Chapter 2, Statistical Learning

What is Statistical Learning

Input variables to a statistical model are often called predictors, independent variablesfeatures, or just variables.

Output variable from a statistical model is often called response, or dependent variable.
In essence, statistical learning refers to a set of approaches for estimating f (error term)

Y=f(X)+ε

Why Estimate f

Estimating f is used for prediction and inference.

The prediction accuracy depends on two quantities, reducible error and irreducible error.

The reducible error measure the prediction error as function of X and predicted Y, which by using appropriate statistical learning technique can be reduced.

The irreducible error measure the prediction error as function of ε and the predicted Y. Its irreducible as no matter how well we estimate Y, we cannot reduce the error introduced by ε. The irreducible error may contain unknown variables, variations and variance of the estimated output Y.

The techniques of statistical learning focus on estimating of f with aim of minimizing the reducible error.

The inference task is about understanding the relations between the output and the predictors. Some questions to ask are

  • Which predictors have strong influence on Y
  • What is the relationship between the response and each predictor
  • Is the relation between Y and predictors representable in linear equation or not

Linear models allow for relatively simple and interpretable inference, but may not yield as accurate predictions as some other approaches

Non-Linear models provide quite accurate predictions for Y , but this comes at the expense of a less interpretable model for which inference is more challenging

How F is estimated

The process here is to observe training data and train a statistical model to find Y ≈ f(X). There are two methods to do so parametric and nonparametric.

Parametric methods involve two step model based approach

  • Define and select a linear model f(X)=WX+b
  • Select a procedure for fitting the train data to the selected model. Example of such procedure is least squares method.

Choosing close model to the true function of f is challenging. Choosing a too far model would result in poor estimation.

Overfitting is a phenomena when the model fits much to the noise in the training data instead of generalizing to capture the underlying f.

Nonparametric methods do not make explicit assumptions about the functional form of f. Instead they seek an estimate close to the training data without being too rough or too wiggly. Advantage of such methods is ability to cover wide range of shapes of f, though this requires training data way more than parametric methods to generalize.

Usually restrictive, simple and linear models are more suitable than flexible models for inference tasks.

Regression problems are about predicting an quantitative response while the classification problems are about predicting a qualitative response.

Model Accuracy Assessment

In regression problems mean square error is commonly used.

screen-shot-2017-01-30-at-7-12-33-pm
Mean Square Error

 

Chapter 1, Introduction

Statistical learning refers to a vast set of tools for understanding data. Types of statistical learning:

  • Supervised statistical learning involves building a statistical model for predicting, or estimating, an output based on one or more inputs
  • Unsupervised statistical learning, there are inputs but no supervising output; nevertheless we can learn relationships and structure from such data
Linear regression is used for predicting quantitative values, such as an individual’s salary
Statistical Learning History:
  • Around 1900, Legendre and Gauss published method of least squares papers (a form of linear regression)
  • In 1936, Fisher proposed linear discriminant analysis for predicting qualitative regression problems.
  • In 1940, various authors proposed logistic regression for solving the regression problem
  • In the early 1970s, Nelder and Wedderburn coined the term generalized linear models for an entire class of statistical learning methods that include both linear and logistic regression as special cases.
  • In 1970, many linear models were available however, fitting a nonlinear relationship was computationally infeasible at that time.
  • In 1980, the computing technology had improved so it can solve nonlinear problems.
  • In mid 1980, Breiman, Friedman, Olshen and Stone introduced classification and regression trees, and were among the first to demonstrate the power of a detailed practical implementation of a method, including cross-validation for model selection
  • In 1986, Hastie and Tibshirani coined the term generalized additive models for a class of nonlinear extensions to generalized linear models
  • In the past years, progress on statistical learning techniques and tools has been marked by the increasing availability of powerful and relatively user-friendly software

Udacity Deep Learning Course Summary

Introduction

Logistic Classifier (Linear Classifier): Takes an input (image pixels) and applies a linear function to them to make predictions.

Linear Function: WX + b = Y, where W is weight matrix, X is input matrix, b is bias and Y is output matrix. The W and b terms are the trained values.

Softmax: is a way to transform scores into probabilities. Increasing the number of training data set would make the classifier more confident and discriminate individual classes clearly.

One-Hot Encoding is when the correct class has probability of 1.0 and other classes 0 probability. This technique is not efficient on large number of classes. The benefit is measuring cross-entropy is easy.

Multinomial Logistic Classification is setting where input is feeded to linear model to logistic to softmax to one-hot encoding.

Gradient Descent is the technique used for finding the value to minimize the loss function. A correct simulation should update Theta0 and Theta1 at same time not after each other.

Learning Rate is variable that controls how big a step the gradient descent takes downhill.

For a Local Minimum the slope is going to be zero and this will result in 0 for derivative term which would result in same value of theta.

Training Dataset is the data set that the classifier uses to train on.

Validation Dataset is the data set used for validation and also it’s possible to reconstruct this dataset until the classifier looks like generalized.

Test Dataset is the data set used for real testing. Data samples that the classifier never saw before.

Rule of 30 is an idea to measure if improvements in the classification model is significant or not.

Usually models use more 30K examples for validation to assure significance. Whenever there’s an improvements more than 0.1% this will be significant, which makes measuring the improvements easy.

The main problem with Gradient Descent is that it’s gets calculated for each example in the training data set, and the derivative term takes long time from processing standpoint. This problem will arise in scalability scenarios where the training data set is huge.

To overcome the scalability and performance issues of gradient descent, using a small random data (to calculate its average loss) set out of the training data set with small learning rate overcomes this problem. That technique is called Stochastic Gradient Descent (SAG).

Use running average of the loss instead of relying solely on the current instance helps reducing the number of steps to converge. This is called momentum technique

Using lower learning rate usually tends to get a more accurate classifier.

Stochastic Gradient Descent Parameters:

  • Initial Learning Rate: the initial value for learning rate
  • Learning Rate Decay: how much learning rate should increase
  • Momentum: the running average of the loss
  • Batch Size: the random sample used by SGD
  • Weight Initialization

ADAGRAD is a technique that eliminates couple of SGD parameters and require batch size and weights only.

Deep Neural Networks

Rectified Linear Units (RELU): non-linear function that’s y=x for y > 0 and 0 otherwise.

Backpropagation is used to calculate the gradient of the cost function to update the weights.

Deep Learning is about adding more hidden layers to the model. This helps because of:

  • Parameter efficiency: uses few parameters but goes deeper.
  • Model Abstraction: going deeper results into layers that learn more generalized and useful shapes, like geometric shape instead of line

Why deep learning has not been used before?

  • Deep learning, requires large dataset which has been available recently
  • Better regularization techniques

Regularization is a technique used to prevent from overfitting

Avoid overfitting:

  • Early termination for the training using validation set as soon as the performance stops improving
  • Using regularization techniques
    • Adding L2 for the weights vector
    • Using Dropout

Dropout (by Geoffrey Hinton) is a regularization technique used to prevent overfitting by stochastically turning off activations for hidden layers. This forces the network not to rely on a specific activation in its training and it learns redundant representation for everything across the input data.

Redundant representations results in a network that need some consensus technique to decide the result.

Convolutional Neural Network

If the structure of the input is known, then there’s a chance to reduce the NN complexity by relying on such fact. For example, if the input is a colored letter then the NN can be simplified by using grayscale instead of adding the colors complexity to the NN.

Features that do not change across time, space or everywhere are called statistical invariant. For example, translation invariance is used to simplify the NN by disregard learning the location of the object in an image.

Weight sharing is used to simplifying learning in a NN by utilizing statistical invariance.

CNN (by Yann LeCun) are NN that share their parameters across the space.

Convolution is the process of applying a mask to a NN in convolution pattern (like image filters) to a given image. This would result into a matrix that has different width, height and depth from the original image.

The layer’s parameters consist of a set of learnable filters (or kernels/patches), which have a small receptive field, but extend through the full depth of the input volume.

An RGB image has three feature maps each corresponds to one color intensity.

Stride is the number of pixels shifted each time the filter is moved

Valid padding is way to cut the image as the size of the filter

Same padding is way to pad zeros at the edge of the image to keep sizes same.

Striding reduces the feature map size but it’s very aggressive and sometimes result in loss of data.  Pooling is a technique used for combining information for an image sample. Two common pooling techniques are max pooling and average pooling.

1×1 convolution is a way to reduce the dimensionality of the NN in a simple way.

Recurrent Neural Network is a NN that’s used with temporal classification/prediction problems.

Long Short Term Memory (LSTM) is a version from RNN that’s being used widely.

Writing a Book Review

Goal/Outcome of the book review

It’s a reaction paper in which strengths and weaknesses of the material are analyzed. It should include a statement of what the author has tried to do, evaluates how well (in the opinion of the reviewer) the author has succeeded, and presents evidence to support this evaluation

What is a book review?

A review is a critical evaluation of a text, event, object, or phenomenon. Reviews can consider books, articles, entire genres or many fields.

Above all, a review makes an argument. The most important element of a review is that it is a commentary, not merely a summary. Typically, reviews are brief. In newspapers and academic journals, they rarely exceed 1000 words

How to write a book review?

In general, a critical assessment, a book review should focus on opinions, not facts and details. Summary should be kept to a minimum, and specific details should serve to illustrate arguments

  • Write a statement giving essential information about the book: title, author, first copyright date, type of book, general subject matter, special features (maps, color plates, etc.), price and ISBN
  • State the author’s purpose in writing the book. If the author didn’t specify ask yourself these questions:
    • Why did the author write on this subject rather than on some other subject?
    • From what point of view is the work written?
    • Was the author trying to give information, to explain something technical, to convince the reader of a belief’s validity by dramatizing it in action?
    • What is the general field or genre, and how does the book fit into it? (Use outside sources to familiarize yourself with the field, if necessary.) Knowledge of the genre means understanding the art form and how it functions.
    • Who is the intended audience?
    • What is the author’s style? Is it formal or informal? Evaluate the quality of the writing style by using some of the following standards: coherence, clarity, originality, forcefulness, correct use of technical words, conciseness, fullness of development, fluidity. Does it suit the intended audience?
    • Scan the Table of Contents, it can help understand how the book is organized and will aid in determining the author’s main ideas and how they are developed – chronologically, topically, etc.
    • How did the book affect you? Were any previous ideas you had on the subject changed, abandoned, or reinforced due to this book? How is the book related to your own course or personal agenda? What personal experiences you’ve had relate to the subject?
    • How well has the book achieved its goal?
    • Would you recommend this book or article to others? Why?
  • State the theme and thesis of the book
    • Theme: The theme is the subject or topic. It is not necessarily the title, and it is usually not expressed in a complete sentence. It expresses a specific phase of the general subject matter
    • Thesis: The thesis is an author’s generalization about the theme, the author’s beliefs about something important, the book’s philosophical conclusion, or the proposition the author means to prove. Express it without metaphor or other figurative language, in one declarative sentence.
  • Explain the method of development-the way the author supports the thesis
    Illustrate your remarks with specific references and quotations. In general, authors tend to use the following methods:
    • Description: The author presents word-pictures of scenes and events by giving specific details that appeal to the five senses, or to the reader’s imagination
    • Narration: The author tells the story of a series of events, usually presented in chronological order
    • Exposition: The author uses explanation and analysis to present a subject or to clarify an idea. Exposition presents the facts about a subject or an issue as clearly and impartially as possible
    • Argument: The author uses the techniques of persuasion to establish the truth of a statement or to convince the reader of its falsity
  • Evaluate the book by:
    • Show whether the author’s main arguments are true
    • Respond to the author’s opinions
    • What do you agree or disagree with? And why?
    • Illustrate whether or not any conclusions drawn are derived logically from the evidence
    • What possibilities does the book suggest?
    • What has the author omitted or what problems were left unsolved?
    • What specific points are not convincing?
    • Compare it with other books on similar subjects or other books by the same as well as different authors.
    • Is it only a reworking of earlier books; a refutation of previous positions?
    • Have newly uncovered sources justified a new approach by the author?
    • Comment on parts of particular interest, and point out anything that seems to give the book literary merit.
    • Relate the book to larger issues.
  • Try to find further information about the author – reputation, qualifications, influences, biographical. IOW, can you discern any connections between the author’s philosophy, life experience and the reviewed book?
  • If relevant, make note of the book’s format – layout, binding, typography, etc
  • Check back matters
    • Is the index accurate?
    • Do footnotes provide important additional information or just redundancy? Are they relevant?
  • Summarize (briefly), analyze, and comment on the book’s content
    • State your general conclusions.
    • Is author’s summary and conclusion is convincing?
    • List the principal topics, and briefly summarize the author’s ideas about these topics, main points, and conclusions (Use specific references and quotations to support your statements)
    • Give recommendation on what’s missing from the book and you feel if it was added that’d be great!
    • Same as above point but what can be taken out to make the book more concise and focused.

When reviewing History and other Nonfiction, ask these questions

  • With what particular subject or period does the book deal?
  • How thorough is the treatment?
  • What were the sources used?
  • Is the account given in broad outline or in detail?
  • Is the style that of reportorial writing, or is there an effort at interpretive writing?
  • What is the point of view or thesis of the author?
  • Is the treatment superficial or profound?
  • For what group is the book intended (textbook, popular, scholarly, etc.)?
  • What part does biographical writing play in the book?
  • Is social history or political history emphasized?
  • Are dates used extensively, and if so, are they used intelligently?
  • Is the book a revision? How does it compare with earlier editions?
  • Are maps, illustrations, charts, etc. used and how are these to be evaluated?

Introducing Website Diagnostic cmdlets in Windows Azure PowerShell

Recently Windows Azure PowerShell has the support of diagnosing azure websites from command line. The experience is pretty smooth and straightforward, let’s get into it step by step:

Download and install Windows Azure PowerShell installer and open it. The article assumes you have already created free trial account.

Download your Windows Azure publish settings file and import it and yes, this’s the most painful thing you’ll do here I promise!

image

 

Create new website called wawsrocks

image

Now we have a website created, to make sure it works fine use Show-AzureWebsite cmdlet like this

image

 

image

To enable the diagnostics on file system just execute Enable-AzureWebsiteApplicationDiagnostic (woof what a long name!). Then you make sure it’s enabled using Get-AzureWebsite cmdlet

image

I’ve picked Verbose logging level among the available options: Information, Error, Verbose and Warning.

At this moment, I can start enjoy watching everything happening on the website by executing Get-AzureWebsiteLog –Tail cmdlet!

image

The logs are saved on the the file system of the website and besides that you can change the behavior to make the logs go to Windows Azure Storage Table just by executing these cmdlets

image

The New-AzureStorageAcccount creates a storage account that will have the logs in a table called WAWSAppLogTable. In the future releases Windows Azure PowerShell will support a cmdlet to query the log entries and filter them.

In case you want to break up with application logging on file, execute this cmdlet

image

If you are so angry :@%#$Q you can also remove the website and the storage account

image

 

The complete listing of the cmdlets used to make this sample is found in this gist.

At the end, I hope you have happy logging weekend!

The Laws of Motion

Game development community call point masses with particles.

The Particle

A particle has a position, but no orientation. In other words, we can’t tell what direction a particle is pointing: it either doesn’t matter or it doesn’t make sense.

The First Two Laws

There are three laws of motion put forward by Newton; The first two laws of motion follow:

  1. An object continues with a constant velocity unless a force acts upon it.
  2. A force acting on an object produces acceleration that is proportional to the object’s mass.

The First Law

The first law tells us what happens if there are no forces around. The object will continue to move with a constant velocity. In the real world, we can’t get away from forces acting on a body; the closest phenomenon that we can imagine is the movement of objects in space.

In a physics engine we could simply assume that there are no forces at work and use Newton-1 directly. To simulate drag, we could add special drag forces. This is fine for the simple engine, but can cause problems with more complex systems. The problem arises because the processor that performs the physics calculations isn’t completely accurate. This inaccuracy can lead to objects getting faster of their own accord.

A better solution is to incorporate a rough approximation of drag directly into the engine. If we need complicated drag (such as aerodynamic drag in a flight simulator or racing game) we can still dot hat the long way, by creating a special drag force.

The Second Law

The second law tells us how forces alter the motion of an object. A force is something that changes the acceleration of an object (i.e., the rate of change of velocity).

The Force Equations

The formula relating the force to the acceleration is the famous

image

The vector form of that is given by

image

Adding Mass to Particles

It is often useful, however, to simulate infinite masses. These are objects that no force of any magnitude can move. They might be just as physically impossible, but they are very useful for immovable objects in a game: the walls or floor, for example, cannot be moved. If we feed an infinite mass into the force equation, then the acceleration will be zero, as we want. As long as such an object has zero initial velocity, it will always stay in the same place.

Unfortunately, we cannot represent a true infinity in most computer languages, and the optimized mathematics instructions on all common game processors do not cope well with infinities.

Note that in last equation we use 1 over the mass to calculate our acceleration. we can speed up our calculations by storing 1 over the mass. We call this the inverse mass. This solves our problem for representing objects of zero or infinite mass: infinite mass objects have a zero inverse mass, which is easy to set. Objects of zero mass would have an infinite inverse mass,which cannot be specified in most programming languages.

Momentum and Velocity

Momentum is the product of velocity and mass. Since mass is normally constant, we can assume that velocity is therefore constant by Newton-1. In the event that a traveling object is changing mass, then its velocity would also be changing, even with no force.

This will be an important distinction when we consider rotations later, however, because rotating objects can change the way their mass is distributed. Under the rotational form of Newton-1, that means a change in rotational speed with no other forces acting.

The Force of Gravity

The formula Newton developed is called the law of universal gravitation:

image

where m1 and m2 are the masses of the two objects, r is the distance between their centers, f is the resulting force, and G is the “universal gravitational constant,” a scaling factor derived from observation of planetary motion.

Because we are only interested in the pull of the Earth, we can simplify the previous equation. First, we can assume that m1 is always constant (i.e., the Earth doesn’t change mass). Second, and less obviously, we can assume that r is also constant. This is due to the huge distances involved. The distance from the surface of the Earth to its center is so huge (6400 km) that there is almost no difference in gravity between standing at sea level and standing on the top of a mountain. For the accuracy we need in a game, we can therefore assume the r parameter is constant. With these assumptions, the equation is simplified to:

image

where m is the mass of the object we are simulating, f is the force, as before, and g is a constant that includes the universal gravitational constant, the mass of the Earth,and its radius:

image

Creating simulations with a g value of 10 ms^−2 can look dull and insipid. Most developers use higher values, from around 15 ms^−2 for shooters (to avoid projectiles having arcs that are too curved) up to 20 ms^−2 typical of driving games. Some developers go further and incorporate the facility to tune the g value on an object-by-object basis.

The acceleration due to gravity can therefore be represented as a vector with the following form:

image

The Integrator

The integrator consists of two parts—one to update the position of the object, and the other to update its velocity. The position will depend on the velocity and acceleration, while the velocity will depend only on the acceleration.

The Mathematics of Particles

Vectors

There are many different kinds of vector spaces with wildly different properties, but for our purposes the only vector spaces we’re interested in are regular (called Euclidean) 2D and 3D space.

Some of the work in a slightly different way to scalar values, and some operations that make sense for scalars (such as division) aren’t defined for vectors.

The Handedness of Space

You can tell which is which using your hands: make a gun shape with your hand,thumb and extended forefinger at right angles to one another. Then, keeping your ring finger and pinky curled up, extend your middle finger so that it is at right angles to the first two. If you label your fingers with the axes in order (thumb is X, forefinger Y, and middle finger Z), then you have a complete set of axes, whether right- or left-handed.

DirectX is left-handed and OpenGL is right handed.

Vectors and Directions

A change in position, given as a vector, can be split into two elements:

image

1- Magnitude (d): the straight-line distance of the change.

2- Unit vector (n): the direction of the change. The vector n represents a change, whose straight-line distance is always 1, in the same direction as the vector a.

We can find d using the 3D version of Pythagoras’s theorem, which has the formula,

image

We can find unit vector by this equation

image

The process of finding just the direction n from a vector is called “normalizing”.

The result of decomposing a vector into its two components is sometimes called
the normal form of the vector”.

Multiplying Vectors

In algebra for scalar values, there is only one kind of multiplication. We write this in various ways, either with no symbol at all (ab), with a dot (a . b), or with a multiplication symbol (a x b).

With vectors these three notations have different meanings, and we have to be more precise. Using no symbol usually denotes a type of multiplication (vector direct product) is not used commonly. The two other notations that we will encounter are called the scalar product (a . b) and the vector product (a x b). First, however, we’ll meet a fourth way of multiplying vectors that uses none of these symbols.

The Component Product

The most obvious product is the least useful. It is used in several places in a physics engine, but despite being quite obvious, it is rarely mentioned in books on vector mathematics. This is because it doesn’t have a simple geometric interpretation—if the two vectors being multiplied together represent positions, then it isn’t clear geometrically how their component product is related to their locations. This isn’t true of the other types of product, as we’ll see.

The Scalar Products (Dot Product)

The dot product is calculated with the following formula

image

The Trigonometry of the Scalar Product

The dot product relates the scalar product to the length of the two vectors and the angle between them:

image

So if we have two normalized vectors then the angle between them is given by Equation

image

If a and b are just regular vectors then the angle would be given by

image

The Geometry of the Scalar Product

If one vector is not normalized, then the size of the scalar product is multiplied by its length. In most cases at least one vector, and often both, will be normalized before performing a scalar product.

The range of a dot product is between 1 and –1 where 1 means that two vectors are identical and –1 means they are totally in opposite direction.

The dot product is most likely be as part of a calculation that needs to find how much one vector lies in the direction of another.

The Vector Product (Cross Product)

The cross product is calculated with this formula

image

The Trigonometry of the Vector Product

Below is the trigonometric equation

image

Commutativity of the Vector Product

You may have noted in the derivation of the vector product that it is not commutative.
In other words, a x b = b x a.

In fact, by comparing the components in the previous equation, we can see that

image

The Geometry of the Vector Product

For a pair of vectors normal of a and b, the magnitude of the vector product represents the component of b that is not in the direction of normal of a.

Because it is easier to calculate the scalar product than the vector product, if we need to know the component of a vector not in the direction of another vector, we are better performing the scalar product and then using the Pythagoras theorem to give the result,

image

where c is the component of b not in the direction of a, and s is the scalar product normal of a . b.

In fact, the vector product is very important geometrically not for its magnitude, but for its direction. In three dimensions, the vector product will point in a direction that is at right angles (i.e., 90 degrees, also called orthogonal) to both of its operands

image

This interpretation shows us an important feature of the vector product: it is only defined in three dimensions. In two dimensions, there is no possible vector at right angles to two nonparallel vectors.

The Orthonormal Basis

In some cases we want to construct a triple of mutually orthogonal vectors, where each vector is at right angles to the other two. Typically we want each of the three vectors to be normalized. This kind of triple vector that is both orthogonal and normalized is called an orthonormal basis.

There are a few ways of doing this. The simplest is to use the cross-product to generate the orthogonal vectors.

The process starts with two nonparallel vectors. The first of these two will not have its direction changed at all: call this a.We cannot change its direction during the process, but if it is not normalized, we will change its magnitude. The other vector, b, may not already be at right angles to a, so it may need to have its direction as well as magnitude changed. One constraint on vector b, however, is that it must not be parallel to vector a. If it is parallel, then we cannot find a unique third vector that is at right angles to both a and b—there are an infinite number of such vectors. The third vector, c, is not given at all, as it is determined entirely from the first two. The algorithm proceeds as follows:

  1. Normalize the starting vector a.
  2. Find vector c by performing the cross-product ca x b.
  3. If vector c has a zero magnitude, then give up: this means that a and b are parallel.
  4. Normalize vector c.
  5. Now we need to ensure that a and b are at right angles to one another. We can do this by recalculating b based on a and c using the cross-product, b = c x a (note the order). The resulting vector b must already be unit length, because both c and a were and we know these are orthogonal.

Calculus

There are two ways of understanding changing quantities: we describe the change itself, or we describe the results of the change. If an object is changing position with time, we need to be able to understand how it is changing position (i.e., its speed, the direction it is moving in, whether it is accelerating or slowing), and the effects of the change (i.e.,where it will be when we come to render it at the next frame of the game).

These two viewpoints are represented by the differential and integral calculus, respectively.

Differential Calculus

we are interested in the rate a quantity is changing with respect to time. This is sometimes informally called its “speed,” but that term is ambiguous. We will call it by the more specific term, “velocity.”

Velocity

it’s computed from this formula

image

where v is the velocity of the object, p’ and p are its positions at the first and second measurements. and delta t is the time that has passed between the two.

If we want to calculate the exact velocity of an object, we could reduce the gap between the first and second measurement. In mathematics, this is written using “limit” notation, as in

image

Rather than use this limit notation, this is more commonly written with a lowercase “d” in place of the delta:

image

Because it is so common in mechanics to be talking about the change with respect to time, this is often simplified even further:

image

Acceleration

This is the change of velocity over time. A positive value for acceleration represents speeding up, a zero acceleration means no change in velocity at all, and negative acceleration represents slowing down. Below is the acceleration formula

image

velocity is the first differential of position, and if we differentiate again we get acceleration, so acceleration is the second differential. Mathematicians often write it in this way:

image

Again the short term for this is

image

Vector Differential Calculus

So far we’ve looked at differentiation purely in terms of a single scalar quantity. For full 3D physics, we need to deal with vector positions rather than scalars. That’s how velocity is represented in 3D

image

And that’s how acceleration is represented:

image

Velocity, Direction and Speed:

The velocity of an object, as we’ve seen, is a vector giving the rate that its position is changing.

The speed of an object is the magnitude of this velocity vector, irrespective of the direction it is moving in.

By decomposing the velocity vector, we can get the speed and the direction of movement:

image

where s is the speed of the object, and image is its direction of movement.

Using the equations for the magnitude and direction of any vector, the speed is given by:

image

and direction by

image

Integral Calculus

In the same way that we obtained velocity from the position using differentiation, we go the other way in integration. If we know the velocity, then we integrate to work out the position at some point in the future. If we know the acceleration, we can find the velocity at any point in time.

If we know that an object is moving with a constant velocity (i.e., no acceleration), and we know this velocity along with how much time has passed, we can update the position of the object using the formula:

image

where image is the constant velocity of the object over the whole time interval.

In the same way, we could update the object’s velocity in terms of its acceleration using the formula:

image

Same could be given in terms of acceleration

image

In game development, integrate means to perform the position or velocity updates.

Vector Integral Calculus

The previous equations can be expressed in vectors like this:

image

and perform the calculation on a component-by-component basis:

image