Handwriting a neural net, part 5 - What Did It Cost?
Now that we’ve done the forward pass, we need to see how far we were off.
This is called putting together a cost function.
Let’s say we’re trying to guess a number. Now let’s say our guess was 4, but our result was 8. We could say something like, we were off by 100 percent. Or we could say we were off by 4.
So let’s use the off by 4 example. But what if our guess was 4 but our result was 0? If we did it mathematically the same way would say that we were off by -4. However, whether we’re off by 4 in the positive direction or we’re off by 4 in the negative direction, we’re still 4 units away from our target. So one solution is, maybe we take the absolute value.
Another solution is, what if we squared our subtraction? This would have the added benefit of make errors much more serious if they get bigger because the get exponentially bigger. So let’s do that. We’ll use an error calculation, also called a cost function, that uses the sum of the squares.
So when we’re doing machine learning, to train it, we know what the correct answer is. Let’s say that it’s 4. And let’s say that our machine guessed 8. We would say error = (y* - output)²
So for our example, y* = 4, and output = 8, so we have error = (4 - 8)² = -4², which = 16. And that’s it!
The goal is to get that error value way down low. When we know that the answer is 4, and the computer says the output is also 4, then our cost function will say our error is 0, we know that our machine may have learned something!