Choose the Right Weights Iteratively

Differentiate the error

Choosing the right weights directly is too difficult. An alternative approach is to iteratively improve the weights by descending the error function and taking small steps. Each step is in the direction of the greatest downward slope from our current position.

This means that the error function didn’t need to sum all the output nodes in the first place. The reason is that the output of a node only depends on the connected links and hence their weights. This fact is sometimes glossed over, and sometimes the error function is simply stated without an explanation.

Here is our simpler expression:

$$\frac{\partial E}{\partial w_{jk}} = \frac{\partial}{\partial w_{jk}}(t_k - o_k)^2$$