Jan 9, 2016

[ML] Debugging and choosing learning rate ⍺

Debugging:
J(ϴ) should decrease after each iteration of running the training set.
i.e
    by checking the J(ϴ) should decrease after each iteration to see if the
    algorithm is correct.

Experience:
    Rather rely on graph to see the convergence instead of counting on
    automatic convergence test.




If with each iteration, the J(ϴ) is increasing, which possibly the learing rate
⍺ is too large, which over shoots.

Remember:
ϴJ = ϴi + ⍺*deferential(J(ϴ))

Choosing the learning rate ⍺:
If ⍺ is too small: slow convergence.
If ⍺ is too large: J(ϴ) may not decrease after each iteration, or might not
    converge. Slow convergence also possible.
Plot the graph to debug.
Choose the ⍺ with 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, ...
    increase the learning rate of 3 folds.




No comments:

Post a Comment

Note: Only a member of this blog may post a comment.