This will require hyperparameter optimization. To do this we must find the optimal value for alpha such that gradient descent converges fast. The value we select for alpha has to be just right so we can learn fast, but not so fast that we miss the minimum. We can completely overshoot the minimum, thus increasing the error of the model.

Another way to think of it is that learning will happen slower.īut being too optimistic, or selecting an alpha value that’s too large, is risky too. The mathematical expression for this can be seen below.įigure 2: Example of minimizing J(w) (Source: MLextend)īeing too conservative, or selecting an alpha value that’s too small, is going to take longer to converge at the minimum. We then use those predictions to calculate the loss of the model - or how wrong the model is. To recap: our machine learning model approximates a target function so we can make predictions. An optimization problem seeks to minimize a loss function.” “A function that maps an event or values of one or more variables onto a real number intuitively representing some “cost” associated with the event. Here’s the definition of a loss function from Wikipedia: More formally, the goal of computing the loss function is to quantify the error between the predictions and the actual label. The responsibility of the loss function is to measure the performance of the model. We will refer to our objective function as the loss function. Most people wouldn’t be bothered if you used the terms interchangeably but the math police would probably pull you up - learn more about the differences between objective, cost, and loss functions.

You may also hear people refer to it as an objective function (the most general term for any function that’s being optimized), or cost function. Before we start wandering around, it’s good to know how we’re performing.

Gradient descent algorithm