CS 507 - Statistical Learning Assignment 2 Fall 2019
Assignment 2: Gaussians and linear regression
1. (15 points) Mathematical preliminaries: Manipulating Gaussian densities Consider the func-
tion f(x) = 1Z exp(ax
2 + bx+ c), with a < 0, and the Gaussian density g(x) = 1√
2piσ2
exp
(
− 12
(
x−µ
σ
)2)
(a) Find the maximum of log(f(x)) as well its curvature (i.e. second derivative) at the maximum.
(b) Show that, for appropriate choices of a, b, c, and Z, f(x) is equivalent to g(x). [Hint: Multiply out
the square term in the exponent of g(x), and equate the coefficients of f(x) and g(x).] Express µ
and σ in terms of a, b, and c.
2. (25 points) Linear regression [Python]. Suppose you are given training-inputs D = {x1, . . . ,xN},
where each xi is an M × 1-dimensional vector, and outputs {y1, . . . , yN} You want to fit a function of
the form f(x, ω) = ω>x =
∑M
i=1 wixi to this data by minimising the cost function
L(ω,D) =
N∑
n=1
(f(xn, ω)− yn)2 + λ
(
ω>ω
)
(a) By taking the derivatives of L with respect to each ωi and setting them to 0, show that this cost
function is minimised for ωλ =
(∑
n xnxn
> + λIM
)−1
(
∑
n xnyn), where I is an identify matrix of
size M .
(b) Load the variables in the file Assignment2.mat and fit the parameters to the data xTrain and
yTrain (where each row of xTrain corresponds to one data-point), using λ = 0, i.e. no regularisa-
tion. Plot the vector ωo that you obtain.
(c) Train multiple versions of your model on the training-set, using values λ = 1, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000.
Make a plot that shows how the least-squares errors of your model on the training and the test set
changes as a function of λ. [Note: Please make clearly labelled plots with meaningful axes.]
(d) Using this plot, decide which value of λ you expect to give you the best generalization performance,
and report the value. Use this model to predict the y-values for the data xValidation, save your
predictions as a variable named yValidationPredicted in a file Assignment2YourName.mat and
submit this file.
(e) Your friendly instructors included a column of ’ones’ for the input data (i.e. the first entry of each
x is 1). Explain why (for this model), including a constant term in the input-data can be useful
and results in a more flexibel regression model.
(f) By inspecting the vector ω obtained for the best model, make a guess as to which of the dimensions
of x are important for predicting y, and which are irrelevant.