FIT5021 - Exam Practice Questions
Dr Mario Boley and Dr Bohan Zhang
Monash University, Faculty of IT
S2-2023
Relation of this collection to the final exam:The collection of questions below is not
equivalent to an exam. Your final exam contains questions worth a total of 60 marks that are
distributed across the six modules of the unit (not necessarily equally). The total time to complete
the exam is 130 minutes and you are expected to complete all questions. Hence, as a rule of thumb,
you should aim to invest no more minutes into a question than two times its marks. For example,
a question with 10 marks is designed to be completed in about 20 minutes. This approach leaves
you an additional 10 minutes as buffer time for the overall exam. The exam is closed book.
How to use this collection to practice:It is fundamentally important that you attempt
every question alone without help before you refer to the teaching material, discuss with your peers,
or even look at the separately published reference solutions. If you are like most students (and
teachers) this approach will be scary because it requires you to confront your gaps in knowledge.
However, this is exactly why it will lead you to a good exam performance (very much in contrast
to just scanning the solutions). It gives yourself the opportunity to really understand what you do
not understand before looking for help. This way, once you receive the information to fill the gap,
you will connect it better with the other things you know, causing it to be available to you under
exam conditions.
Elements of Statistical Learning
1 Classification versus Regression [3 marks]
What is the statistical supervised learning problem on a high level and what is difference between
a classification and a regression problem? Answer in two to three sentences.
2 Goal of Supervised Learning [3 marks]
What is the ultimate goal of statistical supervised learning? How are the concepts of training error
and test error related to this goal? Answer in three to four sentences.
3 Model Selection [6 marks]
Consider the following scenario:
1
•Alice wants to model the species (‘setosa’, ‘versicolor’, or ‘virginica’) of iris flowers as a function
of four variables (their sepal and petal length and width)
•She has collected a dataset of 150 examples
•She wants to use thekNN classifier but she does not know what is a suitable value fork,
hence shes want to choosekfrom the set of candidates{1,2,...,10}based on the collected
data
•Finally, she wants a reliable estimate of the performance of the model that has been learned.
Currently Alice plans to proceed with the following machine learning workflow:
1. Split the data into 10 folds of roughly equal size.
2. Pick the value ofkwith the second best average test error across all folds (for each using the
remaining folds as training data). In particular, she plans to use the second best “to avoid
overfitting”.
3. Use this test error as the final performance estimate.
Answer the following two questions:
(a) Point out in up to two sentences, what is the most substantial problem with Alice’s proposed
workflow and why.
(b) Describe an improved machine learning process that adequately addresses this problem.
4 Normal Distribution and Maximum Likelihood Estimation [10 marks]
The (uni-variate) normal distribution is an extremely important distribution describing the be-
haviour of continuous random variables. It is parameterised by a meanμand a standard deviation
parametersσ(or more typically by the corresponding variance parameterσ
2
). Given a dataset
{x
1
,...,x
N
}of independent realisations of a normal random variableX, we can use the principal
of maximum likelihood to find guesses for the unknown parameters. In particular, these guesses
have simple closed form solutions.
Answer each of the following questions with one to two sentences and give mathematical deriva-
tions as appropriate.
(a) What is the definition of the normal density functionp(x|μ,σ)? What is the key component
of the definition that gives rise to the characteristic bell shape?
(b) What is the key idea of the maximum likelihood estimation of the parametersμandσ, i.e.,
what is the defining property of the maximum likelihood estimatesμ
ML
andσ
ML
.
(c) How can we derive the closed form solution of the maximum likelihood estimation for the
meanμ? Apply this approach to derive it.
(d) How can we derive the closed form solution of the maximum likelihood estimation for the
standard deviationσ? Apply this approach to derive it.
2
Linear Regression
5 Derivation of Squared Error [6 marks]
For fitting the model parameterswof a linear regression model, we used the approach to minimise
the squared error
E(w) =
1
2
N
X
n=1
(t
n
−y
n
)
2
,
where{(x
1
,t
1
),...,(x
N
,t
n
)}is the given training data andy
n
=
P
p
i=1
w
i
φ
i
(x
n
) are the model
predictions. To justify this error function, we showed that it can be derived as a maximum likelihood
parameter estimation for a probabilistic modelp(t|x,w).
Answer each of the following questions with one to two sentences (including mathematical
equations as appropriate).
(a) What is the form of the probabilistic model that we assumed for the regression problem, i.e.,
how are the target values generated given the input vectors?
(b) What is the likelihood function corresponding to this model?
(c) Why is maximising this likelihood function equivalent to minimising the squared error func-
tion?
Linear Classification
6 Logistic Regression [8 marks]
When using the logistic regression model for binary classification, we model the probability of the
positive class (t= 1) given inputxvia the sigmoid transformationσof a linear functionw·xof
model parametersw.
(a) Give the log likelihood function logp(t|x,w) of the logistic regression model for a single data
point (x,t).Hint: We used the fact that we encode the positive class witht= 1and the
negative class witht= 0to give a compact formula.
(b) As a step towards the gradient descent algorithm for logistic regression, derive the partial
derivative of the negative log likelihood (error function)−logp(t|x,w) with respect to pa-
rameterw
i
. Derive the result in individual steps, noting what results you are using (all correct
steps give partial marks).
(c) Extend your result from part (b) to the full gradient of the negative log likelihood when
observing a set ofntraining data points{(x
1
,t
1
),...,(x
N
,t
n
)}.
Latent Variable Models
7 Document Clustering Model [9 marks]
Suppose we are given a collection of documentsD. The data setDis represented as{x
1
,x
2
,x
3
,...,x
N
}
wherex
i
is ad-dimensional “count vector” representing thei-th document, based on bag-of-words
3
and with respect to a word vocabulary of sized. We are interested in fitting a Mixture multinomial
model onto this dataset.
(a) An individual cluster is described by a vector of word occurrence probabilitiesμwhereμ
j
describes the probability of a word in a document to be thej-th word in the vocabulary. Give
a formula of the probabilityp(x|μ) of a count vector x given word occurrence probabilitiesμ
and give a brief explanation of the formula (one to two sentences). Hint: remember that, for
simplicity, we assumed the individual counts to be independent.
(b) Write down the “Q-function”, which is the basis of the Expectation-Maximization (EM)
algorithm for maximizing the log-likelihood. Notice that you do not need to write the EM
algorithm in this part.
(c) Write down the “hard” as well as the ”soft” Expectation-Maximization (EM) algorithm for
estimating the parameters of the model. If necessary, provide enough explanation to under-
stand the algorithm that you have written. Also briefly explain what is the main difference
between hard and soft EM.
Neural Networks
8Forward and Backward Propagation (9 marks)
Given a neural networkf(·) and a datasetD={(x
1
,y
1
),(x
2
,y
2
),...,(x
n
,y
n
)}wherex
i
is a 2-
dimensional vector andy
i
is a scalar value which represents the target.{w
1
,w
2
,...,w
n
}are learnable
parameters.hrepresents a linear unit. For examplet
i
=h
1
w
7
+h
2
w
8
.The error function for training
this neural network is the sum of squared error
E(w) =
1
2
N
X
n=1
(y
i
−t
i
)
2
,
푥
!
푥
"
ℎ
!
ℎ
"
푡
푤
!
푤
"
푤
#
푤
$
푤
%
푤
&
푥
$
푤
'
푤
(
4
(a) Suppose we have a samplex, wherex
1
=0.5,x
2
=0.6,x
3
=0.7. The network parameters are
w
1
=2,w
2
=3
w
3
=2,w
4
=1.5
w
5
=3,w
6
=4
w
7
=6,w
8
=3
Next, let’s suppose the target value y for this example is 4. Write down the forward steps
and the prediction error for this given sample.Hint: you need to write down the detailed
computational steps.
(b) Given the prediction error in the previous question, calculate the gradient ofw
1
, namely
∂E
∂w
1
Please also write down all involved derivatives.
Non-parametric Models
9 Non-parametric Modelling (4 marks)
Answer the following questions in around four sentences: What are non-parametric models and
why are they useful? In particular:
(a) Give at least one example for a non-parametric and one example for a parametric model.
(b) Point out a strength of non-parametric models in terms of the theory of machine learning.
10 Gaussian Process Regression (10 marks)
(a) Provide a mathematical definition of the general Gaussian process regression model, i.e., the
assumed conditional distribution of the observation vectort|X,σ
2
given the input matrix
and noise variance. Make sure to make clear how the model depends on the choice of a kernel
functionk.
(b) Describe a practically efficient and theoretically sound method to choose a suitable kernel
functionk. Describe the high-level idea first in two sentences. Then give one or two key
formulas that can be used to practically implement the method.
Hint: Recall that the sum of two normally distributed random vectors follows again a multi-
variate normal distribution.
(c) Explain in three to four sentences how to conceptually perform predictions and uncertainty
estimation with a Gaussian process regression model. Make sure to mention the general key
idea of prediction and inference with Bayesian models and as well as the idea of how to carry
out the idea specifically for Gaussian process regression. Finally, provide a detailed formula
that can be used to practically compute the required quantities to make a prediction with
uncertainty quantifaction at a test inputx
∗
given the training dataXandt(and all model
choices, i.e., kernel function and noise variance).
Hint: Recall that for a normal random vectorz∼N
k+l
partitioned intoz
1
= (z
1
,...,z
k
)and
z
2
= (z
k+1
,...,z
k+l
), the conditional distribution ofz
2
givenz
1
is normal with
5
–mean vectorμ
2
+Σ
2,1
Σ
−1
1,1
(x
1
−μ
1
)and
–covariance matrixΣ
2,2
−Σ
2,1
Σ
−1
1,1
Σ
1,2
with the partitioned meanμ= (μ
1
,μ
2
)and covariance matrixΣ= [Σ
1
,Σ
2
;Σ
3
,Σ
4
].
6
