1


LABORATORY MANUAL
ECE 403/503
OPTIMIZATION for MACHINE LEARNING



This manual was prepared by
Wu-Sheng Lu

University of Victoria
Department of Electrical and Computer Engineering






© University of Victoria
May 2020
2

Preparing Your Laboratory Report

The objective of the experiments described in this manual is to familiarize the student
with computer simulations and implementation of several optimization as well as data
processing techniques as they are applied to machine learning problems. The primary
software tool required in all experiments is MATLAB to which the student can access
during the laboratory sessions. An appendix, that introduces main functions in MATLAB
and their usage, is included to facilitate the students in preparing their MATLAB code.

PREPARATION

Successful completion of an experiment depends critically on error-free MATLAB
programming. Therefore, preparation prior to the laboratory period is essential.
Specifically, the student should study the description of the experiment and prepare
useable MATLAB codes required in the preparation section(s) before the experiment is
carried out. The student will be required to present the preparation at the beginning of the
lab session.

THE LABORATORY REPORT

A laboratory report is required from each group for each experiment performed. The lab
report should be submitted within one week after the experiment. The front page of the
report is shown on the next page and should be used for each laboratory report.

The report should be divided into the following parts:
(a) Objectives.
(b) Introduction.
(c) Results including relevant MATLAB programs and figures, and description of the
implementations.
(d) Discussion.
(e) Conclusions.








3

UNIVERSITY OF VICTORIA
Department of Electrical and Computer Engineering

ECE 403/503 Optimization for Machine Learning

LABORATORY REPORT

Experiment No:

Title:

Date of Experiment:
(should be as scheduled)

Report Submitted on:

To:

Name: (please print)
4

Experiment 1
Predicting Fuel Consumption of Automobiles

1. Introduction and objective
In this experiment, we build a linear model to predict city-cycle fuel consumption of
automobiles in miles per gallon (MPG). The database, from which the model in question
learns, was from the StatLib library which is maintained at Carnegie Mellon University.
The dataset was used in the 1983 American Statistical Association Exposition. In
literature, the data set was used by Ross Quinlan in his paper:
R. Quinlan, “Combining instance-based and model-based learning,” in Proc. Tenth
International Conference of Machine Learning, pp. 236-243, Univ. of Massachusetts,
Amherst, 1993.
The data set contains 392 samples, each sample is characterized by a vector x with six
components which are numerical values of six features of a car, including number of
cylinders, displacement, horsepower, weight, acceleration, and model year. Associated
with each sample xp is a scalar output yp representing fuel consumption of the car in MPG.
Clearly, here we are dealing with a data set of the form {(xp, yp), p = 1, 2, …, M}
with 6 1p R
x , py R , and M = 392. The objective of the experiment is to develop a
linear model that predicts fuel consumption y of an “unseen” automobile featured by a
new input vector x.
In this experiment, the above data set was divided into two sets – one for training and the
other for testing. The train dataset includes randomly selected 80% of available data -- to
be exact, P = 314 samples and the associated outputs, while the test dataset includes the
remaining T = 78 samples and their outputs.

2. Background
2.1 Least-squares linear regression for prediction
The linear model to be employed in the experiment is based on least-squares linear
regression which was studied in Sec. 1.3.1, see pages 24-28 of the course notes. Below
we summarize the method where the items involved are explicitly specified in accordance
with the above dataset. The linear model is given by
Ty b  w x (E1.1)
where y R , 6 1R w , 6 1R x , and b R . With train data {( , ), 1, 2,...,314}p py p x
(see Sec. 2.3 below), we construct
1 2 314tr
7 314
ˆ
1 1 1 
    


x x x
X and
1
2
tr
314
y
y
y
       
y (E1.2)
It follows from Eq, (1.30) of the course notes that the optimal weight w* and bias b* for
model (E1.1) are given by
5

   1 †tr tr tr tr tr trˆ ˆ ˆ ˆˆ T T  w X X X y X y (E1.3)
where
   † 1tr tr tr trˆ ˆ ˆ ˆT T X X X X
is the pseudo-inverse of trˆ
TX , and ˆ w is defined by ˆ
b



    
w
w .
Remark: MATLAB provides a function named pinv to calculate pseudo-inverse.

2.2 Predicting the outputs of test data
Using the result of (E1.3), the optimally tuned linear predictor is given by
ˆ ˆT Ty b     w x w x (E1.4)
which produces y as a prediction of the output for an input sample x. In particular, we
are interested in applying (E1.4) to the 78 samples from the test data and compare its
predictions with the “ground truth” outputs in terms of root-mean-squared error (RMSE).

2.3 The data set
The data set assumes the form {( , ), 1, 2,...,314}p py p x where each xp is a six-
dimensional column vector. The meaning of each component of xp is as follows:
x1: number of cylinders, in the range [3, 8].
x2: displacement, in the range [68, 455].
x3: horsepower, in the range [46, 230].
x4: weight, in the range [1613, 5140].
x5: acceleration, in the range [8, 24.8]
x6: model year, in the range [70, 82].
The fuel consumption in miles per gallon, yp, is in the range [9, 46.6].
• To get the data set, go to the course website and download and save file D_mpg.mat
into a folder MATLAB can access. (Warning: make sure you download the data set
that was re-posted on May 11, 2020)
• The data file contains a matrix named D_mpg whose size is 7  392. It is important to
stress that D_mpg has been re-organized from the original data set through a random
selection procedure so that its first 314 columns form the train data while the remaining
78 columns form the test data. The code below uses this data matrix to prepare matrix
trXˆ (which is named in matlab as Xh_tr, see below) and vector try (which is named in
matlab as y_tr, see below) defined in (E1.2):




6

y = D_mpg(7,:)';
M = length(y);
P = 314;
T = M - P;
X = D_mpg(1:6,:);
Xh = [X; ones(1,M)];
Xh_tr = Xh(:,1:P);
y_tr = y(1:P);

and data matrix teXˆ and associated output tey for test purpose:
Xh_te = Xh(:,P+1:M);
y_te = y(P+1:M);

3. Procedure
3.1 From the course website download D_mpg.mat
3.2 Prepare matrix trXˆ and try based on Eq. (E1.2) (refer to the code in Sec. 2.3 above).
3.3 Prepare test data teXˆ and associated output tey (refer to the code in Sec. 2.3 above).
3.4 Prepare MATLAB code to compute optimal parameter ˆ
b



    
w
w for the model in
(E1.1) using Eq. (E1.3).
3.5 Apply the optimized model
ˆ ˆT Ty      w x b w x where ˆ
1
    
x
x
to the train and test data respectively, and evaluate the root-mean-squared error
(RMSE) defined by
  1/2314 2train
1
1 ˆ ˆRMSE
314
T
p p
p
y

     w x (E1.5)
and
  1/278 2test
1
1 ˆ ˆRMSE
78
T
t t
t
y

     w x (E1.6)
where {( , ), 1, 2,...,314}p py p x used in (E1.5) are from train data {Xh_tr,y_tr},
while {( , ), 1, 2,...,78}t ty t x used in (E1.6) are from test data {Xh_te,y_te}, both
were prepared in Sec. 2.3. Report your numerical results in terms of RMSEtrain and
RMSEtest.
3.6 For comparison, in a single figure, plot the “ground truth” output y_te as a curve
colored in blue and its prediction ˆ ˆ{ , 1,2,...,78}T t t
 w x as the second curve colored
in red. Comment on your visual inspection of the figure.

Include your MATLAB code in the lab report.