1

LABORATORY MANUAL

ECE 403/503

OPTIMIZATION for MACHINE LEARNING

This manual was prepared by

Wu-Sheng Lu

University of Victoria

Department of Electrical and Computer Engineering

© University of Victoria

May 2020

2

Preparing Your Laboratory Report

The objective of the experiments described in this manual is to familiarize the student

with computer simulations and implementation of several optimization as well as data

processing techniques as they are applied to machine learning problems. The primary

software tool required in all experiments is MATLAB to which the student can access

during the laboratory sessions. An appendix, that introduces main functions in MATLAB

and their usage, is included to facilitate the students in preparing their MATLAB code.

PREPARATION

Successful completion of an experiment depends critically on error-free MATLAB

programming. Therefore, preparation prior to the laboratory period is essential.

Specifically, the student should study the description of the experiment and prepare

useable MATLAB codes required in the preparation section(s) before the experiment is

carried out. The student will be required to present the preparation at the beginning of the

lab session.

THE LABORATORY REPORT

A laboratory report is required from each group for each experiment performed. The lab

report should be submitted within one week after the experiment. The front page of the

report is shown on the next page and should be used for each laboratory report.

The report should be divided into the following parts:

(a) Objectives.

(b) Introduction.

(c) Results including relevant MATLAB programs and figures, and description of the

implementations.

(d) Discussion.

(e) Conclusions.

3

UNIVERSITY OF VICTORIA

Department of Electrical and Computer Engineering

ECE 403/503 Optimization for Machine Learning

LABORATORY REPORT

Experiment No:

Title:

Date of Experiment:

(should be as scheduled)

Report Submitted on:

To:

Name: (please print)

4

Experiment 1

Predicting Fuel Consumption of Automobiles

1. Introduction and objective

In this experiment, we build a linear model to predict city-cycle fuel consumption of

automobiles in miles per gallon (MPG). The database, from which the model in question

learns, was from the StatLib library which is maintained at Carnegie Mellon University.

The dataset was used in the 1983 American Statistical Association Exposition. In

literature, the data set was used by Ross Quinlan in his paper:

R. Quinlan, “Combining instance-based and model-based learning,” in Proc. Tenth

International Conference of Machine Learning, pp. 236-243, Univ. of Massachusetts,

Amherst, 1993.

The data set contains 392 samples, each sample is characterized by a vector x with six

components which are numerical values of six features of a car, including number of

cylinders, displacement, horsepower, weight, acceleration, and model year. Associated

with each sample xp is a scalar output yp representing fuel consumption of the car in MPG.

Clearly, here we are dealing with a data set of the form {(xp, yp), p = 1, 2, …, M}

with 6 1p R

x , py R , and M = 392. The objective of the experiment is to develop a

linear model that predicts fuel consumption y of an “unseen” automobile featured by a

new input vector x.

In this experiment, the above data set was divided into two sets – one for training and the

other for testing. The train dataset includes randomly selected 80% of available data -- to

be exact, P = 314 samples and the associated outputs, while the test dataset includes the

remaining T = 78 samples and their outputs.

2. Background

2.1 Least-squares linear regression for prediction

The linear model to be employed in the experiment is based on least-squares linear

regression which was studied in Sec. 1.3.1, see pages 24-28 of the course notes. Below

we summarize the method where the items involved are explicitly specified in accordance

with the above dataset. The linear model is given by

Ty b w x (E1.1)

where y R , 6 1R w , 6 1R x , and b R . With train data {( , ), 1, 2,...,314}p py p x

(see Sec. 2.3 below), we construct

1 2 314tr

7 314

ˆ

1 1 1

x x x

X and

1

2

tr

314

y

y

y

y (E1.2)

It follows from Eq, (1.30) of the course notes that the optimal weight w* and bias b* for

model (E1.1) are given by

5

1 †tr tr tr tr tr trˆ ˆ ˆ ˆˆ T T w X X X y X y (E1.3)

where

† 1tr tr tr trˆ ˆ ˆ ˆT T X X X X

is the pseudo-inverse of trˆ

TX , and ˆ w is defined by ˆ

b

w

w .

Remark: MATLAB provides a function named pinv to calculate pseudo-inverse.

2.2 Predicting the outputs of test data

Using the result of (E1.3), the optimally tuned linear predictor is given by

ˆ ˆT Ty b w x w x (E1.4)

which produces y as a prediction of the output for an input sample x. In particular, we

are interested in applying (E1.4) to the 78 samples from the test data and compare its

predictions with the “ground truth” outputs in terms of root-mean-squared error (RMSE).

2.3 The data set

The data set assumes the form {( , ), 1, 2,...,314}p py p x where each xp is a six-

dimensional column vector. The meaning of each component of xp is as follows:

x1: number of cylinders, in the range [3, 8].

x2: displacement, in the range [68, 455].

x3: horsepower, in the range [46, 230].

x4: weight, in the range [1613, 5140].

x5: acceleration, in the range [8, 24.8]

x6: model year, in the range [70, 82].

The fuel consumption in miles per gallon, yp, is in the range [9, 46.6].

• To get the data set, go to the course website and download and save file D_mpg.mat

into a folder MATLAB can access. (Warning: make sure you download the data set

that was re-posted on May 11, 2020)

• The data file contains a matrix named D_mpg whose size is 7 392. It is important to

stress that D_mpg has been re-organized from the original data set through a random

selection procedure so that its first 314 columns form the train data while the remaining

78 columns form the test data. The code below uses this data matrix to prepare matrix

trXˆ (which is named in matlab as Xh_tr, see below) and vector try (which is named in

matlab as y_tr, see below) defined in (E1.2):

6

y = D_mpg(7,:)';

M = length(y);

P = 314;

T = M - P;

X = D_mpg(1:6,:);

Xh = [X; ones(1,M)];

Xh_tr = Xh(:,1:P);

y_tr = y(1:P);

and data matrix teXˆ and associated output tey for test purpose:

Xh_te = Xh(:,P+1:M);

y_te = y(P+1:M);

3. Procedure

3.1 From the course website download D_mpg.mat

3.2 Prepare matrix trXˆ and try based on Eq. (E1.2) (refer to the code in Sec. 2.3 above).

3.3 Prepare test data teXˆ and associated output tey (refer to the code in Sec. 2.3 above).

3.4 Prepare MATLAB code to compute optimal parameter ˆ

b

w

w for the model in

(E1.1) using Eq. (E1.3).

3.5 Apply the optimized model

ˆ ˆT Ty w x b w x where ˆ

1

x

x

to the train and test data respectively, and evaluate the root-mean-squared error

(RMSE) defined by

1/2314 2train

1

1 ˆ ˆRMSE

314

T

p p

p

y

w x (E1.5)

and

1/278 2test

1

1 ˆ ˆRMSE

78

T

t t

t

y

w x (E1.6)

where {( , ), 1, 2,...,314}p py p x used in (E1.5) are from train data {Xh_tr,y_tr},

while {( , ), 1, 2,...,78}t ty t x used in (E1.6) are from test data {Xh_te,y_te}, both

were prepared in Sec. 2.3. Report your numerical results in terms of RMSEtrain and

RMSEtest.

3.6 For comparison, in a single figure, plot the “ground truth” output y_te as a curve

colored in blue and its prediction ˆ ˆ{ , 1,2,...,78}T t t

w x as the second curve colored

in red. Comment on your visual inspection of the figure.

Include your MATLAB code in the lab report.