CS5487 – Assignment 3 – Course Project
Antoni Chan
Department of Computer Science
City University of Hong Kong
Proposal due date: Fri, Week 10
Presentation date: TBA, Week 14
Report due date: Fri, Week 14
1 Course Project
The final assignment is a student-defined course project. The goal of the project is to get some
hands-on experience using the course material on your own research problems. If you can’t think
of a project, then you can do the “default” project, which is digit classification (see Section 2).
1.1 Project topic
The goal of the project is to get some hands-on experience using the course material on your own
research problems. Keep in mind that there will only be about 4 weeks to do the project, so the
scope should not be too large. Following the major themes of the course, here are some general
topics for the project:
• regression (supervised learning) – use regression methods (e.g. ridge regression, Gaussian
processes) to model data or predict from data.
• classification (supervised learning) – use classification methods (e.g., SVM, BDR, Logistic
Regression) to learn to distinguish between multiple classes given a feature vector.
• clustering (unsupervised learning) – use clustering methods (e.g., K-means, EM, Mean-Shift)
to discover the natural groups in data.
• visualization (unsupervised learning) – use dimensionality reduction methods (e.g., PCA,
kernel-PCA, non-linear embedding) to visualize the structure of high-dimensional data.
You can pick any one of these topics and apply them to your own problem/data. Before actually
doing the project, you need to write a project proposal so that we can make sure the project is
doable within the 3-4 weeks. I can also give you some pointers to relevant methods, if necessary.
• Can my project be my recently submitted or soon-to-be submitted paper? If you plan to just
turn in the results from your paper, then the answer is no. The project cannot be be work
that you have already done. However, your course project can be based on extending your
work. For example, you can try some models introduced in the course on your data/problem.
1
1.2 Project details
• Group project – all projects should have a group of 2 students. To sign up for a group, go
to Canvas⇒ “People” and then join one of the existing “Project Groups”. If you cannot find
a group, please use the Discussion board.
• Project Proposal – For the first part of the project, you need to write a project proposal.
The project proposal should be at most one page with the following contents: 1) an intro-
duction that briefly states the problem; 2) a precise description of what you plan to do –
e.g., What types of features do you plan to use? What algorithms do you plan to use? What
dataset will you use? How will you evaluate your results? How do you define a good outcome
for the project? – The goal of the proposal is to work out, in your head, what your project
will be. Once the proposal is done, it is just a matter of implementation!
• Project Poster Presentation – Project poster presentations will be at the end of the
semester, before the project due date (Week 14). More details will be sent out later. The
poster presentation is optional. However, if you want to get an “A” on your project, then
you must give a presentation. Put it another way, if you don’t give a presentation, then you
will get at most a “B+” on your project.
• Project Report – The project report is essentially the project proposal with all the details
filled in. The report should have the following contents: 1) introduction – what is the problem?
why is important?; 2) methodology – what algorithms did you use and what are the technical
details? what are the advantages and disadvantages?; 3) experimental setup – what data
did you use? how did you pre-process the data? which algorithms did you run on the data?
what is the metric for evaluation?; 4) experimental results – what were the results? what
insight do you get from these results? what are some typical success and failure cases? –
The project report should be at least 4 pages. There is no upper page limit, but probably it
should not be more than 8 pages long. For group projects, the project report must state the
level of contribution from each project member.
• What to hand in – You need to turn in the following things:
1. Project proposal (due Friday, Week 10).
2. Project report (due Friday, Week 14).
3. Presentation poster (due Friday, Week 14).
4. Source code files (due Friday, Week 14).
Only one group member needs to submit the files on Canvas. You must submit your course
project materials using the Canvas website. Go to “Assignments” ⇒ “Course Project” ⇒
select the appropriate entry.
• Third Party Code – In the course project, you may use 3rd party source code, e.g., libsvm,
etc. If you use 3rd party code, you must acknowledge it with an appropriate reference.
• Grading – The marks for this project will be distributed as follows:
– 16.7% – Project proposal.
– 16.7% – Technical correctness (whether you used the algorithms correctly)
2
– 16.7% – Experiments. More points for thoroughness and testing interesting cases (e.g.,
different parameter settings).
– 16.7% – Analysis of the experiments. More points for insightful observations and anal-
ysis.
– 16.7% – Quality of the written report (organized, complete descriptions, etc).
– 16.7% – Project poster presentation.
Note: Here 16.7% means 5/30
3
2 Default Course Project – Digit Classification
The default project is handwritten digit classification on a subset of the MNIST digits dataset.
• Dataset – The provided dataset is a subset of the MNIST digits. The dataset has 10 classes
(digits 0 through 9) with 4000 images (400 images per class). Each feature vector is a
vectorized image (784 dimensions), containing grayscale values [0, 255]. The original image
dimensions are 28× 28. Here is an example montage of the digits:
The MATLAB file digits4000.mat (or digits4000 *.txt for non-MATLAB users) contains
the following data:
– digits vec – a 784 × 4000 matrix, where each column is a vectorized image, i.e. the
feature vector xi ∈ R784.
– digits labels – a 1× 4000 matrix with the corresponding labels yi ∈ {0, · · · , 9}.
– trainset – a 2× 2000 matrix, where each row is a set of indices to be used for training
the classifier.
– testset – a 2 × 2000 matrix, where each row is the corresponding set of indices to be
used for testing the classifier.
The image above was generated with the following MATLAB code:
4
testX = digits_vec(:,testset(1,:)); % get test data (trial 1)
testXimg = reshape(testX, [28 28 1 2000]); % turn into an image sequence
montage(uint8(testXimg), ’size’, [40 50]); % view as a montage
• Methodology – You can use any technique from the course material, e.g., Bayes classifiers,
Fisher’s Discriminant, SVMs, logistic regression, perceptron, kernel functions, etc. You may
also use other classification techniques not learned in class, but you will need to describe them
in detail in your report. Two useful libraries for classification are “libsvm” and “liblinear”.
You can also pre-process the feature vectors, e.g., using PCA or kPCA to reduce the dimen-
sion, or apply other processing techniques (e.g., normalization or some image processing).
Finally, a common trick for doing multi-class classification using only binary classifiers (e.g.
SVMs) is to use a set of 1-vs-all binary classifiers. Each binary classifier is trained to distin-
guish one digit (+1) vs. the rest of the digits (-1). In this case, there are 10 binary classifiers
total. Given a test example, each binary classifier makes a prediction. Hopefully, only one
classifier has a positive prediction, which can then be selected as the class. If not, then the
classifier that has the most confidence in its prediction is selected. For example, for SVMs
the classifier that places the test example furthest from the margin would be selected. For
logistic regression, the selection would be based on the calculated class probability.
• Evaluation – The classifiers are evaluated over 2 experiment trials. In each trial, 50% of the
data has been set aside for training (and cross-validation of parameters), and the remaining
50% is held out for testing only. The indices of the training set and test sets are given in the
trainset and testset matrices. For a given trial, the same writer does not appear in both
the training and test sets.
For each trial, train a classifier using only the training set data (images and labels). You may
also use the training set to select the optimal model parameters using cross-validation. After
training the classifier, apply the classifier to the test data (images only) to predict the class.
Record the accuracy (number correct predictions / total number) for that trial. Do not tune
the parameters to optimize the test accuracy directly! You can only tune the parameters using
the training set.
As a baseline, a simple nearest-neighbors classifier with Euclidean distance was used on the
test data. The resulting classification accuracy for each experiment trial is:
trial 1 2 mean (std)
1-NN 0.9135 0.9185 0.9160 (0.0035)
In your experiments, which classifier does better? What feature pre-processing helps or hurts
the performance? How does the performance vary with parameter values?
• Bonus Challenge – In the bonus challenge, I will give you a new test set containing my own
handwritten digits, and you will try to classify them using your trained classifiers. Whoever
gets the best performance wins a prize!
5