CS 663 - Machine Learning Spring, 2020

Data Challenge

The purpose of this challenge is to test your ability to write software and models to collect,
normalise, store, analyse and visualise “real world” data. This challenge is designed to mimic
those you may receive upon applying to positions as Data Scientist or Machine Learning
Engineer. You may draw on your work in the lab assignments. The challenge is designed to
take about two hours, but it is not timed. Please deliver your results by the due date.

You may also use any tools or software on your computer, or that are freely available on the
Internet as long as the tool works with a Jupyter notebook. We prefer that you use simpler tools
to more complex ones and that you are “lazy” in the sense of using third party APIs and libraries
as much as possible. The use of obscure, undocumented “black box” libraries is discouraged.

Do as much as you can, as well as you can. Prefer efficient, elegant solutions. Prefer scripted
analysis to unrepeatable use of GUI tools. For data security and transfer time reasons, you have
been given a relatively small data file. Prefer solutions that do not require the full data set to be
stored in memory.

Finally, we are also interested in your ability to work on a team, which means considering how
to package and deliver your results in a way that makes it easy for us to review them. This does
NOT mean you are allowed to discuss with others or use their work, including those enrolled in
this or similar courses. It does mean that undocumented code and data dumps are virtually
useless; commented code and a clear writeup with elegant visuals are ideal. Also consider how
asking targeted questions to members of our team may allow you to get more done in less time.

Background

Health Inspectors from the Health Department of the City and County of San Francisco routinely
conduct inspections of restaurants (“facilities”). ​After conducting an inspection of a facility, a
Health Inspector calculates a score based on the violations observed. Violations can fall into:
● High risk category: records specific violations that directly relate to the transmission of
food borne illnesses, the adulteration of food products and the contamination of
food-contact surfaces
● Moderate risk category: records specific violations that are of a moderate risk to the
public health and safety

CS 663 - Machine Learning - Spring 2020 - Data Challenge 1

● Low risk category: records violations that are low risk or have no immediate risk to the
public health and safety.
These violations may also be graded — i.e. converted to an inspection score — and posted, for
example, on the windows of the facilities. By design, some inspections do not contain violations
or inspection scores.

Data

WIth these instructions, we have provided two CSV files:
● facility_scores_known.csv ​ (9MB): 43,199 facility records plus 1 header
● facility_scores_unknown.csv ​ (2 MB): 10,774 facility records, plus 1 header

Requirements (Process)

There are two (2) parts for this challenge:
1. Predict inspection scores.
2. Explain inspection scores.

Predict inspection scores

You may use the train data to create a model for predicting inspection scores of a facility. The
inspection score for each facility is missing from the test set. You must use a model to predict
the inspection scores of facilities for each instance in this set.

You will submit your prediction for inspection scores, which the grader will compare against the
actual values using MSE (mean squared error). Your prediction must be named “​preds.csv ​”,
a file in CSV format with the one field: inspection score. This file must have one prediction for
each facility appearing in the test.csv file, in order. For example:
76
17

Explain inspection scores

Once your model has been created, you must provide an explanation of what factors best
predict the facility’s score.


CS 663 - Machine Learning - Spring 2020 - Data Challenge 2

Submission

Submit the following to Github (​starter link​):
● Your Jupyter notebook for the implementation of the above
● Your model’s prediction of inspection scores as a CSV file
● A PDF document with an explanation of your process, findings, visualisations, etc.
The submission deadline is 11:59 PM PDT on 13th May, 2020. Late submissions ​will not be
accepted.

Grading

Each submission will be graded as follows:
50% Performance The competitive accuracy (as measured by MSE) of your model 1
as executed on a neutral system.
20% Explanation of
Model
The degree to which your process for finding explanatory features
follows a reasonable process. In addition, the correct identification
of explanatory features according to the following:
20% = Reasonable process and correct features derived
14% = Not following reasonable process or incorrect features
7% = Not following reasonable process and incorrect features
15% Code Quality The degree to which your solution is modular, easy to run, easy to
read and contains comments helpful to a peer or other person with
skills similar to yours.
15% = Completely
9% = Partially
3% = Poorly
10% Process The degree to which your solution follows a reasonable process
and has documented this process.
10% = Completely
7% = Partially: missing process details / module documentation
3% = Poorly: missing several major details / most documentation
5% Execution Time The competitive wall clock execution time of your model as
executed on a neutral, CPU-based system

1 For competitive grading, the submissions with the top performance get a full-credit score (eg. 50/50 on
Performance). Other submissions which do not yield top performance are ranked and graded accordingly.
Your model will be executed once, so be wary of models with varying / random performance.

CS 663 - Machine Learning - Spring 2020 - Data Challenge 3