The University of Melbourne
SWEN90006: Software Testing and Reliability
Assignment 2
Second Semester, 2019
Due Date: 18:00pm, Friday, 18 October, 2019
1 Introduction
This is a group-based assignment, to be completed in groups of 4 (four) of your own choosing.
It is worth 25% of your grade for the subject.
In this group-based assignment, you will gain experience with security testing applied to a small
but non-trivial program, including investigating the trade-offs between different fuzzer designs
and contemporary off-the-shelf greybox fuzzers. You will also learn about memory-error vulner-
abilities in C, and use LLVM’s memory sanitisers.
You will implement a custom fuzzer to find memory-error vulnerabilities in a C program and
insert your own vulnerabilities. Your fuzzer will then “play” competitively against the fuzzers of
other teams to determine whose can find the most vulnerabilities. You will also compare your
fuzzer against LLVM’s general-purpose greybox fuzzer, libFuzzer, and evaluate it using standard
coverage metrics, as well as other metrics that you should decide upon. You will produce a report
detailing your vulnerabilities, your fuzzer’s design and systematically evaluating its performance
and the implications of your design decisions.
Marks are intentionally skewed towards the report (see the marking critera in Section 7), so make
sure you apportion your efforts appropriately in accordance with the marking criteria.
The program under test will be a command-line password manager tool implemented in C, and
inspired by the password manager from Assignment 1.
2 Things To Do Early
To make sure you don’t have last minute problems, make sure you do each of the
following very early on (i.e. once you have assembled your team.)
1. Fork the assignment repo and add team members (see instructions below) plus the admin
user. MAKE SURE YOUR FORKED REPO IS PRIVATE!
2. Make sure everybody in your group can clone the forked repo and push to it
3. Make sure that the automated eligibility checks, run by gitlab when you push to your
repository, are passing.
4. Make sure everybody in your group can build the code and run the various targets (see
instructions below).
1
5. Ask questions on the discussion board if you are having difficulty with these tasks. Prob-
lems with getting the code building or with git/gitlab will not be grounds for
extensions. Make sure you work out how to use the tools early on.
3 Gitlab and Assignment Repository
As with Assignment 1, your team will carry out their work in a git repository hosted on the
gitlab.eng.unimelb.edu.au GitLab service.
3.1 Assignment Repository
The assignment repository is at:
https://gitlab.eng.unimelb.edu.au/tobiasm1/swen90006-a2-2019.
The top-level file README.md documents its structure.
3.2 Working as a Team
The user listed in the “User1” column of the groups spreadsheet will fork the assignment reposi-
tory, on behalf of your group. They should then give permission to the other group members to
push to the repository; although it is up to each team to manage how updates should be made
to their repository. As with Assignment 1, your repositories MUST be private and you
must add the admin user to your project.
Having forked the repository, the “User1” user must then add each of their team members as
Members of their repository, plus the admin user (as in Assignment 1). From the repository’s
page in gitlab, choose Settings then Members, and then invite the other team members to the
repository by specifying their gitlab username and giving them the the appropriate permission
(e.g. the Maintainer permission). They can then clone your repository and push to it just as
you would, allowing your team to work collaboratively.
Also, remember to add the ‘Administrator (@root)’ user as in Assignment 1, giving
them the Developer permission.
4 The Passbook C Implementation
Since the focus of this assignment is security testing for memory-error vulnerabilities, you will be
using a C implementation of a program similar to the program-under-test from Assignment 1.
Specifically, the program is a command-line utility for managing website passwords. It takes
input either from text files or from the terminal (i.e. standard input, aka stdin).
4.1 Commands
Each line of input is a command. Commands conform to the following grammar:
2
::= “get”
“rem”
“put”
“save”
“list”
“masterpw”
Tokens in the grammar above are separated by one or more whitespace characters, namely space,
tab, carriage-return and line-feed. Each is a string of non-whitespace characters.
• For a URL string url, the command “get url ” looks up url in the passbook.
• For a username string username and password string password, the command “put url
username password ” updates the passbook to add a mapping for the URL url to map to
the username and password pair that comprises username and password.
• The command “rem url ” removes any mapping associated with the URL string url.
• For a password string password and filename string filename, the command “save password
filename” saves the current passbook to the file filename protected by the master password
password.
• The command “list” simply lists all mappings in the passbook.
• Finally, for a password string password, the command “masterpw password ” prompts the
user to enter their master password and checks whether the entered password matches the
string password ; if not, the program exits immediately.
4.2 Running the Passbook
Note, once you have built the program (see Section 4.3), the binary lives in the bin/original/
directory, from which the following commands should be run.
The passbook program, passbook, when run with no command line arguments prints out simple
usage information.
$ ./passbook
Usage: ./bin/original/passbook file1 file2 ...
use - to read from standard input
That is, passbook takes as its command line arguments a series of filenames or the special name
“-”, which instructs it to read from standard input (i.e. user keyboard input). It simply processes
each in turn, reading and executing commands line-by-line.
When passbook saves the current passbook to a file (i.e. in response to a “save” command), it
does so by writing to the file a series of commands that, when processed, cause the passbook
state to be reconstituted. This includes checking that the user knows the given master password,
by inserting the “masterpw” command as the first in the file.
For example, we might run the passbook and use it to save two example passwords to the file
passwords.txt under the master password “master_password” as follows:
$ ./passbook -
put http://example.com example_username example_password
put http://example2.com example2_username example2_password
save master_password passwords.txt
3
We can exit the passbook program by pressing CTRL-D, which indicates end-of-file/end-of-input,
or by forcefully interrupting it via CTRL-C.
The contents of the passwords.txt file is then:
masterpw master_password
put http://example.com example_username example_password
put http://example2.com example2_username example2_password
We can then re-run the passbook on this file and then instruct it to take further input from
the user by running “./passbook passwords.txt -” on the command line. It will first prompt
the user to enter the saved master password (“master_password”). If the user enters the master
password correctly, they can then list the passwords via the list command, which in this case
would cause the two saved passwords to be printed:
URL: http://example.com, Username: example_username, Password: example_password
URL: http://example2.com, Username: example2_username, Password: example2_password
4.3 Building
There are two ways to build the software. The first is by using the Engineering IT machines:
• dimefox.eng.unimelb.edu.au
• nutmeg.eng.unimelb.edu.au
• digitalis.eng.unimelb.edu.au
The second is to use your own machine. We discuss each.
4.3.1 Engineering IT Machines
1. ssh to one of the machines, log in with your username and password.
2. Then, ensure that /home/subjects/swen90006/llvm+clang-6.0.0/bin is in your path:
export PATH=$PATH:/home/subjects/swen90006/llvm+clang-6.0.0/bin
You should add that line to the .profile file in your home directory, so that this directory
is always in your path.
3. Test you can run clang-6.0, you should see the error message:
“clang-6.0: error: no input files”
4. Then from the top-level directory of the subject repository, run
source eng_unimelb_setup
which will ensure that the environment variables LDFLAGS and LD_LIBRARY_PATH are set as
required for building on these machines.
4.3.2 Building on your Own Machine
If you want to build on your own machine, you’ll need to install the dependencies yourself.
4
Requirements (aka Dependencies)
Unix-like host: Linux or MacOS The build process and software has been tested on Ubuntu
16.04 LTS and MacOS 10.13.6 High Sierra. It might well work on other somewhat-recent
Unix platforms too.
We strongly recommend using a Virtual Machine running e.g. Ubuntu 16.04 LTS if you
want to build on your own machine. Installing Clang (see below) in this case is very
straightforward.
Clang and LLVM The build process has been tested with Clang+LLVM 6.0
Ubuntu (16.04 LTS)
1. First, enable the universe package repository if you haven’t enabled it already:
sudo add-apt-repository universe
2. Next, just install Clang and LLVM 6.0 via apt-get.
sudo apt-get install libfuzzer-6.0-dev llvm-6.0
MacOS
1. If you don’t have them installed already, first install Xcode Command Line Tools for
your OS version (e.g. “Xcode Command Line Tools (macOS 10.13) for Xcode 9.4.1”),
from https://developer.apple.com/download/more/. Note that we won’t be using
the version of LLVM that ships with Xcode Command Line Tools. However we need
the system include headers and so on that come with it.
2. Download Clang+LLVM 6.0.0 from http://releases.llvm.org/6.0.0/clang+
llvm-6.0.0-x86_64-apple-darwin.tar.xz
3. Extract the resulting file:
tar -xJf clang+llvm-6.0.0-x86_64-apple-darwin.tar.xz
This will put Clang+LLVM into the directory
http:releases.llvm.org6.0.0clang+llvm-6.0.0-x86_64-apple-darwin.tar.xz
4. Try running clang:
cd clang+llvm-6.0.0-x86_64-apple-darwin
./bin/clang-6.0
You should see the message: “clang-6.0: error: no input files”
5. Put clang-6.0 etc. into your path:
export PATH=$PATH:$PWD/bin
You should put this line into the file .profile in your home directory, which ensures
that clang-6.0 will be in your path each time you open a terminal.
6. Test you can run it from any location:
clang-6.0
You should see the same message as before. Note that this time we didn’t need to
specify where clang-6.0 was located because it is now in the path.
5
4.3.3 Building the C Implementation
There is a Makefile for building the C implementation of the program, in the top-level directory.
Assuming you are building with Clang and LLVM 6.0 and have successfully installed them, then
all you should need to do for the initial build is to run ‘make’ from the top-level directory:
$ make
This should give you at least the main binary, passbook in the bin/original directory, as well
as some other targets (see Section 4.3.6 below).
If you are building on dimefox etc. and you get an error message “clang-6.0: error: linker
command failed with exit code 1 (use -v to see invocation)”, it means you probably forgot to do
‘source eng_unimelb_setup’ first.
4.3.4 Disabling Debug Output
The program outputs lots of (debugging) output by default. This can, and probably should, be
turned off by re-building it with the compiler flags set to include ‘-DNDEBUG’.
First remove the old binaries by doing:
make clean
Then re-build, setting the CFLAGS variable to include ‘-DNDEBUG’:
CFLAGS=-DNDEBUG make
4.3.5 Specifying the Compiler to Use
This is not recommended, as a recent version of Clang is required to build the more interesting
versions of the program (see Section 4.3.6 below).
By default, the Makefile tries to build using the compiler clang-6.0. If you need to build using
a different compiler (version), then you can do that by setting the CLANG variable, e.g.:
$ CLANG=clang make
This would cause building using the clang compiler (which is probably some default version of
Clang that might already be present on your machine).
4.3.6 Other Build Targets
Besides the passbook binary, the Makefile knows how to build the other following targets, which
are different versions of the passbook program.
passbook-san the program built with AddressSanitiser enabled, for detecting memory errors.
You should use this to test triggering of your security vulnerabilities (see Section 5.1).
passbook-fuzz a libFuzzer target for the program. When run, this binary uses LLVM’s in-built
greybox fuzzer, libFuzzer, to fuzz the program. You should use this to compare your fuzzer
against (see Section 5.3.2).
passbook-cov the program built to gather coverage information. You should use this to help
evaluate your fuzzer (see Section 5.3.3).
6
To facilitate automated fuzzing and evaluation, these other targets behave slightly differently
than the passbook program. Specifically, they do not perform master password checking when
executing the “masterpw” command and they do not save the output to a file when executing
the “save” command. This allows them to be run in batch mode (i.e. without having to pause
to accept user input) and without modifying the filesystem, which is necessary to facilitate
automated fuzzing and evaluation.
5 Your Tasks
5.1 Task 1: Inserting Memory-Error Vulnerabilities
Your first task is to create 5 vulnerable versions of the software. In the subject repository are 5
identical copies of the program in the directories: src/vuln-1/, src/vuln-2/, . . . src/vuln-5/
Your goal is to construct reasonable vulnerabilities that are non-trivial to exploit. Part of the
marks for your vulnerabilities will be determined by the number of other teams whose fuzzers
can trigger them: the lower this number is, the better for your team up to a point.
Exploitability Your vulnerabilities must be exploitable. By this we mean that if we take
your vulnerable version and compile it with LLVM’s memory error sanitiser, AddressSanitiser,
enabled, then there must exist at least one input that causes AddressSanitiser to report a memory
error when run on your vulnerable version.
Proof of Concept You will prove that each of your vulnerabilities is exploitable by submitting
with it a proof of concept (PoC). A PoC is an input that triggers your vulnerability under
AddressSanitiser as described above.
Your PoCs will live in the poc/ directory. The PoC for vulnerable version i you will place into
the file: vuln-i.poc in that directory, which already exists as an empty file in the repository.
Confirming Exploitability of Your Vulnerabilities You can confirm if your vulnera-
bilities are exploitable as follows. Suppose we have a candidate PoC input for vuln-1 in
poc/vuln-1.poc.
Building with make produces binary for the vuln-1 version in the bin/vuln-1 directory. The
passbook-san binary in that directory is the vuln-1 version compiled with AddressSanitiser
enabled. You can then run
./bin/vuln-1/passbook-san poc/vuln-1.poc
to observe if AddressSanitiser reports a memory error.
Reasonableness To receive full marks, your vulnerabilities must not only be exploitable but
also be reasonable. By this we mean able to be triggered with non-negligible probability. An
example of an unreasonable vulnerability is one that is triggered only when the user enters
a particular URL, username or password, or one that is triggered only if the password is a
palindrome, or other situations that would be almost impossible to trigger via ordinary fuzzing.
Naturally there is no hard and fast rule about reasonableness. We will judge by examining each
of your vulnerable versions manually.
7
Ask on the Discussion Board if you want further clarification, without revealing details of your
vulnerabilities, or email to ask privately.
Where to avoid inserting your vulnerabilities Recall from Section 4.3.6 that the other
targets (including passbook-san) have certain features disabled, namely master password check-
ing and saving to files respectively. Therefore you should not add vulnerabilities to these parts
of the code, since they will be impossible to trigger in passbook-san. Specifically, do not insert
your vulnerabilities in the parts of the code between “#ifdef PASSBOOK_FUZZ . . .#else” and
do not modify or remove these “#ifdef . . .#else . . .#endif” directives.
5.2 Task 2: Design and Implement a Fuzzer
Your team needs to implement a fuzzer that generates input for the program, attempting to
trigger other teams’ (and your own team’s) memory corruption vulnerabilities.
Your fuzzer should be implemented in Java, and lives in the fuzzer/ directory of the repository.
The top-level file for it is already provided as a stub in Fuzzer.java.
It should build with Java 8 and we should be able to build and run it by running the
run_fuzzer.sh (unmodified) from the top-level directory of the repository.
Your fuzzer should place its output in the file fuzz.txt, overwriting that file if it exists already.
You are free to use whatever techniques you like in your fuzzer, including random fuzzing,
mutation fuzzing, generation-based fuzzing, or some combination of these techniques.
When designing your fuzzer, you should consider what techniques are most appropriate to use
and combine, given the function and design of the program under test. A good fuzzer will be
one that is able to fuzz all of the various parts of the software.
We will run your fuzzer multiple times to collect multiple outputs from it, so it should be built
to produce a different output each time it is run (although always in the same file fuzz.txt). If
you want to design your fuzzer to save state between runs, you’ll have to have it save its state
in a custom file that it creates itself, separate to fuzz.txt.
For random numbers, you might wish to use the standard java.util.Random class.
5.3 Task 3: Evaluation
A non-trivial part of this assignment involves you critically evaluating your own fuzzer. We have
provided some initial options available to you when evaluating your fuzzer. However, to get full
marks here we want you to think creatively about how you can use each of them (and any others
that you devise) in a carefully designed experiment from which you draw high quality insights
(see below). Simply using each of these methods is not enough to guarantee a high mark for this
part of the assignment: the options below are starting points to get you thinking. We want to
see creativity and insight, especially in how you design the experimental evaluation.
5.3.1 Triggering Your Own Vulnerabilities
Naturally, your fuzzer’s ability to trigger your own security vulnerabilities is relevant to its
performance. If your fuzzer has created the output fuzz.txt, then you can determine whether
e.g. it triggers vuln-3 by running:
8
./bin/vuln-3/passbook-san fuzz.txt
But of course your vulnerabilities are only a very small sample.
5.3.2 Comparison to Greybox Fuzzing
You should also compare your fuzzer against an off-the-shelf greybox fuzzer like AFL or LLVM’s
libFuzzer.
The Makefile builds a libFuzzer-based fuzzer, with AddressSanitiser enabled, for each of your
vulnerable versions. You can fuzz the ith vulnerable version using libFuzzer by running the
binary ‘passbook-fuzz’ in the bin/vuln-i directory.
Unlike your fuzzer, this fuzzer generates output and runs it directly against the program under
test, and does so repeatedly until it causes a crash (e.g. triggers AddressSanitiser to abort the
program because a memory error occurred). Along the way it builds up a corpus of interesting
inputs that then form a test suite for the program being fuzzed.
Each passbook-fuzz binary accepts various command line arguments to control how the fuzzer
behaves, where the corpus and crash outputs are placed in the filesystem, etc.
You should read the libFuzzer documentation at http://releases.llvm.org/6.0.1/docs/
LibFuzzer.html (for Clang 6.0) to understand what arguments you can pass and how you
can control fuzzing is performed and how results are collected. Note that the Makefile already
handles for you the steps in the “Fuzzer Usage” section of that documentation.
5.3.3 Coverage
You should also evaluate your fuzzer (including against libFuzzer) by gathering source coverage
statistics.
The Makefile already generates a target passbook-cov that automatically collects coverage
statistics, and there is a script get_coverage.sh in the top-level directory for producing coverage
reports from a test suite (corpus).
For example, suppose you have a test suite and you wish to measure the line coverage it achieves.
Suppose your corpus is a set of files in the directory . You can run:
./get_coverage.sh /*
Naturally, consult the LLVM documentation for further information about generating coverage
reports and other coverage statistics. get_coverage.sh should be considered a starting point.
5.3.4 Designing Your Evaluation
You need to carefully design an evaluation (series of experiments and measurements) of your
fuzzer, employing the above tools and any others that you deem worthwhile. We are particularly
interested in seeing results that shed light on the quality and trade-offs made in your fuzzer’s
design.
5.4 Task 4: Report
Along with your repository, you also need to submit a written report.
9
Your report needs to include the following:
Team Members and Responsibilities A list of your team’s members and what each of their
responsibilities were during the project.
Your Vulnerabilities A description of each of your vulnerabilities, under what circumstances
it is triggered, and what its potential consequences might be if present in a real implemen-
tation.
Fuzzer Design A description of your fuzzer’s design, including the techniques it employs, the
trade-offs you made in its design, etc.
Evaluation Your evaluation. This should include a description of the experiments you per-
formed, their results, and your conclusions from them. Remember we want to see you
drawing insights into how your fuzzer’s design affects its performance.
Your report should be submitted as a PDF document and be no longer than 10 A4 pages.
Anything longer than 10 pages we make no promises to read when marking your report.
5.5 Task 5: Individual Report (Optional)
Each team member also has the option to submit an individual report of no more than 0.5 page
(2-3 paragraphs) in plain text. The purpose is to allow individual team members to report on
• What they contributed to the team
• What they perceived other teams members to have contributed
While the project is done as a team, we reserve the right to assess members of a team individually.
The individual report is optional, but you won’t be given the opportunity to submit one after
the deadline if your mark is affected by the individual report of one of your team members.
Anything longer than 0.5 pages will not be read.
6 Fuzzing Tournament
Part of your marks for this assignment will come from us automatically competitively evaluating
your fuzzer and vulnerabilities against those of other teams, following a similar approach to
that used in Assignment 1 for test cases and mutants. The goal of the tournament is to see
whose fuzzer can trigger the most vulnerabilities and which teams’s vulnerabilities are hardest
to trigger.
The tournament will provide valuable feedback on your progress, and will operate on the master
branch of your repository in the same manner as for Assignment 1. Therefore, it is in your
interests to get started early and to keep your master branch in a working state.
10
7 Marking Criteria
7.1 Tournament (5 marks)
We will run your fuzzer some fixed number NUMFUZZ times to produce NUMFUZZ inputs to
the program. (We will determine an appropriate value for NUMFUZZ during the course of the
assignment, but it will likely be no smaller than 10.)
We will then run these against all vulnerable versions of every team to derive a score:
fuzzScore =
k
T
where k is the number of vulnerable versions whose vulnerabilities were triggered by one of
your NUMFUZZ inputs, and T is the highest number of vulnerabilities triggered by any other
team’s fuzzer (i.e. the number of vulnerabilities triggered by the team whose NUMFUZZ inputs
triggered the most vulnerabilities).
The maximum possible score is 1, which will be scaled to 2.5.
For the vulnerabilities, they will be scored identically as to how mutants were scored in Assign-
ment 1.
vulnScore =
Σ5iΣ
N
j ai,j
T ×N
Here N is the total number of teams, and ai,j = 1 if vulnerable version i is not triggered after
being executed on the NUMFUZZ inputs generated by team j’s fuzzer, and T is the highest
number of vulnerable versions of any single team not triggered by any team in the class.
The maximum possible score is 1, which will be scaled to 2.5.
7.2 Report (20 marks)
Criterion Description Marks
Vulnerabilities Reasonableness, clear and concise explanation 2.5
Fuzzer Clear, well motivated design and trade-offs, consider-
ing the various aspects of the program under test
5
Evaluation Design Logical, thorough experimental design, clearly ex-
plained
5
Evaluation Execution Evaluation carried out according to its design 1
Results Presentation Clear presentation of results using appropriate for-
mats (graphs, tables etc.)
1.5
Discussion Discussion of results shows insight, especially into the
consequences of your fuzzer’s design
5
Total 20
11
8 Submission Validation
We will be automatically running your fuzzer and vulnerable versions, and part of your marks
for this assignment will depend on these running correctly on our setup.
As in Assignment 1, each time you push to your repository we will automatically run your fuzzer
and your vulnerable versions against the output from other team’s fuzzers. You should refer
to the logs automatically generated by gitlab to check that your code is working as expected
on our platform and to make sure that any eligibility issues are resolved well in advance of the
submission deadline. Problems here at the time of the submission deadline will not be grounds
for extension and will result in late penalties being applied.
9 Submission
Your team’s submission includes the following parts:
1. Your group’s repository, which we will clone at the time of the assignment deadline.
2. Your group’s report, which should be submitted via the LMS as a PDF document. Select
View/Complete from the Assignment 2 Project Report submission items, and follow the
instructions.
3. Individual reports, which should be submitted via the LMS. Select View/Complete from
the Assignment 2 Individual Report submission
10 Late Submissions
As per 1, the penalty for late submissions is 10% of the assignment mark for each day late, up
to seven days total. After seven days, submissions will no longer be accepted.
Extensions will only be granted for the situations outlined in the policy, meaning that extensions
are not granted if other assignments are due around the same time. It is the responsibility of
individuals to plan the use of their time effectively to meet assignment deadlines.
11 Academic Misconduct
The University misconduct policy2 applies. Students are encouraged to discuss the assignment
topic, but all submitted work must represent the individual’s understanding of the topic.
The subject staff take academic misconduct very seriously. In this subject in the past, we have
successfully prosecuted several students that have breached the university policy. Often this
results in receiving 0 marks for the assessment, and in some cases, has resulted in failure of the
subject.
1See http://www.policy.unimelb.edu.au/schedules/MPF1029-ScheduleA.pdf
2See https://academichonesty.unimelb.edu.au/
12