The University of Melbourne
SWEN90006: Software Testing and Reliability
Assignment 1
Second Semester, 2019
Due Date: 18:00pm, Friday, 4 September, 2020
Introduction
The first assignment deals with input partitioning, boundary-value analysis, and control-flow
testing. You are given a specification and a program that implements that specification. The
aim of this assignment is to test the program using the different techniques, and to analyse the
difference between them.
You are expected to derive and compare test cases, but you are not expected to debug the
program.
The assignment is part laboratory exercise because you are expected to write a JUnit driver
program to run your test cases automatically. Some exploration may be needed here. The
assignment is also part analysis exercise as you are expected to apply the testing techniques to
derive your test cases, and to compare them. Finally, the assignment is also part competition,
as your solutions to various tasks will be evaluated against all other submissions in a small
tournament to measure its effectiveness and completeness.
The assignment is worth 20% of your final mark.
Your Tasks
Task 0 (0 marks) Due 18:00 Friday, 21 August, 2020
Create an account at https://gitlab.eng.unimelb.edu.au/ if you do not already have one.
Fork the source code repository at https://gitlab.eng.unimelb.edu.au/tmiller/
swen90006-a1-2020.git
Once you have forked your repository, set your repository to private. Otherwise, every else can
see your mutants and your code. This can be set in Settings → General → Permissions →
Project Visibility.
Then, Add `Adminstrator (@root)' as a developer to your repository, otherwise we will not be
able to access your code for submission. To do this, go to Settings → Members, under `Select
members to invite', search for `Administrator', then under `Choose a role permission' select
`Developer', and finally click `Add to project'.
Set the Expiration Date as 18 December, 2020.
Finally, submit your repository details here: http://go.unimelb.edu.au/k3qr. We will clone
this repository to run your boundary-value test script for the tournament.
1
Documentation for how to fork, pull, push, merge etc. is available from the Gitlab site: https:
//gitlab.eng.unimelb.edu.au/help/user/index.md.
You will need to be able to: fork, clone, commit, push, and pull your repository.
Task 1
Using the specification, apply equivalence partitioning to derive equivalence classes for the fol-
lowing methods in the API: register and rent.
Document your equivalence partitioning process for each method using only test template trees,
listing the assumptions that you make (if any). You should have two trees: one for each method.
You will be marked only on your test template trees (plus any assumptions listed), so ensure
that they are clear and concise.
You can omit some nodes to improve readability, provided that it is clear what you intend. For
example, if I was testing a book store and I wanted to test all seven Harry Potter books, I would
create nodes for 1 and 7, and then use . . . in between them to represent the other five books.
Note that as part of your input domain, you will have to consider the instance variables. These
are not parameters to any of the methods, but they are inputs.
Do your set of equivalence classes cover the input space? Justify your claim.
Task 2
Select test cases associated with your equivalence classes, and implement these in the JUnit
test driver named tests/Partitioning/swen90006/xilften/PartitioningTests.java. Use
one JUnit test method for each equivalence class. For each test, clearly identify from which
equivalence class it has been selected. Push this script back to you gitlab repository.
Include this as Appendix A of your submission.
NOTE: When implementing tests for one method, you may use other methods to check that the
first method has worked as expected.
Task 3
Conduct a boundary-value analysis for your equivalence classes. Show your working for this.
Select test cases associated with your boundary values.
Task 4
Implement your boundary-value tests in the JUnit test driver called
test/Boundary/swen90006/xilften/BoundaryTests.java. As before, use one JUnit test
method for each test input. Push this script back to you gitlab repository.
Include this as Appendix B of your submission.
Note that the BoundaryTests JUnit script inherits PartitioningTest, which means all tests from
PartitioningTests are including in BoundaryTests. A JUnit test is just a standard public Java
class! You may choose to remove this inheritance, but you may also use it to your advantage to
make the BoundaryTest script easier to create. Overriding an existing test will replace it in the
BoundaryTest script.
2
Task 5
Calculate the coverage score of your two test suites (equivalence partitioning and boundary-value
analysis) using multiple-condition coverage each the two methods, as well as those methods in
Xilften.java that they call; that is, getWeekDays, monthDuration, and dayOfWeek.
Show your working for this coverage calculation in a table that lists each test objective (that is,
each combination for multiple-condition coverage) and one test that achieves this, if any.
NOTE: You do NOT need to draw the control-flow graph for your solution.
You will receive marks for deriving correct coverage scores and showing how you come to this
score. No marks are allocated for having a higher coverage score.
Task 6
Derive five non-equivalent mutants for the Xilften class using only the nine mutation opera-
tors in the Section 4.4 of the subject notes. These mutants should be difficult to find using
testing. Insert each of these mutants into the files programs/mutant-1/swen90006/xilften/,
programs/mutant-2/swen90006/xilften/, etc.
All five mutants must be non-equivalent AND each mutant must be killed by at least one test
in your JUnit BoundaryTest script to demonstrate that they are non-equivalent.
Importantly, do not change anything else about the mutant files except for inserting the mutant.
Each mutant must change exactly one line of Xilften.java.
Task 7
Compare the two sets of test cases (equivalence partitioning and boundary-value analysis) and
their results. Which method did you find was more effective and why? You should consider
the coverage of the valid input/output domain, the coverage achieved, and the mutants it kills.
Limit your comparison to half a page. If your comparison is over half a page, you will be marked
only on the first half page.
The System: Xilften
Xilften is a (fictional) video streaming service.
A user can register an account with xilften, and can rent movies. In this assignment, we deal
only with user registration and with calculating the cost of movie rental.
An API specification is available in the source file programs/original/xilften/Xilften.java
on the gitlab repository above.
Building and Running the Program
The source code has been successfully built and tested on JDK 14 but should also work with
some earlier versions of Java.
The file build.xml contains an Ant build script, should you choose to use it. The README.md
file in the top-level folder has instructions for using this.
3
There are two JUnit test scripts in test/swen90006/xilften. You will need to modify each of
these to complete the tasks below. You can run these by compiling and running as a class, but
you will need to include the library files in the lib/ directory on your classpath.
NOTE: If you find any functional faults in the implementation, please let us know via the
discussion board. We will correct the fault and ask that everyone pull changes. There are not
intended to be any faults in the implementation, but software engineering is hard!
Marking criteria
As part of our marking, we will run your boundary-value analysis JUnit scripts on everyone else's
mutants. You will receive marks for killing other mutants as well as for deriving mutants that
are hard to kill. This will contribute 5 marks to the total.
Criterion Description Marks
Equivalence parti-
tioning
Clear evidence that partitioning the input space to find
equivalence classes has been done systematically and cor-
rectly. Resulting equivalence classes are disjoint and cover
the appropriate input space
7
Boundary-value
analysis
Clear evidence that boundary-value analysis has been ap-
plied systematically and correctly, and all boundaries, in-
cluding on/off points, have been identified
3
Control-flow analy-
sis
Clear evidence that measurement of the control-flow cri-
terion has been done systematically and correctly
2
There is a clear and succinct justification/documentation
of which test covers each objective
2
Discussion Clear demonstration of understanding of the topics used
in the assignment, presented in a logical manner
1
JUnit tests JUnit scripts implement the equivalence partitioning and
boundary-value tests, and find many mutants
2.5
Mutants Selected mutants are valid mutants and are difficult to find
using tests
2.5
Total 20
For the JUnit tests, the score for these will be calculated using the following formula:
junit_score =
k
T
ln(N) + 10
in which N is the number of tests in your test suite, k is the number of mutants that your test
suite kills, and T is the maximum number of mutants killed by any other JUnit test suite1.
The entire pool of mutants are the mutants from all other submissions. Therefore, your score is
the mutant score, divided by ln(N) + 10, which incentivises smaller test suites2. The maximum
1
This ensures that equivalent mutants are not counted.
2
This incentive is to resist the urge to submit a test suite of thousands of tests with the hope of increasing the
score.
4
possible score is 0.1, scaled to be out of 2.5.
For the mutants, the score is:
mutant_score =
∑M
i
∑N
j ai,j
T
in which M is the total number of your mutants, N is the total number of other people's test
suites, ai,j = 1 if mutant i is still alive after executing test suite j, and T ≤M +N is the highest
number of mutants still alive by any student in the class. This is then scaled to be out of 2.5.
Therefore, your score is the inverse of the mutant score of all other students' test suites on your
mutants, which incentivises you to submit hard-to-find mutants, while T normalises the score to
ensure that everyone is rewarded for good mutants.
Important note: We determine that a mutant is found when JUnit contains a failed test.
Because of this, if a JUnit fails a test when applied to the original source code, it will fail on
everyone else's mutants, giving people a 100% score. As such, JUnit suites that fail on the
original source code emulator will be disqualified from the tournament. As noted above, if you
find any faults in the original source code, please let us know via the discussion board.
Submission
JUnit script submission For the JUnit test scripts, we will clone everyone's Gitlab repository
at the due time. We will mark the latest version on the master branch of the repository. To have
any late submissions marked, please email Tim ([email protected]) to let him know to
pull changes from your repository.
Some important instructions:
1. Do NOT change the package names in any of the files.
2. Do NOT change the directory structure.
3. Do NOT add any new files: you should be able to complete the assignment without adding
any new source files.
JUnit scripts will be batch run automatically, so any script that does not follow the instructions
will not run and will not be awarded any marks.
Report submission The remainder of the assignment (test template tree, boundary-value
analysis working, coverage, and discussion) submit a PDF file using the links on the subject
Canvas site. Go to the SWEN90006 Canvas site, select Assignments from the subject menu, and
submit in Assignment 1 report, which will take you to Gradescope to submit.
A video on how to upload a PDF to Gradescope can be found here: https://www.youtube.com/
watch?v=KMPoby5g_nE
Tournament
Each time you push changes to your repository, your code submission (mutants and tests) will
be run against all other students' submissions.
5
The results from your latest run will be available here:
http://murray.cis.unimelb.edu.au:8080/
The first of these will be available from Monday, 24 August.
Note that it takes several minutes for this to run a submission each time it is pushed. Towards
the deadline, there will be many submissions and your results may not update for several hours.
So, push early to get early feedback!
Tips
Some tips to managing the assignment, in particular, the equivalence partitioning:
1. Ensure that you understand the notes before diving into the assignment. Trying to learn
equivalence partitioning or boundary-value analysis on a project this size is difficult.
2. Keep it simple: don't focus on what you think we want to see focus on looking for good
tests and then documenting them. That IS what we want to see.
3. Focus on the requirements: as with any testing effort, focus your testing on the require-
ments, NOT on demonstrating the theory from the notes. Simply look at each requirement
and see which guidelines should apply.
4. If you cannot figure out how to start your test template tree, just start listing tests that
you think are important. Once you have a list, think about putting them into a tree.
Academic Misconduct
The University academic integrity policy (see https://academicintegrity.unimelb.edu.au/)
applies. Students are encouraged to discuss the assignment topic, but all submitted work must
represent the individual's understanding of the topic.
The subject staff take academic misconduct very seriously. In this subject in the past, we have
successfully prosecuted several students that have breached the university policy. Often this
results in receiving 0 marks for the assessment, and in some cases, has resulted in failure of the
subject.
Originality Multiplier
For work that we find is similar to another submission or information found online, an originality
multiplier will be applied to the work. For example, if 20% of the assessment is deemed to have
been taken from another source, the final mark will be multiplied by 0.8.
6