SOFT3202/COMP9202 Assignment
Software Construction and Design 2
Task Description
Software development relies heavily on testing and monitoring to ensure the quality, reliability and
performance of the software in production. In this assignment, you will develop a series of tools that
canbeusedtotestandmonitorsoftwareduringthesoftwaredevelopmentpipeline.
Assignment Breakdown
The requirements for completing this assignment are dependent on the course you are taking. The
assignmentisdueonthe12thofMay,2024.
SOFT3202:
StudentstakingSOFT3202mustcompletethefollowingsectionstoattainthecorrespond-
ingmarks:
Part1 (15%)
Selectoneof:
Part2
Part3 (15%)
Note: Students may not combine marks from Parts 2 and 3. You will only receive marks
forthehigherscoringofeitherimplementation.
COMP9202:
StudentstakingCOMP9202mustcompletethefollowingsectionstoattainthecorrespond-
ingmarks:
Part1 (10%)
Part2 (10%)
Part3 (10%)
Example Submissions
SOFT3202
ExamplesforSOFT3202
Example1:
Part1: 5marks
Part2: 15marks
Totalmarks: 20
Example2:
Part1: 7marks
Part3: 18marks
Totalmarks: 25
Example3:
Part1: 6marks
Part2: 15marks
Part3: 18marks
Totalmarks: 24
COMP9202
ExamplesforCOMP9202
Example1:
Part1: 7marks
Part2: 8marks
Totalmarks: 15
Example2:
Part1: 8marks
Part2: 7marks
Part3: 9marks
Totalmarks: 24
Page2
Part 1: Coverage
Introduction
Thiscomponentrequiresthedevelopmentofacomprehensivetooltoanalyseaprovidedtestsuitefor
a piece of software. The analysis must cover two key white-box testing metrics: statement coverage
and k-path coverage. The aim is to assess the efficacy and thoroughness of the test suite in detecting
faultsandensuringrobustnessinthesoftware.
Objectives
• Implementatoolthattakesaseriesofgiventestinputsandrunsthemonaprogram.
• Report the statement and k-paths coverage for the program when run using the series of test
inputs.
Requirements
1. StatementCoverage
Objective: Determine the percentage of executable statements in the software that are executed by
thetestcasesinthetestsuite.
2. k-PathsCoverage
Objective: Identify and report the number of acyclic paths through the program’s control flow graph
thatarecoveredbythetestsuite.
Input Specifications
Yourprogramshouldtake2command-linearguments:
1. ThepathtoaPythonscript
2. Thepathtoadirectorycontainingasetofinput(.in)files
Itshouldbecalledusingthefollowingcommand:
python coverage.py
Output Specifications
Yourprogramshouldproduceoutputindicating:
Statement Coverage: The percentage of statements executed during testing.
1
k-Path Coverage: The percentage of acyclic paths executed during testing.
2
Forexample:
Statement Coverage: 50%
1
k-Path Coverage: 12%
2
Page3
Part 2: Fuzzing with Mutated Inputs
Introduction
In this part of the assignment, you will develop a fuzzer designed to automate the generation and
mutationoftestinputstomaximizethecoverageofatestsuite. Theprimarygoalistoexpandthetest
coveragebyidentifyingandaddinginputsthatexposenewpathsorstatesinthesoftwareundertest.
Objectives
• Developafuzzercapableofgeneratingandmutatingtestinputs.
• Implementamethodtomeasuretheincreaseintestcoverage.
• Automatetheprocessofenhancingthetestsuitewithinputsthatincreasecoverage.
Requirements
Thistaskrequiresyoutotakeaprogramalongwithaseriesofinputsandmutatetheinputstoachieve
a minimum coverage. You must automatically improve the test suite by adding mutated inputs that
increasecoverage.
Implementation Specifications
• Usethefuzzertoapplymutationstotheinitialsetofinputs.
• For each mutated input, execute the test suite to determine if the mutation results in increased
coverage.
• Ifaninputincreasescoverage(byreachingnewcodepathsorconditionsnotpreviouslytested),
addittoa’population’ofeffectivetestinputs.
• Continue this process until no further increase in coverage is observed, aiming to achieve the
largestpossiblecoverage.
• Writethefinalsetoftestinputsthatcollectivelyprovidethehighestcoverageobservedtoafile.
Input Specifications
Yourprogramshouldtake2command-linearguments:
1. ThepathtoaPythonscript
2. Thepathtoasingletext(.txt)file
Itshouldbecalledusingthefollowingcommand:
python mutation_fuzzer.py
Thetextfilewillcontainasetofinputs,eachonanewline.
Forexample:
Never
1
Gonna
2
Give
3
You
4
Up
5
Page4
Output Specifications
Your program should write back to the provided input (.in) file with exactly the same number of
inputstringsaswasprovidedinitially.
Forexample:
Never
1
Gonna
2
Let
3
You
4
Down
5
Part 3: Grammar-Based Fuzzing
Introduction
Grammar-basedfuzzingisacommonlyusedmethodtotestprogramsthatconsumestructuredinputs,
particularlyinputparsers.
Objectives
• Implementagrammar-basedfuzzertogeneratestructuredinputsfortesting.
• Explorevariousgrammarstructurestohitorexceedacoveragethreshold.
Requirements
This task requires implementing a grammar-based fuzzer capable of generating structured inputs
based on a specified grammar. The goal is to hit or exceed a coverage threshold by generating a
testsuitethateffectivelyteststhetargetprogram.
Implementation Specifications
• Developalgorithmstointerpretgrammarspecificationsandgenerateinputsaccordingly.
• Exploredifferentpathsandoptionswithinthegrammartomaximisecodecoverage.
• Testthegeneratedinputsonthetargetprogramtoassessitscoverage.
• Implementmechanismstoadjustthegenerationprocesstohitorexceedthecoveragethreshold.
Input Specifications
Yourprogramshouldtake3command-linearguments:
1. ThepathtoaPythonscript
2. ThepathtoasinglePython(.py)scriptcontainingthegrammarspecificationsusingthesyntax
taughtinthelecturesandtutorials
3. Thenumberofstringsyourprogramshouldgenerateforthetestsuite
Itshouldbecalledusingthefollowingcommand:
python grammar_fuzzer.py
Thegrammarfileshoulddefinethestructureoftheinputsusingaspecifiedsyntax.
Page5
Output Specifications
The program should generate structured inputs based on the grammar specifications provided and
write them to an output file. The output file should contain the specified number of strings each on a
new line, where each string represents a test input. The generated inputs should cover various paths
andoptionswithinthegrammar,aimingtohitor exceedthecoveragethresholddefinedforthetarget
program.
Forexample,ifthedesirednumberofstringsis100:
input_1
1
input_2
2
...
3
input_100
4
Makesurethatthegeneratedinputscoverasmanygrammarrulesandoptionsaspossibletoeffectively
testthetargetprogramandmeetthecoveragethreshold.
Getting Started
• Review Tutorials and Lectures: Begin by reviewing the tutorials and lectures. Remember
thateverythingyouneedforeachcomponenthasalreadybeencoveredinthisunit.
• Understand the Fundamentals: Go through the revision slides on Ed and make sure that
youunderstandallofthecontentcoveredsofar.
• Ask Questions: If you have any questions or uncertainties about the material covered, don’t
hesitatetoaskonEdforclarificationandaTAwillgetbacktoyoushortly.
Frequently Asked Questions
• Hardcodingwillresultina0foralltasks.
• No external libraries (i.e. those installed using pip or another package manager) may be
used-thisisalimitationofEdstem.
• Youhaveunlimitedattemptsbeforethedeadline.
• Therearepublic,privateandhiddentestcasesforalltasks.
• Test cases will gradually be released over the coming days, and you should check Ed for
announcements.
• YoumayreusetheircodefromQuiz1andanyothertaskfromthisunit.
• Allcode,evenyourown,mustbereferencedaspertheuniversity’spolicy.
• You may structure your program as you wish as long as it is written in Python and gets
calledusingthedescribedcommands.
Page6
Academic Declaration
Bysubmittingthisassignment,youdeclarethefollowing:
I declare that I have read and understood the University of Sydney Student Plagiarism: Coursework
Policy and Procedure, and except where specifically acknowledged, the work contained in this as-
signment/project is my own work, and has not been copied from other sources or been previously
submittedforawardorassessment.
I understand that failure to comply with the Student Plagiarism: Coursework Policy and Procedure
can lead to severe penalties as outlined under Chapter 8 of the University of Sydney By-Law 1999
(as amended). These penalties may be imposed in cases where any significant portion of my submit-
ted work has been copied without proper acknowledgement from other sources, including published
works, the Internet, existing programs, the work of other students, or work previously submitted for
otherawardsorassessments.
I realise that I may be asked to identify those portions of the work contributed by me and required to
demonstrate my knowledge of the relevant material by answering oral questions or by undertaking
supplementary work, either written or in the laboratory, in order to arrive at the final assessment
mark.
I acknowledge that the School of Computer Science, in assessing this assignment, may reproduce
it entirely, may provide a copy to another member of faculty, and/or communicate a copy of this
assignment to a plagiarism checking service or in-house computer program, and that a copy of the
assignment may be maintained by the service or the School of Computer Science for the purpose of
futureplagiarismchecking.
Page7
