1
CSEE4119F24ComputerNetworks
Project1:VideoCDN
0.DeadlinesandUpdates
PreliminaryStageDue:October9th,11:59pm
Updates:
0.DeadlinesandUpdates1
1.Overview2
1.1IntheRealWorld2
1.2YourSystem2
1.3Groupsandcollaborationpolicy4
2.Preliminarystage4
2.1Requirements4
2.2Getyourconnectionsright.5
2.3Forwardingprotocol.5
2.4Runningtheproxy6
2.5Testitout!7
2.6Finalresult7
2.7WhattoSubmitforpreliminarystage8
2.8WheretoSubmit9
3.Finalstage:VideoBitrateAdaptation9
4.DevelopmentEnvironment9
4.1VirtualMachine9
4.2StarterFiles(forfinalstage)11
4.3NetworkSimulation(forfinalstage)11
4.4Apache11
4.5ProgrammingLanguageandPackages11
5.Grading12
Academicintegrity:Zerotoleranceonplagiarism13
2
1.Overview
Inthisproject,youwillexploreaspectsofhowstreamingvideoworks,aswellas
socketprogrammingandHTTP.Inparticular,youwillimplementadaptivebitrate
selection.Theprogramminglanguagesandpackagesarespecifiedinthe
developmentenvironmentsection.
Wewilldothisinmultiplestages:
1.Preliminarystage:buildingasimpleproxy
2.Intermediatestage:requestingandreceivingvideochunks(TBD)
3.Finalstage:implementingadaptivebitratestreaming
1.1IntheRealWorld
Figure1depicts(atahighlevel)whatthissystemlookslikeintherealworld.
ClientstryingtostreamavideofirstissueaDNSquerytoresolvetheservice’s
domainnametoanIPaddressforoneofthecontentserversoperatedbyacontent
deliverynetwork(CDN).TheCDN’sauthoritativeDNSserverselectsthe“best”
contentserverforeachparticularclientbasedon(1)theclient’sIPaddress(from
whichitlearnstheclient’sgeographicornetworklocation)and(2)currentloadon
thecontentservers(whichtheserversperiodicallyreporttotheDNSserver).
OncetheclienthastheIPaddressforoneofthecontentservers,itbegins
requestingchunksofthevideotheuserrequested.Thevideoisencodedatmultiple
bitrates.Astheclientplayerreceivesvideodata,itcalculatesthethroughputofthe
transferandmonitorshowmuchvideoithasbufferedtoplay,anditrequeststhe
highestbitratetheconnectioncansupportwithoutrunningoutofvideointhe
playbackbuffer.
1.2YourSystem
ImplementinganentireCDNisclearlyatallorder,solet’ssimplifythings.First,your
entiresystemwillrunononehost;we’reprovidinganetworksimulatornetsim
3
(describedinDevelopmentEnvironment)thatwillallowyoutorunseveralprocesses
witharbitraryIPaddressesononemachine.Oursimulatoralsoallowsyoutoassign
arbitrarylinkcharacteristics(bandwidthandlatency)tothepathbetweeneachpair
of“endhosts”(processes).Forthisproject,youwilldoyourdevelopmentandtesting
usingavirtualmachine(VM)weprovide.
Browser.You’lluseanoff-the-shelfwebbrowser(e.g.Firefox)toplayvideosserved
byyourCDN(viayourproxy).
Proxy.Ratherthanmodifythevideoplayeritself,youwillimplementadaptivebitrate
selectioninanHTTPproxy.TheplayerrequestschunkswithstandardHTTPGET
requests.Yourproxywillintercepttheseandmodifythemtoretrievewhichever
bitrateyouralgorithmdeemsappropriate.Tosimulatemultipleclients,youwill
launchmultipleinstancesofyourproxy.
WebServer.Videocontentwillbeservedfromanoff-the-shelfwebserver(Apache).
MoredetailisintheDevelopmentEnvironmentsection.Aswiththeproxy,youcan
runmultipleinstancesofApacheondifferentfakeIPaddressestosimulateaCDN
withseveralcontentservers.However,intheassignment,ratherthanusingDNS
redirectionlikeaCDNwould,theproxywillcontactaparticularserverviaitsIP
address(withoutaDNSlookup).Apossible(ungraded)futureextensiontothe
projectcouldincludeimplementingaDNSserverthatdecideswhichservertodirect
theproxyto,basedondistanceornetworkconditionsfromaproxytovariousweb
servers.
4
Theprojectisbrokenupintotwostages(plustheinitialsetuptogetyoureadyfor
thestages):
●Inthepreliminarystage,youwillimplementasimpleproxythatsequentially
handlesclientsandpassesmessagesbackandforthbetweenclientandserver
withoutmodifyingthemessages.
●Inthefinalstage,youwillextendtheproxytoimplementthefullfunctionality
describedabove,withtheproxymodifyingHTTPrequeststoperformbitrate
adaptation.
1.3Groupsandcollaborationpolicy
Thisisanindividualproject,butyoucandiscussitataconceptuallevelwithother
studentsorconsultInternetmaterial(excludingimplementationsofPythonproxies),
aslongasthefinalcodeandconfigurationyousubmitiscompletelyyoursandas
longasyoudonotsharecodeorconfiguration.Beforestartingtheproject,besure
toreadthecollaborationpolicyattheendofthisdocument.
2.Preliminarystage
DonotstartuntilyouhavefinishedsettingupyourVMasdescribedin:
CSEE4119 F24 Tutorial for Setting Up Project 1 VM
Youwillbeimplementingasimpleproxythatacceptsclientconnectionssequentially
(i.e.handlesaclient,and,onceitdisconnects,takescareofthenextclient).Inlater
stages,yourproxywillberequiredtohandleclientconnectionsconcurrently.
Inthispreliminarystage,theproxydoesNOTneedtomodifyanymessagesit
receives,asitwilljustrelaythemessagesbackandforth.Inlaterstages,youwill
enhanceyourproxytomodifymessagesinordertoperformadaptivebitrate
selection.
2.1Requirements
Implementaproxythatforwardsmessagesofanylengthbetweenaclientanda
server.See‘Getyourconnectionsright’and‘Forwardingprotocol’fordetails.Note
5
thattheVMinstanceisrequiredforthefinalstage,butnotthepreliminarystage.You
canworkonthepreliminarystagewithyourlocaldevices.
2.2Getyourconnectionsright.
Yourproxyshouldacceptconnectionsfromclientsandthenopenupanother
connectionwithaserver(seeHowtoruntheproxy).Oncebothconnectionsare
established,theproxyshouldforwardmessagesbetweentheclientandserver.
Youshouldimplementthisintwosteps:
a.Establishaconnectionwithaclient:
YourproxyshouldlistenforconnectionsfromaclientonanyIPaddressonthe
portspecifiedasacommandlineargument(seeHowtoruntheproxy).Your
proxyshouldacceptmultipleconnectionsfromclients.Itisnotrequiredtohandle
themconcurrentlyfornow.Simplyhandlingthemonebyone,sequentially,will
beenough.
b.Establishaconnectionwithaserver:
Oncetheproxygetsconnectedtotheclient,itshouldthenconnecttotheserver.
TheserverIPisprovidedasacommandlineargument.Asfortheportnumber,
use8080.Makesuretocloseconnectionstotheclientandserverwheneitherof
themdisconnects.
(Figure4)Preliminarystructure
2.3Forwardingprotocol.
The“messages”thattheproxyforwardsfollowaparticularstructure(forexample,in
HTTP,youknowthatthereisaheaderandabody).Thisstructureisimportant,as
6
therecipientofthemessagecanknowwheretolooktogetaspecificpieceof
information.Forthepreliminarystage,wekeepourmessagestructureverysimple:
(Figure5)Structureofamessage
Themessagehasabody,andanEndOfMessage(EOM)symbolthatindicatesthe
endofthemessage.Inourcase,wedefineourEOMasthenewlinecharacter‘\n’.
Pleasenotethatdetecting‘\n’isdifferentfromdetectingtheslashcharacter‘\’and
theletter‘n’.
Amessagehastobefullyreceivedbytheproxybeforebeingforwardedtotheother
side.
Hereishowtheforwardingprotocolworksinourcase:
1.Theproxygetsamessagefromtheclientandforwardsittotheserver
2.Theproxyexpectsaresponsefromtheserver,getsit,andforwardsittothe
client
Animportantthingtonoticehereisthatthereisnoasynchronousforwarding(i.e.,
theproxydoesn’tsimplyforwardanymessage,itfirstwaitsforamessagefromthe
client,andthenwaitsforaresponsefromaserver).
2.4Runningtheproxy
YoushouldcreateanexecutablePythonscriptcalledproxyinsidetheproxy
directory(seebelowforadescriptionofthedevelopmentenvironment),which
shouldbeinvokedasfollows:
cd~/csee_4119_abr_project/proxy
./proxy
listen-port:TheTCPportyourproxyshouldlistenonforacceptingconnections
fromtheclient.
7
fake-ip:YourproxyshouldbindtothisIPaddressforoutboundconnectionstothe
server.YoushouldnotbindyourproxylistensockettothisIPaddress—bindthe
listensockettoreceivetraffictothespecifiedportregardlessoftheIPaddress.(i.e.
bycallingmySocket.bind((“”,
Importantnote:Theaboveisaprettyunusualthingtodo.Youmightthink“inlecture,
wesawthattheclientsocketdoesn’tbind,andnow,youaretellingustobindthe
proxy’soutboundsocketwhenconnectingtotheserver”.However,itisnecessary
fortheFinalStage’snetworksimulatortoworkproperly,andsowewilladditinthis
stage.
server-ip:TheIPaddressoftheserver
SeeinstructionsformakingyourscriptexecutableinthesectionHandIn.
2.5Testitout!
Youcantestparts“Getyourconnectionsright”and“Forwardingprotocol”ofyour
proxyimplementationbyusingthenetcattool(ncornetcat,whichisinstalledinthe
VM)presentedinclass,usingbothanetcatclientandanetcatserver.Youshouldbe
abletosendamessagefromtheclientandseeitappearontheserverside.Then,
anyresponsesentfromyourservershouldalsoappearontheclient.Forthefake-ip,
youcanindicate127.0.0.1(localhost)whentestingwithnetcatinstancesthatare
createdonyourmachine.
Rememberthatseeingamessageontheserversidedoesnotmeanyour
implementationis100%correct!Pleasebesuretocomeupwithyourowntest
casesandmakesurethatyourproxyisasexpectedaswedescribedinFinal
Result.
2.6Finalresult
Yourproxyshouldbeabletoforwardamessageofanylengthfromaclienttothe
server,andinturn,forwardtheresponsefromtheserverbacktotheclient.Itshould
supportback-and-forthmessagesuntilonesideclosestheconnection.Afterthe
8
connectionisclosed,itshouldbeabletoacceptanewconnectionfromaclient.You
shouldbeabletotestthebehaviorofyourapplicationbycreatingnetcatinstances.
Notethatthisversionsimplyforwardsmessageswiththestructurespecifiedin
Figure2.Thenextstagewillhaveadifferentmessagestructure:theHTTPmessage
structure,andyou’llhavetoadaptyourproxybasedonyourknowledgeofHTTP
messages.
2.7WhattoSubmitforpreliminarystage
PLEASEPAYATTENTIONTOTHEHANDINSTRUCTURE,ASEVENATYPO
WILLCAUSETHEGRADERTOBREAK,WHICHCANMAKEYOULOSE10
POINTS.
Youwillsubmityourprojectasazippedfilenamed
fileshouldgiveusadirectorynamedhandinwhichshouldonlycontainthe
following:
●proxy—Adirectorynamedproxycontainingonlyyoursourcecode.Thecode
thatyouwanttoexecuteshouldbeanexecutablenamedproxy,asdescribedin
2.3Howtoruntheproxy.Tomakethecodeexecutable,followthesesteps:
1.Add‘#!/usr/bin/envpython3.10’tothetopofyourproxyPythonfile
2.Run‘chmod755proxy’toensurethatthefilehasthecorrectpermissions
tobeexecutablebyus.
(Figure6)Preliminarystagesubmissionfilestructure.
Youmayorganizeyourcodewithintheproxydirectoryasyouseefit.Partofyour
grademaybebasedonhowunderstandable/organized/well-explainedyourcodeis,
butwedonotrequireanyparticularorganizationaswellasitiswell-organized.
9
2.8WheretoSubmit
YouwillsubmityourcodetoGradescope.Ifyouhaveanyquestionsaboutit,please
letusknowASAP.
3.Finalstage:VideoBitrateAdaptation
Detailstobereleasedlater.
4.DevelopmentEnvironment
Fortheproject,weareprovidingavirtualmachine(VM)pre-configuredwiththe
softwareyouwillneed.Westronglyrecommendthatyoudoalldevelopmentand
testinginthisVM;yourcodemustruncorrectlyonthisimageaswewillbeusingit
forgrading.Forexample,somestudentsinpreviousyearsdecidedtowritetheir
codeontheirWindowsenvironment,whichchangedthecontrolcharacterstoCLRF
(UnixusesLF,thusourgradercouldnotruntheircode).Pleasemakesureyour
codeusesLF.ThissectiondescribestheVMandthestartercodeitcontains.
4.1VirtualMachine
WeprovideanimageonGCP(GoogleCloudPlatform)andyoucancreateavirtual
machine(VM)instancebasedonit.PleasefollowthetutorialtosetupyourVM
instanceanddoallyourtestingthere.
IntheeventyouneedtomovefilesbetweentheVMandyourcomputer(e.g.,for
submission),thereareafewoptions.
1.(recommended)Createa(private)Githubrepositorywiththerelevantproject
files.YoucanpushtoyourrepositoryfromtheVMandaccessthefilesfrom
anywhere.
2.IfyouSSHintoyourVMinstanceusingtheGCPdefaultoption(theSSHbutton
onthesamerowoftheinstance),onthetopofyourSSHwindow,therearetwo
arrowbuttonswhichuploadanddownloadfiles.
10
3.Launchyourremotedesktop.Youcanusesidebaroptionstouploadand
downloadfiles.Alternatively,sendthefilesviatheInternet(email,GoogleDrive).
FirefoxisinstalledontheVM.
11
4.2StarterFiles(forfinalstage)
Detailstobereleasedlater.
4.3NetworkSimulation(forfinalstage)
Detailstobereleasedlater.
4.4Apache
Detailstobereleasedlater.
4.5ProgrammingLanguageandPackages
ThisprojectmustbeimplementedinPython3.YourVMinstancecomeswith
Pythonversion3.10.12,whichistheversionwewillusetotestyourcode.Ifthis
choiceoflanguageposesasignificantproblemforyou(i.e.,youhaveneverused
Pythonbefore),pleasecontacttheinstructors.
Torunpython3.10.12ontheVMinstance,pleaseusethefollowingcommand:
python3
Toinstallpythonpackagesforversion3.10.12ontheVMinstance,pleaseuse:
python3-mpip
Forthisproject,youareallowedtousethefollowingpythonpackages:
sys,socket,threading,select,time,re,numpy,subprocess
Usinganunallowedpackageinyourcodemayresultinnocreditbeinggiven.Other
thanthepackageslisted,youmayonlyusethepackageifyouaskonEdDiscussion
andaTAortheprofessorexplicitlyrespondstoyourrequestapprovingtheuse.We
willmaintainapinnedEdposttitled"Project1ListofApproved(andDisallowed)
Packages",sopleasecheckthatpostbeforepostingyourrequest.Ifyouwouldlike
touseapackagenotmentionedhereandareunsureifitwouldbeacceptable,
pleaseaddanewfollowupdiscussionundertheabovementionedpinnedposton
Edatleast3daysinadvanceoftheprojectdeadline.
12
Inyourfollowupdiscussion,youmustmentionthepackagename,thepackage
version,andalinktotheofficialrepositoryofthepackage(e.g.
http://pypi.python.org/pypi).TAswillexamineyourrequestanddetermineifthe
packageisallowedandaddittothelistofallowed/disallowedpackages.
5.Grading
Yourgradewillconsistofthefollowingcomponents:
Proxy(70points)
●Preliminarystageproxying[15pts]
●Finalstageproxyingrunsonbrowser[10pts]
●Finalstageproxy-implementingEWMAthroughputestimator&bitrate
adaptation[33pts]
●DNSServer-correctresponsesandimplementationofwebserverselection[10
pts]
●Codeexecutesasinstructed[-10ptsifwehavetomanuallydebugthings]
Writeup(20points)
●Plotsofutilization,fairness,andsmoothnessfor[10pts]
α∈{0.1, 0.5, 0.9}
●Discussionoftradeoffsforvarying[10pts]
α
Style(10points)
●Codethoroughlycommented
●Codeorganizedandmodular
●READMElistingyourfilesandwhattheycontain
PleasemakesureyourcoderunsasanexecutableandasdescribedinRunningthe
Proxy(PreliminaryStage,FinalStage)beforesubmitting.Codethatdoesnotrun
mayreceiveazeroontheassignment.
WeWILLNOTreleaseourgradingscripts,norspecificallytellyouwhatwewilltest
for.Besuretostresstestyourcodeinavarietyofscenariosusingunexpected
inputs.BesuretoPRECISELYfollowtheinstructionsinthisdocumentforproxy
development:implementexactlywhatisaskedofyou,nomoreandnoless.For
example,donotimplementcommandlineargumentvalidation,wedidn’taskyouto.
13
Academicintegrity:Zerotoleranceonplagiarism
TherulesforColumbiaUniversity,theCSDepartment,andtheEEDepartment(via
SEAS:1and2)apply.Itisyourresponsibilitytocarefullyreadthesepoliciesandask
theprofessor(viaEd)ifyouhaveanyquestionsaboutacademicintegrity.Please
asktheprofessorbeforesubmittingtheassignment,withenoughtimetoresolvethe
issuebeforethedeadline.Amisunderstandingofuniversityorclasspoliciesisnot
anexcuseforviolatingapolicy.
Thisclassrequirescloselyobeyingthepolicyonacademicintegrity,andhaszero
toleranceonplagiarismforallassignments,includingbothprojects/programming
assignmentsandwrittenassignments.Byzerotolerance,wemeanthatthe
minimumpunishmentforplagiarism/cheatingisa0fortheassignment,andallcases
willbereferredtotheDeanofStudents.
Thisassignmentmustbecompletedindividually.Forprogrammingassignments,in
particular,youmustwriteallthecodeyouhandinyourself,exceptforcodethatwe
giveyouaspartoftheassignments.Youarenotallowedtolookatanyoneelse's
solution(includingsolutionsontheInternet,ifthereareany),andyouarenot
allowedtolookatcodefrompreviousyearsoraskpeoplewhotooktheclassin
previousyearsforhelp.Youmaydiscusstheassignmentswithotherstudentsatthe
conceptuallevel,butyoumaynotwritepseudocodetogether,orlookatorcopy
eachother'scode.Helpingotherstudentsviolatethepolicy(forexample,letting
themlookatyourcode)isaviolation,evenifyoucompletedthecodeyourself.
Pleasedonotpublishyourcodeormakeitavailabletofuturestudents--for
example,pleasedonotmakeyourcodevisibleonGithub.Uploadingcourse
materialstositessuchasCourseHero,CheggorGithubisacademicmisconductat
Columbia(seepg10).
Youmaylookatdocumentationfromthetools’websites.However,youmaynotuse
externallibrariesoranyonlinecodeunlessgrantedexplicitpermissionbythe
professororTA.Forwritten(non-programming)answers,ifyouquotematerialfrom
textbooks,journalarticles,manuals,etc.,youmustincludeacitationthatgives
propercredittothesourcetoavoidsuspicionofplagiarism.Ifyouareunsurehowto
properlycite,youcanusethewebtofindreferencesonscientificcitations,orask
fellowstudentsandTAsonEd.
14
YouarenotallowedtouseGitHubCopilot,Kite,orsimilartools.UsingAI-assisted
codingtools(suchasGitHubCopilot)isconsideredacademicdishonesty.
Since"AI-assisted"canbesomewhatvague,Iwillclarifythat"AI-assistedtools"
includes,butisnotlimitedto,anycodingtoolsthatrequireaccesstotheinternetor
morethan2MBofdataforsemanticanalysis(wearesettingthesizebarratherhigh
becausetheGCCexecutableisinexplicablylarge;theintentionisthatyoushould
notbeperforminganalysesusingasizabledataset).
Foreachprogrammingassignment,wewillusesoftwaretocheckforplagiarized
code.
Note:Youmustsetpermissionsonanyhomeworkassignmentssothattheyare
readableonlybyyou.Youmaygetreprimandedforfacilitatingcheatingifyoudonot
followthisrule.
