COSC2673/COSC2793 | Semester 1 2022 Machine Learning & Computational Machine Learning Assignment 2

 Machine Learning Project Weight: 50% of the ��nal course mark Due Date: 23:59, Monday 30 May 2022 Learning Outcomes: This assignment contributes to CLOs: 1, 2, 3, 4, 6 1 Introduction 1.1 Summary In this assignment you will design and create an end-to-end machine learning system for a real-world prob- lem. This assignment is designed for you to apply and practice skills of critical analysis and evaluation to circumstances similar to those found in real-world problems. This is a group project (max 2 students). In this assignment you will: • Design and Create an end-to-end machine learning system • Apply multiple algorithms to a real-world machine learning problem • Analyse and Evaluate the output of the algorithms • Research into extending techniques that are taught in class • Provide an ultimate judgement of the ��nal trained model(s) that you would use in a real-world setting This assignment has the following deliverables: 1. A report (of no more than 5 pages, plus up to 2 pages for appendices) critically analysing your approach and ultimate judgement. 2. An independent evaluation of your model and ultimate judgement (to be included in the report). 3. Your Python scripts, Jupyter notebooks, and software used to build your learning system and produce the models and results. To complete this assignment, you will require skills and knowledge from lecture and lab material for Weeks 1 to 12 (inclusive). You may ��nd that you will be unable to complete some of the activities until you have completed the relevant lab work. However, you will be able to commence work on some sections. Thus, do the work you can initially, and continue to build in new features as you learn the relevant skills. A machine learning learning model cannot be developed within a day or two. Therefore, start early. 1 1.2 Learning Outcomes This assignment contributes to the following course CLOs: • CLO 1: Understand the fundamental concepts and algorithms of machine learning and applications. • CLO 2: Understand a range of machine learning methods and the kinds of problem to which they are suited. • CLO 3: Set up a machine learning con��guration, including processing data and performing feature engineering, for a range of applications. • CLO 4: Apply machine learning software and toolkits for diverse applications. • CLO 6: Understand the ethical considerations involved in the application of machine learning. 1.3 Group Work You may optionally conduct this assignment in pairs (groups of 2). This is because many of the interesting problems can be far more thoroughly investigated by working in pairs. The marking rubric does not evaluate group work or di��erentiate between those working individually or in groups. If you work in a group, both members will receive the same mark. Groups must be formed by no later than Week 9, and registered on Canvas. If there are issues with group work, you must inform the course co-ordinator by no later than 5.00pm Friday of Week 11, otherwise both group members will receive the same grade, with no exceptions. Your group must have a private Git repository that hosts your group's code. This may be on BitBucket or Github. Your group must add your tutor(s) to this repository3. This git repository will be used as the evidence of your individual contribution to your group. 1.4 Academic integrity and plagiarism (standard warning) ! Plagiarism is a very serious o��ence. Academic integrity is about honest presentation of your academic work. It means acknowledging the work of others while developing your own insights, knowledge and ideas. You should take extreme care that you have: • Acknowledged words, data, diagrams, models, frameworks and/or ideas of others you have quoted (i.e. directly copied), summarised, paraphrased, discussed or mentioned in your assessment through the ap- propriate referencing methods • Provided a reference list of the publication details so your reader can locate the source if necessary. This includes material taken from Internet sites. If you do not acknowledge the sources of your material, you may be accused of plagiarism because you have passed o�� the work and ideas of another person without appropriate referencing, as if they were your own. RMIT University treats plagiarism as a very serious o��ence constituting misconduct. Plagiarism covers a variety of inappropriate behaviours, including: • Failure to properly document a source • Copyright material from the internet or databases • Collusion between students For further information on our policies and procedures, please refer to the following: https://www.rmit.edu.au/ students/student-essentials/rights-and-responsibilities/academic-integrity. 2 2 Task Using machine learning in real-world settings involves a more than just running a data set through a particular algorithm. In this assignment, you will design, analyse and evaluate a complete machine learning system. The key aspect of this assignment is the design, analysis, and evaluation of your methodology, investi- gation, and results. This assignment focuses on both the accuracy of your model, and your understanding of your approach and model. For this assignment you have a choice of your project. You may select this project from the list in Section 3, or you may negotiate a project with the course co-ordinator. Regardless of the problem you choose, you must conduct the following tasks: 1. You need to come up with an approach, where each element of the system is justi ed using data analysis, performance analysis and/or knowledge from relevant literature. 2. Investigate various Machine Learning solutions to the problem 3. Make an ultimate judgement 4. Evaluate your ultimate judgement against independent testing data 5. Produce a report of your design, investigation, evaluation and ��ndings 2.1 Investigation Your investigation will require you to design, use, analyse and evaluate an end-to-end machine learning system. You should consider a variety of techniques that have been discussed in class, and techniques you have researched. Your end-to-end system may consist of elements such as: • A well justi��ed evaluation framework. • Pre-processing the data set to make it suitable for providing to various machine learning algorithms. • Carefully selected and justi��ed baseline model(s). • Hyper-parameter setting and tuning to re��ne the model. • Evaluating the trained models, analysing and interpreting the results. Each project features many of these above aspects. Each project also has unique aspects which cover a sample of issues from across machine learning. Additionally, each project has unique mandatory requirement(s), detailed for each project. The details of each project are listed in Section 3. The details in this spec are the minimum requirements. A thorough investigation must consider more ! that the minimum to receive high grades. 2.2 Ultimate Judgement You must make an ultimate judgement of the ��best�� model that you would use and recommend for your particular project. It is up to you to determine the criteria by which you evaluate your model and determine what is means to be ��the best model��. 2.3 Independent Evaluation of your Ultimate Judgement You need to conduct an independent evaluation of your ultimate judgement, using data collected completly outside of the scope of your original training and evaluation. This evaluation simulates how your ultimate judgement would perform if it were deployed in a real-world setting, where you are unable to re-train and adjust the model. ! 3 2.4 Approach, Critical Analysis & Report You must compile a report analysing the approach you have taken in your investigation. Your report: • Must be no longer that 5 pages of text • May contain an additional 2 pages for appendices • Use a single-column layout with no less than size 11pt font • The appendices may only contain citations, ��gures, diagrams, or data tables that provide evidence to support the statements in your report. • Include the name(s) and student id's of the student(s) who wrote the report. Any over length content, or content outside of these requirements will not be marked. For example, if ! you report is too long, ONLY the ��rst 5 pages pages of text will be read and marked. In this report you should analyse elements such as: • Machine learning algorithms that you considered • Why you selected these approaches • Evaluations of the performance of trained model(s) • Your ultimate judgement with supporting analysis and evidence This will allow us to understand your rationale. We encourage you to explore this problem and not just focus on maximising a single performance metric. By the end of your report, we should be convinced that of your ultimate judgement and that you have considered all reasonable aspects in investigating your chosen problem. ! The key aspect of this assignment isn't your code or model, but the thought process behind your work. Remember that good analysis provides factual statements, evidence and justi��cations for conclusions that you draw. A statements such as: ��I did <xyz> because I felt that it was good�� is not analysis. This is an unjusti��ed opinion. Instead, you should aim for statements such as: ��I did <xyz> because it is more e��cient. It is more e��cient because ...�� 3 Pro jects For your project: 1. You may choose from one of the below 3 suggested projects. Each project has unique aspects and will allow you to explore di��erent aspects of the ML ��eld. 2. You may negotiate your own project. Note the special requirements and timeline for negotiating a project. 3.1 Suggested Projects Each project has di��erent requirements, so ensure you are aware of these di��erences. Project 1: Classify Images of Road Tra��c Signs This project is to train a model to classify images of European road tra��c signs. You will be using a modi��ed version of the Belgium Tra��c Sign Classi��cation Benchmark. These are images of road tra��c signs taken from real-world vehicles. Note, this dataset, along with it's sister German TSC dataset, appear in many di��erent forms on various research and ML online resources. The data set for you to use in this assignment has been speci��cally prepared for you, and is provided on Canvas. The dataset consists of 28x28 gray-scale images and you are expected to use the dataset to perform two tasks: • Classify images according to sign-shape, such as diamond, hex, rectangle, round, triangle. • Classify images according to sign-type, such as stop, speed, warning, parking, etc. 4 The correct classi��cation of the images is given by the image sub-directories. Images are ��rst sub-divided by their shape, and then by their sign type. You should also note that some sign types have di��erent individual signs. For example, the speed-sign type has examples of signs of speeds from 10 - 70 mph. You are not required to further sub-divide these signs, but consider all of the di��erent signs as being of the sample type. Your tasks is to investigate classifying the signs using both categories. Requirements • You must investigate at least one supervised machine learning algorithms for each of the two categories (Tasks). That is, you must build at least one model capable of classi��ed the shape of the sign, and at least one model capable of classifying the type of the sign. • You are not required to use separate type(s) of machine learning algorithms, however, a thorough insti- gations should consider di��erent types of algorithms. • You are required to fully train your own algorithms. You may not use pre-trained systems. • You may NOT augment this data set with additional data. • Your ��nal report must conduct an analysis and comparison between classifying the two categories. Independent Evaluation • Your independent evaluation should consist of classifying images of tra��c signs that you have collected. You will need to either take your own digital photographs of tra��c signs, and/or source suitable signs from internet resources. You will need to process these images so they may be used with your trained algorithms. • As part of your evaluation, you should discuss challenges you face in combining this independent data and your models. Project 2: Predict Energy Use This project is to predict the energy use of lights and appliances in a low-energy building using the UCI data set collected in 2017. The dataset for this project is available on Canvas. You should note, the target variable is the total energy use. Requirements • You must investigate, at a minimum, two completely separate types of supervised machine learning algo- rithms, of which at least one method must be a non neural network based algorithm. • You are required to fully train your own algorithms. You may not use pre-trained systems. • Unlike other projects, this data set contains time-series information. To improve your model you may ��nd it necessary to construct additional features to represent the time series information. • You may use feature selection and ensemble learning method, however, these do not count towards the above requirement. Independent Evaluation • As you don't have access a low-energy house, the independent evaluation of this project is di��erent to the other projects. A fundamental feature of the scienti��c process is reproducing existing work, and comparing new results against exist work. • The original data set was published publicly as part of the following paper: Luis M. Candanedo, Veronique Feldheim, Dominique Deramaix, Data driven prediction models of energy use of appliances in a low-energy house, Energy and Buildings, Volume 140, 1 April 2017, Pages 81-97, ISSN 0378-7788 Since publication, a number of papers, and online resources, have been published that use this data set. • Your independent evaluation is to research a number of these published works. Then you must evaluate, compare and contrast your results to these other works. • As part of your evaluation, you should discuss challenges you face in your independent comparison. Project 3: Learning to Switch Tra��c Lights This project is to use reinforcement learning to improve the throughput of tra��c at an intersection with tra��c lights. Most tra��c lights operate on a predetermined phasing (that is, a timing when the lights change), which does not take into account the number of vehicles that are waiting at, or approaching, the intersection. This leads to a sub-optimal throughput of vehicles. For example, some vehicles may be waiting at a red light while the intersection is otherwise clear. A more intelligent policy, can increase the throughput. This project is to use reinforcement learning to ��nd a policy to control the switching of tra��c lights at a simulated intersection, to improve the tra��c throughput. You will need to demonstrate and compare your 5 learnt policy to a more traditional static phasing policy. Requirements • This project will require you to build a simple simulation to conduct the reinforcement learning. It is advisable to build a simulation system that has con��gurable parameters for the ��ow of tra��c. • The simulated intersection must include at least one right turning lane. An example intersection would be the intersection at Victoria Street - Lygon Street. • Due to above work, you only need to use a single form of reinforcement learning to train an optimal policy. • However, you will need to also test the throughput of a static policy, and devise a suitable static policy that might be employed at a real-world intersection. Independent Evaluation • For your independent evaluation you must model a real-world intersection. • You will need to observe the intersection and collect basic information, such as the average throughput, and the rate at which tra��c arrives at the intersection. • You will need to approximate the static control policy of the lights. • You will need to simulate both the static control policy, and learn a suitable policy for this intersection. • We suggest choosing a simple intersection, and observing it over a short window of around 10-15 minutes. You may also wish to simplify the problem by excluding right-turn signals. • In case the social distancing measures do not allow you to observe a simple intersection, some statistics for a test intersection will be provided to you. 3.2 Negotiated Project You may propose and negotiate a project and machine learning problem to investigate, with the course co- ordinator. This project must meet a number of constraints: • The project must be of a suitable complexity and challenge that is similar to the suggested projects. As part of the negotiation, the scope and deliverables of the project will be set. • If the project is using an existing data set, the problem should be phrased in a manner that can be solved by multiple machine learning methods, of which at least two methods will be investigated. • If the project requires a data set to be generated, devised, or collected, this collection should require su��cient e��ort. In these cases, especially for reinforcement learning tasks, only one machine learning method may need to be investigated. • The proposed project must be independent of previously or concurrently assessed work. You may not conduct a project if you have already been assessed on the work, or are concurrently being ! assessed on the work. In general, negotiations will take place via email, during consultation hours, or by appointment. Please note, that the course co-ordinator is not available outside of business hours. All negotiated projects must be ��nalised by no later than 5pm, Friday 9 May. This is the absolute deadline. If you wish to conduct a negotiated project, begin the negotiation process early. A negotiated project may be denied before the deadline if there is insu��cient time for the negotiation process. 4 Additional Information 4.1 Sources of Help Most questions should be asked on Canvas, however, please do not post any code. There is a FAQ, and anything in the FAQ will override what is speci��ed in this speci��cations, if there is ambiguity. Your lecturer is happy to discuss questions and your results with you. Please feel free to come talk to us during consultation, or even a quick question, during lecture break. 6 4.2 Marking Rubric A detailed rubric is attached on canvas. In summary: • Approach 60%; • Ultimate Judgment & Analysis (Independent Evaluation) 20%; • Report Presentation 20%. 4.3 Submission Instructions You must submit all the relevant material as listed below via Canvas. 1. A report (of no more than 5 pages, plus up to 2 pages for appendices) critically analysing your approach and ultimate judgement 2. An independent evaluation of your model and ultimate judgement (included in the report). 3. Your Python scripts, Jupyter notebooks, and software used to build your learning system and produce the models and results. The submission portal on canvas consists of two sub-pages. First page for report submission, the second page for code submission. More information is provided on Canvas. Include only source code in a zip ��le containing your name. We strongly recommend you to attach a README ��le with instructions on how to run your application. Make sure that your assignment can run only with the code included in your zip ��le! 4.4 Late Assessment Policy A penalty of 10% of the maximum mark per day (including weekends) will apply to late assignments up to a maximum of ��ve days. Assignments will not be marked after this time. 7