RMIT Classification: Trusted Machine Learning & Computational Machine Learning COSC 2673 & COSC 2793 Assignment 3 

Due Date Week 14, Friday 10th June 2022, 11:59pm Marks 20% 

1 Overview This assignment is designed to help you become more confident in designing machine learning systems. In this assignment you will conduct a virtual presentation (pre-recorded), presenting a brief summary and critical analysis of the project work that is done in Assessment Task 2, as well as improvements/extensions that could be made for his/her own work based on a literature review of the state-of-the-art approaches. The assignment will consist of: • Critical analysis of an ML technique applied to solve a real-world ML problem. • Conduct a review to identify the state-of-the-art approaches relevant for a particular problem. • Compare and contrast your approach with state of the art approaches to solve similar problems. • Research how to extend the modelling techniques that are taught in class. • Demonstrate theoretical understanding of ML techniques. To complete this assignment, you will require skills and knowledge from lecture and lab material for Weeks 1 to 12 (inclusive). This assignment has ONE deliverable: 1. A recorded video presentation. Assessment Type Individual assignment. Submit online via Canvas → Assignments → Assignment 3. Marks awarded for meeting requirements as closely as possible. Clarifications/updates may be made via announcements/relevant discussion forums. 3 RMIT Classification: Trusted 2 Learning Outcomes This assessment relates to the following course learning outcomes (CLOs): • • • • CLO 1: Understand the fundamental concepts and algorithms of machine learning and applications CLO 2: Understand a range of machine learning methods and the kinds of problem to which they are suited CLO 5: Understand major application areas of machine learning CLO 6: Understand the ethical considerations involved in the application of ma- chine learning. Assessment details 3.1 Task Using machine learning in real-world settings involves more than just running a data set through a particular algorithm. You need to have skill in researching and understanding what others have done to solve similar problems. In this assignment, you will do a critical analysis of a ML project, as well as improvements/extensions that could be made based on a literature review of the state-of-the-art approaches. You will also demonstrate theoretical understanding of ML techniques. The projects you are to investigate for this assignment is given in Section 4. There are two projects and you should select the same project you worked on for assignment 2. Irrespectiveoftheprojectyoushouldrecordapresentationlessthan11minuteslong that includes the following: • (approximately 2 minutes) Present a critical analysis of the model(s) that you have developed. • (approximately 5 minutes) A review of the techniques in literature that are used to solve same/similar problems and compare and contrast your approach tothe approaches found in your review. • (approximately 3 minutes) Discuss how the project can be extended to cover some challenging scenarios. The challenging scenarios that can be considered for each project can be found in section 4. You may use slides in your presentation. However, content on the slides that are not adequately explained (verbally) would not receive any marks. The instructions for recording a presentation is provided separately on the assignment 3 canvas page. Please follow the instructions carefully and make sure the access privileges are configured correctly so that the markers can access the recording. Incorrectly submitted/corrupted recordings or recordings that cannot be accessed will receive zero marks. If the recorded video is over 11 minutes long. Only the part up to 11 minutes will be accessed. The over length part of the recording will be ignored. 2 RMIT Classification: Trusted 3.2 Critical Analysis The first part of your presentation should include a critical analysis of the solution that you developed in assignment 2. The analysis may include: • A brief summary of your solution to the problem. • Other approaches considered while developing the solution. • Analyze the model and its outputs. • Limitations of the model you developed. • Why do you think the approach is adequate to solve the problem. 3.3 Review Next, you should conduct a review to identify the following items and discuss your findings in the presentation. You review should identify: • Other works in literature that solves the same/similar problem as the one you have solved. • Other techniques and algorithms that can be used to solve the problem given to you. See Section 4 for specific details relevant to your project. A good literature review: • Should follow a logical structure. • Should not just provide a list of related papers. Papers should be discussed in relation to why are they relevant for this problem, what is good about them and what are the limitations. • Should discuss literature from peer-reviewed sources. Wikipedia or web discussion forums are not considered as peer-reviewed sources. Remember that good literature review provides factual statements that summaries the work in literature in a way that is useful for the listener to understand the context and follow your rational. Statements such as: “Ying did <xyz>. Chi did <xyz>. Wei did <xyz>” is not a literature review. This is an annotated bibliography. Instead, you should aim for statements such as: “To solve the problem <xyz> Ying did <xyz>, while this method can handle <xyz> it has the limitations <xyz>. To overcome this limitation, Chi proposed <xyz> ...” 3 RMIT Classification: Trusted 3.4 Extensions Finally you should discuss how your system can be extended to handle the specific situation given under section 4. This should include a justified problem formulation and well throughout approach. Your discussion should also include how you would know if the proposed solution is adequate to the task. No experiments (implementations) are needed at this stage. While, this is not the review section and you need to fulfill the requirement (mentioned above: justification, formulation, and discussion), you are allowed to refer to the papers included in the review sections (with proper referencing). 4 Projects The projects you are to investigate for this assignment is given in this section. There are Three projects and you should select the same project you worked on for assignment 2. Each project has different requirements, so ensure you are aware of these differences. Project 1: Classify Images of Road Traffic Signs Summary: Assume you are a machine learning engineer working for a road traffic department. You have just developed a machine learning system that can classify images of road traffic signs. You have used combination of a modified version of the “Belgian Traffic Sign Classification Benchmark” data-set, along with “German TSC” data- set, to develop two ML models to perform the following two tasks: • Classify images according to sign-shape, such as diamond, hex, rectangle, round, triangle. • Classify images according to sign-type, such as stop, speed, warning, parking, etc. The next step is to do a presentation to your management and other ML engineers in your company, critically analyzing your model and discussing the related works and extensions. Your talk should include the following: Critical analysis: Under this section you should analyze the model that you developed and discuss the design choices you made during the model development stage. See Section 3.2 for things that you might discuss under this category. Review: You talk should include a discussion on: • Other works in literature that solves the same/similar problem as the one you have solved. • State-of-the-art techniques used for solving image classification problem. As a starting point, the following two papers are provided to you. You should discuss these in your presentation and compare the approach you have taken to what is discussed in the literature. For higher grades (HD/DI) you should include more paper in your review (see rubric for how many papers to include). The important aspects to be included in the review are discussed in section 3.3. • • J. Zhang, W. Wang, C. Lu, J. Wang, and A. K. Sangaiah, ‘‘Lightweight deep network for traffic sign classification,’’ Annals of Telecommunications, to be published, doi: 10.1007/s12243-019-00731-9. Paper Link D. Ciresan, U. Meier, J. Masci, and J. Schmidhuber. A committee of neural networks for traffic sign classification. InProceedings of International Joint Conference on Neural Networks (IJCNN). 2011. Paper Link 4 RMIT Classification: Trusted Extension: A typical deep neural networks that provide the state-of-the-art performance for traffic sign classification, is highly computational and memory demanding. In real life scenarios, such methods need to be able to perform in real time. Hence, more compact deep neural network architectures for traffic sign classification that are better suited for embedded devices need to be investigated. In other words, to implement such classification methods in real time, a more compact classifier with reduced computational complexity and less memory demand is required. What are the machine learning techniques that are applicable for this task and discuss how you would formulate the problem. No experiments are needed at this stage Project 2: Predict Energy Use Summary: Assume you are a machine learning engineer working for an Energy supply company. You have just developed a machine learning system that can predict the total energy use of lights and appliances in a low-energy building. You have used the UCI data set collected in 2017, to develop two completely separate types of supervised machine learning algorithms (of which at least one method is a non neural network based algorithm). To do so, you have also: • Constructed additional features to represent the time series information. • Might have used feature selection and ensemble learning method (This is what you have done in your submitted assignment-2). The next step is to do a presentation to your management and other ML engineers in your company, critically analyzing your model and discussing the related works and extensions. Your talk should include the following: Critical analysis: Under this section you should analyze the model that you developed and discuss the design choices you made during the model development stage. See Section 3.3 for things that you might discuss under this category. Review: You talk should include a discussion on: • Other works in literature that solves the same/similar problem as the one you have solved. • State-of-the-art techniques used for solving image classification problem As a starting point, the following two papers are provided to you. You should discuss these in your presentation and compare the approach you have taken to what is discussed in the literature. For higher grades (HD/DI) you should include more paper in your review (see rubric for how many papers to include). The important aspects to be included in the review are discussed in section 3.3. • Z.Wang,Y.Wang,R.S.Srinivasan, A novel ensemble learning approach to support building energy use prediction, Energy and Buildings,159(2018), pp.109-122,10.1016/J.ENBUILD.2017.10.085 , Paper Link • Park HY, Lee BH, Son JH, et al. A comparison of neural network-based methods for load forecasting with selected input candidates. In: Proceedings of the 2017 IEEE international conference on industrial technology (ICIT), Toronto, ON, Canada, 22–25 March 2017, pp.1100–1105. New York: IEEE , Paper Link 5 RMIT Classification: Trusted Extension: To make an effective energy management strategy, an accurate load forecasting (energy usage prediction) system is essential. For such system to be suitable for integration with automated building systems managementand intelligence, a system that is consistent, stable, and capable of higher prediction performance is desired. How can we achieve such a goal, alleviate the instability issue, and improve prediction accuracy? What are the machine learning techniques that are applicable for this task and discuss how you would formulate the problem? No experiments are needed at this stage. Project 3: Learning to Switch Traffic Lights Summary: Assume you are a machine learning engineers working for the department of traffic control. You just developed a reinforcement learning based algorithm to find a policy for controlling the switching of traffic lights at a simulated intersection, to improve the traffic thruput.Your algorithms fulfill the bellow requirements: • The simulated intersection has at least one right turning lane • Can be developed at a real-world intersection The next step is to do a presentation to your management and other ML engineers in your company, critically analyzing your model and discussing the related works and extensions. Your talk should include the following: Critical analysis: Under this section you should analyze the model that you developed and discuss the design choices you made during the model development stage (e.g. ele- ments of the reinforcement learning problem). See Section 3.2 for things that you might discuss under this category. Review: You should conduct a review to identify: • Other works in literature that solves the same/similar problem as the one you have solved. • Other techniques and algorithms that can be used to solve the problem given to you. As a starting point, the following two papers are provided to you. You should discuss these in your presentation and compare the approach you have taken to what is discussed in the literature. For higher grades (HD/DI) you should include more paper in your review (see rubric for how many papers to include). The important aspects to be included in the review are discussed in section 3.3. • H.Wei,G.Zheng,H.Yao,Z.Li, Intellilight: a reinforcement learning approach for intelligent traffic light control, 2018 ACM SIGKDD Proceedings of the 24th International Conference on Knowledge Discovery & Data Mining,ACM(2018), pp.2496-2505. Paper Link. • X. Liang, X. Du, G. Wang, and Z. Han, “A deep reinforcement learning network for traffic light cycle control,” IEEE Trans. Veh. Technol., vol. 68, no. 2, pp. 1243–1253, Feb. 2019. Paper Link Extension: You want to advance your system so it can successfully perform in the busy environment sch as intersections with at least 2 road crossing and at least one pedestrian crossing. The improved traffic l ights must take into account the number ofvehicles that are waiting at, or approaching, the intersection. 6 RMIT Classification: Trusted You need to formulate the reinforcement learning problem and discuss the approach you might take. Also discuss the similarity and differences compared to the solutionyou have developed for assignment 2. 5 Submission You have to submit all the relevant material as listed below via Canvas. 1. Share-point link to your recorded presentation. The instruction for recording the presentation will be provided on canvas. Modification to the content pointed by the submitted link, after the due date will incur late penalty. Incorrectly submitted/corrupted recording links or links that cannot be accessed will receive zero marks. After the due date, you will have 5 days to submit your assignment as a late submission. Late submissions will incur a penalty of 10% per day. After these five days, Canvas will be closed and you will lose ALL the assignment marks. Assessment declaration: When you submit work electronically, you agree to the assessment declaration - https:// www.rmit.edu.au/students/student-essentials/assessment-and-exams/assessment/ assessment-declaration 6 Teams Not relevant. This is an individual assignment. 7 Academic integrity and plagiarism (standard warning) Academic integrity is about honest presentation of your academic work. It means acknowledging the work of others while developing your own insights, knowledge and ideas.You should take extreme care that you have: • Acknowledgedwords,data,diagrams,models,frameworksand/orideasofothersyou have quoted (i.e. directly copied), summarized, paraphrased, discussed or mentioned in your assessment through the appropriate referencing methods • Providedareferencelistofthepublicationdetailssoyourreadercanlocatethesource if necessary. This includes material taken from Internet sites. If you do not acknowledge the sources of your material, you may be accused of plagiarism because you have passed off the work and ideas of another person without appropriate referencing, as if they were your own. RMIT University treats plagiarism as a very serious offence constituting misconduct. Plagiarism covers a variety of inappropriate behaviors, including: • Failure to properly document a source • Copyright material from the internet or databases • Collusion between students For further information on our policies and procedures, please refer to the following: https://www.rmit.edu.au/students/student-essentials/rights-and-responsibilities/ academic- integrity. 7 8 Marking guidelines A detailed rubric is attached on canvas. In summary: • Critical analysis 30%; • Review 40%; • Extension 30%; RMIT Classification: Trusted 8 RMIT Classification: Trusted 9