BUCI077H7 Qc Birkbeck College 2020 Page 2 of 7 Assessment Requirements Where is AI headed next? The sudden rise and fall of different techniques have characterised AI research for a long time. Every decade has seen a heated competition between different ideas. MIT Technology Review wanted to visualise these changes. They thus turned to one of the largest open-source databases of scientific papers, known as the arXiv (https://arxiv.org). They downloaded the abstracts of all 16,625 papers available in the “artificial intelligence” section through November 18, 2018, and tracked the words mentioned through the years to see how the field has evolved. Through their analysis, they found three major trends. • a shift toward machine learning during the late 1990s and early 2000s, • a rise in the popularity of neural networks beginning in the early 2010s, and • growth in reinforcement learning in the past few years. Among the top 100 words mentioned, those related to knowledge-based systems—like “logic,” “constraint,” and “rule”—saw the greatest decline. Those related to machine learning—like “data,” “network,” and “performance”—saw the highest growth. BUCI077H7 Qc Birkbeck College 2020 Page 3 of 7 Under the new machine-learning paradigm, the shift to deep learning did not happen immediately. Instead, as their analysis of key terms shows, researchers tested a variety of methods in addition to neural networks, the core machinery of deep learning. Some of the other popular techniques included Bayesian networks, support vector machines, and evolutionary algorithms, all of which take different approaches to finding patterns in data. BUCI077H7 Qc Birkbeck College 2020 Page 4 of 7 As well as the different techniques in machine learning, there are three different types: supervised, unsupervised, and reinforcement learning. In the last few years, however, reinforcement learning, which mimics the process of training animals through punishments and rewards, has seen a rapid uptick of mentions in paper abstracts. However, this idea is not new. For many decades, it did not really work. The supervised-learning people used to make fun of the reinforcement-learning people. However, the pivotal moment came in October 2015, when DeepMind’s AlphaGo, trained with reinforcement learning, defeated the world champion in the ancient game of Go. The effect on the research community was immediate. Description of Task to be completed The MIT Technology Review’s study above analysed historical data (e.g. word frequencies) only meaning no intelligent predictions based on the historic data are made. You are now refining and extending this study by building a machine-learning solution that can predict what techniques will characterise AI research over the next decade. Give answers to the following questions. • Provide an overview of the key concepts of your solution (~250 words) along with a diagram showing a ML pipeline applied to this problem. An ML pipeline is an independently executable workflow of a complete machine learning task/components. Subtasks/subcomponents are encapsulated as a series of steps within the pipeline. Pipelines should focus on machine learning tasks/components such as: database, data collection/preparation, feature extraction, algorithm selection, training, validation and deployment. Each component of your diagram must specifically be named. For example, instead of naming a component ‘Datasets’, it should be named as ‘arXiv’ or ‘Elsevier’. A typical ML pipeline or workflow with different sub-components is shown in the following diagram: • Give details of each component/task in your diagram (100–200 words each). You should provide concise, highly specific answers. Some examples as follows. • Instead of saying ‘normalisation’ you should say what kind of normalisation technique and why you chose to use it. • What features you will extract, from where, how they will be structured/formatted BUCI077H7 Qc Birkbeck College 2020 Page 5 of 7 • Name of specific architecture should be provided (e.g. AlexNet, VGG, ResNet, etc) along with justifications of choice rather than saying ‘Convolutional Neural Network (CNN)’. The following questions may be helpful in providing your answers. Data preparation • Where would the feature/data come from? • How would you select/extract/construct feature? • Where did the labels come from (if needed)? • What kind of data exploration would you do? • How would you clean the data? • Would the classes be balanced (if needed)? How did the distribution change your workflow? • What kind of normalisation would you do? Why? • What would you do about missing data (if any)? • What kind of feature engineering would you do? • How would you split the data into train, validate and test? Training and evaluation • Which ML algorithms would you explore and why? • How would you tune the hyperparameters of the algorithm? • What kind of validation would you do? • What evaluation metric would you use? Why is it the most appropriate one? • If your approach was a classification task, how does a dummy classifier score? • Is interpretability important for your problem? How interpretable is your model? That is, do the learned parameters mean anything, and can we learn from them? • If this was a classification task, are probabilities available in your model and would you use them? • If this was a regression task, have you checked the residuals for normality and homoscedasticity? Page 6 of 7 Marking Scheme Marks will be awarded in the following areas % W e ig h ti n g Marking Descriptors Excellent 80-100% Very Good 70-79% Good 60-69% Satisfactory 50-59% Poor 40-49% Very Poor 0-39% Solution Design (inc. diagram) 30% Full consideration has been given to design alternatives and an entirely appropriate and effective solution results. Very effective creativity in designing a solution to the problem that is as close to an optimum solution as can be expected. Designs fully reflect good principles. Selection and application of principles is very effective, showing insight and creativity. Effective design. Appreciation of principles and has applied these. Evidence of consideration of alternatives and judgement in decisions. Weaknesses in design, but shows some awareness of appropriate principles and can apply these reasonably well with fair (if sometimes basic) justification. Designs are weak but just workable. Limited use of principles in design. Justification for decisions is limited. Little evidence of appropriate design. Design decisions not justified. Preprocessing 10% Strong justifications. Full consideration has been given to each of preprocessing steps and real depth of insight of each step and creativity is shown. Very good justification for selection. Selection and application of preprocessing techniques is very effective, showing insight and creativity. Appreciation of principles. Evidence of consideration of alternatives and judgement in decision. Some weaknesses in the process and can apply these reasonably well with fair justification. Processes are weak and little evidence is provided but just workable. Justification for decisions is limited. Little or no attempt to apply appropriate preprocessing techniques. Selecting features 20% Strong justifications. Full consideration of various techniques in three different categories. Strong evidence of wide reading and research. Very good justifications for selection. Application of feature selection techniques are effective showing insight and creativity. Good justifications. Appreciation of principles. Evidence of consideration of alternatives and judgement in decision. Some weaknesses in the process and can apply these reasonably well with fair justification. Processes are weak and little evidence is provided but just workable. Justification for decisions is limited. Little or no attempt to apply appropriate feature selection techniques. Selecting and refining ML algorithms 20% Full consideration has been given to options for algorithms. Entirely appropriate choices made and expertly applied. Very good Very good selection of algorithms and procedures - competently applied. Very good understanding of Good judgement in selection and application of algorithm and procedures. Good understanding of theories. Reasonable selection and application of algorithm and procedures. Poor but acceptable selection. Decisions may be based on student’s convenience. Appropriate analysis or judgement severely lacking or not provided. Page 7 of 7 Marks will be awarded in the following areas % W e ig h ti n g Marking Descriptors Excellent 80-100% Very Good 70-79% Good 60-69% Satisfactory 50-59% Poor 40-49% Very Poor 0-39% understanding of theories. Strong selection strategies. theories. Justification for selection is sound. Very good selection strategies. Evaluating model 10% Strong justification on evaluation methods. Selected models are fairly evaluated. All chosen evaluation measures correct and complete. Good justification on evaluation methods. All chosen evaluation measures are correct and complete. Selected evaluation methods are appropriate to the objectives. Good justification for selection. Reasonable justification for selection showing some awareness of appropriate principles. Some of the selected method are appropriate, with limited justification. An insufficient awareness of principles with very weak or no justification. Documentation 10% Organisation of work is of a very high standard, likely to be highly stimulating, and at the limits of what may be expected at postgraduate level. Documentation is very well ordered, concise and coherent. Excellent use of diagrams, tables and spaces. Organisation of work is likely to show few mistakes/limitations. Very good use of diagrams, tables and spaces. There is an overall structure evident but does not offer strong flow and progression. Good use of diagrams, tables and spaces. Structure of work is weak or inconsistent. Only the main points are logically organised/linked. Reasonable use of diagrams, tables and spaces. Unstructured and/or incoherent. Illustrations are very poorly presented.
欢迎咨询51作业君
