Homework 1: ML 80629A (FALL 2021) Instructions: • Please include your name and HEC ID with submission. • The homework is due by 11:59pm on the due date. • The homework is worth 20% of the course’s final grade. • Assignments are to be done individually. • All code used to arrive at answers is submitted along with answers. You can convert your jupyter notebook or colab to pdf and upload it. 1 ML Fundamentals (20pt) 1. (4pt) Explain the difference between the training error and the general- ization error. Make sure to describe how to evaluate the generalization error of a model in practice including pitfalls of this approach. 2. (4pt) To increase the size of your training set you first train a model and then use it to obtain labels on an unlabelled test set. You then retrain the model with the data from your train set as well as the data from your test set. Would you expect that your final model would obtain a lower validation error? 3. (4pt) Suggest a method for regularization a K-NN model. Hint: think of what regularization accomplishes in terms of the bias/variance trade off. 4. (4pt) Recall the task of document classification where documents must be classified based on their content. If the documents are encoded in a bag-of-words format, could you use K-NN to classify them? If so, describe a distance function that might be sensible to use. Otherwise, explain why not. 5. (4pt) Describe both the advantages and the disadvantages of using a larger K when doing K-fold cross validation. 1 2 Regression (15pt) • Let’s explore California housing dataset. This dataset was obtained from the StatLib repository. The target variable is the median house value for California districts, expressed in hundreds of thousands of dollars ($100,000). You can load the data from scikit-learn. You can use .DESCR to gather more info about this dataset. 1. (2pt) Perform statistical analysis by using .describe() on the data, what do you notice about the attribute values? 2. (2pt) Look at the distribution of the attribute values by plotting their histograms. Notice anything interesting? 3. (3pt) Perform 10-fold cross-validation (with shuffle=True and random state=20160202) with a LinearRegression model. Report the mean squared error (MSE) on the validation set (averaged across folds). • Two very popular options in Linear Regression are the Lasso method and the Ridge Regression. These models are implemented in sklearn: Lasso and Ridge. Hint: If you’d like to understand Ridge Regression better, you might want to look at this example. 4. (2pt) What one word captures what Lasso and Ridge do? 5. (3pt) Perform 10-fold CV with normal LinearRegression, Lasso, and Ridge. Compare the attribute weights (coef) of each of these meth- ods. What do you observe? 6. (3pt) Report the average MSE for each method on the validation set across folds. How do these variants of linear regression perform compared to each other? Classification (65 pt) In this exercise, you will implement Naive Bayes classifier and neural network models to carry out a text classification task. You will implement various fea- ture representations, i.e., BoW and TF-IDF and then you will do performance analysis of NNs and NB with different hyperparameter settings. The data to download is movie reviews dataset. In each file (train, vali- dation, test), each line consists of a single review text followed by a number indicating the sentiment : 0 for negative and 1 for positive. You would have to split the these data into input texts and corresponding labels.Hint: you can use the function split(’\t’), pandas allows you to easily use this data. 2 3 Feature Representations (5pt) Transform the input text data into the following feature representations to train some models. You can use scikit-learn tools to do this. To reduce the time for training, please set max-features = 10000 to build a vocabulary that only consider the 10k top max features ordered by term frequency across the corpus. Do not change the other parameters. For this question, we ask you for the few lines of code from sklearn that you used to encode (and only encode) the training, validation and test data into (use the default settings): 1. (3pt) Bag-of-Words features (BoW) 2. (2pt) TF-IDF features 4 Naive Bayes (15pt) 1. (5pt) Perform the sentiment classification task with these 2 different types of features. Report which NB you used and what is the validation set accuracy in each case. 2. (10pt) Based on the classification of the model, report 5 words from the texts in the train dataset that can be inferred as positive and 5 words that are negative. 5 Neural Network (30pt) After all the required imports in your code be sure to set the random state=12345. 1. (12pt) From the BoW and TF-IDF features from above, you are going to train different neural network models. Use the option early stopping=True and train the networks by trying all the following hyperparameter combi- nations: • dimension of a first hidden layer: [4,8,16] • dimension of a second hidden layer: [0,4,8] • learning rate (after fixing the above three hyperparameters): [0.1, 0.01, 0.001] • L2 (penalty): [0.001, 0.01, 0.1] What is the best combination of hyperparameters and what is the perfor- mance on the resulting model validation set? 2. (16pt) What did you observe with the impact of different hyperparame- ters on the model accuracy? Write a brief recommendation to follow for someone trying to find the best hyprparameters for their model. 3. (2 pt) Can you recommend any other hyperparameters to tune? 3 6 Comparison (15pt) 1. (4pt) Determine the performance of the simplest baseline, i.e., majority voting, for this task. 2. (5pt) For each case of the feature representation (BoW and TF-IDF), report the best test performance accuracies of the Naive Bayes classifier and Neural Network that you observe. 3. (3pt) Based on these accuracy values, which is the best feature represen- tation for each type of model that we have considered? Can you think of why this is the case? 4. (3pt) Based on these accuracy values, which is the best performing model? Explain briefly why it is the case. 4
欢迎咨询51作业君