School of Electronics and Computer Science
University of Southampton
COMP6245 (2019/20): Foundations of Machine Learning Lab 4 (of 6)
Issue 30 October 2019
Deadline 10 November 2019 (10:00)
Feedback 16 November 2019
In this Lab, you are expected to work independently; i.e. you may only discuss
with or ask questions from a demonstrator or the lecturer. You are, of course, free to
refer to the cited texts or access information from Web-based resources (indeed, this
is recommended).
Objective
• Implementing linear regression
• Regularization using quadratic and sparsity-inducing penalties
• Implementing sparse regression on a realistic problem in chemoinformatics
1 Linear Least Squares Regression:
We will work with the Diabetes dataset from the UCI Machine Learning repository [1]
taken from the package sklearn. Load the data and inspect the features and targets. It
is usually a good idea to plot a few histograms of the targets and pair-wise scatters of the
features in any new problem you are tasked to solve.
• Implement a linear predictor that is solved by the pseudo-inverse method:
a =
(
Y t Y
)−1
Y t f ,
where Y is the N × p input matrix and f is the N × 1 vector of responses.
• Solve the same problem using the linear model from sklearn and compare the
results.
2 Regularization
Tikhonov regularization minimizes the mean squared error with a quadratic penalty on
the weights [2] (available online in https://web.stanford.edu/~hastie/Papers/ESLII.
pdf):
min
a
||f − Y a||22 + γ ||a||22
Derive and implement a regularized regression. Show, using two bar graphs of the weights
side by side to the same scale, how the two solutions differ.
1
Figure 1: Solutions of Linear and Regularized Regressions
3 Sparse Regression
L1 regularization is a method for achieving sparse solutions [3]. It minimizes:
min
a
||f − Y a||22 + γ ||a||1
We will use the sklearn package for implementing its solution. For the Diabetes problem
considered above, solve the lasso problem and plot the resulting weights as a bar graph.
Observe how the number of non-zero weights change with the regularization parameter
γ. Your comparisons should look similar to Fig. 1. In each of these cases, compare the
prediction errors. In the case of the sparse regression, would you say the features with
nonzero weights are more meaningful (to answer, you have to find the source of the data
and look at the variables)?
Regularization Path
In implementing the lasso it is convenient to study the regularization path (Fig. 2, Image
taken from https://scikit-learn.org/stable/auto_examples/linear_model/plot_
lasso_lars.html) Implement and study the regularization path for the six-variable illus-
Figure 2: Regularization Path: How regression coefficients change with hyperparameter.
trative example considered in [3].
4 Solubility Prediction:
We will now look at a large problem of predicting solubility of chemical compounds from
features derived from their molecular structure. Predicting function from structural vari-
2
ables is an important problem because it is easy to define and synthesize small chemical
compounds, but very expensive to test them experimentally. Hence the step known as in
silico screening is increasingly popular. The dataset we will use is from Huuskonen et al.
[4] and the problem has also been considered recently in Pirashvili et al. [5] using more
sophisticated machinery. Have a skim-read through the introductory and results sections
of these papers.
Data used in [4] with several additional features and more compounds is available in the
excel spread sheet Husskonen Solubility Features.xlsx.
• Load the data, split into training and test sets, implement a linear regression and
plot the predicted solubilities against the true solubilities on the training and test
sets. To facilitate comparison, draw the two scatter plots side by side to the same
scale on both axes.
• Implement a lasso regularized solution and plot graphs of how the prediction error
(on the test data) and the corresponding number of non-zero coefficients change with
increasing regularization.
• Are you able to make any comment comparing your results to those claimed in [4]
or [5]?
Report
Write a short report of no more than four pages, summarising your work.
Appendix: Snippets of Code
1. Linear regression on diabetes dataset
from sklearn import datasets
from sklearn.linear_model import LinearRegression
# Load data, inspect and do exploratory plots
#
diabetes = datasets.load_diabetes()
Y = diabetes.data
f = diabetes.target
NumData, NumFeatures = Y.shape
print(NumData, NumFeatures)
print(f.shape)
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12, 4))
ax[0].hist(f, bins=40)
ax[0].set_title("Distribution of Target", fontsize=14)
ax[1].scatter(Y[:,6], Y[:,7], c=’m’, s=3)
ax[1].set_title("Scatter of Two Inputs", fontsize=14)
2. Comparing pseudo-inverse solution to sklearn output
3
# Linear regression using sklearn
#
lin = LinearRegression()
lin.fit(Y, f)
fh1 = lin.predict(Y)
# Pseudo-incerse solution to linear regression
#
a = np.linalg.inv(Y.T @ Y) @ Y.T @ f
fh2 = Y @ a
# Plot predictions to check if they look the same!
#
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(10,5))
ax[0].scatter(f, fh1, c=’c’, s=3)
ax[0].grid(True)
ax[0].set_title("Sklearn", fontsize=14)
ax[1].scatter(f, fh2, c=’m’, s=3)
ax[1].grid(True)
ax[1].set_title("Pseudoinverse", fontsize=14)
3. Tikhanov Tegularizer
gamma = 0.5
aR = np.linalg.inv(Y.T @ Y + gamma*np.identity(NumFeatures)) @ Y.T @ f
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(8,4))
ax[0].bar(np.arange(len(a)), a)
ax[0].set_title(’Pseudo-inverse solution’, fontsize=14)
ax[0].grid(True)
ax[0].set_ylim(np.min(a), np.max(a))
ax[1].bar(np.arange(len(aR)), aR)
ax[1].set_title(’Regularized solution’, fontsize=14)
ax[1].grid(True)
ax[1].set_ylim(np.min(a), np.max(a))
4. Sparsity inducing (lasso) regularizer
from sklearn.linear_model import Lasso
ll = Lasso(alpha=0.2)
ll.fit(Y, f)
yh_lasso = ll.predict(Y)
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(10,4))
ax[0].bar(np.arange(len(a)), a)
ax[0].set_title(’Pseudo-inverse solution’, fontsize=14)
ax[0].grid(True)
ax[0].set_ylim(np.min(a), np.max(a))
ax[1].bar(np.arange(len(ll.coef_)), ll.coef_)
ax[1].set_title(’Lasso solution’, fontsize=14)
ax[1].grid(True)
ax[1].set_ylim(np.min(a), np.max(a))
5. Lasso Regularization path on a synthetic example:
4
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import lasso_path
from sklearn import datasets
# Synthetic data:
# Problem taken from Hastie, et al., Statistical Learning with Sparsity
# Z1, Z2 ~ N(0,1)
# Y = 3*Z1 -1.5*Z2 + 10*N(0,1) Noisy response
# Noisy inputs (the six are in two groups of three each)
# Xj= Z1 + 0.2*N(0,1) for j = 1,2,3, and
# Xj= Z2 + 0.2*N(0,1) for j = 4,5,6.
N = 100
y = np.empty(0)
X = np.empty([0,6])
for i in range(N):
Z1= np.random.randn()
Z2= np.random.randn()
y = np.append(y, 3*Z1 - 1.5*Z2 + 2*np.random.randn())
Xarr = np.array([Z1,Z1,Z1,Z2,Z2,Z2])+ np.random.randn(6)/5
X = np.vstack ((X, Xarr.tolist()))
# Compute regressions with Lasso and return paths
#
alphas_lasso, coefs_lasso, _ = lasso_path(X, y, fit_intercept=False)
# Plot each coefficient
#
fig, ax = plt.subplots(figsize = (8,4))
for i in range(6):
ax.plot(alphas_lasso, coefs_lasso[i,:])
ax.grid(True)
ax.set_xlabel("Regularization")
ax.set_ylabel("Regression Coefficients")
6. Predicting Solubility of Chemical Compounds
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
sol = pd.read_excel("Husskonen_Solubility_Features.xlsx", verbose=False)
print(sol.shape)
colnames = sol.columns
print(colnames)
f = sol["LogS.M."].values
fig, ax = plt.subplots(figsize=(4,4))
ax.hist(f, bins=40, facecolor=’m’)
ax.set_title("Histogram of Log Solubility", fontsize=14)
ax.grid(True)
5
Y = sol[colnames[5:len(colnames)]]
N, p = Y.shape
print(Y.shape)
print(f.shape)
# Split data into training and test sets
#
from sklearn.model_selection import train_test_split
Y_train, Y_test, f_train, f_test = train_test_split(Y, f, test_size=0.3)
# Regularized regression
#
gamma = 2.3
a = np.linalg.inv(Y_train.T @ Y_train + gamma*np.identity(p)) @ Y_train.T @ f_train
fh_train = Y_train @ a.values
fh_test = Y_test @ a.values
# Plot training and test predictions
#
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(10,4))
ax[0].scatter(f_train, fh_train, c=’m’, s=3)
ax[0].grid(True)
ax[0].set_title("Training Data", fontsize=14)
ax[1].scatter(f_test, fh_test, c=’m’, s=3)
ax[1].grid(True)
ax[1].set_title("Test Data", fontsize=14)
# Over to you for implementing Lasso
#
References
[1] K. Bache and M. Lichman, “UCI machine learning repository.” http://archive.ics.
uci.edu/ml, 2013.
[2] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning.
Springer, 2008.
[3] T. Hastie, R. Tibshirani, and M. Wainwright, Statistical Learning with Sparsity: The
Lasso and Generalizations. Chapman & Hall/CRC, 2015.
[4] J. Huuskonen, M. Salo, and J. Taskinen, “Aqueous solubility prediction of drugs based
on molecular topology and neural network modeling,” Journal of Chemical Information
and Computer Sciences, vol. 38, no. 3, pp. 450–456, 1998.
[5] M. Pirashvili, L. Steinberg, B. G. F., M. Niranjan, J. G. Frey, and J. Brodzki, “Im-
proved understanding of aqueous solubility modeling through topological data analy-
sis,” Journal of Cheminformatics, vol. 10, no. 1, p. 54, 2018.
Mahesan Niranjan November 2014
6