Assignment 2
DSC 212: Probability and Statistics for Data Science
Due date: 5:00 pm PST, 13th March, 2023
1. Consider Bernoulli(p) observations 0 0 1 0 1 0 1 1. Plot the posterior distribution for p for the following
prior distributions Beta(1,1), Beta(10,10), Beta(1, 10). Mark the inflection points in the plots.
Note: Hand-drawn plots are sufficient.
2. Let X1, X2, . . . , Xn
i.i.d.∼ Uniform(0,θ). Calculate the posterior distribution corresponding to the prior
density f(θ) ∝ 1θ .
3. In this problem, we derive a general approach—called iterative reweighed least-squares—for obtaining
the empirical risk minimization solution with a smooth loss, and unconstrained linear function class.
Consider minimizing a function L : Rd → R, and assume that L is continuously differentiable up to
second order. The Newton iteration for obtaining the minimizer of L is
θt+1 = θt − [∇2L(θt)]−1∇L(θt) (1)
where ∇2L and ∇L are the Hessian and gradient of L respectively.
(a) Show that iteration (1) is equivalent to minimizing the second-order Taylor expansion of L around
θt.
(b) Let L(θ) =
∑n
i=1 ℓ(x
⊤
i θ, yi) where ℓ : R× Y → R is smooth in its first argument. Argue that
∇L(θ) =
n∑
i=1
αi(θ)xi ∇2L(θ) =
n∑
i=1
wi(θ)xix
⊤
i (2)
for weights αi(θ) and wi(θ) that you specify.
(c) Let X ∈ Rn×d be the matrix whose ith row is x⊤i . Let α(θ) = (α1(θ), . . . , αn(θ)) and Wθ =
diag(w1(θ), . . . , wn(θ)). Show that iteration (1) is equivalent to solving
θt+1 = argmin
θ∈Rd
∥∥∥W 1/2(θt)(Xθ − z(θt)∥∥∥2 (3)
for z(θt) ∈ Rn which you specify. Thus Newton iteration in this context is equivalent to solving a
reweighed least-squares problem at each iteration.
Hint: ∇L(θ) = X⊤α(θ) and ∇2L(θ) = X⊤W (θ)X.
(d) Consider the case where ℓ(t, y) = −yt + log(1 + et) is the logistic loss. Show that in this case,
wi(θ) = σ(θ
⊤xi)(1− σ(θ⊤xi)) and find an expression for zi(θ).
4. (Bayesian linear model) For the fixed design linear model with X = [x1, x2, . . . , xn]⊤ with observations
yi = x
⊤
i β + σεi having i.i.d. standard normal noise εi. Consider the likelihood and prior given below.
fY |β(y|w) ∝ exp(− 1
2σ2
∥y −Xw∥2) (4)
fβ(w) ∝ exp(−∥w∥2Γ) (5)
Here Γ is a positive definite precision matrix and ∥w∥2Γ := w⊤Γw. Find the posterior distribution Pβ|Y .
5. If X ∼ N (µ,Σ) where µ ∈ Rd and Σd×d is a positive definite matrix. Find the distribution of the vector
AX ∈ Rp for a fixed matrix A ∈ Rp×d.
1

欢迎咨询51作业君