The University of Sydney School of Mathematics and Statistics
Solutions to Tutorial Week 11
STAT3023: Statistical Inference Semester 2, 2023 1. Suppose X1,...,Xn are iid N(θ,1) random variables and X ̄ = n1 Pni=1 Xi. If θ0 < θ < θ1 and
0 < C < ∞ show that
as n → ∞. Solution:
?C ̄C?
Pθ θ0+√n<X<θ1−√n →1
Since X ̄ ∼ N (θ, n1 ), √n(X ̄ − θ) ∼ N (0, 1). Therefore, so long as n is big enough that θ 0 + √C ≤ θ 1 − √C
nn
?√? C ? √? ̄ ? √? C ?? n θ0+√n−θ < n X−θ < n θ1−√n−θ
n 2C o2
(i.e. n ≥ θ1−θ0 ), we have
? C ̄ C?
Pθ θ0+√n<X<θ1−√n =Pθ
Since
also, since
= P θ ? C − √ n ( θ − θ 0 ) < √ n ? X ̄ − θ ? < √ n ( θ 1 − θ ) − C = Φ ?√n (θ1 − θ) − C? − Φ ?C − √n (θ − θ0)? .
√
n (θ1 − θ) − C → +∞
Φ?√n(θ1 −θ)−C?→1; √
C− n(θ−θ0)→−∞, Φ ?C − √n (θ − θ0)? → 0 .
Therefore the difference tends to 1, as required.
2. Interval estimation of an exponential rate parameter
Suppose X = (X1, . . . , Xn) consists of iid random variables whose common distribution is expo-
nential with rate θ ∈ Θ = (0, ∞) unknown. Consider the formal decision problem with decision space D = Θ = (0, ∞) and loss function (sequence) L(d|θ) = Ln(d|θ) = 1 n|d − θ| > √C o. This
n
corresponds to obtaining an interval estimate of θ, with risk equal to the non-coverage probability of the interval; the midpoint of the interval is still regarded as an “estimator” of θ though.
Consider using the ordinary maximum likelihood estimator θˆML as the estimator, giving the interval θˆML ± √C .
Solution: The likelihood is
n
fθ(X) = Y?θe−θXi? = θne−Tθ
i=1
where T = Pni=1 Xi; taking logs and differentiating gives
l ′ ( θ ; X ) = nθ − T ; Copyright © 2023 The University of Sydney 1
n
(a) Write down the likelihood and derive θˆML as a function of the Xi’s.
setting equal to zero and solving gives
θˆML = n = 1 ̄ ,
TX where, as usual, X ̄ = T/n denotes the sample mean.
(b) Since X ̄ has a gamma distribution with shape n and rate nθ, the product Yn = θX ̄ has a gamma distribution with shape n and rate n (i.e. its distribution is free of θ). Show that the risk function
Solution:
The risk function is ˆ?ˆC??ˆC?
ˆ R(θ|θML)=Gn
? C ?−1! " 1+θ√n + 1−Gn
Gn(y) = P(Yn ≤ y)
? C ?−1!# 1−θ√n
where
is the CDF of Yn.
R(θ|θML)=Pθ θ<θML−√n +Pθ θ>θML+√n
= P θ = P θ
(1)(1) X ̄ < θ + √C + P θ X ̄ > θ − √C
nn
(θ)(θ) θ X ̄ < θ + √C + P θ θ X ̄ > θ − √C
nn
(1)(1)
=P Yn<1+ C +P Yn>1− C √√
θn θn
? C ?−1! " ? C ?−1!# =Gn 1+θ√n + 1−Gn 1−θ√n .
(c) Determine, for any 0 < a < b < ∞, the maximum risk max R(θ|θˆML).
a≤θ≤b
Solution: Each of the two terms making up the risk is an increasing function of θ and so
the sum of them is also an increasing function of θ. Thus
ˆ ˆ ?C?−1! ?C?−1!
max R(θ|θML)=R(b|θML)=Gn 1+ √ +1−Gn 1− √ . a≤θ≤b bn bn
(d) Determine, for any 0 < a < b < ∞, the limiting maximum risk lim maxR(θ|θˆML).
n→∞ a≤θ≤b
You may use the facts that E(Yn) = 1, Var(Yn) = n1 , and Yn is asymptotically normal.
Solution: Note firstly that for any sequence zn → z, since √n(Yn − 1) →d N (0, 1) we have P?√n(Yn −1)≤zn →Φ(z)
where Φ(·) is the N(0,1) CDF. Consider the first term making up the maximum risk: ?C?−1!( 1)
Gn 1+√ =PYn≤
bn 1+b√n
= P
(√
n(Yn − 1) ≤
√
bn
−C !) b .
C
(√ √ 1 !) =P n(Yn−1)≤n1+C−1
1+C bn
√
Since
this tends to
1 + b√n b
? C? ?C?
Φ−b=1−Φb. ?C?−1!(√ √1!)
In a similar way,
1−Gn1−√ =Pn(Yn−1)>n C−1
θ (n) n+1 showed that the exact risk of this estimator is
θ (n)
−C ! C
b→−, C
bn 1−b√n
Therefore the limiting maximum risk is
lim max R(θ|θML)=2 1−Φ .
= P
n(Yn − 1) > b 1−C
(√ C!)
?C? →1−Φb.
ˆ ? ?C?? n→∞ a≤θ≤b b
3. Suppose X1, . . . , Xn are iid U[0, θ] random variables and that we wish to estimate θ using squared error loss (so the risk is the mean-squared error). Assume that n ≥ 3.
(a) The maximum likelihood estimator of θ is the sample maximum X(n) = maxi=1,...,n Xi. In
last week’s tutorial, using the fact that E (X ) = nθ and Var ?X ? =
nθ2 , we (n+1)2(n+2)
√
bn
n? ?2o Eθ X(n) −θ
2θ2
= (n+2)(n+1) .
Determine, for 0 ≤ a < b < ∞, the limiting maximum (rescaled) risk over [a, b]:
Solution:
lim max n2E n?X n→∞ a≤θ≤b θ (n)
−θ?2o= lim max θ2? 2n2 ? n→∞ a≤θ≤b (n + 2)(n + 1)
lim maxn2En?X −θ?2o. n→∞ a≤θ≤b θ (n)
( 2n2 ) 2
n2 ?1 + 2 ? ?1 + 1 ? nn
(2) 2
?1 + 2 ? ?1 + 1 ? nn
= 2b2 .
(b) In last week’s tutorial we also showed that the unbiased estimator θˆunb = ? n+1 ? X(n) has n
exact risk
For 0 ≤ a < b < ∞ find
?hˆ i2? θ2
Eθ
θunb −θ
= n(n+2). ?hˆ i2?
θunb−θ
= lim b n→∞
= lim b n→∞
lim maxnEθ n→∞ a≤θ≤b
Solution:
lim maxnEθ n→∞ a≤θ≤b
n→∞ a≤θ≤b n(n + 2)
?hˆ i2? 2? n2 ? θunb−θ = lim max θ
= lim b n→∞
= lim b n→∞
( n2 )
(1) 2
?1 + 2 ? n
n2 ?1 + 2 ? n
= b2 .
(c) Show that the Bayes procedure using the “flat prior” weight function w(θ) ≡ 1 is given by
ˆ ?n−1? θflat(X)= n−2 X(n).
Solution:
The likelihood is
fθ(X) =
Yn ?1{0≤Xi ≤θ}? 1?X(1) ≥0 1?X(n) ≤θ
i=1
θ = θn
(where X(1) is the sample minimum). Assuming X(1) ≥ 0, when viewed this is a multiple of
a Pareto density (with shape n − 1 and scale X(n)). (Recall that if X has a Pareto density
with shape α and scale m, then X has pdf f(x) = αmα for x ≥ m.) Since the weight xα+1
function is just w(θ) ≡ 1 this Pareto distribution is also the posterior distribution. Thus the Bayes procedure/estimator is the posterior mean, which is
shape × scale = (n − 1)X(n) . shape−1 (n−2)
(d) Using the expressions for the expectation and variance of X(n) given in part (a) above, determine the variance, bias and thus exact risk of θˆflat.
Solution:
hˆ i?n−1????n−1?nθ Eθ θflat(X) = n−2 Eθ X(n) = n−2 n+1.
Thus the bias is
Biasθ θflat = n−2 Eθ X(n) −θ= (n−2)(n+1)
The variance is given by
Thus the exact risk is
?n−1?2 nθ2
= n−2 (n+1)2(n+2).
?ˆ ? n hˆ io2 = Varθ θflat + Biasθ θflat
θ2 ?n(n−1)2 ? = (n−2)2(n+1)2 n+2 +4
θ2(n−1)2?n 4? = (n−2)2(n+1)2 n+2 + (n−1)2 .
hˆ i ?n − 1? ? ? (n − 1)nθ − θ(n − 2)(n + 1) ?n2 −n−[n2 −n−2]?
= θ (n − 2)(n + 1) = 2θ .
?ˆ ? ?n−1?2
Varθ θflat = n − 2 Varθ ?X(n)?
(n−2)(n+1)
Eθ
?hˆ i2? θflat − θ
(e) Determine,for0≤a<b<∞,
lim maxnEθ
?hˆ i2? θflat−θ
= b2
n4 ?1 − n1 ?2 ?1 − n1 ?2
n+2+(n−1)2 =n?1+n2?+(n−1)2 =?1+n2?+(n−1)2 →1.
(f) Comment on what is interesting about the 3 estimators compared in the previous parts.
Solution:
n→∞ a≤θ≤b
2?hˆi2?
+ 2 (n−1)
lim max n Eθ n→∞a≤θ≤b
since
and
θflat − θ
= lim max θ n→∞a≤θ≤b
2?n2(n−1)2?n 4??
(n−2) (n+1) =limb2?n2(n−1)2 ?n+4??
n+2 n→∞ (n−2)2(n+1)2 n+2 (n−1)2
n2(n − 1)2
(n−2)2(n+1)2 = 4? 2?2? 1?2 = ?
2?2? 1?2 →1 1−n 1+n
n 1−n 1+n ?n4?n414
Solution: The maximum likelihood estimator (the sample maximum) has a bias such that the squared bias and the variance are “of the same order” (i.e. like 1 ) and thus both
n2
contribute to the limiting (maximum, rescaled) risk. The unbiased version θˆunb however removes the bias and only increases the variance slightly, so that in the limit, the (maximum, rescaled) risk is halved. Perhaps most interestingly, the Bayes estimator is “automatically (approximately) bias-corrected”; it is not unbiased, but it is “bias-corrected” enough so that the squared bias is of smaller order than the variance and thus does not contribute to the
limiting (maximum, rescaled) risk.
