Chapter 1

Minimax Optimal Designs for Nonparametric Regression — a further Optimality Property of the Uniform Distribution

Stefanie Biedermann Holger Dette

Abstract

In the common nonparametric regression model y_{i,n} = g(t_{i,n}) + σ(t_{i,n}) ε_{i}, i = 1, …, n, with i.i.d. noise and nonrepeatable design points t_{i}, we consider the problem of choosing an optimal design for the estimation of the mean function g. A minimax approach is adopted which searches for designs minimizing the maximum of the asymptotic integrated mean squared error, where the maximum is taken over an appropriately bounded class of functions (g, σ). The minimax designs are found explicitly, and for certain special cases the optimality of the uniform distribution can be established.

Nonparametric Regression; Kernel Estimation; Locally Optimal Designs; Minimax Designs; Mean Squared Error

1.1 Introduction

Consider the common nonparametric regression modelnonparametric regression model

Y_{i,n} = g(t_{i,n}) + σ(t_{i,n})ε_{i,n}  i = 1, …, n,  (1.1)

where the ε_{i,n} form a triangular array of rowwise independent identically distributed random variables with variance 1 and mean 0, g, σ are unknown smooth functions and {t_{i,n} | i = 1, …, n} is a fixed design in the interval [0, 1]. Much effort has been devoted to the problem of estimating the mean function (see e.g. the recent monographs of Härdle, 1990Härdle, Wand and Jones, 1995Wand and Jones, Fan and Gijbels, 1996Fan and Gijbels), and many of the developed estimation methods are meanwhile standard methods in applied regression analysis. Although it is well known (and intuitively clear) that the asymptotic properties of the various nonparametric estimators depend sensitively on the underlying designs, the problem of designing experiments in the nonparametric setup (1.1) has found much less attention in the literature. This is mainly due to the fact that – as in the case of nonlinear regression models – the optimality criteria usually depend on the unknown mean and variance function. Müller (1984)Müller, H.G. studied optimal designs for estimating derivatives of the mean function by minimizing the asymptotic integrated mean squared errorasymptotic integrated mean squared error of a kernel estimate. He showed that the optimality criteria are local in the sense of Chernoff (1953)Chernoff and the asymptotically optimal designs depend on the unknown mean and variance function. Because of these difficulties Cheng, Hall and Titterington (1998)Cheng, Hall and Titterington proposed a sequential approach for defining an optimal design measure in the context of local linear regression. A different approach was used by Müller (1996)Müller, W.G. who considered the situation, where the unknown response function has to be predicted at a finite number of specified points. This author proposed to minimize a weighted average of the variances of local linear estimates at these points (see Fan and Gijbels, 1996, or Wand and Jones, 1995). A further optimality criterion is considered in the present paper, which is different from the methods considered by the aforementioned authors and mainly motivated by two observations. On the one hand the designs proposed by Müller (1984) are not robust with respect to misspecifications of the mean and variance function, on the other hand there are many situations where sequential designs cannot be applied (e.g. experiments in agriculture). For these reasons, we propose a minimax approach which seeks for designs minimizing the maximum of the asymptotic integrated mean squared error, where the maximum is taken over certain Sobolev balls for the mean and variance function. Section 2 states some basic terminology from nonparametric kernel estimationkernel estimation, which is necessary for the formulation of the minimax optimality criterion. The main results can be found in Section 3 showing that the minimax optimal designs are equal to the weight function used in the definition of the integrated mean squared error. This highlights the particular role of the uniform distribution, which turns out to be the minimax optimal design with respect to the classical integrated mean squared error criterion.

1.2 Asymptotic Representation of the Integrated Mean Squared Error of a Nonparametric Regression Estimator

For the introduction of the optimality criterion, we need an asymptotic representation of the integrated mean squared error of a nonparametric estimate of the mean function g, which is nowadays standard in mathematical statistics (see Härdle, 1990, Wand and Jones, 1995 or Fan and Gijbels, 1996). We assume that the mean function is k-times continuously differentiable, i.e. g ∈ C^{k}[0, 1], and the variance function is Lipschitz continuous of order γ, i.e. σ ∈ Lip_{γ}[0, 1] for some γ ∈ (0, 1]. Following Gasser and Müller (1984)Gasser and Müller, we consider

g_{n,v}(t) = \frac{1}{h^{v+1}(t)} ∑_{i=1}^{n} ∫_{s_{i−1,n}}^{s_{i,n}} K_{v}(\frac{t − x}{h(t)})dx · Y_{i,n}  (2.1)

as an estimate of the v-th derivative g^{(v)} of the regression function (v = 0, 1, …, k− 1). Here K_{v} is a Lipschitz continuous function with compact support, say [−1, 1], such that

\frac{(−1)^{j}}{j!} ∫_{−1}^{1} K_{v}(x)x^{j}dx = \{ \begin{matrix}0 & if & 0 ≤ j < k; j ≠ v \\ 1 & if & j = v \\ B & if & j = k \end{matrix}  (2.2)

where B ≠ 0. The triangular array {s_{i,n} | i = 0, …, n} is defined by s_{0,n} = 0, s_{n,n} = 1, s_{i,n} = (t_{i,n} + t_{i+1,n})/2 (i = 1, …, n − 1), and the design points t_{i,n} are supposed to satisfy a Sacks and Ylvisaker (1970)Sacks and Ylvisaker condition

∫_{0}^{t_{i−1,n}} f(t)dt = \frac{i − 1}{n − 1}  i = 1, …, n  (2.3)

with a positive design density f.

Note that a triangular array of random variables is considered in model (1.1) in contrast to the formulation y_{i} = g(t_{i})+σ(t_{i})ε_{i}, which is maybe more familiar to most workers in experimental design. The reason is a much simpler formulation of the asymptotic properties of the estimate (2.1). For example, if t̃_{i} = \frac{i−1}{n−1} would denote a uniform design, an additional observation at a point t^{∗}, say, would destroy uniformity. On the other hand, the formulation (2.3) using the array t_{i,n} would yield the same design density independent of n ∈ IN.

The quantity h(t) denotes the bandwidth which may depend on a specific point t (local bandwidth). The locally optimal bandwidth minimizes the asymptotic mean squared error at the point t and is given by

h^{∗}(t) = \{\frac{(2v + 1)V σ^{2}(t)}{2(k − v)nB^{2}f(t)(g^{(k)}(t))^{2}} \}^{1/(2k+1)}  (2.4)

where V = ∫_{−1}^{1} K_{v}^{2}(x)dx (see Gasser and Müller, 1984), and it is assumed that g^{(k)}(t) ≠ 0. Insertion of the optimal bandwidth in the weighted integrated mean squared error

E ∫_{0}^{1}{ĝ_{n,v}(t) − g^{(v)}(t)}^{2}w(t)dt

yields (using an appropriate modification of the kernel at the boundary, see Müller, 1984)

n^{2(k−v)/(2k+1)}E ∫_{0}^{1}{ĝ_{n,v}(t) − g^{(v)}(t)}^{2}w(t)dt = c · Ψ_{g,σ}^{(v)}(f) + o(1) , (2.5)

where w is a positive continuous weight function on the interval [0, 1], c is a constant (independent of f) and

Ψ_{g,σ}^{(v)}(f) = ∫_{0}^{1}\{\frac{σ^{2}(t)w(t)}{f(t)}\}^{2(k−v)/(2k+1)}\{(g^{(k)}(t))^{2}w(t)\}^{(2v+1)/(2k+1)}dt. (2.6)

1.3 Optimal Designs minimizing the Maximum Integrated Mean Squared Error

Müller (1984) determined the design density f which minimizes the criterion (2.6) for fixed g, σ^{2}, i.e.

f^{∗}(t) = \frac{φ(t)}{∫_{0}^{1} φ(x)dx}  (3.1)

where φ(t) = [(σ^{2}(t))^{2(k−v)}(g^{(k)}(t))^{2(2v+1)}(w(t))^{2k+1}]^{1/(4k+1−2v)}. The optimal design is local in the sense of Chernoff (1953) and might be not robust with respect to misspecifications of the variance and mean function (see also Example 3.3). For this reason, we propose a minimax criterion for the determination of optimal designs and call a design density f^{∗} minimax optimalminimax optimal for the estimation of the vth derivative of the regression function if it minimizes

max{Ψ_{g,σ}^{(v)}(f) | (g, σ) ∈ F}.  (3.2)

Here F is an appropriate class of functions given either by

F_{2} = {(g, σ) ∈ C^{(k)}[0, 1] × Lip_{γ}[0, 1] |  (3.3)

∫_{0}^{1} σ^{2}(t)dt ≤ ε, ∫_{0}^{1}(g^{(k)}(t))^{2}w(t)dt ≤ η}

or by

F_{∞} = {(g, σ) ∈ C^{(k)}[0, 1] × Lip_{γ}[0, 1] |  (3.4)

sup_{t∈[0,1]} |σ^{2}(t)| ≤ ε, sup_{t∈[0,1]} |g^{(k)}(t)|^{2}w(t) ≤ η}.

Theorem 3.1 Let F be either defined by (3.3) or (3.4). Then the design with density

f^{∗}(t) = \frac{w(t)}{∫_{0}^{1} w(x)dx}

is optimal with respect to the minimax criterion (3.2).

Proof: We consider only the case F = F_{2}, the corresponding result for the supnorm is proved by similar arguments. Without loss of generality it is assumed that ∫_{0}^{1} w(t)dt = 1, i.e. f^{∗} = w. The proof is performed in two steps showing

(1)  sup{Ψ_{g,σ}^{(v)}(f^{∗}) | (g, σ) ∈ F_{2}} = [ε^{2(k−v)}η^{2v+1}]^{1/(2k+1)}
(2)  ∀ f ∃ (g, σ) ∈ F_{2} : Ψ_{g,σ}^{(v)}(f) ≥ [ε^{2(k−v)}η^{2v+1}]^{1/(2k+1)}

(1) The first part follows by a direct application of Hölder’s inequality observing that with the notation p = \frac{2k+1}{2(k−v)}, q = \frac{2k+1}{2v+1} the integrated mean squared error is given by

Ψ_{g,σ}^{(v)}(w) = ∫_{0}^{1} |σ^{2}(t)|^{1/p}|(g^{(k)}(t))^{2}w(t)|^{1/q}dt  (3.5)
≤ {∫_{0}^{1} σ^{2}(t)dt}^{1/p}{∫_{0}^{1}(g^{(k)}(t))^{2}w(t)dt}^{1/q} ≤ ε^{1/p}η^{1/q},

and that there is equality in (3.6) for the functions

σ(t) ≡ \sqrt{ε}; g(t) = \sqrt{η} ∫_{0}^{t} ∫_{0}^{s_{1}} … ∫_{0}^{s_{k−1}} \frac{ds}{\sqrt{w(s)}}  (3.6)

(note that (g, σ) ∈ F_{2}).

(2) Let f denote an arbitrary positive density on the interval [0, 1] and let

p′ = 1 + \frac{1}{p} = \frac{4k + 1 − 2v}{2k + 1} ; q′ = 1 + p = \frac{4k + 1 − 2v}{2(k − v)}  (3.7)

(note that 1/p′ + 1/q′ = 1). For the functions

σ_{∗}(t) = \sqrt{εw(t)}; g_{∗}(t) = \sqrt{η} \frac{t^{k}}{k!}  (3.8)

it follows (observing ∫_{0}^{1} w(t)dt = 1) that (g, σ) ∈ F_{2} and

Ψ_{g_{∗},σ_{∗}}^{(v)}(f) = ε^{1/p}η^{1/q} ∫_{0}^{1} |\frac{w(t)}{f(t)}|^{1/p}w(t)dt

= ε^{1/p}η^{1/q} ∫_{0}^{1}(w(t))^{p′} \frac{dt}{|f(t)|^{p′/q′}} = ε^{1/p}η^{1/q}∥\frac{w}{|f|^{1/q′}} ∥_{p′}^{p′}

≥ ε^{1/p}η^{1/q}\{\frac{∥w∥_{1}}{∥f^{1/q′}∥_{q′}} \}^{p′} = ε^{1/p}η^{1/q}

where ∥s∥_{p} = (∫_{0}^{1} |s(t)|^{p}dt)^{1/p} denotes the L^{p}-norm with respect to the Lebesgue measure and the inequality follows again from Hölder’s inequality. This proves (2) and completes the proof of Theorem 3.1.

□

Example 3.2. Note that for a constant weight function the uniform design turns out to be minimax optimal. This result can be explained intuitively, because in this case the optimization problem is invariant with respect to shifts of the design space. Consequently, the optimal design must be the uniform measure.

Example 3.3. Consider the case k = 2, v = 0 and w(x) ≡ 1. The locally optimal designlocally optimal design (with respect to the asymptotic integrated MSE) is given by (3.1) and we are interested how a misspecification of the mean or variance function affects the performance of this design. To this end we consider the following scenario

(1)  g(x) = e^{x};  σ(x) = 1
(2)  g(x) = e^{x};  σ(x) = e^{x}
(3)  g(x) = e^{x};  σ(x) = e^{2x}
(4)  g(x) = sin x; σ(x) = 1
(5)  g(x) = sin x; σ(x) = e^{x}
(6)  g(x) = sin x; σ(x) = e^{2x}.

For a misspecification we investigate four cases

(A)  g correct;  σ constant
(B)  g correct;  σ^{−1} instead of σ
(C)  g^{2} instead of g; σ correct
(D)  e^{g} instead of g; σ correct

and calculate the locally optimal design from Müller (1984). Note that in cases (A), (B) the variance function is misspecified, while (C), (D) correspond to an incorrect assumption for the mean function. The corresponding results are listed in Table 1.1, which also contains a column for the minimax optimal design which turns out to be the uniform design in these cases. The table shows the asymptotic integrated MSE obtained by the particular design f^{∗} in (3.1) for a misspecification relative to the asymptotic integrated MSE obtained by the design f^{∗} for the correct regression and variance function. The last column shows the corresponding ratio for the minimax optimal design.

A  B  C  D  minimax
1  1  1  0.4650 0.4625 0.9970
2 0.9562 0.8374 0.4519 0.4494 0.9326
3 0.9573 0.8406 0.4404 0.4381 0.9066
4 1 1 0.7193 0.6676 0.9776
5 0.9600 0.8502 0.8289 0.7977 0.8997
6 0.8658 0.5643 0.9379 0.7894 0.7894

Table 1.1: Asymptotic relative efficiency of the optimal design f^{∗} in (3.1) for various misspecifications of the mean and variance function. The last column: minimax design obtained from Theorem 3.1.

We observe that a misspecification of the mean and variance function has a serious impact on the performance of the resulting design. This dependency is even stronger, if the mean function is not adequately specified. The minimax design has a reasonable performance in all considered cases. For these reasons, the locally optimal designs proposed by Müller (1984) should not be used in practice unless information about the variance and mean structure is available. Otherwise the minimax designs proposed in this paper are recommended.

The authors would like to thank I. Gottschlich for typing most parts of this paper with considerable technical expertise. The financial support of the Deutsche Forschungsgemeinschaft (SFB 475, Reduction of complexity in multivariate data structures) is gratefully acknowledged.

Cheng, M.-H., Hall, P., Titterington, D. M. (1998). Optimal design for curve estimation by local linear smoothing. Bernoulli 4, 3-14.

Chernoff, H. (1953). Locally optimal design for estimating parameters. Annals of Mathematical Statistics 24, 586-602.

Fan, J., Gijbels, I. (1996). Local polynomial modelling and its applications. Chapman and Hall, London

Gasser, T., Müller, H. G. (1984). Estimating regression functions and their derivatives by kernel functions. Scandinavian Journal of Statistics 11, 171-185.

Härdle, W. (1990). Applied nonparametric regression. Cambridge University Press, Boston.

Müller, H. G. (1984). Optimal designs for nonparametric kernel regression. Statistics & Probability Letters 2, 285-290.

Müller, W. G. (1996). Optimal design for local fitting. Journal of Statistical Planning and Inference 55, 389-397.

Sacks, J., Ylvisaker, D. (1970). Designs for regression problems for correlated errors. Annals of Mathematical Statistics 41, 2057-2074.

Wand, M. P., Jones, M. C. (1995). Kernel smoothing. Chapman and Hall, London.