APPROXIMATIONS FOR CSISZÁR f–DIVERGENCE VIA MIDPOINT INEQUALITIES

S. S. DRAGOMIR, V. GLUŠČEVIĆ, AND C. E. M. PEARCE

Abstract. Using two midpoint inequalities arising in numerical integration we derive, for an absolutely continuous function f, an approximation for the Csiszár f–divergence of two probability distributions over a finite alphabet. Applications are made to some common divergence measures such as Kullback– Leibler distance, Hellinger discrimination and Renyi α–entropy.

1. Introduction

A common situation in information theory is the following. Two probability distributions p = (p_{1}, …, p_{n}), q = (q_{1}, …, q_{n}) are defined over an alphabet {a_{i}|1 ≤ i ≤ n}, p_{i}, q_{i} being the point probabilities associated with event a_{i} (i = 1, …, n). For example, p, q might represent a priori and a posteriori probability distributions associated with the alphabet.

It is useful to be able to quantify in some way the difference between such distributions p, q. A number of ways have been suggested for doing this. Thus the variational distance (l_{1}-distance) and information divergence (Kullback–Leibler divergence) are defined respectively by

(1.1)  V (p, q) := ∑_{i=1}^{n} |p_{i} − q_{i}|,

(1.2)  D(p, q) := ∑_{i=1}^{n} p_{i} ln \frac{p_{i}}{q_{i}} .

(see for example [10]). As in [1], we define the triangular discrimination between p and q by

(1.3)  Δ(p, q) := ∑_{i=1}^{n} \frac{|p_{i} − q_{i}|^{2}}{p_{i} + q_{i}} .

The Hellinger discrimination h^{2} is given by

(1.4)  h^{2}(p, q) := \frac{1}{2} ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} .

For applications it is important to know how these divergences compare with one another. The basic relations between V, Δ and h^{2} are

\frac{1}{2}V^{2}(p, q) ≤ Δ(p, q) ≤ V (p, q)

1991 Mathematics Subject Classification. 94A17, 26D15.

Key words and phrases. Csiszár f–divergence, midpoint inequalities, divergence measures.

and

2h^{2}(p, q) ≤ Δ(p, q) ≤ 4h^{2}(p, q)

(see LeCam [5] and Dacunha–Castelle [6]). From these we may deduce that

\frac{1}{8}V^{2}(p, q) ≤ h^{2}(p, q) ≤ \frac{1}{2}V (p, q).

The coefficients in these inequalities are best possible (cf. [1]).

The first half of this result has been improved by Kraft [7], who showed that

\frac{1}{8}V^{2}(p, q) ≤ h^{2}(p, q) (1 − \frac{1}{2}h^{2}(p, q)) .

We note also the important inequality

D(p, q) ≥ −2 ln (1 − h^{2}(p, q))

(see Dacunha–Castelle [6]). It follows from this that

D(p, q) ≥ 2h^{2}(p, q).

Again the coefficient 2 is best–possible (see [1]).

Csiszár [2]–[4] has introduced a versatile functional form which subsumes a number of the more popular choices of divergence measures, including those mentioned above. For a convex function f : [0, ∞) → R, the Csiszár f-divergence between p and q is defined by

(1.5)  I_{f}(p, q) := ∑_{i=1}^{n} q_{i}f (\frac{p_{i}}{q_{i}} ) .

Thus the variational distance is given by f(u) = |u−1|, Kullback–Leibler divergence by f(u) = u ln u, triangular discrimination by f(u) = (u−1)^{2}/(u+1) and Hellinger discrimination by f(u) = \frac{1}{2}(\sqrt{u} − 1)^{2}.

Most common choices of f, like the above, satisfy f(1) = 0, so that I_{f}(p, p) = 0. Convexity then ensures that I_{f}(p, q) is nonnegative. However as noted in [11], some additional flexibility for applications can be achieved by not insisting on convexity.

Closely related to the above are questions of approximation. For example, in practice exact values for each p_{i} and q_{i} may not be available, and it is desirable to be able to give an estimate for I_{f}(p, q) or to impose upper bounds on its value with only limited knowledge of p and q. To this end a recent paper [11] derived an approximation theorem by the use of Ostrowski’s integral inequality for absolutely continuous functions. It is convenient to invoke as a benchmark the chi–squared discrepancy measure

D_{χ^{2}}(p, q) := ∑_{i=1}^{n} \frac{p_{i} − q_{i})^{2}}{q_{i}} = ∑_{i=1}^{n} \frac{p_{i}^{2}}{q_{i}} − 1,

which arises from (1.5) as the particular case f(u) = (u − 1)^{2}. The main result derived in [11] is as follows.

Theorem A. Suppose that there exist real numbers r, R with

(1.6)  0 < r ≤ p_{i}/q_{i} ≤ R < ∞ for all i ∈ {1, ..., n}.

Assume that f : [r, R] → R is absolutely continuous on [r, R] and f′ ∈ L_{∞}[r, R], that is, that

∥f′ ∥_{∞} := ess sup_{t∈[r,R]} |f′(t)| < ∞.

Then

|I_{f}(p, q) − \frac{1}{R − r} ∫_{r}^{R} f(t)dt|

≤ [\frac{1}{4} + \frac{1}{(R − r)^{2}} \{D_{χ^{2}}(p, q) + (\frac{R + r}{2} − 1)^{2}\}] (R − r) ∥f′ ∥_{∞}

≤ \frac{1}{2}(R − r) ∥f′ ∥_{∞} .

Suppose that p and q are close in the sense that ε := R − r is small. Then Theorem A provides an approximation for the Csiszár f–divergence of accuracy of order O(ε) with only a mild assumption on f′. However, common choices for f in the engineering literature are frequently quite smooth, so that stronger assumptions are not necessarily restrictive. We can expect that, with appropriate restrictions, approximations to the Csiszár f–divergence can be obtained with higher–order accuracy in ε. In this article we derive approximations of order of accuracy O(ε^{2}) and O(ε^{3}).

We note at the outset that it would appear that a programme of obtaining such approximations can only to a limited extent be achieved in line with the goal in [11] of employing approximants to I_{f}(p, q) that involve p, q through r and R alone. This can achieved with accuracy order O(ε^{2}) but not in general with accuracy order O(ε^{3}).

In Section 2 we discuss some preliminary ideas and state and prove a general proposition useful in questions concerning approximations. In Section 3 we establish the basic results and in Section 4 address as examples some of the more common choices of f. We conclude in Section 5 with an application to mutual information in information theory.

2. Preliminaries

As an introduction to the ideas, suppose f′ is absolutely continuous on [r, R] and f′′ ∈ L_{∞}[r, R]. Then for x ∈ [r, R] we have

f(x) = f(1) + (x − 1)f′(1) + \frac{(x − 1)^{2}}{2} f′′(ξ_{x})

for some ξ_{x} ∈ [r, R]. Setting x = p_{i}/q_{i} gives for each i = 1, …, n that

q_{i}f(p_{i}/q_{i}) = q_{i}f(1) + (p_{i} − q_{i})f′(1) + \frac{(p_{i} − q_{i})^{2}}{2q_{i}} f′′(ξ_{i})

for some ξ_{i} ∈ [r, R]. Since ∑_{i=1}^{n} p_{i} = 1 = ∑_{i=1}^{n} q_{i}, summation over i yields

I_{f}(p, q) = f(1) + \frac{1}{2} ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} f′′(ξ_{i}),

from which we deduce that

|I_{f}(p, q) − f(1)| ≤ \frac{1}{2}D_{χ^{2}}(p, q) ∥f′′ ∥_{∞} .

It was shown in [11] that

(2.1)  D_{χ^{2}}(p, q) ≤ (R − 1)(1 − r).

Also the choices α = R − 1, β = 1 − r in the elementary inequality

αβ ≤ \frac{1}{4}(α + β)^{2}

give (R − 1)(1 − r) ≤ (R − r)^{2}/4, so that (2.1) can be extended to

(2.2)  D_{χ^{2}}(p, q) ≤ (R − 1)(1 − r) ≤ (R − r)^{2}/4.

This leads to

(2.3)  |I_{f}(p, q) − f(1)| ≤ \frac{ε^{2}}{8} ∥f′′ ∥_{∞} ,

in which the approximant does not depend on p, q. This approximation is, however, of limited utility, since as noted f(1) = 0 for many common choices of f.

A similar argument applies when f′′ is absolutely continuous on [r, R] and f′′′ ∈ L_{∞}[r, R]. Paralleling the line of reasoning above leads to

I_{f}(p, q) = f(1) + \frac{1}{2} ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} f′′(1) + \frac{1}{6} ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{3}}{q_{i}^{2}} f′′′(η_{i}),

where each η_{i} ∈ [r, R], whence

|I_{f}(p, q) − f(1) − \frac{1}{2}D_{χ^{2}}(p, q)f′′(1)| ≤ \frac{1}{6}D_{|χ|^{3}}(p, q) ∥f′′′ ∥_{∞} .

Here

D_{|χ|^{3}}(p, q) := ∑_{i=1}^{n} \frac{|p_{i} − q_{i}|^{3}}{q_{i}^{2}}

is the chi–cubed discrepancy, which corresponds to f(u) = (u − 1)^{3}.

It follows at once from (1.6) and ∑_{i=1}^{n} p_{i} = 1 = ∑_{i=1}^{n} q_{i} that r ≤ 1 ≤ R. Hence

(2.4)  |\frac{p_{i}}{q_{i}} − 1| ≤ max(1 − r, R − 1) ≤ R − r for i = 1, …, n,

and from (2.2) we have

(2.5)  D_{|χ|^{3}}(p, q) ≤ (R − r)D_{χ^{2}}(p, q) ≤ (R − r)(R − 1)(1 − r) ≤ \frac{1}{4}(R − r)^{3}.

Therefore

(2.6)  |I_{f}(p, q) − f(1) − \frac{1}{2}D_{χ^{2}}(p, q)f′′(1)| ≤ \frac{ε^{3}}{24} ∥f′′′ ∥_{∞} .

This time the approximant involves p, q nontrivially.

In this article we establish results similar to (2.3) and (2.6) but involving approximants for I_{f}(p, q) for which we can achieve tighter error bounds. Our basic tool is Ostrowski’s integral inequality for absolutely continuous mappings, Theorem B below. For a short proof and applications to numerical integration, the reader is referred to [8].

Theorem B. Assume that g : [a, b] → R is absolutely continuous with g′ ∈ L_{∞}[a, b]. Then

|g(x) − \frac{1}{b − a} ∫_{a}^{b} g(t)dt| ≤ [\frac{1}{4} + (\frac{x − \frac{a+b}{2}}{b − a} )^{2}] (b − a) ∥g′ ∥_{∞}

for all x ∈ [a, b].

We shall also make use of a result which both generalizes and improves (2.5). This concerns the divergence measure

D_{|χ|^{m}}(p, q) := ∑_{i=1}^{n} \frac{|p_{i} − q_{i}|^{m}}{q_{i}^{m−1}}

with m ≥ 1. Proposition 1. Suppose that (1.6) is satisfied with r < R and that m ≥ 1. Then

D_{|χ|^{m}}(p, q) ≤ \frac{(R − 1)(1 − r)}{R − r} [(1 − r)^{m−1} + (R − 1)^{m−1}] ≤ (\frac{R − r}{2} )^{m} .

The first inequality is an equality if and only if p, q form a boundary pair with respect to r and R, that is, for each i either p_{i}/q_{i} = r or p_{i}/q_{i} = R. The second inequality is an equality if and only if R + r = 2, that is, r and R are equidistant from unity.

Proof. Suppose that there exists a value of i ∈ {1, …, n} for which p_{i}/q_{i} = θ with r < θ < R. We may put

p_{i} = r \frac{R − θ}{R − r} q_{i} + R\frac{θ − r}{R − r} q_{i} = p_{i,1} + p_{i,2},

say, and similarly

q_{i} = \frac{R − θ}{R − r} q_{i} + \frac{θ − r}{R − r} q_{i} = q_{i,1} + q_{i,2}.

We then have

p_{i,1}/q_{i,1} = r and p_{i,2}/q_{i,2} = R.

We may replace the probability n–vectors p, q with probability (n + 1)–vectors by replacing p_{i} by the ordered pair p_{i,1}, p_{i,2} and q_{i} by the ordered pair q_{i,1}, q_{i,2}.

The contribution to D_{|χ|^{m}}(p, q) from p_{i} and q_{i} is

\frac{|p_{i} − q_{i}|^{m}}{q_{i}^{m−1}} = q_{i}|1 − θ|^{m}.

The contribution from p_{i,1}, q_{i,1} and p_{i,2}, q_{i,2} after such a replacement is

\frac{R − θ}{R − r} q_{i}(1 − r)^{m} + \frac{θ − r}{R − r} q_{i}(R − 1)^{m}.

We now show that the change gives rise to an increase in the divergence D_{|χ|^{m}}(p, q). We need to prove that

ϕ(θ) := \frac{R − θ}{R − r} (1 − r)^{m} + \frac{θ − r}{R − r} (R − 1)^{m} − |1 − θ|^{m}

is positive if r < θ < R.

First suppose that m > 1. Since

ϕ′′(θ) = −m(m − 1)(1 − θ)^{m−2} for r < θ < 1,

ϕ is strictly concave on (r, 1). Also ϕ(r) = 0 and ϕ(1) > 0, so that ϕ(θ) is positive on (r, 1]. Similarly

ϕ′′(θ) = −m(m − 1)(θ − 1)^{m−2} for 1 < θ < R,

so ϕ is strictly concave on (1, R). Since ϕ(1) > 0 and ϕ(R) = 0, ϕ is positive on [1, R). Thus ϕ(θ) > 0 on (r, R).

Now suppose m = 1. We have

ϕ′(θ) = 2\frac{R − 1}{R − r} > 0 for r < θ < 1

with ϕ(r) = 0, so ϕ is positive on (r, 1]. Further

ϕ′(θ) = −2\frac{1 − r}{R − r} < 0 for 1 < θ < R

with ϕ(R) = 0, so ϕ is also positive on [1, R). Thus we have again that ϕ(θ) > 0 on (r, R).

In either case we have demonstrated the claimed increase in the divergence. It follows that if we seek an upper bound to D_{|χ|^{m}}(p, q), we can without loss of generality restrict attention to boundary pairs p, q. After relabelling if necessary, we assume that

p_{i}/q_{i} = r for i = 1, …, k

and

p_{i}/q_{i} = R for i = k + 1, …, n.

Then

1 = ∑_{i=1}^{n} p_{i} = r ∑_{i=1}^{k} q_{i} + R ∑_{i=k+1}^{n} q_{i} = r ∑_{i=1}^{k} q_{i} + R [1 − ∑_{i=1}^{k} q_{i}]

and we have

∑_{i=1}^{k} q_{i} = \frac{R − 1}{R − r}, ∑_{i=k+1}^{n} q_{i} = \frac{1 − r}{R − r} .

Thus for a boundary pair p, q we have

D_{|χ|^{m}}(p, q) = ∑_{i=1}^{k} \frac{(q_{i} − p_{i})^{m}}{q_{i}^{m−1}} + ∑_{i=k+1}^{n} \frac{(p_{i} − q_{i})^{m}}{q_{i}^{m−1}}

= (1 − r)^{m} ∑_{i=1}^{k} q_{i} + (R − 1)^{m} ∑_{i=k+1}^{n} q_{i}

= \frac{R − 1}{R − r}(1 − r)^{m} + \frac{1 − r}{R − r} (R − 1)^{m}.

Now by elementary calculus we have that for x ≥ 0, y ≥ 0 with x + y = 2c, a constant, the function yx^{m} + xy^{m} takes its maximum value 2c^{m+1} uniquely when x = y = c. Applying this with x = 1 − r, y = R − 1 gives that

\frac{R − 1}{R − r}(1 − r)^{m} + \frac{1 − r}{R − r} (R − 1)^{m} ≤ (\frac{R − 1}{2})^{m} ,

with equality occurring when R − 1 = 1 − r. □

The main part of the result for the case m = 2 is simply (2.2). The result for m = 3 enables us to strengthen (2.6) to

|I_{f}(p, q) − f(1) − \frac{1}{2}D_{χ^{2}}(p, q)f′′(1)| ≤ \frac{ε^{3}}{48} ∥f′′′ ∥_{∞} .

We shall, however, look for a much closer approximant for I_{f}(p, q).

3. Basic results

We assume in what follows that there exist real numbers r, R satisfying (1.6). As noted in the introduction this entails that r ≤ 1 ≤ R.

For f satisfying the conditions of Theorem A, we define f^{∗} : [r, R] → R by

f^{∗}(u) := f(1) + (u − 1)f′ (\frac{1 + u}{2}) .

We remark that this gives f^{∗}(1) = f(1).

Theorem 1. Assume that f : [0, ∞) → R is such that f′ : [r, R] → R is absolutely continuous on [r, R] and f′′ ∈ L_{∞}[r, R]. Then

(3.1)  |I_{f}(p, q) − I_{f^{∗}}(p, q)| ≤ \frac{1}{4} ∥f′′ ∥_{∞} D_{χ^{2}}(p, q)

≤ \frac{1}{4} ∥f′′ ∥_{∞} (R − 1)(1 − r)

≤ \frac{1}{16} ∥f′′ ∥_{∞} (R − r)^{2}.

Proof. Taking x = (a + b)/2 in Theorem B yields the midpoint inequality

|g (\frac{a + b}{2}) (b − a) − ∫_{a}^{b} g(t)dt| ≤ \frac{1}{4} (b − a)^{2} ∥g′ ∥_{∞} .

The choices g = f′, a = 1 and b = x ∈ [r, R] provide

|f(x) − f(1) − (x − 1)f′ (\frac{1 + x}{2})| ≤ \frac{1}{4}(x − 1)^{2} ∥f′′ ∥_{∞}

for all x ∈ [r, R].

Putting x = p_{i}/q_{i} ∈ [r, R] gives

|q_{i}f (\frac{p_{i}}{q_{i}} ) − q_{i}f(1) − (p_{i} − q_{i})f′ (\frac{p_{i} + q_{i}}{2q_{i}} )| ≤ \frac{1}{4} (\frac{p_{i} − q_{i}}{q_{i}} )^{2} q_{i} ∥f′′ ∥_{∞}

for i ∈ {1, ..., n}.

Summing over i and using the generalized triangle inequality leads to

|I_{f}(p, q) − f(1) − ∑_{i=1}^{n}(p_{i} − q_{i})f′ (\frac{p_{i} + q_{i}}{2q_{i}} )| ≤ \frac{1}{4} ∥f′′ ∥_{∞} ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} ,

whence we have the first inequality in (3.1).

The second and third inequalities follow from (2.2). □

Corollary 1. Let f be as in Theorem 1. If ε > 0 and

0 ≤ R − r ≤ 4 · \sqrt{ε/ ∥f′′∥_{∞}},

then

|I_{f}(p, q) − I_{f^{∗}}(p, q)| ≤ ε.

A second corollary emphasizes the approximation aspect for distribution p and q which are close. Corollary 2. Let f : [0, 2] → R be such that the derivative f′ : [0, 2] → R is absolutely continuous and f′′ ∈ L_{∞}[0, 2]. If η ∈ (0, 1) and p(η), q(η) are such that

(3.2)  |\frac{p_{i}(η)}{q_{i}(η)} − 1| ≤ η for all i ∈ \{1, ..., n\} ,

then

(3.3)  I_{f} (p(η), q(η)) = I_{f^{∗}} (p(η), q(η)) + R_{f}(p, q, η)

and the remainder R_{f}(p, q, η) satisfies the estimate

(3.4)  R_{f}(p, q, η) ≤ \frac{1}{4} ∥f′′ ∥_{∞} D_{χ^{2}}(p(η), q(η)) ≤ \frac{1}{4} ∥f′′ ∥_{∞} η^{2}.

Proof. Set R = 1 + η and r = 1 − η in Theorem 1. □

Theorem 2. If f : [a, b] → R is such that f′′ is absolutely continuous and f′′′ ∈ L_{∞}[r, R], then

(3.5) |I_{f} (p, q) − I_{f^{∗}} (p, q)| ≤ \frac{1}{24} ∥f′′′ ∥_{∞} D_{|χ|}^{3}(p, q)

≤ \frac{1}{24} ∥f′′′ ∥_{∞} \frac{(R − 1)(1 − r)}{R − r} [(1 − r)^{2} + (R − 1)^{2}]

≤ \frac{1}{192} ∥f′′′ ∥_{∞} (R − r)^{3}.

The constants are best–possible.

Proof. For g : [a, b] → R such that g′ is absolutely continuous on [a, b] and g′′ ∈ L_{∞}[r, R], we have the midpoint inequality

|g (\frac{a + b}{2}) (b − a) − ∫_{a}^{b} g (t) dt| = \frac{1}{24} (b − a)^{3} ∥g′′ ∥_{∞}

arising in numerical integration. Setting g = f′, a = 1 and b = x ∈ [r, R] supplies

|f(x) − f(1) − (x − 1)f′ (\frac{1 + x}{2})| = \frac{1}{24} |x − 1|^{3} ∥f′′′ ∥_{∞}

for all x ∈ [r, R]. We may proceed as in the proof of Theorem 1 to deduce the first inequality in (3.5). Proposition 1 for m = 3 gives the others. To complete the proof it suffices to prove that the constant 1/24 in the first inequality is best–possible.

Choose f(u) = (u − 1)^{3}, so that I_{f}(p, q) = D_{|χ|^{3}}(p, q). Then

f^{∗}(u) = \frac{3}{4}(u − 1)^{3} = \frac{3}{4}f(u),

so that

|I_{f}(p, q) − I_{f^{∗}}(p, q)| = \frac{1}{4}D_{|χ|^{3}}(p, q).

As ∥f′′′ ∥_{∞} = 6, the first inequality in (3.5) for this choice of f is thus an equality.

□

Corollary 3. Let f be as in Theorem 2. If ε > 0 and

0 ≤ R − r ≤ 4 · \sqrt[3]{3ε/ ∥f′′′∥_{∞}},

then

|I_{f} (p, q) − I_{f^{∗}} (p, q)| ≤ ε.

Also, the following approximation result holds. Corollary 4. Let f : [0, 2] → R be so that f′′′ ∈ L_{∞}[0, 2]. If η ∈ (0, 1) and p(η), q(η) satisfy (3.2), then we have the representation (3.3) and the remainder R_{f}(p, q, η) satisfies the estimate

|R_{f}(p, q, η)| ≤ \frac{1}{24} ∥f′′′ ∥_{∞} η^{3}.

Proof. The result follows from Theorem 2 with r = 1 − η and R = 1 + η. □

4. Applications to some common divergence measures

When f : (0, ∞) → R is the convex map f(u) = u ln u, I_{f}(p, q) becomes the Kullback–Leibler distance D(p, q). We denote I_{f^{∗}}(p, q) by D^{∗}(p, q) and adopt a similar notation for other specific divergences. We have

D^{∗}(p, q) = ∑_{i=1}^{n}(p_{i} − q_{i}) [ln (\frac{p_{i} + q_{i}}{2q_{i}} ) + 1]

= ∑_{i=1}^{n}(p_{i} − q_{i}) ln (\frac{p_{i} + q_{i}}{2q_{i}} ) .

As f′′(u) = 1/u, we have

∥f′′ ∥_{∞} = sup_{u∈[r,R]} |f′′(u)| = 1/r,

and the conclusion of Theorem 1 becomes

(4.1)  |D(p, q) − D^{∗}(p, q)| ≤ \frac{1}{4r} D_{χ^{2}}(p, q) ≤ \frac{(R − 1)(1 − r)}{4r} ≤ \frac{(R − r)^{2}}{16r} .

An example of a concave f : (0, ∞) → R with f(1) = 0 is f(u) = ln u. We have

I_{f}(p, q) = ∑_{i=1}^{n} p_{i} ln \frac{p_{i}}{q_{i}} = −D(q, p)

and

−D^{∗}(q, p) = ∑_{i=1}^{n} (p_{i} − q_{i}) \frac{2q_{i}}{p_{i} + q_{i}} = 2 ∑_{i=1}^{n} q_{i} · \frac{p_{i} − q_{i}}{p_{i} + q_{i}} .

As f′′(u) = −1/u^{2}, we have ∥f′′ ∥_{∞} = 1/r^{2}.

Consequently (3.1) reads

(4.2) |D(q, p) − D^{∗}(q, p)| ≤ \frac{1}{4r^{2}} D_{χ^{2}}(p, q) ≤ \frac{(R − 1)(1 − r)}{4r^{2}} ≤ \frac{1}{16} (\frac{R}{r} − 1)^{2} .

For f : (0, ∞) → R given by f(u) = \frac{1}{2} (\sqrt{u} − 1)^{2}, the f–divergence I_{f}(p, q) becomes the Hellinger discrimination and

h^{2∗}(p, q) = \frac{1}{2} ∑_{i=1}^{n} (p_{i} − q_{i}) \frac{\sqrt{p_{i} + q_{i}} (\sqrt{p_{i} + q_{i}} − \sqrt{2q_{i}})}{p_{i} + q_{i}}.

As f′′(u) = 1/(4u^{2/3}) for u ∈ (0, ∞),

∥f′′ ∥_{∞} = sup_{u∈[r,R]} |f′′(u)| = \frac{1}{4r^{2/3}} .

By Theorem 1 we have

|h^{2}(p, q) − h^{2∗}(p, q)| ≤ \frac{1}{16r^{2/3}} D_{χ^{2}}(p, q) ≤ \frac{(R − 1)(1 − r)}{16r^{2/3}} ≤ \frac{1}{64r^{2/3}} (R − r)^{2} .

The Renyi α–order distance R_{α}(p, q) := ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α} is given by f : (0, ∞) → R

with f(u) = u^{α} (α > 1). This provides an example in which f may or may not be convex and f(1) ≠ 0. We have

R_{α}^{∗}(p, q) = 1 + \frac{α}{2^{α−1}} ∑_{i=1}^{n} (p_{i} − q_{i}) (\frac{p_{i} + q_{i}}{q_{i}} )^{α−1} ,

and as f′′(u) = α(α − 1)u^{α−2}, we have

∥f′′ ∥_{∞} = δ_{α}(r, R) := \{ \begin{matrix}α(α − 1)R^{α−2} & if 2 ≤ α < ∞ \\ α(α − 1)r^{α−2} & if 1 < α ≤ 2. \end{matrix}

Theorem 1 states that

|R_{α}(p, q) − R_{α}^{∗}(p, q)| ≤ \frac{1}{4}δ_{α}(r, R)D_{χ^{2}}(p, q)

≤ \frac{1}{4} × \{ \begin{matrix}α(α − 1)R^{α−2}(R − 1)(1 − r) & if 2 ≤ α < ∞ \\ α(α − 1)r^{α−2}(R − 1)(1 − r) & if 1 < α ≤ 2. \end{matrix}

We now turn to some bounds provided by Theorem 2. Reconsider the mapping f : (0, ∞) → R with f(u) = u ln u. We have f′′′(u) =

−1/u^{2}, ∥f′′′ ∥_{∞} = 1/r^{2} and by Theorem 2

|D(p, q) − D^{∗}(p, q)| ≤ \frac{1}{24r^{2}} D_{|χ|}^{3}(p, q) ≤ \frac{1}{192r^{2}} (R − r)^{3}.

For f(u) = ln u, we have f′′′(u) = 2/(3u^{3}) for u ∈ [r, R] and then ∥f′′′ ∥_{∞} =

2/(3r^{3}). By Theorem 2 we have

|D(q, p) − D^{∗}(q, p)| ≤ \frac{1}{36r^{3}} D_{|χ|}^{3}(p, q) ≤ \frac{1}{288} (\frac{R}{r} − 1)^{3} .

For f : (0, ∞) → R with f(u) = \frac{1}{2} (\sqrt{u} − 1)^{2} , we have f′′′ (u) = −\frac{1}{6} · u^{−5/3} for

u ∈ [r, R] and so ∥f′′′ ∥_{∞} = \frac{1}{6} · r^{−5/3}. Consequently Theorem 2 gives

|h^{2}(q, p) − h^{2∗}(q, p)| ≤ \frac{1}{144r^{5/3}} D_{|χ|}^{3}(p, q) ≤ \frac{1}{1152r^{5/3}} (R − r)^{3} .

Finally for f : (0, ∞) → R with f(u) = u^{α} (α > 1), we have f′′′ (u) = α(α − 1)(α − 2)u^{α−3} and

∥f′′′ ∥_{∞} = η_{α}(r, R) = \{ \begin{matrix}α(α − 1)(α − 2)R^{α−3} & if 3 ≤ α < ∞ \\ α(α − 1)(α − 2)r^{α−3} & if 1 < α ≤ 3. \end{matrix}

Consequently Theorem 2 provides

|R_{α}(p, q) − R_{α}^{∗}(p, q)| ≤ \frac{1}{24}η_{α}(r, R)D_{|χ|}^{3}(p, q)

≤ \frac{1}{192} × \{ \begin{matrix}α(α − 1)(α − 2)R^{α−2}(R − r)^{3} & if 3 ≤ α < ∞ \\ α(α − 1)(α − 2)r^{α−2}(R − r)^{3} & if 1 < α ≤ 3. \end{matrix}

5. Application to mutual information

We consider mutual information, which is a measure of the amount of information one random variable provides about another. It is the reduction of uncertainty about one variable due to the knowledge of the other. Definition 1. Consider two random variables X and Y with a joint probability mass function t(x, y) and marginal probability mass functions p(x) (x ∈ X) and q(y) (y ∈ Y). The mutual information is the relative entropy between the joint distribution and the product distribution, that is,

I(X; Y) = ∑_{x∈X} ∑_{y∈Y} t(x, y) ln [\frac{t(x, y)}{p(x)q(y)}] = D(t(x, y), p(x)q(y))

where D(·, ·) denotes Kullback–Leibler distance.

We assume in what follows that

(5.1)  s ≤ \frac{t(x, y)}{p(x)q(y)} ≤ S for all (x, y) ∈ X × Y.

We have immediately that s ≤ 1 ≤ S.

We may also define mutual information in a chi–squared sense, that is,

I_{χ^{2}}(X; Y) = ∑_{(x,y)∈X ×Y} \frac{t^{2}(x, y)}{p(x)q(y)} − 1.

Using inequality (4.1), we have the following proposition. Proposition 2. If t, p, q satisfy (5.1), we have

|I(X; Y) − ∑_{(x,y)∈X ×Y} [t(x, y) − p(x)q(y)] ln [\frac{t(x, y) + p(x)q(y)}{2p(x)q(y)}]|

≤ \frac{1}{4s}I_{χ^{2}}(X; Y) ≤ \frac{1}{4s}(S − 1)(1 − s) ≤ \frac{1}{16s}(S − s)^{2}.

A similar result follows from (4.2).

References

[1] F. Topsoe, Some inequalities for information divergence and related measures of discrimination, RGMIA Res. Rep. Coll., 2 (1999) No. 1, 85–98.

[2] I. Csiszár, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966) 185–188.

[3] I. Csiszár, Information-type measures of differences of probability distributions and indirect observation, Studia Sci. Math. Hung., 2 (1967) 299–318.

[4] I. Csiszár, On topological properties of f-divergence, Studia Sci. Math. Hung., 2 (1967) 329–339.

[5] L. LeCam, Asymptotic Methods in Statistical Decision Theory, New York: Springer, 1986.

[6] D. Dacunha-Castelle, Ecole d’été de probabilités de Saint–Fleur, VII–1977, Berlin, Heidelberg New York: Springer, 1978.

[7] C. Kraft, Some conditions of consistency and uniform consistency of statistical procedure, Univ. of California Publ. in Statistics, Ph.D. Thesis, 1 (1955), 125–142.

[8] S. S. Dragomir and S. Wang, Application of Ostrowski’s inequality to the estimation of error bounds for some special means and for some numerical quadrature rules, Appl. Math. Lett., 11 (1998), 105-109.

[9] D. S. Mitrinović, Analytic Inequalities, Springer Verlag, 1970.

[10] T. M. Cover and J. A. Thomas, Elements of Information Theory, John Wiley & Sons Inc., 1991.

[11] S. S. Dragomir, V. Gluščević and C. E. M. Pearce, Csiszár’s f–divergence, Ostrowski’s inequality and mutual information, submitted.

School of Communications and Informatics, Victoria University, PO Box 14428, MCMC, Melbourne, Victoria 8001, Australia

  E-mail address: sever@matilda.vu.edu.au

Royal Australian Air Force, ARDU, PO Box 1500, Salisbury SA 5108, Australia

  E-mail address: vgluscev@maths.adelaide.edu.au

Applied Mathematics Department, The University of Adelaide, Adelaide, SA 5005, Australia

  E-mail address: cpearce@maths.adelaide.edu.au