APPROXIMATING CSISZÁR f-DIVERGENCE VIA TWO INTEGRAL IDENTITIES AND APPLICATIONS

N.S. BARNETT, P. CERONE, S. S. DRAGOMIR, AND J. ROUMELIOTIS

Abstract. Some approximations of the Csiszár f−divergence via the use of the integral identities obtained in [8] and [9] and applications are given.

1. Introduction

One of the important issues in many applications of Probability Theory is finding an appropriate measure of distance (or difference or discrimination ) between two probability distributions. A number of divergence measures for this purpose have been proposed and extensively studied by Jeffreys [23], Kullback and Leibler [32], Rényi [42], Havrda and Charvat [21], Kapur [26], Sharma and Mittal [15], Burbea and Rao [5], Rao [41], Lin [34], Csiszár [14], Ali and Silvey [1], Vajda [51], Shioya and Da-te [45] and others (see for example [26] and the references therein).

These measures have been applied in a variety of fields such as: anthropology [37], genetics [39], finance, economics, and political science [47], [48], [43], biology [39], the analysis of contingency tables [20], approximation of probability distributions [27], [24], signal processing [25], [3] and pattern recognition [10], [53]. A number of these measures of distance are specific cases of Csiszár f-divergence and so further exploration of this concept will have a flow on effect to other measures of distance and to areas in which they are applied.

Assume that a set χ and the σ−finite measure μ are given. Consider the set of all probability densities on μ to be Ω := {p|p : χ → ℝ, p (x) ≥ 0, ∫_{χ} p (x) dμ (x) = 1}. The Kullback-Leibler divergence [32] is well known among the information divergences. It is defined as:

(1.1)  D_{KL} (p, q) := ∫_{χ} p (x) log [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω,

where log is to base 2.

In Information Theory and Statistics, various divergences are applied in addition to the Kullback-Leibler divergence. These are the: variation distance D_{v}, Hellinger distance D_{H} [22], χ^{2}−divergence D_{χ}^{2}, α−divergence D_{α}, Bhattacharyya distance D_{B} [4], Harmonic distance D_{Ha}, Jeffreys distance D_{J} [23], triangular discrimination D_{Δ} [49], etc... They are defined as follows:

(1.2)  D_{v} (p, q) := ∫_{χ} |p (x) − q (x)| dμ (x) , p, q ∈ Ω;

Date: September 27, 2000.

1991 Mathematics Subject Classification. Primary 94A99; Secondary 26D15.

Key words and phrases. Csiszár f−divergence, Analytic Inequalities.

(1.3)  D_{H} (p, q) := ∫_{χ} [\sqrt{p (x)} − \sqrt{q (x)}]^{2} dμ (x) , p, q ∈ Ω;

(1.4)  D_{χ}^{2} (p, q) := ∫_{χ} p (x) [(\frac{q (x)}{p (x)})^{2} − 1] dμ (x) , p, q ∈ Ω;

(1.5)  D_{α} (p, q) := \frac{4}{1 − α^{2}} [1 − ∫_{χ} [p (x)]^{\frac{1−α}{2}} [q (x)]^{\frac{1+α}{2}} dμ (x)] , p, q ∈ Ω;

(1.6)  D_{B} (p, q) := ∫_{χ} \sqrt{p (x) q (x)}dμ (x) , p, q ∈ Ω;

(1.7)  D_{Ha} (p, q) := ∫_{χ} \frac{2p (x) q (x)}{p (x) + q (x)}dμ (x) , p, q ∈ Ω;

(1.8)  D_{J} (p, q) := ∫_{χ} [p (x) − q (x)] ln [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω;

(1.9)  D_{Δ} (p, q) := ∫_{χ} \frac{[p (x) − q (x)]^{2}}{p (x) + q (x)} dμ (x) , p, q ∈ Ω.

For other divergence measures, see the paper [26] by Kapur or the book on line [46] by Taneja. For a comprehensive collection of preprints available on line, see the

RGMIA web site http://rgmia.vu.edu.au/papersinfth.html

Csiszár f−divergence is defined as follows [14]

(1.10)  I_{f} (p, q) := ∫_{χ} q (x) f [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω,

where f is convex on (0, ∞). It is assumed that f (u) is zero and strictly convex at u = 1. By appropriately defining this convex function, various divergences are derived. All the above distances (1.1) − (1.9), are particular instances of Csiszár f−divergence. There are also many others which are not in this class (see for example [26] or [46]). For the basic properties of Csiszár f−divergence see [12]- [16].

The main aim of this paper is to point out some representations of Csiszár f−divergence for the function which has the (n − 1) −derivative (n ≥ 1) absolutely continuous by employing two recent integral identities from [8] and [9] involving interior point and end point identities. Estimates for the remainder are also provided.

2. Representation of Csiszár f−Divergence

In [8] (see also [6]), the authors proved the following integral identity generalising the mid-point rule.

CSISZÁR f-DIVERGENCE 3

Lemma 1. Let g : [a, b] → ℝ be a function such that g^{(n−1)} is absolutely continuous. Then for all x ∈ [a, b], we have the identity:

(2.1) ∫_{a}^{b} g (t) dt = ∑_{k=0}^{n−1} \frac{1}{(k + 1)!} [(b − x)^{k+1} + (−1)^{k} (x − a)^{k+1}] g^{(k)} (x)

+ (−1)^{n} ∫_{a}^{b} K_{n} (x, t) g^{(n)} (t) dt,

where the kernel K_{n} : [a, b]^{2} → ℝ is given by

(2.2)  K_{n} (x, t) := \{ \begin{matrix}\frac{(t−a)^{n}}{n!} , & a ≤ t ≤ x ≤ b \\ \frac{(t−b)^{n}}{n!} , & a ≤ x < t ≤ b. \end{matrix}

In particular, if x = \frac{a+b}{2}, then

(2.3)  ∫_{a}^{b} g (t) dt = ∑_{k=0}^{n−1} \frac{1}{2^{k+1}} [\frac{1 + (−1)^{k}}{(k + 1)!} ] (b − a)^{k+1} g^{(k)} (\frac{a + b}{2})

+ (−1)^{n} ∫_{a}^{b} M_{n} (t) g^{(n)} (t) dt,

where

(2.4)  M_{n} (t) := \{ \begin{matrix}\frac{(t−a)^{n}}{n!} , & a ≤ t ≤ \frac{a+b}{2} \\ \frac{(t−b)^{n}}{n!} , & \frac{a+b}{2} < t ≤ b. \end{matrix}

Another integral identity generalising the trapezoid rule is embodied in the following lemma (see [9] or [7]).

Lemma 2. Let g : [a, b] → ℝ be as in Lemma 1. Then for all x ∈ [a, b], we have the representation

∫_{a}^{b} g (t) dt (2.5)

= ∑_{k=0}^{n−1} \frac{1}{(k + 1)!} [(x − a)^{k+1} g^{(k)} (a) + (−1)^{k} (b − x)^{k+1} g^{(k)} (b)]

+\frac{1}{n!} ∫_{a}^{b} (x − t)^{n} g^{(n)} (t) dt,

In particular, if x = \frac{a+b}{2}, then

(2.6) ∫_{a}^{b} g (t) dt = ∑_{k=0}^{n−1} \frac{1}{2^{k+1} (k + 1)!} (b − a)^{k+1} [g^{(k)} (a) + (−1)^{k} g^{(k)} (b)]

+\frac{(−1)^{n}}{n!} ∫_{a}^{b} (t − \frac{a + b}{2})^{n} g^{(n)} (t) dt.

4 N.S. BARNETT, P. CERONE, S. S. DRAGOMIR, AND J. ROUMELIOTIS

Let us consider x = (1 − λ) a + λb, λ ∈ [0, 1], then from (2.1) we obtain ∫_{a}^{b} g (t) dt (2.7)

= ∑_{k=0}^{n−1} \frac{1}{(k + 1)!} [(1 − λ)^{k+1} + (−1)^{k} λ^{k+1}] (b − a)^{k+1} g^{(k)} ((1 − λ) a + λb)
+ (−1)^{n} ∫_{a}^{b} K_{n} ((1 − λ) a + λb, t) g^{(n)} (t) dt,

and from (2.5) we obtain ∫_{a}^{b} g (t) dt (2.8)

= ∑_{k=0}^{n−1} \frac{1}{(k + 1)!} [λ^{k+1}g^{(k)} (a) + (−1)^{k} (1 − λ)^{k+1} g^{(k)} (b)] (b − a)^{k+1}

+\frac{1}{n!} ∫_{a}^{b} [(1 − λ) a + λb − t]^{n} g^{(n)} (t) dt.

We are now able to state and prove the following representation result for the Csiszár f−divergence. Theorem 1. Let f : ℝ → ℝ be a function such that f^{(n)} is absolutely continuous on any [a, b] ⊂ ℝ. If p, q ∈ Ω, then I_{f} (p, q) (2.9)

= f (1) + ∑_{k=0}^{n−1} \frac{1}{(k + 1)!} [(1 − λ)^{k+1} + (−1)^{k} λ^{k+1}]

×I_{(·−1)^{k+1}f^{(k+1)}[(1−λ)+λ·]} (p, q) + (−1)^{n} ∫_{Γ} q (x)

× (∫1^{\frac{p(x)}{q(x)}} K_{n} [\frac{(1 − λ) q (x) + λp (x)}{q (x)}, t] f^{(n+1)} (t) dt) dμ (x) , λ ∈ [0, 1]

and I_{f} (p, q) (2.10)

= f (1) + ∑_{k=0}^{n−1} \frac{λ^{k+1}}{(k + 1)!}f^{(k+1)} (1) D_{k} (p, q)

+ ∑_{k=0}^{n−1} \frac{(−1)^{k} (1 − λ)^{k+1}}{(k + 1)!} I_{(·−1)^{k+1}f^{(k+1)}(·)} (p, q) + \frac{1}{n!} ∫_{Γ} [q (x)]^{−n+1}

× (∫1^{\frac{p(x)}{q(x)}} [λp (x) + [(1 − λ) − t] q (x)]^{n} f^{(n+1)} (t) dt) dμ (x) ,

where

D_{k} (p, q) = ∫_{Γ} [p (x) − q (x)]^{k} [q (x)]^{−k+1} dμ (x) .

Proof. If we apply the identity (2.7) for f′, we get

(2.11)  f (b) = f (a) + ∑_{k=0}^{n−1} \frac{1}{(k + 1)!} [(1 − λ)^{k+1} + (−1)^{k} λ^{k+1}]
× (b − a)^{k+1} f^{(k+1)} ((1 − λ) a + λb)
+ (−1)^{n} ∫_{a}^{b} K_{n} [(1 − λ) a + λb, t] f^{(n+1)} (t) dt.

If in (2.11) we choose b = \frac{p(x)}{q(x)}, x ∈ Γ and a = 1, we get

f (\frac{p (x)}{q (x)}) (2.12)

= f (1) + ∑_{k=0}^{n−1} \frac{1}{(k + 1)!} [(1 − λ)^{k+1} + (−1)^{k} λ^{k+1}]

×\begin{matrix}(p (x) − q (x)) & k+1 \\ [q (x)] & k+1 \end{matrix} · f^{(k+1)} [\frac{(1 − λ) q (x) + λp (x)}{q (x)}]

+ (−1)^{n} ∫1^{\frac{p(x)}{q(x)}} K_{n} [\frac{(1 − λ) q (x) + λp (x)}{q (x)}, t] f^{(n+1)} (t) dt

for all x ∈ Γ.

If we multiply (2.12) by q (x) ≥ 0 (x ∈ Γ), integrate on Γ and take into account that ∫_{Γ} q (x) dμ (x) = 1, then we get the representation (2.9).

If we apply the identity (2.8) for f′, we get

(2.13)  f (b) = f (a) + ∑_{k=0}^{n−1} \frac{1}{(k + 1)!} [λ^{k+1}f^{(k+1)} (a)

+ (−1)^{k} (1 − λ)^{k+1} f^{(k+1)} (b)] (b − a)^{k+1}

+\frac{1}{n!} ∫_{a}^{b} [(1 − λ) a + λb − t]^{n} f^{(n+1)} (t) dt.

If in (2.13) we choose b = \frac{p(x)}{q(x)}, x ∈ Γ and a = 1, we get

f (\frac{p (x)}{q (x)}) (2.14)

= f (1) + ∑_{k=0}^{n−1} \frac{λ^{k+1}}{(k + 1)!}f^{(k+1)} (1) (\frac{p (x)}{q (x)} − 1)^{k+1}

+ ∑_{k=0}^{n−1} \frac{(−1)^{k} (1 − λ)^{k+1}}{(k + 1)!} f^{(k+1)} (\frac{p (x)}{q (x)}) (\frac{p (x)}{q (x)} − 1)^{k+1}

+\frac{1}{n!} ∫1^{\frac{p(x)}{q(x)}} [\frac{(1 − λ) q (x) + λp (x)}{q (x)} − t]^{n} f^{(n+1)} (t) dt,

for all x ∈ Γ.

If we multiply (2.14) by q (x) ≥ 0 (x ∈ Γ), integrate on Γ and take into account that ∫_{Γ} q (x) dμ (x) = 1, we get the representation (2.10).

Remark 1. If in (2.9) we choose λ = 0 or, λ = 1 or, λ = \frac{1}{2}, we get, respectively

(2.15) I_{f} (p, q) = f (1) + ∑_{k=0}^{n−1} \frac{1}{(k + 1)!}f^{(k+1)} (1) D_{k} (p, q) + (−1)^{n} ∫_{Γ} q (x)

× (∫1^{\frac{p(x)}{q(x)}} K_{n} (1, t) f^{(n+1)} (t) dt) dμ (x) ,

I_{f} (p, q) (2.16)

= f (1) + ∑_{k=0}^{n−1} \frac{(−1)^{k}}{(k + 1)!}I_{(·−1)^{k+1}f^{(k+1)}(·)} (p, q)

+ (−1)^{n} ∫_{Γ} q (x) (∫1^{\frac{p(x)}{q(x)}} K_{n} (\frac{p (x)}{q (x)}, t) f^{(n+1)} (t) dt) dμ (x)

and I_{f} (p, q) (2.17)

= f (1) + ∑_{k=0}^{n−1} [\frac{1 + (−1)^{k}}{2^{k+1} (k + 1)!}] I_{(·−1)^{k+1}f^{(k+1)}(\frac{1+·}{2})} (p, q)

+ (−1)^{n} ∫_{Γ} q (x) (∫1^{\frac{p(x)}{q(x)}} K_{n} (\frac{q (x) + p (x)}{2q (x)}, t) f^{(n+1)} (t) dt) dμ (x) .

Remark 2. If in (2.10) we choose λ = 0, or λ = 1 or, λ = \frac{1}{2}, we get, respectively

(2.18) I_{f} (p, q) = f (1) + ∑_{k=0}^{n−1} \frac{(−1)^{k}}{(k + 1)!}I_{(·−1)^{k+1}f^{(k+1)}(·)} (p, q)

+\frac{1}{n!} ∫_{Γ} q (x) (∫1^{\frac{p(x)}{q(x)}} (1 − t)^{n} f^{(n+1)} (t) dt) dμ (x) ,

(2.19) I_{f} (p, q) = f (1) + ∑_{k=0}^{n−1} \frac{f^{(k+1)} (1)}{(k + 1)!}D_{k} (p, q) + \frac{1}{n!} ∫_{Γ} [q (x)]^{−n+1}

× (∫1^{\frac{p(x)}{q(x)}} (p (x) − tq (x))^{n} f^{(n+1)} (t) dt) dμ (x)

and I_{f} (p, q) (2.20)

= f (1) + ∑_{k=0}^{n−1} \frac{f^{(k+1)} (1)}{2^{k+1} (k + 1)!}D_{k} (p, q)

+ ∑_{k=0}^{n−1} \frac{(−1)^{k}}{2^{k+1} (k + 1)!}I_{(·−1)^{k+1}f^{(k+1)}(·)} (p, q) + \frac{1}{n!} ∫_{Γ} [q (x)]^{−n+1}

× (∫1^{\frac{p(x)}{q(x)}} [\frac{1}{2}p (x) + (\frac{1}{2} − t) q (x)]^{n} f^{(n+1)} (t) dt) dμ (x) .

3. Bounds for the Remainder

In this section we point out some bounds for the remainders in the representations (2.9) and (2.10), i.e.,

(3.1) R_{f} (p, q) : = (−1)^{n} ∫_{Γ} q (x) × (∫1^{\frac{p(x)}{q(x)}} K_{n} [\frac{(1 − λ) q (x) + λp (x)}{q (x)}, t]

×f^{(n+1)} (t) dt)dμ (x)

and

(3.2)  R̃_{f} (p, q) : = \frac{1}{n!} ∫_{Γ} [q (x)]^{−n+1} × (∫1^{\frac{p(x)}{q(x)}} [λp (x) +

[(1 − λ) − t] q (x)]^{n}f^{(n+1)} (t) dt)dμ (x) ,

where p, q ∈ Ω, λ ∈ [0, 1] and K_{n} (·, ·) is the kernel defined in equation (2.2). For a, b ∈ ℝ, let us denote

∥f∥_{[a,b],p} := |∫_{a}^{b} |f (t)|^{p} dt|^{\frac{1}{p}} , p ≥ 1

and

∥f∥_{[a,b],∞} := ess sup_{{t∈[a,b] \atop (t∈[b,a])}} |f (t)| .

In order to obtain bounds on R_{f} (p, q) as given in (3.1), we need to consider integrals of the form

I_{1} (z) := ∫_{1}^{z} K_{n} [(1 − λ) · 1 + λz, t] f^{(n+1)} (t) dt, z ∈ (0, ∞) .

Thus

|I_{1} (z)| ≤ |∫_{1}^{z} |K_{n} [(1 − λ) · 1 + λz, t]| |f^{(n+1)} (t)| dt|
≤ ∥f^{(n+1)}∥_{[1,z],∞} |∫_{1}^{z} |K_{n} ((1 − λ) · 1 + λz, t)| dt|

= \frac{1}{n!} ∥f^{(n+1)}∥_{[1,z],∞} |∫_{1}^{(1−λ)·1+λz} |t − 1|^{n} dt + ∫_{(1−λ)·1+λz}^{z} |t − z|^{n} dt|

= \frac{1}{n!} [\frac{|(1 − λ) + λz − 1|^{n+1} + |z − (1 − λ) · 1 − λz|^{n+1}}{n + 1} ] ∥f^{(n+1)}∥_{[1,z],∞}

= \frac{1}{n!} [\frac{λ^{n+1} |z − 1|^{n+1} + (1 − λ)^{n+1} |z − 1|^{n+1}}{n + 1} ] ∥f^{(n+1)}∥_{[1,z],∞}

= \frac{|z − 1|^{n+1}}{(n + 1)!} [λ^{n+1} + (1 − λ)^{n+1}] ∥f^{(n+1)}∥_{[1,z],∞} .

8 N.S. BARNETT, P. CERONE, S. S. DRAGOMIR, AND J. ROUMELIOTIS

Using Hölder’s inequality, we may write for α > 1, \frac{1}{α} + \frac{1}{β} = 1, that:

|I_{1} (z)| ≤ ∥f^{(n+1)}∥_{[1,z],β} |∫_{1}^{z} |K_{n} ((1 − λ) · 1 + λz, t)|^{α} dt|^{\frac{1}{α}} .

However,

|∫_{1}^{z} |K_{n} ((1 − λ) · 1 + λz, t)|^{α} dt|^{\frac{1}{α}}
= \frac{1}{n!} |∫_{1}^{(1−λ)·1+λz} |t − 1|^{αn} dt + ∫_{(1−λ)·1+λz}^{z} |t − z|^{αn} dt|^{\frac{1}{α}}
= \frac{1}{n!} [\frac{|(1 − λ) + λz − 1|^{αn+1} + |z − (1 − λ) · 1 − λz|^{αn+1}}{αn + 1} ]^{\frac{1}{α}}
= \frac{1}{n!} [\frac{λ^{αn+1} |z − 1|^{αn+1} + (1 − λ)^{αn+1} |z − 1|^{αn+1}}{αn + 1} ]^{\frac{1}{α}}
= \frac{|z − 1|^{n+ \frac{1}{α}}}{n! (αn + 1)^{\frac{1}{α}}} [λ^{αn+1} + (1 − λ)^{αn+1}]^{\frac{1}{α}}

and then:

|I_{1} (z)| ≤ ∥f^{(n+1)}∥_{[1,z],β} \frac{|z − 1|^{n+ \frac{1}{α}}}{n! (αn + 1)^{\frac{1}{α}}} [λ^{αn+1} + (1 − λ)^{αn+1}]^{\frac{1}{α}} .

Finally, we observe that

sup_{t∈[1,z]} |K_{n} ((1 − λ) · 1 + λz, t)|
= \frac{1}{n!} max \{((1 − λ) + λz − 1)^{n} + (z − (1 − λ) · 1 − λz)^{n}\}
= \frac{1}{n!} (z − 1)^{n} (max \{λ, 1 − λ\})^{n}
= \frac{1}{n!} |z − 1|^{n} [\frac{1}{2} + |λ − \frac{1}{2}|]^{n}

and then

|I_{1} (z)| ≤ \frac{1}{n!} |z − 1|^{n} [\frac{1}{2} + |λ − \frac{1}{2}|]^{n} ∥f^{(n+1)}∥_{[1,z],1} .

Using the above inequalities, we may state the following result (3.3)

|I_{1} (z)| ≤ \{ \begin{matrix}\frac{|z−1|^{n+1}}{(n+1)!} & [λ^{n+1} + (1 − λ)^{n+1}] & ∥f^{(n+1)}∥ & [1,z],∞ \\ \frac{|z−1|^{n+ \frac{1}{α}}}{n!(αn+1)^{\frac{1}{α}}} & [λ^{αn+1} + (1 − λ)^{αn+1}] & \frac{1}{α} & ∥f^{(n+1)}∥ & [1,z],β \\ α > 1, \frac{1}{α} + \frac{1}{β} = 1 \\ \frac{1}{n!} & |z − 1| & n & [\frac{1}{2} + |λ − \frac{1}{2} |] & n & ∥f^{(n+1)}∥ & [1,z],1 \end{matrix} \} =: κ (z, n)

for all z > 0, n ∈ ℕ.

We are now able to state the following theorem pertaining to the remainder R_{f} (p, q). Theorem 2. Assume that the function f is as in Theorem 1. If p, q ∈ Ω, then we have the inequality

|R_{f} (p, q)| (3.4)

≤ A := \{ \begin{matrix}\frac{1}{(n+1)!} & [λ^{n+1} + (1 − λ)^{n+1}] \\ × ∫_{Γ} \{[q (x)]^{−n} |p (x) − q (x)|^{n+1} ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],∞}\} dμ (x) \\ \frac{1}{n!(αn+1)^{\frac{1}{α}}} & [λ^{αn+1} + (1 − λ)^{αn+1}] & \frac{1}{α} & ∫ & Γ & \{[q (x)]^{−n− \frac{1}{α} +1} \\ × |p (x) − q (x)|^{n+ \frac{1}{α}} ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],β} \} dμ (x) \\ where α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n!} & [\frac{1}{2} + |λ − \frac{1}{2} |] & n & ∫ & Γ & [q (x)] & −n+1 & |p (x) − q (x)| & n \\ × ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],1} dμ (x) . \end{matrix}

Moreover, if we assume that 0 ≤ r ≤ \frac{p(x)}{q(x)} ≤ R < ∞, x ∈ Γ, then the second term in (3.4) can be upper bounded by

B (3.5)

: = \{ \begin{matrix}\frac{1}{(n+1)!} & [λ^{n+1} + (1 − λ)^{n+1}] & ∥f^{(n+1)}∥ & [r,R],∞ \\ × ∫_{Γ} [q (x)]^{−n} |p (x) − q (x)|^{n+1} dμ (x) \\ \frac{1}{n!(αn+1)^{\frac{1}{α}}} & [λ^{αn+1} + (1 − λ)^{αn+1}] & \frac{1}{α} & ∥f^{(n+1)}∥ & [r,R],β \\ × ∫_{Γ} [q (x)]^{−n− \frac{1}{α} +1} |p (x) − q (x)|^{n+ \frac{1}{α}} dμ (x) \\ if α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n!} [\frac{1}{2} + |λ − \frac{1}{2} |]^{n} ∥f^{(n+1)}∥_{[r,R],1} ∫_{Γ} [q (x)]^{−n+1} |p (x) − q (x)|^{n} dμ (x) \end{matrix}

≤ C := \{ \begin{matrix}\frac{1}{(n+1)!} & [λ^{n+1} + (1 − λ)^{n+1}] & ∥f^{(n+1)}∥ & [r,R],∞ & (R − r) & n+1 \\ \frac{1}{n!(αn+1)^{\frac{1}{α}}} & [λ^{αn+1} + (1 − λ)^{αn+1}] & \frac{1}{α} & ∥f^{(n+1)}∥ & [r,R],β & (R − r) & n+ \frac{1}{α} \\ if α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n!} [\frac{1}{2} + |λ − \frac{1}{2} |]^{n} ∥f^{(n+1)}∥_{[r,R],1} (R − r)^{n} . \end{matrix}

The proof of (3.4) follows by the inequality (3.3) choosing z = \frac{p(x)}{q(x)} and integrating.

The proof of (3.5) follows by the fact that |\frac{p(x)}{q(x)} − 1| ≤ R − r for all x ∈ Γ.

We omit the details.

The following corollary may be useful in practical applications. Corollary 1. With the assumptions of Theorem 2, we have the inequality:

|I_{f} (p, q) − f (1) − ∑_{k=0}^{n−1} \frac{f^{(k+1)} (1)}{(k + 1)!}D_{k} (p, q)| (3.6)

≤ \{ \begin{matrix}\frac{1}{(n+1)!} ∫_{Γ} [q (x)]^{−n} |p (x) − q (x)|^{n+1} ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],∞} dμ (x) \\ \frac{1}{n!(αn+1)^{\frac{1}{α}}} & ∫ & Γ & [q (x)] & −n− \frac{1}{α} +1 \\ × |p (x) − q (x)|^{n+ \frac{1}{α}} ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],β} dμ (x) \\ where α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n!} ∫_{Γ} [q (x)]^{−n+1} |p (x) − q (x)|^{n} ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],1} dμ (x) \end{matrix}

= : M_{1}.

Furthermore, if we assume that r ≤ \frac{p(x)}{q(x)} ≤ R < ∞, x ∈ Γ, then we have

M_{1} ≤ \{ \begin{matrix}\frac{1}{(n+1)!} ∥f^{(n+1)}∥_{[r,R],∞} ∫_{Γ} [q (x)]^{−n} |p (x) − q (x)|^{n+1} dμ (x) \\ \frac{1}{n!(αn+1)^{\frac{1}{α}}} & ∥f^{(n+1)}∥ & [r,R],β & ∫ & Γ & [q (x)] & −n− \frac{1}{α} +1 \\ × |p (x) − q (x)|^{n+ \frac{1}{α}} dμ (x) \\ where α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n!} ∥f^{(n+1)}∥_{[r,R],1} ∫_{Γ} [q (x)]^{−n+1} |p (x) − q (x)|^{n} dμ (x) \end{matrix}

= : M_{2}

≤ \{ \begin{matrix}\frac{1}{(n+1)!} & ∥f^{(n+1)}∥ & [r,R],∞ & (R − r) & n+1 \\ \frac{1}{n!(αn+1)^{\frac{1}{α}}} & ∥f^{(n+1)}∥ & [r,R],β & (R − r) & n+ \frac{1}{α} \\ where α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n!} & ∥f^{(n+1)}∥ & [r,R],1 & (R − r) & n \end{matrix} \} =: M_{3}

and

|I_{f} (p, q) − f (1) − ∑_{k=0}^{n−1} \frac{(−1)^{k}}{(k + 1)!}I_{(·−1)^{k+1}f^{(k+1)}(·)} (p, q)|

≤ M_{1} ≤ M_{2} ≤ M_{3}

CSISZÁR f-DIVERGENCE 11

and if 0 ≤ r ≤ \frac{p(x)}{q(x)} ≤ R < ∞, x ∈ Γ, then

|I_{f} (p, q) − f (1) − ∑_{k=0}^{n−1} [\frac{1 + (−1)^{k}}{2^{k+1} (k + 1)!}] I_{(·−1)^{k+1}f^{(k+1)}(\frac{1+·}{2})} (p, q)|

≤ \frac{1}{2^{n}} M_{1} ≤ \frac{1}{2^{n}} M_{2} ≤ \frac{1}{2^{n}} M_{3}.

Now, to obtain the bound on R̃_{f} (p, q) as defined in (3.2), consider the integral

I_{2} (z) := \frac{1}{n!} ∫_{1}^{z} ((1 − λ) · 1 + λz − t)^{n} f^{(n+1)} (t) dt,

from which we have

|I_{2} (z)| ≤ ∥f^{(n+1)}∥_{[1,z],∞} \frac{1}{n!} |∫_{1}^{z} |(1 − λ) · 1 + λz − t|^{n}| dt
= \frac{1}{n!} |∫_{1}^{(1−λ)·1+λz} |(1 − λ) · 1 + λz − t|^{n} dt

+ ∫_{(1−λ)·1+λz}^{z} |(1 − λ) · 1 + λz − t|^{n} dt| ∥f^{(n+1)}∥_{[1,z],∞}

= \frac{1}{n!} · [\frac{|(1 − λ) · 1 + λz − 1|^{n+1} + |(1 − λ) · 1 + λz − z|^{n+1}}{n + 1} ]
× ∥f^{(n+1)}∥_{[1,z],∞}
= \frac{(z − 1)^{n+1}}{(n + 1)!} · [λ^{n+1} + (1 − λ)^{n+1}] · ∥f^{(n+1)}∥_{[1,z],∞} .

Using Hölder’s inequality, we may write, for α > 1, \frac{1}{α} + \frac{1}{β} = 1, that

|I_{2} (z)| ≤ \frac{1}{n!} ∥f^{(n+1)}∥_{[1,z],β} |∫_{1}^{z} |(1 − λ) · 1 + λz − t|^{nα} dt|^{\frac{1}{α}}
= \frac{1}{n!} ∥f^{(n+1)}∥_{[1,z],β} |∫_{1}^{(1−λ)·1+λz} |(1 − λ) · 1 + λz − t|^{nα} dt
+ ∫_{(1−λ)·1+λz}^{z} |(1 − λ) · 1 + λz − t|^{nα} dt|^{\frac{1}{α}}
= \frac{1}{n!} ∥f^{(n+1)}∥_{[1,z],β} [\frac{|(1 − λ) + λz − 1|^{nα+1} + |z − (1 − λ) − λz|^{nα+1}}{nα + 1} ]^{\frac{1}{α}}
= \frac{1}{n!} ∥f^{(n+1)}∥_{[1,z],β} [\frac{λ^{αn+1} |z − 1|^{αn+1} + (1 − λ)^{αn+1} |z − 1|^{αn+1}}{αn + 1} ]^{\frac{1}{α}}
= \frac{|z − 1|^{n+ \frac{1}{α}}}{n! (αn + 1)^{\frac{1}{α}}} · [λ^{αn+1} + (1 − λ)^{αn+1}]^{\frac{1}{α}} · ∥f^{(n+1)}∥_{[1,z],β} .

Finally, we observe that

|I_{2} (z)| ≤ \frac{1}{n!} ∥f^{(n+1)}∥_{[1,z],1} sup_{t∈[1,z]} |(1 − λ) · 1 + λz − t|^{n}
= \frac{1}{n!} ∥f^{(n+1)}∥_{[1,z],1} max \{|(1 − λ) + λz − 1|^{n} + |z − (1 − λ) · 1 − λz|^{n}\}

= \frac{1}{n!} ∥f^{(n+1)}∥_{[1,z],1} (z − 1)^{n} (max \{λ, 1 − λ\})^{n}
= \frac{1}{n!} ∥f^{(n+1)}∥_{[1,z],1} |z − 1|^{n} [\frac{1}{2} + |λ − \frac{1}{2}|]^{n} .

Using the above inequalities, we may state that

(3.7)  |I_{2} (z)| ≤ κ (n, z) ,

where κ (n, z) is defined in (3.3). That is, the bounds for R_{f} (p, q) and R̃_{f} (p, q) are the same.

We may now state the following theorem concerning a bound for the remainder R̃_{f} (p, q). Theorem 3. Assume that the function f is as in Theorem 1. If p, q ∈ ⩽̸, then we have the inequality:

(3.8)  |R̃_{f} (p, q)| ≤ A,

where A is given in (3.4). Moreover, if we assume that 0 ≤ r ≤ \frac{p(x)}{q(x)} ≤ R < ∞, x ∈ Γ, then

(3.9)  A ≤ B ≤ C,

with B and C being as defined in (3.5).

The following corollary may be useful in practical applications. Corollary 2. With the above assumptions, we have

|I_{f} (p, q) − f (1) − ∑_{k=0}^{n−1} \frac{(−1)^{k}}{(k + 1)!}I_{(·−1)^{k+1}f^{(k+1)}(·)} (p, q)| (3.10)

≤ M_{1} ≤ M_{2} ≤ M_{3},

|I_{f} (p, q) − f (1) − ∑_{k=0}^{n−1} \frac{f^{(k+1)} (1)}{(k + 1)!}D_{k} (p, q)| (3.11)

≤ M_{1} ≤ M_{2} ≤ M_{3}

and

|I_{f} (p, q) − f (1) − ∑_{k=0}^{n−1} \frac{f^{(k+1)} (1)}{2^{k+1} (k + 1)!}D_{k} (p, q) (3.12)

− ∑_{k=0}^{n−1} \frac{(−1)^{k}}{2^{k+1} (k + 1)!}I_{(·−1)^{k+1}f^{(k+1)}(·)} (p, q)|

≤ \frac{1}{2^{n}} M_{1} ≤ \frac{1}{2^{n}} M_{2} ≤ \frac{1}{2^{n}} M_{3}

for r ≤ \frac{p(x)}{q(x)} ≤ R, x ∈ Γ, where M_{i} (i = \overline{1, 3}) are as defined in Corollary 1.

Remark 3. If in all the above results we choose f to be a particular function generating the divergences listed at (1.2) – (1.9), then we can obtain many interesting approximations for the above distances. We omit the details.

References

[1] S.M. ALI and S.D. SILVEY, A general class of coefficients of divergence of one distribution from another, J. Roy. Statist. Soc. Sec B, 28 (1966), 131-142.

[2] N.S. BARNETT, S.S. DRAGOMIR and A. SOFO, Better bounds for an inequality of the Ostrowski type with applications, RGMIA Research Report Collection, 3 (1) (2000), Article 1.

[3] M. BETH BASSAT, f-entropies, probability of error and feature selection, Inform. Control, 39 (1978), 227-242.

[4] A. BHATTACHARYYA, On a measure of divergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc., 35 (1943), 99-109.

[5] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy function, IEEE Trans. Inf. Th., 28 (3) (1982), 489-495.

[6] P. CERONE and S.S. DRAGOMIR, Midpoint type rules from an inequalities point of view, Handbook of Analytic-Computational Methods in Applied Mathematics, Editor: G. Anastassiou, CRC Press, N.Y., (2000), 135-200.

[7] P. CERONE and S.S. DRAGOMIR, Trapezoidal type rules from an inequalities point of view, Handbook of Analytic-Computational Methods in Applied Mathematics, Editor: G. Anastassiou, CRC Press, N.Y., (2000), 65-134.

[8] P. CERONE, S.S. DRAGOMIR, and J. ROUMELIOTIS, Some Ostrowski type inequalities for n−time differentiable mappings and applications, Demonstratio Math., 32(2) (1999), 697-712.

[9] P. CERONE, S.S. DRAGOMIR, J. ROUMELIOTIS, and J. ŠUNDE, A new generalisation of the trapezoid formula for n−time differentiable mappings and applications, Demonstratio Math., in press.

[10] C.H. CHEN, Statistical Pattern Recognition, Rocelle Park, New York, Hoyderc Book Co., 1973.

[11] C.K. CHOW and C.N. LIN, Approximating discrete probability distributions with dependence trees, IEEE Trans. Inf. Th., 14 (3) (1968), 462-467.

[12] I. CSISZÁR, Information-type measures of difference of probability distributions and indirect observations, Studia Math. Hungarica, 2 (1967), 299-318.

[13] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[14] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[15] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[16] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[17] D. DACUNHA-CASTELLE, Ecole d′ete de Probability de Saint-Flour, III-1977, Berlin, Heidelberg: Springer 1978.

[18] S.S. DRAGOMIR and S. MABIZELA, Some error estimates in the trapezoidal quadrature rule, RGMIA Research Report Collect., 2 (1999), No. 5, Article 6

(http://rgmia.vu.edu.au/v2n5.html)

[19] S.S. DRAGOMIR and S. WANG, A new inequality of Ostrowski-Grüss type and its applications to the estimation of error bounds for special means and for some numerical quadrature rules, Comp. Math. Appl., 33 (1997), No. 11, 15-20.

[20] D.V. GOKHALE and S. KULLBACK, Information in Contingency Tables, New York, Marcel Dekker, 1978.

[21] J.H. HAVRDA and F. CHARVAT, Quantification method classification process: concept of structural α-entropy, Kybernetika, 3 (1967), 30-35.

[22] E. HELLINGER, Neue Bergrüirdung du Theorie quadratisher Formerus von uneudlichvieleu Veränderlicher, J. für reine and Augeur. Math., 36 (1909), 210-271.

[23] H. JEFFREYS, An invariant form for the prior probability in estimating problems, Proc. Roy. Soc. London, 186 A (1946), 453-461.

[24] T.T. KADOTA and L.A. SHEPP, On the best finite set of linear observables for discriminating two Gaussian signals, IEEE Trans. Inf. Th., 13 (1967), 288-294.

[25] T. KAILATH, The divergence and Bhattacharyya distance measures in signal selection, IEEE Trans. Comm. Technology., Vol COM-15 (1967), 52-60.

[26] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), 1-16.

[27] D. KAZAKOS and T. COTSIDAS, A decision theory approach to the approximation of discrete probability densities, IEEE Trans. Perform. Anal. Machine Intell., 1 (1980), 61-67.

[28] J.H.B. KEMPERMAN, On the optimum note of transmitting information, Ann. Math. Statist., 40 (1969), 2158-2177.

[29] C. KRAFT, Some conditions for consistency and uniform consistency of statistical procedures, Univ. of California Pub. in Statistics, 1 (1955), 125-142.

[30] S. KULLBACK, A lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 13 (1967), 126-127.

[31] S. KULLBACK, Correction to a lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[32] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Ann. Math. Stat., 22 (1951), 79-86.

[33] L. LECAM, Asymptotic Methods in Statistical Decision Theory, New York: Springer, 1986.

[34] J. LIN, Divergence measures based on the Shannon entropy, IEEE Trans. Inf. Th., 37 (1) (1991), 145-151.

[35] J. LIN and S.K.M. WONG, A new directed divergence measure and its characterization, Int. J. General Systems, 17 (1990), 73-81.

[36] H.P. McKEAN, JR., Speed of approach to equilibrium for Koc’s caricature of a Maximilian gas, Arch. Ration. Mech. Anal., 21 (1966), 343-367.

[37] M. MEI, The theory of genetic distance and evaluation of human races, Japan J. Human Genetics, 23 (1978), 341-369.

[38] C.E.M. PEARCE, J. PEČARIĆ, N. UJEVIĆ and S. VAROS̆ANEC, Generaliasations of some inequalities of Ostrowski-Grüss type, Math. Ineq. & Appl., 3 (2000), No. 1, 25-34.

[39] E.C. PIELOU, Ecological Diversity, Wiley, New York, 1975.

[40] M.S. PINSKER, Information and Information Stability of Random variables and processes, (in Russian), Moscow: Izv. Akad. Nouk, 1960.

[41] C.R. RAO, Diversity and dissimilarity coefficients: a unified approach, Theoretic Population Biology, 21 (1982), 24-43.

[42] A. RÉNYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Stat. and Prob., University of California Press, 1 (1961), 547-561.

[43] A. SEN, On Economic Inequality, Oxford University Press, London 1973.

[44] B.D. SHARMA and D.P. MITTAL, New non-additive measures of relative information, Journ. Comb. Inf. Sys. Sci., 2 (4)(1977), 122-132.

[45] H. SHIOYA and T. DA-TE, A generalisation of Lin divergence and the derivative of a new information divergence, Elec. and Comm. in Japan, 78 (7) (1995), 37-40.

[46] I. J. TANEJA, Generalised Information Measures and their Applications

    (http://www.mtm.ufsc.br/~taneja/bhtml/bhtml.html).

[47] H. THEIL, Economics and Information Theory, North-Holland, Amsterdam, 1967.

[48] H. THEIL, Statistical Decomposition Analysis, North-Holland, Amsterdam, 1972.

[49] F. TOPSOE, Some inequalities for information divergence and related measures of discrimination, Res. Rep. Coll., RGMIA, 2 (1) (1999), 85-98.

[50] G.T. TOUSSAINT, Sharper lower bounds for discrimination in terms of variation, IEEE Trans. Inf. Th., 21 (1975), 99-100.

[51] I. VAJDA, Note on discrimination information and variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[52] I. VAJDA, Theory of Statistical Inference and Information, Dordrecht-Boston, Kluwer Academic Publishers, 1989.

[53] V.A. VOLKONSKI and J. A. ROZANOV, Some limit theorems for random function -I, (English Trans.), Theory Prob. Appl., ( USSR), 4 (1959), 178-197.

School of Communications and Informatics, Victoria University of Technology, PO Box 14428, Melbourne City MC 8001, Australia

E-mail address: neil@matilda.vu.edu.au
URL: http://sci.vu.edu.au/staff/neilb.html
E-mail address: pc@matilda.vu.edu.au
URL: http://sci.vu.edu.au/staff/peterc.html
E-mail address: sever@matilda.vu.edu.au
URL: http:// rgmia.vu.edu.au/SSDragomirWeb.html
E-mail address: john.roumeliotis@vu.edu.au
URL: http://dingo.vu.edu.au/~johnr/