APPROXIMATING TWO MAPPINGS ASSOCIATED TO CSISZÁR f−DIVERGENCE VIA TAYLOR’S EXPANSION

N.S. BARNETT, P. CERONE, S.S. DRAGOMIR, AND A. SOFO

Abstract. Using the Taylor representation with integral remainder, we point out some approximations of two mappings generalising Csiszár f−divergence.

1. Introduction

One of the important issues in many applications of Probability Theory is finding an appropriate measure of distance (or difference or discrimination ) between two probability distributions. A number of divergence measures for this purpose have been proposed and extensively studied by Jeffreys [1], Kullback and Leibler [2], Rényi [3], Havrda and Charvat [4], Kapur [5], Sharma and Mittal [6], Burbea and Rao [8], Rao [9], Lin [10], Csiszár [11], Ali and Silvey [13], Vajda [14], Shioya and Da-te [14] and others (see for example [5] and the references therein).

These measures have been applied in a variety of fields such as: anthropology [9], genetics [15], finance, economics, and political science [16], [17], [18], biology [19], the analysis of contingency tables [20], approximation of probability distributions [21], [22], signal processing [23], [24] and pattern recognition [25], [26]. A number of these measures of distance are specific cases of Csiszár f-divergence and so further exploration of this concept will have a flow on effect to other measures of distance and to areas in which they are applied.

Assume that a set Γ and the σ−finite measure μ are given. Consider the set of all probability densities on μ to be Ω := {p|p : Γ → ℝ, p (x) ≥ 0, ∫_{Γ} p (x) dμ (x) = 1}. The Kullback-Leibler divergence [2] is well known among the information divergences. It is defined as:

(1.1)  D_{KL} (p, q) := ∫_{Γ} p (x) log [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω,

where log is to base 2.

In Information Theory and Statistics, various divergences are applied in addition to the Kullback-Leibler divergence. These are the: variation distance D_{v}, Hellinger distance D_{H} [2], χ^{2}−divergence D_{χ}^{2}, α−divergence D_{α}, Bhattacharyya distance D_{B} [3], Harmonic distance D_{Ha}, Jeffreys distance D_{J} [1], triangular discrimination D_{Δ} [36], etc... They are defined as follows:

(1.2)  D_{v} (p, q) := ∫_{Γ} |p (x) − q (x)| dμ (x) , p, q ∈ Ω;

Date: April 11, 2000.

1991 Mathematics Subject Classification. Primary 94Xxx; Secondary 26D15.

Key words and phrases. Csiszár f−divergegence, Taylor’s formula.

(1.3)  D_{H} (p, q) := ∫_{Γ} |\sqrt{p (x)} − \sqrt{q (x)}| dμ (x) , p, q ∈ Ω;

(1.4)  D_{χ}^{2} (p, q) := ∫_{Γ} p (x) [(\frac{q (x)}{p (x)})^{2} − 1] dμ (x) , p, q ∈ Ω;

(1.5)  D_{α} (p, q) := \frac{4}{1 − α^{2}} [1 − ∫_{Γ} [p (x)]^{\frac{1−α}{2}} [q (x)]^{\frac{1+α}{2}} dμ (x)] , p, q ∈ Ω;

(1.6)  D_{B} (p, q) := ∫_{Γ} \sqrt{p (x) q (x)}dμ (x) , p, q ∈ Ω;

(1.7)  D_{Ha} (p, q) := ∫_{Γ} \frac{2p (x) q (x)}{p (x) + q (x)}dμ (x) , p, q ∈ Ω;

(1.8)  D_{J} (p, q) := ∫_{Γ} [p (x) − q (x)] ln [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω;

(1.9)  D_{Δ} (p, q) := ∫_{Γ} \frac{[p (x) − q (x)]^{2}}{p (x) + q (x)} dμ (x) , p, q ∈ Ω.

For other divergence measures, see the paper [5] by Kapur or the book on line [7] by Taneja. For a comprehensive collection of preprints available on line, see the

RGMIA web site http://rgmia.vu.edu.au/papersinfth.html

Csiszár f−divergence is defined as follows [11]

(1.10)  D_{f} (p, q) := ∫_{Γ} p (x) f [\frac{q (x)}{p (x)}] dμ (x) , p, q ∈ Ω,

where f is convex on (0, ∞). It is assumed that f (u) is zero and strictly convex at u = 1. By appropriately defining this convex function, various divergences are derived. All the above distances (1.1) − (1.9), are particular instances of Csiszár f−divergence. There are also many others which are not in this class (see for example [5] or [7]). For the basic properties of Csiszár f−divergence see [8]-[11].

In [12], Lin and Wong (see also [10]) introduced the following divergence

(1.11)  D_{LW} (p, q) := ∫_{Γ} p (x) log [ \frac{p (x)}{\frac{1}{2}p (x) + \frac{1}{2}q (x)}] dμ (x) , p, q ∈ Ω.

This can be represented as follows, using the Kullback-Leibler divergence:

D_{LW} (p, q) = D_{KL} (p, \frac{1}{2}p + \frac{1}{2}q) .

Lin and Wong have established the following inequalities

(1.12)  D_{LW} (p, q) ≤ \frac{1}{2}D_{KL} (p, q) ;
(1.13)  D_{LW} (p, q) + D_{LW} (q, p) ≤ D_{v} (p, q) ≤ 2;
(1.14)  D_{LW} (p, q) ≤ 1.

In [14], Shioya and Da-te improved (1.12) − (1.14) by showing that

D_{LW} (p, q) ≤ \frac{1}{2}D_{v} (p, q) ≤ 1.

APPROXIMATING TWO MAPPINGS ASSOCIATED TO CSISZÁR f−DIVERGENCE 3

For classical and new results in comparing different kinds of divergence measures, see the papers [1]-[14] where further references are given.

In [50], the authors introduced the following divergence measure

(1.15)  H_{f} (p, q; t) := ∫_{χ} p (x) f [\frac{tq (x) + (1 − t) p (x)}{p (x)}] dμ (x) ,

where p, q ∈ Q and t ∈ [0, 1].

It is obvious that this measure can be represented in terms of Csiszár f−divergence, namely, we have the representation

(1.16)  H_{f} (p, q; t) = D_{f} (p, tq + (1 − t) p)

for all p, q ∈ Q and t ∈ [0, 1].

The following properties of H_{f} (·, ·; ·) hold (see [50]). Theorem 1. Assume that the mapping f : [0, ∞) → ℝ is convex and p, q ∈ Q. Then

(i) H_{f} (p, q; ·) is convex on [0, 1];

(ii) We have the inequality

(1.17)  H_{f} (p, q; t) ≤ D_{f} (p, q) for all t ∈ [0, 1]

and the bounds

(1.18)  inf_{t∈[0,1]} H_{f} (p, q; t) = H_{f} (p, q; 0) = 0

and

(1.19)  sup_{t∈[0,1]} H_{f} (p, q; t) = H_{f} (p, q; 1) = D_{f} (p, q) ;

(iii) The mapping H_{f} (p, q; ·) is monotonic nondecreasing on [0, 1].

In the same paper [50], the authors introduced the following divergence

(1.20) F_{f} (p, q; t) = ∫_{χ} ∫_{χ} p (x) p (y) f [t · \frac{q (x)}{p (x)} + (1 − t) · \frac{q (y)}{p (y)}] dμ (x) dμ (y) ,

where p, q ∈ Ω and t ∈ [0, 1].

The properties of this mapping are embodied in the following theorem. Theorem 2. Under the assumptions of Theorem 1, we have

(i) F_{f} (p, q; ·) is symmetrical about \frac{1}{2}, i.e.,

(1.21)  F_{f} (p, q; t) = F_{f} (p, q; 1 − t) for all t ∈ [0, 1] ;

(ii) F_{f} (p, q; ·) is convex on [0, 1]; (iii) We have the bounds

sup_{t∈[0,1]} F_{f} (p, q; t) = F_{f} (p, q; 0) = F_{f} (p, q; 1) = D_{f} (p, q) ; (1.22)

inf_{t∈[0,1]} F_{f} (p, q; t) = F_{f} (p, q; \frac{1}{2}) (1.23)

= ∫_{χ} ∫_{χ} p (x) p (y) f [\frac{q (x) p (y) + p (x) q (y)}{2p (x) p (y)}] dμ (x) dμ (y) ≥ 0;

(iv) F_{f} (p, q; ·) is nondecreasing on [0, \frac{1}{2}] and nonincreasing on [\frac{1}{2}, 1];

(v) We have the inequalities

(1.24)  F_{f} (p, q; t) ≥ max {H_{f} (p, q; t) , H_{f} (p, q; 1 − t)}

for all t ∈ [0, 1]. In this paper we point out some estimates for the divergence measures F_{f} (·, ·; ·)

and H_{f} (·, ·; ·).

2. Some Estimates for n−Time Differentiable Mappings

We use the following lemma (see also [48]). Lemma 1. Let f : I ∈ ℝ → ℝ (I interval of ℝ) be such that f^{(n)} is absolutely continuous on I. Then for all x, a ∈I̊ (I̊ is the interior of I) we have the inequality

|f (x) − f (a) − ∑_{k=1}^{n} \frac{(x − a)^{k}}{k!} f^{(k)} (a)| (2.1)

≤ \{ \begin{matrix}\frac{1}{(n+1)!} ∥f^{(n+1)}∥_{∞} |x − a|^{n+1} & if & f^{(n+1)} ∈ L_{∞} (I) ; \\ _{\frac{1}{n!(nβ+1)^{\frac{1}{β}}}} ∥f^{(n+1)}∥_{α} |x − a|^{n+ \frac{1}{β}} & if & f^{(n+1)} ∈ L_{α} (I) , \\ & & α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} |x − a|^{n} , \end{matrix}

where ∥·∥_{α} (α ∈ [1, ∞]) are the usual Lebesgue norms on I, i.e.,

∥g∥_{α} : = (∫_{I} |g (x)|^{α} dx)^{\frac{1}{α}} , α ≥ 1
∥g∥_{∞} : = ess sup_{x∈I} |g (x)| .

Proof. We start with the Taylor representation with the integral remainder

(2.2)  f (x) = f (a) + ∑_{k=1}^{n} \frac{(x − a)^{k}}{k!} f^{(k)} (a) + \frac{1}{n!} ∫_{a}^{x} (x − t)^{n} f^{(n+1)} (t) dt

for all a, x ∈I̊.

Using the properties of modulus, we have

|f (x) − f (a) − ∑_{k=1}^{n} \frac{(x − a)^{k}}{k!} f^{(k)} (a)| (2.3)

≤ \frac{1}{n!} |∫_{a}^{x} |x − t|^{n} |f^{(n+1)} (t)| dt| =: M (f^{(n+1)}; a, x) .

Obviously, we have

(2.4)  M (f^{(n+1)}; a, x) ≤ ess sup_{t∈I} |f^{(n+1)} (t)| \frac{1}{n!} |∫_{a}^{x} |x − t|^{n} dt|
= \frac{1}{(n + 1)!} ∥f^{(n+1)}∥_{∞} |x − a|^{n+1}

for all a, x ∈I̊.

In addition, by the use of the Hölder integral inequality, we have

(2.5) M (f^{(n+1)}; a, x) ≤ \frac{1}{n!} |∫_{a}^{x} |x − t|^{nβ} dt|^{\frac{1}{β}} |∫_{α}^{β} |f^{(n+1)} (t)|^{α} dt|^{\frac{1}{α}}

= \frac{1}{n!} ∥f^{(n+1)}∥_{α} [\frac{(x − a)^{nβ+1}}{nβ + 1} ]^{\frac{1}{β}}
= \frac{1}{n! (nβ + 1)^{\frac{1}{β}}} ∥f^{(n+1)}∥_{α} |x − a|^{n+ \frac{1}{β}} ,
α > 1, \frac{1}{α} + \frac{1}{β} = 1,

and finally

(2.6)  M (f^{(n+1)}; a, x) ≤ \frac{1}{n!} |x − a|^{n} |∫_{a}^{x} |f^{(n+1)} (t)| dt|
≤ \frac{1}{n!} |x − a|^{n} ∥f^{(n+1)}∥_{1} .

Now, by (2.3) - (2.6), we deduce the desired inequality (2.1).

The following corollary will be useful in what follows.

Corollary 1. Assume that f is as above and a, b ∈I̊. Then for all λ ∈ [0, 1] , we have the inequality:

|f (λb + (1 − λ) a) − f (a) − ∑_{k=1}^{n} \frac{λ^{k} (b − a)^{k}}{k!} f^{(k)} (a)| (2.7)

≤ \{ \begin{matrix}\frac{λ^{n+1}|b−a|^{n+1}}{(n+1)!} ∥f^{(n+1)}∥_{∞} & if & f^{(n+1)} ∈ L_{∞} (I) ; \\ \frac{λ^{n+ \frac{1}{β}} |b−a|^{n+ \frac{1}{β}}}{n!(nβ+1)^{\frac{1}{β}}} ∥f^{(n+1)}∥_{∞} & if & f^{(n+1)} ∈ L_{α} (I) , \\ & & α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{λ^{n}|b−a|^{n}}{n!} ∥f^{(n+1)}∥_{1} . \end{matrix}

We can now point out the following estimation result for the mapping H_{f} (p, q; ·).

Theorem 3. Assume that the mapping f : [0, ∞) → ℝ is such that f^{(n)} is absolutely continuous on [r, R] , where 0 ≤ r ≤ 1 ≤ R < ∞. If p, q ∈ Ω and

(2.8)  r ≤ \frac{q (x)}{p (x)} ≤ R a.e. on χ,

6 N.S. BARNETT, P. CERONE, S.S. DRAGOMIR, AND A. SOFO

then we have the inequality

|H_{f} (p, q; t) − f (1) − ∑_{k=1}^{n} \frac{t^{k}f^{(k)} (1)}{k!}D_{χ}_{k} (p, q)| (2.9)

≤ \{ \begin{matrix}\frac{t^{n+1}∥f^{(n+1)}∥_{∞}}{(n+1)!} D_{|χ|}^{n+1} (p, q) & if & f^{(n+1)} ∈ L_{∞} [r, R] ; \\ \frac{t^{n+ \frac{1}{β}} ∥f^{(n+1)}∥_{α}}{n!(nβ+1)^{\frac{1}{β}}} D_{|χ|}_{n+ \frac{1}{β}} (p, q) & if & f^{(n+1)} ∈ L_{α} [r, R] , \\ & & α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{t^{n}∥f^{(n+1)}∥_{1}}{n!} D_{|χ|}^{n} (p, q) , \end{matrix}

≤ \{ \begin{matrix}\frac{t^{n+1}(R−r)^{n+1}}{(n+1)!} & ∥f^{(n+1)}∥ & ∞ \\ \frac{t^{n+ \frac{1}{β}} (R−r)^{n+ \frac{1}{β}}}{n!(nβ+1)^{\frac{1}{β}}} & ∥f^{(n+1)}∥ & α \\ \frac{t^{n}(R−r)^{n}}{n!} ∥f^{(n+1)}∥_{1} , \end{matrix}

where

D_{χ}_{k} (p, q) := ∫_{χ} \frac{(q (x) − p (x))^{k}}{p^{k−1} (x)} dμ (x) , k = 1, ...

and

D_{|χ|}^{r} (p, q) := ∫_{χ} \frac{|q (x) − p (x)|^{r}}{p^{r−1} (x)} dμ (x) , r ≥ 0

and the Lebesgue α−norms are taken on [r, R].

Proof. Apply inequality (2.1) for λ = t ∈ [0, 1], b = \frac{q(x)}{p(x)}, x ∈ χ and a = 1, to get

|f (t · \frac{q (x)}{p (x)} + (1 − t)) − f (1) − ∑_{k=1}^{n} \frac{t^{k} (\frac{q(x)}{p(x)} − 1)^{k}}{k!} f^{(k)} (1)| (2.10)

≤ \{ \begin{matrix}^{\frac{t^{n+1}|\frac{q(x)}{p(x)} −1|^{n+1}}{(n+1)!}} ∥f^{(n+1)}∥_{∞}; \\ ^{\frac{t^{n+ \frac{1}{β} |\frac{q(x)}{p(x)} −1|^{n+ \frac{1}{β}}}}{n!(nβ+1)^{\frac{1}{β}}}} ∥f^{(n+1)}∥_{α}; \\ ^{\frac{t^{n}|\frac{q(x)}{p(x)} −1|^{n}}{n!}} ∥f^{(n+1)}∥_{1} ; \end{matrix} ≤ \{ \begin{matrix}\frac{t^{n+1}(R−r)^{n+1}}{(n+1)!} ∥f^{(n+1)}∥_{∞}; \\ \frac{t^{n+ \frac{1}{β}} (R−r)^{n+ \frac{1}{β}}}{n!(nβ+1)^{\frac{1}{β}}} ∥f^{(n+1)}∥_{α}; \\ \frac{t^{n}(R−r)^{n}}{n!} ∥f^{(n+1)}∥_{1} , \end{matrix}

for a.e. x ∈ χ.

If we multiply (2.10) by p (x) ≥ 0, integrate on χ and use the properties of the integral, then we get

|∫_{χ} p (x) f (\frac{tq (x) + (1 − t) p (x)}{p (x)}) dμ (x)

−f (1) − ∑_{k=1}^{n} \frac{t^{k}f^{(k)} (1)}{k!} ∫_{χ} \frac{(q (x) − p (x))^{k}}{p^{k−1} (x)} dμ (x)|

≤ \{ \begin{matrix}\frac{t^{n+1}∥f^{(n+1)}∥_{∞}}{(n+1)!} ∫_{χ} \frac{|q(x)−p(x)|^{n+1}}{p^{n}(x)} dμ (x) ; \\ \frac{t^{n+ \frac{1}{β}} ∥f^{(n+1)}∥_{α}}{n!(nβ+1)^{\frac{1}{β}}} ∫_{χ} \frac{|q(x)−p(x)|^{n+ \frac{1}{β}}}{p^{n+ \frac{1}{β} −1}(x)} dμ (x) ; \\ \frac{t^{n}∥f^{(n+1)}∥_{1}}{n!} ∫_{χ} \frac{|q(x)−p(x)|^{n}}{p^{n−1}(x)} dμ (x) , \end{matrix}

≤ \{ \begin{matrix}\frac{t^{n+1}(R−r)^{n+1}}{(n+1)!} & ∥f^{(n+1)}∥ & ∞ \\ \frac{t^{n+ \frac{1}{β}} (R−r)^{n+ \frac{1}{β}}}{n!(nβ+1)^{\frac{1}{β}}} & ∥f^{(n+1)}∥ & α \\ \frac{t^{n}(R−r)^{n}}{n!} ∥f^{(n+1)}∥_{1} , \end{matrix}

and the theorem is proved.

Remark 1. If n = 0, then, basically, for an absolutely continuous mapping f : [r, R] ⊂ [0, ∞) → ℝ, we have:

|D_{f} (p, tq + (1 − t) p) − f (1)| (2.11)

≤ \{ \begin{matrix}t ∥f′∥_{∞} D_{v} (p, q) \\ t^{\frac{1}{β}} ∥f′∥_{α} D_{|α|}_{\frac{1}{β}} (p, q) \\ ∥f′∥ & 1 \end{matrix} ≤ \{ \begin{matrix}t ∥f′∥_{∞} (R − r) \\ t^{\frac{1}{β}} (R − r)^{\frac{1}{β}} · ∥f′∥_{α} \\ ∥f′∥ & 1 \end{matrix}

for all t ∈ [0, 1], where D_{v} (p, q) = ∫_{χ} |p (x) − q (x)| dμ (x). If n = 1, and taking into account that D_{χ} (p, q) = 0, then by (2.5) we get for the mappings whose derivatives f′ are absolutely continuous that

|D_{f} (p, tq + (1 − t) p) − f (1)| (2.12)

≤ \{ \begin{matrix}\frac{t^{2}∥f′∥_{∞}}{2} D_{χ}^{2} (p, q) & if & f′ ∈ L_{∞} [r, R] \\ \frac{t^{\frac{β+1}{β}} ∥f′∥_{α}}{(β+1)^{\frac{1}{β}}} D_{|χ|}_{\frac{β+1}{β}} (p, q) & if & f′ ∈ L_{α} [r, R] \\ t ∥f′∥_{1} D_{v} (p, q) \end{matrix}

for all t ∈ [0, 1].

Of course, if we assume that f is convex and normalised, then the left hand side of both (2.11) and (2.12) will become

0 ≤ D_{f} (p, tq + (1 − t) p)

and the inequalities (2.11) and (2.12) will provide some upper bounds for the mapping H_{f} (p, q; t), t ∈ [0, 1].

Remark 2. If we assume that f′′ is absolutely continuous, then from (2.9) we obtain

|H_{f} (p, q; t) − f (1) − \frac{t^{2}}{2} f′′ (1) D_{χ}^{2} (p, q)| (2.13)

≤ \{ \begin{matrix}\frac{t^{3}∥f′′′∥_{∞}}{6} D_{|χ|}^{3} (p, q) & if & f′′′ ∈ L_{∞} [r, R] ; \\ \frac{t^{\frac{2β+1}{β}} ∥f′′′∥_{α}}{2(2β+1)^{\frac{1}{β}}} D_{|χ|}_{2+ \frac{1}{β}} (p, q) & if & f′′′ ∈ L_{α} [r, R] , \\ & & α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{t^{2}∥f′′′∥_{1}}{2} D_{|χ|}^{2} (p, q) , \end{matrix}

which provides an approximation of H_{f} (p, q; t) by a quadratic in t whose coefficient is dependent on the χ^{2} -distance of p and q.

We also note that Theorem 3 contains, as a particular case (for t = 1), an approximation of the Csiszár f−divergence. Namely,

Corollary 2. With the assumptions of Theorem 3, we have

|D_{f} (p, q) − f (1) − ∑_{k=1}^{n} \frac{f^{(k)} (1)}{k!}D_{χ}_{k} (p, q)| (2.14)

≤ \{ \begin{matrix}\frac{∥f^{(n+1)}∥_{∞}}{(n+1)!} D_{|χ|}^{n+1} (p, q) \\ \frac{∥f^{(n+1)}∥_{α}}{n!(nβ+1)^{\frac{1}{β}}} D_{|χ|}_{n+ \frac{1}{β}} (p, q) \\ \frac{∥f^{(n+1)}∥_{1}}{n!} D_{|χ|}^{n} (p, q) \end{matrix} ≤ \{ \begin{matrix}\frac{∥f^{(n+1)}∥_{∞}}{(n+1)!} & (R − r) & n+1 \\ \frac{∥f^{(n+1)}∥_{α}}{n!(nβ+1)^{\frac{1}{β}}} & (R − r) & n+ \frac{1}{β} \\ \frac{∥f^{(n+1)}∥_{1}}{n!} (R − r)^{n} . \end{matrix}

We also know that for t = \frac{1}{2}, we obtain the generalised Lin-Wong f−divergence

LW_{f} (p, q) := D_{f} (p, \frac{1}{2}p + \frac{1}{2}q)

and so, from (2.9), we may state the following estimation for the Lin-Wong f−divergence.

APPROXIMATING TWO MAPPINGS ASSOCIATED TO CSISZÁR f−DIVERGENCE 9

Corollary 3. With the assumptions of Theorem 3, we have

|LW_{f} (p, q) − f (1) − ∑_{k=1}^{n} \frac{t^{k}f^{(k)} (1)}{2^{k}k!}D_{χ}_{k} (p, q)| (2.15)

≤ \{ \begin{matrix}\frac{∥f^{(n+1)}∥_{∞}}{2^{n+1}(n+1)!}D_{|χ|}^{n+1} (p, q) & if & f^{(n+1)} ∈ L_{∞} [r, R] ; \\ \frac{∥f^{(n+1)}∥_{α}}{2^{n+ \frac{1}{β}} n!(nβ+1)^{\frac{1}{β}}} D_{|χ|}_{n+ \frac{1}{β}} (p, q) & if & f^{(n+1)} ∈ L_{α} [r, R] , \\ & & α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{∥f^{(n+1)}∥_{1}}{2^{n}n!} D_{|χ|}^{n} (p, q) . \end{matrix}

Remark 3. Similar particular cases for n = 0, n = 1 and n = 2 may be stated, but we omit the details.

The following theorem also holds. Theorem 4. Assume that the mapping f : [0, ∞) → ℝ is such that f^{(n)} is absolutely continuous on [r, R], where 0 ≤ r ≤ 1 ≤ R < ∞. If p, q ∈ Ω and

(2.16)  r ≤ \frac{q (x)}{p (x)} ≤ R a.e. on χ,

then we have the inequality

|F_{f} (p, q; t) − D_{f} (p, q) − ∑_{k=1}^{n} \frac{t^{k}}{k!} D_{f^{(k)}}^{(∗)} (p, q)| (2.17)

≤ \{ \begin{matrix}\frac{t^{n+1}}{(n+1)!}D_{n+1}^{(∗)} (p, q) ∥f^{(n+1)}∥_{∞} & if & f^{(n+1)} ∈ L_{∞} [r, R] ; \\ \frac{t^{n+ \frac{1}{β}}}{n!(nβ+1)^{\frac{1}{β}}} D_{n+ \frac{1}{β}}^{(∗)} (p, q) ∥f^{(n+1)}∥_{α} & if & f^{(n+1)} ∈ L_{α} [r, R] , \\ & & α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{t^{n}}{n!} D_{n}^{(∗)} (p, q) ∥f^{(n+1)}∥_{1} \end{matrix}

≤ \{ \begin{matrix}\frac{t^{n+1}}{(n+1)!} (R − r)^{n+1} ∥f^{(n+1)}∥_{∞} & if & f^{(n+1)} ∈ L_{∞} [r, R] ; \\ \frac{t^{n+ \frac{1}{β}}}{n!(nβ+1)^{\frac{1}{β}}} (R − r)^{n+ \frac{1}{β}} ∥f^{(n+1)}∥_{α} & if & f^{(n+1)} ∈ L_{α} [r, R] , \\ & & α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{t^{n}}{n!} (R − r)^{n} ∥f^{(n+1)}∥_{1} , \end{matrix}

where

D_{f^{(k)}}^{(∗)} (p, q) = ∫_{χ} ∫_{χ} \begin{matrix}(det [ \begin{matrix}p (y) & q (y) \\ p (x) & q (x) \end{matrix} ]) & k \\ [p (x)] & k−1 & [p (y)] & k−1 \end{matrix} f^{(k)} (\frac{q (y)}{p (y)}) dμ (x) dμ (y) , k = 1, ....

10 N.S. BARNETT, P. CERONE, S.S. DRAGOMIR, AND A. SOFO

D_{s}^{(∗)} (p, q) = ∫_{χ} ∫_{χ} \begin{matrix}|det [ \begin{matrix}p (y) & q (y) \\ p (x) & q (x) \end{matrix} ]| & s \\ [p (x)] & s−1 & [p (y)] & s−1 \end{matrix} dμ (x) dμ (y) , s > 0

and the α−norms are taken on [r, R]. Proof. We choose in Corollary 1, b = \frac{q(x)}{p(x)}, a = \frac{q(y)}{p(y)}, x, y ∈ χ to obtain

|f (t · \frac{q (x)}{p (x)} + (1 − t) · \frac{q (y)}{p (y)}) − f (\frac{q (y)}{p (y)})

− ∑_{k=1}^{n} \frac{t^{k} (\frac{q(x)}{p(x)} − \frac{q(y)}{p(y)})^{k}}{k!} f^{(k)} (\frac{q (y)}{p (y)})|

≤ \{ \begin{matrix}\frac{t^{n+1}}{(n+1)!} & |\frac{q(x)}{p(x)} − \frac{q(y)}{p(y)} | & n+1 & ∥f^{(n+1)}∥ & ∞ \\ \frac{t^{n+ \frac{1}{β}}}{n!(nβ+1)^{\frac{1}{β}}} & |\frac{q(x)}{p(x)} − \frac{q(y)}{p(y)} | & n+ \frac{1}{β} & ∥f^{(n+1)}∥ & α \\ \frac{t^{n}}{n!} & |\frac{q(x)}{p(x)} − \frac{q(y)}{p(y)} | & n & ∥f^{(n+1)}∥ & 1 \end{matrix}

for all x, y ∈ χ and t ∈ [0, 1], which is clearly equivalent to

|f (\frac{tp (y) q (x) + (1 − t) p (x) q (y)}{p (x) p (y)}) − f (\frac{q (y)}{p (y)}) (2.18)

− ∑_{k=1}^{n} \frac{t^{k}}{k!} · \begin{matrix}(det [ \begin{matrix}p (y) & q (y) \\ p (x) & q (x) \end{matrix} ]) & k \\ [p (x)] & k & [p (y)] & k \end{matrix} f^{(k)} (\frac{q (y)}{p (y)})|

≤ \{ \begin{matrix}\frac{t^{n+1}}{(n+1)!} · |det[ \begin{matrix}p (y) & q (y) \\ p (x) & q (x) \end{matrix} ]|^{n+1}_{[p(x)]^{n+1}[p(y)]^{n+1}} ∥f^{(n+1)}∥_{∞} \\ \frac{t^{n+ \frac{1}{β}}}{n!(nβ+1)^{\frac{1}{β}}} · |det[ \begin{matrix}p (y) & q (y) \\ p (x) & q (x) \end{matrix} ]|^{n+ \frac{1}{β}}_{[p(x)]^{n+ \frac{1}{β}} [p(y)]^{n+ \frac{1}{β}}} ∥f^{(n+1)}∥_{α} \\ \frac{t^{n}}{n!} · |det[ \begin{matrix}p (y) & q (y) \\ p (x) & q (x) \end{matrix} ]|^{n}_{[p(x)]^{n}[p(y)]^{n}} ∥f^{(n+1)}∥_{1} \end{matrix}

for all x, y ∈ χ and t ∈ [0, 1].

If we multiply (2.18) by p (x) p (y) ≥ 0 for x, y ∈ χ, integrate over x and y on χ and use the properties of the integral, we obtain

|∫_{χ} ∫_{χ} p (x) p (y) f (\frac{tp (y) q (x) + (1 − t) p (x) q (y)}{p (x) p (y)}) dμ (x) dμ (y)

− ∫_{χ} ∫_{χ} p (x) p (y) f (\frac{q (y)}{p (y)}) dμ (x) dμ (y)

− ∑_{k=1}^{n} \frac{t^{k}}{k!} ∫_{α} ∫ _{χ} \begin{matrix}(det [ \begin{matrix}p (y) & q (y) \\ p (x) & q (x) \end{matrix} ]) & k \\ [p (x)] & k−1 & [p (y)] & k−1 \end{matrix} f^{(k)} (\frac{q (y)}{p (y)}) dμ (x) dμ (y)|

≤ \{ \begin{matrix}\frac{t^{n+1}}{(n+1)!} ∥f^{(n+1)}∥_{∞} · ∫ _{χ} ∫ _{χ} |det[ \begin{matrix}p (y) & q (y) \\ p (x) & q (x) \end{matrix} ]|^{n+1}_{[p(x)]^{n}[p(y)]^{n}} dμ (x) dμ (y) \\ \frac{t^{n+ \frac{1}{β}}}{n!(nβ+1)^{\frac{1}{β}}} ∥f^{(n+1)}∥_{α} · ∫ _{χ} ∫ _{χ} |det[ \begin{matrix}p (y) & q (y) \\ p (x) & q (x) \end{matrix} ]|^{n+ \frac{1}{β}}_{[p(x)]^{n+ \frac{1}{β} −1}[p(y)]^{n+ \frac{1}{β} −1}} dμ (x) dμ (y) \\ \frac{t^{n}}{n!} ∥f^{(n+1)}∥_{1} · ∫ _{χ} ∫ _{χ} |det[ \begin{matrix}p (y) & q (y) \\ p (x) & q (x) \end{matrix} ]|^{n}_{[p(x)]^{n−1}[p(y)]^{n−1}} dμ (x) dμ (y) , \end{matrix}

which is clearly equivalent to the first inequality in (2.17). The second inequality is obvious by the fact that

|\frac{q (x)}{p (x)} − \frac{q (y)}{p (y)}| ≤ R − r for all x, y ∈ χ.

The theorem is thus completely proved.

References

[1] H. JEFFREYS, An invariant form for the prior probability in estimating problems, Proc. Roy. Soc. London, 186 A (1946), 453-461.

[2] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Ann. Math. Stat., 22 (1951), 79-86.

[3] A. RÉNYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Stat. and Prob., University of California Press, 1 (1961), 547-561.

[4] J.H. HAVRDA and F. CHARVAT, Quantification method classification process: concept of structural α-entropy, Kybernetika, 3 (1967), 30-35.

[5] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), 1-16.

[6] B.D. SHARMA and D.P. MITTAL, New non-additive measures of relative information, Journ. Comb. Inf. Sys. Sci., 2 (4)(1977), 122-132.

[7] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy function, IEEE Trans. Inf. Th., 28 (3) (1982), 489-495.

[8] C.R. RAO, Diversity and dissimilarity coefficients: a unified approach, Theoretic Population Biology, 21 (1982), 24-43.

[9] J. LIN, Divergence measures based on the Shannon entropy, IEEE Trans. Inf. Th., 37 (1) (1991), 145-151.

[10] I. CSISZÁR, Information-type measures of difference of probability distributions and indirect observations, Studia Math. Hungarica, 2 (1967), 299-318.

[11] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[12] S.M. ALI and S.D. SILVEY, A general class of coefficients of divergence of one distribution from another, J. Roy. Statist. Soc. Sec B, 28 (1966), 131-142.

[13] I. VAJDA, Theory of Statistical Inference and Information, Dordrecht-Boston, Kluwer Academic Publishers, 1989.

[14] M. MEI, The theory of genetic distance and evaluation of human races, Japan J. Human Genetics, 23 (1978), 341-369.

[15] A. SEN, On Economic Inequality, Oxford University Press, London 1973.

[16] H. THEIL, Economics and Information Theory, North-Holland, Amsterdam, 1967.

[17] H. THEIL, Statistical Decomposition Analysis, North-Holland, Amsterdam, 1972.

[18] E.C. PIELOU, Ecological Diversity, Wiley, New York, 1975.

[19] D.V. GOKHALE and S. KULLBACK, Information in Contingency Tables, New York, Merul Dekker, 1978.

[20] C.K. CHOW and C.N. LIN, Approximating discrete probability distributions with dependence trees, IEEE Trans. Inf. Th., 14 (3) (1968), 462-467.

[21] D. KAZAKOS and T. COTSIDAS, A decision theory approach to the approximation of discrete probability densities, IEEE Trans. Perform. Anal. Machine Intell., 1 (1980), 61-67.

[22] T.T. KADOTA and L.A. SHEPP, On the best finite set of linear observables for discriminating two Gaussian signals, IEEE Trans. Inf. Th., 13 (1967), 288-294.

[23] T. KAILATH, The divergence and Bhattacharyya distance measures in signal selection, IEEE Trans. Comm. Technology., Vol COM-15 (1967), 52-60.

[24] M. BETH BASSAT, f-entropies, probability of error and feature selection, Inform. Control, 39 (1978), 227-242.

[25] C.H. CHEN, Statistical Pattern Recognition, Rocelle Park, New York, Hoyderc Book Co., 1973.

[26] V.A. VOLKONSKI and J. A. ROZANOV, Some limit theorems for random function -I, (English Trans.), Theory Prob. Appl., ( USSR), 4 (1959), 178-197.

[27] M.S. PINSKER, Information and Information Stability of Random variables and processes, (in Russian), Moscow: Izv. Akad. Nouk, 1960.

[28] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[29] H.P. McKEAN, JR., Speed of approach to equilibrium for Koc’s caricature of a Maximilian gas, Arch. Ration. Mech. Anal., 21 (1966), 343-367.

[30] J.H.B. KEMPERMAN, On the optimum note of transmitting information, Ann. Math. Statist., 40 (1969), 2158-2177.

[31] S. KULLBACK, A lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 13 (1967), 126-127.

[32] S. KULLBACK, Correction to a lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[33] I. VAJDA, Note on discrimination information and variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[34] G.T. TOUSSAINT, Sharper lower bounds for discrimination in terms of variation, IEEE Trans. Inf. Th., 21 (1975), 99-100.

[35] F. TOPSOE, Some inequalities for information divergence and related measures of discrimination, Res. Rep. Coll., RGMIA, 2 (1) (1999), 85-98.

[36] L. LECAM, Asymptotic Methods in Statistical Decision Theory, New York: Springer, 1986.

[37] D. DACUNHA-CASTELLE, Ecole d’ete de Probability de Saint-Flour, III-1977, Berlin, Heidelberg: Springer 1978.

[38] C. KRAFT, Some conditions for consistency and uniform consistency of statistical procedures, Univ. of California Pub. in Statistics, 1 (1955), 125-142.

[39] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[40] E. HELLINGER, Neue Bergrüirdung du Theorie quadratisher Formerus von uneudlichvieleu Veränderlicher, J. für reine and Augeur. Math., 36 (1909), 210-271.

[41] A. BHATTACHARYYA, On a measure of divergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc., 35 (1943), 99-109.

[42] I. J. TANEJA, Generalised Information Measures and their Applications

    (http://www.mtm.ufsc.br/~taneja/bhtml/bhtml.html).

[43] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[44] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[45] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[46] J. LIN and S.K.M. WONG, A new directed divergence measure and its characterization, Int. J. General Systems, 17 (1990), 73-81.

[47] H. SHIOYA and T. DA-TE, A generalisation of Lin divergence and the derivative of a new information divergence, Elec. and Comm. in Japan, 78 (7) (1995), 37-40.

[48] S.S. DRAGOMIR, New estimation of the remainder in Taylor’s formula using Grüss’ type inequalities and applications, Math. Ineq. and Appl., 2 (2) (1999), 183-193.

[49] S.S. DRAGOMIR, An improvement of the remainder estimate in the generalised Taylor’s formula, RGMIA Res. Rep. Coll., 3 (2000), No. 1, Article 1.

[50] N.S. BARNETT, P. CERONE and S.S. DRAGOMIR, Some new inequalities for Hermite-Hadamard difference in information theory, submitted.

[51] N.S. BARNETT, P. CERONE, S.S. DRAGOMIR and A. SOFO, Approximating Csiszár f−divergence by the use of Taylor’s formula with integral remainder, submitted.

[52] M. MATIĆ, J. PEČARIĆ and N. UJEVIĆ, On new estimation of the remainder in generalised Taylor’s formula, Math. Ineq. and Appl., 2 (3) (1999), 343-361.

School of Communications and Informatics, Victoria University of Technology, PO Box 14428, Melbourne City MC, Victoria 8001, Australia

   E-mail address: neil@matilda.vu.edu.au
   URL: http://sci.vu.edu.au/staff/neilb.html
   E-mail address: pc@matilda.vu.edu.au
   URL: http://sci.vu.edu.au/staff/peterc.html
   E-mail address: sever.dragomir@vu.edu.au
   URL: http://rgmia.vu.edu.au/SSDragomirWeb.html
   E-mail address: sofo@matilda.vu.edu.au