APPROXIMATING CSISZÁR f−DIVERGENCE BY THE USE OF TAYLOR’S FORMULA WITH INTEGRAL REMAINDER

N.S. BARNETT, P. CERONE, S.S. DRAGOMIR, AND A. SOFO

Abstract. Some approximations of the Csiszár f−divergence by the use of Taylor’s formula and perturbed Taylor’s formula and some applications for Kullback-Leibler distance are given.

1. Introduction

One of the important issues in many applications of Probability Theory is finding an appropriate measure of distance (or difference or discrimination ) between two probability distributions. A number of divergence measures for this purpose have been proposed and extensively studied by Jeffreys [1], Kullback and Leibler [2], Rényi [3], Havrda and Charvat [4], Kapur [5], Sharma and Mittal [6], Burbea and Rao [7], Rao [8], Lin [9], Csiszár [10], Ali and Silvey [12], Vajda [13], Shioya and Da-te [47] and others (see for example [5] and the references therein).

These measures have been applied in a variety of fields such as: anthropology [8], genetics [14], finance, economics, and political science [15], [16], [17], biology [18], the analysis of contingency tables [19], approximation of probability distributions [20], [21], signal processing [22], [23] and pattern recognition [24], [25]. A number of these measures of distance are specific cases of Csiszár f-divergence and so further exploration of this concept will have a flow on effect to other measures of distance and to areas in which they are applied.

Assume that a set Γ and the σ−finite measure μ are given. Consider the set of all probability densities on μ to be Ω := {p|p : Γ → ℝ, p (x) ≥ 0, ∫_{Γ} p (x) dμ (x) = 1}. The Kullback-Leibler divergence [2] is well known among the information divergences. It is defined as:

(1.1)  D_{KL} (p, q) := ∫_{Γ} p (x) log [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω,

where log is to base 2.

In Information Theory and Statistics, various divergences are applied in addition to the Kullback-Leibler divergence. These are the: variation distance D_{v}, Hellinger distance D_{H} [40], χ^{2}−divergence D_{χ^{2}}, α−divergence D_{α}, Bhattacharyya distance D_{B} [41], Harmonic distance D_{Ha}, Jeffrey’s distance D_{J} [1], triangular discrimination D_{Δ} [35], etc... They are defined as follows:

(1.2)  D_{v} (p, q) := ∫_{Γ} |p (x) − q (x)| dμ (x) , p, q ∈ Ω;

Date: February 21, 2000.

1991 Mathematics Subject Classification. Primary 94Xxx; Secondary 26D15.

Key words and phrases. Csiszár f−divergence, Taylor’s formula, Analytic inequalities.

(1.3)  D_{H} (p, q) := ∫_{Γ} |\sqrt{p (x)} − \sqrt{q (x)}| dμ (x) , p, q ∈ Ω;

(1.4)  D_{χ^{2}} (p, q) := ∫_{Γ} p (x) [(\frac{q (x)}{p (x)})^{2} − 1] dμ (x) , p, q ∈ Ω;

(1.5)  D_{α} (p, q) := \frac{4}{1 − α^{2}} [1 − ∫_{Γ} [p (x)]^{\frac{1−α}{2}} [q (x)]^{\frac{1+α}{2}} dμ (x)] , p, q ∈ Ω;

(1.6)  D_{B} (p, q) := ∫_{Γ} \sqrt{p (x) q (x)}dμ (x) , p, q ∈ Ω;

(1.7)  D_{Ha} (p, q) := ∫_{Γ} \frac{2p (x) q (x)}{p (x) + q (x)}dμ (x) , p, q ∈ Ω;

(1.8)  D_{J} (p, q) := ∫_{Γ} [p (x) − q (x)] ln [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω;

(1.9)  D_{Δ} (p, q) := ∫_{Γ} \frac{[p (x) − q (x)]^{2}}{p (x) + q (x)} dμ (x) , p, q ∈ Ω.

For other divergence measures, see the paper [5] by Kapur or the book on line [42] by Taneja. For a comprehensive collection of preprints available on line, see the

RGMIA web site http://rgmia.vu.edu.au/papersinfth.html

Csiszár f−divergence is defined as follows [10]

(1.10)  D_{f} (p, q) := ∫_{Γ} p (x) f [\frac{q (x)}{p (x)}] dμ (x) , p, q ∈ Ω,

where f is convex on (0, ∞). It is assumed that f (u) is zero and strictly convex at u = 1. By appropriately defining this convex function, various divergences are derived. All the above distances (1.1) − (1.9), are particular instances of Csiszár f−divergence. There are also many others which are not in this class (see for example [5] or [42]). For the basic properties of Csiszár f−divergence see [43]-[45].

In [46], Lin and Wong (see also [9]) introduced the following divergence

(1.11)  D_{LW} (p, q) := ∫_{Γ} p (x) log [ \frac{p (x)}{\frac{1}{2}p (x) + \frac{1}{2}q (x)}] dμ (x) , p, q ∈ Ω.

This can be represented as follows, using the Kullback-Leibler divergence:

D_{LW} (p, q) = D_{KL} (p, \frac{1}{2}p + \frac{1}{2}q) .

Lin and Wong have established the following inequalities

(1.12)  D_{LW} (p, q) ≤ \frac{1}{2}D_{KL} (p, q) ;
(1.13)  D_{LW} (p, q) + D_{LW} (q, p) ≤ D_{v} (p, q) ≤ 2;
(1.14)  D_{LW} (p, q) ≤ 1.

In [47], Shioya and Da-te improved (1.12) − (1.14) by showing that

D_{LW} (p, q) ≤ \frac{1}{2}D_{v} (p, q) ≤ 1.

CSISZÁR f−DIVERGENCE 3

For classical and new results in comparing different kinds of divergence measures, see the papers [1]-[47] where further references are given.

2. The Results

We start with the following result.

Theorem 1. Let f : (0, ∞) → ℝ be n−time differentiable and such that f^{(n)} is absolutely continuous on [r, R], where 0 < r ≤ 1 ≤ R < ∞. Assume that the probability distributions p, q satisfy the condition

(2.1)  r ≤ \frac{p (x)}{q (x)} ≤ R a.e. on Γ.

Then we have the inequalities:

|I_{f} (p, q) − f (1) − ∑_{k=1}^{n} \frac{f^{(k)} (1)}{k!}D_{χ^{k}} (p, q)| (2.2)

≤ \{ \begin{matrix}\frac{1}{(n+1)!} ∥f^{(n+1)}∥_{∞} D_{|χ|}^{n+1} (p, q) & if & f^{(n+1)} ∈ L_{∞} [r, R] ; \\ _{\frac{1}{n!(nβ+1)^{\frac{1}{β}}}} ∥f^{(n+1)}∥_{α} D_{|χ|}_{n+ \frac{1}{β}} (p, q) & if & f^{(n+1)} ∈ L_{α} [r, R] , \\ & & α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} D_{|χ|}^{n} (p, q) ; \end{matrix}

≤ \{ \begin{matrix}\frac{1}{(n+1)!} ∥f^{(n+1)}∥_{∞} (R − r)^{n+1} & if & f^{(n+1)} ∈ L_{∞} [r, R] ; \\ _{\frac{1}{n!(nβ+1)^{\frac{1}{β}}}} ∥f^{(n+1)}∥_{α} (R − r)^{n+ \frac{1}{β}} & if & f^{(n+1)} ∈ L_{α} [r, R] , \\ & & α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} (R − r)^{n}; \end{matrix}

where

D_{χ^{k}} (p, q) : = ∫_{Γ} \frac{(p (x) − q (x))^{k}}{q^{k−1} (x)} dμ (x) ,
D_{|χ|}^{s} (p, q) : = ∫_{Γ} \frac{|p (x) − q (x)|^{s}}{q^{s−1} (x)} dμ (x) (k ∈ ℕ, s ≥ 0)

and ∥·∥_{α} are the usual Lebesgue norms, i.e.,

∥f^{(n+1)}∥_{α} := (∫_{r}^{R} |f^{(n+1)}|^{α} dt)^{\frac{1}{α}} , α ≥ 1, ∥f^{(n+1)}∥_{∞} := ess sup_{t∈[r,R]} |f^{(n+1)} (t)|.

Proof. We start with the Taylor representation with the integral remainder

(2.3)  f (z) = f (a) + ∑_{k=1}^{n} \frac{(z − a)^{k}}{k!} f^{(k)} (a) + \frac{1}{n!} ∫_{a}^{z} (z − t)^{n} f^{(n+1)} (t) dt

for all z, a ∈ (0, ∞).

Using the properties of the modulus, we have

(2.4) |f (z) − f (a) − ∑_{k=1}^{n} \frac{(z − a)^{k}}{k!} f^{(k)} (a)| ≤ \frac{1}{n!} |∫_{a}^{z} |z − t|^{n} |f^{(n+1)} (t)| dt|
: = M (f^{(n+1)}; a, z) .

Now, assume that a, z ∈ [r, R]. Then, obviously

(2.5)  M (f^{(n+1)}; a, z) ≤ sup_{t∈[r,R]} |f^{(n+1)} (t)| \frac{1}{n!} |∫_{a}^{z} |z − t|^{n} dt|
= \frac{1}{(n + 1)!} ∥f^{(n+1)}∥_{∞} |z − a|^{n+1} ,

for all a, z ∈ [r, R].

Also, by the use of Hölder’s integral inequality, we have:-

M (f^{(n+1)}; a, z) (2.6)

≤ \frac{1}{n!} |∫_{a}^{z} |x − t|^{nβ} dt|^{\frac{1}{β}} [∫_{a}^{z} |f^{(n+1)} (t)|^{α} dt]^{\frac{1}{α}}

≤ \frac{1}{n!} ∥f^{(n+1)}∥_{α} [\frac{|z − a|^{nβ+1}}{nβ + 1} ]^{\frac{1}{β}}
= \frac{1}{n! (nβ + 1)^{\frac{1}{β}}} ∥f^{(n+1)}∥_{α} |z − a|^{n+ \frac{1}{β}} , α > 1, \frac{1}{α} + \frac{1}{β} = 1

and, obviously,

(2.7)  M (f^{(n+1)}; a, z) ≤ \frac{1}{n!} |z − a|^{n} |∫_{a}^{z} |f^{(n+1)} (t)| dt|
≤ \frac{1}{n!} |z − a|^{n} ∥f^{(n+1)}∥_{1}

for all z, a ∈ [r, R].

Consequently, by (2.4)-(2.7), we may write (see also [48] for a similar inequality)

|f (z) − f (a) − ∑_{k=1}^{n} \frac{(z − a)^{k}}{k!} f^{(k)} (a)| (2.8)

≤ \{ \begin{matrix}\frac{1}{(n+1)!} ∥f^{(n+1)}∥_{∞} |z − a|^{n+1} & if & f^{(n+1)} ∈ L_{∞} [r, R] ; \\ _{\frac{1}{n!(nβ+1)^{\frac{1}{β}}}} ∥f^{(n+1)}∥_{α} |z − a|^{n+ \frac{1}{β}} & if & f^{(n+1)} ∈ L_{α} [r, R] , \\ & & α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} |z − a|^{n} , \end{matrix}

for all z, a ∈ [r, R].

If in (2.8) choose z = \frac{p(x)}{q(x)}, a = 1, then we obtain

|f (\frac{p (x)}{q (x)}) − f (1) − ∑_{k=1}^{n} \frac{(\frac{p(x)}{q(x)} − 1)^{k}}{k!} f^{(k)} (1)| (2.9)

≤ \{ \begin{matrix}\frac{1}{(n+1)!} & ∥f^{(n+1)}∥ & ∞ & |\frac{p(x)}{q(x)} − 1| & n+1 \\ \frac{1}{n!(nβ+1)^{\frac{1}{β}}} & ∥f^{(n+1)}∥ & α & |\frac{p(x)}{q(x)} − 1| & n+ \frac{1}{β} \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} |\frac{p(x)}{q(x)} − 1|^{n} , \end{matrix}

≤ \{ \begin{matrix}\frac{1}{(n+1)!} & ∥f^{(n+1)}∥ & ∞ & (R − r) & n+1 \\ \frac{1}{n!(nβ+1)^{\frac{1}{β}}} & ∥f^{(n+1)}∥ & α & (R − r) & n+ \frac{1}{β} \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} (R − r)^{n} , \end{matrix}

for a.e. x ∈ Γ.

If we multiply (2.9) by q (x) ≥ 0 and integrate on Γ, and then use the generalised triangle inequality, we may state that

|I_{f} (p, q) − f (1) − ∑_{k=1}^{n} \frac{f^{(k)} (1)}{k!} · ∫_{Γ} \frac{(p (x) − q (x))^{k}}{q^{k−1} (x)} dμ (x)|

≤ \{ \begin{matrix}\frac{1}{(n+1)!} ∥f^{(n+1)}∥_{∞} ∫_{Γ} \frac{|p(x)−q(x)|^{n+1}}{q^{n}(x)} dμ (x) \\ _{\frac{1}{n!(nβ+1)^{\frac{1}{β}}}} ∥f^{(n+1)}∥_{α} ∫_{Γ} \frac{|p(x)−q(x)|^{n+ \frac{1}{β}}}{q^{n+ \frac{1}{β} −1}(x)} dμ (x) \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} ∫_{Γ} \frac{|p(x)−q(x)|^{n}}{q^{n−1}(x)} dμ (x) , \end{matrix}

≤ \{ \begin{matrix}\frac{1}{(n+1)!} & ∥f^{(n+1)}∥ & ∞ & (R − r) & n+1 \\ \frac{1}{n!(nβ+1)^{\frac{1}{β}}} & ∥f^{(n+1)}∥ & α & (R − r) & n+ \frac{1}{β} \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} (R − r)^{n} , \end{matrix}

and the inequality (2.1) is proved. □

The following theorem also holds.

Theorem 2. Let f be as in Theorem 1. If p^{(j)}, q^{(j)} (j = 1, 2) are probability distributions such that

(2.10)  r ≤ \frac{p^{(j)} (x)}{q^{(j)} (x)} ≤ R a.e. on Γ and j = 1, 2.

6 N.S. BARNETT, P. CERONE, S.S. DRAGOMIR, AND A. SOFO

Then

(2.11)  r ≤ \frac{λp^{(1)} (x) + (1 − λ) p^{(2)} (x)}{λq^{(1)} (x) + (1 − λ) q^{(2)} (x)} ≤ R a.e. on Γ and λ ∈ [0, 1]

and we have the inequality, for f^{(n+1)} ∈ L_{α} [r, R], |I_{f} (λp^{(1)} + (1 − λ) p^{(2)}, λq^{(1)} + (1 − λ) q^{(2)}) (2.12)

−λI_{f} (p^{(1)}, q^{(1)}) − (1 − λ) I_{f} (p^{(2)}, q^{(2)})

−λ ∑_{k=1}^{n} \frac{1}{k!} ∫_{Γ} \begin{matrix}(1 − λ) & k \\ [q^{(1)} (x)] & k−1 \end{matrix} (−1)^{k} η^{k} (x) f^{(k)} (\frac{p^{(1)} (x)}{q^{(1)} (x)}) dμ (x)

− (1 − λ) ∑_{k=1}^{n} \frac{1}{k!} ∫_{Γ} \frac{λ^{k}}{[q^{(2)} (x)]^{k−1}} η^{k} (x) f^{(k)} (\frac{p^{(2)} (x)}{q^{(2)} (x)}) dμ (x)|

≤ \{ \begin{matrix}\frac{1}{(n+1)!} ∥f^{(n+1)}∥_{∞} λ (1 − λ) ∫_{Γ} |η (x)|^{n+1} [\begin{matrix}(1−λ) & ^{n} \\ [q^{(1)}(x)] & ^{n} \end{matrix} + \frac{λ^{n}}{[q^{(2)}(x)]^{n}} ] dμ (x) ; \\ _{\frac{1}{n!(nβ+1)^{\frac{1}{β}}}} ∥f^{(n+1)}∥_{α} λ (1 − λ) \\ × ∫_{Γ} |η (x)|^{n+ \frac{1}{β}} [\begin{matrix}(1−λ) & ^{n+ \frac{1}{β} −1} \\ [q^{(1)}(x)] & ^{n+ \frac{1}{β} −1} \end{matrix} + \frac{λ^{n+ \frac{1}{β} −1}}{[q^{(2)}(x)]^{n+ \frac{1}{β} −1}} ] dμ (x) ; \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} λ (1 − λ) ∫_{Γ} |η (x)|^{n} [\begin{matrix}(1−λ) & ^{n−1} \\ [q^{(1)}(x)] & ^{n−1} \end{matrix} + \frac{λ^{n−1}}{[q^{(2)}(x)]^{n−1}} ] dμ (x) , \end{matrix}

where

η (x) = η (λ, p^{(1)}, p^{(2)}, q^{(1)}, q^{(2)}) (x) = \frac{det [ \begin{matrix}p^{(1)} (x) & p^{(2)} (x) \\ q^{(1)} (x) & q^{(2)} (x) \end{matrix} ]}{λq^{(1)} (x) + (1 − λ) q^{(2)} (x)},

for all λ ∈ [0, 1] and x ∈ Γ. Proof. We use the inequality (2.8) to write

|f (\frac{λp^{(1)} (x) + (1 − λ) p^{(2)} (x)}{λq^{(1)} (x) + (1 − λ) q^{(2)} (x)}) − f (\frac{p^{(1)} (x)}{q^{(1)} (x)}) (2.13)

− ∑_{k=1}^{n} \frac{1}{k!} (\frac{λp^{(1)} (x) + (1 − λ) p^{(2)} (x)}{λq^{(1)} (x) + (1 − λ) q^{(2)} (x)} − \frac{p^{(1)} (x)}{q^{(1)} (x)})^{k} f^{(k)} (\frac{p^{(1)} (x)}{q^{(1)} (x)})|

≤ \{ \begin{matrix}\frac{1}{(n+1)!} & ∥f^{(n+1)}∥ & ∞ & |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(1)}(x)}{q^{(1)}(x)} | & n+1 \\ \frac{1}{n!(nβ+1)^{\frac{1}{β}}} & ∥f^{(n+1)}∥ & α & |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(1)}(x)}{q^{(1)}(x)} | & n+ \frac{1}{β} \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(1)}(x)}{q^{(1)}(x)} |^{n} , \end{matrix}

CSISZÁR f−DIVERGENCE 7

and

|f (\frac{λp^{(1)} (x) + (1 − λ) p^{(2)} (x)}{λq^{(1)} (x) + (1 − λ) q^{(2)} (x)}) − f (\frac{p^{(2)} (x)}{q^{(2)} (x)}) (2.14)

− ∑_{k=1}^{n} \frac{1}{k!} (\frac{λp^{(1)} (x) + (1 − λ) p^{(2)} (x)}{λq^{(1)} (x) + (1 − λ) q^{(2)} (x)} − \frac{p^{(2)} (x)}{q^{(2)} (x)})^{k} f^{(k)} (\frac{p^{(2)} (x)}{q^{(2)} (x)})|

≤ \{ \begin{matrix}\frac{1}{(n+1)!} & ∥f^{(n+1)}∥ & ∞ & |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(2)}(x)}{q^{(2)}(x)} | & n+1 \\ \frac{1}{n!(nβ+1)^{\frac{1}{β}}} & ∥f^{(n+1)}∥ & α & |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(2)}(x)}{q^{(2)}(x)} | & n+ \frac{1}{β} \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(2)}(x)}{q^{(2)}(x)} |^{n} . \end{matrix}

If we multiply (2.13) by λq^{(1)} (x) and (2.14) by (1 − λ) q^{(2)} (x) , add them and apply the triangle inequality, we end up with

|(λq^{(1)} (x) + (1 − λ) q^{(2)} (x)) f (\frac{λp^{(1)} (x) + (1 − λ) p^{(2)} (x)}{λq^{(1)} (x) + (1 − λ) q^{(2)} (x)}) (2.15)

−λq^{(1)} (x) f (\frac{p^{(1)} (x)}{q^{(1)} (x)}) − (1 − λ) q^{(2)} (x) f (\frac{p^{(2)} (x)}{q^{(2)} (x)})

−λq^{(1)} (x) ∑_{k=1}^{n} \frac{1}{k!} (\frac{λp^{(1)} (x) + (1 − λ) p^{(2)} (x)}{λq^{(1)} (x) + (1 − λ) q^{(2)} (x)} − \frac{p^{(1)} (x)}{q^{(1)} (x)})^{k} f^{(k)} (\frac{p^{(1)} (x)}{q^{(1)} (x)})

− (1 − λ) ∑_{k=1}^{n} \frac{1}{k!} (\frac{λp^{(1)} (x) + (1 − λ) p^{(2)} (x)}{λq^{(1)} (x) + (1 − λ) q^{(2)} (x)} − \frac{p^{(2)} (x)}{q^{(2)} (x)})^{k} f^{(k)} (\frac{p^{(2)} (x)}{q^{(2)} (x)})|

≤ \{ \begin{matrix}\frac{1}{(n+1)!} & ∥f^{(n+1)}∥ & ∞ & [λq^{(1)} (x) |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(1)}(x)}{q^{(1)}(x)} |^{n+1} \\ + (1 − λ) q^{(2)} (x) |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(2)}(x)}{q^{(2)}(x)} |^{n+1}] \\ \frac{1}{n!(nβ+1)^{\frac{1}{β}}} & ∥f^{(n+1)}∥ & α & [λq^{(1)} (x) |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(1)}(x)}{q^{(1)}(x)} |^{n+ \frac{1}{β}} \\ + (1 − λ) q^{(2)} (x) |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(2)}(x)}{q^{(2)}(x)} |^{n+ \frac{1}{β}} ] \\ \frac{1}{n!} & ∥f^{(n+1)}∥ & 1 & [λq^{(1)} (x) |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(1)}(x)}{q^{(1)}(x)} |^{n} \\ + (1 − λ) q^{(2)} (x) |\frac{λp^{(1)}(x)+(1−λ)p^{(2)}(x)}{λq^{(1)}(x)+(1−λ)q^{(2)}(x)} − \frac{p^{(2)}(x)}{q^{(2)}(x)} |^{n}] , \end{matrix}

8 N.S. BARNETT, P. CERONE, S.S. DRAGOMIR, AND A. SOFO

which is equivalent to

|(λq^{(1)} (x) + (1 − λ) q^{(2)} (x)) f (\frac{λp^{(1)} (x) + (1 − λ) p^{(2)} (x)}{λq^{(1)} (x) + (1 − λ) q^{(2)} (x)}) (2.16)

−λq^{(1)} (x) f (\frac{p^{(1)} (x)}{q^{(1)} (x)}) − (1 − λ) q^{(2)} (x) f (\frac{p^{(2)} (x)}{q^{(2)} (x)})

−λ ∑_{k=1}^{n} \frac{1}{k!} \begin{matrix}(1 − λ) & k & (Det_{2,1}) & k \\ [λq^{(1)} (x) + (1 − λ) q^{(2)} (x)] & k & [q^{(1)} (x)] & k−1 \end{matrix} f^{(k)} (\frac{p^{(1)} (x)}{q^{(1)} (x)})

− (1 − λ) ∑_{k=1}^{n} \frac{1}{k!} \frac{λ^{k} (Det_{1,2})^{k}}{[λq^{(1)} (x) + (1 − λ) q^{(2)} (x)]^{k} [q^{(2)} (x)]^{k−1}} f^{(k)} (\frac{p^{(2)} (x)}{q^{(2)} (x)})|

≤ \{ \begin{matrix}\frac{1}{(n+1)!} & ∥f^{(n+1)}∥ & ∞ & [\frac{λ(1−λ)^{n+1}|Det_{2,1}|^{n+1}}{[λq^{(1)}(x)+(1−λ)q^{(2)}(x)]^{n+1}[q^{(1)}(x)]^{n}} \\ + \frac{(1−λ)λ^{n+1}|Det_{2,1}|^{n+1}}{[λq^{(1)}(x)+(1−λ)q^{(2)}(x)]^{n+1}[q^{(2)}(x)]^{n}} ]; \\ \frac{1}{n!(nβ+1)^{\frac{1}{β}}} & ∥f^{(n+1)}∥ & α & [\frac{λ(1−λ)^{n+ \frac{1}{β}} |Det_{2,1}|^{n+ \frac{1}{β}}}{[λq^{(1)}(x)+(1−λ)q^{(2)}(x)]^{n+ \frac{1}{β}} [q^{(1)}(x)]^{n+ \frac{1}{β} −1}} \\ +\frac{(1−λ)λ^{n+ \frac{1}{β}} |Det_{2,1}|^{n+ \frac{1}{β}}}{[λq^{(1)}(x)+(1−λ)q^{(2)}(x)]^{n+ \frac{1}{β}} [q^{(2)}(x)]^{n+ \frac{1}{β} −1}} ]; \\ \frac{1}{n!} & ∥f^{(n+1)}∥ & 1 & [\frac{λ(1−λ)^{n}|Det_{2,1}|^{n}}{[λq^{(1)}(x)+(1−λ)q^{(2)}(x)]^{n}[q^{(1)}(x)]^{n−1}} \\ +\frac{(1−λ)λ^{n}|Det_{2,1}|^{n}}{[λq^{(1)}(x)+(1−λ)q^{(2)}(x)]^{n}[q^{(2)}(x)]^{n−1}} ]; \end{matrix}

= \{ \begin{matrix}\frac{1}{(n+1)!} ∥f^{(n+1)}∥_{∞} \frac{λ(1−λ)|Det_{2,1}|^{n+1}}{[λq^{(1)}(x)+(1−λ)q^{(2)}(x)]^{n+1}} [\begin{matrix}(1−λ) & ^{n} \\ [q^{(1)}(x)] & ^{n} \end{matrix} + \frac{λ^{n}}{[q^{(2)}(x)]^{n}} ]; \\ \frac{1}{n!(nβ+1)^{\frac{1}{β}}} & ∥f^{(n+1)}∥ & α & \frac{λ(1−λ)|Det_{2,1}|^{n+ \frac{1}{β}}}{[λq^{(1)}(x)+(1−λ)q^{(2)}(x)]^{n+ \frac{1}{β}}} \\ × [\begin{matrix}(1−λ) & ^{n+ \frac{1}{β} −1} \\ [q^{(1)}(x)] & ^{n+ \frac{1}{β} −1} \end{matrix} + \frac{λ^{n+ \frac{1}{β} −1}}{[q^{(2)}(x)]^{n+ \frac{1}{β} −1}} ]; \\ \frac{1}{n!} ∥f^{(n+1)}∥_{1} \frac{λ(1−λ)|Det_{2,1}|^{n}}{[λq^{(1)}(x)+(1−λ)q^{(2)}(x)]^{n}} [\begin{matrix}(1−λ) & ^{n−1} \\ [q^{(1)}(x)] & ^{n−1} \end{matrix} + \frac{λ^{n−1}}{[q^{(2)}(x)]^{n−1}} ] , \end{matrix}

where

Det_{y,z} = det [ \begin{matrix}p^{(y)} (x) & p^{(z)} (x) \\ q^{(y)} (x) & q^{(z)} (x) \end{matrix} ], where (y, z) ∈ \{(1, 2) , (2, 1)\}.

Integrating (2.16) over x ∈ Γ and using the generalised triangle inequality, we deduce the desired inequality (2.12). □

In the recent paper [49], S.S. Dragomir proved the following perturbed Taylor’s formula which is an improvement of a recent result due to Matić, Pečarić and Ujević from [50]. It is instructive to give the details here for the sake of completeness. Lemma 1. Let f : I ⊆ ℝ → ℝ be such that f^{(n)} is absolutely continuous and f^{(n+1)} ∈ L_{2} (I). Then we have

(2.17)  f (z) = ∑_{k=0}^{n} \frac{(z − a)^{k}}{k!} f^{(k)} (a) + \frac{(z − a)^{n+1}}{(n + 1)!} [f^{(n)}; a, z] + G_{n} (f; a, z)

for all z ∈ I, where

[f^{(n)}; a, z] = \frac{f^{(n)} (z) − f^{(n)} (a)}{z − a}

and G_{n} (f; a, z) satisfies the estimate

(2.18) |G_{n} (f; a, z)| ≤ \frac{n (z − a)^{n+1}}{(n + 1)!\sqrt{2n + 1}} [\frac{1}{z − a} ∥f^{(n+1)}∥_{2}^{2} − ([f^{(n)}; a, z])^{2}]^{\frac{1}{2}}

for all z ≥ a.

Proof. Recall Korkine’s identity for the mapping h, g

\frac{1}{z − a} ∫_{a}^{z} h (t) g (t) dt − \frac{1}{(z − a)^{2}} ∫_{a}^{z} h (t) dt · ∫_{a}^{z} g (t) dt (2.19)

= \frac{1}{2 (z − a)} ∫_{a}^{z} ∫_{a}^{z} (h (t) − h (s)) (g (t) − g (s)) dtds.

Using (2.19), we have

∫_{a}^{z} \frac{(z − t)^{n}}{n!} f^{(n+1)} (t) dt − \frac{1}{z − a} ∫_{a}^{z} \frac{(z − t)^{n}}{n!} dt · ∫_{a}^{z} f^{(n+1)} (t) dt

= \frac{1}{2 (z − a)} ∫_{a}^{z} ∫_{a}^{z} (\frac{(z − t)^{n} − (z − s)^{n}}{n!} ) (f^{(n+1)} (t) − f^{(n+1)} (s)) dtds.

and then, using Taylor’s representation (2.3) and the formula (2.17), we may conclude that

G_{n} (f; a, z) (2.20)

= \frac{1}{2 (z − a)} ∫_{a}^{z} ∫_{a}^{z} [\frac{(z − t)^{n} − (z − s)^{n}}{n!} ] (f^{(n+1)} (t) − f^{(n+1)} (s)) dtds.

Now, using the Cauchy-Buniakowski-Schwartz integral inequality for double integrals, we have

|G_{n} (f; a, z)| (2.21)

≤ \frac{1}{2 (z − a)} [∫_{a}^{z} ∫_{a}^{z} [\frac{(z − t)^{n} − (z − s)^{n}}{n!} ]^{2} dtds

× ∫_{a}^{z} ∫_{a}^{z} [f^{(n+1)} (t) − f^{(n+1)} (s)]^{2} dtds]^{\frac{1}{2}} .

Elementary calculations show that

\frac{1}{2 (z − a)^{2}} ∫_{a}^{z} ∫_{a}^{z} [\frac{(z − t)^{n} − (z − s)^{n}}{n!} ]^{2} dtds = \frac{n^{2} (z − a)^{2n}}{[(n + 1)!]^{2} (2n + 1)}

and (see also [50])

\frac{1}{2 (z − a)^{2}} ∫_{a}^{z} ∫_{a}^{z} [f^{(n+1)} (t) − f^{(n+1)} (s)]^{2} dtds

= \frac{1}{z − a} ∥f^{(n+1)}∥_{2}^{2} − ([f^{(n)}; a, z])^{2} ,

and so, by (2.21), we deduce (2.18). □

Now, by the Grüss inequality, we may state that

0 ≤ \frac{1}{z − a} ∫_{a}^{z} [f^{(n+1)} (t)]^{2} dt − (\frac{1}{z − a} ∫_{a}^{z} f^{(n+1)} (t) dt)^{2}
≤ \frac{1}{4} (Γ (z) − γ (z)) ,

where

(2.22)  γ (z) ≤ f^{(n+1)} (t) ≤ Γ (z) for all t ∈ [a, z] ,

then, by Lemma 1, we can obtain the result in [49], showing that (2.18) is an improvement on the pre-Grüss result obtained in [50].

Corollary 1. Let f : I ⊆ ℝ → ℝ be such that f^{(n)} is absolutely continuous and f^{(n+1)} is bounded and satisfies (2.22). Then we have the representation (2.17) and the remainder G_{n} (f; a, z) satisfies the estimate

(2.23)  |G_{n} (f; a, z)| ≤ \frac{n (z − a)^{n+1}}{2 (n + 1)!\sqrt{2n + 1}} (Γ (z) − γ (z))

for all z ≥ a.

If z ≤ a, then a similar bound can be stated and so, in general, for any a ∈ I, we have the representation (2.17) and the bounds

|G_{n} (f; a, z)| (2.24)

≤ \frac{n |z − a|^{n+1}}{(n + 1)!\sqrt{2n + 1}} [\frac{∫_{a}^{z} [f^{(n+1)} (t)]^{2} dt}{z − a} − ([f^{(n)}; a, z])^{2}]^{\frac{1}{2}}

≤ \frac{n |z − a|^{n+1}}{2 (n + 1)!\sqrt{2n + 1}} (Γ (z) − γ (z)) ,

where

Γ := sup_{z∈I} f^{(n+1)} (z) < ∞ and γ := inf_{z∈I} f^{(n+1)} (z) > −∞.

In what follows, we use the estimate (2.24).

Theorem 3. Let f : (0, ∞) → ℝ be n-time differentiable and such that f^{(n)} is absolutely continuous on [r, R], where 0 < r ≤ 1 ≤ R < ∞. Assume that the probability distributions p, q satisfy the condition

(2.25)  r ≤ \frac{p (x)}{q (x)} ≤ R a.e on Γ.

CSISZÁR f−DIVERGENCE 11

Then we have the inequalities

|I_{f} (p, q) − f (1) − [∑_{k=1}^{n} \frac{f^{(k)} (1)}{k!}D_{χ^{k}} (p, q)] (2.26)

− \frac{1}{(n + 1)!}I_{(·−1)^{k}f^{(k)}(·)} (p, q) + \frac{f^{(n)} (1)}{(n + 1)!}D_{χ^{n}} (p, q)|

≤ \frac{n}{(n + 1)!\sqrt{2n + 1}}B (p, q, f^{(n+1)})
≤ \frac{n (Φ − ϕ)}{(n + 1)!\sqrt{2n + 1}}D_{|χ|}^{n+1} (p, q) ≤ \frac{n (Φ − ϕ)}{(n + 1)!\sqrt{2n + 1}} (R − r)^{n+1} ,

where

Φ := sup_{z∈[r,R]} f^{(n+1)} (z) < ∞ and ϕ := inf_{z∈[r,R]} f^{(n+1)} (z) > −∞

and

B (p, q, f^{(n+1)}) := I_{g} (p, q)

where

g (z) = |z − 1|^{n+1} [\frac{1}{z − 1} ∫_{1}^{z} [f^{(n+1)} (t)]^{2} dt − (\frac{f^{(n)} (z) − f^{(n)} (1)}{z − 1})^{2}]^{\frac{1}{2}} .

Proof. Apply the inequality (2.24) for a = 1 and z = \frac{p(x)}{q(x)} to obtain

|f (\frac{p (x)}{q (x)}) − f (1) − ∑_{k=1}^{n} \frac{f^{(k)} (1)}{k!} (\frac{p (x)}{q (x)} − 1)^{k}

−\frac{(\frac{p(x)}{q(x)} − 1)^{n}}{(n + 1)!} [f^{(n)} (\frac{p (x)}{q (x)}) − f^{(n)} (1)]|

≤ \frac{n |\frac{p(x)}{q(x)} − 1|^{n}}{(n + 1)!\sqrt{2n + 1}} \frac{q (x)}{p (x) − q (x)} ∫1^{\frac{p(x)}{q(x)}} [f^{(n+1)} (t)]^{2} dt

− (\frac{f^{(n)} (\frac{p(x)}{q(x)}) − f^{(n)} (1)}{\frac{p(x)}{q(x)} − 1})^{2}]^{\frac{1}{2}} ≤ \frac{n |\frac{p(x)}{q(x)} − 1|^{n+1} (Φ − ϕ)}{(n + 1)!\sqrt{2n + 1}}.

If we multiply by p (x) ≥ 0, integrate over x ∈ Γ and use the generalised triangle inequality, we deduce

|I_{f} (p, q) − f (1) − ∑_{k=1}^{n} \frac{f^{(k)} (1)}{k!}D_{χ^{k}} (p, q)

−\frac{1}{(n + 1)!} ∫_{Γ} q (x) (\frac{p (x)}{q (x)} − 1)^{k} f^{(k)} (\frac{p (x)}{q (x)}) dμ (x) + \frac{f^{(n)} (1)}{(n + 1)!}D_{χ^{n}} (p, q)|

≤ \frac{n (Φ − ϕ)}{(n + 1)!\sqrt{2n + 1}} ∫_{Γ} q (x) |\frac{p (x)}{q (x)} − 1|^{n+1} [\frac{q (x)}{p (x) − q (x)} ∫1^{\frac{p(x)}{q(x)}} [f^{(n+1)} (t)]^{2} dt

−q^{2} (x) \begin{matrix}|f^{(n)} (\frac{p(x)}{q(x)}) − f^{(n)} (1)| & 2 \\ (p (x) − q (x)) & 2 \end{matrix} ]^{\frac{1}{2}} dμ (x)

≤ \frac{n (Φ − ϕ)}{2 (n + 1)!\sqrt{2n + 1}}D_{|χ|}^{n+1} (p, q) ≤ \frac{n (Φ − ϕ)}{2 (n + 1)!\sqrt{2n + 1}} (R − r)^{n+1}

and the theorem is proved. □

3. Some Particular Inequalities

The following proposition holds. Proposition 1. Let p, q be two probability distributions satisfying the condition

(3.1)  0 < r ≤ \frac{p (x)}{q (x)} ≤ R < ∞ a.e on Γ.

Then, for n ≥ 1, we have the inequality

(3.2)  |D_{KL} (q, p) − ∑_{k=1}^{n} \frac{(−1)^{k}}{k!} D_{χ^{k}} (p, q)|

≤ \{ \begin{matrix}\frac{1}{(n + 1) r^{n+1}} D_{|χ|}^{n+1} (p, q) ; \\ \frac{1}{(nβ + 1)^{\frac{1}{β}} [(n + 1) α − 1]^{\frac{1}{α}}} [\frac{R^{(n+1)α−1} − r^{(n+1)α−1}}{R^{(n+1)α−1}r^{(n+1)α−1}} ]^{\frac{1}{α}} D_{|χ|}_{n+ \frac{1}{β}} (p, q) , \\ α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n} · \frac{R^{n} − r^{n}}{R^{n}r^{n}} D_{|χ|}^{n} (p, q) ; \end{matrix}

(3.3) ≤ \{ \begin{matrix}\frac{1}{(n + 1) r^{n+1}} (R − r)^{n+1} ; \\ \frac{1}{(nβ + 1)^{\frac{1}{β}} [(n + 1) α − 1]^{\frac{1}{α}}} [\frac{R^{(n+1)α−1} − r^{(n+1)α−1}}{R^{(n+1)α−1}r^{(n+1)α−1}} ]^{\frac{1}{α}} (R − r)^{n+ \frac{1}{β}} , \\ α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{n} · \frac{R^{n} − r^{n}}{R^{n}r^{n}} (R − r)^{n}; \end{matrix}

Proof. Consider the mapping f (t) = ln t. We have

I_{f} (p, q) = ∫_{Γ} q (x) f (\frac{p (x)}{q (x)}) dμ (x) = ∫_{Γ} q (x) ln (\frac{p (x)}{q (x)}) dμ (x)
= − ∫_{Γ} q (x) ln (\frac{q (x)}{p (x)}) dμ (x) = −D_{KL} (q, p) ,

f^{(k)} (t) = \frac{(−1)^{k−1} (k − 1)!}{t^{k}}, k ∈ ℕ, k ≥ 1

for α > 1 and

∥f^{(n+1)}∥_{∞} : = sup_{t∈[r,R]} |f^{(n+1)} (t)| = n! sup_{t∈[r,R]} \{\frac{1}{t^{n+1}} \} = \frac{n!}{r^{n+1}} ;

∥f^{(n+1)}∥_{α} : = (∫_{r}^{R} |f^{(n+1)} (t)|^{α} dt)^{\frac{1}{α}} = n! [∫_{r}^{R} \frac{dt}{t^{(n+1)α}} ]^{\frac{1}{α}}

= n! [ \frac{t^{−(n+1)α+1}}{− (n + 1) α + 1}dt|_{r}^{R}]^{\frac{1}{α}}

= n! [\frac{R^{(n+1)α−1} − r^{(n+1)α−1}}{[(n + 1) α − 1] R^{(n+1)α−1}r^{(n+1)α−1}} ]^{\frac{1}{α}} .

Applying Theorem 1 and using the above assumptions, we deduce the desired inequality (3.2). □

The following proposition also holds.

Proposition 2. Let p, q be as in the above Proposition 1. Then we have the inequality

|D_{KL} (q, p) − ∑_{k=2}^{n} \frac{(−1)^{k}}{(k − 1) k} D_{χ^{k}} (p, q)| (3.4)

≤ \{ \begin{matrix}\frac{1}{n (n + 1) r^{n+1}} D_{|χ|}^{n+1} (p, q) ; \\ \frac{1}{n (nβ + 1)^{\frac{1}{β}} (nα − 1)^{\frac{1}{α}}} [\frac{R^{nα−1} − r^{nα−1}}{R^{nα−1}r^{nα−1}} ]^{\frac{1}{α}} D_{|χ|}_{n+ \frac{1}{β}} (p, q) , \\ α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{(n − 1) n} · \frac{R^{n−1} − r^{n−1}}{R^{n−1}r^{n−1}} D_{|χ|}^{n} (p, q) ; \end{matrix}

≤ \{ \begin{matrix}\frac{1}{n (n + 1) r^{n+1}} (R − r)^{n+1} ; \\ \frac{1}{n (nβ + 1)^{\frac{1}{β}} (nα − 1)^{\frac{1}{α}}} [\frac{R^{nα−1} − r^{nα−1}}{R^{nα−1}r^{nα−1}} ]^{\frac{1}{α}} (R − r)^{n+ \frac{1}{β}} , \\ α > 1, \frac{1}{α} + \frac{1}{β} = 1; \\ \frac{1}{(n − 1) n} · \frac{R^{n−1} − r^{n−1}}{R^{n−1}r^{n−1}} (R − r)^{n} , \end{matrix}

for x ≥ a.

Proof. Consider the mapping f (t) = t ln (t) . We have

I_{f} (p, q) = ∫_{Γ} q (x) f (\frac{p (x)}{q (x)}) dμ (x) = ∫_{Γ} q (x) \frac{p (x)}{q (x)} ln (\frac{p (x)}{q (x)}) dμ (x)
= ∫_{Γ} p (x) ln (\frac{p (x)}{q (x)}) dμ (x) = D_{KL} (p, q) ,
f^{(1)} (t) = ln t + 1,
f^{(k)} (t) = \frac{(−1)^{k} (k − 2)!}{t^{k−1}}, k ≥ 2
∥f^{(n+1)}∥_{∞} = \frac{(n − 1)!}{r^{n}},
∥f^{(n+1)}∥_{α} = (n − 1)! [\frac{R^{nα−1} − r^{nα−1}}{R^{nα−1}r^{nα−1}} ]^{\frac{1}{α}} , α > 1

and

∥f^{(n+1)}∥_{1} = (n − 2)!\frac{R^{n−1} − r^{n−1}}{R^{n−1}r^{n−1}} .

Applying Theorem 1 for the mapping f (t) = t ln t, we have

|D_{KL} (p, q) − f^{(1)} (1) D_{χ} (p, q) − ∑_{k=2}^{n} \frac{(−1)^{k} (k − 2)!}{k!}D_{χ^{k}} (p, q)|

≤ \{ \begin{matrix}\frac{1}{(n+1)!} \frac{(n−1)!}{r^{n}}D_{|χ|}^{n+1} (p, q) ; \\ _{\frac{1}{n(nβ+1)^{\frac{1}{β}}}} · \frac{(n−1)!}{(nα−1)^{\frac{1}{α}}} [\frac{R^{nα−1}−r^{nα−1}}{R^{nα−1}r^{nα−1}} ]^{\frac{1}{α}} D_{|χ|}_{n+ \frac{1}{β}} (p, q) ; \\ \frac{1}{n!} (n − 2)!\frac{R^{n−1}−r^{n−1}}{R^{n−1}r^{n−1}} D_{|χ|}^{n} (p, q) . \end{matrix}

That is,

|D_{KL} (q, p) − ∑_{k=2}^{n} \frac{(−1)^{k}}{(k − 1) k} D_{χ^{k}} (p, q)|

≤ \{ \begin{matrix}\frac{1}{n(n+1)r^{n+1}} D_{|χ|}^{n+1} (p, q) ; \\ _{\frac{1}{n(nβ+1)^{\frac{1}{β}} (nα−1)^{\frac{1}{α}}}} [\frac{R^{nα−1}−r^{nα−1}}{R^{nα−1}r^{nα−1}} ]^{\frac{1}{α}} D_{|χ|}_{n+ \frac{1}{β}} (p, q) ; \\ \frac{1}{(n−1)n} \frac{R^{n−1}−r^{n−1}}{R^{n−1}r^{n−1}} D_{|χ|}^{n} (p, q) \end{matrix}

and the first inequality in (3.4) is proved.

The second inequality is obvious and we omit the details. □

Remark 1. Similar results can be obtained if we apply Theorem 1 for other particular mappings f, generating the Hellinger, Jeffrey’s, Bhattacharyya, or other divergence measures as considered in the introduction. Remark 2. Perturbed inequalities, such as those stated in Theorem 3, may also be considered. We omit the details.

References

[1] H. JEFFREYS, An invariant form for the prior probability in estimating problems, Proc. Roy. Soc. London, 186 A (1946), 453-461.

[2] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Ann. Math. Stat., 22 (1951), 79-86.

[3] A. RÉNYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Stat. and Prob., University of California Press, 1 (1961), 547-561.

[4] J.H. HAVRDA and F. CHARVAT, Quantification method classification process: concept of structural α-entropy, Kybernetika, 3 (1967), 30-35.

[5] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), 1-16.

[6] B.D. SHARMA and D.P. MITTAL, New non-additive measures of relative information, Journ. Comb. Inf. Sys. Sci., 2 (4)(1977), 122-132.

[7] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy function, IEEE Trans. Inf. Th., 28 (3) (1982), 489-495.

[8] C.R. RAO, Diversity and dissimilarity coefficients: a unified approach, Theoretic Population Biology, 21 (1982), 24-43.

[9] J. LIN, Divergence measures based on the Shannon entropy, IEEE Trans. Inf. Th., 37 (1) (1991), 145-151.

[10] I. CSISZÁR, Information-type measures of difference of probability distributions and indirect observations, Studia Math. Hungarica, 2 (1967), 299-318.

[11] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[12] S.M. ALI and S.D. SILVEY, A general class of coefficients of divergence of one distribution from another, J. Roy. Statist. Soc. Sec B, 28 (1966), 131-142.

[13] I. VAJDA, Theory of Statistical Inference and Information, Dordrecht-Boston, Kluwer Academic Publishers, 1989.

[14] M. MEI, The theory of genetic distance and evaluation of human races, Japan J. Human Genetics, 23 (1978), 341-369.

[15] A. SEN, On Economic Inequality, Oxford University Press, London 1973.

[16] H. THEIL, Economics and Information Theory, North-Holland, Amsterdam, 1967.

[17] H. THEIL, Statistical Decomposition Analysis, North-Holland, Amsterdam, 1972.

[18] E.C. PIELOU, Ecological Diversity, Wiley, New York, 1975.

[19] D.V. GOKHALE and S. KULLBACK, Information in Contingency Tables, New York, Merul Dekker, 1978.

[20] C.K. CHOW and C.N. LIN, Approximating discrete probability distributions with dependence trees, IEEE Trans. Inf. Th., 14 (3) (1968), 462-467.

[21] D. KAZAKOS and T. COTSIDAS, A decision theory approach to the approximation of discrete probability densities, IEEE Trans. Perform. Anal. Machine Intell., 1 (1980), 61-67.

[22] T.T. KADOTA and L.A. SHEPP, On the best finite set of linear observables for discriminating two Gaussian signals, IEEE Trans. Inf. Th., 13 (1967), 288-294.

[23] T. KAILATH, The divergence and Bhattacharyya distance measures in signal selection, IEEE Trans. Comm. Technology., Vol COM-15 (1967), 52-60.

[24] M. BETH BASSAT, f-entropies, probability of error and feature selection, Inform. Control, 39 (1978), 227-242.

[25] C.H. CHEN, Statistical Pattern Recognition, Rocelle Park, New York, Hoyderc Book Co., 1973.

[26] V.A. VOLKONSKI and J. A. ROZANOV, Some limit theorems for random function -I, (English Trans.), Theory Prob. Appl., ( USSR), 4 (1959), 178-197.

[27] M.S. PINSKER, Information and Information Stability of Random variables and processes, (in Russian), Moscow: Izv. Akad. Nouk, 1960.

[28] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[29] H.P. McKEAN, JR., Speed of approach to equilibrium for Koc’s caricature of a Maximilian gas, Arch. Ration. Mech. Anal., 21 (1966), 343-367.

[30] J.H.B. KEMPERMAN, On the optimum note of transmitting information, Ann. Math. Statist., 40 (1969), 2158-2177.

[31] S. KULLBACK, A lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 13 (1967), 126-127.

[32] S. KULLBACK, Correction to a lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[33] I. VAJDA, Note on discrimination information and variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[34] G.T. TOUSSAINT, Sharper lower bounds for discrimination in terms of variation, IEEE Trans. Inf. Th., 21 (1975), 99-100.

[35] F. TOPSOE, Some inequalities for information divergence and related measures of discrimination, Res. Rep. Coll., RGMIA, 2 (1) (1999), 85-98.

[36] L. LECAM, Asymptotic Methods in Statistical Decision Theory, New York: Springer, 1986.

[37] D. DACUNHA-CASTELLE, Ecole d’ete de Probability de Saint-Flour, III-1977, Berlin, Heidelberg: Springer 1978.

[38] C. KRAFT, Some conditions for consistency and uniform consistency of statistical procedures, Univ. of California Pub. in Statistics, 1 (1955), 125-142.

[39] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[40] E. HELLINGER, Neue Bergrüirdung du Theorie quadratisher Formerus von uneudlichvieleu Veränderlicher, J. für reine and Augeur. Math., 36 (1909), 210-271.

[41] A. BHATTACHARYYA, On a measure of divergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc., 35 (1943), 99-109.

[42] I. J. TANEJA, Generalised Information Measures and their Applications

    (http://www.mtm.ufsc.br/~taneja/bhtml/bhtml.html).

[43] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[44] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[45] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[46] J. LIN and S.K.M. WONG, A new directed divergence measure and its characterization, Int. J. General Systems, 17 (1990), 73-81.

[47] H. SHIOYA and T. DA-TE, A generalisation of Lin divergence and the derivative of a new information divergence, Elec. and Comm. in Japan, 78 (7) (1995), 37-40.

[48] S.S. DRAGOMIR, New estimation of the remainder in Taylor’s formula using Grüss’ type inequalities and applications, Math. Ineq. and Appl., 2 (2) (1999), 183-193.

[49] S.S. DRAGOMIR, An improvement of the remainder estimate in the generalised Taylor’s formula, RGMIA Res. Rep. Coll., 3 (2000), No. 1, Article 1.

[50] M. MATIĆ, J. PEČARIĆ and N. UJEVIĆ, On new estimation of the remainder in generalised Taylor’s formula, Math. Ineq. and Appl., 2 (3) (1999), 343-361.

   E-mail address: neil@matilda.vu.edu.au
   E-mail address: pc@matilda.vu.edu.au

School of Communications and Informatics, Victoria University of Technology, PO Box 14428, Melbourne City MC 8001,, Victoria, Australia.

   E-mail address: sever.dragomir@vu.edu.au
   URL: http://rgmia.vu.edu.au/SSDRagomirWeb.html
   E-mail address: sofo@matilda.vu.edu.au