APPROXIMATING CSISZÁR f−DIVERGENCE VIA A GENERALISED TAYLOR FORMULA

S.S. DRAGOMIR AND V. GLUŠCEVIĆ

Abstract. Some approximation of the Csiszár f−divergence by the use of a generalised Taylor formula and applications are given.

1. Introduction

One of the important issues in many applications of Probability Theory is finding an appropriate measure of distance (or difference or discrimination ) between two probability distributions. A number of divergence measures for this purpose have been proposed and extensively studied by Jeffreys [1], Kullback and Leibler [2], Rényi [3], Havrda and Charvat [4], Kapur [5], Sharma and Mittal [6], Burbea and Rao [7], Rao [8], Lin [9], Csiszár [10], Ali and Silvey [12], Vajda [13], Shioya and Da-te [47] and others (see for example [5] and the references therein).

These measures have been applied in a variety of fields such as: anthropology [8], genetics [14], finance, economics, and political science [15], [16], [17], biology [18], the analysis of contingency tables [19], approximation of probability distributions [20], [21], signal processing [22], [23] and pattern recognition [24], [25]. A number of these measures of distance are specific cases of Csiszár f-divergence and so further exploration of this concept will have a flow on effect to other measures of distance and to areas in which they are applied.

Assume that a set Γ and the σ−finite measure μ are given. Consider the set of all probability densities on μ to be Ω := {p|p : Γ → ℝ, p (x) ≥ 0, ∫_{Γ} p (x) dμ (x) = 1}. The Kullback-Leibler divergence [2] is well known among the information divergences. It is defined as:

D_{KL} (p, q) := ∫_{Γ} p (x) log [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω, (1.1)

where log is to base 2.

In Information Theory and Statistics, various divergences are applied in addition to the Kullback-Leibler divergence. These are the: variation distance D_{v}, Hellinger distance D_{H} [40], χ^{2}−divergence D_{χ}^{2}, α−divergence D_{α}, Bhattacharyya distance D_{B} [41], Harmonic distance D_{Ha}, Jeffrey’s distance D_{J} [1], triangular discrimination D_{Δ} [35], etc... They are defined as follows:

D_{v} (p, q) := ∫_{Γ} |p (x) − q (x)| dμ (x) , p, q ∈ Ω; (1.2)

Date: October 25, 2000.

1991 Mathematics Subject Classification. Primary 94Xxx; Secondary 26D15.

Key words and phrases. Csiszár f−divergence, Taylor formula.

D_{H} (p, q) := ∫_{Γ} |\sqrt{p (x)} − \sqrt{q (x)}| dμ (x) , p, q ∈ Ω; (1.3)

D_{χ}^{2} (p, q) := ∫_{Γ} p (x) [(\frac{q (x)}{p (x)})^{2} − 1] dμ (x) , p, q ∈ Ω; (1.4)

D_{α} (p, q) := \frac{4}{1 − α^{2}} [1 − ∫_{Γ} [p (x)]^{\frac{1−α}{2}} [q (x)]^{\frac{1+α}{2}} dμ (x)] , p, q ∈ Ω; (1.5)

D_{B} (p, q) := ∫_{Γ} \sqrt{p (x) q (x)}dμ (x) , p, q ∈ Ω; (1.6)

D_{Ha} (p, q) := ∫_{Γ} \frac{2p (x) q (x)}{p (x) + q (x)}dμ (x) , p, q ∈ Ω; (1.7)

D_{J} (p, q) := ∫_{Γ} [p (x) − q (x)] ln [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω; (1.8)

D_{Δ} (p, q) := ∫_{Γ} \frac{[p (x) − q (x)]^{2}}{p (x) + q (x)} dμ (x) , p, q ∈ Ω. (1.9)

For other divergence measures, see the paper [5] by Kapur or the book on line [42] by Taneja. For a comprehensive collection of preprints available on line, see the

RGMIA web site http://rgmia.vu.edu.au/papersinfth.html

Csiszár f−divergence is defined as follows [10]

I_{f} (p, q) := ∫_{Γ} p (x) f [\frac{q (x)}{p (x)}] dμ (x) , p, q ∈ Ω, (1.10)

where f is convex on (0, ∞). It is assumed that f (u) is zero and strictly convex at u = 1. By appropriately defining this convex function, various divergences are derived. All the above distances (1.1) − (1.9), are particular instances of Csiszár f−divergence. There are also many others which are not in this class (see for example [5] or [42]). For the basic properties of Csiszár f−divergence see [43]-[45].

In [46], Lin and Wong (see also [9]) introduced the following divergence

D_{LW} (p, q) := ∫_{Γ} p (x) log [ \frac{p (x)}{\frac{1}{2}p (x) + \frac{1}{2}q (x)}] dμ (x) , p, q ∈ Ω. (1.11)

This can be represented as follows, using the Kullback-Leibler divergence:

D_{LW} (p, q) = D_{KL} (p, \frac{1}{2}p + \frac{1}{2}q) .

Lin and Wong have established the following inequalities

D_{LW} (p, q) ≤ \frac{1}{2}D_{KL} (p, q) ; (1.12)

D_{LW} (p, q) + D_{LW} (q, p) ≤ D_{v} (p, q) ≤ 2; (1.13)

D_{LW} (p, q) ≤ 1. (1.14)

In [47], Shioya and Da-te improved (1.12) − (1.14) by showing that

D_{LW} (p, q) ≤ \frac{1}{2}D_{v} (p, q) ≤ 1.

CSISZÁR f−DIVERGENCE 3

For classical and new results in comparing different kinds of divergence measures, see the papers [1]-[47] where further references are given.

2. Representation of Csiszár f−Divergence

We may state the following result which is a reformulation of Theorem 1 in [50]:

Theorem 1. Let {S_{n} (·, ·)}_{n∈ℕ} be a sequence of polynomials of two variables satisfying the condition:

\frac{∂S_{n} (t, x)}{∂t} = S_{n−1} (t, x) , S_{0} (t, x) = 1 for x, t ∈ ℝ and n ∈ ℕ. (2.1)

Then we have the identity

(2.2) f (x) = f (a) + ∑_{k=1}^{n} (−1)^{k+1} [S_{k} (x, x) f^{(k)} (x) − S_{k} (a, x) f^{(k)} (a)]
+R_{n} (f; a, x) ,

where

R_{n} (f; a, x) := (−1)^{n} ∫_{a}^{x} S_{n} (t, x) f^{(n+1)} (t) dt

and f : I → ℝ is such that f^{(n)} is absolutely continuous on I.

1. If in (2.2) we set S_{n} (t, x) = \frac{1}{n!} (t − x)^{n} (n ∈ ℕ), then we get the Taylor’s identity [48]:

f (x) = f (a) + ∑_{k=1}^{n} \frac{(x − a)^{k}}{k!} f^{(k)} (a) + \frac{1}{n!} ∫_{a}^{x} (x − t)^{n} f^{(n+1)} (t) dt, x ∈ I. (2.3)

2. If in (2.2) we set S_{n} (t, x) = \frac{1}{n!} (t − \frac{a+x}{2})^{n} , (n ∈ ℕ), then we have the identity [50]:

(2.4)  f (x) = f (a) + ∑_{k=1}^{n} \frac{(x − a)^{k}}{2^{k}k!} [f^{(k)} (a) + (−1)^{k+1} f^{(k)} (x)]

+\frac{(−1)^{n}}{n!} ∫_{a}^{x} (x − \frac{a + x}{2})^{n} f^{(n+1)} (t) dt, x ∈ I.

3. If in (2.2) we set S_{n} (t, x) = \frac{(x−a)^{n}}{n!} B_{n} (\frac{t−a}{x−a}), n ∈ ℕ, S_{0} (t, x) = 1, where B_{n} denotes the Bernoulli polynomials and B_{n} := B_{n} (0) are the Bernoulli numbers, then we have the following representation [50]

(2.5)  f (x) = f (a) + \frac{x − a}{2} [f′ (x) + f′ (a)]

− \begin{matrix}[ & \frac{n}{2} & ] \\ ∑ \\ k=1 \end{matrix} \frac{(x − a)^{2k}}{(2k)!} B_{2k} [f^{(2k)} (x) − f^{(2k)} (a)]

+ (−1)^{n} \frac{(x − a)^{n}}{n!} ∫_{a}^{x} B_{n} (\frac{t − a}{x − a}) f^{(n+1)} (t) dt.

4. If in (2.2) we set S_{n} (t, x) = \frac{(x−a)^{n}}{n!} E_{n} (\frac{t−a}{x−a}), n ∈ ℕ, S_{0} (t, x) = 1, where E_{n} (t) denotes the Euler polynomials and B_{n} are the Bernoulli numbers, then we have the representation [50]

(2.6)  f (x) = f (a) + 2 \begin{aligned}[ & \frac{n+1}{2} & ] \\ & ∑ \\ & k=1 \end{aligned} \frac{(x − a)^{2k−1} (4^{k} − 1)}{(2k)!}

×B_{2k} [f^{(2k−1)} (x) + f^{(2k−1)} (a)]

+ (−1)^{n} \frac{(x − a)^{n}}{n!} ∫_{a}^{x} E_{n} (\frac{t − a}{x − a}) f^{(n+1)} (t) dt.

We are able now to point out the following representation for the Csiszár f−divergence.

Theorem 2. Let {S_{n} (t, z)}_{n∈ℕ} and f : I ⊆ ℝ → ℝ be as in Theorem 1. If p, q ∈ Ω, then we have the representation

(2.7)  I_{f} (p, q) = f (1) + ∑_{k=1}^{n} (−1)^{k+1} [I_{S_{k}(·,·)f^{(k)}(·)} (p, q)
− I_{S_{k}(1,·)f^{(k)}(1)} (p, q)] + R_{f} (p, q) ,

where the remainder R_{f} (p, q) can be given by

R_{f} (p, q) = (−1)^{n} ∫_{Γ} q (x) (∫1^{\frac{p(x)}{q(x)}} S_{n} (t, \frac{p (x)}{q (x)}) f^{(n+1)} (t) dt) dμ (x) . (2.8)

Proof. From the representation (2.2), we may write that

(2.9) f (\frac{p (x)}{q (x)}) = f (1) + ∑_{k=1}^{n} (−1)^{k+1} [S_{k} (\frac{p (x)}{q (x)}, \frac{p (x)}{q (x)}) f^{(k)} (\frac{p (x)}{q (x)})

−S_{k} (1, \frac{p (x)}{q (x)}) f^{(k)} (1)]

+ (−1)^{n} ∫1^{\frac{p(x)}{q(x)}} S_{n} (t, \frac{p (x)}{q (x)}) f^{(n+1)} (t) dt

for all x ∈ Γ.

If we multiply (2.9) by q (x) ≥ 0 (x ∈ Γ), integrate on Γ and take into account that ∫_{Γ} q (x) dμ (x) = 1, then we obtain the desired representation (2.7).

The following particular cases are important in applications.

1. If we use the representation (2.3), we get

(2.10) I_{f} (p, q) = f (1) + ∑_{k=1}^{n} \frac{f^{(k)} (1)}{k!}D_{k} (p, q)

+\frac{1}{n!} ∫_{Γ} q (x) [∫1^{\frac{p(x)}{q(x)}} (\frac{p (x)}{q (x)} − t)^{n} f^{(n+1)} (t) dt] dμ (x)

for all p, q ∈ Ω, where

D_{k} (p, q) := ∫_{Γ} q^{−k+1} (x) (p (x) − q (x))^{k} dμ (x) , k = 1, …, n.

2. From the identity (2.4), we may get that

I_{f} (p, q) (2.11)

= f (1) + ∑_{k=1}^{n} \frac{f^{(k)} (1)}{2^{k}k!}D_{k} (p, q) + ∑_{k=1}^{n} \frac{(−1)^{k+1}}{2^{k}k!} I_{(·−1)^{k}f^{(k)}(·)} (p, q)

+\frac{1}{n!} ∫_{Γ} q (x) [∫1^{\frac{p(x)}{q(x)}} (1 − \frac{p (x) + q (x)}{q (x)})^{n} f^{(n+1)} (t) dt] dμ (x)

for all p, q ∈ Ω.

3. If we use the identity (2.5), we may obtain:

(2.12) I_{f} (p, q) = f (1) + ∫_{Γ} [\frac{p (x) − q (x)}{2}] f′ (\frac{p (x)}{q (x)}) dμ (x)

+ \begin{matrix}[ & \frac{n}{2} & ] \\ ∑ \\ k=1 \end{matrix} \frac{f^{(2k)} (1)}{(2k)!}D_{2k} (p, q) − \begin{matrix}[ & \frac{n}{2} & ] \\ ∑ \\ k=1 \end{matrix} \frac{B_{2k}}{(2k)!}I_{(·−1)^{2k}f^{(2k)}(·)} (p, q)

+\frac{(−1)^{n}}{n!} ∫_{Γ} q^{−n+1} (x) (p (x) − q (x))^{n}

× [∫1^{\frac{p(x)}{q(x)}} B_{n} ( \frac{t − 1}{\frac{p(x)}{q(x)} − 1}) f^{(n+1)} (t) dt] dμ (x)

for all p, q ∈ Ω.

4. Finally, by the use of identity (2.6), we may write that

(2.13)  I_{f} (p, q) = f (1) + 2 \begin{aligned}[ & \frac{n+1}{2} & ] \\ & ∑ \\ & k=1 \end{aligned} \frac{(4^{k} − 1) B_{2k}f^{(k)} (1)}{(2k)!}D_{2k−1} (p, q)

+2 \begin{aligned}[ & \frac{n+1}{2} & ] \\ & ∑ \\ & k=1 \end{aligned} \frac{B_{2k} (4^{k} − 1)}{(2k)!}I_{(·−1)^{2k−1}f^{(2k−1)}} (p, q)

+\frac{(−1)^{n}}{n!} ∫_{Γ} q^{−n+1} (x) (p (x) − q (x))^{n}

× [∫1^{\frac{p(x)}{q(x)}} E_{n} ( \frac{t − 1}{\frac{p(x)}{q(x)} − 1}) f^{(n+1)} (t) dt] dμ (x)

for all p, q ∈ Ω.

3. Bounds for the Remainder

For a, b ∈ ℝ, we denote

∥f∥_{[a,b],p} := |∫_{a}^{b} |f (t)|^{p} dt|^{\frac{1}{p}} if p ∈ [1, ∞)

and

∥f∥_{[a,b],∞} := ess sup_{{t∈[a,b] \atop (t∈[b,a])}} |f (t)| .

It is obvious that the order a < b or a > b is irrelevant in the definitions of the above Lebesgue p−norms.

The following general theorem involving the estimation of the remainder R_{f} (p, q) holds.

Theorem 3. Assume that {S_{n} (t, x)}_{n∈ℕ} and f are as in Theorem 1. If p, q ∈ Ω, then we have the inequality

|R_{f} (p, q)| ≤ \{ \begin{matrix}∫ _{Γ} q (x) ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],∞} × ∥S_{n} (·, \frac{p(x)}{q(x)})∥_{[1, \frac{p(x)}{q(x)}],1} dμ (x) , \\ ∫ _{Γ} q (x) ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],α} × ∥S_{n} (·, \frac{p(x)}{q(x)})∥_{[1, \frac{p(x)}{q(x)}],β} dμ (x) , \\ if α > 1, \frac{1}{α} + \frac{1}{β} = 1, \\ ∫ _{Γ} q (x) ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],1} × ∥S_{n} (·, \frac{p(x)}{q(x)})∥_{[1, \frac{p(x)}{q(x)}],∞} dμ (x) . \end{matrix} (3.1)

Proof. We have that

|R_{f} (p, q)| ≤ ∫_{Γ} q (x) |∫1^{\frac{p(x)}{q(x)}} S_{n} (t, \frac{p (x)}{q (x)}) f^{(n+1)} (t) dt| dμ (x) . (3.2)

Now, observe that

|∫1^{\frac{p(x)}{q(x)}} S_{n} (t, \frac{p (x)}{q (x)}) f^{(n+1)} (t) dt| (3.3)

≤ ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],∞} × ∥S_{n} (·, \frac{p (x)}{q (x)})∥_{[1, \frac{p(x)}{q(x)}],1}

and, by Hölder’s inequality for α > 1, \frac{1}{α} + \frac{1}{β} = 1,

|∫1^{\frac{p(x)}{q(x)}} S_{n} (t, \frac{p (x)}{q (x)}) f^{(n+1)} (t) dt| (3.4)

≤ |∫1^{\frac{p(x)}{q(x)}} |S_{n} (t, \frac{p (x)}{q (x)})|^{β} dt|^{\frac{1}{β}} × ||∫1^{\frac{p(x)}{q(x)}} f^{(n+1)} (t)|^{α} dt|^{\frac{1}{α}}

= ∥S_{n} (·, \frac{p (x)}{q (x)})∥_{[1, \frac{p(x)}{q(x)}],β} × ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],α} .

Finally,

|∫1^{\frac{p(x)}{q(x)}} S_{n} (t, \frac{p (x)}{q (x)}) f^{(n+1)} (t) dt| (3.5)

≤ ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],1} × ∥S_{n} (·, \frac{p (x)}{q (x)})∥_{[1, \frac{p(x)}{q(x)}],∞}

for all x ∈ Γ.

Using (3.2) and (3.3) – (3.5), we deduce (3.1).

CSISZÁR f−DIVERGENCE 7

Remark 1. If we assume that 0 ≤ r ≤ \frac{p(x)}{q(x)} ≤ R < ∞ for all x ∈ Γ, then obviously r ≤ 1 ≤ R and the right side of the inequality (3.1) may be upper bounded by

\{ \begin{matrix}∥f^{(n+1)}∥_{[r,R],∞} ∫ _{Γ} q (x) ∥S_{n} (·, \frac{p(x)}{q(x)})∥_{[1, \frac{p(x)}{q(x)}],1} dμ (x) , \\ ∥f^{(n+1)}∥_{[r,R],α} ∫ _{Γ} q (x) ∥S_{n} (·, \frac{p(x)}{q(x)})∥_{[1, \frac{p(x)}{q(x)}],β} dμ (x) , \\ ∥f^{(n+1)}∥_{[r,R],1} ∫ _{Γ} q (x) ∥S_{n} (·, \frac{p(x)}{q(x)})∥_{[1, \frac{p(x)}{q(x)}],∞} dμ (x) , \end{matrix}

If we choose some particular instances of polynomials S_{n} (·, ·) we may compute the Lebesgue norm ∥S_{n} (·, \frac{p(x)}{q(x)})∥_{s}, s ∈ [1, ∞], obtaining more explicit bounds for the remainder R_{f} (p, q).

1. If we choose S_{n} (t, z) = \frac{1}{n!} (t − z)^{n}, then

∥S_{n} (·, z)∥_{[1,z],1} = \frac{1}{n!} |∫_{1}^{z} |t − z|^{n} dt| = \frac{1}{(n + 1)!} |z − 1|^{n+1} ,

∥S_{n} (·, z)∥_{[1,z],α} = \frac{1}{n!} |∫_{1}^{z} |t − z|^{αn} dt|^{\frac{1}{α}}
= \frac{1}{n!} [\frac{|z − 1|^{αn+1}}{αn + 1} ]^{\frac{1}{α}} = \frac{|z − 1|^{n+^{\frac{1}{α}}}}{n! (αn + 1)^{\frac{1}{α}}} ,

and

∥S_{n} (·, z)∥_{[1,z],∞} = \frac{1}{n!} |z − 1|^{n} .

Consequently, we may state the following corollary which is useful in practice. Corollary 1. Let f be as in Theorem 1. Then, for p, q ∈,

I_{f} (p, q) = f (1) + ∑_{k=1}^{n} \frac{f^{(k)} (1)}{k!}D_{k} (p, q) + R_{f} (p, q) , (3.6)

where

D_{k} (p, q) := ∫_{Γ} q^{−k+1} (x) (p (x) − q (x))^{k} dμ (x)

and the remainder R_{f} (p, q) satisfies the bound |R_{f} (p, q)| (3.7)

≤ \{ \begin{matrix}\frac{1}{(n + 1)!} ∫ _{Γ} |p (x) − q (x)|^{n+1} [q (x)]^{−n} ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],∞} dμ (x) , \\ \frac{1}{n! (αn + 1)^{\frac{1}{α}}} ∫ _{Γ} |p (x) − q (x)|^{n+ \frac{1}{α}} [q (x)]^{−n− \frac{1}{α} +1} ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],β} dμ (x) , \\ if α > 1, \frac{1}{α} + \frac{1}{β} = 1, \\ \frac{1}{n!} ∫ _{Γ} |p (x) − q (x)|^{n} [q (x)]^{−n+1} ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],1} dμ (x) . \end{matrix}

Moreover, if 0 ≤ r ≤ \frac{p(x)}{q(x)} ≤ R < ∞ for all x ∈ Γ, then the right hand side of (3.7) can be upper bounded by

\{ \begin{matrix}\frac{∥f^{(n+1)}∥_{[r,R],∞}}{(n + 1)!} ∫ _{Γ} |p (x) − q (x)|^{n+1} [q (x)]^{−n} dμ (x) , \\ \frac{∥f^{(n+1)}∥_{[r,R],β}}{n! (αn + 1)^{\frac{1}{α}}} ∫ _{Γ} |p (x) − q (x)|^{n+ \frac{1}{α}} [q (x)]^{−n− \frac{1}{α} +1} dμ (x) , \\ if α > 1, \frac{1}{α} + \frac{1}{β} = 1, \\ \frac{∥f^{(n+1)}∥_{[r,R],1}}{n!} ∫ _{Γ} |p (x) − q (x)|^{n} [q (x)]^{−n+1} dμ (x) , \end{matrix} (3.8)

≤ \{ \begin{matrix}\frac{∥f^{(n+1)}∥_{[r,R],∞} (R − r)^{n+1}}{(n + 1)!} , \\ \frac{∥f^{(n+1)}∥_{[r,R],β} (R − r)^{n+ \frac{1}{α}}}{n! (αn + 1)^{\frac{1}{α}}} , \\ \frac{∥f^{(n+1)}∥_{[r,R],1} (R − r)^{n}}{n!} . \end{matrix}

2. If we choose S_{n} (t, z) = \frac{1}{n!} (t − \frac{1+z}{2} )^{n}, then

∥S_{n} (·, z)∥_{[1,z],1} = \frac{1}{n!} |∫_{1}^{z} |t − \frac{1 + z}{2} |^{n} dt| .

If we assume that z ≥ 1, then

∫_{1}^{z} |t − \frac{1 + z}{2} |^{n} dt = ∫1^{\frac{1+z}{2}} (\frac{1 + z}{2} − t)^{n} dt + ∫ z_{\frac{1+z}{2}} (t − \frac{1 + z}{2} )^{n} dt
= \frac{1}{n + 1} (\frac{z − 1}{2})^{n+1} + \frac{1}{n + 1} (\frac{z − 1}{2})^{n+1}
= \frac{(z − 1)^{n+1}}{(n + 1) 2^{n}} .

If we assume that z ≤ 1, then

∫_{z}^{1} |t − \frac{1 + z}{2} |^{n} dt = \frac{(1 − z)^{n+1}}{(n + 1) 2^{n}}

and thus, we may state that

∥S_{n} (·, z)∥_{[1,z],1} = \frac{1}{(n + 1)!} · \frac{|z − 1|^{n+1}}{2^{n}} .

CSISZÁR f−DIVERGENCE 9

Similarly, we have

∥S_{n} (·, z)∥_{[1,z],α} = \frac{1}{n!} |∫_{1}^{z} |t − \frac{1 + z}{2} |^{nα} dt|^{\frac{1}{α}} = \frac{1}{n!} [\frac{|z − 1|^{nα+1}}{(nα + 1) 2^{nα}} ]^{\frac{1}{α}}
= \frac{1}{n!} · \frac{|z − 1|^{n+ \frac{1}{α}}}{(nα + 1)^{\frac{1}{α}} 2^{n}} , α ≥ 1

and

∥S_{n} (·, z)∥_{[1,z],∞} = \frac{1}{n!} · \frac{|z − 1|^{n}}{2^{n}} .

Consequently, we may state the following corollary which is useful in practice. Corollary 2. Let f be as in Theorem 1. Then, for p, q ∈ Ω,

(3.9)  I_{f} (p, q) = f (1) + ∑_{k=1}^{n} \frac{f^{(k)} (1)}{2^{k}k!}D_{k} (p, q)

+ ∑_{k=1}^{n} \frac{(−1)^{k+1}}{2^{k}k!} I_{(·−1)^{k}f^{(k)}(·)} (p, q) + R̃_{f} (p, q)

and the remainder R̃_{f} (p, q) satisfies the bound |R̃_{f} (p, q)| (3.10)

≤ \{ \begin{matrix}\frac{1}{(n + 1)!2^{n}} ∫ _{Γ} [q (x)]^{−n} |p (x) − q (x)|^{n+1} ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],∞} dμ (x) , \\ \frac{1}{n!2^{n} (αn + 1)^{\frac{1}{α}}} ∫ _{Γ} [q (x)]^{−n+1− \frac{1}{α}} |p (x) − q (x)|^{n+ \frac{1}{α}} ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],β} dμ (x) , \\ if α > 1, \frac{1}{α} + \frac{1}{β} = 1, \\ \frac{1}{n!2^{n}} ∫ _{Γ} [q (x)]^{−n+1} |p (x) − q (x)|^{n} ∥f^{(n+1)}∥_{[1, \frac{p(x)}{q(x)}],1} dμ (x) . \end{matrix}

Moreover, if 0 ≤ r ≤ \frac{p(x)}{q(x)} ≤ R < ∞ for all x ∈ Γ, then the right hand side of (3.10) can be upper bounded by:

\{ \begin{matrix}\frac{∥f^{(n+1)}∥_{[r,R],∞}}{(n + 1)!2^{n}} ∫ _{Γ} |p (x) − q (x)|^{n+1} [q (x)]^{−n} dμ (x) , \\ \frac{∥f^{(n+1)}∥_{[r,R],β}}{2^{n}n! (αn + 1)^{\frac{1}{α}}} ∫ _{Γ} |p (x) − q (x)|^{n+ \frac{1}{α}} [q (x)]^{−n− \frac{1}{α} +1} dμ (x) , \\ if α > 1, \frac{1}{α} + \frac{1}{β} = 1, \\ \frac{∥f^{(n+1)}∥_{[r,R],1}}{2^{n}n!} ∫ _{Γ} |p (x) − q (x)|^{n} [q (x)]^{−n+1} dμ (x) , \end{matrix} (3.11)

≤ \{ \begin{matrix}\frac{∥f^{(n+1)}∥_{[r,R],∞} (R − r)^{n+1}}{(n + 1)!2^{n}} , \\ \frac{∥f^{(n+1)}∥_{[r,R],β} (R − r)^{n+ \frac{1}{α}}}{n! (αn + 1)^{\frac{1}{α}} 2^{n}} , α > 1, \frac{1}{α} + \frac{1}{β} = 1, \\ \frac{∥f^{(n+1)}∥_{[r,R],1} (R − r)^{n}}{n!} . \end{matrix}

References

[1] H. JEFFREYS, An invariant form for the prior probability in estimating problems, Proc. Roy. Soc. London, 186 A (1946), 453-461.

[2] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Ann. Math. Stat., 22 (1951), 79-86.

[3] A. RÉNYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Stat. and Prob., University of California Press, 1 (1961), 547-561.

[4] J.H. HAVRDA and F. CHARVAT, Quantification method classification process: concept of structural α-entropy, Kybernetika, 3 (1967), 30-35.

[5] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), 1-16.

[6] B.D. SHARMA and D.P. MITTAL, New non-additive measures of relative information, Journ. Comb. Inf. Sys. Sci., 2 (4)(1977), 122-132.

[7] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy function, IEEE Trans. Inf. Th., 28 (3) (1982), 489-495.

[8] C.R. RAO, Diversity and dissimilarity coefficients: a unified approach, Theoretic Population Biology, 21 (1982), 24-43.

[9] J. LIN, Divergence measures based on the Shannon entropy, IEEE Trans. Inf. Th., 37 (1) (1991), 145-151.

[10] I. CSISZÁR, Information-type measures of difference of probability distributions and indirect observations, Studia Math. Hungarica, 2 (1967), 299-318.

[11] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[12] S.M. ALI and S.D. SILVEY, A general class of coefficients of divergence of one distribution from another, J. Roy. Statist. Soc. Sec B, 28 (1966), 131-142.

[13] I. VAJDA, Theory of Statistical Inference and Information, Dordrecht-Boston, Kluwer Academic Publishers, 1989.

[14] M. MEI, The theory of genetic distance and evaluation of human races, Japan J. Human Genetics, 23 (1978), 341-369.

[15] A. SEN, On Economic Inequality, Oxford University Press, London 1973.

[16] H. THEIL, Economics and Information Theory, North-Holland, Amsterdam, 1967.

[17] H. THEIL, Statistical Decomposition Analysis, North-Holland, Amsterdam, 1972.

[18] E.C. PIELOU, Ecological Diversity, Wiley, New York, 1975.

[19] D.V. GOKHALE and S. KULLBACK, Information in Contingency Tables, New York, Merul Dekker, 1978.

[20] C.K. CHOW and C.N. LIN, Approximating discrete probability distributions with dependence trees, IEEE Trans. Inf. Th., 14 (3) (1968), 462-467.

[21] D. KAZAKOS and T. COTSIDAS, A decision theory approach to the approximation of discrete probability densities, IEEE Trans. Perform. Anal. Machine Intell., 1 (1980), 61-67.

[22] T.T. KADOTA and L.A. SHEPP, On the best finite set of linear observables for discriminating two Gaussian signals, IEEE Trans. Inf. Th., 13 (1967), 288-294.

[23] T. KAILATH, The divergence and Bhattacharyya distance measures in signal selection, IEEE Trans. Comm. Technology., Vol COM-15 (1967), 52-60.

[24] M. BETH BASSAT, f-entropies, probability of error and feature selection, Inform. Control, 39 (1978), 227-242.

[25] C.H. CHEN, Statistical Pattern Recognition, Rocelle Park, New York, Hoyderc Book Co., 1973.

[26] V.A. VOLKONSKI and J. A. ROZANOV, Some limit theorems for random function -I, (English Trans.), Theory Prob. Appl., ( USSR), 4 (1959), 178-197.

[27] M.S. PINSKER, Information and Information Stability of Random variables and processes, (in Russian), Moscow: Izv. Akad. Nouk, 1960.

[28] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[29] H.P. McKEAN, JR., Speed of approach to equilibrium for Koc’s caricature of a Maximilian gas, Arch. Ration. Mech. Anal., 21 (1966), 343-367.

[30] J.H.B. KEMPERMAN, On the optimum note of transmitting information, Ann. Math. Statist., 40 (1969), 2158-2177.

[31] S. KULLBACK, A lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 13 (1967), 126-127.

[32] S. KULLBACK, Correction to a lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[33] I. VAJDA, Note on discrimination information and variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[34] G.T. TOUSSAINT, Sharper lower bounds for discrimination in terms of variation, IEEE Trans. Inf. Th., 21 (1975), 99-100.

[35] F. TOPSOE, Some inequalities for information divergence and related measures of discrimination, Res. Rep. Coll., RGMIA, 2 (1) (1999), 85-98.

[36] L. LECAM, Asymptotic Methods in Statistical Decision Theory, New York: Springer, 1986.

[37] D. DACUNHA-CASTELLE, Ecole d’ete de Probability de Saint-Flour, III-1977, Berlin, Heidelberg: Springer 1978.

[38] C. KRAFT, Some conditions for consistency and uniform consistency of statistical procedures, Univ. of California Pub. in Statistics, 1 (1955), 125-142.

[39] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[40] E. HELLINGER, Neue Bergrüirdung du Theorie quadratisher Formerus von uneudlichvieleu Veränderlicher, J. für reine and Augeur. Math., 36 (1909), 210-271.

[41] A. BHATTACHARYYA, On a measure of divergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc., 35 (1943), 99-109.

[42] I. J. TANEJA, Generalised Information Measures and their Applications

    (http://www.mtm.ufsc.br/~taneja/bhtml/bhtml.html).

[43] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[44] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[45] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[46] J. LIN and S.K.M. WONG, A new directed divergence measure and its characterization, Int. J. General Systems, 17 (1990), 73-81.

[47] H. SHIOYA and T. DA-TE, A generalisation of Lin divergence and the derivative of a new information divergence, Elec. and Comm. in Japan, 78 (7) (1995), 37-40.

[48] S.S. DRAGOMIR, New estimation of the remainder in Taylor’s formula using Grüss’ type inequalities and applications, Math. Ineq. and Appl., 2 (2) (1999), 183-193.

[49] S.S. DRAGOMIR, An improvement of the remainder estimate in the generalised Taylor’s formula, RGMIA Res. Rep. Coll., 3 (2000), No. 1, Article 1.

[50] M. MATIĆ, J. PEČARIĆ and N. UJEVIĆ, On new estimation of the remainder in generalised Taylor’s formula, Math. Ineq. and Appl., 2 (3) (1999), 343-361.

School of Communications and Informatics, Victoria University of Technology, PO Box 14428, Melbourne City MC, Victoria 8001, Australia.

   E-mail address: sever@matilda.vu.edu.au
   URL: http://rgmia.vu.edu.au/SSDragomirWeb.html

Royal Australian Airforce, ARDU, PO Box 1500, Salisbury SA 5108, Australia.

   E-mail address: vgluscev@spam.adelaide.edu.au