OTHER INEQUALITIES FOR CSISZÁR DIVERGENCE AND APPLICATIONS

S.S. DRAGOMIR

Abstract. In this paper we point out some new inequalities for Csiszár f−divergence and apply them for particular instances of distances between two probability distributions.

1. Introduction

One of the important issues in many applications of Probability Theory is finding an appropriate measure of distance (or difference or discrimination ) between two probability distributions. A number of divergence measures for this purpose have been proposed and extensively studied by Jeffreys [1], Kullback and Leibler [2], Rényi [3], Havrda and Charvat [4], Kapur [5], Sharma and Mittal [6], Burbea and Rao [7], Rao [8], Lin [9], Csiszár [10], Ali and Silvey [12], Vajda [13], Shioya and Da-te [40] and others (see for example [5] and the references therein).

These measures have been applied in a variety of fields such as: anthropology [8], genetics [14], finance, economics, and political science [15], [16], [17], biology [18], the analysis of contingency tables [19], approximation of probability distributions [20], [21], signal processing [22], [23] and pattern recognition [24], [25]. A number of these measures of distance are specific cases of Csiszár f-divergence and so further exploration of this concept will have a flow on effect to other measures of distance and to areas in which they are applied.

Assume that a set χ and the σ−finite measure μ are given. Consider the set of all probability densities on μ to be Ω := {p|p : χ → ℝ, p (x) ≥ 0, ∫_{χ} p (x) dμ (x) = 1}. The Kullback-Leibler divergence [2] is well known among the information divergences. It is defined as:

D_{KL} (p, q) := ∫_{χ} p (x) log [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω, (1.1)

where log is to base 2.

In Information Theory and Statistics, various divergences are applied in addition to the Kullback-Leibler divergence. These are the: variation distance D_{v}, Hellinger distance D_{H} [1], χ^{2}−divergence D_{χ}^{2}, α−divergence D_{α}, Bhattacharyya distance D_{B} [2], Harmonic distance D_{Ha}, Jeffreys distance D_{J} [1], triangular discrimination D_{Δ} [35], etc... They are defined as follows:

D_{v} (p, q) := ∫_{χ} |p (x) − q (x)| dμ (x) , p, q ∈ Ω; (1.2)

Date: June 20, 2000.

1991 Mathematics Subject Classification. Primary 26D15; Secondary 94A99.

Key words and phrases. Csiszár Divergence.

D_{H} (p, q) := ∫_{χ} [\sqrt{p (x)} − \sqrt{q (x)}]^{2} dμ (x) , p, q ∈ Ω; (1.3)

D_{χ}^{2} (p, q) := ∫_{χ} p (x) [(\frac{q (x)}{p (x)})^{2} − 1] dμ (x) , p, q ∈ Ω; (1.4)

D_{α} (p, q) := \frac{4}{1 − α^{2}} [1 − ∫_{χ} [p (x)]^{\frac{1−α}{2}} [q (x)]^{\frac{1+α}{2}} dμ (x)] , p, q ∈ Ω; (1.5)

D_{B} (p, q) := ∫_{χ} \sqrt{p (x) q (x)}dμ (x) , p, q ∈ Ω; (1.6)

D_{Ha} (p, q) := ∫_{χ} \frac{2p (x) q (x)}{p (x) + q (x)}dμ (x) , p, q ∈ Ω; (1.7)

D_{J} (p, q) := ∫_{χ} [p (x) − q (x)] ln [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω; (1.8)

D_{Δ} (p, q) := ∫_{χ} \frac{[p (x) − q (x)]^{2}}{p (x) + q (x)} dμ (x) , p, q ∈ Ω. (1.9)

For other divergence measures, see the paper [5] by Kapur or the book on line [6] by Taneja. For a comprehensive collection of preprints available on line, see the

RGMIA web site http://rgmia.vu.edu.au/papersinfth.html

Csiszár f−divergence is defined as follows [10]

D_{f} (p, q) := ∫_{χ} p (x) f [\frac{q (x)}{p (x)}] dμ (x) , p, q ∈ Ω, (1.10)

where f is convex on (0, ∞). It is assumed that f (u) is zero and strictly convex at u = 1. By appropriately defining this convex function, various divergences are derived. All the above distances (1.1) − (1.9), are particular instances of Csiszár f−divergence. There are also many others which are not in this class (see for example [5] or [6]). For the basic properties of Csiszár f−divergence see [7]-[10].

2. The Results

We start with the following result.

Theorem 1. Let ϕ : [0, ∞) → ℝ be a convex mapping on the interval [r, R] ⊂ [0, ∞) with r ≤ 1 ≤ R. If p, q ∈ Ω and r ≤ \frac{p(y)}{q(y)} ≤ R for all y ∈ χ, then we have the inequality

I_{ϕ} (p, q) ≤ \frac{R − 1}{R − r} · ϕ (r) + \frac{1 − r}{R − r} · ϕ (R) . (2.1)

Proof. As ϕ is convex on [r, R], we may write that

ϕ (tr + (1 − t) R) ≤ tϕ (r) + (1 − t) ϕ (R) (2.2)

for all t ∈ [0, 1].

Choose t = \frac{R−x}{R−r}, x ∈ [r, R]. Then 1 − t = \frac{x−r}{R−r} and from (2.2) we deduce (see also [46, p. 98])

ϕ (x) ≤ \frac{R − x}{R − r} · ϕ (r) + \frac{x − r}{R − r} · ϕ (R) (2.3)

for all x ∈ [r, R], as a simple calculation shows that \frac{R−x}{R−r} · r + \frac{x−r}{R−r} · R = x. Put in (2.3) x = \frac{p(y)}{q(y)}, y ∈ χ, to get

ϕ (\frac{p (y)}{q (y)}) ≤ \frac{R − \frac{p(y)}{q(y)}}{R − r} · ϕ (r) + \frac{\frac{p(y)}{q(y)} − r}{R − r} · ϕ (R) (2.4)

for all y ∈ χ.

If we multiply (2.4) by q (y) ≥ 0, integrate on χ and take into account that

∫_{χ} p (y) dμ (y) = ∫_{χ} q (y) dμ (y) = 1

then by (2.4) we obtain (2.1).

The following result also holds.

Theorem 2. Let ϕ : [0, ∞) → ℝ be differentiable convex on [r, R] and p, q be as in Theorem 1. Then we have the inequality:

(2.5)  0 ≤ \frac{R − 1}{R − r} · ϕ (r) + \frac{1 − r}{R − r} · ϕ (R) − I_{ϕ} (p, q)
≤ \frac{ϕ′ (R) − ϕ′ (r)}{R − r} · [(R − 1) (1 − r) − D_{χ}^{2} (p, q)]
≤ \frac{1}{4} (R − r) [ϕ′ (R) − ϕ′ (r)] ,

where D_{χ}^{2} (·, ·) is the chi-square divergence.

Proof. Since the mapping ϕ is differentiable convex, we can write

ϕ (u) − ϕ (v) ≥ ϕ′ (v) (u − v) (2.6)

for all u, v ∈ (r, R).

Now, assume that α, β ≥ 0 and α + β > 0. Then, by (2.6), we have

(2.7)  ϕ (\frac{αa + βb}{α + β}) − ϕ (a) ≥ ϕ′ (a) (\frac{αa + βb}{α + β} − a)
= \frac{β}{α + β} · ϕ′ (a) (b − a)

and

(2.8)  ϕ (\frac{αa + βb}{α + β}) − ϕ (b) ≥ ϕ′ (b) (\frac{αa + βb}{α + β} − b)
= −\frac{α}{α + β} · ϕ′ (b) (b − a) .

Now, if we multiply (2.7) by α and (2.8) by β and add the obtained results, we get

(α + β) ϕ (\frac{αa + βb}{α + β}) − αϕ (a) − βϕ (b) ≥ \frac{αβ}{α + β} (b − a) (ϕ′ (a) − ϕ′ (b))

which is equivalent to:

(2.9)  0 ≤ \frac{αϕ (a) + βϕ (b)}{α + β} − ϕ (\frac{αa + βb}{α + β})
≤ \begin{matrix}α & β \\ (α + β) & 2 \end{matrix} (ϕ′ (b) − ϕ′ (a)) (b − a) .

Now, if in (2.9) we choose α = R − x, β = x − r, a = r, b = R, then we obtain

(2.10)  0 ≤ \frac{(R − x) ϕ (r) + (x − r) ϕ (R)}{R − r} − ϕ (x)
≤ \frac{(R − x) (x − r)}{R − r} (ϕ′ (R) − ϕ′ (r)) .

If in (2.10), we choose x = \frac{p(y)}{q(y)} and then multiply with q (y) we get

\frac{(Rq (y) − p (y)) ϕ (r) + (p (y) − rq (y)) ϕ (R)}{R − r} − q (y) ϕ (\frac{p (y)}{q (y)}) (2.11)

≤ \frac{(Rq (y) − p (y)) (p (y) − rq (y))}{(R − r) q (y)} (ϕ′ (R) − ϕ′ (r))

for all y ∈ χ.

If we integrate (2.11) on χ and take into consideration that

∫_{χ} p (y) dμ (y) = ∫_{χ} q (y) dμ (y) = 1,

we get

\frac{(R − 1) ϕ (r) + (1 − r) ϕ (R)}{R − r} − I_{ϕ} (p, q) (2.12)

≤ \frac{(ϕ′ (R) − ϕ′ (r))}{R − r} ∫_{χ} \frac{(Rq (y) − p (y)) (p (y) − rq (y))}{q (y)}dμ (y) .

However,

0 ≤ ∫_{χ} \frac{(Rq (y) − p (y)) (p (y) − rq (y))}{q (y)}dμ (y)
= R − ∫_{χ} \frac{p^{2} (y)}{q (y)}dμ (y) − rR + r = R + r − rR − 1 − D_{χ}^{2} (p, q)
= (R − 1) (1 − r) − D_{χ}^{2} (p, q) .

As

(R − 1) (1 − r) ≤ \frac{1}{4} (R − r)^{2} and D_{χ}^{2} (p, q) ≥ 0,

the last inequality is obvious.

The following results also holds.

Theorem 3. Assume that the function Ψ : [0, ∞) → ℝ is twice differentiable on [r, R] and

m ≤ Ψ′′ (t) ≤ M for all t ∈ [r, R] . (2.13)

If the probability distributions p, q ∈ Ω satisfy the conditions of Theorem 1, then we have the inequality:

\frac{1}{2}m [(R − 1) (1 − r) − D_{χ}^{2} (p, q)] (2.14)

≤ \frac{R − 1}{R − r} · Ψ (r) + \frac{1 − r}{R − r} · Ψ (R) − I_{Ψ} (p, q)
≤ \frac{1}{2}M [(R − 1) (1 − r) − D_{χ}^{2} (p, q)] .

Proof. Define the function Φ_{m} : [0, ∞) → ℝ, Φ_{m} (t) = Ψ (t) − \frac{1}{2}mt^{2}. Then Φ_{m} is twice differentiable and Φ′′_{m} (t) = Ψ′′ (t) − m ≥ 0, t ∈ [r, R], which shows that Φ_{m} is convex on [r, R].

If we write the inequality (2.1) for the convex mapping Φ_{m}, we obtain

I_{Ψ− \frac{1}{2} m(·)^{2}} (p, q) ≤ \frac{R − 1}{R − r} [Ψ (r) − \frac{1}{2}mr^{2}] + \frac{1 − r}{R − r} [Ψ (R) − \frac{1}{2}mR^{2}] . (2.15)

However,

I_{Ψ− \frac{1}{2} m(·)^{2}} (p, q)
= I_{Ψ} (p, q) − \frac{1}{2}m [∫_{χ} \frac{p^{2} (y)}{q (y)}dμ (y) − 1 + 1]
= I_{Ψ} (p, q) − \frac{1}{2}mD_{χ}^{2} (p, q) − \frac{1}{2}m

and then, by (2.15), we can get

\frac{R − 1}{R − r} · Ψ (r) + \frac{1 − r}{R − r} · Ψ (R) − I_{Ψ} (p, q) (2.16)

≥ \frac{1}{2}mR^{2} · \frac{(1 − r)}{R − r} + \frac{1}{2}mr^{2} · \frac{(R − 1)}{R − r} − \frac{1}{2}mD_{χ}^{2} (p, q) − \frac{1}{2}m

Nonetheless, the right hand side of (2.16) is

\frac{1}{2}m [(R − 1) (1 − r) − D_{χ}^{2} (p, q)]

and the first inequality in (2.14) is obtained.

The second inequality follows by a similar argument applied for the mapping Φ_{m} (t) := \frac{1}{2}Mt^{2} − Ψ (t). We omit the details.

Corollary 1. With the assumptions in Theorem 3, and if m ≥ 0, then

(2.17)  0 ≤ \frac{1}{2}m [(R − 1) (1 − r) − D_{χ}^{2} (p, q)]
≤ \frac{R − 1}{R − r} · Ψ (r) + \frac{1 − r}{R − r} · Ψ (R) − I_{ϕ} (p, q) .

Proof. We only have to prove the fact that

D_{χ}^{2} (p, q) ≤ (R − 1) (1 − r) , (2.18)

which follows by the fact that (see the proof of Theorem 2)

0 ≤ ∫_{χ} \frac{(Rq (y) − p (y)) (p (y) − rq (y))}{q (y)}dμ (y)
= (R − 1) (1 − r) − D_{χ}^{2} (p, q) .

3. Applications for Particular Divergences

Before we point out some applications of the above results, we would like to recall the following special means:

L (α, β) := \{ \begin{matrix}β & if α = β; \\ \frac{β−α}{ln β−ln α} & if β ≠ α, α, β > 0 (logarithmic mean) \end{matrix}

and

I (α, β) := \{ \begin{matrix}β & if α = β; \\ \frac{1}{e} (\begin{matrix}β & ^{β} \\ α & ^{α} \end{matrix} )^{\frac{1}{β−α}} & if β ≠ α, (identric mean). \end{matrix}

1. Kullback-Leibler Divergence. Consider the convex mapping ϕ : (0, ∞) → ℝ, ϕ (t) = t ln t. Then

I_{ϕ} (p, q) = ∫_{χ} p (x) ln [\frac{p (x)}{q (x)}] dμ (x) = D (p, q) ,

where D (p, q) is the Kullback-Leibler distance.

Proposition 1. Let p, q ∈ Ω with the property that:

r ≤ \frac{p (y)}{q (y)} ≤ R for all y ∈ χ. (3.1)

Then we have the inequality

D (p, q) ≤ ln I (r, R) − \frac{G^{2} (r, R)}{L (r, R)} + 1, (3.2)

where I (·, ·) is the identric mean, L (·, ·) is the logarithmic mean and G (·, ·) is the usual geometric mean.

Proof. We apply Theorem 1 for ϕ (t) = t ln t to get

D (p, q) ≤ \frac{R − 1}{R − r}r ln r + \frac{1 − r}{R − r} R ln R
= \frac{R ln R − r ln r}{R − r} − rR · \frac{ln R − ln r}{R − r}
= ln I (r, R) + 1 − \frac{G^{2} (r, R)}{L (r, R)}

and the inequality (3.2) is proved.

Proposition 2. With the assumptions of Proposition 1, we have

(3.3)  0 ≤ ln I (r, R) − \frac{G^{2} (r, R)}{L (r, R)} + 1 − D (p, q)
≤ \frac{(R − 1) (1 − r) − D_{χ}^{2} (p, q)}{L (r, R)}.

The proof follows by Theorem 2 applied for ϕ (t) = t ln t, and taking into account that

\frac{ϕ′ (R) − ϕ′ (r)}{R − r} = \frac{1}{L (r, R)}.

Using Theorem 3, we may be able to improve the inequality (3.3) as follows.

Proposition 3. Let p, q ∈ Ω satisfy the condition (3.1). Then we have the inequality:

\frac{1}{2R} [(R − 1) (1 − r) − D_{χ}^{2} (p, q)] (3.4)

≤ ln I (r, R) − \frac{G^{2} (r, R)}{L (r, R)} + 1 − D (p, q)
≤ \frac{1}{2r} [(R − 1) (1 − r) − D_{χ}^{2} (p, q)] .

Proof. We have ϕ′′ (t) = \frac{1}{t}, t ∈ [r, R] and then

\frac{1}{R} ≤ ϕ′′ (t) ≤ \frac{1}{r}, t ∈ [r, R] .

Applying Theorem 3 for ϕ (t) = t ln t, we obtain (3.4).

Now, assume that ϕ (t) = − ln t, which is a convex mapping as well.

We have

I_{ϕ} (p, q) = − ∫_{χ} q (y) ln [\frac{p (y)}{q (y)}] dμ (y)
= ∫_{χ} q (y) ln [\frac{q (y)}{p (y)}] dμ (y) = D (q, p) .

Using Theorem 1, we may state the following proposition.

Proposition 4. Let p, q ∈ Ω with the property that (3.1) holds. Then we have the inequality:

D (q, p) ≤ ln I (\frac{1}{r}, \frac{1}{R}) − \frac{1}{L (r, R)} + 1. (3.5)

Proof. Applying the inequality (2.1) for ϕ (t) = − ln t, we may write that

D (q, p)
≤ \frac{(R − 1) (− ln r) + (1 − r) (− ln R)}{R − r}
= \frac{r ln R − R ln r}{R − r} − \frac{ln R − ln r}{R − r} = \frac{rR (\frac{1}{R} ln R − \frac{1}{r} ln r)}{R − r} − \frac{1}{L (r, R)}
= \frac{\frac{1}{r} ln \frac{1}{r} − \frac{1}{R} ln \frac{1}{R}}{\frac{1}{r} − \frac{1}{R}} − \frac{1}{L (r, R)} = ln I (\frac{1}{r}, \frac{1}{R}) + 1 − \frac{1}{L (r, R)}

and the inequality (3.5) is proved.

Proposition 5. Let p, q be as in Proposition 1. Then

(3.6)  0 ≤ ln I (\frac{1}{r}, \frac{1}{R}) − \frac{1}{L (r, R)} + 1 − D (q, p)
≤ \frac{1}{G^{2} (r, R)} [(R − 1) (1 − r) − D_{χ}^{2} (p, q)] .

The proof follows by Theorem 2 applied for the function ϕ (t) = − ln t, and taking into account that

\frac{ϕ′ (R) − ϕ′ (r)}{R − r} = \frac{1}{rR} = \frac{1}{G^{2} (r, R)}.

The inequality (3.6) can be improved as follows.

Proposition 6. Let p, q be as in Proposition 1. Then

\frac{1}{2R^{2}} [(R − 1) (1 − r) − D_{χ}^{2} (p, q)] (3.7)

≤ ln I (\frac{1}{r}, \frac{1}{R}) − \frac{1}{L (r, R)} + 1 − D (q, p)
≤ \frac{1}{2r^{2}} [(R − 1) (1 − r) − D_{χ}^{2} (p, q)] .

The proof is obvious by Theorem 3, taking into account that ϕ′′ (t) = \frac{1}{t^{2}} and \frac{1}{R^{2}} ≤ ϕ′′ (t) ≤ \frac{1}{r^{2}} for all t ∈ [r, R].

2. Hellinger discrimination. Consider the convex mapping ϕ : [0, ∞) → ℝ, ϕ (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}. Then

I_{ϕ} (p, q) = \frac{1}{2} ∫_{χ} q (x) (\sqrt{\frac{p (x)}{q (x)}} − 1)^{2} dμ (x)

= \frac{1}{2} ∫_{χ} (\sqrt{p (x)} − \sqrt{q (x)})^{2} dμ (x) = h^{2} (p, q) ,

where h^{2} (p, q) is the Hellinger discrimination.

Proposition 7. With the assumptions of Proposition 1, we have

h^{2} (p, q) ≤ \frac{(\sqrt{R} − 1) (1 − \sqrt{r})}{\sqrt{R} + \sqrt{r}}. (3.8)

Proof. We apply Theorem 1 for ϕ (t) = \frac{1}{2} (\sqrt{t} − 1)^{2} to get

h^{2} (p, q)

≤ \frac{(R − 1) \frac{1}{2} (\sqrt{r} − 1)^{2} + (1 − r) \frac{1}{2} (\sqrt{R} − 1)^{2}}{R − r}

= \frac{\frac{1}{2} (\sqrt{R} − 1) (\sqrt{r} − 1)}{R − r} [(\sqrt{R} + 1) (1 − \sqrt{r}) + (1 + \sqrt{r}) (\sqrt{R} − 1)]

= \frac{(\sqrt{R} − 1) (\sqrt{r} − 1) (\sqrt{R} − \sqrt{r})}{R − r} = \frac{(\sqrt{R} − 1) (1 − \sqrt{r})}{\sqrt{R} + \sqrt{r}},

and the inequality (3.8) is proved.

Using Theorem 2, we may state the following proposition as well.

Proposition 8. With the assumptions of Proposition 1, we have

(3.9)  0 ≤ \frac{(\sqrt{R} − 1) (1 − \sqrt{r})}{\sqrt{R} + \sqrt{r}} − h^{2} (p, q)

≤ \frac{1}{4 (r − R) A (\sqrt{r}, \sqrt{R})} [(R − 1) (1 − r) − D_{χ}^{2} (p, q)] ,

where A (·, ·) is the arithmetic mean.

The proof is obvious by Theorem 2 applied for ϕ (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}, taking into account that ϕ′ (t) = \frac{1}{2} − \frac{1}{2\sqrt{t}}, and

\frac{ϕ′ (R) − ϕ′ (r)}{R − r} = \frac{\sqrt{R} − \sqrt{r}}{2\sqrt{rR} (R − r)} = \frac{1}{2\sqrt{rR} (\sqrt{R} + \sqrt{r})}.

Finally, by the use of Theorem 3, we may state:

Proposition 9. Assume that p, q ∈ Ω are as in Proposition 1. Then

\frac{1}{8\sqrt{R^{3}}} [(R − 1) (1 − r) − D_{χ}^{2} (p, q)] (3.10)

≤ \frac{(\sqrt{R} − 1) (1 − \sqrt{r})}{\sqrt{R} + \sqrt{r}} − h^{2} (p, q)

≤ \frac{1}{8\sqrt{r^{3}}} [(R − 1) (1 − r) − D_{χ}^{2} (p, q)] .

The proof follows by Theorem 3 applied for the mapping ϕ (t) = \frac{1}{2} (\sqrt{t} − 1)^{2} for which ϕ′′ (t) = \frac{1}{4\sqrt{t^{3}}} and, obviously,

\frac{1}{4\sqrt{R^{3}}} ≤ ϕ′′ (t) ≤ \frac{1}{4\sqrt{r^{3}}} for all t ∈ [r, R] .

References

[1] H. JEFFREYS, An invariant form for the prior probability in estimating problems, Proc. Roy. Soc. London, 186 A (1946), 453-461.

[2] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Ann. Math. Stat., 22 (1951), 79-86.

[3] A. RÉNYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Stat. and Prob., University of California Press, 1 (1961), 547-561.

[4] J.H. HAVRDA and F. CHARVAT, Quantification method classification process: concept of structural α-entropy, Kybernetika, 3 (1967), 30-35.

[5] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), 1-16.

[6] B.D. SHARMA and D.P. MITTAL, New non-additive measures of relative information, Journ. Comb. Inf. Sys. Sci., 2 (4)(1977), 122-132.

[7] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy function, IEEE Trans. Inf. Th., 28 (3) (1982), 489-495.

[8] C.R. RAO, Diversity and dissimilarity coefficients: a unified approach, Theoretic Population Biology, 21 (1982), 24-43.

[9] J. LIN, Divergence measures based on the Shannon entropy, IEEE Trans. Inf. Th., 37 (1) (1991), 145-151.

[10] I. CSISZÁR, Information-type measures of difference of probability distributions and indirect observations, Studia Math. Hungarica, 2 (1967), 299-318.

[11] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[12] S.M. ALI and S.D. SILVEY, A general class of coefficients of divergence of one distribution from another, J. Roy. Statist. Soc. Sec B, 28 (1966), 131-142.

[13] I. VAJDA, Theory of Statistical Inference and Information, Dordrecht-Boston, Kluwer Academic Publishers, 1989.

[14] M. MEI, The theory of genetic distance and evaluation of human races, Japan J. Human Genetics, 23 (1978), 341-369.

[15] A. SEN, On Economic Inequality, Oxford University Press, London 1973.

[16] H. THEIL, Economics and Information Theory, North-Holland, Amsterdam, 1967.

[17] H. THEIL, Statistical Decomposition Analysis, North-Holland, Amsterdam, 1972.

[18] E.C. PIELOU, Ecological Diversity, Wiley, New York, 1975.

[19] D.V. GOKHALE and S. KULLBACK, Information in Contingency Tables, New York, Marcel Dekker, 1978.

[20] C.K. CHOW and C.N. LIN, Approximating discrete probability distributions with dependence trees, IEEE Trans. Inf. Th., 14 (3) (1968), 462-467.

[21] D. KAZAKOS and T. COTSIDAS, A decision theory approach to the approximation of discrete probability densities, IEEE Trans. Perform. Anal. Machine Intell., 1 (1980), 61-67.

[22] T.T. KADOTA and L.A. SHEPP, On the best finite set of linear observables for discriminating two Gaussian signals, IEEE Trans. Inf. Th., 13 (1967), 288-294.

[23] T. KAILATH, The divergence and Bhattacharyya distance measures in signal selection, IEEE Trans. Comm. Technology., Vol COM-15 (1967), 52-60.

[24] M. BETH BASSAT, f-entropies, probability of error and feature selection, Inform. Control, 39 (1978), 227-242.

[25] C.H. CHEN, Statistical Pattern Recognition, Rocelle Park, New York, Hoyderc Book Co., 1973.

[26] V.A. VOLKONSKI and J. A. ROZANOV, Some limit theorems for random function -I, (English Trans.), Theory Prob. Appl., ( USSR), 4 (1959), 178-197.

[27] M.S. PINSKER, Information and Information Stability of Random variables and processes, (in Russian), Moscow: Izv. Akad. Nouk, 1960.

[28] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[29] H.P. McKEAN, JR., Speed of approach to equilibrium for Koc’s caricature of a Maximilian gas, Arch. Ration. Mech. Anal., 21 (1966), 343-367.

[30] J.H.B. KEMPERMAN, On the optimum note of transmitting information, Ann. Math. Statist., 40 (1969), 2158-2177.

[31] S. KULLBACK, A lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 13 (1967), 126-127.

[32] S. KULLBACK, Correction to a lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[33] I. VAJDA, Note on discrimination information and variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[34] G.T. TOUSSAINT, Sharper lower bounds for discrimination in terms of variation, IEEE Trans. Inf. Th., 21 (1975), 99-100.

[35] F. TOPSOE, Some inequalities for information divergence and related measures of discrimination, Res. Rep. Coll., RGMIA, 2 (1) (1999), 85-98.

[36] L. LECAM, Asymptotic Methods in Statistical Decision Theory, New York: Springer, 1986.

[37] D. DACUNHA-CASTELLE, Ecole d’ete de Probability de Saint-Flour, III-1977, Berlin, Heidelberg: Springer 1978.

[38] C. KRAFT, Some conditions for consistency and uniform consistency of statistical procedures, Univ. of California Pub. in Statistics, 1 (1955), 125-142.

[39] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[40] E. HELLINGER, Neue Bergrüirdung du Theorie quadratisher Formerus von uneudlichvieleu Veränderlicher, J. für reine and Augeur. Math., 36 (1909), 210-271.

[41] A. BHATTACHARYYA, On a measure of divergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc., 35 (1943), 99-109.

[42] I. J. TANEJA, Generalised Information Measures and their Applications

    (http://www.mtm.ufsc.br/~taneja/bhtml/bhtml.html).

[43] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[44] J. LIN and S.K.M. WONG, A new directed divergence measure and its characterization, Int. J. General Systems, 17 (1990), 73-81.

[45] H. SHIOYA and T. DA-TE, A generalisation of Lin divergence and the derivative of a new information divergence, Elec. and Comm. in Japan, 78 (7) (1995), 37-40.

[46] J. PEČARIĆ, F. PROSCHAN and Y.L. TONG, Convex Functions, Partial Orderings and Statistical Applications, Academic Press Inc., 1992.

School of Communications and Informatics, Victoria University of Technology, Melbourne City MC, Victoria 8001, Australia.

   E-mail address: sever@matilda.vu.edu.au
   URL: http://rgmia.vu.edu.au/SSDragomirWeb.html