SOME INEQUALITIES FOR TWO CSISZÁR DIVERGENCES AND APPLICATIONS

S.S. DRAGOMIR

Abstract. Some inequalities for the Csiszár divergences of two mappings with applications to the variational distance, Kullback-Leibler distance, Hellinger discrimination, Chi-Square distance, Bhattacharyya distance, Jeffreys divergence, etc... are given.

1. Introduction

Given a convex function f : [0, ∞) → ℝ, the f−divergence functional

I_{f} (p, q) = ∑_{i=1}^{n} q_{i}f (\frac{p_{i}}{q_{i}} ) , (1.1)

was introduced by Csiszár [1]-[2] as a generalized measure of information, a “distance function” on the set of probability distribution ℙ^{n}. The restriction here to discrete distributions is only for convenience, similar results hold for general distributions. As in Csiszár [1]-[2], we interpret undefined expressions by

f (0) = lim_{t→0+} f (t) , 0 f (\frac{0}{0}) = 0,

0 f (\frac{a}{0}) = lim_{ε→0+} εf (\frac{a}{ε}) = a lim_{t→∞} \frac{f(t)}{t}, a > 0.

The following results (Theorems 1 and 2, and Corollary 1) were essentially given by Csiszár and Körner [3].

Theorem 1. (Joint Convexity) If f : [0, ∞) → ℝ is convex, then I_{f} (p, q) is jointly convex in p and q.

Theorem 2. (Jensen’s inequality) Let f : [0, ∞) → ℝ be convex. Then for any

p, q ∈ ℝ_{+}^{n} with P_{n} := ∑_{i=1}^{n} p_{i} > 0, Q_{n} := ∑_{i=1}^{n} q_{i} > 0, we have the inequality

I_{f} (p, q) ≥ Q_{n}f (\frac{P_{n}}{Q_{n}} ) . (1.2)

If f is strictly convex, equality holds in (1.2) iff

\frac{p_{1}}{q_{1}} = \frac{p_{2}}{q_{2}} = ... = \frac{p_{n}}{q_{n}} . (1.3)

It is natural to consider the following corollary.

Date: January 24, 2000.

1991 Mathematics Subject Classification. Primary 94Xxx; Secondary 26D15.

Key words and phrases. Csiszár f−divergence, Variational distance, Kullback-Leibler distance, Hellinger discrimination.

2 S.S. DRAGOMIR

Corollary 1. (Nonnegativity) Let f : [0, ∞) → ℝ be convex and normalised, i.e., f (1) = 0. (1.4) Then for any p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}, we have the inequality I_{f} (p, q) ≥ 0. (1.5) If f is strictly convex, equality holds in (1.5) iff p_{i} = q_{i} for all i ∈ {1, ..., n}. (1.6)

In particular, if p, q are probability vectors, then Corollary 1 shows that, for strictly convex and normalized f : [0, ∞) → ℝ that I_{f} (p, q) ≥ 0 and I_{f} (p, q) = 0 iff p = q. (1.7)

We now give some more examples of divergence measures in Information Theory which are particular cases of Csiszár f−divergences.

1. Kullback-Leibler distance ([12]). The Kullback-Leibler distance D (·, ·) is defined by

D (p, q) := ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) . (1.8)

If we choose f (t) = t ln t, t > 0, then obviously I_{f} (p, q) = D (p, q) . (1.9)

2. Variational distance (l_{1}−distance). The variational distance V (·, ·) is defined by

V (p, q) := ∑_{i=1}^{n} |p_{i} − q_{i}| . (1.10)

If we choose f (t) = |t − 1|, t ∈ ℝ_{+}, then we have

I_{f} (p, q) = V (p, q) . (1.11)

3. Hellinger discrimination ([13]). The Hellinger discrimination h^{2} (·, ·) is defined by

h^{2} (p, q) := \frac{1}{2} ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} . (1.12)

It is obvious that if f (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}, then

I_{f} (p, q) = h^{2} (p, q) . (1.13)

4. Triangular discrimination ([24]). We define triangular discrimination between p and q by

Δ (p, q) = ∑_{i=1}^{n} \frac{|p_{i} − q_{i}|^{2}}{p_{i} + q_{i}} . (1.14)

It is obvious that if f (t) = \frac{(t−1)^{2}}{t+1} , t ∈ (0, ∞), then

I_{f} (p, q) = Δ (p, q) . (1.15)

5. χ^{2}−distance. We define the χ^{2}−distance (chi-square distance) by

D_{χ}^{2} (p, q) := ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} . (1.16)

It is clear that if f (t) = (t − 1)^{2}, t ∈ [0, ∞), then

I_{f} (p, q) = D_{χ}^{2} (p, q) (1.17)

6. Rényi α−order entropy ([14]). The α−order entropy (α > 1) is defined by

R_{α} (p, q) := ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}. (1.18)

It is obvious that if f (t) = t^{α} (t ∈ (0, ∞)) , then

I_{f} (p, q) = R_{α} (p, q) . (1.19)

For other examples of divergence measures, see the paper [22] by J.N. Kapur, where further references are given.

2. The Results

In the recent paper [28], the author proved the following inequality for Csiszár f−divergence:

Theorem 3. Let Φ : ℝ_{+} → ℝ be differentiable convex. Then for all p, q ∈ ℝ_{+}^{n} we have the inequality:

Φ′ (1) (P_{n} − Q_{n}) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) , (2.1)

where P_{n} := ∑_{i=1}^{n} p_{i} > 0, Q_{n} := ∑_{i=1}^{n} q_{i} > 0 and Φ′ : (0, ∞) → ℝ is the derivative of

Φ. If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n) , then the equality holds in (2.9) iff p = q,

If we assume that P_{n} = Q_{n} and Φ is normalised, then we obtain the simpler inequality

0 ≤ I_{Φ} (p, q) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) . (2.2)

Applications for particular divergences which are instances of Csiszár f−divergence were also given.

A similar result of the above theorem has been presented in another paper by the author [29].

Theorem 4. Let Φ, p, q be as in Theorem 3. Then we have the inequality

0 ≤ I_{Φ} (p, q) − Q_{n}Φ (\frac{P_{n}}{Q_{n}} ) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − \frac{P_{n}}{Q_{n}} I_{Φ′} (p, q) . (2.3)

If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n), then the equality holds in (2.3) iff \frac{p_{1}}{q_{1}} = ... = \frac{p_{n}}{q_{n}}.

Obviously, if P_{n} = Q_{n} and Φ is normalised, then (2.3) becomes (2.2).

The following result concerning an upper and a lower bound for the Csiszár f−divergence in terms of the Kullback-Leibler distance D (p, q) holds.

As in [30], we will say that the mapping f : C ⊂ ℝ → ℝ, where C is an interval (in [30], the definition was considered in general normed spaces), is

(i) α−lower convex on C if f − \frac{α}{2} · |·|^{2} is convex on C; (ii) β−upper convex on C if \frac{β}{2} · |·|^{2} − f is convex on C; (iii) (m, M) −convex on C (with m ≤ M) if it is both m−lower convex and

M−upper convex. In [30], amongst others, the author has proved the following result for Csiszár

f−divergence.

Theorem 5. Let Φ : ℝ_{+} → ℝ and p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}.

(i) If Φ is α-lower convex on ℝ_{+}, then we have the inequality

\frac{α}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) . (2.4)

(ii) If Φ is β−upper convex on ℝ_{+}, then we have the inequality

I_{Φ} (p, q) − Q_{n}Φ (1) ≤ \frac{β}{2} · D_{χ}^{2} (p, q) . (2.5)

(iii) If Φ is (m, M) −convex on ℝ_{+}, then we have the following sandwich inequality

\frac{m}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ \frac{M}{2} · D_{χ}^{2} (p, q) , (2.6)

where D_{χ}^{2} (·, ·) is the χ^{2}−divergence.

Of course, if Φ is normalised, i.e., Φ (1) = 0 and p, q are probability distributions, then we get the simpler inequalities:

\frac{α}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) , I_{Φ} (p, q) ≤ \frac{β}{2} · D_{χ}^{2} (p, q) (2.7)

and

\frac{m}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) ≤ \frac{M}{2} · D_{χ}^{2} (p, q) . (2.8)

In [30], some applications for particular instances of Csiszár f−divergences were also given.

We start with the following result.

Theorem 6. Let f, g : [0, ∞) → ℝ be two mappings such that f (1) = g (1) = 0. If there exists the real constants m, M such that

(2.9)  m |f (x) − f (y)| ≤ |g (x) − g (y)| ≤ M |f (x) − f (y)|
for all x, y ∈ [r, R] ⊂ (0, ∞) ,

then we have the inequality:

mI_{|f|} (p, q) ≤ I_{|g|} (p, q) ≤ MI_{|f|} (p, q) (2.10)

for all p, q probability distributions with 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ for all i ∈ \{1, ..., n\}.

Proof. By (2.9) it follows that

(2.11) m |f (\frac{p_{i}}{q_{i}} )| = m |f (\frac{p_{i}}{q_{i}} ) − f (1)| ≤ |g (\frac{p_{i}}{q_{i}} ) − g (1)| = |g (\frac{p_{i}}{q_{i}} )|
≤ M |f (\frac{p_{i}}{q_{i}} ) − f (1)| = M |f (\frac{p_{i}}{q_{i}} )|

for all i ∈ {1, ..., n}.

If we multiply (2.11) by q_{i} ≥ 0 and sum the obtained inequalities, we may deduce (2.10).

Corollary 2. Assume that the mappings f, g : [0, ∞) → ℝ are as above and f, g are differentiable on (r, R) with f′ (t) ≠ 0 for t ∈ (r, R) and

−∞ < m = inf_{t∈(r,R)} |\frac{g′ (t)}{f′ (t)}| , sup_{t∈(r,R)} |\frac{g′ (t)}{f′ (t)}| = M < ∞, (2.12)

then we have the inequality (2.10) for all p, q as above.

Proof. We use the following Cauchy’s theorem:

If γ, ϕ : [a, b] → ℝ are continuous and differentiable on (a, b) and ϕ′ (t) ≠ 0 for all t ∈ (a, b), then there exists a c ∈ [a, b] such that

\frac{γ (b) − γ (a)}{ϕ (b) − ϕ (a)} = \begin{matrix}γ & ′ & (c) \\ ϕ & ′ & (c) \end{matrix}.

Now, suppose that x, y ∈ [r, R] and x < y. Then, by Cauchy’s theorem, we have

m ≤ |\frac{g (x) − g (y)}{f (x) − f (y)}| = |\frac{g′ (z)}{f′ (z)}| ≤ M

and then we can conclude that for any x, y ∈ [r, R] we have

m |f (x) − f (y)| ≤ |g (x) − g (y)| ≤ M |f (x) − f (y)| .

Applying Theorem 6, we deduce (2.10).

The following corollary for the variational distance holds.

Corollary 3. Let g : [0, ∞) → ℝ be a mapping such that g (1) = 0. If there exists the real constants n, N such that

n |x − y| ≤ |g (x) − g (y)| ≤ N |x − y| for all x, y ∈ [r, R] , (2.13)

then we have the inequality

nV (p, q) ≤ I_{|g|} (p, q) ≤ NV (p, q) (2.14)

for any probability distribution p, q with 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ for all i ∈ \{1, ..., n\}.

The proof is obvious by Theorem 6, choosing f (x) = x − 1.

Corollary 4. Assume that the mapping g is continuous on [a, b] and differentiable on (a, b) and

−∞ < n = inf_{t∈(r,R)} |g′ (t)| , sup_{t∈(r,R)} |g′ (t)| = N < ∞.

Then we have the inequality (2.14) for all p, q as above.

6 S.S. DRAGOMIR

3. Some Particular Cases in Terms of the Variational Distance We start with the following result.

Proposition 1. Let 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ (i = 1, ..., n). Then we have the inequality

(3.1) 0 ≤ KL (p, q) ≤ \{ \begin{matrix}[ln R + 1] V (p, q) & if & r ≥ e^{−1}, \\ max \{ln R + 1; |ln R + 1|\} V (p, q) & if & r < e^{−1}. \end{matrix}

Proof. Consider the mapping g : (0, ∞) → ℝ, g (t) = t ln t. Then g′ (t) = ln t + 1 and obviously,

M := sup_{t∈(r,R)} |g′ (t)| = \{ \begin{matrix}ln R + 1 & if & r ≥ e^{−1}, \\ max \{ln R + 1; |ln R + 1|\} & if & r < e^{−1}. \end{matrix}

Applying Corollary 4, we can state

∑_{i=1}^{n} q_{i} |\frac{p_{i}}{q_{i}} ln (\frac{p_{i}}{q_{i}} )| ≤ NV (p, q) .

By the generalised triangle inequality, we have

KL (p, q) = ∑_{i=1}^{n} p_{i} ln (\frac{p_{i}}{q_{i}} ) = |∑_{i=1}^{n} p_{i} ln (\frac{p_{i}}{q_{i}} )| ≤ ∑_{i=1}^{n} p_{i} |ln (\frac{p_{i}}{q_{i}} )| ≤ NV (p, q)

and the inequality (3.1) is proved.

Let us introduce the modified Kullback-Leibler distance

|KL| (p, q) = ∑_{i=1}^{n} p_{i} |ln (\frac{p_{i}}{q_{i}} )| .

Then obviously, K (p, q) ≤ |KL| (p, q) for all p, q ∈ ℙ^{n}. (3.2) For this modified distance, we may prove the following as well. Proposition 2. Let 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ (i = 1, ..., n). Then we have the inequality (ln r + 1) V (p, q) ≤ |KL| (p, q) ≤ (ln R + 1) V (p, q) , (3.3) provided that r ≥ e^{−1}. Proof. The second inequality in (2.11) has been proven above.

For the first inequality, we can apply Corollary 4 by observing that for g (t) = t ln t, and r ≥ e^{−1},

inf_{t∈[r,R]} |g′ (t)| = ln r + 1.

We omit the details.

The following proposition also holds.

INEQUALITIES FOR TWO CSISZÁR DIVERGENCES 7

Proposition 3. Let 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ (i = 1, ..., n). Then we have the inequality:

KL (q, p) ≤ \frac{1}{r}V (p, q) . (3.4)

Proof. Consider the mapping g : (0, ∞) → ℝ, g (t) = ln t. Then g′ (t) = \frac{1}{t} and obviously,

M := sup_{t∈[r,R]} |g′ (t)| = \frac{1}{r}.

Applying Corollary 3, we can state:

∑_{i=1}^{n} q_{i} |ln (\frac{p_{i}}{q_{i}} )| ≤ \frac{1}{r}V (p, q) .

By the generalised triangle inequality, we have

K (q, p) = ∑_{i=1}^{n} q_{i} ln (\frac{q_{i}}{p_{i}} ) = |∑_{i=1}^{n} q_{i} ln (\frac{q_{i}}{p_{i}} )|
≤ ∑_{i=1}^{n} q_{i} |ln (\frac{p_{i}}{q_{i}} )| ≤ \frac{1}{r}V (p, q)

and the proposition is proved.

The following result for the modified Kullback-Leibler distance also holds. Proposition 4. Let p, q be as above in Proposition 3. Then we have the inequality

\frac{1}{R}V (p, q) ≤ |KL| (q, p) ≤ \frac{1}{r}V (p, q) . (3.5)

Proof. The second inequality in (3.5) has been proven above. The first inequality follows by the first inequality in Corollary 4 by taking into account that

m = inf_{t∈(r,R)} |g′ (t)| = \frac{1}{R}.

Now, the following result for Hellinger discrimination holds. Proposition 5. Let 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ (i = 1, ..., n) . Then we have the inequality:

[\frac{\sqrt{R} − \sqrt{r}}{4\sqrt{rR}} − |\frac{\sqrt{R} + \sqrt{r}}{4\sqrt{rR}} − \frac{1}{2}|] V (p, q) (3.6)

≤ h^{2} (p, q) ≤ [\frac{\sqrt{R} − \sqrt{r}}{4\sqrt{rR}} + |\frac{\sqrt{R} + \sqrt{r}}{4\sqrt{rR}} − \frac{1}{2}|] V (p, q) .

Proof. Consider the mapping g : (0, ∞) → ℝ, g (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}. Then obviously,

g′ (t) = \frac{1}{2} · \frac{\sqrt{t} − 1}{\sqrt{t}}, t ∈ (0, ∞)

and

n = inf_{t∈[r,R]} |g′ (t)| = min {|g′ (r)| , |g′ (R)|}
= \frac{|g′ (r)| + |g′ (R)| − ||g′ (r)| − |g′ (R)||}{2}
= \frac{\sqrt{R} − \sqrt{r}}{4\sqrt{rR}} − |\frac{\sqrt{R} + \sqrt{r}}{4\sqrt{rR}} − \frac{1}{2}|

and

N = sup_{t∈[r,R]} |g′ (t)| = max {|g′ (r)| , |g′ (R)|}
= \frac{|g′ (r)| + |g′ (R)| + ||g′ (r)| − |g′ (R)||}{2}
= \frac{\sqrt{R} − \sqrt{r}}{4\sqrt{rR}} + |\frac{\sqrt{R} + \sqrt{r}}{4\sqrt{rR}} − \frac{1}{2}|

respectively.

As g (t) ≥ 0, then obviously

I_{|g|} (p, q) = I_{g} (p, q) = h^{2} (p, q) .

Using (2.14), we obtain (3.6).

Remark 1. The inequality (3.6) is equivalent to

|h^{2} (p, q) − \frac{\sqrt{R} − \sqrt{r}}{4\sqrt{rR}} V (p, q)| ≤ |\frac{\sqrt{R} + \sqrt{r}}{4\sqrt{rR}} − \frac{1}{2}| V (p, q) (3.7)

Now, we point out some inequalities for the chi-square distance.

Proposition 6. Let 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ (i = 1, ..., n). Then we have the inequality

[R − r − |R + r − 2|] V (p, q) ≤ D_{χ}^{2} (p, q) ≤ [R − r + |R + r − 2|] V (p, q) . (3.8)

Proof. Consider the mapping g : (0, ∞) → ℝ, g (t) = (t − 1)^{2}. Then obviously g′ (t) = 2 (t − 1) and

n = inf_{t∈[r,R]} |g′ (t)| = min {|g′ (r)| , |g′ (R)|}
= R − r − |R + r − 2|

and

N = R − r + |R + r − 2| .

Using the inequality (2.14), and taking into account that g (t) ≥ 0, t ∈ ℝ, and

I_{g} (p, q) = ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} = D_{χ}^{2} (p, q) ,

we deduce (3.8).

Remark 2. The inequality (3.8) is equivalent with

|D_{χ}^{2} (p, q) − (R − r) V (p, q)| ≤ |R + r − 2| V (p, q) . (3.9)

INEQUALITIES FOR TWO CSISZÁR DIVERGENCES 9

We point out now some inequalities for the Bhattacharyya distance. Proposition 7. Let 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ (i = 1, ..., n). Then we have the inequality:

0 ≤ 1 − B (p, q) ≤ \frac{1}{2\sqrt{r}} V (p, q) . (3.10)

Proof. Consider the mapping g (t) = 1 − \sqrt{t}, t ∈ (0, ∞). Then g (1) = 0, g′ (t) = −\frac{1}{2\sqrt{t}} and

N = sup_{t∈[r,R]} |g′ (t)| = sup_{t∈[r,R]} \frac{1}{2\sqrt{t}} = \frac{1}{2\sqrt{r}} .

Applying Corollary 4, we may state

∑_{i=1}^{n} q_{i} |1 − \sqrt{\frac{p_{i}}{q_{i}}} | ≤ \frac{1}{2\sqrt{r}} V (p, q) ,

which is equivalent to

∑_{i=1}^{n} |q_{i} − \sqrt{p_{i}q_{i}}| ≤ \frac{1}{2\sqrt{r}} V (p, q) . (3.11)

Using the generalised triangle inequality, we obtain

∑_{i=1}^{n} |q_{i} − \sqrt{p_{i}q_{i}}| ≥ |∑_{i=1}^{n} (q_{i} − \sqrt{p_{i}q_{i}})|
= |1 − B (p, q)| = 1 − B (p, q) .

If we define the following distance B̃ (p, q) := ∑_{i=1}^{n} \sqrt{q_{i}} |\sqrt{q_{i}} − \sqrt{p_{i}}| , then we may state the following proposition as well. Proposition 8. Assume that p_{i}, q_{i}, r, R are as above. Then

\frac{1}{2\sqrt{R}}V (p, q) ≤ B̃ (p, q) ≤ \frac{1}{2\sqrt{r}} V (p, q) . (3.12)

The proof is obvious by Corollary 3 applied for the mapping g (t) = 1 − \sqrt{t}. Now, let us consider the harmonic distance

M (p, q) := ∑_{i=1}^{n} \frac{2p_{i}q_{i}}{p_{i} + q_{i}} .

The following proposition holds. Proposition 9. Assume that p_{i}, q_{i}, r, R are as above. Then we have the inequality:

0 ≤ 1 − M (p, q) ≤ \frac{2}{(r + 1)^{2}} V (p, q) . (3.13)

Proof. Consider the mapping g (t) = 1 − \frac{2t}{t+1}. Then g (1) = 0, g′ (t) = −\frac{2}{(t+1)^{2}} and

N := sup_{t∈[r,R]} |g′ (t)| = \frac{2}{(r + 1)^{2}} .

Applying Corollary 4, we can state that

∑_{i=1}^{n} q_{i} |1 − \frac{2\frac{p_{i}}{q_{i}}}{\frac{p_{i}}{q_{i}} + 1}| ≤ \frac{2}{(r + 1)^{2}} V (p, q) ,

which is clearly equivalent to:

∑_{i=1}^{n} \frac{q_{i} |p_{i} − q_{i}|}{p_{i} + q_{i}} ≤ \frac{2}{(r + 1)^{2}} V (p, q) . (3.14)

Using the generalised triangle inequality, we get (3.13).

If we introduce the divergence measure:

M̃ (p, q) := ∑_{i=1}^{n} q_{i} · \frac{|p_{i} − q_{i}|}{p_{i} + q_{i}} = I_{l} (p, q) ,

where l (t) = \frac{|t−1|}{t+1}, t > 0, then we have the following proposition.

Proposition 10. With the above assumptions, we have

\frac{2}{(R + 1)^{2}} V (p, q) ≤ M̃ (p, q) ≤ \frac{2}{(r + 1)^{2}} V (p, q) . (3.15)

Finally, let us consider the Jeffreys distance

J (p, q) = ∑_{i=1}^{n} (p_{i} − q_{i}) ln (\frac{p_{i}}{q_{i}} ) .

The following proposition holds.

Proposition 11. Assume that 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞. Then we have the inequality:

[\frac{R − r}{2rR} + ln \sqrt{\frac{R}{r}} − |\frac{R − r}{2rR} − ln \sqrt{rR} − 1|] V (p, q) (3.16)

≤ J (p, q) ≤ [\frac{R − r}{2rR} + ln \sqrt{\frac{R}{r}} + |\frac{R − r}{2rR} − ln \sqrt{rR} − 1|] V (p, q) .

Proof. Consider the mapping g (t) = (t − 1) ln t, t > 0. Then, obviously g′ (t) = ln t − \frac{1}{t} + 1, g′′ (t) = \frac{t+1}{t^{2}}, which shows that g′ (·) is strictly increasing on (0, ∞) and g′ (1) = 0. Then

n = inf_{t∈[r,R]} |g′ (t)| = min {|g′ (r)| , |g′ (R)|}

= \frac{R − r}{2rR} + ln \sqrt{\frac{R}{r}} − |\frac{R + r}{2rR} − ln \sqrt{rR} − 1|

and

N = sup_{t∈[r,R]} |g′ (t)| = max {|g′ (r)| , |g′ (R)|}

= \frac{R − r}{2rR} + ln \sqrt{\frac{R}{r}} + |\frac{R + r}{2rR} − ln \sqrt{rR} − 1| .

INEQUALITIES FOR TWO CSISZÁR DIVERGENCES 11

In addition, as

I_{|g|} (p, q) = ∑_{i=1}^{n} q_{i} |(\frac{p_{i}}{q_{i}} − 1)| |ln (\frac{p_{i}}{q_{i}} )| = ∑_{i=1}^{n} |p_{i} − q_{i}| |ln p_{i} − ln q_{i}|
= ∑_{i=1}^{n} (p_{i} − q_{i}) (ln p_{i} − ln q_{i}) = I (p, q) ,

then by (2.14), we deduce (3.16). Remark 3. The above inequality (3.16) is equivalent to

|J (p, q) − [\frac{R − r}{2rR} + ln \sqrt{\frac{R}{r}}] V (p, q)| ≤ |\frac{R + r}{2rR} − ln \sqrt{rR} − 1| V (p, q) . (3.17)

4. Other Particular Cases

Let us consider the modified Kullback-Leibler divergence

|KL| (q, p) := ∑_{i=1}^{n} q_{i} |ln (\frac{p_{i}}{q_{i}} )| ,

where p, q ∈ ℙ^{n}.

We point out some estimates in terms of |KL|. Proposition 12. Assume that 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ (i = 1, ..., n). Then we have the inequality

0 ≤ KL (p, q) ≤ [\frac{R − r}{2} + ln \sqrt{\frac{R^{R}}{r^{r}}} + |\frac{r + R}{2} + ln \sqrt{R^{R}r^{r}}|] |KL| (q, p) . (4.1)

Proof. Consider the mappings g (t) = t ln t, f (t) = ln t, t > 0. Then h (t) := \frac{g′(t)}{f′(t)} = t ln t + t.

We observe that

M = sup_{t∈[r,R]} |h (t)| = max {|h (r)| , |h (R)|}
= \frac{R − r}{2} + ln \sqrt{\frac{R^{R}}{r^{r}}} + |\frac{r + R}{2} + ln \sqrt{R^{R}r^{r}}| .

Applying Corollary 2, we may write

∑_{i=1}^{n} q_{i} |\frac{p_{i}}{q_{i}} ln (\frac{p_{i}}{q_{i}} )| ≤ M ∑_{i=1}^{n} q_{i} |ln (\frac{p_{i}}{q_{i}} )| = M |KL| (q, p)

and as, by the generalised triangle inequality, we have

∑_{i=1}^{n} p_{i} |ln (\frac{p_{i}}{q_{i}} )| ≥ |KL (p, q)| = KL (p, q) ≥ 0,

the inequality (4.1) is proved.

We now compare the Hellinger discrimination with |KL|.

Proposition 13. Let p_{i}, q_{i}, r, R be as in Proposition 12. Then we have the inequality:

\frac{1}{2} [\frac{R − r}{2} − \frac{\sqrt{R} − \sqrt{r}}{2} − |\frac{\sqrt{r} + \sqrt{R}}{2} − \frac{r + R}{2}|] |KL| (q, p) (4.2)

≤ h^{2} (p, q) ≤ \frac{1}{2} [\frac{R − r}{2} − \frac{\sqrt{R} − \sqrt{r}}{2} + |\frac{\sqrt{r} + \sqrt{R}}{2} − \frac{r + R}{2}|] |KL| (q, p) .

Proof. Consider the mappings g (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}, f (t) = ln t, t > 0. Then

h (t) := \frac{g′ (t)}{f′ (t)} = \frac{1}{2} (\frac{\sqrt{t} − 1}{\sqrt{t}}) · t = \frac{1}{2} (\sqrt{t} − 1) \sqrt{t}, t > 0.

We observe that

m = inf_{t∈[r,R]} |h (t)| = min {|h (r)| , |h (R)|}

= \frac{1}{2} [\frac{R − r}{2} − \frac{\sqrt{R} − \sqrt{r}}{2} − |\frac{\sqrt{r} + \sqrt{R}}{2} − \frac{r + R}{2}|]

and, analogously,

M = sup_{t∈[r,R]} |h (t)| = max {|h (r)| , |h (R)|}

= \frac{1}{2} [\frac{R − r}{2} − \frac{\sqrt{R} − \sqrt{r}}{2} + |\frac{\sqrt{r} + \sqrt{R}}{2} − \frac{r + R}{2}|] .

Now, as g (t) ≥ 0, we have

I_{|g|} (p, q) = I_{g} (p, q) = h^{2} (p, q)

and then, by Corollary 2, we deduce (4.2).

Remark 4. The above inequality is equivalent with

|h^{2} (p, q) − \frac{1}{2} [\frac{R − r}{2} − \frac{\sqrt{R} − \sqrt{r}}{2} ] |KL (q, p)|| (4.3)

≤ |\frac{\sqrt{r} + \sqrt{R}}{2} − \frac{r + R}{2}| |KL| (q, p) .

We now compare the Chi-square distance with |KL|. The following proposition holds.

Proposition 14. Let p_{i}, q_{i}, r, R be as above. Then

[(R − r) (R + r − 1) − |R + r − (R^{2} + r^{2})|] |KL| (q, p) (4.4)

≤ D_{χ}^{2} (p, q) ≤ [(R − r) (R + r − 1) + |R + r − (R^{2} + r^{2})|] |KL| (q, p) .

Proof. Consider the mappings g (t) = (t − 1)^{2}, f (t) = ln t, t > 0. Then h (t) = \frac{g′(t)}{f′(t)} = 2t (t − 1).

We observe that

m = inf_{t∈[r,R]} |h (t)| = \frac{1}{2} [2r (1 − r) + 2R (R − 1) − |2r (1 − r) − 2R (R − 1)|]
= [r − r^{2} + R^{2} − R − |r − r^{2} − R^{2} + R|]
= R^{2} − r^{2} − (R − r) − |R + r − (R^{2} + r^{2})|
= (R − r) (R + r − 1) − |R + r − (R^{2} + r^{2})|

and

M = sup_{t∈[r,R]} (h (t)) = (R − r) (R + r − 1) + |R + r − (R^{2} + r^{2})| .

Now, as g (t) ≥ 0, we have

I_{|g|} (p, q) = I_{g} (p, q) = D_{χ}^{2} (p, q)

and then, by Corollary 2, we deduce (4.4).

Remark 5. The above inequality is equivalent with

|D_{χ}^{2} (p, q) − (R − r) (R + r − 1) |KL| (q, p)| (4.5)

≤ |R + r − (R^{2} + r^{2})| |KL| (q, p) .

References

[1] I. CSISZÁR, Information measures: A critical survey, Trans. 7th Prague Conf. on Info. Th., Statist. Decis. Funct., Random Processes and 8th European Meeting of Statist., Volume B, Academia Prague, 1978, 73-86.

[2] I. CSISZÁR, Information-type measures of difference of probability functions and indirect observations, Studia Sci. Math. Hungar, 2 (1967), 299-318.

[3] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[4] J.H. JUSTICE (editor), Maximum Entropy and Bayssian Methods in Applied Statistics, Cambridge University Press, Cambridge, 1986.

[5] J.N. KAPUR, On the roles of maximum entropy and minimum discrimination information principles in Statistics, Technical Address of the 38th Annual Conference of the Indian Society of Agricultural Statistics, 1984, 1-44.

[6] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy functions, IEEE Transactions on Information Theory, 28 (1982), 489-495.

[7] R.G. GALLAGER, Information Theory and Reliable Communications, J. Wiley, New York, 1968.

[8] C.E. SHANNON, A mathematical theory of communication, Bull. Sept. Tech. J., 27 (1948), 370-423 and 623-656.

[9] B.R. FRIEDEN, Image enhancement and restoration, Picture Processing and Digital Filtering (T.S. Huang, Editor), Springer-Verlag, Berlin, 1975.

[10] R.M. LEAHY and C.E. GOUTIS, An optimal technique for constraint-based image restoration and mensuration, IEEE Trans. on Acoustics, Speech and Signal Processing, 34 (1986), 1692-1642.

[11] S. KULLBACK, Information Theory and Statistics, J. Wiley, New York, 1959.

[12] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[13] R. BERAN, Minimum Hellinger distance estimates for parametric models, Ann. Statist., 5 (1977), 445-463.

[14] A. RENYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Statist. Prob., Vol. 1, University of California Press, Berkeley, 1961.

[15] S.S. DRAGOMIR and N.M. IONESCU, Some converse of Jensen’s inequality and applications, Anal. Num. Theor. Approx., 23 (1994), 71-78.

[16] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s discrete inequality for differentiable convex mappings and applications in information theory, Math. Comput. Modelling, 24 (2) (1996), 1-11.

[17] S.S. DRAGOMIR and C.J. GOH, Some counterpart inequalities in for a functional associated with Jensen’s inequality, J. of Ineq. & Appl., 1 (1997), 311-325.

[18] S.S. DRAGOMIR and C.J. GOH, Some bounds on entropy measures in information theory, Appl. Math. Lett., 10 (1997), 23-28.

[19] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s continuous inequality and applications in information theory, RGMIA Preprint,

    http://matilda.vu.edu.au/~rgmia/InfTheory/Continuse.dvi

[20] M. MATIĆ, Jensen’s inequality and applications in Information Theory (in Croatian), Ph.D. Thesis, Univ. of Zagreb, 1999.

[21] S.S. DRAGOMIR, J. ŠUNDE and M. SCHOLZ, Some upper bounds for relative entropy and applications, Comp. and Math. with Appl. ( in press).

[22] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), No. 1, 1-16.

[23] D.S. MITRINOVIĆ, J.E. PEČARIĆ and A.M. FINK, Classical and New Inequalities in Analysis, Kluwer Academic Publishers, 1993.

[24] F. TOPSOE, Some inequalities for information divergence and related measures of discrimination, Res. Rep. Coll. RGMIA, 2 (1) (1999), 85-98.

[25] D. DACUNHA-CASTELLE, Ecole d’ete de Probabilités de Saint-Fleour, III-1997, Berlin, Heidelberg: Springer 1978.

[26] H. JEFFREYS, An invariant form for the prior probability in estimation problems, Proc. Roy. Soc. London, Ser. A, 1986 (1946), 453-461.

[27] A. BHATTACHARYYA, On a measure of divergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc., 35 (1943), 99-109.

[28] S.S. DRAGOMIR, Some inequalities for the Csiszár Φ-divergence, submitted.

[29] S.S. DRAGOMIR, A converse inequality for the Csiszár Φ-divergence, submitted.

[30] S.S. DRAGOMIR, Some inequalities for (m, M)-convex mappings and applications for the Csiszár Φ−divergence in information theory, submitted.

School of Communications and Informatics, P.O. Box 14428, Victoria University of Technology

   E-mail address: sever.dragomir@vu.edu.au
   URL: http://rgmia.vu.edu.au/SSDragomirWeb.html