UPPER AND LOWER BOUNDS FOR CSISZÁR f−DIVERGENCE

IN TERMS OF THE KULLBACK-LEIBLER DISTANCE AND APPLICATIONS

S.S. DRAGOMIR

Abstract. In this paper we point out an upper and a lower bound for the Csiszár f−divergence of two discrete random variables in terms of the Kullback-Leibler distance. Some particular cases for Hellinger and triangular discimination, χ^{2} -distance and the Rényi α−entropy, etc. are considered.

1. Introduction

Given a convex function f : [0, ∞) → ℝ, the f−divergence functional

I_{f} (p, q) = ∑_{i=1}^{n} q_{i}f (\frac{p_{i}}{q_{i}} ) , (1.1)

was introduced by Csiszár [1]-[2] as a generalized measure of information, a “distance function” on the set of probability distribution ℙ^{n}. The restriction here to discrete distributions is only for convenience, similar results hold for general distributions. As in Csiszár [1]-[2], we interpret undefined expressions by

f (0) = lim_{t→0+} f (t) , 0 f (\frac{0}{0}) = 0,

0 f (\frac{a}{0}) = lim_{ε→0+} εf (\frac{a}{ε}) = a lim_{t→∞} \frac{f(t)}{t}, a > 0.

The following results (Theorems 1 and 2, and Corollary 1) were essentially given by Csiszár and Körner [3].

Theorem 1. (Joint Convexity) If f : [0, ∞) → ℝ is convex, then I_{f} (p, q) is jointly convex in p and q.

Theorem 2. (Jensen’s inequality) Let f : [0, ∞) → ℝ be convex. Then for any

p, q ∈ ℝ_{+}^{n} with P_{n} := ∑_{i=1}^{n} p_{i} > 0, Q_{n} := ∑_{i=1}^{n} q_{i} > 0, we have the inequality

I_{f} (p, q) ≥ Q_{n}f (\frac{P_{n}}{Q_{n}} ) . (1.2)

If f is strictly convex, equality holds in (1.2) iff

\frac{p_{1}}{q_{1}} = \frac{p_{2}}{q_{2}} = ... = \frac{p_{n}}{q_{n}} . (1.3)

It is natural to consider the following corollary.

Date: December 10, 1999.

1991 Mathematics Subject Classification. Primary 94Xxx, 26D15.

Key words and phrases. Csiszár f−divergence, Divergence measures in Information Theory, Jensen’s Inequality, Hellinger and triangular discimination.

2 S.S. DRAGOMIR

Corollary 1. (Nonnegativity) Let f : [0, ∞) → ℝ be convex and normalised, i.e., f (1) = 0. (1.4) Then for any p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}, we have the inequality I_{f} (p, q) ≥ 0. (1.5) If f is strictly convex, equality holds in (1.5) iff p_{i} = q_{i} for all i ∈ {1, ..., n}. (1.6)

In particular, if p, q are probability vectors, then Corollary 1 shows that, for strictly convex and normalized f : [0, ∞) → ℝ that I_{f} (p, q) ≥ 0 and I_{f} (p, q) = 0 iff p = q. (1.7)

We now give some more examples of divergence measures in Information Theory which are particular cases of Csiszár f−divergences.

1. Kullback-Leibler distance ([12]). The Kullback-Leibler distance D (·, ·) is defined by

D (p, q) := ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) . (1.8)

If we choose f (t) = t ln t, t > 0, then obviously I_{f} (p, q) = D (p, q) . (1.9)

2. Variational distance (l_{1}−distance). The variational distance V (·, ·) is defined by

V (p, q) := ∑_{i=1}^{n} |p_{i} − q_{i}| . (1.10)

If we choose f (t) = |t − 1|, t ∈ ℝ_{+}, then we have

I_{f} (p, q) = V (p, q) . (1.11)

3. Hellinger discrimination ([13]). The Hellinger discrimination h^{2} (·, ·) is defined by

h^{2} (p, q) := \frac{1}{2} ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} . (1.12)

It is obvious that if f (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}, then

I_{f} (p, q) = h^{2} (p, q) . (1.13)

4. Triangular discrimination ([24]). We define triangular discrimination between p and q by

Δ (p, q) = ∑_{i=1}^{n} \frac{|p_{i} − q_{i}|^{2}}{p_{i} + q_{i}} . (1.14)

It is obvious that if f (t) = \frac{(t−1)^{2}}{t+1} , t ∈ (0, ∞), then

I_{f} (p, q) = Δ (p, q) . (1.15)

BOUNDS FOR CSISZÁR f−DIVERGENCE 3

5. χ^{2}−distance. We define the χ^{2}−distance (chi-square distance) by

D_{χ}^{2} (p, q) := ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} . (1.16)

It is clear that if f (t) = (t − 1)^{2}, t ∈ [0, ∞), then

I_{f} (p, q) = D_{χ}^{2} (p, q) (1.17)

6. Rényi α−order entropy ([14]). The α−order entropy (α > 1) is defined by

R_{α} (p, q) := ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}. (1.18)

It is obvious that if f (t) = t^{α} (t ∈ (0, ∞)) , then

I_{f} (p, q) = R_{α} (p, q) . (1.19)

For other examples of divergence measures, see the paper [22] by J.N. Kapur, where further results are given.

2. Some Inequalities Between Csiszár f−Divergence and the Kullback-Leibler Distance

In the recent paper [28], the author proved the following inequality for Csiszár f−divergence:

Theorem 3. Let Φ : ℝ_{+} → ℝ be differentiable convex. Then for all p, q ∈ ℝ_{+}^{n} we have the inequality:

Φ′ (1) (P_{n} − Q_{n}) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) , (2.1)

where P_{n} := ∑_{i=1}^{n} p_{i} > 0, Q_{n} := ∑_{i=1}^{n} q_{i} > 0 and Φ′ : (0, ∞) → ℝ is the derivative of

Φ. If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n) , then the equality holds in (2.9) iff p = q,

If we assume that P_{n} = Q_{n} and Φ is normalised, then we obtain the simpler inequality

0 ≤ I_{Φ} (p, q) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) . (2.2)

Applications for particular divergences which are instances of Csiszár f−divergence were also given.

A similar result of the above theorem has been presented in antoher paper by the author [29].

Theorem 4. Let Φ, p, q be as in Theorem 3. Then we have the inequality

0 ≤ I_{Φ} (p, q) − Q_{n}Φ (\frac{P_{n}}{Q_{n}} ) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − \frac{P_{n}}{Q_{n}} I_{Φ′} (p, q) . (2.3)

If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n), then the equality holds in (2.3) iff \frac{p_{1}}{q_{1}} = ... = \frac{p_{n}}{q_{n}}.

Obviously, if P_{n} = Q_{n} and Φ is normalised, then (2.3) becomes (2.2).

The following result concerning an upper and a lower bound for the Csiszár f−divergence in terms of the Kullback-Leibler distance D (p, q) holds.

As in [30], we will say that the mapping f : C ⊂ ℝ → ℝ, where C is an interval (in [30], the definition was considered in general normed spaces), is

(i) α−lower convex on C if f − \frac{α}{2} · |·|^{2} is convex on C; (ii) β−upper convex on C if \frac{β}{2} · |·|^{2} − f is convex on C; (iii) (m, M) −convex on C (with m ≤ M) if it is both m−lower convex and

M−upper convex. In [30], amongst others, the author has proved the following result for Csiszár

f−divergence.

Theorem 5. Let Φ : ℝ_{+} → ℝ and p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}.

(i) If Φ is α-lower convex on ℝ_{+}, then we have the inequality

\frac{α}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) . (2.4)

(ii) If Φ is β−upper convex on ℝ_{+}, then we have the inequality

I_{Φ} (p, q) − Q_{n}Φ (1) ≤ \frac{β}{2} · D_{χ}^{2} (p, q) . (2.5)

(iii) If Φ is (m, M) −convex on ℝ_{+}, then we have the following sandwich inequality

\frac{m}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ \frac{M}{2} · D_{χ}^{2} (p, q) , (2.6)

where D_{χ}^{2} (·, ·) is the χ^{2}−divergence.

Of course, if Φ is normalised, i.e., Φ (1) = 0 and p, q are probability distributions, then we get the simpler inequalities:

\frac{α}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) , I_{Φ} (p, q) ≤ \frac{β}{2} · D_{χ}^{2} (p, q) (2.7)

and

\frac{m}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) ≤ \frac{M}{2} · D_{χ}^{2} (p, q) . (2.8)

In [30], some applications for particular instances of Csiszár f−divergences were also given.

The following result concerning an upper and a lower bound for the Csiszár f−divergence in terms of the Kullback-Leibler distance D (p, q) holds. This result complements, in a sense, the results presented above in Theorem 5.

Theorem 6. Assume that the generating mapping f : (0, ∞) → ℝ is normalised, i.e., f (1) = 0 and satisfies the assumptions

(i) f is twice differentiable on (r, R) , where 0 ≤ r ≤ 1 ≤ R ≤ ∞; (ii) there exists the real constants m, M such that

m ≤ tf′′ (t) ≤ M for all t ∈ (r, R) . (2.9)

If p, q are discrete probability distributions satisfying the assumption

r ≤ r_{i} := \frac{p_{i}}{q_{i}} ≤ R for all i ∈ \{1, ..., n\} , (2.10)

then we have the inequality

mD (p, q) ≤ I_{f} (p, q) ≤ MD (p, q) . (2.11)

Proof. Define the mapping F_{m} : (0, ∞) → ℝ, F_{m} (t) = f (t) − mt ln t. Then F_{m} (·) is normalised, twice differentiable and since

F′′_{m} (t) = f′′ (t) − \frac{m}{t} = \frac{1}{t} (tf′′ (t) − m) ≥ 0 (2.12)

for all t ∈ (r, R), it follows that F_{m} (·) is convex on (r, R) . Applying the nonnegativity property of the Csiszár f−divergence functional for F_{m} (·) and the linearity property, we may state that

(2.13)  0 ≤ I_{F_{m}} (p, q) = I_{f} (p, q) − mI_{(·) ln(·)} (p, q)
= I_{f} (p, q) − mD (p, q)

from where results the first inequality in (2.11).

Define F_{M} : (0, ∞) → ℝ, F_{M} (t) := Mt ln t−f (t), which is obviously normalised, twice differentiable and by (2.9), convex on (r, R). Applying the nonnegativity property of Csiszár f−divergence for F_{M}, we obtain the second part of (2.11).

Remark 1. If in (2.9) we have the strict inequality “<” for any t ∈ (r, R), then the mappings F_{m} and F_{M} are strictly convex and the case of equality holds in (2.11) iff p = q.

The following theorem concerning the convexity property of the Csiszár f−divergence also holds.

Theorem 7. Assume that f satisfies the assumptions (i) and (ii) from Theorem 6. If p^{(j)}, q^{(j)} (j = 1, 2) are probability distributions satisfying (2.10), i.e.,

(2.14) r ≤ \frac{p_{i}^{(j)}}{q_{i}^{(j)}} ≤ R for all i ∈ \{1, ..., n\} and j ∈ \{1, 2\} ,

then

(2.15) r ≤ \frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ≤ R for all i ∈ \{1, ..., n\} and λ ∈ [0, 1]

and

m[D (λp^{(1)} + (1 − λ) p^{(2)}, λq^{(1)} + (1 − λ) q^{(2)}) (2.16)

−λD (p^{(1)}, q^{(1)}) − (1 − λ) D (p^{(2)}, q^{(2)}) ]
≤ I_{f} (λp^{(1)} + (1 − λ) p^{(2)}, λq^{(1)} + (1 − λ) q^{(2)})
−λI_{f} (p^{(1)}, q^{(1)}) − (1 − λ) I_{f} (p^{(2)}, q^{(2)})
≤ M[D (λp^{(1)} + (1 − λ) p^{(2)}, λq^{(1)} + (1 − λ) q^{(2)})
−λD (p^{(1)}, q^{(1)}) − (1 − λ) D (p^{(2)}, q^{(2)}) ]

for all λ ∈ [0, 1].

Proof. By (2.14), we have

rλq_{i}^{(1)} ≤ λp_{i}^{(1)} ≤ λRq_{i}^{(1)} for all i ∈ {1, ..., n} (2.17)

and

r (1 − λ) q_{i}^{(2)} ≤ (1 − λ) p_{i}^{(2)} ≤ R (1 − λ) q_{i}^{(2)} for all i ∈ {1, ..., n} . (2.18)

Summing the above (2.17) and (2.18), we obtain (2.15).

It is already known that the mappings F_{m}, F_{M} as defined in Theorem 6 are convex and normalised.

Applying the “Joint Convexity Principle” for I_{F_{m}} (·, ·), i.e.,

I_{F_{m}} (λ (p^{(1)}, q^{(1)}) + (1 − λ) (p^{(2)}, q^{(2)})) (2.19)

≤ λI_{F_{m}} (p^{(1)}, q^{(1)}) + (1 − λ) I_{F_{m}} (p^{(2)}, q^{(2)})

and rearranging the terms, we may end up with the first inequality in (2.16).

The second inequality follows likewise if we apply the same property to the Csiszár f−divergence I_{F_{M}} (·, ·) .

We omit the details.

Remark 2. If m > 0 in (2.9), then the inequality (2.11) is a better result than the positivity property of Csiszár divergence. The same will apply for the joint convexity of Csiszár divergence if m > 0.

Using the inequality (2.2) which holds for Φ differentiable convex and normalised functions for p, q probability distributions, we can state the following theorem as well.

Theorem 8. Let f : [0, ∞) → ℝ be a normalised mapping, i.e., f (1) = 0 and satisfies the assumptions:

(i) f is twice differentiable on (r, R), where 0 ≤ r ≤ 1 ≤ R ≤ ∞; (ii) there exists the constants m, M such that

m ≤ tf′′ (t) ≤ M for all t ∈ (r, R) . (2.20)

If p, q are discrete probability distributions satisfying the assumption

r ≤ r_{i} = \frac{p_{i}}{q_{i}} ≤ R for all i ∈ \{1, ..., n\}, (2.21)

then we have the inequality

I_{f′} (\frac{p^{2}}{q} , p) − I_{f′} (p, q) − MD (q, p) (2.22)

≤ I_{f} (p, q)
≤ I_{f′} (\frac{p^{2}}{q} , p) − I_{f′} (p, q) − mD (q, p) .

Proof. We know (see the proof of Theorem 6) that the mapping F_{m} : (0, ∞) → ℝ, F_{m} (t) = f (t) − mt ln t is normalised, twice differentiable and convex on (r, R).

If we apply the second inequality from (2.2) for F_{m}, we may write:

I_{F_{m}} (p, q) ≤ I_{F′_{m}} (\frac{p^{2}}{q} , p) − I_{F′_{m}} (p, q) . (2.23)

However,

I_{F_{m}} (p, q) = I_{f} (p, q) − mD (q, p) ,
I_{F′_{m}} (\frac{p^{2}}{q} , p) = I_{f′(·)−m[ln(·)+1]} (\frac{p^{2}}{q} , p)
= I_{f′} (\frac{p^{2}}{q} , p) − mI_{ln(·)} (\frac{p^{2}}{q} , p) − m
= I_{f′} (\frac{p^{2}}{q} , p) + mD (p, \frac{p^{2}}{q} ) − m

and

I_{F′_{m}} (p, q) = I_{f′} (p, q) + mD (q, p) − m.

Consequently, by (2.23), we have

I_{f} (p, q) − mD (p, q)
≤ I_{f′} (\frac{p^{2}}{q} , p) + mD (p, \frac{p^{2}}{q} ) − m − I_{f′} (p, q) − mD (q, p) + m
= I_{f′} (\frac{p^{2}}{q} , p) + m (D (p, \frac{p^{2}}{q} ) − D (q, p)) − I_{f′} (p, q) .

As a simple computation shows that D (p, \frac{p^{2}}{q} ) = −D (p, q) , the second inequality in (2.22) is proved.

Consider F_{M} (t) := Mt ln t − f (t), which is obviously normalised, twice differentiable and convex on (r, R).

If we apply the second inequality from (2.2) for F_{M}, we may write:

I_{F_{M}} (p, q) ≤ I_{F′_{M}} (\frac{p^{2}}{q} , p) − I_{F′_{M}} (p, q) . (2.24)

However,

I_{F_{M}} (p, q) = MD (p, q) − I_{f} (p, q) ;
I_{F′_{M}} (\frac{p^{2}}{q} , p) = −MD (p, \frac{p^{2}}{q} ) + M − I_{f′} (\frac{p^{2}}{q} , p);
I_{F′_{M}} (p, q) = −MD (q, p) + M − I_{f′} (p, q)

and then, by (2.24), we get

MD (p, q) − I_{f} (p, q)
≤ −MD (p, \frac{p^{2}}{q} ) + M − I_{f′} (\frac{p^{2}}{q} , p) + MD (q, p) − M + I_{f′} (p, q) ,

which is equivalent with the first part of (2.22).

Remark 3. The inequality (2.22) is obviously equivalent with the following one:

mD (q, p) ≤ I_{f′} (\frac{p^{2}}{q} , p) − I_{f′} (p, q) − I_{f} (p, q) ≤ MD (q, p) .

The above results have natural applications when the Kullback-Leibler distance is compared with a number of other divergence measures arising in Information Theory.

3. Some Particular Cases

Using Theorem 6, we shall be able to point out the following particular cases which are of interest in Information Theory.

Proposition 1. Let p, q be two probability distributions with the property that

0 < r ≤ \frac{p_{i}}{q_{i}} = r_{i} ≤ R < ∞ for all i ∈ \{1, ..., n\} . (3.1)

Then we have the inequality

\frac{1}{R}D (p, q) ≤ D (q, p) ≤ \frac{1}{r}D (p, q) . (3.2)

Proof. Consider the mapping f : [r, R] → ℝ, f (t) = − ln t. Define g (t) = tf′′ (t) = t · (\frac{1}{t^{2}} ) = \frac{1}{t}. Then obviously

sup_{t∈[r,R]} g (t) = \frac{1}{r} and inf_{t∈[r,R]} g (t) = \frac{1}{R}.

Also,

I_{f} (p, q) = − ∑_{i=1}^{n} q_{i} ln (\frac{p_{i}}{q_{i}} ) = ∑_{i=1}^{n} q_{i} ln (\frac{q_{i}}{p_{i}} )
= D (q, p) .

Now, using (2.11) with m = \frac{1}{R} and M = \frac{1}{r}, we deduce the desired inequality.

Corollary 2. With the above assumptions for p and q, we have:

r ≤ \frac{D (p, q)}{D (q, p)} ≤ R. (3.3)

Corollary 3. Assume that p, q satisfy the condition

|\frac{p_{i}}{q_{i}} − 1| ≤ ε for all i ∈ \{1, ..., n\} . (3.4)

Then we have the inequality

|\frac{D (p, q)}{D (q, p)} − 1| ≤ ε.

The following proposition connecting the χ^{2}−distance with the Kullback-Leibler distance holds.

Proposition 2. Let p, q be two probability distributions satisfying the condition (3.1). Then we have the inequality:

2r ≤ \frac{D_{χ}^{2} (p, q)}{D (p, q)} ≤ 2R. (3.5)

Proof. Consider the mapping f : [r, R] → ℝ, f (t) = (t − 1)^{2} . Define g (t) = tf′′ (t) = 2t. Then, obviously,

sup_{t∈[r,R]} g (t) = 2R and inf_{t∈[r,R]} g (t) = 2r.

Since

I_{f} (p, q) = D_{χ}^{2} (p, q) ,

then, applying (2.11) for m = 2r and M = 2R, we deduce the desired inequality.

Remark 4. The following inequality is well known in the literature

D (p, q) ≤ D_{χ}^{2} (p, q) . (3.6)

For a simple proof of this fact as well as for different applications in Information Theory, see [27]. Now, observe that from the first inequality in (3.5), we have

D (p, q) ≤ \frac{1}{2r} D_{χ}^{2} (p, q) . (3.7)

We remark that if \frac{1}{2r} ≤ 1 i.e., r ≥ \frac{1}{2}, the inequality (3.7) is better than (3.6).

The following corollary is obvious.

Corollary 4. Assume that the probability distributions p, q satisfy the condition (3.4). Then

\frac{1}{2} |\frac{D_{χ}^{2} (p, q)}{D (p, q)} − 2| ≤ ε. (3.8)

The following inequality connecting the Kullback-Leibler distance with the Hellinger discrimination holds.

Proposition 3. Assume that the probability distributions p, q satisfy the condition (3.1). Then we have the inequality:

\frac{1}{4\sqrt{R}}D (p, q) ≤ h^{2} (p, q) ≤ \frac{1}{4\sqrt{r}} D (p, q) . (3.9)

Proof. Consider the mapping f (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}. Then f′ (t) = \frac{1}{2} − \frac{1}{2\sqrt{t}} and f′′ (t) = \frac{1}{4\sqrt{t^{3}}} . Define g : [r, R] → ℝ given by

g (t) = tf′′ (t) = \frac{1}{4\sqrt{t}}.

Then, obviously

sup_{t∈[r,R]} g (t) = \frac{1}{4\sqrt{r}} and inf_{t∈[r,R]} g (t) = \frac{1}{4\sqrt{R}}.

Since

I_{f} (p, q) = h^{2} (p, q) ,

then by (2.11) for m = \frac{1}{4\sqrt{R}} and M = \frac{1}{4\sqrt{r}} , we deduce the desired inequality (3.9).

Remark 5. The following inequality is well known in the literature (see for example [25]):

D (p, q) ≥ 2h^{2} (p, q) (3.10)

for any p, q probability distribution.

From the second inequality in (3.9), we have

D (p, q) ≥ 4\sqrt{r}h^{2} (p, q) . (3.11)

We remark that if 4\sqrt{r} ≥ 2, i.e., r ≥ \frac{1}{4}, then the inequality in (3.11) is better than (3.10).

The following result establishes a connection between the triangular discrimination Δ and the Kullback-Leibler distance.

Proposition 4. Assume that the probability distributions p, q satisfy the condition (3.1).

(i) If 0 < r ≤ \frac{1}{2}, then we have

8 min \{\frac{r}{(r + 1)^{3}} , \frac{R}{(R + 1)^{3}} \} D (p, q) ≤ Δ (p, q) ≤ \frac{32}{27}D (p, q) . (3.12)

(ii) If \frac{1}{2} < r < 1, then we have

\frac{8R}{(R + 1)^{3}} D (p, q) ≤ Δ (p, q) ≤ \frac{8r}{(r + 1)^{3}} D (p, q) . (3.13)

Proof. Consider the mapping f (t) = \frac{(t−1)^{2}}{t+1} . We have

f′ (t) = 1 − \frac{4}{(t + 1)^{2}}

and

f′′ (t) = \frac{8}{(t + 1)^{3}} .

Define

g : [r, R] → ℝ, g (t) = tf′′ (t) = \frac{8t}{(t + 1)^{3}} , t ∈ [r, R] .

We have

g′ (t) = \frac{8 (1 − 2t)}{(t + 1)^{4}},

which shows that g has the maximum realized at t_{0} = \frac{1}{2} and

max_{t∈(0,∞)} g (t) = g (\frac{1}{2}) = \frac{32}{27}.

We have the two cases:

1) If 0 < r ≤ \frac{1}{2}, then

sup_{t∈[r,R]} g (t) = \frac{32}{27} and

inf_{t∈[r,R]} g (t) = min [g (r) , g (R)] = min \{\frac{8r}{(r + 1)^{3}} , \frac{8R}{(R + 1)^{3}} \} .

2) If \frac{1}{2} < r < 1, then

sup_{t∈[r,R]} g (t) = g (r) = \frac{8r}{(r + 1)^{3}} and
inf_{t∈[r,R]} g (t) = g (R) = \frac{8R}{(R + 1)^{3}} .

Applying the inequality (2.11), we deduce (3.12) and (3.13). We omit the details.

Remark 6. It is clear, by the above arguments, that for every probability distribution we have the inequality

Δ (p, q) ≤ \frac{32}{27}D (p, q) . (3.14)

We know that (see Topsoe [24])

2h^{2} (p, q) ≤ Δ (p, q) ≤ 4h^{2} (p, q) . (3.15)

Now, as D (p, q) ≥ 2h^{2} (p, q), then we obtain

Δ (p, q) ≤ 2D (p, q) , (3.16)

which is not as good as our result (3.14).

Let us compare the Rényi α−entropy with the Kullback-Leibler distance.

The following proposition holds.

Proposition 5. Assume that the probability distributions p, q satisfy the condition (3.1). Then

α (α − 1) r^{α−1}D (p, q) + 1 ≤ R_{α} (p, q) ≤ α (α − 1) R^{α−1}D (p, q) + 1 (3.17)

for α > 1.

Proof. Consider the mapping f : (0, ∞) → ℝ, f (t) = t^{α} − 1, α > 1. Then f′ (t) = αt^{α−1} and f′′ (t) = α (α − 1) t^{α−2}. Define g : [r, R] → ℝ, g (t) = tf′′ (t) = α (α − 1) t^{α−1}. It is obvious that

sup_{t∈[r,R]} g (t) = α (α − 1) R^{α−1} and inf_{t∈[r,R]} g (t) = α (α − 1) r^{α−1}.

Now, observe that f (1) = 0, i.e., f is normalised and so we can apply the inequality (2.11) getting

α (α − 1) r^{α−1}D (p, q) ≤ ∑_{i=1}^{n} q_{i} [(\frac{p_{i}}{q_{i}} )^{α} − 1] ≤ α (α − 1) R^{α−1}D (p, q) ,

i.e.,

α (α − 1) r^{α−1}D (p, q) + 1 ≤ R_{α} (p, q) ≤ α (α − 1) R^{α−1}D (p, q) + 1

and the proposition is proved.

We define the Bhattacharyya distance by (see [27])

B (p, q) := ∑_{i=1}^{n} \sqrt{p_{i}q_{i}}.

The following proposition holds.

Proposition 6. Assume that the probability distributions p, q satisfy the condition (3.1). Then

4\sqrt{r} [1 − B (p, q)] ≤ D (p, q) ≤ 4\sqrt{R} [1 − B (p, q)] . (3.18)

Proof. Consider the mapping f : (0, ∞) → ℝ, f (t) = \sqrt{t}−1. Then f is normalised, f′ (t) = \frac{1}{2}t^{− \frac{1}{2}} , f′′ (t) = −\frac{1}{4}t^{− \frac{3}{2}} . Define g : [r, R] → ℝ, g (t) = tf′′ (t) = −\frac{1}{4}t^{− \frac{1}{2}} . It is obvious that

sup_{t∈[r,R]} g (t) = g (R) = −\frac{1}{4\sqrt{R}}, inf_{t∈[r,R]} g (t) = g (r) = −\frac{1}{4\sqrt{r}} .

12 S.S. DRAGOMIR

Applying the inequality (2.11), we have:

−\frac{1}{4\sqrt{r}} D (p, q) ≤ ∑_{i=1}^{n} q_{i} (\sqrt{\frac{p_{i}}{q_{i}}} − 1) ≤ −\frac{1}{4\sqrt{R}}D (p, q) ,

i.e.,

1 − \frac{1}{4\sqrt{r}} D (p, q) ≤ B (p, q) ≤ 1 − \frac{1}{4\sqrt{R}}D (p, q) ,

which is equivalent to (3.18).

We define the harmonic distance by

M (p, q) := ∑_{i=1}^{n} \frac{2p_{i}q_{i}}{p_{i} + q_{i}} .

The following proposition holds. Proposition 7. Assume that p, q are two discrete probability distributions. Then

1 − \frac{16}{27}D (p, q) ≤ M (p, q) ≤ 1. (3.19)

Proof. Consider the mapping f : (0, ∞) → ℝ, f (t) = \frac{2t}{t+1} −1. Then f is normalised

f′ (t) = \frac{2}{(t + 1)^{2}} , f′′ (t) = \frac{−4}{(t + 1)^{3}} .

Define g : [r, R] → ℝ, g (t) = tf′′ (t) = \frac{−4t}{(t+1)^{3}} . Then

g′ (t) = \frac{4 (2t − 1)}{(t + 1)^{4}}.

It is clear that g is monotonic decreasing on [0, \frac{1}{2}) and monotonic increasing on (\frac{1}{2}, ∞). We have

inf_{t∈(0,∞)} g (t) = g (\frac{1}{2}) = −\frac{16}{27},
sup_{t∈(0,∞)} g (t) = 0.

Applying the inequality (2.11) for m = −\frac{16}{27} and M = 0, we deduce

−\frac{16}{27}D (p, q) ≤ ∑_{i=1}^{n} q_{i} \{[ \frac{2\frac{p_{i}}{q_{i}}}{\frac{p_{i}}{q_{i}} + 1}] − 1\} ≤ 0,

which is equivalent to

−\frac{16}{27}D (p, q) ≤ M (p, q) − 1 ≤ 0

and the inequality (3.19) is proved.

The above result can be improved if we know more information about r_{i} := \frac{p_{i}}{q_{i}} (i = 1, ..., n). We can state the following proposition. Proposition 8. Assume that p, q satisfy the condition (2.10).

BOUNDS FOR CSISZÁR f−DIVERGENCE 13

(i) If r ∈ (0, \frac{1}{2}), then

(3.20)  1 − \frac{16}{27}D (p, q) ≤ M (p, q)
≤ 1 − 4 min \{\frac{r}{(r + 1)^{3}} , \frac{R}{(R + 1)^{3}} \} D (p, q) .

(ii) If r ∈ [\frac{1}{2}, 1), then

1 − \frac{4r}{(r + 1)^{3}} D (p, q) ≤ M (p, q) ≤ 1 − \frac{4R}{(R + 1)^{3}} D (p, q) . (3.21)

Proof. (i) If r ∈ (0, \frac{1}{2}), then

−\frac{16}{27} ≤ g (t) ≤ max \{g (r) , g (R)\}
= max \{−\frac{4r}{(r + 1)^{3}} , −\frac{4R}{(R + 1)^{3}} \}
= −4 min \{\frac{r}{(r + 1)^{3}} , \frac{R}{(R + 1)^{3}} \} , t ∈ [r, R] .

and then, applying (2.11), we may write

−\frac{16}{27}D (p, q) ≤ M (p, q) − 1 ≤ −4 min \{\frac{r}{(r + 1)^{3}} , \frac{R}{(R + 1)^{3}} \} D (p, q) ,

and the inequality (3.20) is proved. (ii) If r ∈ [\frac{1}{2}, 1), then

g (r) ≤ g (t) ≤ g (R) for all t ∈ [r, R] ,

that is,

−\frac{4r}{(r + 1)^{3}} ≤ g (t) ≤ −\frac{4R}{(R + 1)^{3}} , t ∈ [r, R] .

Applying (2.11), we deduce (3.21). Let us consider the J-divergence defined by [26]

J (p, q) : = ∑_{i=1}^{n} (p_{i} − q_{i}) log (\frac{p_{i}}{q_{i}} )
= ∑_{i=1}^{n} q_{i} (\frac{p_{i}}{q_{i}} − 1) log (\frac{p_{i}}{q_{i}} )
= I_{f} (p, q) ,

where f : (0, ∞) → ℝ, f (x) = (x − 1) ln x.

The following proposition also holds. Proposition 9. Assume that p, q satisfy the condition (2.10). Then

\frac{R + 1}{R}D (p, q) ≤ J (p, q) ≤ \frac{r + 1}{r}D (p, q) . (3.22)

Proof. Consider f (t) = (t − 1) ln t. Then f′ (t) = ln t − \frac{1}{t} + 1 and f′′ (t) = \frac{t+1}{t^{2}}. Define g (t) = tf′′ (t) = 1 + \frac{1}{t}. Then obviously

sup_{t∈[r,R]} g (t) = 1 + \frac{1}{r}, inf_{t∈[r,R]} g (t) = 1 + \frac{1}{R}.

Now, using the inequality (2.11), for M = \frac{r+1}{1}, m = \frac{R+1}{R}, we obtain the desired result.

Remark 7. Similar results can be obtained by applying Theorem 8, but we omit the details.

References

[1] I. CSISZÁR, Information measures: A critical survey, Trans. 7th Prague Conf. on Info. Th., Statist. Decis. Funct., Random Processes and 8th European Meeting of Statist., Volume B, Academia Prague, 1978, 73-86.

[2] I. CSISZÁR, Information-type measures of difference of probability functions and indirect observations, Studia Sci. Math. Hungar, 2 (1967), 299-318.

[3] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[4] J.H. JUSTICE (editor), Maximum Entropy and Bayssian Methods in Applied Statistics, Cambridge University Press, Cambridge, 1986.

[5] J.N. KAPUR, On the roles of maximum entropy and minimum discrimination information principles in Statistics, Technical Address of the 38th Annual Conference of the Indian Society of Agricultural Statistics, 1984, 1-44.

[6] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy functions, IEEE Transactions on Information Theory, 28 (1982), 489-495.

[7] R.G. GALLAGER, Information Theory and Reliable Communications, J. Wiley, New York, 1968.

[8] C.E. SHANNON, A mathematical theory of communication, Bull. Sept. Tech. J., 27 (1948), 370-423 and 623-656.

[9] B.R. FRIEDEN, Image enhancement and restoration, Picture Processing and Digital Filtering (T.S. Huang, Editor), Springer-Verlag, Berlin, 1975.

[10] R.M. LEAHY and C.E. GOUTIS, An optimal technique for constraint-based image restoration and mensuration, IEEE Trans. on Acoustics, Speech and Signal Processing, 34 (1986), 1692-1642.

[11] S. KULLBACK, Information Theory and Statistics, J. Wiley, New York, 1959.

[12] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[13] R. BERAN, Minimum Hellinger distance estimates for parametric models, Ann. Statist., 5 (1977), 445-463.

[14] A. RENYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Statist. Prob., Vol. 1, University of California Press, Berkeley, 1961.

[15] S.S. DRAGOMIR and N.M. IONESCU, Some converse of Jensen’s inequality and applications, Anal. Num. Theor. Approx., 23 (1994), 71-78.

[16] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s discrete inequality for differentiable convex mappings and applications in information theory, Math. Comput. Modelling, 24 (2) (1996), 1-11.

[17] S.S. DRAGOMIR and C.J. GOH, Some counterpart inequalities in for a functional associated with Jensen’s inequality, J. of Ineq. & Appl., 1 (1997), 311-325.

[18] S.S. DRAGOMIR and C.J. GOH, Some bounds on entropy measures in information theory, Appl. Math. Lett., 10 (1997), 23-28.

[19] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s continuous inequality and applications in information theory, RGMIA Preprint,

http://matilda.vu.edu.au/~rgmia/InfTheory/Continuse.dvi

[20] M. MATIĆ, Jensen’s inequality and applications in Information Theory (in Croatian), Ph.D. Thesis, Univ. of Zagreb, 1999.

[21] S.S. DRAGOMIR, J. ŠUNDE and M. SCHOLZ, Some upper bounds for relative entropy and applications, Comp. and Math. with Appl. ( in press).

[22] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), No. 1, 1-16.

[23] D.S. MITRINOVIĆ, J.E. PEČARIĆ and A.M. FINK, Classical and New Inequalities in Analysis, Kluwer Academic Publishers, 1993.

[24] F. TOPSOE, Some inequalities for information divergence and related measures of discrimination, Res. Rep. Coll. RGMIA, 2 (1) (1999), 85-98.

[25] D. DACUNHA-CASTELLE, Ecole d’ete de Probabilités de Saint-Fleour, III-1997, Berlin, Heidelberg: Springer 1978.

[26] H. JEFFREYS, An invariant form for the prior probability in estimation problems, Proc. Roy. Soc. London, Ser. A, 1986 (1946), 453-461.

[27] A. BHATTACHARYYA, On a measure of divergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc., 35 (1943), 99-109.

[28] S.S. DRAGOMIR, Some inequalities for the Csiszár Φ-divergence, submitted.

[29] S.S. DRAGOMIR, A converse inequality for the Csiszár Φ-divergence, submitted.

[30] S.S. DRAGOMIR, Some inequalities for (m, M)-convex mappings and applications for the Csiszár Φ−divergence in Information Theory, submitted.

School of Communications and Informatics, Victoria University of Technology, PO Box 14428, Melbourne City MC, Victoria 8001. Australia

   E-mail address: sever.dragomir@vu.edu.au
   URL: http://melba.vu.edu.au/~rgmia/SSDragomirWeb.html