UPPER AND LOWER BOUNDS FOR CSISZÁR f−DIVERGENCE IN TERMS OF HELLINGER DISCRIMINATION AND APPLICATIONS

S.S. DRAGOMIR

Abstract. In this paper we point out an upper and a lower bound for the Csiszár f−divergence of two discrete random variables in terms of the Hellinger discrimination. Some paricular cases for the Kullback-Leibler distance, triangular discrimination, χ^{2}−distance and the Rényi α−entropy, etc.. are considered.

1. Introduction

Given a convex function f : [0, ∞) → ℝ, the f−divergence functional

(1.1)  I_{f} (p, q) = ∑_{i=1}^{n} q_{i}f (\frac{p_{i}}{q_{i}} ) ,

was introduced by Csiszár [1]-[2] as a generalized measure of information, a “distance function” on the set of probability distribution ℙ^{n}. The restriction here to discrete distributions is only for convenience, similar results hold for general distributions. As in Csiszár [1]-[2], we interpret undefined expressions by

f (0) = lim_{t→0+} f (t) , 0 f (\frac{0}{0}) = 0,

0 f (\frac{a}{0}) = lim_{ε→0+} εf (\frac{a}{ε}) = a lim_{t→∞} \frac{f(t)}{t}, a > 0.

The following results (Theorems 1 and 2, and Corollary 1) were essentially given by Csiszár and Körner [3]. Theorem 1. (Joint Convexity) If f : [0, ∞) → ℝ is convex, then I_{f} (p, q) is jointly convex in p and q. Theorem 2. (Jensen’s inequality) Let f : [0, ∞) → ℝ be convex. Then for any

p, q ∈ ℝ_{+}^{n} with P_{n} := ∑_{i=1}^{n} p_{i} > 0, Q_{n} := ∑_{i=1}^{n} q_{i} > 0, we have the inequality

(1.2)  I_{f} (p, q) ≥ Q_{n}f (\frac{P_{n}}{Q_{n}} ) .

If f is strictly convex, equality holds in (1.2) iff

(1.3)  \frac{p_{1}}{q_{1}} = \frac{p_{2}}{q_{2}} = ... = \frac{p_{n}}{q_{n}} .

It is natural to consider the following corollary.

Date: December 10, 1999.

1991 Mathematics Subject Classification. Primary 94Xxx, 26D15.

Key words and phrases. Csiszár f−Divergence, Divergence measures in Information Theory, Jensen’s Inequality.

Corollary 1. (Nonnegativity) Let f : [0, ∞) → ℝ be convex and normalised, i.e.,

(1.4)  f (1) = 0.

Then for any p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}, we have the inequality

(1.5)  I_{f} (p, q) ≥ 0.

If f is strictly convex, equality holds in (1.5) iff

(1.6)  p_{i} = q_{i} for all i ∈ {1, ..., n}.

In particular, if p, q are probability vectors, then Corollary 1 shows that, for strictly convex and normalized f : [0, ∞) → ℝ that

(1.7)  I_{f} (p, q) ≥ 0 and I_{f} (p, q) = 0 iff p = q.

We now give some more examples of divergence measures in Information Theory which are particular cases of Csiszár f−divergences.

(1) Kullback-Leibler distance ([12]). The Kullback-Leibler distance D (·, ·) is defined by

(1.8)  D (p, q) := ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) .

If we choose f (t) = t ln t, t > 0, then obviously

(1.9)  I_{f} (p, q) = D (p, q) .

(2) Variational distance (l_{1}−distance). The variational distance V (·, ·) is defined by

(1.10)  V (p, q) := ∑_{i=1}^{n} |p_{i} − q_{i}| .

If we choose f (t) = |t − 1|, t ∈ ℝ_{+}, then we have

(1.11)  I_{f} (p, q) = V (p, q) .

(3) Hellinger discrimination ([13]). The Hellinger discrimination h^{2} (·, ·) is defined by

(1.12)  h^{2} (p, q) := \frac{1}{2} ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} .

It is obvious that if f (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}, then

(1.13)  I_{f} (p, q) = h^{2} (p, q) .

(4) Triangular discrimination ([24]). We define triangular discrimination between p and q by

(1.14)  Δ (p, q) = ∑_{i=1}^{n} \frac{|p_{i} − q_{i}|^{2}}{p_{i} + q_{i}} .

It is obvious that if f (t) = \frac{(t−1)^{2}}{t+1} , t ∈ (0, ∞), then

(1.15)  I_{f} (p, q) = Δ (p, q) .

BOUNDS FOR CSISZÁR f−DIVERGENCE 3

(5) χ^{2}−distance. We define the χ^{2}−distance (chi-square distance) by

(1.16)  D_{χ^{2}} (p, q) := ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} .

It is clear that if f (t) = (t − 1)^{2}, t ∈ [0, ∞), then

(1.17)  I_{f} (p, q) = D_{χ^{2}} (p, q)

(6) Rényi α−order entropy ([14]). The α−order entropy (α > 1) is defined by

(1.18)  R_{α} (p, q) := ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}.

It is obvious that if f (t) = t^{α} (t ∈ (0, ∞)) , then

(1.19)  I_{f} (p, q) = R_{α} (p, q) .

For other examples of divergence measures, see the paper [22] by J. N. Kapur where further references are given.

2. Some Inequalities Between Csiszár f−Divergence and Hellinger Discrimination

In the recent paper [28], the author proved the following inequality for Csiszár f−divergence: Theorem 3. Let Φ : ℝ_{+} → ℝ be differentiable convex. Then for all p, q ∈ ℝ_{+}^{n} we have the inequality:

(2.1)  Φ′ (1) (P_{n} − Q_{n}) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) ,

where P_{n} := ∑_{i=1}^{n} p_{i} > 0, Q_{n} := ∑_{i=1}^{n} q_{i} > 0 and Φ′ : (0, ∞) → ℝ is the derivative of

Φ. If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n) , then the equality holds in (2.9) iff p = q,

If we assume that P_{n} = Q_{n} and Φ is normalised, then we obtain the simpler

(2.2)  0 ≤ I_{Φ} (p, q) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) .

Applications for particular divergences which are instances of Csiszár f−divergence were also given.

Asimilar result of the above theorem has been presented in antoher paper by the author [29]. Theorem 4. Let Φ, p, q be as in Theorem 3. Then we have the inequality

(2.3)  0 ≤ I_{Φ} (p, q) − Q_{n}Φ (\frac{P_{n}}{Q_{n}} ) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − \frac{P_{n}}{Q_{n}} I_{Φ′} (p, q) .

If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n), then the equality holds in (2.3) iff \frac{p_{1}}{q_{1}} = ... = \frac{p_{n}}{q_{n}}.

Obviously, if P_{n} = Q_{n} and Φ is normalised, then (2.3) becomes (2.2).

The following result concerning an upper and a lower bound for the Csiszár f−divergence in terms of the Kullback-Leibler distance D (p, q) holds.

As in [30], we will say that the mapping f : C ⊂ ℝ → ℝ, where C is an interval, (in [30], the definition was considered in general normed spaces) is

(i) α−lower convex on C if f − \frac{α}{2} · |·|^{2} is convex on C; (ii) β−upper convex on C if \frac{β}{2} · |·|^{2} − f is convex on C; (iii) (m, M) −convex on C (with m ≤ M) if it is both m−lower convex and

M−upper convex. In [30], amongst others, the author has proved the following result for Csiszár

f−divergence. Theorem 5. Let Φ : ℝ_{+} → ℝ and p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}.

(i) If Φ is α-lower convex on ℝ_{+}, then we have the inequality

(2.4)  \frac{α}{2} · D_{χ^{2}} (p, q) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) .

(ii) If Φ is β−upper convex on ℝ_{+}, then we have the inequality

(2.5)  I_{Φ} (p, q) − Q_{n}Φ (1) ≤ \frac{β}{2} · D_{χ^{2}} (p, q) .

(iii) If Φ is (m, M) −convex on ℝ_{+}, then we have the following sandwich inequality

(2.6)  \frac{m}{2} · D_{χ^{2}} (p, q) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ \frac{M}{2} · D_{χ^{2}} (p, q) ,

where D_{χ^{2}} (·, ·) is the χ^{2}−divergence. Of course, if Φ is normalised, i.e., Φ (1) = 0 and p, q are probability distributions,

then we get the simpler inequalities:

(2.7)  \frac{α}{2} · D_{χ^{2}} (p, q) ≤ I_{Φ} (p, q) , I_{Φ} (p, q) ≤ \frac{β}{2} · D_{χ^{2}} (p, q)

and

(2.8)  \frac{m}{2} · D_{χ^{2}} (p, q) ≤ I_{Φ} (p, q) ≤ \frac{M}{2} · D_{χ^{2}} (p, q) .

In [30], some applications for particular instances of Csiszár f−divergences were also given.

The following result concerning an upper and a lower bound for the Csiszár f−divergence in terms of the Hellinger discrimination h^{2} (p, q) holds. These results will complement, in a sense, the ones presented above. Theorem 6. Assume that the generating mapping f : (0, ∞) → ℝ is normalized, i.e., f (1) = 0 and satisfies the assumptions

(i) f is twice differentiable on (r, R), where 0 ≤ r ≤ 1 ≤ R ≤ ∞, (ii) there exists the real constants m, M such that

(2.9)  m ≤ t^{\frac{3}{2}} f′′ (t) ≤ M for all t ∈ (r, R) .

If p, q are discrete probability distributions verifying the assumption

(2.10)  r ≤ r_{i} := \frac{p_{i}}{q_{i}} ≤ R for all i ∈ \{1, ..., n\} ,

then we have the inequality

(2.11)  4mh^{2} (p, q) ≤ I_{f} (p, q) ≤ 4Mh^{2} (p, q) .

Proof. Define the mapping H_{m} : (0, ∞) → ℝ, H_{m} (t) = f (t) − 2m (\sqrt{t} − 1)^{2}. Then H_{m} (·) is normalised, twice differentiable and since

(2.12)  H′′_{m} (t) = f′′ (t) − \frac{m}{t^{\frac{3}{2}}} = \frac{1}{t^{\frac{3}{2}}} (t^{\frac{3}{2}} f′′ (t) − m) ≥ 0

for all t ∈ (a, b), implied by the first inequality in (2.9). Thus, the mapping H_{m} (·) is convex on (r, R) .

Applying the nonnegativity property of the Csiszár f−divergence functional for H_{m} (·) and the linearity, we may state that

(2.13)  0 ≤ I_{H_{m}} (p, q) = I_{f} (p, q) − 2mI_{(\sqrt{·}−1)}^{2} (p, q)
= I_{f} (p, q) − 4mh^{2} (p, q) ,

from where we get the first inequality in (2.11).

Define H_{M} : (0, ∞) → ℝ, H_{M} (t) = 2M (\sqrt{t} − 1)^{2} − f (t) which obviously is normalised, twice differentiable and, by (2.9), convex on (r, R).

Applying the nonnegativity property of Csiszár f−divergence for I_{H_{M}}, we obtain the second part of (2.11). □

The following theorem concerning the convexity property of the Csiszár f−divergence also holds.

Theorem 7. Assume that f satisfies the assumptions (i) and (ii) from Theorem 6. If p^{(j)}, q^{(j)} (j = 1, 2) are probability distributions satisfying (2.10), that is,

(2.14)  r ≤ \frac{p_{i}^{(j)}}{q_{i}^{(j)}} ≤ R for all i ∈ \{1, ..., n\} and j ∈ \{1, 2\} ,

then

(2.15)  r ≤ \frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ≤ R for all i ∈ \{1, ..., n\} and λ ∈ [0, 1]

and

4m[h^{2} (λp^{(1)} + (1 − λ) p^{(2)}, λq^{(1)} + (1 − λ) q^{(2)}) (2.16)

−λh^{2} (p^{(1)}, q^{(1)}) − (1 − λ) h^{2} (p^{(2)}, q^{(2)}) ]
≤ I_{f} (λp^{(1)} + (1 − λ) p^{(2)}, λq^{(1)} + (1 − λ) q^{(2)})
−λI_{f} (p^{(1)}, q^{(1)}) − (1 − λ) I_{f} (p^{(2)}, q^{(2)})
≤ 4M[h^{2} (λp^{(1)} + (1 − λ) p^{(2)}, λq^{(1)} + (1 − λ) q^{(2)})
−λh^{2} (p^{(1)}, q^{(1)}) − (1 − λ) h^{2} (p^{(2)}, q^{(2)}) ]

for all λ ∈ [0, 1].

Proof. By (2.14), we have

(2.17)  rλq_{i}^{(1)} ≤ λp_{i}^{(1)} ≤ λRq_{i}^{(1)} for all i ∈ {1, ..., n}

and

(2.18)  r (1 − λ) q_{i}^{(2)} ≤ (1 − λ) p_{i}^{(2)} ≤ R (1 − λ) q_{i}^{(2)} for all i ∈ {1, ..., n} .

Summing the above (2.17) and (2.18), we obtain (2.15).

It is already known that the mappings H_{m}, H_{M} as defined in Theorem 6 are convex and normalised.

Applying the “Joint Convexity Principle” for I_{H_{m}} (·, ·), i.e.,

I_{H_{m}} (λ (p^{(1)}, q^{(1)}) + (1 − λ) (p^{(2)}, q^{(2)})) (2.19)

≤ λI_{H_{m}} (p^{(1)}, q^{(1)}) + (1 − λ) I_{H_{m}} (p^{(2)}, q^{(2)})

and rearranging the terms, we end up with the first inequality in (2.16).

The second inequality follows likewise if we apply the same property to the Csiszár f−divergence I_{H_{M}} (·, ·).

We omit the details. □

Remark 1. If m > 0 in (2.9), then the inequality (2.11) is a better result than the positivity property of Csiszár divergence. The same will apply for the joint convexity of Csiszár divergence if m > 0.

Using the inequality (2.2) which holds for Φ, a differentiable convex and normalised function, for p, q probability distributions, we can state the following theorem as well. Theorem 8. Let f : [0, ∞) → ℝ be a normalised mapping. That is, f (1) = 0 and satisfies the assumption

(i) f is twice differentiable on (r, R), where 0 ≤ r ≤ 1 ≤ R ≤ ∞; (ii) there exists the constants m, M such that

(2.20)  m ≤ t^{\frac{3}{2}} f′′ (t) ≤ M for all t ∈ (r, R) .

If p, q are discrete probability distributions verifying the assumption

(2.21)  r ≤ r_{i} := \frac{p_{i}}{q_{i}} ≤ R for all i ∈ \{1, ..., n\},

then we have the inequality

I_{f′} (\frac{p^{2}}{q} , p) − I_{f′} (p, q) − 2MC (p, q) + 4Mh^{2} (p, q) (2.22)

≤ I_{f} (p, q)
≤ I_{f′} (\frac{p^{2}}{q} , p) − I_{f′} (p, q) − 2mC (p, q) + 4mh^{2} (p, q) ,

where C (p, q) := ∑_{i=1}^{n} (q_{i} − p_{i}) \sqrt{\frac{q_{i}}{p_{i}}}.

Proof. We know (see the proof of Theorem 6), that the mapping H_{m} : [0, ∞) → ℝ, H_{m} (t) := f (t) − 2m (\sqrt{t} − 1)^{2} is normalised, twice differentiable and convex on (r, R).

If we apply the second inequality from (2.2) for H_{m}, we may write

(2.23)  I_{H_{m}} (p, q) ≤ I_{H′_{m}} (\frac{p^{2}}{q} , p) − I_{H′_{m}} (−p, q) .

However,

I_{H_{m}} (p, q) = I_{f} (p, q) − 4mh^{2} (p, q) ,
I_{H′_{m}} (\frac{p^{2}}{q} , p) = I_{f′(·)−4m( \frac{1}{2} − \frac{1}{2\sqrt{·}} )} (\frac{p^{2}}{q} , p)
= I_{f′} (\frac{p^{2}}{q} , p) − 2m + 2mI_{\frac{1}{\sqrt{·}}} (\frac{p^{2}}{q} , p)

= I_{f′} (\frac{p^{2}}{q} , p) − 2m + 2m ∑_{i=1}^{n} p_{i} (\frac{1}{\sqrt{\frac{p_{i}^{2}}{q_{i}} · \frac{1}{p_{i}}}} )

= I_{f′} (\frac{p^{2}}{q} , p) − 2m + 2m ∑_{i=1}^{n} p_{i}\sqrt{\frac{q_{i}}{p_{i}}}

= I_{f′} (\frac{p^{2}}{q} , p) − 2m + 2m ∑_{i=1}^{n} \sqrt{p_{i}q_{i}}

and

I_{H′_{m}} (p, q) = I_{f′} (p, q) − 2m + 2mI_{\frac{1}{\sqrt{·}}} (p, q)

= I_{f′} (p, q) − 2m + 2m ∑_{i=1}^{n} q_{i} (\frac{1}{\sqrt{\frac{p_{i}}{q_{i}}}} )

= I_{f′} (p, q) − 2m + 2m ∑_{i=1}^{n} q_{i}\sqrt{\frac{q_{i}}{p_{i}}}

and then, by (2.23), we obtain

I_{f} (p, q) − 4mh^{2} (p, q)
≤ I_{f′} (\frac{p^{2}}{q} , p) − 2m + 2m ∑_{i=1}^{n} p_{i}\sqrt{\frac{q_{i}}{p_{i}}} − I_{f′} (p, q) + 2m − 2m ∑_{i=1}^{n} q_{i}\sqrt{\frac{q_{i}}{p_{i}}} ,

which is equivalent to the second inequality in (2.22).

If we consider H_{M} (t) := 2M (\sqrt{t} − 1)^{2} − f (t), t ≥ 0, then we observe that H_{M} (·) is normalised, twice differentiable and convex on (r, R). Applying the second inequality from (2.2), we deduce the first part of (2.22). □

The above results have natural applications when the Hellinger distance is compared with a number of other divergence measures arising in Information Theory.

3. Some Particular Cases

Using Theorem 6, we shall be able to point out the following particular cases which are of interest in Information Theory.

Proposition 1. Let p, q be two probability distributions with the property that

(3.1)  0 < r ≤ \frac{p_{i}}{q_{i}} =: q_{i} ≤ R < ∞ for all i ∈ \{1, ..., n\} .

Then we have the inequality

(3.2)  4\sqrt{r}h^{2} (p, q) ≤ D (p, q) ≤ 4\sqrt{R}h^{2} (p, q) .

Proof. Consider the mapping f : (0, ∞) → ℝ, f (t) = t ln t. Then

f′′ (t) = \frac{1}{t}, t ∈ (0, ∞) .

Consider the mapping g : [r, R] → ℝ, g (t) = t^{\frac{3}{2}} · \frac{1}{t} = t^{\frac{1}{2}} . Then

inf_{t∈[r,R]} g (t) = \sqrt{r}, sup_{t∈[r,R]} g (t) = \sqrt{R}.

Therefore, applying the inequality (2.11) with m = \sqrt{r}, M = \sqrt{R}, we obtain (3.2). □

Remark 2. The following inequality is well known in the literature (see for example Dacunha-Castelle [25]):

(3.3)  D (p, q) ≥ 2h^{2} (p, q)

for any p, q probability distributions. From the first inequality in (3.2) we have the inequality

(3.4)  D (p, q) ≥ 4\sqrt{r}h^{2} (p, q)

We remark that if 4\sqrt{r} ≥ 2, i.e., r ≥ \frac{1}{4}, then the inequality (3.4) is better than (3.3).

The following proposition also holds. Proposition 2. Let p, q be two probability distributions with the property (3.1). Then we have the inequality:

(3.5)  \frac{4}{\sqrt{R}}h^{2} (p, q) ≤ D (q, p) ≤ \frac{4}{\sqrt{r}} h^{2} (p, q) .

Proof. Consider the mapping f : [r, R] → ℝ, f (t) = − ln t. Define g (t) = t^{\frac{3}{2}} f′′ (t) = \frac{1}{\sqrt{t}}. Then obviously

sup_{t∈[r,R]} g (t) = \frac{1}{\sqrt{r}} , inf_{t∈[r,R]} g (t) = \frac{1}{\sqrt{R}}.

In addition,

I_{f} (p, q) = − ∑_{i=1}^{n} q_{i} ln (\frac{p_{i}}{q_{i}} ) = ∑_{i=1}^{n} q_{i} ln (\frac{q_{i}}{p_{i}} ) = D (q, p) .

Now, using the inequality (3.2), we get the desired inequality (3.5). □

The following result for the χ^{2}−distance also holds. Proposition 3. Let p, q be two probability distributions satisfying the condition (3.1). Then we have the inequality:

(3.6)  8r^{\frac{3}{2}} h^{2} (p, q) ≤ D_{χ^{2}} (p, q) ≤ 8R^{\frac{3}{2}} h^{2} (p, q) .

Proof. Consider the mapping f : (0, ∞) → ℝ, f (t) = (t − 1)^{2}. Define g : [r, R] → ℝ, g (t) = t^{\frac{3}{2}} f′′ (t) = 2t^{\frac{3}{2}} . Obviously,

sup_{t∈[r,R]} g (t) = 2R^{\frac{3}{2}} and inf_{t∈[r,R]} g (t) = 2r^{\frac{3}{2}} .

Since

I_{f} (p, q) = D_{χ^{2}} (p, q) ,

then, applying the inequality (2.11) with m = 2r^{\frac{3}{2}} , M = 2R^{\frac{3}{2}} , we get the desired inequality (3.6). □

Now, let us consider the J−divergence [26]

J (p, q) = ∑_{i=1}^{n} (p_{i} − q_{i}) ln (\frac{p_{i}}{q_{i}} ) .

The following proposition holds. Proposition 4. Let p, q be two probability distributions. Then we have the inequality

(3.7)  8h^{2} (p, q) ≤ J (p, q) .

Proof. Consider the mapping f : (0, ∞) → ℝ, f (t) = (t − 1) ln t. Define g : [r, R] → ℝ,

g (t) = t^{\frac{3}{2}} f′′ (t) = t^{\frac{1}{2}} + \frac{1}{t^{\frac{1}{2}}} ≥ 2,

which shows that

inf_{t∈(0,∞)} g (t) = 2.

Since

I_{f} (p, q) = J (p, q) ,

then, applying the inequality (2.11) with m = 2, we get the desired inequality. □

If we know more about r_{i} := \frac{p_{i}}{q_{i}} (i = 1, ..., n), i.e., the condition (3.1) holds, then we can point out an upper bound for the J (·, ·) as follows: Proposition 5. If 0 < r ≤ r_{i} ≤ R < ∞ for all i ∈ {1, ..., n}, then we have the inequality

(3.8)  J (p, q) ≤ 4 max \{\sqrt{r} + \frac{1}{\sqrt{r}} , \sqrt{R} + \frac{1}{\sqrt{R}}\} h^{2} (p, q) .

Proof. As above, we have

g (t) = t^{\frac{1}{2}} + \frac{1}{t^{\frac{1}{2}}} .

For the mapping h (u) = u + \frac{1}{u}, we have

h′ (u) = \frac{u^{2} − 1}{u^{2}},

which shows that the mapping is strictly decreasing on (0, 1) and strictly increasing on (1, ∞) Therefore

sup_{t∈[r,R]} g (t) = max [g (r) , g (R)] = max \{\sqrt{r} + \frac{1}{\sqrt{r}} , \sqrt{R} + \frac{1}{\sqrt{R}}\} .

Applyig Theorem 6 we deduce the desired result. □

Remark 3. We observe that

\sqrt{R} + \frac{1}{\sqrt{R}} − \sqrt{r} − \frac{1}{\sqrt{r}} = \begin{matrix}(\sqrt{R} − \sqrt{r}) & (\sqrt{rR} − 1) \\ √ & \overline{rR} \end{matrix}

and then the inequality (3.8) can be rewritten in the equivalent form as

(3.9)  J (p, q) ≤ 4h^{2} (p, q) × \{ \begin{matrix}\sqrt{R} + \frac{1}{\sqrt{R}} & if & R ≥ \frac{1}{r} \\ \sqrt{r} + \frac{1}{\sqrt{r}} & if & 1 ≤ R < \frac{1}{r} \end{matrix} .

Let us now consider the harmonic distance

M (p, q) := ∑_{i=0}^{n} \frac{2p_{i}q_{i}}{p_{i} + q_{i}} .

The following proposition holds.

Proposition 6. Let p, q be two probability distributions. Then we have the inequality

(3.10)  0 ≤ 1 − M (p, q) ≤ \frac{1}{2}h^{2} (p, q) .

Proof. Consider the function f : (0, ∞) → ℝ, f (t) = 1 − \frac{2t}{t+1}. Then

f′ (t) = −\frac{2}{(1 + t)^{2}} , f′′ (t) = \frac{4}{(t + 1)^{3}} .

Define the mapping

g (t) = t^{\frac{3}{2}} f′′ (t) = \frac{4t^{\frac{3}{2}}}{(t + 1)^{3}} .

A simple calculation shows that

g′ (t) = \frac{6\sqrt{t} (1 − t)}{(t + 1)^{4}}.

Consequently, the mapping g is increasing on the interval (0, 1) and decreasing on (1, ∞). Moreover,

sup_{t∈(0,∞)} = g (1) = \frac{1}{2} and I_{f} (p, q) = 1 − M (p, q) .

Applying the inequality (2.11) for M = \frac{1}{2}, we deduce (3.10). □

If we know more about r_{i} := \frac{p_{i}}{q_{i}}, that is, the condition (3.1) holds, then we can improve the first inequality in (3.10) as follows.

Proposition 7. Assume that the probability distributions p, q satisfy (3.1). Then we have the inequality

(3.11)  16 min \{\frac{r^{\frac{3}{2}}}{(r + 1)^{3}} , \frac{R^{\frac{3}{2}}}{(R + 1)^{3}} \} h^{2} (p, q) ≤ 1 − M (p, q) .

Proof. Taking into account that the mapping g (t) = \frac{4t^{\frac{3}{2}}}{(t+1)^{3}} is monotonic increasing on (0, 1) and decreasing on (1, ∞), we may assert that

inf_{t∈[r,R]} g (t) = min \{g (r) , g (R)\} = 4 min \{\frac{r^{\frac{3}{2}}}{(r + 1)^{3}} , \frac{R^{\frac{3}{2}}}{(R + 1)^{3}} \} .

Using the inequality (2.11), we deduce the desired lower bound (3.11). □

Remark 4. Similar results can be stated by Applying Theorem 8, but we omit the details.

References

[1] I. CSISZÁR, Information measures: A critical survey, Trans. 7th Prague Conf. on Info. Th., Statist. Decis. Funct., Random Processes and 8th European Meeting of Statist., Volume B, Academia Prague, 1978, 73-86.

[2] I. CSISZÁR, Information-type measures of difference of probability functions and indirect observations, Studia Sci. Math. Hungar, 2 (1967), 299-318.

[3] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[4] J.H. JUSTICE (editor), Maximum Entropy and Bayssian Methods in Applied Statistics, Cambridge University Press, Cambridge, 1986.

[5] J.N. KAPUR, On the roles of maximum entropy and minimum discrimination information principles in Statistics, Technical Address of the 38th Annual Conference of the Indian Society of Agricultural Statistics, 1984, 1-44.

[6] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy functions, IEEE Transactions on Information Theory, 28 (1982), 489-495.

[7] R.G. GALLAGER, Information Theory and Reliable Communications, J. Wiley, New York, 1968.

[8] C.E. SHANNON, A mathematical theory of communication, Bull. Sept. Tech. J., 27 (1948), 370-423 and 623-656.

[9] B.R. FRIEDEN, Image enhancement and restoration, Picture Processing and Digital Filtering (T.S. Huang, Editor), Springer-Verlag, Berlin, 1975.

[10] R.M. LEAHY and C.E. GOUTIS, An optimal technique for constraint-based image restoration and mensuration, IEEE Trans. on Acoustics, Speech and Signal Processing, 34 (1986), 1692-1642.

[11] S. KULLBACK, Information Theory and Statistics, J. Wiley, New York, 1959.

[12] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[13] R. BERAN, Minimum Hellinger distance estimates for parametric models, Ann. Statist., 5 (1977), 445-463.

[14] A. RENYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Statist. Prob., Vol. 1, University of California Press, Berkeley, 1961.

[15] S.S. DRAGOMIR and N.M. IONESCU, Some converse of Jensen’s inequality and applications, Anal. Num. Theor. Approx., 23 (1994), 71-78.

[16] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s discrete inequality for differentiable convex mappings and applications in information theory, Math. Comput. Modelling, 24 (2) (1996), 1-11.

[17] S.S. DRAGOMIR and C.J. GOH, Some counterpart inequalities in for a functional associated with Jensen’s inequality, J. of Ineq. & Appl., 1 (1997), 311-325.

[18] S.S. DRAGOMIR and C.J. GOH, Some bounds on entropy measures in information theory, Appl. Math. Lett., 10 (1997), 23-28.

[19] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s continuous inequality and applications in information theory, RGMIA Preprint,

http://matilda.vu.edu.au/~rgmia/InfTheory/Continuse.dvi

[20] M. MATIĆ, Jensen’s inequality and applications in Information Theory (in Croatian), Ph.D. Thesis, Univ. of Zagreb, 1999.

[21] S.S. DRAGOMIR, J. ŠUNDE and M. SCHOLZ, Some upper bounds for relative entropy and applications, Comp. and Math. with Appl. ( in press).

[22] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), No. 1, 1-16.

[23] D.S. MITRINOVIĆ, J.E. PEČARIĆ and A.M. FINK, Classical and New Inequalities in Analysis, Kluwer Academic Publishers, 1993.

[24] F. TOPSOE, Some inequalities for information divergence and related measures of discrimination, Res. Rep. Coll. RGMIA, 2 (1) (1999), 85-98.

[25] D. DACUNHA-CASTELLE, Ecole d’ete de Probabilités de Saint-Fleour, III-1997, Berlin, Heidelberg: Springer 1978.

[26] H. JEFFREYS, An invariant form for the prior probability in estimation problems, Proc. Roy. Soc. London, Ser. A, 1986 (1946), 453-461.

[27] A. BHATTACHARYYA, On a measure of divergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc., 35 (1943), 99-109.

[28] S.S. DRAGOMIR, Some inequalities for the Csiszár Φ-divergence, submitted.

[29] S.S. DRAGOMIR, A converse inequality for the Csiszár Φ-divergence, submitted.

[30] S.S. DRAGOMIR, Some inequalities for (m, M)-convex mappings and applications for the Csiszár Φ−divergence in information theory, submitted.

School of Communicactions and Informatics, Victoria Univeristy of Technology, PO Box 14428, Melbourne City MC, Victoria 8001, Australia.

   E-mail address: sever.dragomir@vu.edu.au
   URL: http://melba.vu.edu.au/~rgmia/SSdragomirWeb.html