A CONVERSE INEQUALITY FOR THE CSISZÁR Φ−DIVERGENCE

S.S. DRAGOMIR

Abstract. A converse inequality for the Csiszár Φ−divergence and applications for some particular distance functions such as: Kullback-Leibler distance, Renyi distance, χ^{2}−distance, Bhattacharyya distance, J-distance, etc. are given.

1. Introduction

Given a convex function Φ : ℝ_{+} → ℝ_{+}, the Φ- divergence functional

I_{Φ} (p, q) := ∑_{i=1}^{n} q_{i}Φ (\frac{p_{i}}{q_{i}} ) (1.1)

was introduced in Csiszár [1], [2] as a generalized measure of information, a “distance function” on the set of probability distributions ℙ^{n}. The restriction here to discrete distribution is only for convenience, similar results hold for general distributions.

As in Csiszár [2], we interpret undefined expressions by

Φ (0) = lim_{t→0+} Φ (t) , 0Φ (\frac{0}{0}) = 0
0Φ (\frac{a}{0}) = lim_{ε→0+} Φ (\frac{a}{ε}) = a lim_{t→∞} \frac{Φ (t)}{t}, a > 0.

The following results were essentially given by Csiszár and Körner [3].

Theorem 1. If Φ : ℝ_{+} → ℝ is convex, then I_{Φ} (p, q) is jointly convex in p and q.

The following lower bound for the Φ−divergence functional also holds.

Theorem 2. Let Φ : ℝ_{+} → ℝ_{+} be convex. Then for every p, q ∈ ℝ_{+}^{n}, we have the inequality:

I_{Φ} (p, q) ≥ ∑_{i=1}^{n} q_{i}Φ (\frac{∑_{i=1}^{n} p_{i}}{∑_{i=1}^{n} q_{i}} ) . (1.2)

If Φ is strictly convex, equality holds in (1.2) iff

\frac{p_{1}}{q_{1}} = \frac{p_{2}}{q_{2}} = ... = \frac{p_{n}}{q_{n}} . (1.3)

Date: September 7, 1999.

1991 Mathematics Subject Classification. Primary 94Xxx; Secondary 26D15.

Key words and phrases. The Csiszár Φ−Divergence, Kullback-Leibler distance, Rényi α−entropy.

2 S.S. DRAGOMIR

Corollary 1. Let Φ : ℝ_{+} → ℝ be convex and normalized, i.e., Φ (1) = 0. (1.4) Then for any p, q ∈ ℝ_{+}^{n} with

∑_{i=1}^{n} p_{i} = ∑_{i=1}^{n} q_{i} (1.5)

we have the inequality I_{Φ} (p, q) ≥ 0. (1.6) If Φ is strictly convex, the equality holds in (1.6) iff p_{i} = q_{i} for all i ∈ {1, ..., n}.

In particular, if p, q are probability vectors, then (1.5) is assured. Corollary 1 then shows, for strictly convex and normalized Φ : ℝ_{+} → ℝ, I_{Φ} (p, q) ≥ 0 for all p, q ∈ ℙ^{n}. (1.7) The equality holds in (1.7) iff p = q.

These are “distance properties”. However, I_{Φ} is not a metric: It violates the triangle inequality, and is asymmetric, i.e, for general p, q ∈ ℝ_{+}^{n}, I_{Φ} (p, q) ≠ I_{Φ} (q, p).

In the examples below we obtain, for suitable choices of the kernel Φ, some of the best known distance functions I_{Φ} used in mathematical statistics [4]-[5], information theory [6]-[8] and signal processing [9]-[10]. Example 1. (Kullback-Leibler) For Φ (t) := t log t, t > 0 (1.8) the Φ−divergence is

I_{Φ} (p, q) = ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) , (1.9)

the Kullback-Leibler distance [11]-[12]. Example 2. (Hellinger) Let

Φ (t) = \frac{1}{2} (1 − \sqrt{t})^{2} , t > 0. (1.10)

Then I_{Φ} gives the Hellinger distance [13]

I_{Φ} (p, q) = \frac{1}{2} ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} , (1.11)

which is symmetric. Example 3. (Renyi) For α > 1, let Φ (t) = t^{α}, t > 0. (1.12) Then

I_{Φ} (p, q) = ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}, (1.13)

which is the α−order entropy [14].

Example 4. (χ^{2}−distance) Let

Φ (t) = (t − 1)^{2} , t > 0. (1.14)

Then

I_{Φ} (p, q) = ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} (1.15)

is the χ^{2}−distance between p and q.

Finally, we have

Example 5. (Variational distance). Let Φ (t) = |t − 1| , t > 0. The corresponding divergence, called the variational distance, is symmetric,

I_{Φ} (p, q) = ∑_{i=1}^{n} |p_{i} − q_{i}| .

For other examples of divergence measures, see the paper [22] by J.N. Kapur, where further references are given.

2. Some Converse Inequalities for the Csiszár Φ−Divergence

We start with the following converse of Jensen’s discrete inequality established in 1994 in [15] by Dragomir and Ionescu.

Lemma 1. Let f : I ⊆ ℝ → ℝ be a differentiable convex mapping on I. If x_{i} ∈I̊,

t_{i} ≥ 0 with T_{n} := ∑_{i=1}^{n} t_{i} > 0 (i = 1, ..., n) , then we have the inequality

(2.1)  0 ≤ \frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}f (x_{i}) − f (\frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i})

≤ \frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i}f′ (x_{i}) − \frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i} · \frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}f′ (x_{i}) ;

where f′ :I̊→ ℝ is the derivative of f on I̊. If t_{i} > 0 (i = 1, ..., n) and the mapping f is strictly convex on I̊, then the case of equality holds in (2.1) iff x_{1} = x_{2} = ... = x_{n}.

Proof. As f : I → ℝ is convex on I̊, then we have the following inequality,

f (x) − f (y) ≥ f′ (y) (x − y) for all x, y ∈ I̊. (2.2)

Choose in (2.2) x = \frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i} ∈ I̊, y = x_{j}, j = 1, ...., n to obtain

f (\frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i}) − f (x_{j}) ≥ f′ (x_{j}) (\frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i} − x_{j}) (2.3)

for all j ∈ {1, ..., n} .

If we multiply (2.3) by t_{j} ≥ 0 and sum over j from 0 to n, we deduce

∑_{j=1}^{n} t_{j} [f (\frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i}) − f (x_{j})] (2.4)

≥ ∑_{j=1}^{n} t_{j}f′ (x_{j}) (\frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i} − x_{j}) .

However, a simple calculation shows that

∑_{j=1}^{n} t_{j} [f (\frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i}) − f (x_{j})] = T_{n}f (\frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i}) − ∑_{i=1}^{n} t_{i}f (x_{i})

and

∑_{j=1}^{n} t_{j}f′ (x_{j}) (\frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i} − x_{j})

= \frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i} ∑_{i=1}^{n} t_{i}f′ (x_{i}) − ∑_{i=1}^{n} t_{i}x_{i}f′ (x_{i})

and then, by (2.4), we can state

T_{n}f (\frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i}) − ∑_{i=1}^{n} t_{i}f (x_{i})

≥ \frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i} ∑_{i=1}^{n} t_{i}f′ (x_{i}) − ∑_{i=1}^{n} t_{i}x_{i}f′ (x_{i}) .

Dividing by T_{n} > 0, we obtain the second inequality in (2.1).

If f is strictly convex, then the equality holds in (2.2) iff x = y. Using this and an obvious argument, we obtain the fact that the equality holds in (2.1) iff x_{1} = ... = x_{n}.

We omit the details.

Remark 1. If, in the above lemma we drop the differentiability condition, then can choose instead of f′ (x_{i}) any number l = l (x_{i}) ∈ [f′_{−} (x_{i}) , f′_{+} (x_{i})] and the inequality still remains valid. This follows by the fact that for a convex mapping f : ℝ_{+} → ℝ_{+}, we have

f (x) − f (y) ≥ l (y) (x − y) for x, y ∈ ℝ_{+},

where l (y) ∈ [f′_{−} (y) , f′_{+} (y)] and f′_{−} (y) is the left derivative of f in y and f′_{+} (y) is the right derivative of f in y. We omit the details.

Remark 2. For an extension of this theorem for convex mappings of several variables see the paper [16] by Dragomir and Goh where further applications in Information Theory for Shannon’s Entropy, Mutual Information, Conditional Entropy, etc. are given. An integral version of this result can be found in the paper [19] by Dragomir and Goh where further applications for the continuous case of the Shannon Entropy have been given. Extensions of the above results for convex mappings defined on convex sets in linear spaces and particularly in normed spaces can be found in the recent Ph.D. Dissertation [20] of M. Matić where other applications in Information Theory for Shannon’s Entropy have been considered.

A CONVERSE INEQUALITY 5

In the recent paper [23], we proved the following theorem. Theorem 3. Let Φ : ℝ_{+} → ℝ_{+} be differentiable convex. Then for all p, q ∈ ℝ_{+}^{n} we have the inequality

Φ′ (1) (P_{n} − Q_{n}) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ I_{Φ′} (\frac{p^{2}}{q} , q) − I_{Φ′} (p, q) , (2.5)

where P_{n} := ∑_{i=1}^{n} p_{i} > 0, Q_{n} := ∑_{i=1}^{n} q_{i} > 0 and Φ′ is the derivative of Φ. If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n), then the equality holds in (2.5) iff p = q.

The following converse inequality for the Csiszár Φ−divergence also holds. Theorem 4. Let Φ, p and q be as in Theorem 2. Then we have the inequality:

(2.6)  0 ≤ I_{Φ} (p, q) − Q_{n}Φ (\frac{P_{n}}{Q_{n}} )
≤ I_{Φ′} (\frac{p^{2}}{q} , p) − \frac{P_{n}}{Q_{n}} I_{Φ′} (p, q) ,

where Q_{n} := ∑_{i=1}^{n} q_{i} > 0, P_{n} := ∑_{i=1}^{n} p_{i} > 0 and Φ′ is the derivative of Φ. If Φ is strictly convex, and p_{i}, q_{i} > 0, (i = 1, ..., n), then the equality holds in (2.6) iff \frac{p_{1}}{q_{1}} = ... = \frac{p_{n}}{q_{n}} . Proof. Choose in Lemma 1 f = Φ, t_{i} := q_{i} and x_{i} = \frac{p_{i}}{q_{i}} to obtain

0 ≤ \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i}Φ (\frac{p_{i}}{q_{i}} ) − Φ (\frac{P_{n}}{Q_{n}} )
≤ \frac{1}{Q_{n}} ∑_{i=1}^{n} p_{i}I_{Φ′} (\frac{p_{i}}{q_{i}} ) − \frac{1}{Q_{n}} ∑_{i=1}^{n} p_{i} · \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i}I_{Φ′} (\frac{p_{i}}{q_{i}} ) ,

which is equivalent to (2.6). The case of equality is obvious.

The following corollary is natural. Corollary 2. Let Φ : ℝ_{+} → ℝ be differentiable convex and normalized. Then, for any p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}, we have the converse of the positivity inequality (1.6)

0 ≤ I_{Φ} (p, q) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) . (2.7)

The equality holds in (2.7) for a strictly convex mapping Φ iff p = q.

3. Applications for some Particular Φ−Divergences Let us consider the Kullback-Leibler distance given by (1.9)

KL (p, q) := ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) . (3.1)

Choose the convex mapping Φ (t) = − log (t) , t > 0. For this mapping we have the Csiszár Φ−divergence

I_{Φ} (p, q) = ∑_{i=1}^{n} q_{i} [− log (\frac{p_{i}}{q_{i}} )] = ∑_{i=1}^{n} q_{i} log (\frac{q_{i}}{p_{i}} ) = KL (q, p) .

6 S.S. DRAGOMIR

The following inequality holds. Proposition 1. Let p, q ∈ ℝ_{+}^{n}. Then we have the inequality

(3.2)  0 ≤ KL (q, p) − Q_{n} log (\frac{Q_{n}}{P_{n}} ) ≤ \frac{1}{Q_{n}} [P_{n} ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} − Q_{n}^{2}]
= \frac{1}{2Q_{n}} ∑_{i,j=1}^{n} p_{i}p_{j} (\frac{q_{i}}{p_{i}} − \frac{q_{j}}{p_{j}} )^{2} .

The case of equality holds iff \frac{p_{1}}{q_{1}} = ... = \frac{p_{n}}{q_{n}} . Proof. If Φ (t) = − log t, then Φ′ (t) = −\frac{1}{t}, t > 0. We have

I_{Φ′} (\frac{p^{2}}{q} , p) = ∑_{i=1}^{n} p_{i} · [\frac{1}{(\frac{p_{i}^{2}}{q_{i}} ) · \frac{1}{p_{i}}} ] = −Q_{n},
I_{Φ′} (p, q) = ∑_{i=1}^{n} q_{i} · [− \frac{1}{\frac{p_{i}}{q_{i}}} ] = − ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}}

and then from (2.6), we can state that

0 ≤ KL (q, p) + Q_{n} log (\frac{P_{n}}{Q_{n}} )
≤ −Q_{n} + \frac{P_{n}}{Q_{n}} ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} = \frac{1}{Q_{n}} [P_{n} ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} − Q_{n}^{2}] ,

i.e.,

0 ≤ KL (q, p) − Q_{n} log (\frac{Q_{n}}{P_{n}} ) ≤ \frac{1}{Q_{n}} [P_{n} ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} − Q_{n}^{2}] .

On the other hand, we observe that

\frac{1}{2} ∑_{i,j=1}^{n} p_{i}p_{j} (\frac{q_{i}}{p_{i}} − \frac{q_{j}}{p_{j}} )^{2} = \frac{1}{2} ∑_{i,j=1}^{n} p_{i}p_{j} (\frac{q_{i}^{2}}{p_{i}^{2}} − 2\frac{q_{i}q_{j}}{p_{i}p_{j}} + \frac{q_{j}^{2}}{p_{j}^{2}} )
= \frac{1}{2} [ ∑_{i,j=1}^{n} p_{j} \frac{q_{i}^{2}}{p_{i}} − 2 ∑_{i,j=1}^{n} q_{i}q_{j} + ∑_{i,j=1}^{n} p_{i} \frac{q_{j}^{2}}{p_{j}} ]
= P_{n} ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} − Q_{n}^{2}

and the inequality (3.2) is proved.

The case of equality follows by the strict convexity of the mapping − log. Corollary 3. If we assume that P_{n} = Q_{n} in (3.2), we have

0 ≤ KL (q, p) ≤ ∑_{i=1}^{n} [\frac{q_{i}^{2} − p_{i}^{2}}{p_{i}} ] = \frac{1}{2Q_{n}} ∑_{i,j=1}^{n} p_{i}p_{j} (\frac{q_{i}}{p_{i}} − \frac{q_{j}}{p_{j}} )^{2} (3.3)

with equality iff p = q.

Remark 3. We know that the χ^{2}−distance between p and q is (see (1.15))

D_{χ}^{2} (p, q) := ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} = (∑_{i=1}^{n} \frac{p_{i}^{2} − q_{i}^{2}}{q_{i}} if P_{n} = Q_{n}) .

As

D_{χ}^{2} (q, p) = ∑_{i=1}^{n} \frac{(q_{i} − p_{i})^{2}}{p_{i}} = (∑_{i=1}^{n} \frac{q_{i}^{2} − p_{i}^{2}}{p_{i}} if P_{n} = Q_{n}) ,

then the inequality (3.3) can be rewritten in the following manner

0 ≤ KL (q, p) ≤ D_{χ}^{2} (q, p) . (3.4)

Corollary 4. Let p and q be two probability distributions. Then

0 ≤ KL (q, p) ≤ D_{χ}^{2} (q, p) (3.5)

with equality iff p = q.

Remark 4. For a direct proof of the inequality (3.5) see [21] where further bounds are also given.

Now, let us consider the α−order entropy of Renyi (see (1.13))

D_{α} (p, q) := ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}, α > 1 (3.6)

and p, q ∈ ℝ_{+}^{n}.

We know that Renyi’s entropy is actually the Csiszár Φ−divergence for the convex mapping Φ (t) = t^{α}, α > 1, t > 0 (see Example 3).

The following proposition holds.

Proposition 2. Let p, q ∈ ℝ_{+}^{n}. Then we have the inequality

(3.7)  0 ≤ D_{α} (p, q) − P_{n}^{α}Q_{n}^{1−α}
≤ \frac{α}{Q_{n}} [Q_{n}D_{α} (p, q) − P_{n}D_{α} (q^{\frac{2−α}{α}} , \frac{1}{p})] .

The case of equality holds iff \frac{p_{1}}{q_{1}} = ... = \frac{p_{n}}{q_{n}} .

Proof. If Φ (t) = t^{α}, then Φ′ (t) = αt^{α−1}.

We have

(3.8)  I_{Φ′} (\frac{p^{2}}{q} , p) = ∑_{i=1}^{n} p_{i} · [α (\frac{p_{i}^{2}}{q_{i}p_{i}} )^{α−1}]

= α ∑_{i=1}^{n} p_{i} (\frac{p_{i}}{q_{i}} )^{α−1} = α ∑_{i=1}^{n} q_{i}^{1−α}p_{i}^{α}
= αD_{α} (p, q)

and

I_{Φ′} (p, q) = ∑_{i=1}^{n} q_{i} [α (\frac{p_{i}}{q_{i}} )^{α−1}] = ∑_{i=1}^{n} p_{i}^{α−1}q_{i}^{2−α} = αD_{α} (q^{\frac{2−α}{α}} , \frac{1}{p}) . (3.9)

Using the inequality (2.6), we can state that

0 ≤ D_{α} (p, q) − Q_{n} (\frac{P_{n}}{Q_{n}} )^{α}
≤ αD_{α} (p, q) − α\frac{P_{n}}{Q_{n}} D_{α} (q^{\frac{2−α}{α}} , \frac{1}{p})
= \frac{α}{Q_{n}} [Q_{n}D_{α} (p, q) − P_{n}D_{α} (q^{\frac{2−α}{α}} , \frac{1}{p})]

and the inequality (3.7) is obtained.

The case if equality follows by the fact that the mapping Φ (t) = t^{α} (α > 1, t > 0) is strictly convex on (0, ∞) .

Corollary 5. If we assume that P_{n} = Q_{n} in (3.7), we obtain

0 ≤ D_{α} (p, q) − P_{n} ≤ α [D_{α} (p, q) − D_{α} (q^{\frac{2−α}{α}} , \frac{1}{p})] (3.10)

with equality iff p = q. In particular, if p, q are probability distributions, then

0 ≤ D_{α} (p, q) − 1 ≤ α [D_{α} (p, q) − D_{α} (q^{\frac{2−α}{α}} , \frac{1}{p})] (3.11)

with equality iff p = q.

Now, let us consider the Bhattacharyya distance given by (see [20])

B (p, q) = ∑_{i=1}^{n} \sqrt{p_{i}q_{i}}.

If we consider the convex mapping Φ : (0, ∞) → ℝ, Φ (t) = −\sqrt{t}, then

I_{Φ} (p, q) = − ∑_{i=1}^{n} q_{i}\sqrt{\frac{p_{i}}{q_{i}}} = − ∑_{i=1}^{n} \sqrt{p_{i}q_{i}} = −B (p, q) .

The following inequality holds.

Proposition 3. Let p, q ∈ ℝ_{+}^{n}. Then we have the inequality

0 ≤ \sqrt{Q_{n}P_{n}} − B (p, q) ≤ \frac{1}{2} [\frac{P_{n}}{Q_{n}} ∑_{i=1}^{n} q_{i}\sqrt{\frac{q_{i}}{p_{i}}} − B (p, q)] . (3.12)

The case of equality holds iff \frac{p_{1}}{q_{1}} = ... = \frac{p_{n}}{q_{n}}.

Proof. If Φ (t) = −\sqrt{t}, then Φ′ (t) = −\frac{1}{2} · \frac{1}{\sqrt{t}}, t > 0. We have

I_{Φ′} (\frac{p^{2}}{q} , p) = ∑_{i=1}^{n} p_{i} (−\frac{1}{2} · \frac{1}{\sqrt{\frac{p_{i}^{2}}{q_{i}p_{i}}}} ) = −\frac{1}{2} ∑_{i=1}^{n} \sqrt{q_{i}p_{i}} = −\frac{1}{2}B (p, q) ,

I_{Φ′} (p, q) = ∑_{i=1}^{n} q_{i} (−\frac{1}{2} · \frac{1}{\sqrt{\frac{p_{i}}{q_{i}}}} ) = −\frac{1}{2} ∑_{i=1}^{n} q_{i}\sqrt{\frac{q_{i}}{p_{i}}}

and then, from (2.6), we can state that

0 ≤ −B (p, q) + Q_{n}\sqrt{\frac{P_{n}}{Q_{n}}} ≤ \frac{1}{2}B (p, q) + \frac{1}{2} \frac{P_{n}}{Q_{n}} ∑_{i=1}^{n} q_{i}\sqrt{\frac{q_{i}}{p_{i}}} ,

which is equivalent to (3.12).

The case of equality follows by the strict convexity of the mapping Φ.

Remark 5. The second inequality in (3.12) is equivalent to

\sqrt{P_{n}Q_{n}} ≤ \frac{1}{2} [\frac{P_{n}}{Q_{n}} ∑_{i=1}^{n} q_{i}\sqrt{\frac{q_{i}}{p_{i}}} + B (p, q)] (3.13)

with equality iff \frac{p_{1}}{q_{1}} = ... = \frac{p_{n}}{q_{n}}.

Corollary 6. If we assume that P_{n} = Q_{n} in (3.12), then we obtain

0 ≤ P_{n} − B (p, q) ≤ \frac{1}{2} [∑_{i=1}^{n} q_{i}\sqrt{\frac{q_{i}}{p_{i}}} − B (p, q)]

with equality iff p = q.

Another important divergence measure in Information Theory is the J-divergence defined by (see for example [2])

J (p, q) = ∑_{i=1}^{n} (p_{i} − q_{i}) log (\frac{p_{i}}{q_{i}} ) .

Let us observe that the mapping Φ (t) := (t − 1) ln t, t ∈ (0, ∞) , has the derivatives

Φ′ (t) = ln t − \frac{1}{t} + 1, t > 0
Φ′′ (t) = \frac{t + 1}{t^{2}} > 0 for t > 0,

which shows that Φ is convex on (0, 1) and

I_{Φ} (p, q) = ∑_{i=1}^{n} q_{i} (\frac{p_{i}}{q_{i}} − 1) ln (\frac{p_{i}}{q_{i}} ) = J (p, q) .

We can state the following proposition.

Proposition 4. Let p, q ∈ ℝ_{+}^{n}. Then we have the inequality

(3.14)  0 ≤ J (p, q) − (P_{n} − Q_{n}) ln (\frac{P_{n}}{Q_{n}} )

≤ KL (p, q) + \frac{P_{n}}{Q_{n}} KL (q, p) + \frac{P_{n}}{Q_{n}} ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} − Q_{n}.

The case of equality holds iff \frac{p_{1}}{q_{1}} = ... = \frac{p_{n}}{q_{n}}.

Proof. We have

I_{Φ′} (\frac{p^{2}}{q} , p) = ∑_{i=1}^{n} p_{i} [ln (\frac{p_{i}^{2}}{q_{i}p_{i}} ) − \frac{q_{i}p_{i}}{p_{i}^{2}} + 1]

= ∑_{i=1}^{n} p_{i} ln (\frac{p_{i}}{q_{i}} ) − Q_{n} + P_{n}
= KL (p, q) − Q_{n} + P_{n},
I_{Φ′} (p, q) = ∑_{i=1}^{n} q_{i} [ln (\frac{p_{i}}{q_{i}} ) − \frac{q_{i}}{p_{i}} + 1]

= −KL (q, p) − ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} + Q_{n}

and then, from (2.6), we can state that

0 ≤ J (p, q) − Q_{n} (\frac{P_{n}}{Q_{n}} − 1) ln (\frac{P_{n}}{Q_{n}} )

≤ KL (p, q) − Q_{n} + P_{n} − \frac{P_{n}}{Q_{n}} [−KL (q, p) − ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} + Q_{n}]

= KL (p, q) + P_{n} + \frac{P_{n}}{Q_{n}} ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} − P_{n} − Q_{n}

= KL (p, q) + \frac{P_{n}}{Q_{n}} KL (q, p) + \frac{P_{n}}{Q_{n}} ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} − Q_{n}

and the inequality (3.14) is obtained.

4. Further Bounds via the Discrete Grüss Inequality

The following inequality is well known in the literature as the weighted discrete Grüss inequality.

Lemma 2. Let a_{i}, b_{i}, p_{i} ∈ ℝ (i = 1, ..., n) be such that p_{i} ≥ 0 with P_{n} := ∑_{i=1}^{n} p_{i} > 0

and

a ≤ a_{i} ≤ A, b ≤ b_{i} ≤ B for all i ∈ {1, ..., n} . (4.1)

Then we have the inequality

|\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}a_{i}b_{i} − \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}a_{i} · \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}b_{i}| ≤ \frac{1}{4} (A − a) (B − b) (4.2)

and the constant \frac{1}{4} is the best possible one in the sense that it cannot be replaced by a smaller constant independent of n.

Using this result, we can point out the following counterpart of Jensen’s inequality.

Lemma 3. Let f, x_{i}, t_{i} be as in Lemma 1. If there exist the real constants m, M ∈I̊ such that m ≤ x_{i} ≤ M for all i ∈ {1, ..., n} , then we have the inequality:

0 ≤ \frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}f (x_{i}) − f (\frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i}) ≤ \frac{1}{4} (M − m) (f′ (M) − f′ (m)) , (4.3)

where f′ :I̊→ ℝ is the derivative of f.

Proof. Define in Grüss’ inequality a_{i} := x_{i} and b_{i} := f′ (x_{i}) . As f is convex, f′ is monotonic nondecreasing and therefore f′ (m) ≤ b_{i} ≤ f′ (M) . Applying the Grüss inequality, we can state that

\frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i}f′ (x_{i}) − \frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}x_{i} · \frac{1}{T_{n}} ∑_{i=1}^{n} t_{i}f′ (x_{i})

≤ \frac{1}{4} (M − m) (f′ (M) − f′ (m))

and by the inequality (2.1) we deduce (4.3).

Remark 6. Similar results can be obtained by using other Grüss type inequalities. However, we omit the details.

The following converse inequality for the Csiszár Φ−divergence holds.

Theorem 5. Let Φ, p and q be as in Theorem 2. If there exist the real numbers r, R such that 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ for all i ∈ \{1, ..., n\} , then we have the inequality

0 ≤ I_{Φ} (p, q) − Q_{n}Φ (\frac{P_{n}}{Q_{n}} ) ≤ \frac{1}{4} (R − r) (Φ′ (R) − Φ′ (r)) Q_{n}. (4.4)

Proof. Apply Lemma 3 for f = Φ, t_{i} = q_{i} and x_{i} = \frac{p_{i}}{q_{i}} (i = 1, ..., n) .

The following particular inequalities can be noted as well:

(4.5) 0 ≤ KL (p, q) − P_{n} log (\frac{P_{n}}{Q_{n}} ) ≤ \frac{P_{n}}{4} (R − r) [log (R) − log (r)]
≤ \frac{P_{n}}{4} · \begin{matrix}(R − r) & 2 \\ √ & \overline{Rr} \end{matrix} .

Indeed, if we choose Φ (t) = t log t in (4.4), we obtain the Kullback-Leibler divergence. The last inequality in (4.5) follows by the well known inequality between the geometric mean G (a, b) = \sqrt{ab} and the logarithmic mean L (a, b) := \frac{b−a}{log b−log a} (a, b > 0, a ≠ b) which says that

L (a, b) ≥ G (a, b) for all a, b > 0, a ≠ b. (4.6)

In addition, if in (4.4) we put Φ (t) = − log t, we deduce

0 ≤ KL (q, p) − Q_{n} log (\frac{Q_{n}}{P_{n}} ) ≤ \frac{Q_{n}}{4} · \frac{(R − r)^{2}}{Rr} , (4.7)

provided that 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ for all i ∈ \{1, ..., n\} .

Now, if in (4.4) we choose Φ (t) = t^{α} (α > 1) , t > 0, then we get the following inequality for the α−order Renyi entropy

0 ≤ D_{α} (p, q) − P_{n}^{α}Q_{n}^{1−α} ≤ \frac{α}{4} · Q_{n} (R − r) (R^{α−1} − r^{α−1}) , (4.8)

provided that 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ for all i ∈ \{1, ..., n\} .

If we apply Theorem 5 for the Bhattacharyya distance, we get the inequality

0 ≤ \sqrt{P_{n}Q_{n}} − B (p, q) ≤ \frac{1}{8} · \begin{matrix}(R − r) & (\sqrt{R} − \sqrt{r}) \\ √ & \overline{rR} \end{matrix} · Q_{n}, (4.9)

provided that 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ for all i ∈ \{1, ..., n\}.

Finally, if we apply Theorem 5 for the J-divergence, we may obtain the inequality

0 ≤ J (p, q) − (P_{n} − Q_{n}) ln \frac{P_{n}}{Q_{n}} ≤ \frac{1}{4} (R − r) (ln \frac{R}{r} + \frac{R − r}{rR} ) Q_{n}, (4.10)

provided that 0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞ for all i ∈ \{1, ..., n\}.

References

[1] I. CSISZÁR, Information measures: A critical survey, Trans. 7th Prague Conf. on Info. Th., Statist. Decis. Funct., Random Processes and 8th European Meeting of Statist., Volume B, Academia Prague, 1978, 73-86.

[2] I. CSISZÁR, Information-type measures of difference of probability functions and indirect observations, Studia Sci. Math. Hungar, 2 (1967), 299-318.

[3] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[4] J.H. JUSTICE (editor), Maximum Entropy and Bayssian Methods in Applied Statistics, Cambridge University Press, Cambridge, 1986.

[5] J.N. KAPUR, On the roles of maximum entropy and minimum discrimination information principles in Statistics, Technical Address of the 38th Annual Conference of the Indian Society of Agricultural Statistics, 1984, 1-44.

[6] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy functions, IEEE Transactions on Information Theory, 28 (1982), 489-495.

[7] R.G. GALLAGER, Information Theory and Reliable Communications, J. Wiley, New York, 1968.

[8] C.E. SHANNON, A mathematical theory of communication, Bull. Sept. Tech. J., 27 (1948), 370-423 and 623-656.

[9] B.R. FRIEDEN, Image enhancement and restoration, Picture Processing and Digital Filtering (T.S. Huang, Editor), Springer-Verlag, Berlin, 1975.

[10] R.M. LEAHY and C.E. GOUTIS, An optimal technique for constraint-based image restoration and mensuration, IEEE Trans. on Acoustics, Speech and Signal Processing, 34 (1986), 1692-1642.

[11] S. KULLBACK, Information Theory and Statistics, J. Wiley, New York, 1959.

[12] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[13] R. BERAN, Minimum Hellinger distance estimates for parametric models, Ann. Statist., 5 (1977), 445-463.

[14] A. RENYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Statist. Prob., Vol. 1, University of California Press, Berkeley, 1961.

[15] S.S. DRAGOMIR and N.M. IONESCU, Some converse of Jensen’s inequality and applications, Anal. Num. Theor. Approx., 23 (1994), 71-78.

[16] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s discrete inequality for differentiable convex mappings and applications in information theory, Math. Comput. Modelling, 24 (2) (1996), 1-11.

[17] S.S. DRAGOMIR and C.J. GOH, Some counterpart inequalities in for a functional associated with Jensen’s inequality, J. of Ineq. & Appl., 1 (1997), 311-325.

[18] S.S. DRAGOMIR and C.J. GOH, Some bounds on entropy measures in information theory, Appl. Math. Lett., 10 (1997), 23-28.

[19] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s continuous inequality and applications in information theory, RGMIA Preprint,

http://matilda.vu.edu.au/~rgmia/InfTheory/Continuse.dvi

[20] M. MATIĆ, Jensen’s inequality and applications in Information Theory (in Croatian), Ph.D. Thesis, Univ. of Zagreb, 1999.

[21] S.S. DRAGOMIR, J. ŠUNDE and M. SCHOLZ, Some upper bounds for relative entropy and applications, Comp. and Math. with Appl. ( in press).

[22] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), No. 1, 1-16.

[23] S.S. DRAGOMIR, Some inequalities for the Csiszár Φ-divergence, submitted.

School of Commuincations and Informatics, Victoria University of Technology, PO Box 14428, Melbourne City MC 8001, Victoria, Australia

   E-mail address: sever@matilda.vu.edu.au
   URL: http://rgmia.vu.edu.au/SSDragomirWeb.html