SOME INEQUALITIES FOR THE CSISZÁR Φ-DIVERGENCE

S.S. DRAGOMIR

Abstract. Some inequalities for the Csiszár Φ-divergence and applications for the Kullback-Leibler, Rényi, Hellinger and Bhattacharyya distances in Information Theory are given.

1. Introduction

Given a convex function Φ : ℝ_{+} → ℝ_{+}, the Φ-divergence functional

I_{Φ} (p, q) := ∑_{i=1}^{n} q_{i}Φ (\frac{p_{i}}{q_{i}} ) (1.1)

was introduced in Csiszár [1], [2] as a generalized measure of information, a “distance function” on the set of probability distributions ℙ^{n}. The restriction here to discrete distribution is only for convenience, similar results hold for general distributions.

As in Csiszár [2], we interpret undefined expressions by

Φ (0) = lim_{t→0+} Φ (t) , 0Φ (\frac{0}{0}) = 0,
0Φ (\frac{a}{0}) = lim_{ε→0+} Φ (\frac{a}{ε}) = a lim_{t→∞} \frac{Φ (t)}{t}, a > 0.

The following results were essentially given by Csiszár and Körner [3].

Theorem 1. If Φ : ℝ_{+} → ℝ is convex, then I_{Φ} (p, q) is jointly convex in p and q.

The following lower bound for the Φ−divergence functional also holds.

Theorem 2. Let Φ : ℝ_{+} → ℝ_{+} be convex. Then for every p, q ∈ ℝ_{+}^{n}, we have the inequality:

I_{Φ} (p, q) ≥ ∑_{i=1}^{n} q_{i}Φ (\frac{∑_{i=1}^{n} p_{i}}{∑_{i=1}^{n} q_{i}} ) . (1.2)

If Φ is strictly convex, equality holds in (1.2) iff

\frac{p_{1}}{q_{1}} = \frac{p_{2}}{q_{2}} = ... = \frac{p_{n}}{q_{n}} . (1.3)

Date: November 16, 1999.

1991 Mathematics Subject Classification. Primary 94Xxx; Secondary 26D15.

Key words and phrases. The Csiszár Φ-Divergence, Kullback-Leibler distance, Rényi α−entropy, Hellinger discrimination, Bhattacharyya distance.

2 S.S. DRAGOMIR

Corollary 1. Let Φ : ℝ_{+} → ℝ be convex and normalized, i.e., Φ (1) = 0. (1.4) Then for any p, q ∈ ℝ_{+}^{n} with

∑_{i=1}^{n} p_{i} = ∑_{i=1}^{n} q_{i}, (1.5)

we have the inequality I_{Φ} (p, q) ≥ 0. (1.6) If Φ is strictly convex, equality holds in (1.6) iff p_{i} = q_{i} for all i ∈ {1, ..., n}.

In particular, if p, q are probability vectors, then (1.5) is assured. Corollary 1 then shows, for strictly convex and normalized Φ : ℝ_{+} → ℝ, that I_{Φ} (p, q) ≥ 0 for all p, q ∈ ℙ^{n}. (1.7) The equality holds in (1.7) iff p = q.

These are “distance properties”. However, I_{Φ} is not a metric: It violates the triangle inequality, and is asymmetric, i.e, for general p, q ∈ ℝ_{+}^{n}, I_{Φ} (p, q) ≠ I_{Φ} (q, p).

In the examples below we obtain, for suitable choices of the kernel Φ, some of the best known distance functions I_{Φ} used in mathematical statistics [4]-[5], information theory [6]-[8] and signal processing [9]-[10]. Example 1. (Kullback-Leibler) For Φ (t) := t log t, t > 0; (1.8) the Φ−divergence is

I_{Φ} (p, q) = ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) , (1.9)

the Kullback-Leibler distance [11]-[12]. Example 2. (Hellinger) Let

Φ (t) = \frac{1}{2} (1 − \sqrt{t})^{2} , t > 0. (1.10)

Then I_{Φ} gives the Hellinger distance [13]

I_{Φ} (p, q) = \frac{1}{2} ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} , (1.11)

which is symmetric. Example 3. (Renyi) For α > 1, let Φ (t) = t^{α}, t > 0. (1.12) Then

I_{Φ} (p, q) = ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}, (1.13)

which is the α−order entropy [14].

Example 4. (χ^{2}−distance) Let

Φ (t) = (t − 1)^{2} , t > 0. (1.14)

Then

I_{Φ} (p, q) = ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} (1.15)

is the χ^{2}−distance between p and q.

Finally, we have

Example 5. (Variation distance). Let Φ (t) = |t − 1| , t > 0. The corresponding divergence, called the variation distance, is symmetric,

I_{Φ} (p, q) = ∑_{i=1}^{n} |p_{i} − q_{i}| .

For other examples of divergence measures, see the paper [22] by J.N. Kapur, where further references are given.

2. Other Inequalities for the Csiszár Φ-Divergence

We start with the following result.

Theorem 3. Let Φ : ℝ_{+} → ℝ be differentiable convex. Then for all p, q ∈ ℝ_{+}^{n} we have the inequality

Φ′ (1) (P_{n} − Q_{n}) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) , (2.1)

where P_{n} := ∑_{i=1}^{n} p_{i} > 0, Q_{n} := ∑_{i=1}^{n} q_{i} > 0 and Φ′ : (0, ∞) → ℝ is the derivative of

Φ. If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n), then the equality holds in (2.1) iff p = q.

Proof. As Φ is differentiable convex on ℝ_{+}, then we have the inequality

Φ′ (y) (y − x) ≥ Φ (y) − Φ (x) ≥ Φ′ (x) (y − x) (2.2)

for all x, y ∈ ℝ_{+}.

Choose in (2.2) y = \frac{p_{i}}{q_{i}} and x = 1, to obtain

Φ′ (\frac{p_{i}}{q_{i}} ) (\frac{p_{i}}{q_{i}} − 1) ≥ Φ (\frac{p_{i}}{q_{i}} ) − Φ (1) ≥ Φ′ (1) (\frac{p_{i}}{q_{i}} − 1) (2.3)

for all i ∈ {1, ..., n}.

Now, if we multiply (2.3) by q_{i} ≥ 0 (i = 1, ..., n) and sum over i from 1 to n, we can deduce

∑_{i=1}^{n} (p_{i} − q_{i}) Φ′ (\frac{p_{i}}{q_{i}} ) ≥ I_{Φ} (p, q) − Q_{n}Φ (1) ≥ Φ′ (1) (P_{n} − Q_{n})

and as

∑_{i=1}^{n} (p_{i} − q_{i}) Φ′ (\frac{p_{i}}{q_{i}} ) = I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) ,

the inequality in (2.1) is thus obtained.

The case of equality holds in (2.2) for a strictly convex mapping iff x = y and so the equality holds in (2.1) iff \frac{p_{i}}{q_{i}} = 1 for all i ∈ \{1, ..., n\}, and the theorem is proved.

Remark 1. In the above theorem, if we would like to drop the differentiability condition, we can choose instead of Φ′ (x) any number l = l (x) ∈ [Φ′_{−} (x) , Φ′_{+} (x)] and the inequality will still be valid. This follows by the fact that for the convex mapping Φ : ℝ_{+} → ℝ_{+} we have

l_{2} (x) (x − y) ≥ Φ (x) − Φ (y) ≥ l_{1} (y) (x − y) , x, y ∈ (0, ∞) ;

where l_{1} (y) ∈ [Φ′_{−} (y) , Φ′_{+} (y)] and l_{2} (x) ∈ [Φ′_{−} (x) , Φ′_{+} (x)], where Φ′_{−} is the left and Φ′_{+} is the right derivative of Φ respectively. We omit the details.

The following corollary is a natural consequence of the above theorem.

Corollary 2. Let Φ : ℝ_{+} → ℝ_{+} be convex and normalized. If Φ′ (1) (P_{n} − Q_{n}) ≥ 0, then we have the positivity inequality

0 ≤ I_{Φ} (p, q) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) . (2.4)

The equality holds in (2.4) for a strictly convex mapping Φ iff p = q.

Remark 2. Corollary 2 shows that the positivity inequality (1.6) holds for a larger class of (p, q) ∈ ℝ_{+}^{n} than that one considered in Corollary 1, namely, for (p, q) ∈ {ℝ_{+}^{n} × ℝ_{+}^{n} : P_{n} = Q_{n}}.

We have the following theorem as well.

Theorem 4. Assume that Φ is differentiable convex on (0, ∞). If p^{(j)}, q^{(j)} (j = 1, 2) are probability distributions, then for all λ ∈ [0, 1] we have the inequality

(2.5) 0 ≤ λI_{Φ} (p^{(1)}, q^{(1)}) + (1 − λ) I_{Φ} (p^{(2)}, q^{(2)})
−I_{Φ} (λp^{(1)} + (1 − λ) q^{(1)}, λp^{(2)} + (1 − λ) q^{(2)})

≤ λ (1 − λ) ∑_{i=1}^{n} \frac{| \begin{matrix}p_{i}^{(1)} & p_{i}^{(2)} \\ q_{i}^{(1)} & q_{i}^{(2)} \end{matrix} |}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} [Φ′ (\frac{p_{i}^{(1)}}{q_{i}^{(1)}} ) − Φ′ (\frac{p_{i}^{(2)}}{q_{i}^{(2)}} )] ,

where Φ′ is the derivative of Φ.

Proof. Using the inequality (2.2) , we may state

Φ′ (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ) (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(1)}}{q_{i}^{(1)}} ) (2.6)

≥ Φ (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ) − Φ (\frac{p_{i}^{(1)}}{q_{i}^{(1)}} )

≥ Φ′ (\frac{p_{i}^{(1)}}{q_{i}^{(1)}} ) (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(1)}}{q_{i}^{(1)}} )

and

Φ′ (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ) (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(2)}}{q_{i}^{(2)}} ) (2.7)

≥ Φ (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ) − Φ (\frac{p_{i}^{(2)}}{q_{i}^{(2)}} )

≥ Φ′ (\frac{p_{i}^{(2)}}{q_{i}^{(2)}} ) (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(2)}}{q_{i}^{(2)}} ) .

Multiply (2.6) by λq_{i}^{(1)} and (2.7) by (1 − λ) q_{i}^{(2)} and add the obtained inequalities to get

∑_{i=1}^{n} Φ′ (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ) [λq_{i}^{(1)} (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(1)}}{q_{i}^{(1)}} ) (2.8)

+ (1 − λ) q_{i}^{(2)} (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(2)}}{q_{i}^{(2)}} )]

≥ I_{Φ} (λp^{(1)} + (1 − λ) p^{(2)}, λq^{(1)} + (1 − λ) q^{(2)})

−λI_{Φ} (p^{(1)}, q^{(1)}) − (1 − λ) I_{Φ} (p^{(2)}, q^{(2)})

≥ ∑_{i=1}^{n} [λq_{i}^{(1)}Φ′ (\frac{p_{i}^{(1)}}{q_{i}^{(1)}} ) (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(1)}}{q_{i}^{(1)}} )

+ (1 − λ) q_{i}^{(2)}Φ′ (\frac{p_{i}^{(2)}}{q_{i}^{(2)}} ) (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(2)}}{q_{i}^{(2)}} )] .

However,

λq_{i}^{(1)} (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(1)}}{q_{i}^{(1)}} )

+ (1 − λ) q_{i}^{(2)} (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(2)}}{q_{i}^{(2)}} )

= −\frac{λ (1 − λ) | \begin{matrix}p_{i}^{(1)} & p_{i}^{(2)} \\ q_{i}^{(1)} & q_{i}^{(2)} \end{matrix} |}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} + \frac{λ (1 − λ) | \begin{matrix}p_{i}^{(1)} & p_{i}^{(2)} \\ q_{i}^{(1)} & q_{i}^{(2)} \end{matrix} |}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} = 0,

which shows that the first membership in (2.8) is zero.

In addition,

λq_{i}^{(1)} (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(1)}}{q_{i}^{(1)}} ) = −\frac{λ (1 − λ) | \begin{matrix}p_{i}^{(1)} & p_{i}^{(2)} \\ q_{i}^{(1)} & q_{i}^{(2)} \end{matrix} |}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}},

and

(1 − λ) q_{i}^{(2)} (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(2)}}{q_{i}^{(2)}} ) = −\frac{λ (1 − λ) | \begin{matrix}p_{i}^{(1)} & p_{i}^{(2)} \\ q_{i}^{(1)} & q_{i}^{(2)} \end{matrix} |}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}},

and then, the second membership in (2.4) is

−λ (1 − λ) ∑_{i=1}^{n} \frac{| \begin{matrix}p_{i}^{(1)} & p_{i}^{(2)} \\ q_{i}^{(1)} & q_{i}^{(2)} \end{matrix} |}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} [Φ′ (\frac{p_{i}^{(1)}}{q_{i}^{(1)}} ) − Φ′ (\frac{p_{i}^{(2)}}{q_{i}^{(2)}} )] ,

which proves the theorem. Remark 3. The first inequality in (2.5) is actually the joint convexity property of I_{Φ} (·, ·) which has been proven here in a different manner than in [3].

3. Applications for Some Particular Φ−Divergences Let us consider the Kullback-Leibler distance given by (1.9)

KL (p, q) := ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) . (3.1)

Consider the convex mapping Φ (t) = − log t, t > 0. For this mapping we have the Csiszár Φ−divergence

(3.2)  I_{Φ} (p, q) = ∑_{i=1}^{n} q_{i} [− log (\frac{p_{i}}{q_{i}} )]
= ∑_{i=1}^{n} q_{i} log (\frac{q_{i}}{p_{i}} ) = KL (q, p) .

The following inequality holds. Proposition 1. Let p, q ∈ ℝ^{n}. Then we have the inequality

Q_{n} − P_{n} ≤ KL (q, p) ≤ ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} − Q_{n}. (3.3)

The case of equality holds iff p = q. Proof. Since Φ (t) = − log t, then Φ′ (t) = −\frac{1}{t}, t > 0. We have

I_{Φ′} (\frac{p^{2}}{q} , p) = ∑_{i=1}^{n} p_{i} · [−\frac{1}{(\frac{p_{i}^{2}}{q_{i}} ) · \frac{1}{p_{i}}} ] = −Q_{n},
I_{Φ′} (p, q) = ∑_{i=1}^{n} q_{i} · [− \frac{1}{\frac{p_{i}}{q_{i}}} ] = − ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} ,

and then, from (2.1), we get

− (P_{n} − Q_{n}) ≤ KL (q, p) ≤ −Q_{n} + ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} ,

which is the desired inequality (3.3).

The case of equality is obvious taking into account that − log is a strictly convex mapping on (0, ∞).

The following result for the Kullback-Leibler distance also holds.

Proposition 2. Let p, q ∈ ℝ^{n}. Then we have the inequality

P_{n} − Q_{n} ≤ KL (p, q) ≤ P_{n} − Q_{n} + KL (q, p) − KL (p, \frac{p^{2}}{q} ) . (3.4)

The case of equality holds iff p = q.

Proof. As Φ (t) = t log (t), then Φ′ (t) = log t + 1. We have

I_{Φ} (p, q) = KL (p, q) ,
I_{Φ′} (\frac{p^{2}}{q} , p) = I_{log(·)+1} (\frac{p^{2}}{q} , p) = P_{n} + I_{log(·)} (\frac{p^{2}}{q} , p) .

As we know that I_{− log(·)} (p, q) = KL (q, p) (see (3.2)), then we have that

I_{log(·)} (\frac{p^{2}}{q} , p) = −KL (p, \frac{p^{2}}{q} ) .

In addition, we have

I_{Φ′} (p, q) = I_{log(·)+1} (p, q) = Q_{n} + I_{log(·)} (p, q)
= Q_{n} − KL (q, p)

and then, by (2.1), we can state that

P_{n} − Q_{n} ≤ KL (p, q) ≤ P_{n} − Q_{n} − KL (p, \frac{p^{2}}{q} ) Q_{n} + KL (q, p)

and the inequality (3.4) is obtained.

The case of equality holds from the fact that the mapping Φ (t) = t log t is strictly convex on (0, ∞).

Now, let us consider the α−order entropy of Rényi (see (1.13))

D_{α} (p, q) := ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}, α > 1 (3.5)

and p, q ∈ ℝ_{+}^{n}.

We know that Rényi’s entropy is actually the Csiszár Φ−divergence for the convex mapping Φ (t) = t^{α}, α > 1, t > 0 (see Example 3).

The following proposition holds.

Proposition 3. Let p, q ∈ ℝ_{+}^{n}. Then we have the inequality

α (P_{n} − Q_{n}) ≤ D_{α} (p, q) − Q_{n} ≤ α [D_{α} (p, q) − D_{α} (q^{\frac{2−α}{α}} , p^{−1})] . (3.6)

The case of equality holds iff p = q.

Proof. Since Φ (t) = t^{α}, then Φ′ (t) = αt^{α−1}.

We have

I_{Φ′} (\frac{p^{2}}{q} , p) = ∑_{i=1}^{n} p_{i} [α · (\frac{p_{i}^{2}}{q_{i}p_{i}} )^{α−1}]

= α ∑_{i=1}^{n} p_{i} (\frac{p_{i}}{q_{i}} )^{α−1} = α ∑_{i=1}^{n} q_{i}^{1−α}p_{i}^{α} = αD_{α} (p, q)

and

I_{Φ′} (p, q) = ∑_{i=1}^{n} q_{i} [α · (\frac{p_{i}}{q_{i}} )^{α−1}]

= α ∑_{i=1}^{n} p_{i}^{α−1}q_{i}^{2−α} = αD_{α} (q^{\frac{2−α}{α}} , \frac{1}{p}) .

Using the inequality (2.1), we have

α (P_{n} − Q_{n}) ≤ D_{α} (p, q) − Q_{n} ≤ α [D_{α} (p, q) − D_{α} (q^{\frac{2−α}{α}} , \frac{1}{p})]

and the inequality (3.6) is proved.

The case of equality holds since the mapping Φ (t) = t^{α} is strictly convex on (0, ∞) for α > 1.

Consider now the Hellinger discrimination (see for example [22])

h^{2} (p, q) = \frac{1}{2} ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} ,

where p, q ∈ ℝ_{+}^{n}.

We know that Hellinger discrimination is actually the Csiszár Φ−divergence for the convex mapping Φ (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}.

We may state the following proposition.

Proposition 4. Let p, q ∈ ℝ_{+}^{n}. Then we have the inequality

0 ≤ h^{2} (p, q) ≤ \frac{1}{2} [P_{n} − Q_{n}] + \frac{1}{2} [∑_{i=1}^{n} q_{i} (\sqrt{\frac{q_{i}}{p_{i}}} − \sqrt{\frac{p_{i}}{q_{i}}} )] . (3.7)

The equality holds iff p = q.

Proof. As Φ (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}, we have Φ′ (t) = \frac{1}{2} − \frac{1}{2\sqrt{t}} and Φ′′ (t) = \frac{1}{4} · \frac{1}{\sqrt{t^{3}}} > 0 (t ∈ (0, ∞)) which shows that Φ is indeed strictly convex on (0, ∞).

We also have:

I_{Φ} (p, q) = h^{2} (p, q) ,
I_{Φ′} (\frac{p^{2}}{q} , p) = ∑_{i=1}^{n} p_{i} [\frac{1}{2} − \frac{1}{2\sqrt{\frac{p_{i}^{2}}{q_{i}p_{i}}}} ]

= \frac{1}{2}P_{n} − \frac{1}{2} ∑_{i=1}^{n} \sqrt{p_{i}q_{i}} = \frac{1}{2} [P_{n} − ∑_{i=1}^{n} \sqrt{p_{i}q_{i}}]

I_{Φ′} (p, q) = ∑_{i=1}^{n} q_{i} [\frac{1}{2} − \frac{1}{2\sqrt{\frac{p_{i}}{q_{i}}}} ] = \frac{1}{2} [Q_{n} − ∑_{i=1}^{n} q_{i}\sqrt{\frac{q_{i}}{p_{i}}} ]

and as Φ′ (1) = 0 and Φ (1) = 0, then, by (2.1) applied for Φ as above, we deduce (3.7). The case of equality is obvious by the strict convexity of Φ.

Consider now the Bhattacharyya distance ( see for example [22])

B (p, q) = ∑_{i=1}^{n} \sqrt{p_{i}q_{i}},

where p, q ∈ ℝ_{+}^{n}.

We know that for the convex mapping f (t) = −\sqrt{t}, we have

I_{Φ} (p, q) = − ∑_{i=1}^{n} q_{i}\sqrt{\frac{p_{i}}{q_{i}}} = −B (p, q) .

We may state the following proposition.

Proposition 5. Let p, q ∈ ℝ_{+}^{n}. Then we have the inequality

\frac{1}{2} (Q_{n} − P_{n}) ≤ Q_{n} − B (p, q) ≤ \frac{1}{2} ∑_{i=1}^{n} q_{i} (\sqrt{\frac{q_{i}}{p_{i}}} − \sqrt{\frac{p_{i}}{q_{i}}} ) . (3.8)

The case of equality holds iff p = q.

Proof. As Φ (1) = −\sqrt{t}, t > 0, then Φ′ (t) = −\frac{1}{2\sqrt{t}} and Φ′′ (t) = \frac{1}{4\sqrt{t^{3}}} , t > 0, which also shows that Φ (·) is strictly convex on (0, ∞). We also have

I_{Φ′} (\frac{p^{2}}{q} , p) = ∑_{i=1}^{n} p_{i} [−\frac{1}{2\sqrt{\frac{p_{i}^{2}}{q_{i}p_{i}}}} ] = −\frac{1}{2} ∑_{i=1}^{n} \sqrt{p_{i}q_{i}} = −\frac{1}{2}B (p, q) ,

I_{Φ′} (p, q) = −\frac{1}{2} ∑_{i=1}^{n} q_{i} \frac{1}{\sqrt{\frac{p_{i}}{q_{i}}}} = −\frac{1}{2} ∑_{i=1}^{n} q_{i}\sqrt{\frac{q_{i}}{p_{i}}}

and as Φ′ (1) = −\frac{1}{2}, Φ (1) = −1, then by (2.1) applied for the mapping Φ as defined above, we deduce (3.8).

The case of equality is obvious by the strict convexity of Φ.

4. Further Bounds for the Case when P_{n} = Q_{n}

The following inequality of Grüss type is well known in the literature as the Biernacki, Pidek and Ryll-Nardzewski inequality (see for example [23]).

Lemma 1. Let a_{i}, b_{i} (i = 1, ..., n) be real numbers such that

a ≤ a_{i} ≤ A, b ≤ b_{i} ≤ B for all i ∈ {1, ..., n} . (4.1)

Then we have the inequality:

|\frac{1}{n} ∑_{i=1}^{n} a_{i}b_{i} − \frac{1}{n^{2}} ∑_{i=1}^{n} a_{i} ∑_{i=1}^{n} b_{i}| ≤ \frac{1}{n} [\frac{n}{2}] (1 − \frac{1}{n} [\frac{n}{2}]) (A − a) (B − b) , (4.2)

where [x] denotes the integer part of x.

The following inequality holds.

Theorem 5. Let Φ : ℝ_{+} → ℝ_{+} be differentiable convex. If p, q ∈ ℝ_{+}^{n} are such that P_{n} = Q_{n} and

m ≤ p_{i} − q_{i} ≤ M, i = 1, ..., n (4.3)

0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞, i = 1, ..., n, (4.4)

then we have the inequality

0 ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ [\frac{n}{2}] (1 − \frac{1}{n} [\frac{n}{2}]) (M − m) (Φ′ (R) − Φ′ (r)) . (4.5)

Proof. From (2.1) we have

0 ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ ∑_{i=1}^{n} (p_{i} − q_{i}) Φ′ (\frac{p_{i}}{q_{i}} ) . (4.6)

Applying (4.2) we have

|\frac{1}{n} ∑_{i=1}^{n} (p_{i} − q_{i}) Φ′ (\frac{p_{i}}{q_{i}} ) − \frac{1}{n^{2}} ∑_{i=1}^{n} (p_{i} − q_{i}) ∑_{i=1}^{n} Φ′ (\frac{p_{i}}{q_{i}} )| (4.7)

≤ \frac{1}{n} [\frac{n}{2}] (1 − \frac{1}{n} [\frac{n}{2}]) (M − m) (Φ′ (R) − Φ′ (r))

as the mapping Φ′ is monotonic nondecreasing, and then

Φ′ (r) ≤ Φ′ (\frac{p_{i}}{q_{i}} ) ≤ Φ′ (R) for all i ∈ \{1, ..., n\} .

As ∑_{i=1}^{n} (p_{i} − q_{i}) = 0, we deduce by (4.6) and (4.7) the desired result (4.5).

The following inequalities for particular distances are valid.

1. If p, q ∈ ℝ_{n}^{+} are such that the conditions (4.3) and (4.4) hold, then we have the inequalities

0 ≤ KL (q, p) ≤ [\frac{n}{2}] (1 − \frac{1}{n} [\frac{n}{2}]) (M − m) \frac{R − r}{rR} , (4.8)

SOME INEQUALITIES FOR THE CSISZÁR Φ-DIVERGENCE 11

and

0 ≤ KL (p, q) ≤ [\frac{n}{2}] (1 − \frac{1}{n} [\frac{n}{2}]) (M − m) [log (\frac{R}{r})] . (4.9)

2. If p, q are as in (4.3) and (4.4), we have the inequality (α ≥ 1)

0 ≤ D_{α} (p, q) − Q_{n} ≤ α [\frac{n}{2}] (1 − \frac{1}{n} [\frac{n}{2}]) (M − m) (R^{α−1} − r^{α−1}) . (4.10)

3. If p, q are as in (4.3) and (4.4), we have the inequality

0 ≤ h^{2} (p, q) ≤ \frac{1}{2} [\frac{n}{2}] (1 − \frac{1}{n} · [\frac{n}{2}]) (M − m) \frac{\sqrt{R} − \sqrt{r}}{\sqrt{rR}} . (4.11)

4. Under the above assumptions for p and q, we have

0 ≤ Q_{n} − B (p, q) ≤ \frac{1}{2} [\frac{n}{2}] (1 − \frac{1}{n} [\frac{n}{2}]) (M − m) \frac{\sqrt{R} − \sqrt{r}}{\sqrt{rR}} . (4.12)

Using the following Grüss’ weighted inequality. Lemma 2. Assume that a_{i}, b_{i} (i = 1, ..., n) are as in Lemma 1. If q_{i} ≥ 0, ∑_{i=1}^{n} q_{i} = 1, then we have the inequality

|∑_{i=1}^{n} q_{i}a_{i}b_{i} − ∑_{i=1}^{n} q_{i}a_{i} ∑_{i=1}^{n} q_{i}b_{i}| ≤ \frac{1}{4} (A − a) (B − b) . (4.13)

We may prove the following converse inequality as well. Theorem 6. Let Φ : ℝ_{+} → ℝ_{+} be differentiable convex. If p, q ∈ ℝ_{+}^{n} are such that P_{n} = Q_{n} and

0 < r ≤ \frac{p_{i}}{q_{i}} ≤ R < ∞, i = 1, ..., n, (4.14)

then we have the inequality

0 ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ \frac{1}{4} (R − r) [Φ′ (R) − Φ′ (r)] . (4.15)

Proof. From (2.1) we have

(4.16)  0 ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ ∑_{i=1}^{n} (p_{i} − q_{i}) Φ′ (\frac{p_{i}}{q_{i}} )
= ∑_{i=1}^{n} q_{i} (\frac{p_{i}}{q_{i}} − 1) Φ′ (\frac{p_{i}}{q_{i}} ) .

As Φ′ (·) is monotonic nondecreasing, then

Φ′ (r) ≤ Φ′ (\frac{p_{i}}{q_{i}} ) ≤ Φ′ (R) for all i ∈ \{1, ..., n\}.

Applying (4.13) for a_{i} = \frac{p_{i}}{q_{i}} − 1, b_{i} = Φ′ (\frac{p_{i}}{q_{i}} ), we obtain

|∑_{i=1}^{n} q_{i} (\frac{p_{i}}{q_{i}} − 1) Φ′ (\frac{p_{i}}{q_{i}} ) − ∑_{i=1}^{n} q_{i} (\frac{p_{i}}{q_{i}} − 1) ∑_{i=1}^{n} q_{i}Φ′ (\frac{p_{i}}{q_{i}} )| (4.17)

≤ \frac{1}{4} (R − r) [Φ′ (R) − Φ′ (r)]

and as

∑_{i=1}^{n} q_{i} (\frac{p_{i}}{q_{i}} − 1) = 0,

then, by (4.16) and (4.17) we deduce (4.15).

The following inequalities for particular distances are valid.

1. If p, q are such that P_{n} = Q_{n} and (4.14) holds, then

0 ≤ KL (q, p) ≤ \frac{(R − r)^{2}}{4rR} (4.18)

and

0 ≤ KL (q, p) ≤ \frac{1}{4} (R − r)^{2} ln (\frac{R}{r}) . (4.19)

2. With the same assumptions for p, q, we have

(4.20) 0 ≤ D_{α} (p, q) − Q_{n} ≤ \frac{α}{4} (R − r) (R^{α−1} − r^{α−1}) (α ≥ 1) ;

(4.21) 0 ≤ h^{2} (p, q) ≤ \frac{1}{8} (R − r) \frac{\sqrt{R} − \sqrt{r}}{\sqrt{Rr}}

and

0 ≤ Q_{n} − B (p, q) ≤ \frac{1}{8} (R − r) \frac{\sqrt{R} − \sqrt{r}}{\sqrt{Rr}} . (4.22)

Remark 4. Any other Grüss type inequality can be used to provide different bounds for the difference

Δ := ∑_{i=1}^{n} (p_{i} − q_{i}) Φ′ (\frac{p_{i}}{q_{i}} ) .

We omit the details.

References

[1] I. CSISZÁR, Information measures: A critical survey, Trans. 7th Prague Conf. on Info. Th., Statist. Decis. Funct., Random Processes and 8th European Meeting of Statist., Volume B, Academia Prague, 1978, 73-86.

[2] I. CSISZÁR, Information-type measures of difference of probability distributions and indirect observations, Studia Sci. Math. Hungar, 2 (1967), 299-318.

[3] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorems for Discrete Memoryless Systems, Academic Press, New York, 1981.

[4] J.H. JUSTICE (editor), Maximum Entropy and Bayssian Methods in Applied Statistics, Cambridge University Press, Cambridge, 1986.

[5] J.N. KAPUR, On the roles of maximum entropy and minimum discrimination information principles in Statistics, Technical Address of the 38th Annual Conference of the Indian Society of Agricultural Statistics, 1984, 1-44.

[6] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy functions, IEEE Transactions on Information Theory, 28 (1982), 489-495.

[7] R.G. GALLAGER, Information Theory and Reliable Communications, J. Wiley, New York, 1968.

[8] C.E. SHANNON, A mathematical theory of communication, Bull. Sept. Tech. J., 27 (1948), 370-423 and 623-656.

[9] B.R. FRIEDEN, Image enhancement and restoration, Picture Processing and Digital Filtering (T.S. Huang, Editor), Springer-Verlag, Berlin, 1975.

[10] R.M. LEAHY and C.E. GOUTIS, An optimal technique for constraint-based image restoration and mensuration, IEEE Trans. on Acoustics, Speech and Signal Processing, 34 (1986), 1692-1642.

[11] S. KULLBACK, Information Theory and Statistics, J. Wiley, New York, 1959.

[12] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[13] R. BERAN, Minimum Hellinger distance estimates for parametric models, Ann. Statist., 5 (1977), 445-463.

[14] A. RENYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Statist. Prob., Vol. 1, University of California Press, Berkeley, 1961.

[15] S.S. DRAGOMIR and N.M. IONESCU, Some converse of Jensen’s inequality and applications, Anal. Num. Theor. Approx., 23 (1994), 71-78.

[16] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s discrete inequality for differentiable convex mappings and applications in information theory, Math. Comput. Modelling, 24 (2) (1996), 1-11.

[17] S.S. DRAGOMIR and C.J. GOH, Some counterpart inequalities in for a functional associated with Jensen’s inequality, J. of Ineq. & Appl., 1 (1997), 311-325.

[18] S.S. DRAGOMIR and C.J. GOH, Some bounds on entropy measures in information theory, Appl. Math. Lett., 10 (1997), 23-28.

[19] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s continuous inequality and applications in information theory, RGMIA Preprint,

http://matilda.vu.edu.au/~rgmia/InfTheory/Continuse.dvi

[20] M. MATIĆ, Jensen’s inequality and applications in Information Theory (in Croatian), Ph.D. Thesis, Univ. of Zagreb, 1999.

[21] S.S. DRAGOMIR, J. ŠUNDE and M. SCHOLZ, Some upper bounds for relative entropy and applications, Comp. and Math. with Appl. (in press).

[22] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), No. 1, 1-16.

[23] D.S. MITRINOVIĆ, J.E. PEČARIĆ and A.M. FINK, Classical and New Inequalities in Analysis, Kluwer Academic Publishers, 1993.

School of Communications and Informatics, Victoria University of Technology, PO Box 14428, Melbourne City MC 8001, Victoria, Australia.

E-mail address: sever.dragomir@vu.edu.au
URL: http://rgmia.vu.edu.au/SSDragomirWeb.html