SOME INEQUALITIES FOR (m, M)-CONVEX MAPPINGS AND

APPLICATIONS FOR THE CSISZÁR Φ−DIVERGENCE IN INFORMATION THEORY

S. S. DRAGOMIR

Abstract. Some inequalities of Jensen’s type and their applications for the Csiszár f-divergence are given.

1. Introduction

Given a convex function Φ : ℝ_{+} → ℝ_{+}, the Φ- divergence functional

I_{Φ} (p, q) := ∑_{i=1}^{n} q_{i}Φ (\frac{p_{i}}{q_{i}} ) (1.1)

was introduced in Csiszár [1] as a generalized measure of information, a “distance function” on the set of probability distributions ℙ^{n}. The restriction here to discrete distribution is only for convenience, similar results hold for general distributions.

As in Csiszár [2], we interpret undefined expressions by

Φ (0) = lim_{t→0+} Φ (t) , 0Φ (\frac{0}{0}) = 0,
0Φ (\frac{a}{0}) = lim_{ε→0+} Φ (\frac{a}{ε}) = a lim_{t→∞} \frac{Φ (t)}{t}, a > 0.

The following results were essentially given by Csiszár and Körner [3].

Theorem 1. If Φ : ℝ_{+} → ℝ is convex, then I_{Φ} (p, q) is jointly convex in p and q.

The following lower bound for the Φ−divergence functional also holds.

Theorem 2. Let Φ : ℝ_{+} → ℝ_{+} be convex. Then for every p, q ∈ ℝ_{+}^{n}, we have the inequality:

I_{Φ} (p, q) ≥ ∑_{i=1}^{n} q_{i}Φ (\frac{∑_{i=1}^{n} p_{i}}{∑_{i=1}^{n} q_{i}} ) . (1.2)

If Φ is strictly convex, equality holds in (1.2) iff

\frac{p_{1}}{q_{1}} = \frac{p_{2}}{q_{2}} = ... = \frac{p_{n}}{q_{n}} . (1.3)

Date: November 18, 1999.

1991 Mathematics Subject Classification. Primary 94Xxx; Secondary 26D15.

Key words and phrases. The Csiszár Φ−Divergence, Kullback-Leibler distance, Rényi α−entropy.

2 S. S. DRAGOMIR

Corollary 1. Let Φ : ℝ_{+} → ℝ be convex and normalized, i.e., Φ (1) = 0. (1.4) Then for any p, q ∈ ℝ_{+}^{n} with

∑_{i=1}^{n} p_{i} = ∑_{i=1}^{n} q_{i} (1.5)

we have the inequality I_{Φ} (p, q) ≥ 0. (1.6) If Φ is strictly convex, equality holds in (1.5) iff p_{i} = q_{i} for all i ∈ {1, ..., n}.

In particular, if p, q are probability vectors, then (1.5) is assured. Corollary 1 then shows, for strictly convex and normalized Φ : ℝ_{+} → ℝ, I_{Φ} (p, q) ≥ 0 for all p, q ∈ ℙ^{n}. (1.7) The equality holds in (1.7) iff p = q.

These are “distance properties”. However, I_{Φ} is not a metric: It violates the triangle inequality, and is asymmetric, i.e, for general p, q ∈ ℝ_{+}^{n}, I_{Φ} (p, q) ≠ I_{Φ} (q, p).

In the examples below we obtain, for suitable choices of the kernel Φ, some of the best known distance functions I_{Φ} used in mathematical statistics [4]-[5], information theory [6]-[8] and signal processing [9]-[10]. Example 1. (Kullback-Leibler) For Φ (t) := t log t, t > 0 (1.8) the Φ−divergence is

I_{Φ} (p, q) = ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) , (1.9)

the Kullback-Leibler distance [11]-[12]. Example 2. (Hellinger) Let

Φ (t) = \frac{1}{2} (1 − \sqrt{t})^{2} , t > 0. (1.10)

Then I_{Φ} gives the Hellinger distance [13]

I_{Φ} (p, q) = \frac{1}{2} ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} , (1.11)

which is symmetric. Example 3. (Renyi) For α > 1, let Φ (t) = t^{α}, t > 0. (1.12) Then

I_{Φ} (p, q) = ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}, (1.13)

which is the α−order entropy [14].

SOME INEQUALITIES FOR (m, M)-CONVEX MAPPINGS 3

Example 4. (χ^{2}−distance) Let

Φ (t) = (t − 1)^{2} , t > 0. (1.14)

Then

(1.15)  I_{Φ} (p, q) = ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} = ∑_{i=1}^{n} \frac{p_{i}^{2}}{q_{i}} − 2P_{n} + Q_{n}

(= ∑_{i=1}^{n} \frac{p_{i}^{2} − q_{i}^{2}}{q_{i}} if P_{n} = Q_{n})

is the χ^{2}−distance between p and q.

Finally, we have

Example 5. (Variation distance). Let Φ (t) = |t − 1| , t > 0. The corresponding divergence, called the variation distance, is symmetric,

I_{Φ} (p, q) = ∑_{i=1}^{n} |p_{i} − q_{i}| .

For other examples of divergence measures, see the paper [22] by J.N. Kapur, where further references are given.

2. Some Inequalities of Jensen’s Type

Let (X, ∥·∥) be a real or complex normed linear space, C ⊆ X a convex subset of X and f : C → ℝ.

Definition 1. Let α, β ∈ ℝ.

(i) The mapping f will be called α−lower convex on C if f − \frac{α}{2} · ∥·∥^{2} is a convex mapping on C;

(ii) The mapping f will be called β−upper convex on C if \frac{β}{2} · ∥·∥^{2} − f is a convex

mapping on C; (iii) The mapping f will be called (α, β) − convex on C if it is both α−lower convex

and β-upper convex on C.

Note that if f is (α, β)-convex on C, then α ≤ β.

Indeed, if f is (α, β) −convex, then f − \frac{α}{2} ·∥·∥^{2} and \frac{β}{2} ·∥·∥^{2} −f are convex, which clearly implies that the sum

f − \frac{α}{2} · ∥·∥^{2} + \frac{β}{2} · ∥·∥^{2} − f = \frac{β − α}{2} · ∥·∥^{2}

is convex, and then β ≥ α.

Taking into account the above, when we talk about an (α, β) −convex function, we can assume without loss of generality that α = m ≤ M = β.

The following theorem holds.

Theorem 3. Let f : C ⊆ X → ℝ, C be convex on X, x_{i} ∈ C, p_{i} ≥ 0 (i = 1, ..., n)

with P_{n} := ∑_{i=1}^{n} p_{i} > 0.

(i) If f is α−lower convex on C, then we have the following inequality (for α ≥ 0 - refinement of Jensen’s inequality)

\frac{α}{2} · \frac{1}{P_{n}^{2}} [P_{n} ∑_{i=1}^{n} p_{i} ∥x_{i}∥^{2} − ∥∑_{i=1}^{n} p_{i}x_{i}∥^{2}] (2.1)

≤ \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}f (x_{i}) − f (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i}) .

(ii) If f is β−upper convex on C, then we have the following inequality (which is a counterpart of Jensen’s inequality if f is convex)

\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}f (x_{i}) − f (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i}) (2.2)

≤ \frac{β}{2} · \frac{1}{P_{n}^{2}} [P_{n} ∑_{i=1}^{n} p_{i} ∥x_{i}∥^{2} − ∥∑_{i=1}^{n} p_{i}x_{i}∥^{2}] .

(iii) If f is (m, M) −convex on C, then we have the following sandwich inequality

\frac{m}{2} · \frac{1}{P_{n}^{2}} [P_{n} ∑_{i=1}^{n} p_{i} ∥x_{i}∥^{2} − ∥∑_{i=1}^{n} p_{i}x_{i}∥^{2}] (2.3)

≤ \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}f (x_{i}) − f (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i})

≤ \frac{M}{2} · \frac{1}{P_{n}^{2}} [P_{n} ∑_{i=1}^{n} p_{i} ∥x_{i}∥^{2} − ∥∑_{i=1}^{n} p_{i}x_{i}∥^{2}] .

Proof. If g : C ⊆ X → ℝ is a convex mapping on C, x_{i} ∈ C, p_{i} ≥ 0 with P_{n} > 0, then, Jensen’s inequality holds

g (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i}) ≤ \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}g (x_{i}) . (2.4)

(i) Let g (x) = f (x) − \frac{α}{2} · ∥x∥^{2}. Then g is convex on C and by (2.4) we get

f (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i}) − \frac{α}{2} ∥\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i}∥^{2} ≤ \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i} [f (x_{i}) − \frac{α}{2} ∥x_{i}∥^{2}] ,

which is clearly equivalent with (2.1). (ii) Let h (x) = \frac{β}{2} · ∥x∥^{2} − f (x). Then h is convex on C, and by (2.4) for h, we

obtain (2.2). (iii) Follows by (i) and (ii).

The following corollary for inner product spaces holds.

Corollary 2. Let (X, ⟨·, ·⟩) be an inner product, ∥·∥ := \sqrt{⟨·, ·⟩}, C ⊆ X a convex subset on X, f : C → ℝ, and x_{i} ∈ C, p_{i} ≥ 0 (i = 1, ..., n) with P_{n} > 0.

(i) If f is α−lower convex on C, then

\frac{α}{P_{n}^{2}} ∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{i} − x_{j}∥^{2} ≤ \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}f (x_{i}) − f (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i}) . (2.5)

(ii) If f is β−upper convex on C, then

\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}f (x_{i}) − f (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i}) ≤ \frac{β}{P_{n}^{2}} ∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{i} − x_{j}∥^{2} . (2.6)

(iii) If f is (m, M) −convex on C, then

(2.7) m · \frac{1}{P_{n}^{2}} ∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{i} − x_{j}∥^{2} ≤ \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}f (x_{i}) − f (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i})

≤ M · \frac{1}{P_{n}^{2}} ∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{i} − x_{j}∥^{2} .

Proof. The argument follows by Theorem 3 taking into account that for inner products, we have:

∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{i} − x_{j}∥^{2}

= \frac{1}{2} ∑_{i,j=1}^{n} p_{i}p_{j} ∥x_{i} − x_{j}∥^{2} = \frac{1}{2} ∑_{i,j=1}^{n} p_{i}p_{j} [∥x_{i}∥^{2} − 2 Re ⟨x_{i}, x_{j}⟩ + ∥x_{j}∥^{2}]

= \frac{1}{2} [∑_{i=1}^{n} p_{i} ∥x_{i}∥^{2} P_{n} − 2 Re ⟨∑_{i=1}^{n} p_{i}x_{i}, ∑_{j=1}^{n} p_{j}x_{j}⟩ + P_{n} ∑_{j=1}^{n} p_{j} ∥x_{j}∥^{2}]

= P_{n} ∑_{i=1}^{n} p_{i} ∥x_{i}∥^{2} − ∥∑_{i=1}^{n} p_{i}x_{i}∥^{2} .

Remark 1. The case of the mappings defined on real intervals have been obtained by Andrica and Raşa in [23].

Further more, let us assume that

Δ (x) := max_{1≤i<j≤n} ∥x_{i} − x_{j}∥

and

δ (x) := min_{1≤i<j≤n} ∥x_{i} − x_{j}∥ .

The following corollary also holds.

Corollary 3. Let X, f, C, x_{i}, p_{i} (i = 1, ..., n) be as in Corollary 2.

(i) If f is α−lower convex on C with α > 0, then we have the following refinement of Jensen’s inequality:

0 < \frac{α}{2} (1 − \frac{∑_{i=1}^{n} p_{i}^{2}}{P_{n}^{2}} ) δ^{2} (x) ≤ \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}f (x_{i}) − f (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i}) . (2.8)

(ii) If f is convex and β−upper convex on C, then we have the following converse inequality:

0 ≤ \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}f (x_{i}) − f (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i}) ≤ \frac{β}{2} (1 − \frac{∑_{i=1}^{n} p_{i}^{2}}{P_{n}^{2}} ) Δ^{2} (x) . (2.9)

(iii) If f is (m, M) −convex on C with m > 0, then we have the following estimate

(2.10) \frac{m}{2} (1 − \frac{∑_{i=1}^{n} p_{i}^{2}}{P_{n}^{2}} ) δ^{2} (x) ≤ \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}f (x_{i}) − f (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i})

≤ \frac{M}{2} (1 − \frac{∑_{i=1}^{n} p_{i}^{2}}{P_{n}^{2}} ) Δ^{2} (x) .

To prove the above corollary we use the following lemma which is interesting in itself too.

Lemma 1. Let (X, ⟨·, ·⟩) be an inner product space and x_{i} ∈ X, p_{i} ≥ 0 (i = 1, ..., n) with P_{n} > 0. Then we have the inequality

(2.11)  (1 − \frac{∑_{i=1}^{n} p_{i}^{2}}{P_{n}^{2}} ) δ^{2} (x) ≤ \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i} ∥x_{i}∥^{2} − ∥\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i}∥^{2}

≤ (1 − \frac{∑_{i=1}^{n} p_{i}^{2}}{P_{n}^{2}} ) Δ^{2} (x) .

Proof. As above, we have

(2.12) \frac{1}{P_{n}} ∑_{i=1}^{n} p_{i} ∥x_{i}∥^{2} − ∥\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i}∥^{2} = \frac{1}{2P_{n}^{2}} ∑_{i,j=1}^{n} p_{i}p_{j} ∥x_{i} − x_{j}∥^{2}

= \frac{1}{P_{n}^{2}} ∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{i} − x_{j}∥^{2} .

Now, obviously

(2.13)  δ^{2} (x) \frac{1}{P_{n}^{2}} ∑_{1≤i<j≤n} p_{i}p_{j} ≤ \frac{1}{P_{n}^{2}} ∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{i} − x_{j}∥^{2}

≤ Δ^{2} (x) \frac{1}{P_{n}^{2}} ∑_{1≤i<j≤n} p_{i}p_{j}.

On the other hand,

∑_{i,j=1} p_{i}p_{j} = (∑_{i=1}^{n} p_{i})^{2} = P_{n}^{2}

and

∑_{i,j=1}^{n} p_{i}p_{j} = ∑_{1≤i<j≤n} p_{i}p_{j} + ∑_{1≤j<i≤n} p_{i}p_{j} + ∑_{i=j=1}^{n} p_{i}p_{j}

= 2 ∑_{1≤i<j≤n} p_{i}p_{j} + ∑_{i=1}^{n} p_{i}^{2},

from where we obtain

∑_{1≤i<j≤n} p_{i}p_{j} = \frac{1}{2} (P_{n}^{2} − ∑_{i=1}^{n} p_{i}^{2}) . (2.14)

Now, using (2.12) - (2.14), we get the desired inequality (2.11).

3. Applications for Csiszár Φ−Divergence

In the recent paper [24], the author proved the following inequality for Csiszár f−divergence:

Theorem 4. Let Φ : ℝ_{+} → ℝ be differentiable convex. Then for all p, q ∈ ℝ_{+}^{n} we have the inequality:

Φ′ (1) (P_{n} − Q_{n}) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) , (3.1)

where P_{n} := ∑_{i=1}^{n} p_{i} > 0, Q_{n} := ∑_{i=1}^{n} q_{i} > 0 and Φ′ : (0, ∞) → ℝ is the derivative of

Φ. If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n) , then the equality holds in (2.1) iff p = q,

If we assume that P_{n} = Q_{n} and Φ is normalised, then we obtain the simpler inequality

0 ≤ I_{Φ} (p, q) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) . (3.2)

Applications for particular divergences which are instances of Csiszár f−divergence were also given.

A similar result to the above theorem has been presented in another paper by the author [25].

Theorem 5. Let Φ, p, q be as in Theorem 4. Then we have the inequality

0 ≤ I_{Φ} (p, q) − Q_{n}Φ (\frac{P_{n}}{Q_{n}} ) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − \frac{P_{n}}{Q_{n}} I_{Φ′} (p, q) . (3.3)

If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n), then the equality holds in (3.3) iff \frac{p_{1}}{q_{1}} = ... = \frac{p_{n}}{q_{n}}.

Obviously, if P_{n} = Q_{n} and Φ is normalised, then (3.3) becomes (3.2).

In this section we apply some of the above results for the Csiszár Φ−divergence and obtain other inequalities that are simialr, in a sense, with those presented above.

Theorem 6. Let Φ : ℝ_{+} → ℝ and p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}.

8 S. S. DRAGOMIR

(i) If Φ is α−lower convex on ℝ_{+}, then we have the inequality

\frac{α}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) . (3.4)

(ii) If Φ is β−upper convex on ℝ_{+}, then we have the inequality

I_{Φ} (p, q) − Q_{n}Φ (1) ≤ \frac{β}{2} · D_{χ}^{2} (p, q) . (3.5)

(iii) If Φ is (m, M) −convex on ℝ_{+}, then we have the following inequality

\frac{m}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ \frac{M}{2} · D_{χ}^{2} (p, q) , (3.6)

where D_{χ}^{2} (p, q) is the χ^{2}−distance, i.e.,

D_{χ}^{2} (p, q) = ∑_{i=1}^{n} \frac{p_{i}^{2} − q_{i}^{2}}{q_{i}} .

Proof. We use Theorem 3 in which we put f = Φ, C = ℝ_{+}, X = ℝ_{+}, p_{i} → q_{i}, x_{i} = \frac{p_{i}}{q_{i}}. We have

\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}f (x_{i}) − f (\frac{1}{P_{n}} ∑_{i=1}^{n} p_{i}x_{i})

= \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i}Φ (\frac{p_{i}}{q_{i}} ) − Φ (\frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i} · \frac{p_{i}}{q_{i}} ) = \frac{1}{Q_{n}} I_{Φ} (p, q) − Φ (1)

and

\frac{1}{P_{n}^{2}} [P_{n} ∑_{i=1}^{n} p_{i}x_{i}^{2} − (∑_{i=1}^{n} p_{i}x_{i})^{2}]
= \frac{1}{Q_{n}^{2}} [Q_{n} ∑_{i=1}^{n} q_{i} (\frac{p_{i}}{q_{i}} )^{2} − (∑_{i=1}^{n} q_{i} · \frac{p_{i}}{q_{i}} )^{2}] = \frac{1}{Q_{n}^{2}} [Q_{n} ∑_{i=1}^{n} \frac{p_{i}^{2}}{q_{i}} − P_{n}^{2}]
= \frac{1}{Q_{n}} D_{χ}^{2} (p, q) .

Now, by (2.1)-(2.3), we obtain the desired results (3.4)-(3.6). Corollary 4. Let Φ : ℝ_{+} → ℝ be normalised, i.e., Φ (1) = 0, and p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}.

(i) If Φ is α−lower convex on ℝ_{+}, then we have the inequality

\frac{α}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) ; (3.7)

(ii) If Φ is β-upper convex on ℝ_{+}, then we have the inequality

I_{Φ} (p, q) ≤ \frac{β}{2} · D_{χ}^{2} (p, q) ; (3.8)

(iii) If Φ is (m, M) −convex on ℝ_{+}, then we have the sandwich inequality

\frac{m}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) ≤ \frac{M}{2} · D_{χ}^{2} (p, q) . (3.9)

In practical applications, it is important to have sufficient conditions for the mapping Φ so that it will be α−lower convex, β−upper convex, or (m, M)-convex on a certain interval I of ℝ_{+}.

The following proposition holds.

Proposition 1. Let Φ : I ⊆ ℝ_{+} → ℝ be a differentiable mapping on I̊ (I̊ is the interior of I).

(i) If there exists an α ∈ ℝ such that

(3.10) \frac{Φ′ (x_{2}) − Φ′ (x_{1})}{x_{2} − x_{1}} ≥ α for all x_{2} > x_{1}, x_{2}, x_{1} ∈ I̊;

then Φ is α−lower convex on I; (ii) If there exists a β ∈ ℝ such that

(3.11) \frac{Φ′ (x_{2}) − Φ′ (x_{1})}{x_{2} − x_{1}} ≤ β for all x_{2} > x_{1}, x_{2}, x_{1} ∈ I̊;

then Φ is β−upper convex on I; (iii) If there exists m, M ∈ ℝ such that m < M and

(3.12) m ≤ \frac{Φ′ (x_{2}) − Φ′ (x_{1})}{x_{2} − x_{1}} ≤ M for all x_{2} > x_{1}, x_{2}, x_{1} ∈ I̊,

then Φ is (m, M) −convex on I.

Proof. It is well known that a differentiable mapping g : I → ℝ is convex iff g is monotonic nondecreasing on I̊, i.e., g (x_{2}) ≥ g (x_{1}) for all x_{2} > x_{1}, x_{1}, x_{2} ∈I̊.

Applying this criterion to the mapping Φ (x)− \frac{α}{2} ·x^{2}, \frac{β}{2} ·x^{2} −Φ (x), Φ (x)−m· \frac{x^{2}}{2} , M · \frac{x^{2}}{2} − Φ (x), we obtain the desired results.

Proposition 2. Let Φ : I ⊆ ℝ_{+} → ℝ be a twice differentiable mapping on I̊.

(i) If there exists an α ∈ ℝ such that

(3.13) Φ′′ (x) ≥ α for all x ∈ I̊,

then Φ is α−lower convex on I; (ii) If there exists a β ∈ ℝ such that

(3.14) Φ′′ (x) ≤ β for all x ∈ I̊,

then Φ is β−upper convex on I; (iii) If there exists m, M ∈ ℝ such that m < M and

(3.15) m ≤ Φ′′ (x) ≤ M for all x ∈ I̊,

then Φ is (m, M) −convex on I.

The proof is obvious by the well known fact that a twice differentiable mapping g : I → ℝ is convex iff g′′ (x) ≥ 0 for all x ∈I̊.

We omit the details.

In what follows, we will apply the previous result to some well known particular information measures which are Csiszár Φ−divergences for some appropriate choices of the mapping Φ.

10 S. S. DRAGOMIR

4. Applications for Some Particular Φ−Divergences Let us consider the Kullback-Leibler distance given by (1.9)

KL (p, q) := ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) . (4.1)

Consider the convex mapping Φ (t) = − log t, t > 0. For this mapping we have the Csiszár Φ-divergence

I_{Φ} (p, q) = ∑_{i=1}^{n} q_{i} [− log (\frac{p_{i}}{q_{i}} )] = ∑_{i=1}^{n} q_{i} log (\frac{q_{i}}{p_{i}} ) = KL (q, p) .

The following result holds. Proposition 3. Let p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}. Denote r_{i} := \frac{p_{i}}{q_{i}}, i = 1, ..., n.

(i) If r_{i} ≤ R < ∞, i = 1, ..., n, then we have the inequality

KL (q, p) ≥ \frac{1}{2R^{2}} D_{χ}^{2} (p, q) . (4.2)

(ii) If r_{i} ≥ r > 0, then we have the inequality

\frac{1}{2r^{2}} D_{χ}^{2} (p, q) ≥ KL (q, p) . (4.3)

(iii) If 0 < m ≤ r_{i} ≤ M < ∞, then we have

\frac{1}{2M^{2}} D_{χ}^{2} (p, q) ≤ KL (q, p) ≤ \frac{1}{2m^{2}} D_{χ}^{2} (p, q) . (4.4)

Proof. As Φ (t) = − log t, then Φ′ (t) = −\frac{1}{t}, Φ′′ (t) = \frac{1}{t^{2}} and if t ∈ [a, b] ⊂ (0, ∞), then \frac{1}{b^{2}} ≤ Φ′′ (t) ≤ \frac{1}{a^{2}} , t ∈ [a, b]. Using this and Theorem 6, we deduce the desired results.

We know that for Φ (t) = t log t, we have

I_{Φ} (p, q) = KL (p, q) .

Therefore, if we apply Theorem 6 for the mapping Φ (t) = t log t, we can state the following proposition. Proposition 4. Let p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n} and r_{i} := \frac{p_{i}}{q_{i}} (i = 1, ..., n).

(i) If r_{i} ≤ R < ∞, (i = 1, ..., n), then we have the inequality

\frac{1}{2R}D_{χ}^{2} (p, q) ≤ KL (p, q) . (4.5)

(ii) If r_{i} ≥ r > 0, then we have the inequality

KL (p, q) ≤ \frac{1}{2r} D_{χ}^{2} (p, q) . (4.6)

(iii) If 0 < m ≤ r_{i} ≤ M < ∞, (i = 1, ..., n), then we have the double inequality

\frac{1}{2M} D_{χ}^{2} (p, q) ≤ KL (p, q) ≤ \frac{1}{2m}D_{χ}^{2} (p, q) . (4.7)

SOME INEQUALITIES FOR (m, M)-CONVEX MAPPINGS 11

Now, let us consider the Rényi α−distance (α > 1)

D_{α} (p, q) := ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}, α > 1, p, q ∈ ℝ_{+}^{n}. (4.8)

We know that the Rényi α−distance is actually the Csiszár Φ−divergence for the mapping Φ (t) = t^{α}, α > 1 (see Example 3).

The following proposition holds. Proposition 5. Let p, q ∈ ℝ_{+}^{n}, P_{n} = Q_{n} and r_{i} := \frac{p_{i}}{q_{i}}, i = 1, ..., n.

(i) We have the inequality

(4.9) D_{α} (p, q) − P_{n} ≥ \{ \begin{matrix}\frac{α(α−1)r^{α−2}}{2} D_{χ}^{2} (p, q) \\ if α ∈ [2, ∞) and r_{i} ≥ r > 0, i = 1, ..., n, \\ \frac{α(α−1)R^{α−2}}{2} D_{χ}^{2} (p, q) \\ if α ∈ (1, 2) and r_{i} ≤ R (i = 1, ..., n) . \end{matrix}

(ii) We have the inequality

(4.10) D_{α} (p, q) − P_{n} ≤ \{ \begin{matrix}\frac{α(α−1)}{2}R^{α−2}D_{χ}^{2} (p, q) \\ if α ∈ [2, ∞) and r_{i} ≤ R (i = 1, ..., n) , \\ \frac{α(α−1)}{2}r^{α−2}D_{χ}^{2} (p, q) \\ if α ∈ (1, 2) and r_{i} ≥ R (i = 1, ..., n) . \end{matrix}

(iii) If m ≤ r_{i} ≤ M and α ∈ [2, ∞), then we have

\frac{α (α − 1)}{2}m^{α−2}D_{χ}^{2} (p, q) ≤ D_{α} (p, q) − P_{n} ≤ \frac{α (α − 1)}{2}M^{α−2}D_{χ}^{2} (p, q) . (4.11)

If α ∈ (1, 2), then the reverse inequality holds in (4.11). Let us consider the following Hellinger discrimination [13]

h^{2} (p, q) := \frac{1}{2} ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} .

Consider the mapping Φ (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}. Then obviously

I_{Φ} (p, q) = h^{2} (p, q) .

We can state the following proposition. Proposition 6. Let p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}. Denote r_{i} := \frac{p_{i}}{q_{i}} (i = 1, ..., n).

(i) If r_{i} ≤ R < ∞ (i = 1, ..., n), then we have the inequality

\frac{1}{8\sqrt{R^{3}}} D_{χ}^{2} (p, q) ≤ h^{2} (p, q) . (4.12)

(ii) If r_{i} ≥ r > 0, then we have the inequality

h^{2} (p, q) ≤ \frac{1}{8\sqrt{r^{3}}} D_{χ}^{2} (p, q) . (4.13)

12 S. S. DRAGOMIR

(iii) If 0 < m ≤ r_{i} ≤ M < ∞, then we have

\frac{1}{8\sqrt{M^{3}}} D_{χ}^{2} (p, q) ≤ h^{2} (p, q) ≤ \frac{1}{8\sqrt{m^{3}}} D_{χ}^{2} (p, q) . (4.14)

Proof. As Φ (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}, we have Φ′ (t) = \frac{1}{2} − \frac{1}{2\sqrt{t}}, Φ′′ (t) = \frac{1}{4\sqrt{t^{3}}} and then

α = inf_{0<t≤R} Φ′′ (t) = \frac{1}{4\sqrt{R^{3}}} ,
β = sup_{t≥r} Φ′′ (t) = \frac{1}{4\sqrt{r^{3}}} .

Using Theorem 6 for Φ as above, we obtain the desired inequalities.

Now, let us consider the Bhattacharyya distance (see for example [22])

B (p, q) = ∑_{i=1}^{n} \sqrt{p_{i}q_{i}}.

Consider the mapping Φ (t) = −\sqrt{t}. Then obviously

Φ′ (t) = −\frac{1}{2\sqrt{t}}, t > 0;
Φ′′ (t) = \frac{1}{4\sqrt{t^{3}}} , t > 0,

and

I_{Φ} (p, q) = −B (p, q) .

The following proposition holds. Proposition 7. Let p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}. Denote r_{i} := \frac{p_{i}}{q_{i}} (i = 1, ..., n).

(i) If r_{i} ≤ R < ∞ (i = 1, ..., n), then we have the inequality

\frac{1}{8\sqrt{R^{3}}} D_{χ}^{2} (p, q) ≤ P_{n} − B (p, q) . (4.15)

(ii) If r_{i} ≥ r > 0 (i = 1, ..., n), then we have the inequality

P_{n} − B (p, q) ≤ \frac{1}{8\sqrt{r^{3}}} D_{χ}^{2} (p, q) (4.16)

(iii) If 0 < m ≤ r_{i} ≤ M < ∞, then

\frac{1}{8\sqrt{M^{3}}} D_{χ}^{2} (p, q) ≤ P_{n} − B (p, q) ≤ \frac{1}{8\sqrt{m^{3}}} D_{χ}^{2} (p, q) . (4.17)

The proof follows by Theorem 2 applied for the mapping Φ (t) = −\sqrt{t}.

Now, let us consider the Harmonic distance (see for example [22])

M (p, q) := ∑_{i=1}^{n} \frac{2p_{i}q_{i}}{p_{i} + q_{i}} .

If Φ (t) = −\frac{2t}{t+1}, t ∈ (0, ∞), then obviously,

f′ (t) = −\frac{2}{(t + 1)^{2}} , t > 0
f′′ (t) = \frac{4}{(t + 1)^{3}} , t > 0

SOME INEQUALITIES FOR (m, M)-CONVEX MAPPINGS 13

and

I_{Φ} (p, q) = −M (p, q) .

The following proposition holds. Proposition 8. Let p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}. Denote r_{i} := \frac{p_{i}}{q_{i}} (i = 1, ..., n).

(i) If r_{i} ≤ R < ∞ (i = 1, ..., n), then we have the inequality:

\frac{2}{(R + 1)^{3}} D_{χ}^{2} (p, q) ≤ P_{n} − M (p, q) ; (4.18)

(ii) If r_{i} ≥ r > 0, then

P_{n} − M (p, q) ≤ \frac{2}{(r + 1)^{3}} D_{χ}^{2} (p, q) ; (4.19)

(iii) If 0 < m ≤ r_{i} ≤ M, then we have the sandwich inequality

\frac{2}{(M + 1)^{3}} D_{χ}^{2} (p, q) ≤ P_{n} − M (p, q) ≤ \frac{2}{(m + 1)^{3}} D_{χ}^{2} (p, q) . (4.20)

The proof follows by Theorem 6 applied for the mapping Φ (t) = −\frac{2t}{t+1}. Finally, let us consider the Jeffreys distance (see for example [22])

J (p, q) = ∑_{i=1}^{n} (p_{i} − q_{i}) ln (\frac{p_{i}}{q_{i}} ) .

If Φ (t) = (t − 1) ln t, t ∈ (0, ∞), then obviously,

f′ (t) = ln t − \frac{1}{t} + 1; t > 0;
f′′ (t) = \frac{t + 1}{t^{2} + 1}

and I_{Φ} (p, q) = J (p, q).

The following proposition holds. Proposition 9. Let p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}. Denote r_{i} := \frac{p_{i}}{q_{i}} (i = 1, ..., n).

(i) If r_{i} ≤ R < ∞ (i = 1, ..., n), then we have the inequality

\frac{R + 1}{2R^{2}}D_{χ}^{2} (p, q) ≤ J (p, q) ; (4.21)

(ii) If r_{i} ≥ r > 0 (i = 1, ..., n), then we have the inequality

J (p, q) ≤ \frac{r + 1}{2r^{2}}D_{χ}^{2} (p, q) ; (4.22)

(iii) If 0 < m ≤ r_{i} ≤ M < ∞, then we have the sandwich inequality

\frac{M + 1}{2M^{2}}D_{χ}^{2} (p, q) ≤ J (p, q) ≤ \frac{m + 1}{2m^{2}}D_{χ}^{2} (p, q) . (4.23)

References

[1] I. CSISZÁR, Information measures: A critical survey, Trans. 7th Prague Conf. on Info. Th., Statist. Decis. Funct., Random Processes and 8th European Meeting of Statist., Volume B, Academia Prague, 1978, 73-86.

[2] I. CSISZÁR, Information-type measures of difference of probability functions and indirect observations, Studia Sci. Math. Hungar, 2 (1967), 299-318.

[3] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[4] J.H. JUSTICE (editor), Maximum Entropy and Boyssian Methods in Applied Statistics, Cambridge University Press, Cambridge, 1986.

[5] J.N. KAPUR, On the roles of maximum entropy and minimum discrimination information principles in Statistics, Technical Address of the 38th Annual Conference of the Indian Society of Agricultural Statistics, 1984, T-44.

[6] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy functions, IEEE Transactions on Information Theory, 28 (1982), 489-495.

[7] R.G. GALLAGER, Information Theory and Reliable Communications, J. Wiley, New York, 1968.

[8] C.E. SHANNON, A mathematical theory of communication, Bull. Sept. Tech. J., 27 (1948), 370-423 and 623-656.

[9] B.R. FRIEDEN, Image enhancement and restoration, Picture Processing and Digital Filtering (T.S. Huang, Editor), Springer-Verlag, Berlin, 1975.

[10] R.M. LEAHY and C.E. GOUTIS, An optimal technique for constraint-based image restoration and mensuration, IEEE Trans. on Acoustics, Speech and Signal Processing, 34 (1986), 1692-1642.

[11] S. KULLBACK, Information Theory and Statistics, J. Wiley, New York, 1959.

[12] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[13] R. BERAH, Minimum Hellinger distance estimates for parametric models, Ann. Statist., 5 (1977), 445-463.

[14] A. RENYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Statist. Prob., Vol. 1, University of California Press, Berkeley, 1961.

[15] S.S. DRAGOMIR and N.M. IONESCU, Some converse of Jensen’s inequality and applications, Anal. Num. Theor. Approx., 23 (1994), 71-78.

[16] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s discrete inequality for differentiable convex mappings and applications in information theory, Math. Comput. Modelling, 24 (2) (1996), 1-11.

[17] S.S. DRAGOMIR and C.J. GOH, Some counterpart inequalities in for a functional associated with Jensen’s inequality, J. of Ineq. & Appl., 1 (1997), 311-325.

[18] S.S. DRAGOMIR and C.J. GOH, Some bounds on entropy measures in information theory, Appl. Math. Lett., 10 (1997), 23-28.

[19] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s continuous inequality and applications in information theory, RGMIA Preprint,

http://matilda.vu.edu.au/~rgmia/InfTheory/Continuse.dvi

[20] M. MATIĆ, Jensen’s inequality and applications in Information Theory (in Croatian), Ph.D. Thesis, Univ. of Zagreb, 1999.

[21] S.S. DRAGOMIR, J. ŠUNDE and M. SCHOLZ, Some upper bounds for relative entropy and applications, Comp. and Math. with Appl. ( in press).

[22] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), No. 1, 1-16.

[23] D. ANDRICA and I. RAŞA, The Jensen inequality: refinements and applications, Anal. Numer. Theor. Approx., 74 (1985), 105-108.

[24] S.S. DRAGOMIR, Some inequalities for the Csiszár Φ-divergence, submitted.

[25] S.S. DRAGOMIR, A converse inequality for the Csiszár Φ-divergence, submitted.

School of Communications and Informatics, Victoria University of Technology, PO Box 14428, Melbourne City MC 8001, Victoria, Australia

E-mail address: sever.dragomir@vu.edu.au
URL: http://rgmia.vu.edu.au/SSDragomirWeb.html