SOME INEQUALITIES FOR THE CSISZÁR ϕ−DIVERGENCE

WHEN ϕ IS AN L- LIPSCHITZIAN FUNCTION AND APPLICATIONS

S.S. DRAGOMIR

Abstract. Some inequalities of Jessen’s type for vector valued Lipschitzian functions and applications for the discrete Csiszár ϕ−divergence are given.

1. Introduction

Given a convex function Φ : ℝ_{+} → ℝ_{+}, the Φ- divergence functional

(1.1)  I_{Φ} (p, q) := ∑_{i=1}^{n} q_{i}Φ (\frac{p_{i}}{q_{i}} ) ,

was introduced in Csiszár [1], [2] as a generalized measure of information, a “distance function” on the set of probability distributions ℙ^{n}. The restriction here to discrete distribution is only for convenience, similar results hold for general distributions.

As in Csiszár [2], we interpret undefined expressions by

Φ (0) = lim_{t→0+} Φ (t) , 0Φ (\frac{0}{0}) = 0,
0Φ (\frac{a}{0}) = lim_{ε→0+} Φ (\frac{a}{ε}) = a lim_{t→∞} \frac{Φ (t)}{t}, a > 0.

The following results were essentially given by Csiszár and Körner [3]. Theorem 1. If Φ : ℝ_{+} → ℝ is convex, then I_{Φ} (p, q) is jointly convex in p and q.

The following lower bound for the Φ−divergence functional also holds. Theorem 2. Let Φ : ℝ_{+} → ℝ_{+} be convex. Then for every p, q ∈ ℝ_{+}^{n}, we have the inequality:

(1.2)  I_{Φ} (p, q) ≥ ∑_{i=1}^{n} q_{i}Φ (\frac{∑_{i=1}^{n} p_{i}}{∑_{i=1}^{n} q_{i}} ) .

If Φ is strictly convex, equality holds in (1.2) iff

(1.3)  \frac{p_{1}}{q_{1}} = \frac{p_{2}}{q_{2}} = ... = \frac{p_{n}}{q_{n}} .

Date: June 20, 2000.

1991 Mathematics Subject Classification. Primary 26D15, 46A99.

Key words and phrases. Csiszár ϕ−Divergence, L- Lipschitzian Function.

Corollary 1. Let Φ : ℝ_{+} → ℝ be convex and normalized, i.e.,

(1.4)  Φ (1) = 0.

Then for any p, q ∈ ℝ_{+}^{n} with

(1.5)  ∑_{i=1}^{n} p_{i} = ∑_{i=1}^{n} q_{i}

we have the inequality

(1.6)  I_{Φ} (p, q) ≥ 0.

If Φ is strictly convex, the equality holds in (1.6) iff p_{i} = q_{i} for all i ∈ {1, ..., n}.

In particular, if p, q are probability vectors, then (1.5) is assured. Corollary 1 then shows, for strictly convex and normalized Φ : ℝ_{+} → ℝ,

(1.7)  I_{Φ} (p, q) ≥ 0 for all p, q ∈ ℙ^{n}.

The equality holds in (1.7) iff p = q.

These are “distance properties”. However, I_{Φ} is not a metric: It violates the triangle inequality, and is asymmetric, i.e, for general p, q ∈ ℝ_{+}^{n}, I_{Φ} (p, q) ≠ I_{Φ} (q, p).

In the examples below we obtain, for suitable choices of the kernel Φ, some of the best known distance functions I_{Φ} used in mathematical statistics [4]-[5], information theory [6]-[8] and signal processing [9]-[10]. Example 1. (Kullback-Leibler) For

(1.8)  Φ (t) := t log t, t > 0

the Φ−divergence is

(1.9)  I_{Φ} (p, q) = ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) ,

the Kullback-Leibler distance [11]-[12]. Example 2. (Hellinger) Let

(1.10)  Φ (t) = (1 − \sqrt{t})^{2} , t > 0.

Then I_{Φ} gives the Hellinger distance [13]

(1.11)  I_{Φ} (p, q) = ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} ,

which is symmetric. Example 3. (Renyi) For α > 1, let

(1.12)  Φ (t) = t^{α}, t > 0.

Then

(1.13)  I_{Φ} (p, q) = ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α},

which is the α−order entropy [14].

CSISZÁR ϕ−DIVERGENCE 3

Example 4. (χ^{2}−distance) Let

(1.14)  Φ (t) = (t − 1)^{2} , t > 0.

Then

(1.15)  I_{Φ} (p, q) = ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}}

is the χ^{2}−distance between p and q.

Finally, we have Example 5. (Variational distance). Let Φ (t) = |t − 1| , t > 0. The corresponding divergence, called the variational distance, is symmetric,

I_{Φ} (p, q) = ∑_{i=1}^{n} |p_{i} − q_{i}| .

For other examples of divergence measures, see the paper [22] by J.N. Kapur, where further references are given.

2. Some Inequalities for Lipschitzian Mappings. The Case of l_{∞}−Norm

2.1. Some Inequalities. We start with the following result. Theorem 3. Let X,Y be two normed linear spaces with the norms ∥·∥ and |·| respectively. If F : X → Y is L - Lipschitzian, that is,

(2.1)  |F (x) − F (y)| ≤ L ∥x − y∥ for all x, y ∈ X,

then for all x_{i} ∈ X, p_{i} ≥ 0 with ∑_{i=1}^{n} p_{i} = 1 (i = 1, ..., n), we have the inequality

(2.2)  |F (∑_{i=1}^{n} p_{i}x_{i}) − ∑_{i=1}^{n} p_{i}F (x_{i})| ≤ L max_{k=1,...,n−1} ∥Δx_{k}∥ ∑_{i,j=1}^{n} p_{i}p_{j} |i − j| .

Proof. As F is L−Lipschitzian, we can choose x = ∑_{i=1}^{n} p_{i}x_{i} and y = x_{j} (j = 1, ..., n) in (2.1) to get

|F (∑_{i=1}^{n} p_{i}x_{i}) − F (x_{j})| ≤ L ∥∑_{i=1}^{n} p_{i}x_{i} − x_{j}∥ (2.3)

= L ∥∑_{i=1}^{n} p_{i} (x_{i} − x_{j})∥ ≤ L ∑_{i=1}^{n} p_{i} ∥x_{i} − x_{j}∥ .

Multiplying (2.3) by p_{j} ≥ 0 and summing over j from 1 to n, we deduce

(2.4)  ∑_{j=1}^{n} p_{j} |F (∑_{i=1}^{n} p_{i}x_{i}) − F (x_{j})| ≤ L ∑_{i,j=1}^{n} p_{i}p_{j} ∥x_{j} − x_{i}∥ .

By the generalised triangle inequality we have

(2.5)  ∑_{j=1}^{n} p_{j} |F (∑_{i=1}^{n} p_{i}x_{i}) − F (x_{j})| ≥ |F (∑_{i=1}^{n} p_{i}x_{i}) − ∑_{j=1}^{n} p_{j}F (x_{j})| .

4 S.S. DRAGOMIR

Also, it is obvious that

(2.6) ∑_{i,j=1}^{n} p_{i}p_{j} ∥x_{i} − x_{j}∥ = 2 ∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{j} − x_{i}∥ .

Now, observe that, for i < j, we have

x_{j} − x_{i} = ∑_{k=i}^{j−1} Δx_{k},

where Δx_{k} := x_{k+1} − x_{k} is the forward difference.

Therefore, by the generalised triangle inequality we have

∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{j} − x_{i}∥ (2.7)

= ∑_{1≤i<j≤n} p_{i}p_{j} ∥∑_{k=i}^{j−1} Δx_{k}∥ ≤ ∑_{1≤i<j≤n} p_{i}p_{j} ∑_{k=i}^{j−1} ∥Δx_{k}∥
≤ ∑_{1≤i<j≤n} p_{i}p_{j} (j − i) max_{k=1,...,n−1} ∥Δx_{k}∥
= max_{k=1,...,n−1} ∥Δx_{k}∥ ∑_{1≤i<j≤n} p_{i}p_{j} (j − i)
= \frac{1}{2} max_{k=1,...,n−1} ∥Δx_{k}∥ ∑_{i,j=1}^{n} p_{i}p_{j} |j − i| .

Using (2.4) - (2.7) we deduce the desired inequality (2.2). Corollary 2. With the above assumptions for F and x_{i} (i = 1, ..., n), we have the inequality

(2.8)  |F (\frac{1}{n} ∑_{i=1}^{n} x_{i}) − \frac{1}{n} ∑_{i=1}^{n} F (x_{i})| ≤ L · \frac{n^{2} − 1}{3n} max_{k=1,...,n−1} ∥Δx_{k}∥ .

Proof. We choose p_{i} = \frac{1}{n} (i = 1, ..., n) in (2.2) and have to compute

I := ∑_{i,j=1}^{n} |i − j| .

We observe that

∑_{j=1}^{n} |i − j| = ∑_{j=1}^{i} |i − j| + ∑_{j=i+1}^{n} |i − j| = ∑_{j=1}^{i} (i − j) + ∑_{j=i+1}^{n} (j − i)
= i^{2} − \frac{i (i + 1)}{2} + ∑_{j=1}^{n} j − i (n − i)
= i^{2} − (n + 1) i + \frac{n (n + 1)}{2} = \frac{n^{2} − 1}{4} + (i − \frac{n + 1}{2})^{2} .

Then

I = ∑_{i=1}^{n} (∑_{j=1}^{n} |i − j|) = ∑_{i=1}^{n} [i^{2} − (n + 1) i + \frac{n (n + 1)}{2}]

= \frac{(n − 1) n (n + 1)}{3},

and the inequality (2.8) is proved.

The following corollary provides a counterpart of the generalised triangle inequality in normed spaces.

Corollary 3. Let (X, ∥·∥) be a normed space and x_{i} ∈ X, p_{i} ≥ 0 (i = 1, ..., n) with ∑_{i=1}^{n} p_{i} = 1. Then we have the inequality

(2.9)  0 ≤ ∑_{i=1}^{n} p_{i} ∥x_{i}∥ − ∥∑_{i=1}^{n} p_{i}x_{i}∥ ≤ max_{k=1,...,n−1} ∥Δx_{k}∥ ∑_{i,j=1}^{n} p_{i}p_{j} |j − i|

and in particular

(2.10)  0 ≤ ∑_{i=1}^{n} ∥x_{i}∥ − ∥∑_{i=1}^{n} x_{i}∥ ≤ \frac{n^{2} − 1}{3} max_{k=1,...,n−1} ∥Δx_{k}∥ .

The proof is obvious by Theorem 3 on choosing F : X → ℝ, F (X) = ∥x∥ which is L−Lipschitzian with L = 1 as

|∥x∥ − ∥y∥| ≤ ∥x − y∥ for all x, y ∈ X.

2.2. Applications for Csiszár ϕ−Divergence. The following theorem holds.

Theorem 4. Let ϕ : ℝ_{+} → ℝ be L−Lipschitzian on ℝ_{+}. Then for all p, q ∈ ℝ_{+}^{n}, we have the inequality

(2.11) |I_{ϕ} (p, q) − Q_{n}ϕ (\frac{P_{n}}{Q_{n}} )| ≤ L max_{k=1,...,n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} | \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

Proof. We apply inequality (2.2) for F = ϕ and p_{i} = \frac{q_{i}}{Q_{n}}, x_{i} = \frac{p_{i}}{q_{i}} to get

|ϕ (\frac{∑_{i=1}^{n} p_{i}}{Q_{n}} ) − \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i}ϕ (\frac{p_{i}}{q_{i}} )|

≤ L max_{k=1,...,n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} | \frac{1}{Q_{n}^{2}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

from where we obtain (2.11).

Corollary 4. Let ϕ : ℝ_{+} → ℝ be L−Lipschitzian and normalised. Then for any p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}, we have the inequality

(2.12)  0 ≤ |I_{ϕ} (p, q)| ≤ L max_{k=1,...,n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} | \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

6 S.S. DRAGOMIR

Corollary 5. Let ϕ : ℝ_{+} → ℝ be differentiable convex, with a bounded derivative, that is, ∥ϕ′∥_{∞} := sup_{t≥0} |ϕ′ (t)| < ∞. Then we have the inequality:

(2.13)  0 ≤ I_{ϕ} (p, q) − Q_{n}ϕ (\frac{P_{n}}{Q_{n}} )
≤ ∥ϕ′∥_{∞} max_{k=1,...,n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} | \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

Moreover, if ϕ is normalised and P_{n} = Q_{n}, then

(2.14)  0 ≤ I_{ϕ} (p, q) ≤ ∥ϕ′∥_{∞} max_{k=1,...,n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} | \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

Recall the Kullback-Leibler distance given by (1.9)

(2.15)  KL (p, q) := ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) .

We observe that, for the convex mapping ϕ (t) := − log (t), t > 0,

(2.16)  I_{ϕ} (p, q) = ∑_{i=1}^{n} q_{i} [− log (\frac{p_{i}}{q_{i}} )] = ∑_{i=1}^{n} q_{i} log (\frac{q_{i}}{p_{i}} ) = KL (q, p) .

The following proposition for the Kullback-Leibler distance holds. Proposition 1. Let p, q ∈ ℝ_{+}^{n} satisfy the condition

(2.17)  0 < m ≤ r_{k} := \frac{p_{k}}{q_{k}} for all k = 1, …, n.

Then, we have the inequality

(2.18)  0 ≤ KL (q, p) − Q_{n} log (\frac{Q_{n}}{P_{n}} )
≤ \frac{1}{m} max_{k=1,...,n−1} |r_{k+1} − r_{k}| \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

Proof. As ϕ (t) = − log (t), then ϕ′ (t) = −\frac{1}{t}, t > 0. The restriction of ϕ′ in the interval [m, ∞) is bounded and ∥ϕ′∥_{∞} = sup_{t∈[m,∞)} |\frac{1}{t} | = \frac{1}{m} < ∞. Applying the inequality (2.13), we deduce (2.18). Remark 1. If we assume that P_{n} = Q_{n}, then m ≤ 1 and (2.18) becomes

(2.19)  0 ≤ KL (q, p) ≤ \frac{1}{m} max_{k=1,...,n−1} |r_{k+1} − r_{k}| \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

We also know that for ϕ (t) = t log t, t > 0, the Csiszár ϕ−divergence is (see (1.9)), ϕ (p, q) = KL (p, q) = ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ).

The following proposition also holds.

Proposition 2. Let p, q ∈ ℝ_{+}^{n} satisfy the condition

(2.20)  0 < m ≤ r_{k} ≤ M < ∞ for all k = 1, …, n.

Then we have the inequality

(2.21)  0 ≤ KL (q, p) − P_{n} log (\frac{P_{n}}{Q_{n}} )
≤ max {|log (Ml)| , |log (ml)|}

× max_{k=1,...,n−1} |r_{k+1} − r_{k}| \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

Proof. For the mapping ϕ (t) = t log t, t > 0, we have ϕ′ (t) = log (t) + 1. On the interval [m, M] we have

log (m) + 1 ≤ ϕ′ (M) ≤ log (M) + 1, t ∈ [m, M] .

Applying the inequality (2.13), we deduce (2.21).

Remark 2. If we assume that P_{n} = Q_{n}, then m ≤ 1 ≤ M and (2.21) becomes

(2.22)  0 ≤ KL (q, p)
≤ max {|log (Ml)| , |log (ml)|}

× max_{k=1,...,n−1} |r_{k+1} − r_{k}| \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

Let ϕ (t) = (\sqrt{t} − 1)^{2}, t > 0. Then I_{ϕ} gives the Hellinger distance

H_{e} (p, q) = ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} .

Using Corollary 5, we can state the following proposition. Proposition 3. Let p, q ∈ ℝ_{+}^{n} satisfy the condition (2.20). Then we have the inequality

(2.23)  0 ≤ H_{e} (p, q) − (\sqrt{P_{n}} − \sqrt{Q_{n}})^{2}

≤ max \{\frac{|\sqrt{m} − 1|}{\sqrt{m}}, \frac{|\sqrt{M} − 1|}{\sqrt{M}}\}

× max_{k=1,...,n−1} |r_{k+1} − r_{k}| \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

Proof. As ϕ (t) = (\sqrt{t} − 1)^{2}, t > 0, then ϕ′ (t) = 1− \frac{1}{\sqrt{t}}. If we consider the mapping ϕ′ restricted to the interval [m, M] ⊂ (0, ∞), then we observe that

\frac{|\sqrt{m} − 1|}{\sqrt{m}} ≤ ϕ′ (t) ≤ \frac{|\sqrt{M} − 1|}{\sqrt{M}}, t ∈ [m, M]

and thus

∥ϕ′∥_{∞} ≤ max \{\frac{|\sqrt{m} − 1|}{\sqrt{m}}, \frac{|\sqrt{M} − 1|}{\sqrt{M}}\} .

Remark 3. If we assume that P_{n} = Q_{n}, then m ≤ 1 ≤ M and (2.23) becomes

(2.24)  0 ≤ H_{e} (p, q)
≤ max \{\frac{|1 − \sqrt{m}|}{\sqrt{m}}, \frac{|\sqrt{M} − 1|}{\sqrt{M}}\}

× max_{k=1,...,n−1} |r_{k+1} − r_{k}| \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

Consider now the mapping ϕ (t) = t^{α}, α > 1, t > 0 and the α−order entropy by Rényi Re_{α} (p, q) := ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}.

We can state the following proposition.

Proposition 4. Let p, q ∈ ℝ_{+}^{n} be such that

(2.25)  0 < r_{k} ≤ M < ∞ for all k = 1, …, n.

Then we have the inequality

(2.26)  0 ≤ Re_{α} (p, q) − P_{n}^{α}Q_{n}^{1−α}
≤ αM^{α−1} max_{k=1,...,n−1} |r_{k+1} − r_{k}| \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

In particular, if P_{n} = Q_{n}, then M ≥ 1 and (2.26) becomes

(2.27)  0 ≤ Re_{α} (p, q) − Q_{n}
≤ αM^{α−1} max_{k=1,...,n−1} |r_{k+1} − r_{k}| \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

The proof is obvious by Corollary 5 applied for ϕ (t) = t^{α}, and we omit the details.

Finally, if we consider the χ^{2}−distance (see (1.15))

D_{χ}^{2} (p, q) = ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}}

obtained from the Csiszár ϕ−divergence for the choice ϕ (t) = (t − 1)^{2}, t > 0, then we can state the following proposition as well.

Proposition 5. Let p, q ∈ ℝ_{+}^{n} fulfill the properties of (2.20). Then we have the converse inequality

(2.28)  0 ≤ D_{χ}^{2} (p, q) − \frac{1}{Q_{n}} (P_{n} − Q_{n})^{2}
≤ 2 max {|m − 1| , |M − 1|}

× max_{k=1,...,n−1} |r_{k+1} − r_{k}| \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

CSISZÁR ϕ−DIVERGENCE 9

In particular, if P_{n} = Q_{n}, then m ≤ 1 ≤ M and (2.28) becomes

(2.29)  0 ≤ D_{χ}^{2} (p, q)
≤ 2 max {1 − m, M − 1}

× max_{k=1,...,n−1} |r_{k+1} − r_{k}| \frac{1}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |i − j| .

3. The Case of l_{1}−Norm

3.1. Some Inequalities. We start with the following result. Theorem 5. Let X,Y be two normed linear spaces with the norms ∥·∥ and |·| respectively. If F : X → Y is L - Lipschitzian, that is,

(3.1)  |F (x) − F (y)| ≤ L ∥x − y∥ for all x, y ∈ X,

then for all x_{i} ∈ X, p_{i} ≥ 0 with ∑_{i=1}^{n} p_{i} = 1 (i = 1, ..., n), we have the inequality

(3.2)  |F (∑_{i=1}^{n} p_{i}x_{i}) − ∑_{i=1}^{n} p_{i}F (x_{i})| ≤ L ∑_{i=1}^{n} p_{i} (1 − p_{i}) ∑_{k=1}^{n−1} ∥Δx_{k}∥ .

Proof. As F is L−Lipschitzian, we can state (see the proof of Theorem 3) that

(3.3)  |F (∑_{i=1}^{n} p_{i}x_{i}) − ∑_{j=1}^{n} p_{j}F (x_{j})| ≤ 2L ∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{j} − x_{i}∥ .

in (3.1). Now, observe that, for i < j, we have

x_{j} − x_{i} = ∑_{k=i}^{j−1} Δx_{k},

where Δx_{k} := x_{k+1} − x_{k} is the forward difference.

Therefore, as in the proof of Theorem 3, we have

∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{j} − x_{i}∥ (3.4)

= ∑_{1≤i<j≤n} p_{i}p_{j} ∥∑_{k=i}^{j−1} Δx_{k}∥ ≤ ∑_{1≤i<j≤n} p_{i}p_{j} ∑_{k=i}^{j−1} ∥Δx_{k}∥
≤ ∑_{1≤i<j≤n} p_{i}p_{j} ∑_{k=1}^{n−1} ∥Δx_{k}∥ .

Now, if we put

I = ∑_{1≤i<j≤n} p_{i}p_{j},

we observe that

1 = ∑_{i,j=1}^{n} p_{i}p_{j} = 2 ∑_{1≤i<j≤n} p_{i}p_{j} + ∑_{i=1}^{n} p_{i}^{2} = 2I + ∑_{i=1}^{n} p_{i}^{2}

from where we deduce

I = \frac{1}{2} ∑_{i=1}^{n} p_{i} (1 − p_{i}) .

Now, using (3.3) - (3.4) we deduce the desired inequality (2.2).

Corollary 6. With the above assumptions for F and x_{i} (i = 1, ..., n), we have the inequality

(3.5)  |F (\frac{1}{n} ∑_{i=1}^{n} x_{i}) − \frac{1}{n} ∑_{i=1}^{n} F (x_{i})| ≤ L · \frac{n − 1}{n} ∑_{k=1}^{n−1} ∥Δx_{k}∥ .

The proof is obvious by the above theorem, choosing p_{i} = \frac{1}{n} (i = 1, ..., n).

The following corollary provides a counterpart of the generalised triangle inequality. Corollary 7. Let (X, ∥·∥) be a normed space and x_{i} ∈ X, p_{i} ≥ 0 (i = 1, ..., n) with ∑_{i=1}^{n} p_{i} = 1. Then we have the inequality

(3.6)  0 ≤ ∑_{i=1}^{n} p_{i} ∥x_{i}∥ − ∥∑_{i=1}^{n} p_{i}x_{i}∥ ≤ ∑_{i=1}^{n} p_{i} (1 − p_{i}) ∑_{k=1}^{n−1} ∥Δx_{k}∥

and in particular

(3.7)  0 ≤ ∑_{i=1}^{n} ∥x_{i}∥ − ∥∑_{i=1}^{n} x_{i}∥ ≤ (n − 1) ∑_{k=1}^{n−1} ∥Δx_{k}∥ .

The proof is obvious by Theorem 5 on choosing F : X → ℝ, F (X) = ∥x∥ which is L−Lipschitzian with L = 1 as

|∥x∥ − ∥y∥| ≤ ∥x − y∥ for all x, y ∈ X.

3.2. Applications for Csiszár ϕ−Divergence. The following theorem holds. Theorem 6. Let ϕ : ℝ_{+} → ℝ be L−Lipschitzian on ℝ_{+}. Then for all p, q ∈ ℝ_{+}^{n}, we have the inequality

(3.8)  |I_{ϕ} (p, q) − Q_{n}ϕ (\frac{P_{n}}{Q_{n}} )| ≤ \frac{L}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} | .

Proof. We apply inequality (3.2) for F = ϕ and p_{i} = \frac{q_{i}}{Q_{n}}, x_{i} = \frac{p_{i}}{q_{i}} (i = 1, …, n) to get

|ϕ (\frac{P_{n}}{Q_{n}} ) − \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i}ϕ (\frac{p_{i}}{q_{i}} )| ≤ L ∑_{i=1}^{n} \frac{q_{i} (Q_{n} − q_{i})}{Q_{n}^{2}} ∑_{k=1}^{n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} | .

from where we obtain (3.8).

Corollary 8. Let ϕ : ℝ_{+} → ℝ be L−Lipschitzian and normalised. Then for any p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}, we have the inequality

(3.9)  0 ≤ |I_{ϕ} (p, q)| ≤ \frac{L}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} | .

Corollary 9. Let ϕ : ℝ_{+} → ℝ be differentiable convex, with a bounded derivative. Then we have the inequality:

(3.10) 0 ≤ I_{ϕ} (p, q) − Q_{n}ϕ (\frac{P_{n}}{Q_{n}} ) ≤ \frac{∥ϕ′∥_{∞}}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} | .

Moreover, if ϕ is normalised and P_{n} = Q_{n}, then

(3.11)  0 ≤ I_{ϕ} (p, q) ≤ \frac{∥ϕ′∥_{∞}}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} | .

Now, let us recall the Kullback-Leibler distance

(3.12)  KL (p, q) := ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) .

We observe that, for the convex mapping ϕ (t) := − log (t), t > 0,

(3.13)  I_{ϕ} (p, q) = ∑_{i=1}^{n} q_{i} [− log (\frac{p_{i}}{q_{i}} )] = ∑_{i=1}^{n} q_{i} log (\frac{q_{i}}{p_{i}} ) = KL (q, p) .

The following proposition for the Kullback-Leibler distance holds. Proposition 6. Let p, q ∈ ℝ_{+}^{n} satisfy the condition

(3.14)  0 < m ≤ r_{k} := \frac{p_{k}}{q_{k}} for all k = 1, …, n.

Then, we have the inequality

(3.15)  0 ≤ KL (q, p) − Q_{n} log (\frac{Q_{n}}{P_{n}} )
≤ \frac{1}{mQ_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |r_{k+1} − r_{k}| .

Proof. As ϕ (t) = − log (t), then ϕ′ (t) = −\frac{1}{t}, t > 0. The restriction of ϕ′ in the interval [m, ∞) is bounded and ∥ϕ′∥_{∞} = sup_{t∈[m,∞)} |\frac{1}{t} | = \frac{1}{m} < ∞. Applying the

inequality (3.10), we deduce (3.15).

Remark 4. If we assume that P_{n} = Q_{n}, then m ≤ 1 and (3.15) becomes

(3.16)  0 ≤ KL (q, p) ≤ \frac{1}{mQ_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |r_{k+1} − r_{k}| .

We also know that for ϕ (t) = t log t, t > 0, the Csiszár ϕ−divergence is

ϕ (p, q) = KL (p, q) = ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) .

The following proposition also holds. Proposition 7. Let p, q ∈ ℝ_{+}^{n} satisfy the condition

(3.17)  0 < m ≤ r_{k} ≤ M < ∞ for all k = 1, …, n.

Then we have the inequality

(3.18) 0 ≤ KL (q, p) − P_{n} log (\frac{P_{n}}{Q_{n}} )
≤ \frac{max \{|log (Ml)| , |log (ml)|\}}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |r_{k+1} − r_{k}| .

Proof. For the mapping ϕ (t) = t log t, t > 0, we have ϕ′ (t) = log (t) + 1. On the interval [m, M] we have

log m + 1 ≤ ϕ′ (t) ≤ log M + 1, t ∈ [m, M]

and hence

|ϕ′ (t)| ≤ max {|log (Ml)| , |log (ml)|} , t ∈ [m, M] .

Applying the inequality (3.10), we deduce (3.18).

Remark 5. If we assume that P_{n} = Q_{n}, then (2.21) becomes

(3.19) 0 ≤ KL (q, p)
≤ \frac{max \{|log (Ml)| , |log (ml)|\}}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |r_{k+1} − r_{k}| .

Let ϕ (t) = (\sqrt{t} − 1)^{2}, t > 0. Then I_{ϕ} gives the Hellinger distance

He (p, q) = ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} ,

Using Corollary 9, we may state the following proposition. Proposition 8. Assume that p, q ∈ ℝ_{+}^{n} satisfy the condition (3.17). Then we have the inequality

0 ≤ He (p, q) − (\sqrt{P_{n}} − \sqrt{Q_{n}})^{2} (3.20)

≤ max \{\frac{|\sqrt{m} − 1|}{\sqrt{m}}, \frac{|\sqrt{M} − 1|}{\sqrt{M}}\} \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |r_{k+1} − r_{k}| .

Proof. As ϕ (t) = (\sqrt{t} − 1)^{2}, t > 0, then ϕ′ (t) = 1− \frac{1}{\sqrt{t}}. If we consider the mapping ϕ′ restricted to the interval [m, M] ⊂ (0, ∞), then we observe that

\frac{|\sqrt{m} − 1|}{\sqrt{m}} ≤ ϕ′ (t) ≤ \frac{|\sqrt{M} − 1|}{\sqrt{M}}, t ∈ [m, M]

and thus

∥ϕ′∥_{∞} ≤ max \{\frac{|\sqrt{m} − 1|}{\sqrt{m}}, \frac{|\sqrt{M} − 1|}{\sqrt{M}}\} .

Remark 6. If we assume that P_{n} = Q_{n}, then m ≤ 1 ≤ M and (2.23) becomes

0 ≤ He (p, q) (3.21)

≤ max \{\frac{1 − \sqrt{m}}{\sqrt{m}}, \frac{\sqrt{M} − 1}{\sqrt{M}}\} \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |r_{k+1} − r_{k}| .

Consider now the mapping ϕ (t) = t^{α}, α > 1, t > 0 and the α−order entropy by Rényi Re_{α} (p, q) := ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}.

We can state the following proposition.

Proposition 9. Let p, q ∈ ℝ_{+}^{n} be such that

(3.22)  0 < r_{k} ≤ M < ∞ for all k = 1, …, n.

Then we have the inequality

(3.23)  0 ≤ Re_{α} (p, q) − P_{n}^{α}Q_{n}^{1−α}
≤ αM^{α−1} \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |r_{k+1} − r_{k}| .

In particular, if P_{n} = Q_{n}, then M ≥ 1 and (3.23) becomes

(3.24)  0 ≤ Re_{α} (p, q) − Q_{n}
≤ αM^{α−1} \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |r_{k+1} − r_{k}| .

The proof is obvious by Corollary 9 applied for ϕ (t) = t^{α}, and we omit the details.

Finally, if we consider the χ^{2}−distance

D_{χ}^{2} (p, q) := ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}}

obtained from the Csiszár ϕ−divergence for the choice ϕ (t) = (t − 1)^{2}, then we can state the following proposition too. Proposition 10. Let p, q ∈ ℝ_{+}^{n} fulfill the conditions of (2.20). Then we have the inequality

(3.25) 0 ≤ D_{χ}^{2} (p, q) − \frac{1}{Q_{n}} (P_{n} − Q_{n})^{2}

≤ 2 max \{|m − 1| , |M − 1|\} \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |r_{k+1} − r_{k}| .

In particular, if P_{n} = Q_{n}, then (3.25) becomes

(3.26) 0 ≤ D_{χ}^{2} (p, q)
≤ 2 max \{1 − m, M − 1\} \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i} (Q_{n} − q_{i}) ∑_{k=1}^{n−1} |r_{k+1} − r_{k}| .

4. The Case of l_{p}−Norm

4.1. Some Inequalities. We start with the following result. Theorem 7. Let X, Y, F, (p_{i})_{i=\overline{1,n}} be as in Theorem 5. Then we have the inequality:

(4.1) |F (∑_{i=1}^{n} p_{i}x_{i}) − ∑_{i=1}^{n} p_{i}F (x_{i})| ≤ L ∑_{i,j=1}^{n} p_{i}p_{j} |j − i|^{\frac{1}{q}} (∑_{k=1}^{n−1} ∥Δx_{k}∥^{p})^{\frac{1}{p}} ,

where p > 1, \frac{1}{p} + \frac{1}{q} = 1.

Proof. As in the proof of Theorem 3, we have

(4.2)  |F (∑_{i=1}^{n} p_{i}x_{i}) − ∑_{j=1}^{n} p_{j}F (x_{j})| ≤ 2L ∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{j} − x_{i}∥ .

Also,

(4.3)  ∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{j} − x_{i}∥ ≤ ∑_{1≤i<j≤n} p_{i}p_{j} ∑_{k=i}^{j−1} ∥Δx_{k}∥ .

Using Hölder’s discrete inequality, we may write for p > 1, \frac{1}{p} + \frac{1}{q} = 1, that

∑_{k=i}^{j−1} ∥Δx_{k}∥ ≤ (∑_{k=i}^{j−1} 1)^{\frac{1}{q}} (∑_{k=i}^{j−1} ∥Δx_{k}∥^{p})^{\frac{1}{p}} = (j − i)^{\frac{1}{q}} (∑_{k=i}^{j−1} ∥Δx_{k}∥^{p})^{\frac{1}{p}}

≤ (j − i)^{\frac{1}{q}} (∑_{k=1}^{n−1} ∥Δx_{k}∥^{p})^{\frac{1}{p}}

and then, by (4.3), we get

∑_{1≤i<j≤n} p_{i}p_{j} ∥x_{j} − x_{i}∥ ≤ ∑_{1≤i<j≤n} p_{i}p_{j} (j − i)^{\frac{1}{q}} (∑_{k=1}^{n−1} ∥Δx_{k}∥^{p})^{\frac{1}{p}}

= \frac{1}{2} ∑_{i,j=1}^{n} p_{i}p_{j} |j − i|^{\frac{1}{q}} (∑_{k=1}^{n−1} ∥Δx_{k}∥^{p})^{\frac{1}{p}} .

Using (4.2) we deduce (4.1).

Corollary 10. Let (X, ∥·∥) be a normed space and x_{i} ∈ X, p_{i} ≥ 0 (i = 1, …, n) with ∑_{i=1}^{n} p_{i} = 1. Then we have the inequality:

(4.4)  0 ≤ ∑_{i=1}^{n} p_{i} ∥x_{i}∥ − ∥∑_{i=1}^{n} p_{i}x_{i}∥ ≤ ∑_{i,j=1}^{n} p_{i}p_{j} |j − i|^{\frac{1}{q}} (∑_{k=1}^{n−1} ∥Δx_{k}∥^{p})^{\frac{1}{p}} .

4.2. Applications for Csiszár ϕ−Divergence. The following result for Csiszár f−divergence holds.

Theorem 8. Let ϕ : ℝ_{+} → ℝ be L−Lipschitzian on ℝ_{+}. Then for all p, q ∈ ℝ_{+}^{n}, we have the inequality:

(4.5) |I_{ϕ} (p, q) − Q_{n}ϕ (\frac{P_{n}}{Q_{n}} )| ≤ \frac{L}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |j − i|^{\frac{1}{q}} (∑_{k=1}^{n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} |^{p})^{\frac{1}{p}} ,

where p > 1, \frac{1}{p} + \frac{1}{q} = 1.

Proof. We apply inequality (4.1) for F = ϕ, p_{i} = \frac{q_{i}}{Q_{n}}, x_{i} = \frac{p_{i}}{q_{i}} to get

|ϕ (\frac{P_{n}}{Q_{n}} ) − \frac{1}{Q_{n}} ∑_{i=1}^{n} q_{i}ϕ (\frac{p_{i}}{q_{i}} )|

≤ L ∑_{i,j=1}^{n} \frac{q_{i}}{Q_{n}} · \frac{q_{j}}{Q_{n}} |j − i|^{\frac{1}{q}} (∑_{k=1}^{n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} |^{p})^{\frac{1}{p}}

from where we obtain (4.5).

Corollary 11. Let ϕ : ℝ_{+} → ℝ be L−Lipschitzian and normalised. Then for all p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}, we have

(4.6)  0 ≤ |I_{ϕ} (p, q)| ≤ \frac{c}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |j − i|^{\frac{1}{q}} (∑_{k=1}^{n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} |^{p})^{\frac{1}{p}} .

Corollary 12. Let ϕ : ℝ_{+} → ℝ be differentiable convex with a bounded derivative. Then

(4.7)  0 ≤ I_{ϕ} (p, q) − Q_{n}ϕ (\frac{P_{n}}{Q_{n}} )

≤ \frac{∥ϕ′∥_{∞}}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |j − i|^{\frac{1}{q}} (∑_{k=1}^{n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} |^{p})^{\frac{1}{p}} .

Moreover, if ϕ is normalised and P_{n} = Q_{n}, then

(4.8)  0 ≤ I_{ϕ} (p, q) ≤ \frac{∥ϕ′∥_{∞}}{Q_{n}} ∑_{i,j=1}^{n} q_{i}q_{j} |j − i|^{\frac{1}{q}} (∑_{k=1}^{n−1} |\frac{p_{k+1}}{q_{k+1}} − \frac{p_{k}}{q_{k}} |^{p})^{\frac{1}{p}} .

Remark 7. Inequalities for particular divergences as in the previous two sections can be stated, but we omit the details.

References

[1] I. CSISZÁR, Information measures: A critical survey, Trans. 7th Prague Conf. on Info. Th., Statist. Decis. Funct., Random Processes and 8th European Meeting of Statist., Volume B, Academia Prague, 1978, 73-86.

[2] I. CSISZÁR, Information-type measures of difference of probability functions and indirect observations, Studia Sci. Math. Hungar, 2 (1967), 299-318.

[3] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[4] J.H. JUSTICE (editor), Maximum Entropy and Bayssian Methods in Applied Statistics, Cambridge University Press, Cambridge, 1986.

[5] J.N. KAPUR, On the roles of maximum entropy and minimum discrimination information principles in Statistics, Technical Address of the 38th Annual Conference of the Indian Society of Agricultural Statistics, 1984, 1-44.

[6] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy functions, IEEE Transactions on Information Theory, 28 (1982), 489-495.

[7] R.G. GALLAGER, Information Theory and Reliable Communications, J. Wiley, New York, 1968.

[8] C.E. SHANNON, A mathematical theory of communication, Bull. Sept. Tech. J., 27 (1948), 370-423 and 623-656.

[9] B.R. FRIEDEN, Image enhancement and restoration, Picture Processing and Digital Filtering (T.S. Huang, Editor), Springer-Verlag, Berlin, 1975.

[10] R.M. LEAHY and C.E. GOUTIS, An optimal technique for constraint-based image restoration and mensuration, IEEE Trans. on Acoustics, Speech and Signal Processing, 34 (1986), 1692-1642.

[11] S. KULLBACK, Information Theory and Statistics, J. Wiley, New York, 1959.

[12] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[13] R. BERAN, Minimum Hellinger distance estimates for parametric models, Ann. Statist., 5 (1977), 445-463.

[14] A. RENYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Statist. Prob., Vol. 1, University of California Press, Berkeley, 1961.

[15] S.S. DRAGOMIR and N.M. IONESCU, Some converse of Jensen’s inequality and applications, Anal. Num. Theor. Approx., 23 (1994), 71-78.

[16] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s discrete inequality for differentiable convex mappings and applications in information theory, Math. Comput. Modelling, 24 (2) (1996), 1-11.

[17] S.S. DRAGOMIR and C.J. GOH, Some counterpart inequalities in for a functional associated with Jensen’s inequality, J. of Ineq. & Appl., 1 (1997), 311-325.

[18] S.S. DRAGOMIR and C.J. GOH, Some bounds on entropy measures in information theory, Appl. Math. Lett., 10 (1997), 23-28.

[19] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s continuous inequality and applications in information theory, RGMIA Preprint,

    http://matilda.vu.edu.au/~rgmia/InfTheory/Continuse.dvi

[20] M. MATIĆ, Jensen’s inequality and applications in Information Theory (in Croatian), Ph.D. Thesis, Univ. of Zagreb, 1999.

[21] S.S. DRAGOMIR, J. ŠUNDE and M. SCHOLZ, Some upper bounds for relative entropy and applications, Comp. and Math. with Appl. ( in press).

[22] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), No. 1, 1-16.

[23] S.S. DRAGOMIR, Some inequalities for the Csiszár Φ-divergence, submitted.

School of Communications and Informatics, Victoria University of Technology, Melbourne City MC, Victoria 8001, Australia.

   E-mail address: sever@matilda.vu.edu.au
   URL: http://rgmia.vu.edu.au/SSDragomirWeb.html