A REFINEMENT OF JENSEN’S INEQUALITY WITH APPLICATIONS FOR f-DIVERGENCE MEASURES

S.S. DRAGOMIR

Abstract. A refinement of the discrete Jensen’s inequality for convex functions defined on a convex subset in linear spaces is given. Application for f-divergence measures including the Kullback-Leibler and Jeffreys divergences are provided as well.

1. Introduction

The Jensen inequality for convex functions plays a crucial role in the Theory of Inequalities due to the fact that other inequalities such as that arithmetic meangeometric mean inequality, Hölder and Minkowski inequalities, Ky Fan’s inequality etc. can be obtained as particular cases of it.

Let C be a convex subset of the linear space X and f a convex set on C. If p = (p_{1}, …, p_{n}) is a probability sequence and x = (x_{1}, …, x_{n}) ∈ C^{n}, then

(1.1)  f (∑_{i=1}^{n} p_{i}x_{i}) ≤ ∑_{i=1}^{n} p_{i}f (x_{i}) ,

is well known in the literature as Jensen’s inequality.

In 1989, J. Pečarić and the author obtained the following refinement of (1.1):

(1.2)  f (∑_{i=1}^{n} p_{i}x_{i}) ≤ ∑_{i_{1},...,i_{k+1}=1}^{n} p_{i_{1}} … p_{i_{k+1}}f (\frac{x_{i_{1}} + ⋯ + x_{i_{k+1}}}{k + 1} )

≤ ∑_{i_{1},...,i_{k}=1}^{n} p_{i_{1}} … p_{i_{k}}f (\frac{x_{i_{1}} + ⋯ + x_{i_{k}}}{k} )

≤ ⋯ ≤ ∑_{i=1}^{n} p_{i}f (x_{i}) ,

for k ≥ 1 and p,x as above.

Date: 14 August, 2007. Kullback-Leibler divergence, Jeffreys divergence.

2000 Mathematics Subject Classification. 26D15, 94A17.

Key words and phrases. Convex functions, Jensen’s inequality, f-divergence measures,

If q_{1}, …, q_{k} ≥ 0 with ∑_{j=1}^{k} q_{j} = 1, then the following refinement obtained in 1994 by the author [6] also holds:

(1.3)  f (∑_{i=1}^{n} p_{i}x_{i}) ≤ ∑_{i_{1},...,i_{k}=1}^{n} p_{i_{1}} … p_{i_{k}}f (\frac{x_{i_{1}} + ⋯ + x_{i_{k}}}{k} )

≤ ∑_{i_{1},...,i_{k}=1}^{n} p_{i_{1}} … p_{i_{k}}f (q_{1}x_{i_{1}} + ⋯ + q_{k}x_{i_{k}})

≤ ∑_{i=1}^{n} p_{i}f (x_{i}) ,

where 1 ≤ k ≤ n and p, x are as above.

For other refinements and applications related to Ky Fan’s inequality, the arithmetic mean-geometric mean inequality, the generalised triangle inequality etc., see [3]-[8].

The main aim of the present paper is to establish a different refinement of the Jensen inequality for convex functions defined on linear spaces. Natural applications for the generalised triangle inequality in normed spaces and for the arithmetic mean-geometric mean inequality for positive numbers are given. Further applications for f-divergence measures of Csiszár with particular instances for the total variation distance, χ^{2}-divergence, Kullback-Leibler and Jeffreys divergences are provided as well.

2. General Results

The following result may be stated.

Theorem 1. Let f : C → ℝ be a convex function on the convex subset C of the linear space X, x_{i} ∈ C, p_{i} > 0, i ∈ {1, …, n} with ∑_{i=1}^{n} p_{i} = 1. Then

(2.1) f (∑_{j=1}^{n} p_{j}x_{j}) ≤ min_{k∈\{1,...,n\}} [(1 − p_{k}) f (\frac{∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}}{1 − p_{k}} ) + p_{k}f (x_{k})]

≤ \frac{1}{n} [∑_{k=1}^{n} (1 − p_{k}) f (\frac{∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}}{1 − p_{k}} ) + ∑_{k=1}^{n} p_{k}f (x_{k})]

≤ max_{k∈\{1,...,n\}} [(1 − p_{k}) f (\frac{∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}}{1 − p_{k}} ) + p_{k}f (x_{k})]

≤ ∑_{j=1}^{n} p_{j}f (x_{j}) .

In particular,

(2.2)  f (\frac{1}{n} ∑_{j=1}^{n} x_{j}) ≤ \frac{1}{n} min_{k∈\{1,...,n\}} [(n − 1) f (\frac{∑_{j=1}^{n} x_{j} − x_{k}}{n − 1} ) + f (x_{k})]

≤ \frac{1}{n^{2}} [(n − 1) ∑_{k=1}^{n} f (\frac{∑_{j=1}^{n} x_{j} − x_{k}}{n − 1} ) + ∑_{k=1}^{n} f (x_{k})]

JENSEN’S INEQUALITY WITH APPLICATIONS FOR f-DIVERGENCE MEASURES 3

≤ \frac{1}{n} max_{k∈\{1,...,n\}} [(n − 1) f (\frac{∑_{j=1}^{n} x_{j} − x_{k}}{n − 1} ) + f (x_{k})]

≤ \frac{1}{n} ∑_{j=1}^{n} f (x_{j}) .

Proof. For any k ∈ {1, …, n} , we have

∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k} = ∑_{{j=1 \atop j≠k}}^{n} p_{j}x_{j} = \frac{∑_{{j=1 \atop j≠k}}^{n} p_{j}}{∑_{{j=1 \atop j≠k}}^{n} p_{j}} ∑_{{j=1 \atop j≠k}}^{n} p_{j}x_{j} = (1 − p_{k}) · \frac{1}{∑_{{j=1 \atop j≠k}}^{n} p_{j}} ∑_{{j=1 \atop j≠k}}^{n} p_{j}x_{j}

which implies that

(2.3)  \frac{∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}}{1 − p_{k}} = \frac{1}{∑_{{j=1 \atop j≠k}}^{n} p_{j}} ∑_{{j=1 \atop j≠k}}^{n} p_{j}x_{j} ∈ C

for each k ∈ {1, …, n} , since the right side of (2.3) is a convex combination of the elements x_{j} ∈ C, j ∈ {1, …, n} \ {k} .

Taking the function f on (2.3) and applying the Jensen inequality, we get successively

f (\frac{∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}}{1 − p_{k}} ) = f ( \frac{1}{∑_{{j=1 \atop j≠k}}^{n} p_{j}} ∑_{{j=1 \atop j≠k}}^{n} p_{j}x_{j}) ≤ \frac{1}{∑_{{j=1 \atop j≠k}}^{n} p_{j}} ∑_{{j=1 \atop j≠k}}^{n} p_{j}f (x_{j})

= \frac{1}{1 − p_{k}} [∑_{j=1}^{n} p_{j}f (x_{j}) − p_{k}f (x_{k})]

for any k ∈ {1, …, n} , which implies

(2.4)  (1 − p_{k}) f (\frac{∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}}{1 − p_{k}} ) + p_{k}f (x_{k}) ≤ ∑_{j=1}^{n} p_{j}f (x_{j})

for each k ∈ {1, …, n} .

Utilising the convexity of f, we also have

(2.5) (1 − p_{k}) f (\frac{∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}}{1 − p_{k}} ) + p_{k}f (x_{k})

≥ f [(1 − p_{k}) · \frac{∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}}{1 − p_{k}} + p_{k}x_{k}] = f (∑_{j=1}^{n} p_{j}x_{j})

for each k ∈ {1, …, n} .

Taking the minimum over k in (2.5), utilising the fact that

min_{k∈\{1,...,n\}} α_{k} ≤ \frac{1}{n} ∑_{k=1}^{n} α_{k} ≤ max_{k∈\{1,...,n\}} α_{k}

and then taking the maximum in (2.4), we deduce the desired inequality (2.1).

The following corollary may be stated as well.

Corollary 1. Let f : C → ℝ be a convex function on the convex subset C, 0 ∈ C, y_{j} ∈ X and q_{j} > 0, j ∈ {1, …, n} with ∑_{j=1}^{n} q_{j} = 1. If y_{j} − ∑_{l=1}^{n} q_{l}y_{l} ∈ C for any j ∈ {1, …, n} , then

f (0) (2.6)

≤ min_{k∈\{1,...,n\}} \{(1 − q_{k}) f [\frac{q_{k}}{1 − q_{k}} (∑_{l=1}^{n} q_{l}y_{l} − y_{k})] + q_{k}f (y_{k} − ∑_{l=1}^{n} q_{l}y_{l})\}

≤ \frac{1}{n} \{∑_{l=1}^{n} (1 − q_{k}) f [\frac{q_{k}}{1 − q_{k}} (∑_{l=1}^{n} q_{l}y_{l} − y_{k})] + ∑_{l=1}^{n} q_{k}f (y_{k} − ∑_{l=1}^{n} q_{l}y_{l})\}

≤ max_{k∈\{1,...,n\}} \{(1 − q_{k}) f [\frac{q_{k}}{1 − q_{k}} (∑_{l=1}^{n} q_{l}y_{l} − y_{k})] + q_{k}f (y_{k} − ∑_{l=1}^{n} q_{l}y_{l})\}

≤ ∑_{j=1}^{n} q_{j}f (y_{j} − ∑_{l=1}^{n} q_{l}y_{l}) .

In particular, if y_{j} − \frac{1}{n} ∑_{l=1}^{n} y_{l} ∈ C for any j ∈ \{1, …, n\} , then

f (0) (2.7)

≤ \frac{1}{n} min_{k∈\{1,...,n\}} \{(n − 1) f [\frac{1}{n − 1} (\frac{1}{n} ∑_{l=1}^{n} y_{l} − y_{k})] + f (y_{k} − \frac{1}{n} ∑_{l=1}^{n} y_{l})\}

≤ \frac{1}{n^{2}} \{(n − 1) ∑_{k=1}^{n} f [\frac{1}{n − 1} (\frac{1}{n} ∑_{l=1}^{n} y_{l} − y_{k})] + ∑_{k=1}^{n} f (y_{k} − \frac{1}{n} ∑_{l=1}^{n} y_{l})\}

≤ \frac{1}{n} max_{k∈\{1,...,n\}} \{(n − 1) f [\frac{1}{n − 1} (\frac{1}{n} ∑_{l=1}^{n} y_{l} − y_{k})] + f (y_{k} − \frac{1}{n} ∑_{l=1}^{n} y_{l})\}

≤ \frac{1}{n} ∑_{j=1}^{n} f (y_{j} − \frac{1}{n} ∑_{l=1}^{n} y_{l}) .

The above results can be applied for various convex functions related to celebrated inequalities as mentioned in the introduction.

Application 1. If (X, ∥·∥) is a normed linear space and p ≥ 1, then the function f : X → ℝ, f (x) = ∥x∥^{p} is convex on X. Now, on applying Theorem 1 and

JENSEN’S INEQUALITY WITH APPLICATIONS FOR f-DIVERGENCE MEASURES 5

Corollary 1 for x_{i} ∈ X, p_{i} > 0, i ∈ {1, …, n} with ∑_{i=1}^{n} p_{i} = 1, we get:

(2.8) ∥∑_{j=1}^{n} p_{j}x_{j}∥^{p} ≤ min_{k∈{1,...,n}} [(1 − p_{k})^{1−p} ∥∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}∥^{p} + p_{k} ∥x_{k}∥^{p}]

≤ \frac{1}{n} [∑_{k=1}^{n} (1 − p_{k})^{1−p} ∥∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}∥^{p} + ∑_{k=1}^{n} p_{k} ∥x_{k}∥^{p}]

≤ max_{k∈{1,...,n}} [(1 − p_{k})^{1−p} ∥∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}∥^{p} + p_{k} ∥x_{k}∥^{p}]

≤ ∑_{j=1}^{n} p_{j} ∥x_{j}∥^{p}

and

(2.9) max_{k∈{1,...,n}} {[(1 − p_{k})^{1−p} p_{k}^{p} + p_{k}] ∥x_{k} − ∑_{l=1}^{n} p_{l}x_{l}∥^{p}}

≤ ∑_{j=1}^{n} p_{j} ∥x_{k} − ∑_{l=1}^{n} p_{l}x_{l}∥^{p} .

In particular, we have the inequality:

(2.10)  ∥\frac{1}{n} ∑_{j=1}^{n} x_{j}∥^{p} ≤ \frac{1}{n} min_{k∈\{1,...,n\}} [(n − 1)^{1−p} ∥∑_{j=1}^{n} x_{j} − x_{k}∥^{p} + ∥x_{k}∥^{p}]

≤ \frac{1}{n^{2}} [(n − 1)^{1−p} ∑_{k=1}^{n} ∥∑_{j=1}^{n} x_{j} − x_{k}∥^{p} + ∑_{k=1}^{n} ∥x_{k}∥^{p}]

≤ \frac{1}{n} max_{k∈\{1,...,n\}} [(n − 1)^{1−p} ∥∑_{j=1}^{n} x_{j} − x_{k}∥^{p} + ∥x_{k}∥^{p}]

≤ \frac{1}{n} ∑_{j=1}^{n} ∥x_{j}∥^{p}

and

(2.11) [(n − 1)^{1−p} + 1] max_{k∈\{1,...,n\}} ∥x_{k} − \frac{1}{n} ∑_{l=1}^{n} x_{l}∥^{p} ≤ ∑_{j=1}^{n} ∥x_{j} − \frac{1}{n} ∑_{l=1}^{n} x_{l}∥^{p} .

If we consider the function h_{p} (t) := (1 − t)^{1−p} t^{p} + t, p ≥ 1, t ∈ [0, 1) , then we observe that

h′_{p} (t) = 1 + pt^{p−1} (1 − t)^{1−p} + (p − 1) t^{p} (1 − t)^{−p} ,

which shows that h_{p} is strictly increasing on [0, 1) . Therefore,

min_{k∈{1,...,n}} {(1 − p_{k})^{1−p} p_{k}^{p} + p_{k}} = p_{m} + (1 − p_{m})^{1−p} p_{m}^{p},

where p_{m} := min_{k∈{1,...,n}} p_{k}. By (2.9), we then obtain the following inequality:

(2.12) [p_{m} + (1 − p_{m})^{1−p} · p_{m}^{p}] max_{k∈{1,...,n}} ∥x_{k} − ∑_{l=1}^{n} p_{l}x_{l}∥^{p}

≤ ∑_{j=1}^{n} p_{j} ∥x_{j} − ∑_{l=1}^{n} p_{l}x_{l}∥^{p} .

Application 2. Let x_{i}, p_{i} > 0, i ∈ {1, …, n} with ∑_{i=1}^{n} p_{i} = 1. The following inequality is well known in the literature as the arithmetic mean-geometric mean inequality:

(2.13)  ∑_{j=1}^{n} p_{j}x_{j} ≥ ∏_{j=1}^{n} x_{j}^{p_{j}}.

The equality case holds in (2.13) iff x_{1} = ⋯ = x_{n}.

Applying the inequality (2.1) for the convex function f : (0, ∞) → ℝ, f (x) = − ln x and performing the necessary computations, we derive the following refinement of (2.13):

(2.14)  ∑_{i=1}^{n} p_{i}x_{i} ≥ max_{k∈\{1,...,n\}} \{(\frac{∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}}{1 − p_{k}} )^{1−p_{k}} · x_{k}^{p_{k}} \}

≥ ∏_{k=1}^{n} [(\frac{∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}}{1 − p_{k}} )^{1−p_{k}} · x_{k}^{p_{k}} ]^{\frac{1}{n}}

≥ min_{k∈\{1,...,n\}} \{(\frac{∑_{j=1}^{n} p_{j}x_{j} − p_{k}x_{k}}{1 − p_{k}} )^{1−p_{k}} · x_{k}^{p_{k}} \} ≥ ∏_{i=1}^{n} x_{i}^{p_{i}}.

In particular, we have the inequality:

\frac{1}{n} ∑_{i=1}^{n} x_{i} ≥ max_{k∈\{1,...,n\}} \{(\frac{∑_{j=1}^{n} x_{j} − x_{k}}{n − 1} )^{\frac{n−1}{n}} · xk^{\frac{1}{n}} \}

≥ ∏_{k=1}^{n} [(\frac{∑_{j=1}^{n} x_{j} − x_{k}}{n − 1} )^{\frac{n−1}{n}} · xk^{\frac{1}{n}} ]^{\frac{1}{n}}

≥ min_{k∈\{1,...,n\}} \{(\frac{∑_{j=1}^{n} x_{j} − x_{k}}{n − 1} )^{\frac{n−1}{n}} · xk^{\frac{1}{n}} \} ≥ (∏_{i=1}^{n} x_{i})^{\frac{1}{n}} .

3. Applications for f-Divergences

Given a convex function f : [0, ∞) → ℝ, the f-divergence functional

(3.1)  I_{f} (p, q) := ∑_{i=1}^{n} q_{i}f (\frac{p_{i}}{q_{i}} ) ,

where p = (p_{1}, …, p_{n}) , q = (q_{1}, …, q_{n}) are positive sequences was introduces by Csiszár in [1], as a generalised measure of information, a “distance function” on the set of probability distributions ℙ^{n}. As in [1], we interpret undefined expressions by

f (0) = lim_{t→0+} f (t) ,  0f (\frac{0}{0}) = 0,
0f (\frac{a}{0}) = lim_{q→0+} f (\frac{a}{q}) = a lim_{t→∞} \frac{f (t)}{t}, a > 0.

The following results were essentially given by Csiszár and Körner [2]: (i) If f is convex, then I_{f} (p, q) is jointly convex in p and q; (ii) For every p, q ∈ R_{+}^{n}, we have

(3.2)  I_{f} (p, q) ≥ ∑_{j=1}^{n} q_{j}f (\frac{∑_{j=1}^{n} p_{j}}{∑_{j=1}^{n} q_{j}} ) .

If f is strictly convex, equality holds in (3.2) iff

\frac{p_{1}}{q_{1}} = \frac{p_{2}}{q_{2}} = ⋯ = \frac{p_{n}}{q_{n}} .

If f is normalised, i.e., f (1) = 0, then for every p, q ∈ R_{+}^{n} with ∑_{i=1}^{n} p_{i} =

∑_{i=1}^{n} q_{i}, we have the inequality
(3.3)  I_{f} (p, q) ≥ 0.

In particular, if p, q ∈ ℙ^{n}, then (3.3) holds. This is the well-known positive property of the f-divergence.

The following refinement of (3.3) may be stated.

Theorem 2. For any p, q ∈ ℙ^{n}, we have the inequalities

(3.4)  I_{f} (p, q) ≥ max_{k∈\{1,...,n\}} [(1 − q_{k}) f (\frac{1 − p_{k}}{1 − q_{k}} ) + q_{k}f (\frac{p_{k}}{q_{k}} )]

≥ \frac{1}{n} [∑_{k=1}^{n} (1 − q_{k}) f (\frac{1 − p_{k}}{1 − q_{k}} ) + ∑_{k=1}^{n} q_{k}f (\frac{p_{k}}{q_{k}} )]

≥ min_{k∈\{1,...,n\}} [(1 − q_{k}) f (\frac{1 − p_{k}}{1 − q_{k}} ) + q_{k}f (\frac{p_{k}}{q_{k}} )] ≥ 0,

provided f : [0, ∞) → ℝ is convex and normalised on [0, ∞) .

The proof is obvious by Theorem 1 applied for the convex function f : [0, ∞) → ℝ and for the choice x_{i} = \frac{p_{i}}{q_{i}} , i ∈ \{1, …, n\} and the probabilities q_{i}, i ∈ \{1, …, n\} .

If we consider a new divergence measure R_{f} (p, q) defined by

(3.5)  R_{f} (p, q) := \frac{1}{n − 1} ∑_{k=1}^{n} (1 − q_{k}) f (\frac{1 − p_{k}}{1 − q_{k}} )

and call it the reverse f−divergence, we observe that

(3.6)  R_{f} (p, q) = I_{f} (r, t)

with

r = (\frac{1 − p_{1}}{n − 1} , …, \frac{1 − p_{n}}{n − 1} ) , t = (\frac{1 − q_{1}}{n − 1} , …, \frac{1 − q_{n}}{n − 1} ) (n ≥ 2) .

With this notation, we can state the following corollary of the above proposition.

Corollary 2. For any p, q ∈ ℙ^{n}, we have

(3.7)  I_{f} (p, q) ≥ R_{f} (p, q) ≥ 0.

The proof is obvious by the second inequality in (3.4) and the details are omitted. In what follows, we point out some particular inequalities for various instances of

divergence measures such as: the total variation distance, χ^{2}-divergence, Kullback-Leibler divergence, Jeffreys divergence.

The total variation distance is defined by the convex function f (t) = |t − 1| , t ∈ ℝ and given in:

(3.8)  V (p, q) := ∑_{j=1}^{n} q_{j} |\frac{p_{j}}{q_{j}} − 1| = ∑_{j=1}^{n} |p_{j} − q_{j}| .

The following improvement of the positivity inequality for the total variation distance can be stated as follows.

Proposition 1. For any p, q ∈ ℙ^{n}, we have the inequality:

(3.9)  V (p, q) ≥ 2 max_{k∈{1,...,n}} |p_{k} − q_{k}| (≥ 0) .

The proof follows by the first inequality in (3.7) for f (t) = |t − 1| , t ∈ ℝ.

The K. Pearson χ^{2}-divergence is obtained for the convex function f (t) = (1 − t)^{2} , t ∈ ℝ and given by

(3.10)  χ^{2} (p, q) := ∑_{j=1}^{n} q_{j} (\frac{p_{j}}{q_{j}} − 1)^{2} = ∑_{j=1}^{n} \frac{(p_{j} − q_{j})^{2}}{q_{j}} .

Proposition 2. For any p, q ∈ ℙ^{n},

(3.11)  χ^{2} (p, q) ≥ max_{k∈\{1,...,n\}} \{\frac{(p_{k} − q_{k})^{2}}{q_{k} (1 − q_{k})}\} ≥ 4 max_{k∈\{1,...,n\}} (p_{k} − q_{k})^{2} (≥ 0) .

Proof. On applying the first inequality in Theorem 2 for the function f (t) = (1 − t)^{2} , t ∈ ℝ, we get

χ^{2} (p, q) ≥ max_{k∈\{1,...,n\}} \{(1 − q_{k}) (\frac{1 − p_{k}}{1 − q_{k}} − 1)^{2} + q_{k} (\frac{p_{k}}{q_{k}} − 1)^{2}\}

= max_{k∈\{1,...,n\}} \{\frac{(p_{k} − q_{k})^{2}}{q_{k} (1 − q_{k})}\} .

Since

q_{k} (1 − q_{k}) ≤ \frac{1}{4} [q_{k} + (1 − q_{k})]^{2} = \frac{1}{4},

then

\frac{(p_{k} − q_{k})^{2}}{q_{k} (1 − q_{k})} ≥ 4 (p_{k} − q_{k})^{2}

for each k ∈ {1, …, n} , which proves the last part of (3.11).

The Kullback-Leibler divergence can be obtained for the convex function f : (0, ∞) → ℝ, f (t) = t ln t and is defined by

(3.12)  KL (p, q) := ∑_{j=1}^{n} q_{j} · \frac{p_{j}}{q_{j}} ln (\frac{p_{j}}{q_{j}} ) = ∑_{j=1}^{n} p_{j} ln (\frac{p_{j}}{q_{j}} ) .

Proposition 3. For any p, q ∈ ℙ^{n}, we have:

(3.13)  KL (p, q) ≥ ln [ max_{k∈\{1,...,n\}} \{(\frac{1 − p_{k}}{1 − q_{k}} )^{1−p_{k}} · (\frac{p_{k}}{q_{k}} )^{p_{k}}\}]

≥ ln [ max_{k∈\{1,...,n\}} \{\frac{(1 − q_{k}) q_{k}}{q_{k} (1 − p_{k})^{2} + (1 − q_{k}) p_{k}^{2}} \}] .

Proof. The first inequality is obvious by Theorem 2. Utilising the inequality between the geometric mean and the harmonic mean,

x^{α}y^{1−α} ≥ \frac{1}{\frac{α}{x} + \frac{1−α}{y}} , x, y > 0, α ∈ [0, 1]

we have

(\frac{1 − p_{k}}{1 − q_{k}} )^{1−p_{k}} · (\frac{p_{k}}{q_{k}} )^{p_{k}} ≥ \frac{1}{(1 − p_{k}) · (\frac{1−p_{k}}{1−q_{k}} ) + p_{k} · \frac{p_{k}}{q_{k}}}

= \frac{(1 − q_{k}) q_{k}}{q_{k} (1 − p_{k})^{2} + (1 − q_{k}) p_{k}^{2}} ,

for any k ∈ {1, …, n} , which implies the second part of (3.13).

Another divergence measure that is of importance in Information Theory is the Jeffreys divergence

(3.14)  J (p, q) := ∑_{j=1}^{n} q_{j} · (\frac{p_{j}}{q_{j}} − 1) ln (\frac{p_{j}}{q_{j}} ) = ∑_{j=1}^{n} (p_{j} − q_{j}) ln (\frac{p_{j}}{q_{j}} ) ,

which is an f-divergence for f (t) = (t − 1) ln t, t > 0.

Proposition 4. For any p, q ∈ ℙ^{n}, we have:

(3.15)  J (p, q) ≥ max_{k∈\{1,...,n\}} \{(q_{k} − p_{k}) ln [\frac{(1 − p_{k}) q_{k}}{(1 − q_{k}) p_{k}} ]\}

≥ max_{k∈\{1,...,n\}} [\frac{(q_{k} − p_{k})^{2}}{p_{k} + q_{k} − 2p_{k}q_{k}} ] ≥ 0.

Proof. Writing the first inequality in Theorem 2 for f (t) = (t − 1) ln t, we have

J (p, q) ≥ max_{k∈\{1,...,n\}} \{(1 − q_{k}) [(\frac{1 − p_{k}}{1 − q_{k}} − 1) ln (\frac{1 − p_{k}}{1 − q_{k}} )] + q_{k} (\frac{p_{k}}{q_{k}} − 1) ln (\frac{p_{k}}{q_{k}} )\}

= max_{k∈\{1,...,n\}} \{(q_{k} − p_{k}) ln (\frac{1 − p_{k}}{1 − q_{k}} ) − (q_{k} − p_{k}) ln (\frac{p_{k}}{q_{k}} )\}

= max_{k∈\{1,...,n\}} \{(q_{k} − p_{k}) ln [\frac{(1 − p_{k}) q_{k}}{(1 − q_{k}) p_{k}} ]\} ,

proving the first inequality in (3.15).

Utilising the elementary inequality for positive numbers,

\frac{ln b − ln a}{b − a} ≥ \frac{2}{a + b}, a, b > 0

we have

(q_{k} − p_{k}) [ln (\frac{1 − p_{k}}{1 − q_{k}} ) − ln (\frac{p_{k}}{q_{k}} )]

= (q_{k} − p_{k}) · \frac{ln (\frac{1−p_{k}}{1−q_{k}} ) − ln (\frac{p_{k}}{q_{k}} )}{\frac{1−p_{k}}{1−q_{k}} − \frac{p_{k}}{q_{k}}} · [\frac{1 − p_{k}}{1 − q_{k}} − \frac{p_{k}}{q_{k}} ]

= \frac{(q_{k} − p_{k})^{2}}{q_{k} (1 − q_{k})} · \frac{ln (\frac{1−p_{k}}{1−q_{k}} ) − ln (\frac{p_{k}}{q_{k}} )}{\frac{1−p_{k}}{1−q_{k}} − \frac{p_{k}}{q_{k}}}

≥ \frac{(q_{k} − p_{k})^{2}}{q_{k} (1 − q_{k})} · \frac{2}{\frac{1−p_{k}}{1−q_{k}} − \frac{p_{k}}{q_{k}}} = \frac{2 (q_{k} − p_{k})^{2}}{p_{k} + q_{k} − 2p_{k}q_{k}} ≥ 0,

for each k ∈ {1, …, n} , giving the second inequality in (3.15).

References

[1] I. CSISZÁR, Information-type measures of differences of probability distributions and indirect observations, Studia Sci. Math. Hung., 2 (1967), 299-318.

[2] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorems for Discrete Memoryless Systems, Academic Press, New York, 1981.

[3] S.S. DRAGOMIR, An improvement of Jensen’s inequality, Bull. Math. Soc. Sci. Math. Roumanie, 34(82) (1990), No. 4, 291-296.

[4] S.S. DRAGOMIR, Some refinements of Ky Fan’s inequality, J. Math. Anal. Appl., 163(2) (1992), 317-321.

[5] S.S. DRAGOMIR, Some refinements of Jensen’s inequality, J. Math. Anal. Appl., 168(2) (1992), 518-522.

[6] S.S. DRAGOMIR, A further improvement of Jensen’s inequality, Tamkang J. Math., 25(1) (1994), 29-36.

[7] S.S. DRAGOMIR, A new improvement of Jensen’s inequality, Indian J. Pure and Appl. Math., 26(10) (1995), 959-968.

[8] S.S. DRAGOMIR, J. PEČARIĆ and L.E. PERSSON, Properties of some functionals related to Jensen’s inequality, Acta Math. Hung., 70(1-2) (1996), 129-143.

[9] J. PEČARIĆ and S.S. DRAGOMIR, A refinements of Jensen inequality and applications, Studia Univ. Babeş-Bolyai, Mathematica, 24(1) (1989), 15-19.

School of Computer Science and Mathematics, Victoria University, PO Box 14428, Melbourne City Mail Centre, VIC, 8001, Australia.

   E-mail address: sever.dragomir@vu.edu.au
   URL: http://rgmia.vu.edu.au/dragomir