Bounds for the Normalised Jensen Functional

Sever S. Dragomir

Abstract. New inequalities for the general case of convex functions defined on linear spaces which improve the famous Jensen’s inequality are established. Particular instances in the case of normed spaces and for complex and real n-tuples are given. Refinements of Shannon’s inequality and the positivity of Kullback-Leibler divergence are obtained.

1. Introduction

Jensen’s inequality for convex function is one of the most known and extensively used inequality in various filed of Modern Mathematics. It is a source of many classical inequalities including the generalised triangle inequality, the arithmetic mean-geometric mean-harmonic mean inequality, the positivity of relative entropy in Information Theory, Schannon’s inequality, Ky Fan’s inequality, Levinson’s inequality and other results. For classical and contemporary developments related to the Jensen inequality, see [6], [7] and [3] where further references are provided.

To be more specific, we recall that, if X is a linear space and C ⊆ X a convex subset in X, then for any convex function f : C → ℝ and any z_{i} ∈ C, r_{i} ≥ 0 for i ∈ {1, ..., k} , k ≥ 2 with ∑_{i=1}^{k} r_{i} = R_{k} > 0 one has the weighted Jensen’s inequality:

(J)  \frac{1}{R_{k}} ∑_{i=1}^{k} r_{i}f (z_{i}) ≥ f (\frac{1}{R_{k}} ∑_{i=1}^{k} r_{i}z_{i}) .

If f : f : C → ℝ is strictly convex and r_{i} > 0 for i ∈ {1, ..., k} then the equality case hods in (J) if and only if z_{1} = ... = z_{n}.

The main aim of the present note is to provide an elementary refinement of this classical inequality and point out a few applications in relation with some fundamental inequalities in various fields of Mathematics.

1991 Mathematics Subject Classification. Primary 26D15; Secondary26D10.

Key words and phrases. Jensen’s Inequality, Convex Functions,

2. Bounds for the Normalised Jensen Functional

By P_{n} we denote the set of all nonnegative n-tuples (p_{1}, ..., p_{n}) with the property that ∑_{i=1}^{n} p_{i} = 1. Consider the normalised Jensen functional

J_{n} (f, x, p) = ∑_{i=1}^{n} p_{i}f (x_{i}) − f (∑_{i=1}^{n} p_{i}x_{i}) ≥ 0

where f : C → ℝ be a convex function on the convex set C and x = (x_{1}, ..., x_{n}) ∈ C^{n} and p ∈P_{n}.

The following result holds.

Theorem 1. If p, q ∈P_{n}, q_{i} > 0 for each i ∈ {1, ..., n} then

(2.1) max_{1≤i≤n} \{\frac{p_{i}}{q_{i}} \} J_{n} (f, x, q) ≥ J_{n} (f, x, p) ≥ min_{1≤i≤n} \{\frac{p_{i}}{q_{i}} \} J_{n} (f, x, q) (≥ 0) .

Proof. We give here a direct proof based on the Jensen inequality for appropriate choices of the elements in (J). The reader is invited to try to find other proofs, eventually simpler than this one.

Denote m := min_{1≤i≤n} \{\frac{p_{i}}{q_{i}} \} and observe that 0 ≤ m ≤ 1.

If we apply Jensen’s inequality (J) for k = n + 1,

z_{1} = ∑_{i=1}^{n} q_{i}x_{i}, z_{j+1} = x_{j} ∈ C, j ∈ {1, ..., n};

r_{1} = m, r_{j+1} = (\frac{p_{j}}{q_{j}} − m) q_{j} ≥ 0, j ∈ \{1, ..., n\}

for which

∑_{j=1}^{n+1} r_{j} = m + ∑_{j=1}^{n} (\frac{p_{j}}{q_{j}} − m) q_{j} = 1

then we have

mf (∑_{i=1}^{n} q_{i}x_{i}) + ∑_{j=1}^{n} (\frac{p_{j}}{q_{j}} − m) q_{j}f (x_{j}) (2.2)

≥ f [m (∑_{i=1}^{n} q_{i}x_{i}) + ∑_{j=1}^{n} (\frac{p_{j}}{q_{j}} − m) q_{j}x_{j}] .

Since

mf (∑_{i=1}^{n} q_{i}x_{i}) + ∑_{j=1}^{n} (\frac{p_{j}}{q_{j}} − m) q_{j}f (x_{j})

= ∑_{j=1}^{n} p_{j}f (x_{j}) − m [∑_{j=1}^{n} q_{j}f (x_{j}) − f (∑_{j=1}^{n} q_{j}x_{j})]

and

f [m (∑_{i=1}^{n} q_{i}x_{i}) + ∑_{j=1}^{n} (\frac{p_{j}}{q_{j}} − m) q_{j} (x_{j})] = f (∑_{j=1}^{n} p_{j}x_{j})

hence by (2.2) we get the second inequality in (2.1).

BOUNDS FOR THE NORMALISED JENSEN FUNCTIONAL 3

Now, let M := max_{1≤i≤n} \{\frac{p_{i}}{q_{i}} \} and observe that M ≥ 1.

If we apply Jensen’s inequality (J) for k = n + 1,

z_{j} = x_{j}, z_{n+1} = ∑_{i=1}^{n} p_{i}x_{i}, j ∈ {1, ..., n} ,
r_{j} = \frac{1}{M} (M − \frac{p_{j}}{q_{j}} ) q_{j}, r_{n+1} = \frac{1}{M} , j ∈ \{1, ..., n\}

for which

∑_{j=1}^{n+1} r_{j} = \frac{1}{M} ∑_{j=1}^{n} (M − \frac{p_{j}}{q_{j}} ) q_{j} + \frac{1}{M} = 1,

then we have

\frac{1}{M} ∑_{j=1}^{n} (M − \frac{p_{j}}{q_{j}} ) q_{j}f (x_{j}) + \frac{1}{M} f (∑_{i=1}^{n} p_{i}x_{i}) (2.3)

≥ f [\frac{1}{M} ∑_{j=1}^{n} (M − \frac{p_{j}}{q_{j}} ) q_{j}x_{j} + \frac{1}{M} ∑_{i=1}^{n} p_{i}x_{i}] .

Since

\frac{1}{M} ∑_{j=1}^{n} (M − \frac{p_{j}}{q_{j}} ) q_{j}f (x_{j}) + \frac{1}{M} f (∑_{i=1}^{n} p_{i}x_{i})

= ∑_{j=1}^{n} q_{j}f (x_{j}) − \frac{1}{M} [∑_{j=1}^{n} p_{j}f (x_{j}) − (∑_{i=1}^{n} p_{i}x_{i})]

and

f [\frac{1}{M} ∑_{j=1}^{n} (M − \frac{p_{j}}{q_{j}} ) q_{j}x_{j} + \frac{1}{M} ∑_{i=1}^{n} p_{i}x_{i}] = f (∑_{j=1}^{n} p_{j}x_{j}) ,

hence by (2.3) we deduce the first part of (2.1) and the proof is complete.

If we consider for the uniform distribution u = (\frac{1}{n}, ..., \frac{1}{n}) the unweighted Jensen functional

J_{n} (f, x) := J_{n} (f, x, u) = \frac{1}{n} ∑_{i=1}^{n} f (x_{i}) − f (\frac{1}{n} ∑_{i=1}^{n} x_{i})

then we can state the following particular case of interest as well:

Corollary 1. If p ∈P_{n}, then

(2.4)  n max_{1≤i≤n} {p_{i}} J_{n} (f, x) ≥ J_{n} (f, x, p) ≥ n min_{1≤i≤n} {p_{i}} J_{n} (f, x) (≥ 0) .

3. Applications in Normed Spaces

Let (X, ∥·∥) be a real or complex normed linear space. It is well known that the function f_{p} : X → ℝ, f_{p} (x) = ∥x∥^{p} , p ≥ 1 is convex on X. Applying the results obtained above one may easily state the following inequalities:

max_{1≤i≤n} \{\frac{p_{i}}{q_{i}} \} [∑_{j=1}^{n} q_{j} ∥x_{j}∥^{p} − ∥∑_{j=1}^{n} q_{j}x_{j}∥^{p}] (3.1)

≥ ∑_{j=1}^{n} p_{j} ∥x_{j}∥^{p} − ∥∑_{j=1}^{n} p_{j}x_{j}∥^{p}

≥ min_{1≤i≤n} \{\frac{p_{i}}{q_{i}} \} [∑_{j=1}^{n} q_{j} ∥x_{j}∥^{p} − ∥∑_{j=1}^{n} q_{j}x_{j}∥^{p}] (≥ 0)

and

max_{1≤i≤n} {p_{i}} [∑_{j=1}^{n} ∥x_{j}∥^{p} − n^{1−p} ∥∑_{j=1}^{n} x_{j}∥^{p}] ≥ ∑_{j=1}^{n} p_{j} ∥x_{j}∥^{p} − ∥∑_{j=1}^{n} p_{j}x_{j}∥^{p} (3.2)

≥ min_{1≤i≤n} {p_{i}} [∑_{j=1}^{n} ∥x_{j}∥^{p} − n^{1−p} ∥∑_{j=1}^{n} x_{j}∥^{p}] (≥ 0)

for all p ≥ 1.

If in (3.2) we choose p_{j} := \frac{1}{∥x_{j}∥}, where x_{j} ∈ X⧹ \{0\} , j ∈ \{1, ..., n\} , then we get

max_{1≤j≤n} \{∥x_{j}∥\} [∑_{j=1}^{n} ∥x_{j}∥^{p−1} − ∥∑_{j=1}^{n} \frac{x_{j}}{∥x_{j}∥}∥^{p}] (3.3)

≥ ∑_{j=1}^{n} ∥x_{j}∥^{p} − n^{1−p} ∥∑_{j=1}^{n} x_{j}∥^{p}

≥ min_{1≤j≤n} \{∥x_{j}∥\} [∑_{j=1}^{n} ∥x_{j}∥^{p−1} − ∥∑_{j=1}^{n} \frac{x_{j}}{∥x_{j}∥}∥^{p}] .

We remark that, for p = 1 one may get out of the previous results the following inequalities that are intimately related with the generalised triangle inequality in normed spaces:

max_{1≤i≤n} \{\frac{p_{i}}{q_{i}} \} [∑_{j=1}^{n} q_{j} ∥x_{j}∥ − ∥∑_{j=1}^{n} q_{j}x_{j}∥] ≥ ∑_{j=1}^{n} p_{j} ∥x_{j}∥ − ∥∑_{j=1}^{n} p_{j}x_{j}∥ (3.4)

≥ min_{1≤i≤n} \{\frac{p_{i}}{q_{i}} \} [∑_{j=1}^{n} q_{j} ∥x_{j}∥ − ∥∑_{j=1}^{n} q_{j}x_{j}∥] (≥ 0) ,

max_{1≤i≤n} {p_{i}} [∑_{j=1}^{n} ∥x_{j}∥ − ∥∑_{j=1}^{n} x_{j}∥] ≥ ∑_{j=1}^{n} p_{j} ∥x_{j}∥ − ∥∑_{j=1}^{n} p_{j}x_{j}∥ (3.5)

≥ min_{1≤i≤n} {p_{i}} [∑_{j=1}^{n} ∥x_{j}∥ − ∥∑_{j=1}^{n} x_{j}∥] (≥ 0)

and

max_{1≤j≤n} \{∥x_{j}∥\} [n − ∥∑_{j=1}^{n} \frac{x_{j}}{∥x_{j}∥}∥] ≥ ∑_{j=1}^{n} ∥x_{j}∥ − ∥∑_{j=1}^{n} x_{j}∥ (3.6)

≥ min_{1≤j≤n} \{∥x_{j}∥\} [n − ∥∑_{j=1}^{n} \frac{x_{j}}{∥x_{j}∥}∥] .

We note that the inequality (3.6) has been obtained in a different way by M. Kato. K.-S. Saito & T. Tamura in [4] where an analysis of the equality case for strictly convex spaces has been performed as well.

Remark 1. Let ℂ be the field of complex numbers. If z = Re z + i Im z, then by |·|_{p} : ℂ → [0, ∞), p ∈ [1, ∞] we define the p−modulus of z as

|z|_{p} := \{ \begin{matrix}max \{|Re z| , |Im z|\} & if p = ∞, \\ (|Re z|^{p} + |Im z|^{p})^{\frac{1}{p}} & if p ∈ [1, ∞), \end{matrix}

where |a| , a ∈ ℝ is the usual modulus of the real number a.

For p = 2, we recapture the usual modulus of a complex number, i.e.,

|z|_{2} = \sqrt{|Re z|^{2} + |Im z|^{2}} = |z| , z ∈ ℂ.

It is well known that (ℂ, |·|_{p}) , p ∈ [1, ∞] is a Banach space over the real number field ℝ.

It is obvious that all above inequalities hold for |·|_{p}, the nonnegative n-tuples p, q and the complex numbers z_{1}, ..., z_{n}. The details are omitted.

4. The Geometric Mean-Arithmetic Mean Inequality

Although the inequality between the arithmetic and geometric means

\frac{a + b}{2} ≥ \sqrt{ab}, a, b ≥ 0;

was probably known in antiquity, the general result for weighted means seems to have first appeared in print in the nineteenth century, in the notes of Cauchy’s course given at the École Royale in 1821, page 315.

If p ∈P_{n}, a ≥ 0 and

G_{n} (p, a) := ∏_{i=1}^{n} a_{i}^{p_{i}}, A_{n} (p, a) := ∑_{i=1}^{n} p_{i}a_{i}

then

(4.1)  A_{n} (p, a) ≥ G_{n} (p, a) .

If all p_{i} > 0, then the equality case holds in (4.1) if and only if a is constant, i.e., all the components are equal to a constant k ∈ ℝ .

For classical and recent results related to the GA-inequality, see Chapter II of the book [1].

Applying the inequality (2.1) for the choices f : (0, ∞) → ℝ, f (x) = − ln x one can state the following inequality

(4.2)  (\frac{A_{n} (q, a)}{G_{n} (q, a)})^{max^{1≤i≤n}^{\{ \frac{p_{i}}{q_{i}} \}}} ≥ \frac{A_{n} (p, a)}{G_{n} (p, a)} ≥ (\frac{A_{n} (q, a)}{G_{n} (q, a)})^{min^{1≤i≤n}^{\{ \frac{p_{i}}{q_{i}} \}}} .

If, for the uniform distribution u = (\frac{1}{n}, ..., \frac{1}{n}) , we consider

A_{n} (a) = A_{n} (u, a) , G_{n} (a) = G_{n} (u, a) ,

then we have

(4.3)  (\frac{A_{n} (a)}{G_{n} (a)})^{n max_{1≤i≤n}\{p_{i}\}} ≥ \frac{A_{n} (p, a)}{G_{n} (p, a)} ≥ (\frac{A_{n} (a)}{G_{n} (a)})^{n min_{1≤i≤n}\{p_{i}\}} .

Since there is the obvious relation between the arithmetic mean and the harmonic mean

H_{n} (p, a) := \frac{1}{∑{n \atop i=1} \frac{p_{i}}{a_{i}}} = A_{n} (p,\frac{1}{a})^{−1}

where \frac{1}{a} := (\frac{1}{a_{1}} , ..., \frac{1}{a_{n}} ) and a_{i} are all positive, one can derive a similar inequality between the harmonic and geometric means. The details are omitted.

5. Inequalities for Shannon’s Entropy

Let X be a random variable with the range R = {x_{1}, ..., x_{n}} and the probability distribution p_{1}, ..., p_{n} (p_{i} > 0, i = 1, ..., n) . Define the Shannon entropy by

H (X) := − ∑_{i=1}^{n} p_{i} ln p_{i}.

The following theorem is well known in the literature and concerns the maximum possible value of H (X) in terms of the size of R [5, p. 27].

Theorem 2. Let X be defined as above. Then

(5.1)  0 ≤ H (X) ≤ ln n.

Furthermore, H (X) = 0 iff p_{i} = 1 for some i and H (X) = ln n iff p_{i} = \frac{1}{n} for all i ∈ {1, ..., n} .

This fundamental result may be improved as follows:

Proposition 1. Let X be defined as above. If p_{i} > 0 for each i ∈ {1, ..., n} , then

(5.2)  0 ≤ n min_{1≤i≤n} \{p_{i}\} ln [\frac{A_{n} (\frac{1}{p})}{G_{n} (\frac{1}{p})}]

≤ ln n − H (X) ≤ n max_{1≤i≤n} \{p_{i}\} ln [\frac{A_{n} (\frac{1}{p})}{G_{n} (\frac{1}{p})}] .

Proof. It follows from Corollary 1 for the choices f (t) = − ln t, x_{i} = 1/p_{i} and appropriate elementary calculations. The details are omitted.

6. Inequalities for the Relative Entropy

The relative entropy is a measure of the distance between two distributions. In statistics, it arises as an expected logarithm of the likelihood ratio. The relative entropy D(p∥q) is a measure of the inefficiency of assuming that the distribution is q when the true distribution is p. For example, if we knew the true distribution of the random variable, then we could construct a code with average description length H(p). If, instead, we used the code for a distribution q, we would need H(p) + D(p∥q) bits on the average to describe the random variable [2, p. 18].

Definition 1. The relative entropy or Kullback-Leibler distance between two probability distributions p and q is defined by

D(p∥q) := ∑_{i=1}^{n} p_{i} ln (\frac{p_{i}}{q_{i}} ) = E_{p} ln (\frac{p(X)}{q(X)}) .

In the above definition, we use the convention (based on continuity arguments) that 0 ln (\frac{0}{q}) = 0 and p ln (\frac{p}{0}) = ∞.

It is well-known that relative entropy is always non-negative and is zero if and only if p = q. However, it is not a true distance between distributions since it is not symmetric and does not satisfy the triangle inequality.

The following theorem is of fundamental importance [2, p. 26].

Theorem 3. (Information Inequality) Let p, q, be two probability distributions. Then

(6.1)  D(p∥q) ≥ 0

with equality if and only if

(6.2)  p_{i} = q_{i} for all x ∈ {1, ..., n}.

This fundamental result may be improved as follows.

Proposition 2. Let p, q, be two probability distributions. Define the χ^{2} divergence of p, q by the quantity

χ^{2} (p, q) := ∑_{i=1}^{n} p_{i} (\frac{q_{i}}{p_{i}} − 1)^{2} = ∑_{i=1}^{n} \frac{q_{i}^{2}}{p_{i}} − 1.

If p_{i}, q_{i} > 0 for all x ∈ {1, ..., n}, then

max_{1≤i≤n} \{\frac{p_{i}}{q_{i}} \} [D(q∥p) − ln (χ^{2} (p, q) + 1)]
≥ D(p∥q)
≥ min_{1≤i≤n} \{\frac{p_{i}}{q_{i}} \} [D(q∥p) − ln (χ^{2} (p, q) + 1)] (≥ 0) .

Proof. The proof follows by Theorem 1 on choosing f (t) = − ln t, x_{i} = \frac{q_{i}}{p_{i}} and performing appropriate elementary calculations. The details are omitted.

References

[1] P.S. BULLEN, Handbook of Mean and Their Inequalities, Kluwer Academic Publishers, Dordrecht/Boston/London, 2003.

[2] T.M. COVER and J.A. THOMAS, Elements of Information Theory, John Wiley & Sons Inc., 1991.

[3] S.S. DRAGOMIR and C.E.M. PEARCE, Selected Topics on Hermite-Hadamard Inequalities and Applications, RGMIA Monographs, Victoria University, 2000. (ONLINE:

   http://rgmia.vu.edu.au/monographs/).

[4] M. KATO, K.-S. SAITO and T. TAMURA, Sharp traiangle inequality and its reverse in Banach spaces, submitted.

[5] R.J. McELIECE, The Theory of Information and Coding, Addison Wesley Publishing Company, Reading, 1977.

[6] D.S. MITRINOVIĆ, J.E. PEČARIĆ and A.M. FINK, Classical and New Inequalities in Analysis, Kluwer Academic Publishers, Dordrecht, 1993.

[7] J. E. PEČARIĆ, F. PROSCHAN and Y.L. TONG, Convex Functions, Partial Orderings and Statistical Applications, Academic Press, 1992.

School of Computer Science & Mathematics, Victoria University, Melbourne, Victoria, Australia

    E-mail address: sever@csm.vu.edu.au
    URL: http://rgmia.vu.edu.au/dragomir/