AN UPPER BOUND FOR THE CSISZÁR f−DIVERGENCE IN

TERMS OF THE VARIATIONAL DISTANCE AND APPLICATIONS

S.S. DRAGOMIR

Abstract. In this paper we point out an upper bound for the Csiszár f−divergence of two discrete random variables in terms of the generalized r−variational distance. Some particular cases for Kullback-Leibler distance, Hellinger and triangular discrimination, χ^{2}−distance, etc.. are considered.

1. Introduction

Given a convex function f : [0, ∞) → ℝ, the f−divergence functional

I_{f} (p, q) = ∑_{i=1}^{n} q_{i}f (\frac{p_{i}}{q_{i}} ) , (1.1)

was introduced by Csiszár [1]-[2] as a generalized measure of information, a “distance function” on the set of probability distribution ℙ^{n}. The restriction here to discrete distributions is only for convenience, similar results hold for general distributions. As in Csiszár [1]-[2], we interpret undefined expressions by

f (0) = lim_{t→0+} f (t) , 0 f (\frac{0}{0}) = 0,

0 f (\frac{a}{0}) = lim_{ε→0+} εf (\frac{a}{ε}) = a lim_{t→∞} \frac{f(t)}{t}, a > 0.

The following results (Theorems 1 and 2, and Corollary 1) were essentially given by Csiszár and Körner [3].

Theorem 1. (Joint Convexity) If f : [0, ∞) → ℝ is convex, then I_{f} (p, q) is jointly convex in p and q.

Theorem 2. (Jensen’s inequality) Let f : [0, ∞) → ℝ be convex. Then for any

p, q ∈ ℝ_{+}^{n} with P_{n} := ∑_{i=1}^{n} p_{i} > 0, Q_{n} := ∑_{i=1}^{n} q_{i} > 0, we have the inequality

I_{f} (p, q) ≥ Q_{n}f (\frac{P_{n}}{Q_{n}} ) . (1.2)

If f is strictly convex, equality holds in (1.2) iff

\frac{p_{1}}{q_{1}} = \frac{p_{2}}{q_{2}} = ... = \frac{p_{n}}{q_{n}} . (1.3)

It is natural to consider the following corollary.

Date: December 13, 1999.

1991 Mathematics Subject Classification. Primary 94Xxx, 26D15.

Key words and phrases. Csiszár f−divergence, Variational Distance, Divergence Measures in Information Theory, Jensen’s inequality.

2 S.S. DRAGOMIR

Corollary 1. (Nonnegativity) Let f : [0, ∞) → ℝ be convex and normalised, i.e., f (1) = 0. (1.4) Then for any p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}, we have the inequality I_{f} (p, q) ≥ 0. (1.5) If f is strictly convex, equality holds in (1.5) iff p_{i} = q_{i} for all i ∈ {1, ..., n}. (1.6)

In particular, if p, q are probability vectors, then Corollary 1 shows that, for strictly convex and normalized f : [0, ∞) → ℝ that I_{f} (p, q) ≥ 0 and I_{f} (p, q) = 0 iff p = q. (1.7)

We now give some more examples of divergence measures in Information Theory which are particular cases of Csiszár f−divergences.

1. Kullback-Leibler distance ([12]). The Kullback-Leibler distance D (·, ·) is defined by

D (p, q) := ∑_{i=1}^{n} p_{i} log (\frac{p_{i}}{q_{i}} ) . (1.8)

If we choose f (t) = t ln t, t > 0, then obviously I_{f} (p, q) = D (p, q) . (1.9)

2. Variational distance (l_{1}−distance). The variational distance V (·, ·) is defined by

V (p, q) := ∑_{i=1}^{n} |p_{i} − q_{i}| . (1.10)

If we choose f (t) = |t − 1|, t ∈ ℝ_{+}, then we have

I_{f} (p, q) = V (p, q) . (1.11)

3. Hellinger discrimination ([13]). The Hellinger discrimination h^{2} (·, ·) is defined by

h^{2} (p, q) := \frac{1}{2} ∑_{i=1}^{n} (\sqrt{p_{i}} − \sqrt{q_{i}})^{2} . (1.12)

It is obvious that if f (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}, then

I_{f} (p, q) = h^{2} (p, q) . (1.13)

4. Triangular discrimination ([24]). We define triangular discrimination between p and q by

Δ (p, q) = ∑_{i=1}^{n} \frac{|p_{i} − q_{i}|^{2}}{p_{i} + q_{i}} . (1.14)

It is obvious that if f (t) = \frac{(t−1)^{2}}{t+1} , t ∈ (0, ∞), then

I_{f} (p, q) = Δ (p, q) . (1.15)

5. χ^{2}−distance. We define the χ^{2}−distance (chi-square distance) by

D_{χ}^{2} (p, q) := ∑_{i=1}^{n} \frac{(p_{i} − q_{i})^{2}}{q_{i}} . (1.16)

It is clear that if f (t) = (t − 1)^{2}, t ∈ [0, ∞), then

I_{f} (p, q) = D_{χ}^{2} (p, q) (1.17)

6. Rényi α−order entropy ([14]). The α−order entropy (α > 1) is defined by

R_{α} (p, q) := ∑_{i=1}^{n} p_{i}^{α}q_{i}^{1−α}. (1.18)

It is obvious that if f (t) = t^{α} (t ∈ (0, ∞)) , then

I_{f} (p, q) = R_{α} (p, q) . (1.19)

For other examples of divergence measures, see the paper [22] by J.N. Kapur, where further references are given.

2. An Upper Bound for Csiszár f−Divergence

In the recent paper [28], the author proved the following inequality for Csiszár f−divergence:

Theorem 3. Let Φ : ℝ_{+} → ℝ be differentiable convex. Then for all p, q ∈ ℝ_{+}^{n}, we have the inequality:

Φ′ (1) (P_{n} − Q_{n}) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) , (2.1)

where P_{n} := ∑_{i=1}^{n} p_{i} > 0, Q_{n} := ∑_{i=1}^{n} q_{i} > 0 and Φ′ : (0, ∞) → ℝ is the derivative of

Φ. If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n) , then the equality holds in (2.9) iff p = q,

If we assume that P_{n} = Q_{n} and Φ is normalised, then we obtain the simpler

0 ≤ I_{Φ} (p, q) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − I_{Φ′} (p, q) . (2.2)

Applications for particular divergences which are instances of Csiszár f−divergence were also given.

Asimilar result of the above theorem has been presented in antoher paper by the author [29].

Theorem 4. Let Φ, p, q be as in Theorem 3. Then we have the inequality

0 ≤ I_{Φ} (p, q) − Q_{n}Φ (\frac{P_{n}}{Q_{n}} ) ≤ I_{Φ′} (\frac{p^{2}}{q} , p) − \frac{P_{n}}{Q_{n}} I_{Φ′} (p, q) . (2.3)

If Φ is strictly convex and p_{i}, q_{i} > 0 (i = 1, ..., n), then the equality holds in (2.3) iff \frac{p_{1}}{q_{1}} = ... = \frac{p_{n}}{q_{n}}.

Obviously, if P_{n} = Q_{n} and Φ is normalised, then (2.3) becomes (2.2).

The following result concerning an upper and a lower bound for the Csiszár f−divergence in terms of the Kullback-Leibler distance D (p, q) holds.

As in [30], we will say that the mapping f : C ⊂ ℝ → ℝ, where C is an interval (in [30], the definition was considered in general normed spaces), is

(i) α−lower convex on C if f − \frac{α}{2} · |·|^{2} is convex on C; (ii) β−upper convex on C if \frac{β}{2} · |·|^{2} − f is convex on C; (iii) (m, M) −convex on C (with m ≤ M) if it is both m−lower convex and

M−iupper convex. In [30], amongst others, the author has proved the following result for Csiszár

f−divergence.

Theorem 5. Let Φ : ℝ_{+} → ℝ and p, q ∈ ℝ_{+}^{n} with P_{n} = Q_{n}.

(i) If Φ is α-lower convex on ℝ_{+}, then we have the inequality

\frac{α}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) . (2.4)

(ii) If Φ is β−upper convex on ℝ_{+}, then we have the inequality

I_{Φ} (p, q) − Q_{n}Φ (1) ≤ \frac{β}{2} · D_{χ}^{2} (p, q) . (2.5)

(iii) If Φ is (m, M) −convex on ℝ_{+}, then we have the following sandwich inequality

\frac{m}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) − Q_{n}Φ (1) ≤ \frac{M}{2} · D_{χ}^{2} (p, q) , (2.6)

where D_{χ}^{2} (·, ·) is the χ^{2}−divergence.

Of course, if Φ is normalised, i.e., Φ (1) = 0 and p, q are probability distributions, then we get the simpler inequalities:

\frac{α}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) , I_{Φ} (p, q) ≤ \frac{β}{2} · D_{χ}^{2} (p, q) (2.7)

and

\frac{m}{2} · D_{χ}^{2} (p, q) ≤ I_{Φ} (p, q) ≤ \frac{M}{2} · D_{χ}^{2} (p, q) . (2.8)

In [30], some applications for particular instances of Csiszár f−divergences were also given.

Define the generalised r−variational distance by

V_{r} (p, q) := ∑_{i=1}^{n} q_{i}^{1−r} |p_{i} − q_{i}|^{r} , (2.9)

where p, q are probability distributions and r ∈ (0, 1]. Note that for r = 1, we recapture the usual variational distance (or l_{1}-distance).

The following theorem holds.

Theorem 6. Let f : [0, ∞) → ℝ be a convex mapping and such that f (1) = 0, i.e., f is normalised, and f is of r − H−Hölder type on [r, R] , i.e.,

|f (x) − f (y)| ≤ H |x − y|^{r} for all x, y ∈ [r, R] . (2.10)

Then we have the inequality

0 ≤ I_{f} (p, q) ≤ HV_{r} (p, q) . (2.11)

Proof. We choose in (2.10) x = \frac{p_{i}}{q_{i}}, y = 1 (i = 1, ..., n) to get

|f (\frac{p_{i}}{q_{i}} ) − f (1)| ≤ H |\frac{p_{i}}{q_{i}} − 1|^{r} , (2.12)

for all i ∈ {1, ..., n} .

If we multiply (2.12) by q_{i}, sum the obtained inequalities and use the generalised triangle inequality, we obtain

0 ≤ I_{f} (p, q) = ∑_{i=1}^{n} q_{i}f (\frac{p_{i}}{q_{i}} ) = |∑_{i=1}^{n} q_{i}f (\frac{p_{i}}{q_{i}} ) − ∑_{i=1}^{n} q_{i}f (1)|

≤ H ∑_{i=1}^{n} q_{i} |\frac{p_{i}}{q_{i}} − 1|^{r} = HV_{r} (p, q)

and the inequality (2.11) is proved.

Remark 1. If we assume that f is convex, normalised and L−Lipschitzian on [r, R], i.e., r = 1 and H = L, then we have the inequality

0 ≤ I_{f} (p, q) ≤ LV_{r} (p, q) , (2.13)

where V (p, q) is the usual variational distance.

The practical result is embodied in the following corollary.

Corollary 2. If the mapping f : [r, R] → ℝ is convex, normalised, absolutely

continuous on [r, R] and f′ ∈ L_{∞} [r, R], i.e., ∥f′∥_{∞} := ess sup_{t∈[r,R]} |f′ (t)| < ∞,

then we have the inequality

0 ≤ I_{f} (p, q) ≤ ∥f′∥_{∞} V_{r} (p, q) . (2.14)

The following theorem holds.

Theorem 7. Assume that f is as in Theorem 6. If p^{(j)}, q^{(j)} (j = 1, 2) are probability distributions satisfying the condition

r ≤ \frac{p_{i}^{(j)}}{q_{i}^{(j)}} ≤ R for all i ∈ \{1, ..., n\} and j ∈ \{1, 2\} ,

then obviously

r ≤ \frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ≤ R for all i ∈ \{1, ..., n\} and λ ∈ [0, 1]

and we have the inequality

(2.15)  0 ≤ I_{f} (λp^{(1)} + (1 − λ) p^{(2)}, λq^{(1)} + (1 − λ) q^{(2)})
−λI_{f} (p^{(1)}, q^{(1)}) − (1 − λ) I_{f} (p^{(2)}, q^{(2)})

≤ Hλ^{r} (1 − λ)^{r} ∑_{i=1}^{n} \begin{matrix}|det [ \begin{matrix}p_{i}^{(1)} & p_{i}^{(2)} \\ q_{i}^{(1)} & q_{i}^{(2)} \end{matrix} ]| & r \\ [λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}] & r \end{matrix}

× [λ^{1−r} [q_{i}^{(1)}]^{1−r} + (1 − λ)^{1−r} [q_{i}^{(2)}]^{1−r}]

6 S.S. DRAGOMIR

for all λ ∈ [0, 1] . Proof. If we choose in (2.10)

x = \frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} and y = \frac{p_{i}^{(1)}}{q_{i}^{(1)}} (i = 1, ..., n)

we get

|f (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ) − f (\frac{p_{i}^{(1)}}{q_{i}^{(1)}} )| (2.16)

≤ H |\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(1)}}{q_{i}^{(1)}} |^{r}
= H \begin{matrix}|λq_{i}^{(1)}p_{i}^{(1)} + (1 − λ) q_{i}^{(1)}p_{i}^{(2)} − λp_{i}^{(1)}q_{i}^{(1)} − (1 − λ) p_{i}^{(1)}q_{i}^{(2)}| & r \\ [q_{i}^{(1)}] & r & [λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}] & r \end{matrix}
= \frac{H (1 − λ)^{r} |q_{i}^{(1)}p_{i}^{(2)} − p_{i}^{(1)}q_{i}^{(2)}|^{r}}{[q_{i}^{(1)}]^{r} [λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}]^{r}} .

If we multiply (2.16) by λq_{i}^{(1)}, we obtain

|λq_{i}^{(1)}f (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ) − λq_{i}^{(1)}f (\frac{p_{i}^{(1)}}{q_{i}^{(1)}} )| (2.17)

≤ \frac{Hλ (1 − λ)^{r} [q_{i}^{(1)}]^{1−r} |q_{i}^{(1)}p_{i}^{(2)} − p_{i}^{(1)}q_{i}^{(2)}|^{r}}{[λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}]^{r}}

for all i ∈ {1, ..., n} and λ ∈ [0, 1].

If in (2.10) we choose x as above but

y = \frac{p_{i}^{(2)}}{q_{i}^{(2)}} (i = 1, ..., n) ,

we get

|f (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ) − f (\frac{p_{i}^{(2)}}{q_{i}^{(2)}} )| (2.18)

≤ H |\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} − \frac{p_{i}^{(2)}}{q_{i}^{(2)}} |^{r}
= H \begin{matrix}|λp_{i}^{(1)}q_{i}^{(2)} + (1 − λ) p_{i}^{(2)}q_{i}^{(2)} − λp_{i}^{(2)}q_{i}^{(1)} − (1 − λ) q_{i}^{(2)}p_{i}^{(2)}| & r \\ [λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}] & r & [q_{i}^{(2)}] & r \end{matrix}
= \frac{H (1 − λ)^{r} |p_{i}^{(1)}q_{i}^{(2)} − p_{i}^{(2)}q_{i}^{(1)}|^{r}}{[q_{i}^{(2)}]^{r} [λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}]^{r}} .

If we multiply (2.18) by (1 − λ) q_{i}^{(2)}, we obtain

|(1 − λ) q_{i}^{(2)}f (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ) − (1 − λ) q_{i}^{(2)}f (\frac{p_{i}^{(2)}}{q_{i}^{(2)}} )| (2.19)

≤ \frac{Hλ^{r} (1 − λ) [q_{i}^{(2)}]^{1−r} |q_{i}^{(1)}p_{i}^{(2)} − p_{i}^{(1)}q_{i}^{(2)}|^{r}}{[λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}]^{r}}

for all i ∈ {1, ..., n} and λ ∈ [0, 1].

If we add (2.13) and (2.19) and use the triangle inequality, we get

|[λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}] f (\frac{λp_{i}^{(1)} + (1 − λ) p_{i}^{(2)}}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}} ) (2.20)

−λq_{i}^{(1)}f (\frac{p_{i}^{(1)}}{q_{i}^{(1)}} ) − (1 − λ) q_{i}^{(2)}f (\frac{p_{i}^{(2)}}{q_{i}^{(2)}} )|

≤ \frac{Hλ^{r} (1 − λ)^{r} |det [ \begin{matrix}p_{i}^{(1)} & p_{i}^{(2)} \\ q_{i}^{(1)} & q_{i}^{(2)} \end{matrix} ]|^{r}}{[λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}]^{r}} [λ^{1−r} [q_{i}^{(1)}]^{1−r} + (1 − λ)^{1−r} [q_{i}^{(2)}]^{1−r}]

for all i ∈ {1, ..., n} and λ ∈ [0, 1].

If we sum (2.20) over i from 1 to n and use the generalised triangle inequality, we obtain the desired inequality (2.15).

Remark 2. If we assume that f is L−Lipschitzian, then the inequality (2.15) becomes

(2.21)  0 ≤ I_{f} (λp^{(1)} + (1 − λ) p^{(2)}, λq^{(1)} + (1 − λ) q^{(2)})
−λI_{f} (p^{(1)}, q^{(1)}) − (1 − λ) I_{f} (p^{(2)}, q^{(2)})

≤ Lλ (1 − λ) ∑_{i=1}^{n} \frac{|det [ \begin{matrix}p_{i}^{(1)} & p_{i}^{(2)} \\ q_{i}^{(1)} & q_{i}^{(2)} \end{matrix} ]|}{λq_{i}^{(1)} + (1 − λ) q_{i}^{(2)}},

for all λ ∈ [0, 1] .

3. Some Particular Cases

Using the inequality (2.14), i.e.,

0 ≤ I_{f} (p, q) ≤ ∥f′∥_{∞} V (p, q) , (3.1)

provided that f is absolutely continuous on [r, R] and f′ ∈ L_{∞} [r, R] , ∥f′∥_{∞} := ess sup_{t∈[r,R]} |f′ (t)|, we shall be able to point out a number of interesting inequalities

between different divergence measures in Information Theory.

Proposition 1. Let p, q be two probability distributions with the property that

0 < r ≤ \frac{p_{i}}{q_{i}} =: r_{i} < R < ∞ for all i ∈ \{1, ..., n\} . (3.2)

Then we have the inequality

(3.3) 0 ≤ D (p, q) ≤ \{ \begin{matrix}[ln \sqrt{\frac{R}{r}} + |1 + ln \sqrt{rR}|] V (p, q) & if & 0 < r ≤ e^{−1} \\ (1 + ln R) V (p, q) & if & e^{−1} < r < 1 \end{matrix} .

Proof. Consider the mapping f : (0, ∞) → ℝ, f (t) = t ln t. Then

f′ (t) = ln (et) .

Therefore

∥f′∥_{∞} = sup_{t∈[r,R]} |f′ (t)| = max {|ln (er)| , ln (eR)} .

Now, observe that

1. If 0 < r ≤ e^{−1}, then |ln (er)| = ln (er) = −1 − ln r and then

max \{|ln (er)| , ln (eR)\} = \frac{−1 − ln r + ln R + 1 + |ln r + 1 + 1 + ln R|}{2}
= ln (\frac{R}{2})^{\frac{1}{2}} + |1 + ln \sqrt{rR}| .

2. If e^{−1} < r < 1, then |ln (er)| = 1 + ln r and then

max {|ln (er)| , ln (eR)} = max {ln (er) , ln (eR)} = 1 + ln R and the proposition is proved.

We also have the following proposition. Proposition 2. Let p, q be as in Proposition 1. Then we have the inequality:

0 ≤ D (q, p) ≤ \frac{1}{r}V (p, q) . (3.4)

Proof. Consider the mapping f (t) = − ln t, t ∈ (0, ∞). Then

f′ (t) = −\frac{1}{t} and ∥f′∥_{∞} = \frac{1}{r}.

Since

I_{f} (p, q) = − ∑_{i=1}^{n} q_{i} ln (\frac{p_{i}}{q_{i}} ) = ∑_{i=1}^{n} q_{i} ln (\frac{q_{i}}{p_{i}} ) = D (q, p) ,

the inequality (3.1) gives the desired result (3.4).

We point out now a bound for the χ^{2}−divergence. Proposition 3. Let p, q be as above. Then 0 ≤ D_{χ}^{2} (p, q) ≤ 2 (R − 1) V (p, q) . (3.5) Proof. Consider the mapping f : (0, ∞) → ℝ, f (t) = (t − 1)^{2}. As f′ (t) = 2 (t − 1), it follows that ∥f′∥_{∞} = 2 (R − 1). Using (3.1), we obtain (3.5).

The following result for Hellinger discrimination also holds.

Proposition 4. Assume that the probability distributions p, q satisfy the condition (3.1). Then we have the inequality

0 ≤ h^{2} (p, q) ≤ \frac{1}{4} [\frac{\sqrt{R} − \sqrt{r}}{\sqrt{rR}} + |\frac{\sqrt{R} + \sqrt{r}}{\sqrt{rR}} − 2|] V (p, q) . (3.6)

Proof. Consider the mapping f : (0, ∞) → ℝ, f (t) = \frac{1}{2} (\sqrt{t} − 1)^{2}. Then

f′ (t) = \frac{1}{2} − \frac{1}{2\sqrt{t}} and f′′ (t) = \frac{1}{4\sqrt{t^{3}}} , t ∈ (0, ∞) .

Then

∥f′∥_{∞} = sup_{t∈[r,R]} |f′ (t)| = max \{|\frac{\sqrt{r} − 1}{2\sqrt{r}}| , |\frac{\sqrt{R} − 1}{2\sqrt{R}}|\}

= max \{\frac{1 − \sqrt{r}}{2\sqrt{r}} , \frac{\sqrt{R} − 1}{2\sqrt{R}}\}

= \frac{1}{4} [\frac{\sqrt{R} − \sqrt{r}}{\sqrt{rR}} + |2 − \frac{\sqrt{R} + \sqrt{r}}{\sqrt{rR}} |] .

Using (3.1), we deduce (3.6).

Now, consider the Bhattacharyya distance which is defined by [27]

B (p, q) = ∑_{i=1}^{n} \sqrt{p_{i}q_{i}}. (3.7)

The following proposition holds.

Proposition 5. Assume that p, q are probability distributions. Then we have the inequality

0 ≤ 1 − B (p, q) ≤ V_{\frac{1}{2}} (p, q) , (3.8)

where V_{\frac{1}{2}} (p, q) = ∑_{i=1}^{n} \sqrt{q_{i} (p_{i} − q_{i})} is the \frac{1}{2}−variational distance.

Proof. Consider the mapping f : [0, ∞) → [0, ∞) given by

f (t) = −\sqrt{t} + 1.

Obviously,

|f (x) − f (y)| = |\sqrt{x} − \sqrt{y}| ≤ \sqrt{|x − y|}, for all x, y ∈ [0, ∞),

which shows that f is of \frac{1}{2}−Hölder type with the constant H = 1.

Applying Theorem 6, we deduce (3.8).

Another inequality for Bhattacharyya distance in terms of the variational distance V is embodied in the following proposition.

Proposition 6. Assume that the probability distributions p, q satisfy the the condition:

0 < r ≤ \frac{p_{i}}{q_{i}} for all i ∈ \{1, ..., n\} . (3.9)

Then we have the inequality

0 ≤ 1 − B (p, q) ≤ \frac{1}{2\sqrt{r}} V (p, q) . (3.10)

Proof. For the mapping f (t) = −\sqrt{t} + 1, we have f′ (t) = −\frac{1}{2\sqrt{t}} and ∥f′∥_{∞} = sup_{t∈[0,∞)} = \frac{1}{2\sqrt{r}}. Applying (3.1), we deduce (3.11).

Now, we consider another useful divergence measure in Information Theory which is known as the Harmonic distance:

M (p, q) := ∑_{i=1}^{n} \frac{2p_{i}q_{i}}{p_{i} + q_{i}} . (3.11)

Proposition 7. Assume that the probability distributions p, q satisfy the condition (3.9). Then we have the inequality

0 ≤ 1 − M (p, q) ≤ \frac{2}{(r + 1)^{2}} V (p, q) . (3.12)

Proof. Consider the mapping f : [0, ∞) → ℝ, f (t) = 1 − \frac{2t}{t+1}. Obviously f′ (t) =

−\frac{2}{(t+1)^{2}} , f′′ (t) = \frac{4}{(t+1)^{3}} and ∥f′∥_{∞} = sup_{t∈[r,∞)} |f′ (t)| = \frac{2}{(r+1)^{2}} . As

I_{f} (p, q) = 1 − M (p, q) ,

then by (3.1) we deduce the desired inequality (3.12).

Another useful divergence measure is the Jeffreys J−divergence defined by [26]

J (p, q) = ∑_{i=1}^{n} (p_{i} − q_{i}) ln (\frac{p_{i}}{q_{i}} ) .

We can state the following proposition as well.

Proposition 8. Assume that the probability distributions p, q satisfy the condition:

\frac{p_{i}}{q_{i}} ≤ R < ∞, (i = 1, ..., n) . (3.13)

Then we have the inequality

0 ≤ J (p, q) ≤ (ln R − \frac{1}{R} + 1) V (p, q) . (3.14)

Proof. Consider the mapping f (t) = (t − 1) ln t, t > 0. Then

f′ (t) = ln t − \frac{1}{t} + 1; t ∈ (0, ∞)
f′′ (t) = \frac{t + 1}{t^{2}}, t ∈ (0, ∞)

which shows that

∥f′∥_{∞} = sup_{t∈(0,R]} |f′ (t)| = f′ (R) = ln R − \frac{1}{R} + 1.

As

I_{f} (p, q) = J (p, q) ,

then by (3.1) we deduce (3.14).

Finally, the following result for the triangular discrimination holds (for the definition of triangular discrimination, see [24]).

Proposition 9. If p, q are such that the condition (3.13) holds, then

0 ≤ Δ (p, q) ≤ \begin{matrix}(R − 1) & (R + 3) \\ (R + 1) & 2 \end{matrix}V (p, q) ≤ V (p, q) . (3.15)

The proof is obvious by (3.2) applied for the mapping f (t) = \frac{(t−1)^{2}}{t+1} , and we omit the details.

References

[1] I. CSISZÁR, Information measures: A critical survey, Trans. 7th Prague Conf. on Info. Th., Statist. Decis. Funct., Random Processes and 8th European Meeting of Statist., Volume B, Academia Prague, 1978, 73-86.

[2] I. CSISZÁR, Information-type measures of difference of probability functions and indirect observations, Studia Sci. Math. Hungar, 2 (1967), 299-318.

[3] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[4] J.H. JUSTICE (editor), Maximum Entropy and Bayssian Methods in Applied Statistics, Cambridge University Press, Cambridge, 1986.

[5] J.N. KAPUR, On the roles of maximum entropy and minimum discrimination information principles in Statistics, Technical Address of the 38th Annual Conference of the Indian Society of Agricultural Statistics, 1984, 1-44.

[6] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy functions, IEEE Transactions on Information Theory, 28 (1982), 489-495.

[7] R.G. GALLAGER, Information Theory and Reliable Communications, J. Wiley, New York, 1968.

[8] C.E. SHANNON, A mathematical theory of communication, Bull. Sept. Tech. J., 27 (1948), 370-423 and 623-656.

[9] B.R. FRIEDEN, Image enhancement and restoration, Picture Processing and Digital Filtering (T.S. Huang, Editor), Springer-Verlag, Berlin, 1975.

[10] R.M. LEAHY and C.E. GOUTIS, An optimal technique for constraint-based image restoration and mensuration, IEEE Trans. on Acoustics, Speech and Signal Processing, 34 (1986), 1692-1642.

[11] S. KULLBACK, Information Theory and Statistics, J. Wiley, New York, 1959.

[12] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[13] R. BERAN, Minimum Hellinger distance estimates for parametric models, Ann. Statist., 5 (1977), 445-463.

[14] A. RENYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Statist. Prob., Vol. 1, University of California Press, Berkeley, 1961.

[15] S.S. DRAGOMIR and N.M. IONESCU, Some converse of Jensen’s inequality and applications, Anal. Num. Theor. Approx., 23 (1994), 71-78.

[16] S.S. DRAGOMIR and C.J. GOH, A counterpart of Jensen’s discrete inequality for differentiable convex mappings and applications in information theory, Math. Comput. Modelling, 24 (2) (1996), 1-11.

[17] S.S. DRAGOMIR and C.J. GOH, Some counterpart inequalities in for a functional associated with Jensen’s inequality, J. of Ineq. & Appl., 1 (1997), 311-325.

[18] S.S. DRAGOMIR and C.J. GOH, Some bounds on entropy measures in information theory, Appl. Math. Lett., 10 (1997), 23-28.

[19] S.S DRAGOMIR and C.J. GOH, A counterpart of Jensen’s continuous inequality and applications in information theory, RGMIA Preprint,

http://matilda.vu.edu.au/~rgmia/InfTheory/Continuse.dvi

[20] M. MATIĆ, Jensen’s inequality and applications in Information Theory (in Croatian), Ph.D. Thesis, Univ. of Zagreb, 1999.

[21] S.S. DRAGOMIR, J. ŠUNDE and M. SCHOLZ, Some upper bounds for relative entropy and applications, Comp. and Math. with Appl. in press).

[22] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), No. 1, 1-16.

[23] D.S. MITRINOVIĆ, J.E. PEČARIĆ and A.M. FINK, Classical and New Inequalities in Analysis, Kluwer Academic Publishers, 1993.

[24] F. TOPSOE, Some inequalities for information divergence and related measures of discrimination, Res. Rep. Coll. RGMIA, 2 (1) (1999), 85-98.

[25] D. DACUNHA-CASTELLE, Ecole d’ete de Probabilités de Saint-Fleour, III-1997, Berlin, Heidelberg: Springer 1978.

[26] H. JEFFREYS, An invariant form for the prior probability in estimation problems, Proc. Roy. Soc. London, Ser. A, 1986 (1946), 453-461.

[27] A. BHATTACHARYYA, On a measure of divergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc., 35 (1943), 99-109.

[28] S.S. DRAGOMIR, Some inequalities for the Csiszár Φ-divergence, submitted.

[29] S.S. DRAGOMIR, A converse inequality for the Csiszár Φ-divergence, submitted.

[30] S.S. DRAGOMIR, Some inequalities for (m, M)-convex mappings and applications for the Csiszár Φ−divergence in information theory, submitted.

School of Communications and Informatics, Victoria University of Technology, PO Box 14428, Melbourne City MC, Victoria 8001, Australia.

   E-mail address: sever.dragomir@vu.edu.au
   URL: http://melba.vu.edu.au/~rgmia/SSDragomirWeb.html