A GENERALIZED f−DIVERGENCE FOR PROBABILITY VECTORS AND APPLICATIONS

S.S. DRAGOMIR

Abstract. A generalized f−divergence for probability vectors and some fundamental inequalities are pointed out.

1. Introduction

One of the important issues in many applications of Probability Theory is finding an appropriate measure of distance (or difference or discrimination ) between two probability distributions. A number of divergence measures for this purpose have been proposed and extensively studied by Jeffreys [31], Kullback and Leibler [42], Rényi [20], Havrda and Charvat [25], Kapur [44], Sharma and Mittal [5], Burbea and Rao [41], Rao [34], Lin [10], Csiszár [11], Ali and Silvey [52], Vajda [37], Shioya and Da-te [21] and others (see for example [44] and the references therein).

These measures have been applied in a variety of fields such as: anthropology [34], genetics [43], finance, economics, and political science [47], [48], [39], biology [19], the analysis of contingency tables [9], approximation of probability distributions [26], [23], signal processing [24], [3] and pattern recognition [8], [53]. A number of these measures of distance are specific cases of Csiszár f-divergence and so further exploration of this concept will have a flow on effect to other measures of distance and to areas in which they are applied.

Assume that a set χ and the σ−finite measure μ are given. Consider the set of all probability densities on μ to be Ω := {p|p : χ → ℝ, p (x) ≥ 0, ∫ p (x) dμ (x) = 1}. The Kullback-Leibler divergence [42] is well known among the information divergences. It is defined as:-

(1.1)  D_{KL} (p, q) := ∫_{χ} p (x) log [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω,

where log is to base 2.

In Information Theory and Statistics, various divergences are applied in addition to the Kullback-Leibler divergence. These are the: variation distance D_{v}, Hellinger distance D_{H} [31], χ^{2}−divergence D_{χ}^{2}, α−divergence D_{α}, Bhattacharyya distance D_{B} [42], Harmonic distance D_{Ha}, Jeffreys distance D_{J} [31], triangular discrimination D_{Δ} [33], etc... They are defined as follows:

(1.2)  D_{v} (p, q) := ∫_{χ} |p (x) − q (x)| dμ (x) , p, q ∈ Ω;

Date: March 20, 2000.

1991 Mathematics Subject Classification. Primary 94Xxx; Secondary 26D15.

Key words and phrases. A Generalized f−Divergence, Jensen’s Inequality.

(1.3)  D_{H} (p, q) := ∫_{χ} [\sqrt{p (x)} − \sqrt{q (x)}]^{2} dμ (x) , p, q ∈ Ω;

(1.4)  D_{χ}^{2} (p, q) := ∫_{χ} p (x) [(\frac{q (x)}{p (x)})^{2} − 1] dμ (x) , p, q ∈ Ω;

(1.5)  D_{α} (p, q) := \frac{4}{1 − α^{2}} [1 − ∫_{χ} [p (x)]^{\frac{1−α}{2}} [q (x)]^{\frac{1+α}{2}} dμ (x)] , p, q ∈ Ω;

(1.6)  D_{B} (p, q) := ∫_{χ} \sqrt{p (x) q (x)}dμ (x) , p, q ∈ Ω;

(1.7)  D_{Ha} (p, q) := ∫_{χ} \frac{2p (x) q (x)}{p (x) + q (x)}dμ (x) , p, q ∈ Ω;

(1.8)  D_{J} (p, q) := ∫_{χ} [p (x) − q (x)] ln [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω;

(1.9)  D_{Δ} (p, q) := ∫_{χ} \frac{[p (x) − q (x)]^{2}}{p (x) + q (x)} dμ (x) , p, q ∈ Ω.

For other divergence measures, see the paper [25] by Kapur or the book on line [5] by Taneja. For a comprehensive collection of preprints available on line, see the RGMIA web site

http://rgmia.vu.edu.au/papersinfth.html

Csiszár f−divergence is defined as follows [11]

(1.10)  D_{f} (p, q) := ∫_{χ} p (x) f [\frac{q (x)}{p (x)}] dμ (x) , p, q ∈ Ω,

where f is convex on (0, ∞). It is assumed that f (u) is zero and strictly convex at u = 1. By appropriately defining this convex function, various divergences are derived. All the above distances (1.1) − (1.9), are particular instances of Csiszár f−divergence. There are also many others which are not in this class (see for example [44] or [5]). For the basic properties of Csiszár f−divergence see [41]-[11].

In [1], Lin and Wong (see also [10]) introduced the following divergence

(1.11)  D_{LW} (p, q) := ∫_{χ} p (x) log [ \frac{p (x)}{\frac{1}{2}p (x) + \frac{1}{2}q (x)}] dμ (x) , p, q ∈ Ω.

This can be represented as follows, using the Kullback-Leibler divergence:

D_{LW} (p, q) = D_{KL} (p, \frac{1}{2}p + \frac{1}{2}q) .

Lin and Wong have established the following inequalities

(1.12)  D_{LW} (p, q) ≤ \frac{1}{2}D_{KL} (p, q) ;

(1.13)  D_{LW} (p, q) + D_{LW} (q, p) ≤ D_{v} (p, q) ≤ 2;

A GENERALIZED f−DIVERGENCE 3

(1.14)  D_{LW} (p, q) ≤ 1.

In [37], Shioya and Da-te improved (1.12) − (1.14) by showing that

D_{LW} (p, q) ≤ \frac{1}{2}D_{v} (p, q) ≤ 1.

For classical and new results in comparing different kinds of divergence measures, see the papers [31]-[38] where further references are given.

2. The Generalized f−Divergence and Some Inequalities

Let (p_{1}, ..., p_{n}) and (q_{1}, ..., q_{n}) be such that p_{i}, q_{i} ∈ Ω (i = 1, ..., n). Define the f−divergence between two vectors by

D̄_{f} (p_{1}, ..., p_{n}, q_{1}, ..., q_{n}) (2.1)

: = ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n}) f [ \frac{\frac{q_{1}(x_{1})}{p_{1}(x_{1})} + ... + \frac{q_{n}(x_{n})}{p_{n}(x_{n})}}{n} ] dμ (x_{1}) ...dμ (x_{n})

provided that the multiple integral exists and is finite and f : [0, ∞) → ℝ is a given mapping.

Usually, the mapping is assumed to be convex and normalised.

The following simple result holds. Theorem 1. Assume that the mapping f : [0, ∞) → ℝ is convex. Then we have the inequalities

(2.2)  f (1) ≤ D̄_{f} (p_{1}, ..., p_{n+1}, q_{1}, ..., q_{n+1})
≤ D̄_{f} (p_{1}, ..., p_{n}, q_{1}, ..., q_{n}) ≤ ... ≤ D̄_{f} (p_{1}, p_{2}, q_{1}, q_{2})
≤ D̄_{f} (p_{1}, q_{1}) ,

provided that p_{i}, q_{i} ∈ Ω, i ∈ ℕ and D_{f} (·, ·) is the usual f−Csiszár divergence.

Proof. Using Jensen’s inequality for multiple integrals, we obtain

∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n}) f [\frac{1}{n} (\frac{q_{1} (x_{1})}{p_{1} (x_{1})} + ... + \frac{q_{n} (x_{n})}{p_{n} (x_{n})})] dμ (x_{1}) ...dμ (x_{n})

≥ f [∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n}) · \frac{1}{n} (\frac{q_{1} (x_{1})}{p_{1} (x_{1})} + ... + \frac{q_{n} (x_{n})}{p_{n} (x_{n})}) dμ (x_{1}) ...dμ (x_{n})]
= f (1)

because a simple calculation shows that

\frac{1}{n} ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n}) (\frac{q_{1} (x_{1})}{p_{1} (x_{1})} + ... + \frac{q_{n} (x_{n})}{p_{n} (x_{n})}) dμ (x_{1}) ...dμ (x_{n})

= \frac{1}{n} [∫_{χ} q_{1} (x_{1}) dμ (x_{1}) ∫_{χ} p_{2} (x_{2}) dμ (x_{2}) ... ∫_{χ} p_{n} (x_{n}) dμ (x_{n})

+ ∫_{χ} p_{1} (x_{1}) dμ (x_{1}) ... ∫_{χ} q_{n} (x_{n}) dμ (x_{n})] = 1

and the first inequality in (2.2) is proved.

4 S.S. DRAGOMIR

If we apply Jensen’s inequality for y_{1}, ..., y_{n+1}, we may write

(2.3)  \frac{1}{n + 1} [f (y_{1}) + ... + f (y_{n+1})] ≥ f (\frac{y_{1} + ... + y_{n+1}}{n + 1} ) .

Choose

y_{1} = \frac{z_{1} + z_{2} + ... + z_{n−1} + z_{n}}{n} ,
y_{2} = \frac{z_{2} + z_{3} + ... + z_{n} + z_{n+1}}{n} ,
⋮
y_{n+1} = \frac{z_{n+1} + z_{1} + ... + z_{n−1}}{n} .

Then

\frac{1}{n + 1} (y_{1} + ... + y_{n+1}) = \frac{n (z_{1} + z_{2} + ... + z_{n+1})}{n (n + 1)}
= \frac{z_{1} + z_{2} + ... + z_{n} + z_{n+1}}{n + 1}

and then, by (2.3) we have the inequality

\frac{1}{n + 1} [f (\frac{z_{1} + z_{2} + ... + z_{n−1} + z_{n}}{n} ) + ... + f (\frac{z_{n+1} + z_{1} + ... + z_{n−1}}{n} )] (2.4)

≥ f (\frac{z_{1} + z_{2} + ... + z_{n} + z_{n+1}}{n + 1} )

for all z_{1}, ..., z_{n+1} ∈ [0, ∞).

If we put in (2.4) z_{i} = \frac{p_{i}(x_{i})}{q_{i}(x_{i})} (i = 1, ..., n + 1) and if we multiply with p_{1} (x_{1}) ...p_{n+1} (x_{n+1}) ≥ 0 and then integrate over χ^{n+1}, we may write

\frac{1}{n + 1} [∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n+1} (x_{n+1}) (2.5)

×f [\frac{1}{n} (\frac{p_{1} (x_{1})}{q_{1} (x_{1})} + ... + \frac{p_{n} (x_{n})}{q_{n} (x_{n})})] dμ (x_{1}) ...dμ (x_{n+1})

... + ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n+1} (x_{n+1})

× f [\frac{1}{n} (\frac{p_{n+1} (x_{n+1})}{q_{n+1} (x_{n+1})} + \frac{p_{1} (x_{1})}{q_{1} (x_{1})} + ... + \frac{p_{n−1} (x_{n−1})}{q_{n−1} (x_{n−1})})] dμ (x_{1}) ...dμ (x_{n+1})]

≥ ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n+1} (x_{n+1})

×f [\frac{1}{n + 1} (\frac{p_{1} (x_{1})}{q_{1} (x_{1})} + ... + \frac{p_{n+1} (x_{n+1})}{q_{n+1} (x_{n+1})})] dμ (x_{1}) ...dμ (x_{n+1})

= D̄_{f} (p_{1}, ..., p_{n+1}, q_{1}, ..., q_{n+1})

A GENERALIZED f−DIVERGENCE 5

and as

\frac{1}{n + 1} [∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n+1} (x_{n+1})

×f [\frac{1}{n} (\frac{p_{1} (x_{1})}{q_{1} (x_{1})} + ... + \frac{p_{n} (x_{n})}{q_{n} (x_{n})})] dμ (x_{1}) ...dμ (x_{n+1})

= ... = ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n+1} (x_{n+1})

×f [\frac{1}{n} (\frac{p_{n+1} (x_{n+1})}{q_{n+1} (x_{n+1})} + ... + \frac{p_{n−1} (x_{n−1})}{q_{n−1} (x_{n−1})})] dμ (x_{1}) ...dμ (x_{n+1})

= D̄_{f} (p_{1}, ..., p_{n}, q_{1}, ..., q_{n}) ,

then, from (2.5), we deduce the second inequality for all n, and the theorem is thus proved.

For f : [0, ∞) → ℝ, f (x) = (x − 1)^{2}, we obtain the χ^{2}−distance

D_{χ}^{2} (p_{1}, q_{1}) = ∫_{χ} p_{1} (x) (\frac{q_{1} (x)}{p_{1} (x)} − 1)^{2} dμ (x) = ∫_{α} \frac{q_{1}^{2} (x) − p_{1}^{2} (x)}{p_{1} (x)}dμ (x) , p, q ∈ Ω.

The following corollary holds. Corollary 1. Let p_{i}, q_{i} ∈ Ω (i ∈ ℕ) and define the χ^{2}−divergence by D̄_{χ}^{2} (p_{1}, ..., p_{n}; q_{1}, ..., q_{n}) (2.6)

: = ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n}) [\frac{(\frac{q_{1}(x_{1})}{p_{1}(x_{1})} + ... + \frac{q_{n}(x_{n})}{p_{n}(x_{n})})}{n} − 1]^{2} dμ (x_{1}) ...dμ (x_{n}) .

Then

(2.7)  D̄_{χ}^{2} (p_{1}, ..., p_{n}; q_{1}, ..., q_{n}) = \frac{1}{n^{2}} ∑_{i=1}^{n} D_{χ}^{2} (p_{i}, q_{i})

and

(2.8)  0 ≤ D̄_{χ}^{2} (p_{1}, ..., p_{n+1}; q_{1}, ..., q_{n+1}) ≤ D̄_{χ}^{2} (p_{1}, ..., p_{n}; q_{1}, ..., q_{n})
≤ ... ≤ D̄_{χ}^{2} (p_{1}, p_{2}, q_{1}, q_{2}) ≤ D̄_{χ}^{2} (p_{1}, q_{1})

for all n ∈ ℕ, n ≥ 1.

6 S.S. DRAGOMIR

Proof. We observe that

D̄_{χ}^{2} (p_{1}, ..., p_{n}; q_{1}, ..., q_{n})
= \frac{1}{n^{2}} ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n})

× [(\frac{q_{1} (x_{1})}{p_{1} (x_{1})} − 1) + ... + (\frac{q_{n} (x_{n})}{p_{n} (x_{n})} − 1)]^{2} dμ (x_{1}) ...dμ (x_{n})

= \frac{1}{n^{2}} ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n}) [∑_{i=1}^{n} (\frac{q_{i} (x_{i})}{p_{i} (x_{i})} − 1)^{2}

+ 2 ∑_{1≤i<j≤n} (\frac{q_{i} (x_{i})}{p_{i} (x_{i})} − 1) (\frac{q_{j} (x_{j})}{p_{j} (x_{j})} − 1)] dμ (x_{1}) ...dμ (x_{n})

= \frac{1}{n^{2}} ∑_{i=1}^{n} ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n}) (\frac{q_{i} (x_{i})}{p_{i} (x_{i})} − 1)^{2} dμ (x_{1}) ...dμ (x_{n})

+\frac{2}{n^{2}} ∑_{1≤i<j≤n} ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n})

× (\frac{q_{i} (x_{i})}{p_{i} (x_{i})} − 1) (\frac{q_{j} (x_{j})}{p_{j} (x_{j})} − 1) dμ (x_{1}) ...dμ (x_{n}) .

However, for any i, we have

∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n}) (\frac{q_{i} (x_{i})}{p_{i} (x_{i})} − 1)^{2} dμ (x_{1}) ...dμ (x_{n})

= ∫_{χ} p_{1} (x_{1}) dμ (x_{1}) ... ∫_{χ} p_{i} (x_{i}) dμ (x_{i}) ... ∫_{χ} p_{n} (x_{n}) dμ (x_{n})
= ∫_{χ} p_{i} (x_{i}) (\frac{q_{i} (x_{i})}{p_{i} (x_{i})} − i)^{2} dμ (x_{i}) = D̄_{χ}^{2} (p_{i}, q_{i})

and (for i ≠ j)

∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n}) (\frac{q_{i} (x_{i})}{p_{i} (x_{i})} − 1) (\frac{q_{j} (x_{j})}{p_{j} (x_{j})} − 1) dμ (x_{1}) ...dμ (x_{n})

= ∫_{χ} p_{1} (x_{1}) dμ (x_{1}) ... ∫_{χ} p_{i} (x_{i}) (\frac{q_{i} (x_{i})}{p_{i} (x_{i})} − 1) dμ (x_{i})

... ∫_{χ} p_{j} (x_{j}) (\frac{q_{j} (x_{j})}{p_{j} (x_{j})} − 1) dμ (x_{j})

... ∫_{χ} p_{n} (x_{n}) dμ (x_{n}) = 0 as

∫_{χ} p_{i} (x_{i}) (\frac{q_{i} (x_{i})}{p_{i} (x_{i})} − 1) dμ (x_{i}) = ∫_{χ} p_{j} (x_{j}) (\frac{q_{j} (x_{j})}{p_{j} (x_{j})} − 1) dμ (x_{j}) = 0

and then the representation (2.7) is proved. The sequence of inequalities in (2.8) follows by (2.2) and we omit the details.

For f : [0, ∞) → ℝ, f (x) = e^{x}, define the exp-divergence by

(2.9)  D_{exp(·)} (p_{1}, q_{1}) := ∫_{χ} p_{1} (x) exp (\frac{q_{1} (x)}{p_{1} (x)}) dμ (x) , p, q ∈ Ω.

The following corollary holds. Corollary 2. Let p_{i}, q_{i} ∈ Ω (i ∈ ℕ) and define the exp-divergence by

D̄_{exp} (p_{1}, ..., p_{n}; q_{1}, ..., q_{n}) (2.10)

: = ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n}) exp ( \frac{\frac{q_{1}(x_{1})}{p_{1}(x_{1})} + ... + \frac{q_{n}(x_{n})}{p_{n}(x_{n})}}{n} ) dμ (x_{1}) ...dμ (x_{n}) .

Then

(2.11)  D̄_{exp} (p_{1}, ..., p_{n}; q_{1}, ..., q_{n}) = ∏_{i=1}^{n} D_{[exp(·)]}_{\frac{1}{n}} (p_{i}, q_{i})

and

(2.12) 1 ≤ D̄_{exp} (p_{1}, ..., p_{n+1}; q_{1}, ..., q_{n+1}) ≤ D̄_{exp} (p_{1}, ..., p_{n}; q_{1}, ..., q_{n})
≤ ... ≤ D̄_{exp} (p_{1}, p_{2}, q_{1}, q_{2}) ≤ D_{exp} (p_{1}, q_{1}) ,

for all n ≥ 1.

Proof. We observe that

D̄_{exp} (p_{1}, ..., p_{n}; q_{1}, ..., q_{n})
= ∫_{χ} ... ∫_{χ} p_{1} (x_{1}) ...p_{n} (x_{n}) exp [\frac{q_{1} (x_{1})}{p_{1} (x_{1})}] ... exp [\frac{q_{n} (x_{n})}{p_{n} (x_{n})}] dμ (x_{1}) ...dμ (x_{n})

= ∏_{i=1}^{n} ∫_{χ} p_{i} (x_{i}) exp [\frac{q_{i} (x_{i})}{np_{i} (x_{i})}] dμ (x_{i})

= ∏_{i=1}^{n} ∫_{χ} p_{i} (x_{i}) (exp \frac{q_{i} (x_{i})}{p_{i} (x_{i})})^{\frac{1}{n}} dμ (x_{i}) = ∏_{i=1}^{n} D_{[exp(·)]}_{\frac{1}{n}} (p_{i}, q_{i})

and the identity (2.11) is proved.

The sequence of inequalities (2.12) follows by (2.2).

References

[1] S.M. ALI and S.D. SILVEY, A general class of coefficients of divergence of one distribution from another, J. Roy. Statist. Soc. Sec B, 28 (1966), 131-142.

[2] N.S. BARNETT, S.S. DRAGOMIR and A. SOFO, Better bounds for an inequality of the Ostrowski type with applications, RGMIA Research Report Collection, 3 (1) (2000), Article 1.

[3] M. BETH BASSAT, f-entropies, probability of error and feature selection, Inform. Control, 39 (1978), 227-242.

[4] A. BHATTACHARYYA, On a measure of divergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc., 35 (1943), 99-109.

[5] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy function, IEEE Trans. Inf. Th., 28 (3) (1982), 489-495.

[6] P. CERONE and S.S. DRAGOMIR, Midpoint type rules from an inequalities point of view, RGMIA Research Report Collect., 2 (1999), No. 7, Article 1

(http://rgmia.vu.edu.au/v2n7.html)

[7] P. CERONE and S.S. DRAGOMIR, Trapezoidal type rules from an inequalities point of view, RGMIA Research Report Collect., 2 (1999), No. 6, Article 8

(http://rgmia.vu.edu.au/v2n6.html)

[8] C.H. CHEN, Statistical Pattern Recognition, Rocelle Park, New York, Hoyderc Book Co., 1973.

[9] C.K. CHOW and C.N. LIN, Approximating discrete probability distributions with dependence trees, IEEE Trans. Inf. Th., 14 (3) (1968), 462-467.

[10] I. CSISZÁR, Information-type measures of difference of probability distributions and indirect observations, Studia Math. Hungarica, 2 (1967), 299-318.

[11] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[12] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[13] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[14] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[15] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[16] D. DACUNHA-CASTELLE, Ecole d′ete de Probability de Saint-Flour, III-1977, Berlin, Heidelberg: Springer 1978.

[17] S.S. DRAGOMIR and S. MABIZELA, Some error estimates in the trapezoidal quadrature rule, RGMIA Research Report Collect., 2 (1999), No. 5, Article 6

(http://rgmia.vu.edu.au/v2n5.html)

[18] S.S. DRAGOMIR and S. WANG, A new inequality of Ostrowski-Grüss type and its applications to the estimation of error bounds for special means and for some numerical quadrature rules, Comp. Math. Appl., 33 (1997), No. 11, 15-20.

[19] D.V. GOKHALE and S. KULLBACK, Information in Contingency Tables, New York, Marcel Dekker, 1978.

[20] J.H. HAVRDA and F. CHARVAT, Quantification method classification process: concept of structural α-entropy, Kybernetika, 3 (1967), 30-35.

[21] E. HELLINGER, Neue Bergrüirdung du Theorie quadratisher Formerus von uneudlichvieleu Veränderlicher, J. für reine and Augeur. Math., 36 (1909), 210-271.

[22] H. JEFFREYS, An invariant form for the prior probability in estimating problems, Proc. Roy. Soc. London, 186 A (1946), 453-461.

[23] T.T. KADOTA and L.A. SHEPP, On the best finite set of linear observables for discriminating two Gaussian signals, IEEE Trans. Inf. Th., 13 (1967), 288-294.

[24] T. KAILATH, The divergence and Bhattacharyya distance measures in signal selection, IEEE Trans. Comm. Technology., Vol COM-15 (1967), 52-60.

[25] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), 1-16.

[26] D. KAZAKOS and T. COTSIDAS, A decision theory approach to the approximation of discrete probability densities, IEEE Trans. Perform. Anal. Machine Intell., 1 (1980), 61-67.

[27] J.H.B. KEMPERMAN, On the optimum note of transmitting information, Ann. Math. Statist., 40 (1969), 2158-2177.

[28] C. KRAFT, Some conditions for consistency and uniform consistency of statistical procedures, Univ. of California Pub. in Statistics, 1 (1955), 125-142.

[29] S. KULLBACK, A lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 13 (1967), 126-127.

[30] S. KULLBACK, Correction to a lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[31] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Ann. Math. Stat., 22 (1951), 79-86.

[32] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[33] L. LECAM, Asymptotic Methods in Statistical Decision Theory, New York: Springer, 1986.

[34] J. LIN, Divergence measures based on the Shannon entropy, IEEE Trans. Inf. Th., 37 (1) (1991), 145-151.

[35] J. LIN and S.K.M. WONG, A new directed divergence measure and its characterization, Int. J. General Systems, 17 (1990), 73-81.

[36] H.P. McKEAN, JR., Speed of approach to equilibrium for Koc’s caricature of a Maximilian gas, Arch. Ration. Mech. Anal., 21 (1966), 343-367.

[37] M. MEI, The theory of genetic distance and evaluation of human races, Japan J. Human Genetics, 23 (1978), 341-369.

[38] C.E.M. PEARCE, J. PEČARIĆ, N. UJEVIĆ and S. VAROS̆ANEC, Generaliasations of some inequalities of Ostrowski-Grüss type, Math. Ineq. & Appl., 3 (2000), No. 1, 25-34.

[39] E.C. PIELOU, Ecological Diversity, Wiley, New York, 1975.

[40] M.S. PINSKER, Information and Information Stability of Random variables and processes, (in Russian), Moscow: Izv. Akad. Nouk, 1960.

[41] C.R. RAO, Diversity and dissimilarity coefficients: a unified approach, Theoretic Population Biology, 21 (1982), 24-43.

[42] A. RÉNYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Stat. and Prob., University of California Press, 1 (1961), 547-561.

[43] A. SEN, On Economic Inequality, Oxford University Press, London 1973.

[44] B.D. SHARMA and D.P. MITTAL, New non-additive measures of relative information, Journ. Comb. Inf. Sys. Sci., 2 (4)(1977), 122-132.

[45] H. SHIOYA and T. DA-TE, A generalisation of Lin divergence and the derivative of a new information divergence, Elec. and Comm. in Japan, 78 (7) (1995), 37-40.

[46] I. J. TANEJA, Generalised Information Measures and their Applications

    (http://www.mtm.ufsc.br/~taneja/bhtml/bhtml.html).

[47] H. THEIL, Economics and Information Theory, North-Holland, Amsterdam, 1967.

[48] H. THEIL, Statistical Decomposition Analysis, North-Holland, Amsterdam, 1972.

[49] F. TOPSOE, Some inequalities for information divergence and related measures of discrimination, Res. Rep. Coll., RGMIA, 2 (1) (1999), 85-98.

[50] G.T. TOUSSAINT, Sharper lower bounds for discrimination in terms of variation, IEEE Trans. Inf. Th., 21 (1975), 99-100.

[51] I. VAJDA, Note on discrimination information and variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[52] I. VAJDA, Theory of Statistical Inference and Information, Dordrecht-Boston, Kluwer Academic Publishers, 1989.

[53] V.A. VOLKONSKI and J. A. ROZANOV, Some limit theorems for random function -I, (English Trans.), Theory Prob. Appl., ( USSR), 4 (1959), 178-197.

School of Commuincations and Informatics, Victoria University of Technology, PO Box 14428, Melbourne City MC, Victoria 8001, Australia.

   E-mail address: sever.dragomir@vu.edu.au
   URL: http://rgmia.vu.edu.au/SSDragomirWeb.html