ON THE p−LOGARITHMIC AND α−POWER DIVERGENCE MEASURES IN INFORMATION THEORY

S.S. DRAGOMIR

Abstract. In this paper we introduce the concepts of p−logarithmic and α−power divergence measures and point out a number of basic results.

1. Introduction

One of the important issues in many applications of Probability Theory is finding an appropriate measure of distance (or difference or discrimination ) between two probability distributions. A number of divergence measures for this purpose have been proposed and extensively studied by Jeffreys [2], Kullback and Leibler [3], Rényi [4], Havrda and Charvat [5], Kapur [6], Sharma and Mittal [7], Burbea and Rao [8], Rao [9], Lin [10], Csiszár [11], Ali and Silvey [13], Vajda [14], Shioya and Da-te [48] and others (see for example [6] and the references therein).

These measures have been applied in a variety of fields such as: anthropology [9], genetics [15], finance, economics, and political science [16], [17], [18], biology [19], the analysis of contingency tables [20], approximation of probability distributions [21], [22], signal processing [23], [24] and pattern recognition [25], [26].

Assume that a set χ and the σ−finite measure μ are given. Consider the set of all probability densities on μ to be Ω := {p|p : χ → ℝ, p (x) ≥ 0, ∫_{χ} p (x) dμ (x) = 1}. The Kullback-Leibler divergence [3] is well known among the information divergences. It is defined as:

(1.1)  D_{KL} (p, q) := ∫_{χ} p (x) log [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω,

where log is to base 2.

In Information Theory and Statistics, various divergences are applied in addition to the Kullback-Leibler divergence. These are the: variation distance D_{v}, Hellinger distance D_{H} [41], χ^{2}−divergence D_{χ}^{2}, α−divergence D_{α}, Bhattacharyya distance D_{B} [42], Harmonic distance D_{Ha}, Jeffrey’s distance D_{J} [2], triangular discrimination D_{Δ} [36], etc... They are defined as follows:

(1.2)  D_{v} (p, q) := ∫_{χ} |p (x) − q (x)| dμ (x) , p, q ∈ Ω;

(1.3)  D_{H} (p, q) := ∫_{χ} |\sqrt{p (x)} − \sqrt{q (x)}| dμ (x) , p, q ∈ Ω;

Date: March 15, 2000.

1991 Mathematics Subject Classification. Primary 94Xxx; Secondary 26D15.

Key words and phrases. Divergence measures in Information Theory.

(1.4)  D_{χ}^{2} (p, q) := ∫_{χ} p (x) [(\frac{q (x)}{p (x)})^{2} − 1] dμ (x) , p, q ∈ Ω;

(1.5)  D_{α} (p, q) := \frac{4}{1 − α^{2}} [1 − ∫_{χ} [p (x)]^{\frac{1−α}{2}} [q (x)]^{\frac{1+α}{2}} dμ (x)] , p, q ∈ Ω;

(1.6)  D_{B} (p, q) := ∫_{χ} \sqrt{p (x) q (x)}dμ (x) , p, q ∈ Ω;

(1.7)  D_{Ha} (p, q) := ∫_{χ} \frac{2p (x) q (x)}{p (x) + q (x)}dμ (x) , p, q ∈ Ω;

(1.8)  D_{J} (p, q) := ∫_{χ} [p (x) − q (x)] ln [\frac{p (x)}{q (x)}] dμ (x) , p, q ∈ Ω;

(1.9)  D_{Δ} (p, q) := ∫_{χ} \frac{[p (x) − q (x)]^{2}}{p (x) + q (x)} dμ (x) , p, q ∈ Ω.

For other divergence measures, see the paper [6] by Kapur or the book on line [43] by Taneja. For a comprehensive collection of preprints available on line, see the

RGMIA web site http://rgmia.vu.edu.au/papersinfth.html

Csiszár f−divergence is defined as follows [11]

(1.10)  D_{f} (p, q) := ∫_{χ} p (x) f [\frac{q (x)}{p (x)}] dμ (x) , p, q ∈ Ω,

where f is convex on (0, ∞). It is assumed that f (u) is zero and strictly convex at u = 1. By appropriately defining this convex function, various divergences are derived. All the above distances (1.1) − (1.9), are particular instances of Csiszár f−divergence. There are also many others which are not in this class (see for example [6] or [43]). For the basic properties of Csiszár f−divergence see [44]-[46].

In [47], Lin and Wong (see also [10]) introduced the following divergence

(1.11)  D_{LW} (p, q) := ∫_{χ} p (x) log [ \frac{p (x)}{\frac{1}{2}p (x) + \frac{1}{2}q (x)}] dμ (x) , p, q ∈ Ω.

This can be represented as follows, using the Kullback-Leibler divergence:

D_{LW} (p, q) = D_{KL} (p, \frac{1}{2}p + \frac{1}{2}q) .

Lin and Wong have established the following inequalities

(1.12)  D_{LW} (p, q) ≤ \frac{1}{2}D_{KL} (p, q) ;

(1.13)  D_{LW} (p, q) + D_{LW} (q, p) ≤ D_{v} (p, q) ≤ 2;

(1.14)  D_{LW} (p, q) ≤ 1.

p−LOGARITHMIC AND α−POWER DIVERGENCE MEASURES 3

In [48], Shioya and Da-te improved (1.12) − (1.14) by showing that

D_{LW} (p, q) ≤ \frac{1}{2}D_{v} (p, q) ≤ 1.

For classical and new results in comparing different kinds of divergence measures, see the papers [2]-[48] where further references are given.

2. Some New Divergence Measures

We define the p−Logarithmic means by (see [1, p. 346])

(2.1)  L_{p} (a, b) = \{ \begin{matrix}[\frac{b^{p+1}−a^{p+1}}{(p+1)(b−a)}]^{\frac{1}{p}} , & p ≠ −1, 0, \\ \frac{b−a}{ln b−ln a}, & p = −1, \\ \frac{1}{e} (\frac{b^{b}}{a^{a}} )^{\frac{1}{b−a}} , & p = 0, \end{matrix} a ≠ b, a, b > 0

L_{p} (a, a) = a.

Where convenient, L_{−1} (a, b) −the logarithmic mean, will be written as just L (a, b). The case p = 0 is also called the identric mean, i.e., L_{0} (a, b) and will be denoted by I (a, b). Of course, we will also define L_{∞} (a, b) = max {a, b} and L_{−∞} = min {a, b} to complete the scale.

It is easily checked that the definitions in the above scale are consistent in the sense that lim_{p→0} L_{p} (a, b) = I (a, b) and lim_{p→±∞} L_{p} (a, b) = L_{±∞} (a, b).

We define the p−logarithmic divergence measure, or simply the L_{p}−divergence measure, by

(2.2) D_{L_{p}} (q, r) = \{ \begin{matrix}∫_{χ} [\frac{[q(x)]^{p+1}−[r(x)]^{p+1}}{(p+1)(q(x)−r(x))} ]^{\frac{1}{p}} dμ (x) , & if p ≠ −1, 0, \\ ∫_{χ} [\frac{q(x)−r(x)}{ln q(x)−ln r(x)}] dμ (x) , & if p = −1, \\ \frac{1}{e} ∫_{χ} [\begin{matrix}[q(x)] & ^{q(x)} \\ [r(x)] & ^{r(x)} \end{matrix} ]^{\frac{1}{q(x)−r(x)}} dμ (x) , & if p = 0, \end{matrix} q, r ∈ Ω

(2.3)  D_{+∞} (q, r) : = ∫_{χ} max {q (x) , r (x)} dμ (x) , q, r ∈ Ω,
(2.4)  D_{−∞} (q, r) : = ∫_{χ} min {q (x) , r (x)} dμ (x) , q, r ∈ Ω.

We observe that

(2.5)  D_{+∞} (q, r) = ∫_{χ} \frac{q (x) + r (x) + |q (x) − r (x)|}{2}dμ (x) = 1 + \frac{1}{2}D_{v} (q, r)

and similarly,

(2.6)  D_{−∞} (q, r) = 1 − \frac{1}{2}D_{v} (q, r) .

Since L_{p} (a, b) = L_{p} (b, a) for all a, b > 0 and p ∈ [−∞, ∞], we can conclude that the L_{p}−divergence measures are symmetrical.

Now, if we consider the continuous mappings (which are not necessarily convex)

(2.7) f_{p} (x) : = \{ \begin{matrix}[\frac{x^{p+1}−1}{(p+1)(x−1)}]^{\frac{1}{p}} , & x ∈ (0, 1) ∪ (1, ∞) , p ≠ −1, 0; \\ \frac{x−1}{ln x−1}, & x ∈ (0, 1) ∪ (1, ∞) , p = −1; \\ \frac{1}{e}x^{\frac{x}{x−1}} , & x ∈ (0, 1) ∪ (1, ∞) , p = 0; \\ 1 & if x = 1, \end{matrix}

f_{+∞} (x) = 1 + \frac{1}{2} |x − 1| ,
f_{−∞} (x) = 1 − \frac{1}{2} |x − 1| ,

and taking into account that L_{p} (a, b) = aL_{p} (1, \frac{b}{a}) for all a, b > 0 and p ∈ [−∞, ∞], we deduce that

(2.8)  D_{f_{p}} (q, r) = ∫_{χ} q (x) f_{p} (\frac{r (x)}{q (x)}) dμ (x)
= ∫_{χ} q (x) L_{p} (\frac{r (x)}{q (x)}, 1) dμ (x)
= ∫_{χ} L_{p} (r (x) , q (x)) dμ (x) = D_{L_{p}} (q, r)

for all q, r ∈ Ω, which shows that the L_{p}−divergence measure can be interpreted as f−Csiszár divergences for f = f_{p}, which are not necessarily convex.

The following result is well known in the theory of p−logarithmic means (see for example [1, p. 347]). Lemma 1. We have

(2.9)  L_{−2} (a, b) = G (a, b) ,
(2.10)  L_{− \frac{1}{2}} (a, b) = \frac{1}{2} (A (a, b) + G (a, b)) ,
(2.11)  L_{1} (a, b) = A (a, b) ,
(2.12)  L_{−3} (a, b) = [H (a, b) G^{2} (a, b)]^{\frac{1}{3}} ,

and the monotonicity property

(2.13)  L_{−∞} (a, b) ≤ L_{r} (a, b) ≤ L_{s} (a, b) ≤ L_{+∞} (a, b)

with equality iff a = b, where −∞ ≤ r < b ≤ ∞, and A (a, b) is the arithmetic mean, G (a, b) is the geometric mean and H (a, b) is the harmonic mean of a, b. In particular, we have

(2.14)  G (a, b) ≤ L (a, b) ≤ \frac{1}{2} [A (a, b) + G (a, b)] ≤ I (a, b) ≤ A (a, b)

with equality iff a = b.

Now, using (2.9) - (2.12), we observe that

(2.15)  D_{L_{−2}} (q, r) = ∫_{χ} \sqrt{r (x) q (x)}dμ (x)
= D_{B} (p, q) (Bhattacharyya distance)

(2.16)  D_{L_{− \frac{1}{2}}} (q, r) = ∫_{χ} \frac{1}{2} [\frac{q (x) + r (x)}{2} + \sqrt{q (x) r (x)}] dμ (x)
= \frac{1}{2} + \frac{1}{2}D_{B} (p, q)

(2.17) D_{L_{−3}} (q, r) = ∫_{χ} [\frac{2r (x) q (x)}{r (x) + q (x)} · r (x) q (x)]^{\frac{1}{3}} dμ (x)

= \sqrt[3]{2} ∫_{χ} \frac{r^{\frac{2}{3}} (x) q^{\frac{2}{3}} (x)}{[r (x) + q (x)]^{\frac{1}{3}}} dμ (x) =: D_{[HG^{2}]}_{\frac{1}{3}} (p, q)

for all q, r ∈ Ω.

Using Lemma 1, we can state the following fundamental theorem regarding the position of the L_{p}−divergence measures.

Theorem 1. For any q, r ∈ Ω, we have the inequality

(2.18)  1 − \frac{1}{2}D_{v} (r, q) ≤ D_{L_{u}} (r, q) ≤ D_{L_{s}} (r, q) ≤ 1 + \frac{1}{2}D_{v} (r, q)

for all −∞ ≤ u < s ≤ ∞. In particular, we have

(2.19)  1 − \frac{1}{2}D_{v} (p, q) ≤ D_{[HG^{2}]}_{\frac{1}{3}} (p, q) ≤ D_{B} (p, q) ≤ D_{L} (p, q)
≤ \frac{1}{2} + \frac{1}{2}D_{B} (p, q) ≤ D_{I} (p, q) ≤ 1,

where

D_{L} (r, q) := ∫_{χ} [\frac{r (x) − q (x)}{ln r (x) − ln q (x)}] dμ (x) is the Logarithmic divergence

and

D_{I} (r, q) = \frac{1}{e} ∫_{χ} [\begin{matrix}[r (x)] & r(x) \\ [q (x)] & q(x) \end{matrix} ]^{\frac{1}{r(x)−q(x)}} dμ (x) is the Identric divergence.

Remark 1. From (2.18), we can conclude the following inequality for the L_{p}−divergence measure in terms of the variation distance

(2.20)  |D_{L_{s}} (r, q) − 1| ≤ \frac{1}{2}D_{v} (r, q) , r, q ∈ Ω

for all s ∈ [−∞, ∞]. The constant \frac{1}{2} is sharp.

Indeed, if we assume that (2.20) holds with another constant c > 0, i.e.,

(2.21)  |D_{L_{s}} (r, q) − 1| ≤ cD_{v} (r, q) ,

then, choosing s = ∞, we obtain

|1 + \frac{1}{2}D_{v} (r, q) − 1| ≤ cD_{v} (r, q) for all r, q ∈ Ω,

which implies that c ≥ \frac{1}{2}.

For r ∈ ℝ, we define the α−th power mean of the positive numbers a, b by (see [1, p. 133])

(2.22)  M^{[α]} (a, b) := \{ \begin{matrix}(\frac{a^{α}+b^{α}}{2} )^{\frac{1}{α}} & if & α ≠ 0, α ≠ ±∞; \\ \sqrt{ab} & if & α = 0; \\ max \{a, b\} & if & α = +∞; \\ min \{a, b\} & if & α = −∞. \end{matrix}

We define the α−power divergence measure by (2.23)

D_{M^{^{[}^{α}^{]}}} (p, q) := \{ \begin{matrix}∫_{χ} [\frac{p^{α}(x)+q^{α}(x)}{2}]^{\frac{1}{α}} dμ (x) & if & α ≠ 0, α ≠ ±∞; \\ ∫_{χ} \sqrt{p (x) q (x)}dμ (x) & if & α = 0; (Bhattacharyya distance) \\ 1 + \frac{1}{2}D_{v} (p, q) & if & α = +∞; \\ 1 − \frac{1}{2}D_{v} (p, q) & if & α = −∞. \end{matrix}

Since M^{[α]} (a, b) = M^{[α]} (b, a) for all a, b > 0 and α ∈ [−∞, ∞], we can conclude that the α−power divergences are symmetrical. Now, if we consider the continuous mappings (which are not necessarily convex)

(2.24)  f_{α} (x) := \{ \begin{matrix}[\frac{x^{α}+1}{2}]^{\frac{1}{α}} & if & α ≠ 0, α ≠ ±∞; \\ \sqrt{x} & if & α = 0, x ∈ (0, ∞) ; \\ 1 + \frac{1}{2} |x − 1| & if & α = +∞; \\ 1 − \frac{1}{2} |x − 1| & if & α = −∞ \end{matrix}

and taking into account that M^{[α]} (a, b) = aM^{[α]} (1, \frac{b}{a}), we deduce that

(2.25)  D_{f_{α}} (p, q) = ∫_{α} p (x) f_{α} (\frac{q (x)}{p (x)}) dμ (x)
= ∫_{χ} p (x) M^{[α]} (\frac{q (x)}{p (x)}, 1) dμ (x)
= ∫_{χ} M^{[α]} (p (x) , q (x)) dμ (x) = D_{M^{^{[}^{α}^{]}}} (p, q)

for all p, r ∈ Ω, which shows that the α−power divergence measures can be interpreted as f−Csiszár divergences for f = f_{α}, which are not necessarily convex.

The following result concerning the fundamental property of the α−power means holds (see [1, p. 133 and p. 159]). Lemma 2. Let a, b > 0. Then

(2.26)  M^{[−∞]} (a, b) ≤ M^{[α]} (a, b) ≤ M^{[β]} (a, b) ≤ M^{[+∞]} (a, b)

for −∞ ≤ α < β ≤ ∞. Also,

(2.27)  lim_{α→0} M^{[α]} (a, b) = G (a, b) , lim_{α→±∞} M^{[α]} (a, b) = M^{[±∞]} (a, b) .

In particular

(2.28)  M^{[−1]} (a, b) = H (a, b) .

Using the above lemma, we can state the following theorem concerning the location of the α−power divergence measure. Theorem 2. For any p, q ∈ Ω, we have:

(2.29)  1 − \frac{1}{2}D_{v} (p, q) ≤ D_{M^{^{[}^{α}^{]}}} (p, q) ≤ D_{M^{^{[}^{β}^{]}}} (p, q) ≤ 1 + \frac{1}{2}D_{v} (p, q)

for −∞ ≤ α < β ≤ ∞. In particular, we have

(2.30)  1 − \frac{1}{2}D_{v} (p, q) ≤ D_{Ha} (p, q) ≤ D_{B} (p, q) ≤ 1 + \frac{1}{2}D_{v} (p, q) ,

where D_{Ha} (p, q) is the Harmonic divergence and D_{B} (p, q) is the Bhattacharyya distance. Remark 2. From (2.29), we may conclude the following inequalities for the αpower divergence measures in terms of the variational distance

(2.31)  |D_{M^{^{[}^{α}^{]}}} (p, q) − 1| ≤ \frac{1}{2}D_{v} (p, q)

for any p, q ∈ Ω and α ∈ [−∞, ∞] and the constant \frac{1}{2} is sharp.

In what follows, by the use of a result by Pittenger [1, p. 349], we obtain inequalities that are related to logarithmic and power means: Theorem 3. Let a, b > 0, −∞ < r < ∞ and define

r_{1} = min \{\frac{r + 2}{3}, r · \frac{ln 2}{ln r + 1}\} , r > −1, r ≠ 0
= min \{\frac{2}{3}, ln 2\} , r = 0
= min \{\frac{r + 2}{3}, 0\} , r ≤ −1,

with r_{2} as defined above, but with max instead of min, then

(2.32)  M^{[r_{1}]} (a, b) ≤ L_{r} (a, b) ≤ M^{[r_{2}]} (a, b) ,

with equality iff a = b or r = 1, −\frac{1}{2} or −2. The values r_{1} and r_{2} are sharp.

We are able to establish the following relationship between the power divergence and the generalized logarithmic divergence. Theorem 4. For any p, q ∈ Ω, we have:

(2.33)  D_{M^{[r_{1}]}} (p, q) ≤ D_{L_{r}} (p, q) ≤ D_{M^{[r_{2}]}} (p, q) ,

where r_{1}, r_{2} are as defined above. Remark 3. a) In the case r = −1, the corresponding inequality in (2.32) is

due to Lin (see [1, p. 349]) and for this r we obtain from (2.33)

(2.34)  B (p, q) ≤ D_{L} (p, q) ≤ D_{M^{[^{\frac{1}{3}}]}} (p, q) , p, q ∈ Ω.

b) If −2 < r < −\frac{1}{2} or r > 1, then r_{2} = \frac{r+2}{3} and the right hand side of (2.33) becomes

(2.35)  D_{L_{r}} (p, q) ≤ D_{M^{[^{\frac{r+2}{3}}]}} (p, q) , p, q ∈ Ω.

Using the above means, we can imagine all the sets of other divergences that can be constructed by the use of different contributions of these means. For example, we can define for p, q ∈ Ω

D_{(AG)}_{\frac{1}{2}} (p, q) : = ∫_{χ} \sqrt{A (p (x) , q (x)) G (p (x) , q (x))}dμ (x) ;

D_{(LI)}_{\frac{1}{2}} (p, q) : = ∫_{χ} \sqrt{L (p (x) , q (x)) I (p (x) , q (x))}dμ (x)

or even

D_{(GI)}_{\frac{1}{2}} (p, q) := ∫_{χ} \sqrt{G (p (x) , q (x)) I (p (x) , q (x))}dμ (x) .

Using Alzer’s result for means (see [1, p. 350]) Theorem 5. If a, b > 0, we have

\sqrt{A (a, b) G (a, b)} < \sqrt{L (a, b) I (a, b)} < M^{[^{\frac{1}{2}}]} (a, b) ;
L (a, b) + I (a, b) < A (a, b) + G (a, b) ;
\sqrt{G (a, b) I (a, b)} < L (a, b) < \frac{1}{2} [G (a, b) + I (a, b)] ,

and we may state the following theorem concerning the above divergence measures. Theorem 6. For any p, q ∈ Ω, we have

(2.36)  D_{(AG)}_{\frac{1}{2}} (p, q) < D_{(LI)}_{\frac{1}{2}} (p, q) < D_{M^{[^{\frac{1}{2}}]}} (p, q) ,
(2.37)  D_{L} (p, q) + D_{I} (p, q) < 1 + B (p, q) ,
(2.38)  D_{(GI)}_{\frac{1}{2}} (p, q) < D_{I} (p, q) < \frac{1}{2} [B (p, q) + D_{I} (p, q)] .

Remark 4. In this way, we have shown that any result for special means can be imported for the divergence measure generated by these means, providing a very rich universe of facts in comparing the new distances with the other distances which have already become classics in Information Theory: Bhattacharyya distance B (p, q), Harmonic distance Ha (p, q) or variation distance D_{v} (p, q), etc.

References

[1] P.S. BULLEN, D.S MITRINOVIĆ and P.M. VASIĆ, Means and Their Inequalities, Kluwer Academic Publishers, 1988.

[2] H. JEFFREYS, An invariant form for the prior probability in estimating problems, Proc. Roy. Soc. London, 186 A (1946), 453-461.

[3] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Ann. Math. Stat., 22 (1951), 79-86.

[4] A. RÉNYI, On measures of entropy and information, Proc. Fourth Berkeley Symp. Math. Stat. and Prob., University of California Press, 1 (1961), 547-561.

[5] J.H. HAVRDA and F. CHARVAT, Quantification method classification process: concept of structural α-entropy, Kybernetika, 3 (1967), 30-35.

[6] J.N. KAPUR, A comparative assessment of various measures of directed divergence, Advances in Management Studies, 3 (1984), 1-16.

[7] B.D. SHARMA and D.P. MITTAL, New non-additive measures of relative information, Journ. Comb. Inf. Sys. Sci., 2 (4)(1977), 122-132.

[8] I. BURBEA and C.R. RAO, On the convexity of some divergence measures based on entropy function, IEEE Trans. Inf. Th., 28 (3) (1982), 489-495.

[9] C.R. RAO, Diversity and dissimilarity coefficients: a unified approach, Theoretic Population Biology, 21 (1982), 24-43.

[10] J. LIN, Divergence measures based on the Shannon entropy, IEEE Trans. Inf. Th., 37 (1) (1991), 145-151.

[11] I. CSISZÁR, Information-type measures of difference of probability distributions and indirect observations, Studia Math. Hungarica, 2 (1967), 299-318.

[12] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[13] S.M. ALI and S.D. SILVEY, A general class of coefficients of divergence of one distribution from another, J. Roy. Statist. Soc. Sec B, 28 (1966), 131-142.

[14] I. VAJDA, Theory of Statistical Inference and Information, Dordrecht-Boston, Kluwer Academic Publishers, 1989.

[15] M. MEI, The theory of genetic distance and evaluation of human races, Japan J. Human Genetics, 23 (1978), 341-369.

[16] A. SEN, On Economic Inequality, Oxford University Press, London 1973.

[17] H. THEIL, Economics and Information Theory, North-Holland, Amsterdam, 1967.

[18] H. THEIL, Statistical Decomposition Analysis, North-Holland, Amsterdam, 1972.

[19] E.C. PIELOU, Ecological Diversity, Wiley, New York, 1975.

[20] D.V. GOKHALE and S. KULLBACK, Information in Contingency Tables, New York, Marcel Dekker, 1978.

[21] C.K. CHOW and C.N. LIN, Approximating discrete probability distributions with dependence trees, IEEE Trans. Inf. Th., 14 (3) (1968), 462-467.

[22] D. KAZAKOS and T. COTSIDAS, A decision theory approach to the approximation of discrete probability densities, IEEE Trans. Perform. Anal. Machine Intell., 1 (1980), 61-67.

[23] T.T. KADOTA and L.A. SHEPP, On the best finite set of linear observables for discriminating two Gaussian signals, IEEE Trans. Inf. Th., 13 (1967), 288-294.

[24] T. KAILATH, The divergence and Bhattacharyya distance measures in signal selection, IEEE Trans. Comm. Technology., Vol COM-15 (1967), 52-60.

[25] M. BETH BASSAT, f-entropies, probability of error and feature selection, Inform. Control, 39 (1978), 227-242.

[26] C.H. CHEN, Statistical Pattern Recognition, Rocelle Park, New York, Hoyderc Book Co., 1973.

[27] V.A. VOLKONSKI and J. A. ROZANOV, Some limit theorems for random function -I, (English Trans.), Theory Prob. Appl., ( USSR), 4 (1959), 178-197.

[28] M.S. PINSKER, Information and Information Stability of Random variables and processes, (in Russian), Moscow: Izv. Akad. Houk, 1960.

[29] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[30] H.P. McKEAN, JR., Speed of approach to equilibrium for Koc’s caricature of a Maximilian gas, Arch. Ration. Mech. Anal., 21 (1966), 343-367.

[31] J.H.B. KEMPERMAN, On the optimum note of transmitting information, Ann. Math. Statist., 40 (1969), 2158-2177.

[32] S. KULLBACK, A lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 13 (1967), 126-127.

[33] S. KULLBACK, Correction to a lower bound for discrimination information in terms of variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[34] I. VAJDA, Note on discrimination information and variation, IEEE Trans. Inf. Th., 16 (1970), 771-773.

[35] G.T. TOUSSAINT, Sharper lower bounds for discrimination in terms of variation, IEEE Trans. Inf. Th., 21 (1975), 99-100.

[36] F. TOPSOE, Some inequalities for information divergence and related measures of discrimination, Res. Rep. Coll., RGMIA, 2 (1) (1999), 85-98.

[37] L. LECAM, Asymptotic Methods in Statistical Decision Theory, New York: Springer, 1986.

[38] D. DACUNHA-CASTELLE, Ecole d’ete de Probability de Saint-Flour, III-1977, Berlin, Heidelberg: Springer 1978.

[39] C. KRAFT, Some conditions for consistency and uniform consistency of statistical procedures, Univ. of California Pub. in Statistics, 1 (1955), 125-142.

[40] S. KULLBACK and R.A. LEIBLER, On information and sufficiency, Annals Math. Statist., 22 (1951), 79-86.

[41] E. HELLINGER, Neue Bergrüirdung du Theorie quadratisher Formerus von uneudlichvieleu Veränderlicher, J. für reine and Augeur. Math., 36 (1909), 210-271.

[42] A. BHATTACHARYYA, On a measure of divergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc., 35 (1943), 99-109.

[43] I. J. TANEJA, Generalised Information Measures and their Applications

    (http://www.mtm.ufsc.br/~taneja/bhtml/bhtml.html).

[44] I. CSISZÁR, A note on Jensen’s inequality, Studia Sci. Math. Hung., 1 (1966), 185-188.

[45] I. CSISZÁR, On topological properties of f−divergences, Studia Math. Hungarica, 2 (1967), 329-339.

[46] I. CSISZÁR and J. KÖRNER, Information Theory: Coding Theorem for Discrete Memoryless Systems, Academic Press, New York, 1981.

[47] J. LIN and S.K.M. WONG, A new directed divergence measure and its characterization, Int. J. General Systems, 17 (1990), 73-81.

[48] H. SHIOYA and T. DA-TE, A generalisation of Lin divergence and the derivative of a new information divergence, Elec. and Comm. in Japan, 78 (7) (1995), 37-40.

School of Communications and Informatics, Victoria University of Technology, PO 14428, Melbourne City MC, Victoria, 8001, Australia

   E-mail address: sever.dragomir@vu.edu.au