Measures of dependence

How to quantitatively measure the dependence between two random variables?

Methods

Many methods can be obtained by using a statistical distance to compare the joint distribution with the product of the marginal distributions, e.g.

  • MI from KL divergence
  • Mutual dependence from Hellinger distance
  • Distance covariance from weighted \(L^2\) metric on characteristic functions

Literature

Surveys

  • Josse, Holmes, 2016: Measuring multivariate association and beyond (doi, arxiv)
    • Focus on RV coefficient and distance covariance but see Sec. 4 for other measures
  • de Siqueira Santos et al, 2014: A comparative study of statistical methods used to identify dependencies between gene expression signals (doi)
  • Wagner & Eckhoff, 2015: Technical privacy metrics: a systematic survey (arxiv)