Rare and revealing: A new method for uncovering hidden patterns in data

In disciplines as diverse as public health, economics, genomics, and causal discovery, it’s important to determine dependence patterns in data, which describe how one variable influences another. While researchers have developed methods to do this, current tests often fail to identify dependence in small regions of data where the relationships between variables differ from those in most of the data. This characteristic, termed ‘rare dependence’ in a recent study by researchers from MBZUAI and other institutions, throws off many of today’s data analysis tools.

To address this problem, the researchers developed a new method that has the potential to be used in fields that analyze observational data. By using kernel-based conditional independence testing via sample importance reweighting, it identifies regions in data that exhibit rare dependence and assigns comparatively more importance to them.

The team tested their algorithm on synthetic and real-world data and found that it successfully identified relations between variables even in the presence of rare dependence, illustrating its potential to help researchers gain new insights from data analysis.

Yiqing Li, a research associate at MBZUAI, is a co-author of the study and recently presented the findings at the International Conference on Machine Learning (ICML) in Vancouver. Yewei Xia, Xiaofei Wang, Zhengming Chen, Liuhua Peng, Mingming Gong, and Kun Zhang are co-authors.

What’s difficult about rare dependence?

Rare dependence occurs frequently in real-world datasets and it can also be an issue in AI-related fields like causal inference, feature selection, and self-supervised learning. In medicine, drugs can sometimes have unexpected effects that manifest only in a small subpopulation. For example, opioids have been found to actually make the pain worse in a subset of patients. Current independence tests struggle in these situations.

Li and her colleagues describe their approach as assigning more “attention to the dependent sub-samples, successfully detecting rare dependence.” Data points that are determined to exhibit more significant dependence patterns are automatically given more weight.

The researchers tested their method on two synthetic datasets. They compared their method to several other independence tests, including the Hilbert-Schmidt Independence Criterion (HSIC) which pays attention to all samples in a dataset in the same way.

For both datasets, their method controlled for type I errors, a kind of “false positive,” where a test determines that variables are dependent when they are actually independent. Theirs did so with higher testing power in both cases, which the researchers say confirms the need for reweighting a subset of samples in the case of rare dependence.

A scatter plot of data points generated by the researchers to evaluate independence tests in the presence of rare dependence, indicated by the red box. Traditional independence tests like HSIC evaluate the whole sample, which can produce misleading results.