Unsupervised Kernel Dimension Reduction

Unsupervised dimensionality reduction of a high-dimensional variable X can be seen as identifying a compact representation Z such that Z can be used to reconstruct X well. How to infer such Z without specifying the reconstruction function? We apply the framework of kernel dimension reduction, originally designed for supervised problems, to this problem. Concretely, kernel-based measures of independence are used to derive low-dimensional representations that maximally capture information in covariates in order to predict responses which are the same as the covariates. Our empirical studies show that the resulting compact representation yields meaningful and appealing visualization and clustering of data. Furthermore, when used in conjunction with supervised learners for classification, our methods lead to lower classification errors than state-of-the-art methods, especially when embedding data in spaces of very few dimensions.

Related publications

Discriminative learning of Bayesian latent structure models

Probabilistic topic models (and their extensions) have become popular as models of latent structures in collections of text documents or images. These models are usually treated as generative models and trained using maximum likelihood estimation, an approach which may be suboptimal in the context of an overall classification problem. In this project, we show how to train Latent Dirichlet Allocation (LDA) discriminatively by maximizing the conditional likelihood of side information such as labels. Our empirical study shows that the predictive power of the discriminatively learned LDA improves significantly over that of unsupervised LDA.

Left. 2-d embedding of LDA topics on the 2o-newsgroup data shows mixed clusterings of documents from different (color-coded) newsgroups. Right. Discriminatively trained LDA shows much clearly separated clusters of documents.

Related publications

Learning parts based representation for speech and audios

An auditory scene, composed of overlapping acoustic sources, can be viewed as a complex object whose constituent parts are the individual sources. In this project, we investigate how the technique of nonnegative matrix factorization (NMF) can be used to learn parts from voices. These parts correspond to harmonic stacks of periodic components in voices, which give rise to the perception of pitches.

Related publications