Learning the parts of objects by non-negative matrix factorization (Letters to Nature)

Daniel D. Lee*& H. Sebastian Seung*†
*Bell Laboratories, Lucent Technologies, Murray Hill, New Jersey 07974, USA
†Department of Brain and Cognitive Sciences, Massachusetts Institute of
Technology, Cambridge, Massachusetts 02139, USA

 

The differences between PCA, VQ and NMF arise from different constraints imposed on the matrix factorsWandH.

In VQ, each column ofHis constrained to be a unary vector, with one element equal to unity and the other elements equal to zero. In other words, every face (column ofV) is approximated by a single basis image (column ofW) in the factorizationV≈WH. Such a unary encod-ing for a particular face is shown next to the VQ basis in Fig. 1. This unary representation forces VQ to learn basis images that are prototypical faces.

PCA constrains the columns ofWto be orthonormal and the rows ofHto be orthogonal to each other. This relaxes the unary constraint of VQ, allowing a distributed representation in which each face is approximated by a linear combination of all the basis images, or eigenfaces. A distributed encoding of a particular face is shown next to the eigenfaces in Fig. 1. Although eigenfaces have a statistical interpretation as the directions of largest variance, many of them do not have an obvious visual interpretation. This is because PCA allows the entries ofWandHto be of arbitrary sign. As the eigenfaces are used in linear combinations that generally involve complex cancellations between positive and negative numbers, many individual eigenfaces lack intuitive meaning.

NMF does not allow negative entries in the matrix factors W and H. Unlike the unary constraint of VQ, these non-negativity con-straints permit the combination of multiple basis images to repre-sent a face. But only additive combinations are allowed, because the non-zero elements ofWandHare all positive. In contrast to PCA, no subtractions can occur. For these reasons, the non-negativity constraints are compatible with the intuitive notion of combining parts to form a whole, which is how NMF learns a parts-based representation.