We present a face identification algorithm that automatically processes an unknown image by locating and identifying the face. The heart of the algorithm is the use of pursuit filters. A matching pursuit filter is an adapted wavelet expansion, where the expansion is adapted to both the data and the pattern recognition problem being addressed. For identification, the filters find the features that differentiate among faces, whereas, for detection, the filters encode the similarities among faces. The filters are designed though a simultaneous decomposition of a training set into a two-dimensional (2-D) wavelet expansion. This yields a representation that is explicitly 2-D and encodes information locally. The algorithm uses coarse to fine processing to locate a small set of key facial features, which are restricted to the nose and eye regions of the Face. The result is an algorithm that is robust to variations in facial expression, hair style, and the surrounding environment. Based on the locations of the facial features, the identification module searches the data base for the identity of the unknown face using matching pursuit filters to make the identification. The algorithm was demonstrated on three sets of images. The first set was images from the FERET data base. The second set was infrared and visible images of the same people. This demonstration was done to compare performance on infrared and visible images individually, and on fusing the results from both modalities. The third set was mugshot data from a law enforcement application