Nonnegative matrix factorization (NMF),  a well-known technique  to find  parts-based representations of nonnegative data, has been widely studied. In reality,  ordinal relations often exist among data,  such as data i is more related to j than to q.  Such relative order is naturally available, and more importantly, it truly reflects the latent data structure.  Preserving the ordinal relations enables us to find structured representations of data that are faithful to the relative order, so that the learned representations become  more discriminative. However, current NMFs pay no attention to this. In this paper, we make the first attempt towards incorporating the ordinal relations and  propose a novel ranking preserving nonnegative matrix factorization (RPNMF) approach, which enforces the learned representations to be ranked according to the relations. We derive  iterative updating rules to solve RPNMF's objective function with  convergence guaranteed.  Experimental results with several datasets for clustering and classification have demonstrated that RPNMF achieves greater performance against the state-of-the-arts,  not only  in terms of  accuracy, but also interpretation of orderly data structure.