Predicting users’ activity and location preferences is of great significance in location based services. Considering that users’ activity and location preferences interplay with each other, many scholars tried to figure out the relation between users’ activities and locations for improving prediction performance. However, most previous works enforce a rigid human-defined modeling strategy to capture these two factors, either activity purpose controlling location preference or spatial region determining activity preference. Unlike existing methods, we introduce spatial-activity topics as the latent factor capturing both users’ activity and location preferences. We propose Multi-task Context Aware Recurrent Neural Network to leverage the spatial activity topic for activity and location prediction. More specifically, a novel Context Aware Recurrent Unit is designed to integrate the sequential dependency and temporal regularity of spatial activity topics. Extensive experimental results based on real-world public datasets demonstrate that the proposed model significantly outperforms state-of-the-art approaches.