In this big-data era, vast amount of continuously arriving data can be found in various fields, such as sensor networks, network management, web and financial applications. To process such data, algorithms are usually challenged by its complex structure and high volume. Representation learning facilitates the data operation by providing a condensed description of patterns underlying the data. Knowledge discovery based on the new representations will then be computationally efficient, and to certain extent be more effective due to the removal of noise and irrelevant information in the step of representation learning. In this paper, we will briefly review state-of-the-art techniques for extracting representation and discovering knowledge from streaming and temporal data, and demonstrate their performance at addressing several real application problems.