The field of Large Language Models (LLMs) is moving beyond simply scaling the parameters during pre-training. Instead, researchers are focusing on Test-Time Scaling (TTS), which is a mechanism to allocate computational resources dynamically during the inference process. In the rapidly emerging field of Agentic LLMs, TTS has also shown great value in enabling agents to handle complex tasks that static models cannot handle. Despite all the substantial work in this field, it still lacks a comprehensive survey summarizing these advances. To the best of our knowledge, this paper presents the first systematic review of TTS tailored for Agentic LLMs. To this end, we propose a novel RAIE taxonomy along four scaling dimensions: (i) Reasoning optimizes the entire thought process through search algorithms and self-verification; (ii) Acting involves collecting information through external tools and environmental feedback; (iii) Interaction copes with complex tasks by forming various multi-agent networks; (iv) Evolution enables agents to continuously update their memory and strategies. Furthermore, we also introduce a task-oriented guideline for choosing the best TTS strategy. Finally, we sum up challenges and future directions to inspire further studies in this field. A curated list of papers and associated code will be made publicly available.