Learned agents that control traffic signals or call software tools are usually trained in simulators or fixed benchmarks, yet must act in worlds that differ. Many reliability concerns (robustness, safety, alignment, etc.) become one problem once agents are written as Markov decision processes (MDPs): a gap between the world an agent was built for and the world it acts in. The sim-to-real gap decomposes along state, observation, action, transition, and reward; multi-agent deployment further adds topology, population, and task gaps as neighbors learn and workloads scale. Using reinforcement learning (RL) for traffic control and large language model (LLM) agents for tool use and mobile GUIs as running examples, this paper traces how mismatches appear on each channel and how mitigations from one community (domain randomization, grounded action transformation, perturbation benchmarks) transfer to the other. Because the gap rarely closes, reliable deployment needs selective human oversight; uncertainty quantification can flag when to escalate, localize errors at the answer, reasoning, step, or agent level, and support sim-to-real transfer in RL. This paper opens an agenda for a shared MDP vocabulary, uncertainty-aware oversight, and scaling human-AI collaboration across volume, complexity, and expertise.