As AI systems are increasingly deployed in high-stakes settings, two concerns become particularly important: cautiousness (awareness of model limitations for risk-aware decisions) and explainability (providing human-understandable justifications). My PhD research develops methods that enable both, from theoretical and application perspectives. On the theoretical side, I formalize cautious and interpretable models for multiple uncertainty frameworks, with algorithms for explanation and preference elicitation under uncertainty. On the application side, I implement these methods in post-stroke rehabilitation, designing AI systems that analyze multimodal clinical data to support individualized rehabilitation planning.