Letting AI agents interact in multi-agent settings introduces significant complexity in predicting and interpreting their collective behavior, with profound implications for trustworthy AI adoption in research and society. We present FAIRGAME (Framework for AI Agents Bias Recognition using Game Theory), an open-source framework that simulates game-theoretic scenarios with LLM-based agents to systematically uncover biases arising from model choice, language, agent personality, and more. Applied to the Prisoner’s Dilemma and Battle of the Sexes across four LLMs and five human languages, FAIRGAME reveals inconsistencies across LLM models and consistent deviations from game-theoretic predictions; it also quantifies LLM-specific behavioral tendencies through a novel scoring system. Our results show that LLMs draw on prior world knowledge beyond payoff matrices, and that language and personality significantly shape strategic outcomes, supporting the use of reproducible and controlled simulation pipelines to predict the interacting behavior of LLM agents.