Rating AI Models for Robustness Through a Causal Lens

1 Problem

1.1 Motivation

AI models have advanced rapidly and are now widely accessible through chatbots and other applications. However, their growing complexity has introduced opacity, making them harder to interpret, especially in critical domains like healthcare and education. These models often learn correlations rather than causal relationships, limiting their trustworthiness. Robustness refers to a model's ability to maintain performance under varying conditions, such as input perturbations or the presence of protected attributes. My work measures robustness by quantifying statistical bias, confounding bias, and the impact of perturbations on model performance. Existing methods often overlook confounders and fail to isolate the true effects of perturbations. ARC addresses these gaps and generalizes easily to other AI models. In contrast to existing correlation-based robustness or fairness metrics, ARC provides a causally grounded approach that supports direct comparison across systems.

1.2 Research Questions

The causal model M used by ARC, is shown in Figure 1. Arrowheads indicate the causal direction from cause to effect. If Protected Attribute (Z) is a common cause of both Perturbation (P) and Outcome (O), it introduces a spurious correlation between P and O, a confounding effect, making Z the confounder. This is undesirable as it obscures the true causal impact of P on O, leading to potentially misleading conclusions about model behavior. ARC aims to address the following research questions:

RQ1: Can we identify and quantify the effect of protected attributes on the model's outcome in the absence of confounders, indicating statistical bias?

RQ2: Can we identify and quantify the effect of protected attributes on the relationship between perturbations and model outcomes, indicating confounding bias?

RQ3: Can we quantify the causal impact of perturbations on the model's outcome?

RQ4: Can the robustness of a composite system be inferred from the raw scores of its individual components (e.g., a sentiment model combined with a translator)?

RQ5: Can existing eXplainable AI (XAI) approaches complement ARC and work in tandem with it?

1.3 Contributions

Through our previous works on causality-based rating, we addressed RQ1-RQ3 for AI models across different tasks, including sentiment analysis [Lakkaraju et al., 2024b], timeseries forecasting [Lakkaraju et al., 2024a; Lakkaraju et al., 2025], binary classification, and group recommendation. The latter two are integrated into the ARC tool along with the former two, available here: http://casy.cse.sc.edu/causal rating.

2 Approach

ARC takes input data and applies domain-relevant perturbations (for e.g., introducing null values in numerical timeseries data which can happen due to incomplete transmission) to obtain predictions, which are then evaluated using causality-based metrics we introduced in [Lakkaraju et al., 2023; Lakkaraju et al., 2024b] to compute raw scores. The raw scores are used to rate the AI models under each perturbation, and ratings are assigned by grouping scores into discrete levels.

2.1 Evaluation of Approach

In [Lakkaraju et al., 2025], we conducted a user study presenting time-series model prediction errors alongside our computed ratings, which helped users more easily compare model robustness. In future work, another user study will be necessary to evaluate the effectiveness of combining our rating framework with XAI approaches. To assess the reliability of our causal analyses, we plan to use Rosenbaum sensitivity analysis [Rosenbaum, 1987] to estimate the magnitude of hidden bias needed to invalidate our findings.

2.2 Impact of Research

My research could have the following impact:

1. Enabling robustness-based comparisons of AI models, including both bias and sensitivity to perturbations, across diverse tasks, which could encourage benchmarking platforms like Hugging Face to adopt more meaningful robustness metrics.
2. Providing a practical framework for analyzing composite models, which are common in real-world applications.
3. Translating complex causal analyses into interpretable scores and ratings, allowing non-expert users to make informed decisions and supporting model selection and auditing.

3 FutureWork

We explored RQ4 stated in Section 1.2 in [Lakkaraju et al., 2023] by combining machine translation systems with sentiment analysis models. However, we did not establish a formal relationship such as an inequality or functional bound between the raw scores of individual components and the composite system. Deriving such a relationship, even approximate (e.g., bounding the composite score by the min or max of component scores), could be critical for understanding and debugging systems like chatbots built from multiple AI models. RQ5 remains an open direction. ARC shares similarities with global XAI approaches but differs in a fundamental way. Unlike traditional global XAI methods [Ribeiro et al., 2016; Sundararajan et al., 2017; Lundberg and Lee, 2017], ARC takes a causally-grounded, hypothesis-driven approach, generating raw scores based on user-defined hypotheses. For example, a user might want to test whether a specific gender causally affects the sentiment predicted by a model for sentences such as "Amanda is feeling depressed." Global XAI methods help identify features that consistently influence model predictions. When many features are present, such insights can guide ARC to prioritize which variables to test for causal impact, rather than evaluating all features equally. The ultimate goal is to position ARC as a decisionsupport framework that aligns with the six desirable criteria for effective decision aids outlined in [Miller, 2023], which are grounded in the ten cardinal decision issues proposed by [Yates and Potworowski, 2012].