Focus-LIME: Surgical Interpretation of Long-Context Large Language Models via Proxy-Based Neighborhood Selection

1 Introduction

As Large Language Models (LLMs) scale to handle massive context windows [Ding et al., 2024], they are entrusted with increasingly complex tasks, from auditing legal contracts to debugging extensive codebases. Given the opaque nature of these models, explaining their behavior has emerged as a critical area of research [Valvoda and Cotterell, 2024; Guo et al., 2024]. In particular, practitioners require explainability at varying levels of granularity [Elisha et al., 2025]. Consequently, there is a persistent demand for faithful, fine-grained explanations that can pinpoint the exact locus of model decisions, such as locating a specific important clause or word within a 50-page document. In this paper, we propose to generate surgical local explanations for LLMs by curating the neighborhood of explanations with proxy models.

To illustrate the dilemma of explaining long-context LLMs, consider the motivating example from the Contract Understanding Atticus Dataset (CUAD) [Hendrycks et al., 2021] shown in Figure 1. Here, a user queries a 6,000-word contract to identify clauses designating the specific state or national laws applicable to the agreement. On one hand, concept-based explanation methods [Ghorbani et al., 2019; Liu et al., 2024b; Ludan et al., 2024] scale effectively to long contexts by operating at a high level of abstraction. In this scenario, while they might correctly identify the model's reliance on "Jurisdictional Clause" concepts, they inherently lack the granularity to localize specific tokens, leaving the user unable to verify the exact locus of the decision. Conversely, feature-based methods like LIME [Ribeiro et al., 2016] theoretically offer this surgical precision but collapse under the curse of dimensionality. When applied to this extensive context, standard LIME suffers from severe attribution dilution: it dissipates the sampling budget across thousands of irrelevant tokens, drowning the critical signal in background noise. To generate faithful explanations, LIME would require an impractically large number of model queries, also resulting in prohibitive computational costs. Our goal is to bridge this gap by strictly curating the neighborhood, enabling the explainer to filter out semantic noise and surgically focus on the decisive evidence. Consequently, our method can precisely identify that specific words like "Governing", "Law", and "Illinois" are the key drivers of the model's decision.

∗Corresponding Author

To address the intractability of fine-grained attribution in long-context scenarios, we propose Focus-LIME, a framework designed to restore the surgical feature attribution capabilities of LIME for LLMs. Our approach fundamentally reimagines the role of efficiency-oriented techniques. While recent research [Zhao et al., 2025; Jiang et al., 2023; Liu et al., 2026] has explored using proxy models to generate explanations entirely to reduce inference costs, this strategy inherently risks producing unfaithful explanations. Acknowledging this limitation, we draw inspiration from proxy-based efficiency but shift the paradigm from "replacement" to "guidance." We employ the proxy model strictly as a "preliminary scout" for neighborhood selection, tasked with filtering out irrelevant background features. Crucially, we then perform the final perturbation analysis using the target model, but exclusively within this optimized, low-dimensional neighborhood. This "Scout then Scrutinize" strategy aims to preserve fidelity to the target model's decision boundary, while obtaining computational scalability from proxy-based methods.

We extensively evaluate Focus-LIME on long-context benchmarks, including the CUAD and Qasper [Dasigi et al., 2021] datasets. Empirical results demonstrate that our method significantly outperforms both original LIME and pure-proxy baselines in terms of explanation fidelity. Compared with curating neighborhoods using target models, Focus-LIME achieves comparable fidelity while requiring notably fewer model queries, showcasing its efficiency.

In summary, our contributions are as follows:

• We propose Focus-LIME, a model-agnostic framework that enables surgical feature-attribution interpretation for long-context LLMs by decoupling neighborhood selection (via proxy) from attribution (via target).
• We instantiate this framework by designing a coarse-tofine neighborhood contraction algorithm that effectively leverages proxy models to filter semantic noise without sacrificing target fidelity.
• We evaluate the effectiveness of our approach on longcontext benchmarks, showing that it makes surgical explanations practical and provides faithful explanations to users.

2 Background and RelatedWork

In this section, we introduce the necessary background and survey prior studies relevant to our work.

2.1 Large Language Models

A large language model can be regarded as a probabilistic function f that, given a sequence of tokens x = [x1, x2, . . . , xt], produces a probability distribution over the set of possible next tokens, written as f(x). Formally, the mapping is f : V ∗ → R|V| , where V denotes the vocabulary, and the output corresponds to a distribution over all vocabulary elements. Since most LLMs operate with an upper bound on input length n, we typically assume the sequence x ∈ Xn . Today, LLMs are used in a wide range of applications, including medical [Guo et al., 2024] and legal [Valvoda and Cotterell, 2024], which often involve processing long document inputs and require reliable explanations to ensure their trustworthiness.

2.2 Local Model-Agnostic Explanation Techniques

A model-agnostic local explanation method t takes as input a predictive model f together with a specific instance x, and produces an explanation gf,x that characterizes how the model behaves around x.

In this work, our attention is mainly on attribution-style approaches, which are the most widely used explanation form. These methods assign numerical relevance scores to the input features of x ∈ X, aiming to capture how much each feature contributes to the output f(x). The explanation gf,x can thus be represented as a vector of attributions a = [a1, a2, . . . , an], where each coefficient ai quantifies the effect of feature xi on the model's decision. Formally, we can describe an attribution-based explanation method t as a function

where F denotes the set of predictive models, Xn is the input domain, and n is the dimensionality of the instance x.

Local explanations describe the target model's behavior in a local neighborhood around a specific input [Dwivedi et al.,

2023]. The neighborhood is usually defined as a set of perturbed samples generated by perturbing the input [Ribeiro et al., 2016; Lundberg and Lee, 2017; Liu and Zhang, 2025; Dwivedi et al., 2023]. Perturbations are typically represented in a binary masking space, where each perturbed instance is encoded by a vector z ∈ {0, 1}n . Each entry zi indicates whether the corresponding feature xi of the original input x is preserved (zi = 1) or removed/replaced (zi = 0). Formally, there exists a mapping function

that reconstructs a perturbed sample in the original input space. The neighborhood N of the input x is then defined as

When processing long sentence inputs, the number of features n can be very large, leading to a large and highdimensional neighborhood. Many methods have been proposed to handle this challenge. Some methods [Santhiappan et al., 2024; Achtibat et al., 2024] use attention to narrow the neighborhood, but they require access to the internal model structure, which is not applicable to closed-source LLMs. Some other methods [Shankaranarayana and Runje, 2019; Tan et al., 2023; Yu et al., 2026] try to improve the sampling strategy, but they still provide explanations in a large neighborhood, which can lead to unfaithful explanations.

2.3 Desiderata of Explanations

In this paper, we focus on the fidelity and cost of explanations [Yeh et al., 2019; Burger et al., 2023].

On one hand, explanations should faithfully reflect the model's decision-making process. High fidelity indicates that the explanation accurately captures how the model arrives at its predictions. This ensures that users can trust the explanation and apply it to various practical scenarios, such as model debugging and bias detection.

On the other hand, when explaining large language models, the cost of generating explanations becomes a critical consideration, as querying these models can be computationally expensive. Therefore, it is essential to develop explanation methods that can deliver high-fidelity explanations while adhering to practical computational budgets. In this work, we aim to improve the fidelity of explanations without increasing the overall computational cost.

3 Methodology

In this section, we introduce the Focus-LIME framework for generating surgical explanations for long-context LLMs via proxy-based neighborhood selection.

3.1 ProblemFormulation

Following the standard setting of local model-agnostic attribution-based explanation methods [Ribeiro et al., 2016; Lundberg and Lee, 2017], let fT denote the target model and ga denote a local surrogate parameterized by the attribution vector a. The goal is to estimate the optimal attribution vector a∗ that minimizes the local approximation error L over the neighborhood:

where ϕ maps a binary perturbation mask back to the original input space, πx is the sampling distribution around x, and Ω(a) is a complexity regularizer.

The Cost-Fidelity Dilemma. The fidelity of the local explanation relies heavily on the sample density within the neighborhood. To learn a faithful linear surrogate ga, the number of perturbed samples K must scale with the feature dimension n to ensure the regression stabilizes.

However, because querying large language models incurs significant computational costs, end users often face a budget B that limits the total number of tokens that can be processed. Under a computational budget B, the maximum allowable samples Kmax = ⌊B/C(fT )⌋ is fixed, where C(fT ) denotes the average cost of evaluating one perturbed sample with the target model, measured in processed tokens. In long-context scenarios, as n grows, the sample density ρ = Kmax/n becomes vanishingly small, making the estimation of a∗ increasingly unreliable. The limited samples fail to cover the high-dimensional decision boundary, causing the variance of the estimator a∗ to explode. We term this phenomenon attribution dilution, where the resulting explanation becomes indistinguishable from noise.

Proposed Formulation. To resolve this, we introduce a computationally efficient proxy model fP for the target model fT (where C(fP ) is defined analogously and C(fP ) ≪ C(fT )) and decompose the problem into two stages. Instead of optimizing over the entire neighborhood N, we seek to identify an active neighborhood subspace S ⊆ N that contains the most information. Our objective is reformulated as:

where π′x is the sampling distribution restricted to S. The subspace S is determined by the proxy fP to filter out irrelevant dimensions (semantic noise), allowing the target model fT to focus its limited sampling budget on the critical decision boundary. In this case, the sample density ρ′ = Kmax/|S| can be maintained at a reasonable level, enabling faithful estimation of a∗focused .

3.2 The Focus-LIME Framework

To address the dilemma of vanishing sample density, we propose Focus-LIME, a framework that implements a coarseto-fine neighborhood curation strategy. The overall pipeline is illustrated in Figure 2. Our core insight is to decompose the explanation process into two distinct phases: first using a proxy model to scout the relevant neighborhood without additional target model queries, and then using the target model to scrutinize the fine-grained attributions within this focused region.

Phase I: Proxy-Guided Scouting. This phase aims to identify an Active Neighborhood Subspace S ⊂ N that contains the most relevant features, thereby filtering out background noise. We formulate Focus-LIME as a flexible framework where the scouting policy is a modular component. Depending on the nature of the proxy model and the task, various selection strategies can be employed (e.g., gradient-based saliency [Sundararajan et al., 2017], attention [Santhiappan et al., 2024; Achtibat et al., 2024], and so on). In our experiments, we instantiate this as an iterative refinement process in a fully model-agnostic manner:

1. Initialization: We initialize the candidate set C(0) as the entire input document x. Let t = 1 be the current iteration step.
2. Hierarchical Decomposition: At step t, we decompose the surviving candidates from the previous step C(t-1) into finer-grained units U(t) = {u1, u2, . . . , um}. For example, in the first iteration, we decompose the document into paragraphs; in the second, we decompose the selected paragraphs into sentences.
3. Proxy Valuation: We employ LIME to compute an attribution score for each unit in U(t) using the proxy model fP . We follow [Liu et al., 2026] to choose the proxy model.
4. Top-k Filtering: We select the top-kt units with the highest relevance scores to form the refined set C(t) , where kt is the retention budget at iteration t.
5. Termination Check: We check if the total token length of C(t) satisfies the density constraint or a user-defined stop condition (e.g., maximum number of iterations). If the constraint is met, the iteration terminates, and C(t) is converted into the final binary Focus Mask mfocus. Otherwise, we increment t and repeat from Step 2.

Through this iterative pruning, Focus-LIME effectively "zooms in" on the decision boundary, discarding massive amounts of irrelevant context.

Phase II: Target-Based Scrutinizing. Once the iterative scouting terminates, we obtain a converged Focus Mask mfocus ∈ {0, 1}n , which delineates the important region from the irrelevant context. In this phase, we engage the target model fT to perform fine-grained attribution. Unlike standard LIME, which suffers from dilution by sampling uniformly across the entire high-dimensional space, FocusLIME restricts the perturbation logic as follows:

1. Constrained Perturbation: We generate a set of K perturbed samples Z ′ = {z′1 , . . . , z′K } using a MaskConditioned Generator. For each sample z′ ∈ {0, 1}n , the value of the i-th token z′i is determined by:

This strategy ensures that the irrelevant context remains static, while the perturbation is concentrated exclusively within the active region identified using the proxy model.

1. Target Valuation: We query the expensive target model fT with these constrained samples to obtain the prediction vector y′ . Crucially, because the dimensionality of the perturbation space has been reduced from n (total tokens) to nactive, where nactive := ∑ni=1 mfocus[i], the sample density ρ = K/nactive is restored to a high level, ensuring the stability of the subsequent regression.
2. Explanation Generation: Finally, following the standard LIME procedure, we fit a weighted linear regression model to approximate the target model's behavior within the focused neighborhood.

where πS(z′ ) is the kernel weight within the focused neighborhood, L is the same local fitting loss used in Eq. 1, and only the weights corresponding to the active region (mfocus[i] = 1) are optimized. The resulting attribution vector w∗ provides a surgical explanation: it identifies the precise tokens that most significantly influenced the target model's prediction within the longcontext input.

3.3 Summary

In summary, we presented Focus-LIME, a framework designed to overcome the intractability of explaining longcontext LLMs. By recognizing that not all tokens in a massive document require equal scrutiny, we reformulated the local explanation problem as a two-stage process:

• Scout (Phase I): Utilizing a cost-efficient proxy model to iteratively filter out semantic noise and identify a lowdimensional Active Neighborhood.
• Scrutinize (Phase II): Leveraging the target model to perform surgical attribution exclusively within this optimized subspace.

This approach resolves the fundamental conflict between attribution dilution and computational budget in existing model-agnostic feature-attribution methods. This ensures that the final explanation achieves high fidelity to the target model's behavior while keeping the inference cost manageable.

4 Empirical Study

In this chapter, we present a comprehensive empirical evaluation of Focus-LIME on challenging long-context benchmarks. Specifically, we aim to answer the following research questions:

• RQ1 (Faithfulness): Does Focus-LIME generate more faithful explanations compared to original methods (e.g., LIME) and pure-proxy baselines in long-context scenarios?
• RQ2 (Mechanism Verification): Does the neighborhood curation process mitigate attribution dilution, thereby improving explanation fidelity?
• RQ3 (Human Alignment): Does Focus-LIME effectively align with human reasoning by accurately localizing the ground-truth evidence spans annotated by experts?

4.1 RQ1: Faithfulness Evaluation

Experimental Setup

Datasets. We use two long-context document QA datasets in our experiments:

• CUAD [Hendrycks et al., 2021] is a legal document QA dataset that requires models to answer questions based on long legal contracts.
• Qasper [Dasigi et al., 2021] is a long-context QA dataset that contains scientific papers.

We use the yes/no question answering tasks in both datasets for evaluation, retaining examples whose input context exceeds 5,000 tokens. Considering the cost of target-model queries, we evaluate on a subset of 500 examples from each dataset.

Models. We evaluate our method on different combinations of target and proxy models:

• Target Models (fT ): We use GPT-4o [Achiam et al., 2023] and DeepSeek V3 [Liu et al., 2024a] as target models.
• Proxy Models (fP ): After the screening process, we use Qwen3-30B-A3B [Yang et al., 2025a] and Qwen2.532B [Yang et al., 2025b] as proxy models for all target LLMs; these models can run on consumer-grade GPUs.

Baselines. We compare Focus-LIME against the following baselines:

• Original LIME: Applies LIME directly to the full document. For each test, we sample 1000 perturbed samples.
• Proxy Explanation: Following [Liu et al., 2026], this baseline uses the proxy model to generate explanations. Because the proxy model is much cheaper than the target model, we sample 10,000 perturbed samples for each test.
• Focus-LIME (w/o Proxy Model): An ablation of Focus-LIME that skips the proxy-based neighborhood curation step and directly applies constrained perturbation to the full document.

Metrics. To evaluate the faithfulness of explanations, we conduct deletion experiments [Ribeiro et al., 2016; Lundberg and Lee, 2017; Sun et al., 2023], which measure the change in model output when the most important features (as identified by the explanation) are removed. We use Area Over most relevant first perturbation curve (AOPC) as the metric. Specifically, AOPC is defined as:

where pf(y|x) is the probability that f outputs y given the input x, T is the set of all test inputs, and x(k) is the input x with the top k most important features removed. A higher AOPCk indicates a better explanation. We report AOPC10, AOPC50, and AOPC100 and AOPC := ∑100 k=1 AOP Ck 100 in our experiments.

Results

Tables 1 and 2 present the faithfulness evaluation results on the CUAD and Qasper datasets, respectively. We observe that in the absence of neighborhood curation, both Standard LIME and Pure Proxy baselines struggle significantly in long-context scenarios. Specifically, they yield negligible AOPC scores, failing to identify truly decisive features due to severe attribution dilution. In contrast, by incorporating the neighborhood curation process, Focus-LIME restores the tractability of the explanation task, making it practical and useful by substantially enhancing fidelity.

Furthermore, Focus-LIME has performance comparable to the ablation without the proxy model, which indicates that using proxy models for neighborhood curation does not sacrifice explanation fidelity. Additionally, Focus-LIME achieves similar performance with different proxy models (Qwen330B-A3B and Qwen2.5-32B), indicating that our method is robust to the choice of proxy model.

4.2 RQ2: MechanismVerification

While RQ1 has demonstrated the superior faithfulness of Focus-LIME on long-context benchmarks, we further investigate the underlying mechanism by verifying our hypothesis that narrowing the neighborhood can improve the fidelity of local explanations in general scenarios. This analysis examines whether users can also use this strategy to improve explanation fidelity beyond long-context scenarios.

Experimental Setup. We conducted an experiment on a text classification dataset: the Large Movie Review Dataset (IMDb) [Maas et al., 2011], which contains movie reviews with an average length of 228 words. We use Qwen3-235BA22B as the target model, and use Qwen3-30B-A3B as the proxy model to optimize the neighborhood. We optimize the neighborhood by greedily fixing the least important feature according to the explanation generated by LIME in each iteration, and evaluate the fidelity of explanations along the path of narrowing the neighborhood. We use AOPC as the fidelity metric, and compare the fidelity of explanations on the original neighborhood and the optimal neighborhood found on the path of narrowing the neighborhood.

Results. Figure 3 shows the deletion fidelity (AOPC) of explanations on the original and optimal neighborhoods when deleting the top 1 to 100 most important features. On average, using the optimal neighborhood improves fidelity by 11.4% relative to the original neighborhood. When deleting the 10 most important features, using the optimal neighborhood improves fidelity by 32.2% relative to the original neighborhood.

Moreover, Figure 4 provides a more fine-grained analysis of the fidelity improvement. Figure 4(a) provides an overview of the relationship between fidelity improvement, the number of deleted features, and the percentage of examples. Specifically, Figure 4(b) shows that over 20% of the examples can achieve more than 10% relative improvement in fidelity by using the optimal neighborhood across different deletion ratios, and when deleting less than 10 features, over 50% of the examples can achieve more than 10% relative improvement in fidelity. Figure 4(c) further shows that using the optimal neighborhood can significantly improve fidelity, especially when deleting a small number of features.

4.3 RQ3: Alignment with Human Evidence

RQ3 investigates the practical utility of Focus-LIME: can it serve as a reliable tool for human users to locate critical information? To answer this, we utilize the ground-truth annotations provided in the CUAD dataset. A high-quality explanation for a robust model should align with these human annotations.

Quantitative Analysis: Evidence Localization. We treat the top-k words identified by the explainer as a "retrieval set" and evaluate their overlap with the human ground truth using Recall. As the length of human-annotated evidence varies across examples, we choose k as a relative ratio of the evidence length, specifically 100%, 150%, or 200% of the total number of words in the human-annotated evidence. In this experiment, we follow the setting of RQ1, using GPT-4o as the target model and Qwen3-30B-A3B and Qwen2.5-32B as proxy models.

As shown in Table 3, our methods can effectively localize the ground-truth evidence spans annotated by experts. In Figure 5, we present an example where Focus-LIME successfully identifies the critical clauses that determine the model's decision, demonstrating its practical utility in assisting users as they navigate and verify long documents.

5 Conclusion

In this paper, we introduced Focus-LIME, a framework designed to resolve the critical challenge of surgical word-level attribution dilution in long-context LLMs. By leveraging a cost-efficient proxy to rigorously curate the search space, our method restores the tractability of surgical interpretation, allowing the target model to focus its sampling budget on the most relevant subspace. Extensive experiments demonstrate that Focus-LIME makes word-level explanations for longcontext LLMs practical and has the potential to improve explanation fidelity in more general settings.

Ethical Statement

This work presents an interpretability method that is generally not harmful, though its explanations could potentially be misused to probe model weaknesses or support adversarial attacks.

Acknowledgments

This work was sponsored by the National Natural Science Foundation of China (NSFC) under Grant No. W2411051.