Neurosymbolic Framework for Concept-Driven Logical Reasoning in Skeleton-Based Human Action Recognition

1 Introduction

Skeleton-based human activity recognition (HAR) provides an efficient, privacy-preserving alternative to video-based methods by focusing on joint coordinates that isolate human structure from environmental noise [Duan et al., 2022;

* Corresponding author, now at School of Computing Technologies, RMIT

† Extended version of this paper can be found at: Arxiv

Liu et al., 2023; Asif et al., 2023]. While deep learning has advanced classification accuracy, these models remain black boxes, problematic for high-stakes domains such as healthcare and rehabilitation where transparent decision-making is essential [Ilyas et al., 2025; Mehta et al., 2023].

Neurosymbolic AI has gained traction for combining neural perception with formal logic interpretability [Garcez et al., 2002; Bhuyan et al., 2024]. However, its application to skeleton-based HAR remains largely unexplored due to fundamental integration challenges. This work addresses these challenges to achieve interpretable skeleton-based HAR.

Symbol Grounding. A key challenge is bridging continuous joint trajectories and discrete symbolic predicates. Traditional joint heuristics (e.g., knee angle < 90o for bent knee) [Okamoto and Parmar, 2024] are brittle, as small pose errors or variations in viewpoint and execution can prevent predicate activation for reliable HAR. Inspired by Concept Bottleneck Models (CBMs) [Koh et al., 2020], we instead learn abstract motion concepts and construct an LLM-generated concept bank of spatial primitives (e.g., arm raise, knee bend) that serve as symbolic predicates. We also enforce semantic grounding by aligning GCN-extracted skeleton features with pretrained text encoder based concept embeddings.

Temporal Dynamics. Actions are inherently temporal, varying in speed, duration, and execution order. WEARINGGLASSES and TAKEOFFGLASSES share similar static poses (hands near the face, arm engagement) but differ in temporal dynamics: the former involves grasping and upward motion, while the latter requires release and downward motion. Purely spatial logic cannot capture such distinctions, and incorporating temporal logical rules into neurosymbolic frameworks is challenging. To address this, we introduce explicit temporal concepts capturing motion direction (e.g., motion upward, motion downward), hand state transitions (e.g., hand grasp, hand release), and execution order (e.g., seq hands first). Our Spatio-Temporal Concept Decoder (STC-Decoder) then independently decodes spatial and temporal concepts, enabling distingushing semantically similar actions such as (arm raise ∧ hand grasp ∧ motion upward) ⇒ WEARGLASSES versus (arm lower ∧ hand release ∧ motion downward) ⇒ TAKEOFFGLASSES (Fig 1(b)).

End-to-End Learning. Discrete Boolean operators block gradient propagation, preventing joint optimization of perception and reasoning components [Schrouff et al., 2021]. While differentiable fuzzy logic [Barbiero et al., 2023] shows promise, scalability to large-scale, fine-grained skeletonbased HAR remains open. We employ differentiable logic layers with soft AND/OR operations [Perfilieva, 2002] and gradient grafting [Wang et al., 2023], enabling discrete inference with end-to-end differentiability. Final predictions aggregate triggered rules via learned weights, defining actions as logical compositions of motion concepts.

Together, these components form an end-to-end neurosymbolic framework (Fig 1(b)) in which a GCN encoder extracts spatio-temporal features with sequence-concept alignment, the STC-Decoder predicts interpretable concepts, and differentiable logic layers compose them via first-order operations (∧, ∨, ¬) to derive action predictions for action recognition. Grounding predictions in logical rules over concepts provides both local (concepts) and global interpretability (rules). Experiments on NTU RGB+D 60/120 [Shahroudy et al., 2016; Liu et al., 2019] and NW-UCLA [Wang et al., 2014] demonstrate we achieve competitive performance with transparent explanations. To summarize, our major contributions are:

• REASON (Rule-based Explainable Action via Symbolic cONcepts), a neurosymbolic framework for skeleton-based HAR that formulates action recognition as logical reasoning over learnable motion concepts and integrates concept-grounded representations with differentiable first-order logic for end-to-end training.
• A Spatio-Temporal Concept Decoder (STC-Decoder) that extracts spatial body-configuration and temporal motion-dynamics concepts, yielding interpretable abstractions robust to execution variability.
• A concept bank of spatial and dynamic concepts for actions available in NTU RGB+D and NW-UCLA.

2 RelatedWork

Skeleton-based HAR. ST-GCN [Yan et al., 2018] established the GCN paradigm by modeling joints as graph nodes with spatio-temporal convolutions. Subsequent work (Fig 1(a)-(i),(ii)) addresses topology rigidity: 2s-AGCN [Shi et al., 2019] learns adaptive joint relationships, CTRGCN [Chen et al., 2021] refines topology channel-wise, and InfoGCN [Chi et al., 2022] incorporates informationtheoretic objectives. Recent architectures like HD-GCN [Lee et al., 2023] and BlockGCN [Zhou et al., 2024] capture hierarchical interactions. Self-supervised approaches (Fig 1(a)- (iv)) including SkeletonMAE [Wu et al., 2023] and MotionBERT [Zhu et al., 2023] improve generalization via masked reconstruction. Despite strong performance, these models lack interpretable action semantics.

Vision-Language for HAR. Recent works have used LLMs for skeleton-based HAR (Fig 1(a)-(iii),(iv)). SUGAR [Ye et al., 2025] and LLM-AR [Qu et al., 2024] leverage LLM priors for classification or incorporate action prompts [Xiang et al., 2023]. However, these methods yield generic, noninstance-specific explanations and cannot distinguish execution variants (e.g., arms-first vs. legs-first), and are impractical for edge deployment. In contrast, our framework learns efficient, compositional logical rules over motion predicates that capture multiple execution pathways.

Explainable HAR and Neurosymbolic AI. Prior interpretable HAR work has focused on RGB video via post-hoc concept attribution [Kowal et al., 2024] or neurosymbolic pipelines with expert-defined rules [Okamoto and Parmar, 2024], inheriting privacy concerns and sensitivity to environmental noise. While neurosymbolic systems enable traceable reasoning [Hitzler and Sarker, 2022], skeleton data exhibit substantial subject and viewpoint variability, making handcrafted rules brittle. We address this with learnable concept predicates combined with differentiable logic layers to discover action-defining rules from data.

Concept Bottleneck Models. CBMs constrain representations to interpretable concept layers, enabling intervention and debugging [Koh et al., 2020], with extensions like Logic CBMs [Vemuri et al., 2025] incorporating differentiable logic for concept compositions. Recent work further automates concept bank construction using LLM priors [Mehta et al., 2025; Jiang et al., 2025a], reducing manual effort. Despite success in static image classification tasks [Jiang et al., 2025b], CBMs have not addressed skeleton-based HAR, which requires joint modeling of spatial and temporal dynamics. We bridge this gap by formulating skeleton-based HAR as logical reasoning over learnable motion concepts.

3 Method

Our work centres on the question: Can concepts serve as an intermediate representation for neurosymbolic skeletonbased HAR? We address this by first constructing a concept bank of spatial and temporal motion primitives (Sec. 3.1), and then developing a framework to predict concepts from skeleton sequences (Sec. 3.2& 3.3) and composing them via differentiable logic layers (Sec. 3.4& 3.5) for interpretable HAR.

3.1 Concept Bank Generation

Unlike image classification where concepts correspond to static attributes (e.g., a zebra has stripes), action recognition requires capturing spatio-temporal dynamics. Furthermore, execution variability within an action (e.g., armassisted v/s arm-free JUMP) makes manual concept enumeration infeasible. We address this by construcing a concept bank C = Cs ∪ Ct ∪ Cint of spatial, temporal, and interactive motion concepts through a three-stage LLM-based pipeline.

Fine-grained Action Description Generation

Following HAKE [Li et al., 2022], we decompose actions into six body parts: P = {head, hand, arm, hip, leg, foot}. We then prompt an LLM (GPT-3) to generate part-wise motion descriptions Da = {d(p) a }p∈P for each action a. For example, JUMP yields: {Leg: bends at knees then pushes explosively; Arm:

extends overhead; Foot: pushes off ground}.

From Action Descriptions to Abstract Concepts

Generated raw descriptions are action-specific. Thus, using them directly could cause vocabulary explosion [Jiang et al., 2025a]. Hence, we convert descriptions into canonical concepts by extracting {verb-direction-modifier} patterns from all actions, then embedding them via a sentence encoder ϕ, and clustering them with k-means. Cluster representatives form spatial concepts Cs (e.g., arm raise, leg jump, hand grasp). Additionally, we define temporal concepts Ct capturing motion direction, sequence, and dynamics (e.g., motion upward, hand first, speed fast), and an interaction concept for mutual actions.

Concept-Action Association and Refinement

We construct an association matrix M ∈ {0, 1}|A|×|C| where Ma,c = 1 indicates concept c is active for action a. Using LLaMA-3.2 7B [Grattafiori et al., 2024], we match each action's descriptions to semantically appropriate concepts, allowing multiple concepts per body part for compositional movements (e.g., JUMP activates both leg squat and leg jump). To ensure discriminability, we verify unique concept signatures and invoke refinement for ambiguous pairs (e.g., WEARGLASSES vs. TAKEOFFGLASSES are distinguished via opposing direction concepts, see Figure 1). Our final concept bank comprises |C| = 74 concepts (52 spatial + 21 temporal + 1 interaction). Implementation details, prompts, pseudocode and the complete vocabulary are provided in Appendix A.

3.2 Skeleton Representation Learning

After constructing the concept bank, we describe the overall framework (Fig 2(a)). Skeleton sequences are first encoded using a GCN and aligned with concept semantics via a concept-grounded text encoder and a contrastive objective.

Skeleton Encoder Let X ∈ RT×V×3 denote an input skeleton sequence with T frames, V joints and (x,y,z) coordinates. We model the skeleton as a spatio-temporal graph and process it through a GCN backbone (e.g., Hyper-GCN [Zhou et al., 2025]) with multi-scale temporal convolutions. Crucially, we retain full spatio-temporal resolution without pooling, outputting features F ∈ RT×V×D to preserve finegrained motion information for concept prediction.

The encoder feeds into three branches (Fig 2(a)): (B1) the STC-Decoder for concept prediction, (B2) a text alignment branch where pooled features ¯F = Pool(F) are aligned with concept embeddings, and (B3) a part-divergence branch promoting discriminative representations across anatomical regions.

Body-Part Feature Divergence We partition joints into subsets {Vp}p∈P and extract part-specific features Fp ∈ RT×|Vp|×D . To prevent feature collapse across body parts, we introduce a divergence loss that penalizes high cosine similarity between pooled part representations. Let ¯fp = Pool(Fp) ∈ RD denote the feature of part (p). The loss minimizes off-diagonal similarities across all part pairs:

This regularization encourages each body part to learn discriminative features aligned with its corresponding concept subset Cp ⊂ Cspatial.

Concept-Grounded Text Encoder To ground skeleton features in semantic space, we encode textual descriptions using CLIP [Radford et al., 2021] with LoRA fine-tuning [Hu et al., 2022]. Crucially, rather than generic action labels, we construct concept-grounded prompts incorporating motion primitives from our concept bank. For action a with associated concepts {ci} from matrix M, we construct: "A person performing [action], where the [part1] [concept1], the [part2] [concept2], ...". For example, JUMP yields: "A person performing JUMP, where the arms swing upward, the knees bend, and the body moves upward." This formulation ensures text embeddings ea = ftext(τa) encode the same compositional semantics as the concept vocabulary.

We then align skeleton and text representations via contrastive learning. Projected skeleton features z = hskel(¯F) and concept embeddings t = htext(ea) are aligned through:

where τ is a learnable temperature. This objective encourages skeleton features to cluster with their concept-grounded descriptions, reinforcing the compositional structure.

3.3 STC-Decoder

The spatio-temporal concept decoder (STC-Decoder, Fig. 2b) maps continuous skeleton sequences to discrete concept activations, bridging skeleton representations and symbolic predicates. As actions activate multiple concepts, we adopt a multi-label design with separate spatial and temporal pathways.

Spatial-Temporal Feature Decoupling. Given encoder features F ∈ RT×V ×D , we decouple them through dimensionwise aggregation:

Spatial features Fs preserve per-joint information by aggregating across time, while temporal features Ft retain motion dynamics by aggregating across joints.

Group-Decoding with Concept Queries. Predicting all |C| concepts jointly poses an optimization challenge: concepts exhibit heterogeneous semantic relationships, some are mutually exclusive (motion upward vs. motion downward), others frequently co-occur (arm raise, hand grasp), and many are conditionally dependent on body-part context. Learning these complex inter-concept relationships in a single framework is difficult.

We address this by partitioning concepts into G groups, where each group predicts g = ⌈|C|/G⌉ concepts. Rather than manually assigning semantic roles, we let each group learn to specialize through end-to-end training, groups naturally discover coherent concept subsets (e.g., related bodypart configurations or temporal patterns) that exhibit homogeneous co-occurrence, making each sub-problem easier to optimize.

We initialize separate learnable query matrices Qs ∈ RGs×D and Qt ∈ RGt×D for spatial and temporal branches. Each branch performs cross-attention followed by a feedforward network, where group queries learn to selectively aggregate skeleton information relevant to their respective concept subsets (Appendix C).

Group Fully-Connected Layer. A Group-FC layer [Ridnik et al., 2023] projects the refined group queries to per-concept predictions. Each group k ∈ {0, . . . , G-1} maintains its own projection Wk ∈ RD×g , responsible for predicting g concepts. For concept i, we determine its group index k = ⌊i/g⌋ and position within the group j = i mod g, then compute:

This factorization encourages each group query to specialize in its concept subset through gradient-driven learning. Aggregating spatial and temporal logits yields:

where ˆ ci denotes the predicted probability of concept ci, providing soft symbolic grounding for the logic layers.

Concept Prediction Loss. Concept prediction is supervised using the action-concept matrixM. For an action label a, the target concept vector is c∗ = M[a, :] ∈ {0, 1}|C| , and training minimizes binary cross-entropy Lconcept = BCE(c∗ , ˆc). This supervision grounds continuous skeleton features into interpretable concept activations ˆc, which serve as symbolic predicates for logical composition.

3.4 Differentiable First-Order Logic Reasoning

Symbolic reasoning requires binary predicates, whereas our predicted concepts ˆc ∈ [0, 1]|C| provide soft activations over the motion vocabulary. We therefore binarize concept predictions and compose them using first-order logic, while also employing continuous relaxations to enable end-to-end learning and preserve discrete, interpretable rules.

Concept Binarization

Concept leakage [Mahinpei et al., 2021] occurs when downstream layers exploit spurious information in soft activations rather than intended semantics. We therefore employ a concept activation binarizer which applies a threshold ¯ ci = ⊮[ˆ ci > 0.5], yielding ¯c ∈ {0, 1}|C| . Furthermore, to enable logical negation of concepts, we augment the predicate space as:

allowing rules involving both presence and absence of concepts (e.g., leg jump ∧ ¬leg static).

Logical Layer Architecture

We compose concept predicates into action-discriminative rules through stacked logical layers (Fig. 2(c)). Each layer N (l) contains two sub-layers: conjunction (AND) and disjunction (OR), enabling arbitrary propositional formulas in disjunctive normal form [Wang et al., 2023]. Let n(l-1) denote the input (with n(0) = p). The conjunction (A(l) ) and disjunction layers (B(l) ) compute:

whereW(∧) ,W(∨) ∈ {0, 1} are learnable switchboards (i.e., binary adjacency matrices) specifying which predicates participate in each conjunction/disjunction. By stacking L layers with outputs n(l) = [a(l) ; b(l) ], the network learns compositional rules of increasing complexity, yielding rule activations r ∈ {0, 1}R.

Continuous Relaxation and Gradient Grafting

Discrete Boolean operations in logical layer produce zero gradients. For end-to-end training, we adopt differentiable logic relaxations [Barbiero et al., 2023] that approximate AND/OR via product-based formulations. During training, we maintain continuous weights ˜W(∧) , ˜W(∨) ∈ [0, 1] and compute soft operations:

When weights and inputs are binary, these reduce to standard AND/OR (Appendix D). Logical layer training employs gradient grafting [Wang et al., 2023] (Fig 2(a)): the forward pass uses binarized concepts ¯c and discrete weights W = ⊮[ ˜W > 0.5] for interpretable rule evaluation, while backpropagation computes gradients through the continuous relaxations (Eq. 8and 9) to update soft weights ˜W. For the concept binarizer, the Straight-Through Estimator [Bengio et al., 2013] passes gradients unchanged: ∂L/∂ˆ ci ≈ ∂L/∂¯ ci , as shown in Fig 2(a).

3.5 Rule-Based Action Classification

The rule activations r ∈ {0, 1}R encode which logical concept combinations are present. A linear classifier maps these to action predictions ˆy = Vr + b, where V ∈ R|A|×R, associates rules with actions, and b ∈ R|A| is a bias term. Post-training, we extract human-readable rules by tracing adjacency matrices back to concept combinations, yielding expressions like r1 : leg jump ∧ ¬arm swing ∧ motion upward. This ensures both local interpretability (instance-specific concept activations) and global interpretability (logical rules characterizing each action class). The details for post-training rule interpretation are provided in Appendix D.

Overall Training Objective The framework is trained endto-end by combining all loss terms: L = Ltask + αLconcept + βLalign +γLdiv +λ∥ ˜W∥1, where the ℓ1 regularization on ˜W enforces sparse, interpretable rules. At inference, concepts are binarized, rules are evaluated discretely, and actions are predicted via triggered rules.

4 Experiments

We evaluate our neurosymbolic framework on standard skeleton-based HAR benchmarks, focusing on: (i) performance competitiveness, (ii) interpretability of learned rules, and (iii) component-wise contributions.

4.1 Datasets

We evaluate on three benchmarks: NTU RGB+D 60 (56,880 sequences, 60 classes) under Cross-Subject (X-Sub) and Cross-View (X-View) protocols; NTU RGB+D 120 (114,480 sequences, 120 classes) under Cross-Subject (X-Sub) and Cross-Setup (X-Set) protocols; and NW-UCLA (1,494 sequences, 10 classes) following protocol to train on cameras 1&2, test on camera 3.

4.2 Implementation Details

We follow [Chen et al., 2021] for preprocessing, using random rotation, spatial flip, and temporal cropping with uniform 64-frame sampling. Training uses AdamW for 400 epochs with cosine decay on a single A6000 GPU.

We employ staggered warmup to prevent spurious rule formation: the GCN/CLIP/STC-Decoder warm up for 5 epochs while logic layers remain frozen for 15 epochs. This delay ensures logic layers learn from converged concept predictions rather than random activations. Logic layers use 10× higher learning rate (10-4 vs. 10-5 ) to traverse from dense initialization to sparse binary connectivity against ℓ1 regularization. Full hyperparameters are in Appendix F.

4.3 Comparison with State-of-the-Art

Table 1 compares against GCN-based, Transformer-based, and LLM-augmented methods. Beyond performance, we highlight whether methods require two-stage training (separate pretraining) and whether they provide interpretable predictions.

Our core neurosymbolic framework (REASON) achieves competitive performance while being the only single-stage, fully interpretable approach. In 4-stream configuration, we attain 94.3% on NTU-60 X-Sub and 92.8% on NTU-120 X-Set, surpassing black-box methods (Hyper-GCN, BlockGCN). Critically, the concept-rule reasoning chain remains fully auditable, distinguishing our approach from prior work. LLM Re-ranking (Optional). To enable fair comparison with LLM-augmented methods (SUGAR, LLM-AR), we evaluate an optional re-ranking stage. Unlike these methods that use generic action descriptions, we ground a LoRAfinetuned LLaMA-3.2 7B in instance-specific concept activations and top-K classifier candidates (Appendix G). With LLM re-ranking, we achieve 95.6% (X-Sub) and 93.2% (XSet), outperforming two-stage LLM methods, as we reason over interpretable concepts rather than opaque features.

Interpretable Reasoning via Concept-Rule Chains

Beyond accuracy, our framework provides transparent reasoning through learned concept-rule chains. Figure 3 visualizes the complete reasoning pipeline for representative actions. Skeleton poses are rendered with joint sizes scaled by concept activation confidence, revealing which body parts drive predictions. For BRUSHINGTEETH, the model activates intuitive concepts (hand lift to face, duration sustained, rhythm regular) and applies learned rules such as r1 : duration sustained ∧ hand grasp ∧ ¬head static (+0.37). This end-to-end transparency, from skeleton input to concept activations to logical rule firings, enables systematic analysis of both correct classifications and failure cases, a capability absent in existing methods.

4.4 Ablation Studies

We conduct ablations on NTU RGB+D 120 (Table 2). Backbone Generalization (Table 2a). Our framework is encoder-agnostic: across ST-GCN, 2s-AGCN, CTR-GCN, ST-GCN++, and Hyper-GCN, our approach consistently improves performance, demonstrating plug-and-play applicability for existing GCN encoders.

Component Analysis (Table 2b). Each module contributes incrementally: text alignment (Lalign) adds +0.9-1.2%, part divergence (Ldiv) +0.4-1.0%, and STC-Decoder +0.2-1.6%. STC-Decoder Configuration (Table 2c). Group-decoding with Gs = 8, Gt = 4 achieves optimal accuracy while cutting cross-attention cost ∼6× vs. per-concept queries (G = 74). Concept Vocabulary (Table 2d). Spatial concepts alone achieve 78.39%; adding temporal concepts yields +7.8% (86.15%), demonstrating that motion dynamics are critical for distinguishing actions with similar poses. Interaction concepts provide a further +1.2%.

Logic Layer Capacity (Table 2e). The 128-node configuration is optimal; smaller (64) lacks expressiveness while larger (256/512) overfits. Crucially, disabling negation causes catastrophic failure (49.87%), many actions require ¬concepts (e.g., STANDING needs ¬leg walk). Skip connections contribute +3.6% by preserving atomic concept access.

Concept Intervention (Table 2f). Replacing mispredicted concepts ˆ ci with ground-truth c∗i improves accuracy monotonically: +1.9% (1 concept), +3.9% (2), +5.7% (3). This confirms that logic layers capture faithful concept-action relationships and errors stem from concept misprediction rather than flawed rules, enabling human-in-the-loop correction. Overall process of concept intervention is in Appendix H.

Feature Space Analysis Figure 4 compares t-SNE embeddings of Hyper-GCN features and STC-Decoder concept activations. Actions overlapping in the Hyper-GCN space (e.g., READING/WRITING, PUTON/TAKEOFFSHOE, THUMBUP/DOWN) are well separated in the concept space by discriminative temporal concepts (e.g., motion forward vs. motion backward), with similar separation for mutual actions (e.g., EXCHANGINGTHINGS, SHAKINGHANDS).

(a) Backbone ablation

(e) Logic layer capacity

Figure 4: t-SNE visualization on NTU RGB+D 120. Hyper-GCN features (top) and our STC-decoder features (bottom). (a) Semantically similar action pairs. (b) Two-person interaction actions.

5 Conclusion

In this work, we presented REASON, a neurosymbolic framework that formulates skeleton-based HAR as logical reasoning over learnable motion concepts. Through concept grounding, spatio-temporal decoding, and differentiable logic, REASON achieves state-of-the-art performance while maintaining fully transparent concept-rule reasoning chains, establishing neurosymbolic learning as a promising paradigm for interpretable skeleton-based HAR.