Human-Readable Neuro-Fuzzy Networks from Frequent Yet Discernible Patterns in Reward-Based Environments

1 Introduction

Reinforcement Learning (RL) balances online exploration and exploitation to tackle reward-based environments. If exploration is prohibited, offline RL may train an agent to maximize an expected reward using only past data [Levine et al., 2020]. Inducing policies represented by Deep Neural Networks (DNNs) with RL for complex environments has achieved great success, but suffers from a lack of transparency and sample inefficiency [Casillas et al., 2013]. Fuzzy RL tackles the lack of transparency and sample inefficiency by "bootstrapping" policies with written instructions from an expert (i.e., a "warm start") via IF-THEN rules describing an approximate/imprecise causality called fuzzy logic rules [Lee, 1990]. Neural systems incorporating this a priori expert symbolic knowledge are a type of neuro-symbolic network called Neuro-Fuzzy Networks (NFNs). While the NFNs are more transparent and sample efficient, automatically building them remains an ongoing research endeavor [Aghaeipoor and Javidi, 2019]. This challenge is further exacerbated by the potential conflict between achieving interpretability and ensuring accuracy in NFNs within complex domains [Casillas et al., 2013]; further compounding the difficulty in creating NFNs is the well-known symbol grounding problem [Harnad, 1990] found in symbolic reasoning - the issue of connecting [human] language (the symbols) to percepts [Mooney, 2008].

Our prior systematic design process (in Section 3) selforganizes NFNs using unsupervised methods and is capable of offline, model-free Fuzzy RL [Hostetter et al., 2023b]. This work successfully addressed a common challenge in NFN design by preventing the fuzzy logic rules from growing linearly with the available training data. Additionally, it enabled human-in-the-loop interaction with the agent's transparent knowledge base facilitated by the NFN.

However, two major flaws exist in our prior work [Hostetter et al., 2023b]: (1) the meaning of fuzzy logic rules' conditions cannot be adjusted during training, and (2) the number of conditions in each fuzzy logic rule grows linearly with respect to the environment's attributes. Issue (1) limits the NFN's potential for increasing the fuzzy logic rules' effectiveness, and issue (2) detrimentally affects the readability of fuzzy logic rules as the number of input attributes increases.

Our proposed work in this paper addresses issues (1) and (2) by yielding a simpler and more robust linguistic fuzzy rule base (i.e., a knowledge base) that maintains global semantics [Casillas et al., 2013] without harming performance. Before fuzzy logic rule simplification, we adjust the symbols' meaning by ensuring the NFNs' internal fuzzy representations are capable of recovering the original percepts to mitigate issue (1). Then, fuzzy logic rules are simplified in a general approach compatible with Fuzzy RL by leveraging fuzzy information granulation [Zadeh, 1997] and extending social network analysis to retain only frequent-yet-discernible (FYD) patterns; this addresses issue (2).

To assess how effectively FYD addresses the two major issues, we compare NFNs simplified by FYD to DNNs and our prior work [Hostetter et al., 2023b] in classic control tasks. Then, we empirically investigate its effectiveness in realworld higher-dimensional settings by teaching students probability principles within an intelligent tutoring system (ITS) against these same controls as well as Efficient CLAuse-wIse Rule Extraction (ECLAIRE) - an efficient, polynomial-time rule extraction algorithm to decompose large DNNs into rulebased models [Zarlenga et al., 2021]. Sample code demonstrating FYD is public (MIT license) [Hostetter, 2025]. 1

2 Background

Fuzzy logic (abbrev. "f.ℓ."), f.ℓ. rules, linguistic variables, and their linguistic terms are all by-products of fuzzy set theory [Zadeh, 1965] - the mathematical study of an uncertainty called impreciseness [Klir and Yuan, 1995]. Often, fuzzy set theory is mistakenly compared to probability theory, but they handle different types of uncertainty - in fact, they may complement one another [Zadeh and Aliev, 2018].

1 https://github.com/johnHostetter/IJCAI-2025-FYD

For a fuzzy set, element membership, µ, is [typically] between 0 and 1 [Klir and Yuan, 1995] - unlike "traditional" sets, where an element either belongs or it does not. Fuzzy sets more accurately reflect the nuances and spectrum to which vague or imprecise symbols/descriptions may apply to an element. For example, understanding the nuance/degree to which a person belongs to "the set of tall people" is often overlooked or ignored in set theories assuming bivalence. Here, we determine elements' membership to fuzzy sets by using the Gaussian membership function for two reasons: (1) it easily allows gradient descent [Tung et al., 2011], and (2) NFNs that use Gaussians are fuzzy basis functions capable of universal function approximation as proven with the StoneWeierstrass theorem [Wang and Mendel, 1992a].

Constrained fuzzy sets with semantics are linguistic terms, and linguistic variables can only take on values that are linguistic terms [Klir and Yuan, 1995]. An implication between the assignment of linguistic variable(s) to some linguistic term(s) (i.e., premises) and other linguistic variable(s) assigned to some other linguistic term(s) (i.e., consequents) is a f.ℓ. rule. Degree of applicability/activation of a f.ℓ. rule depends on the information's relevancy or membership in the premises. A collection of f.ℓ. rules is the knowledge base, where a f.ℓ. system may be built to infer outputs from input stimuli; an NFN is a computationally efficient f.ℓ. system capable of back-propagation [Lin and Lee, 1991]. We propose a framework to self-organize NFNs capable of policy induction by means of methods such as model-free offline RL [Kumar et al., 2020] or imitation learning [Torabi et al., 2018].

Assume state s belongs to a n-dimensional continuous space, ∏ni Si. Then, i indexes the state domain and corresponds to the ith state attribute. Action space, A, is a discrete and finite set. We will also demonstrate our methods on an example dataset (Table 1) throughout Sections 3 and 4. Building our NFNs requires no knowledge regarding the action taken or the reward received, only the state information.

3 Self-Organizing Neuro-Fuzzy Networks

Our prior work [Hostetter et al., 2023b] introduced a systematic design process called CLIP-ECM-Wang-Mendel (CEW) [Hostetter and Chi, 2023] to self-organize an NFN from data while remaining flexible to various learning paradigms. Sections 3.1, 3.2, and 3.3 describe each step to construct components which ultimately form an NFN. Section 3.1 creates membership functions in a data-driven, incremental manner; this effectively discovers logical propositions that describe "vague" symbolic concepts (e.g., slow velocity). Section 3.2 finds "exemplary" stimuli for f.ℓ. rule candidacy. Section 3.3 uses results from Sections 3.1 and 3.2 to yield possible f.ℓ. rule premises (i.e., linguistic descriptions or qualitative assessments of the stimuli); then, a preliminary NFN is selforganized and can be trained similarly to a DNN.

3.1 CreatingMembership Functions

Our f.ℓ. rules will map linguistic terms that describe the environment's current state to Q-values of the available actions. Fuzzy sets represent these linguistic terms (e.g., adjectives or concepts) and may be discovered using Categorical Learning Induced Partitioning (CLIP) [Tung et al., 2011].

CLIP is a quick, single-pass, computationally efficient algorithm to incrementally create fuzzy sets and does not require a predetermined linguistic term count. Given the first state, s, in the data, D, CLIP will create a fuzzy set in each ith domain as in Fig. 1.1. Initially, this fuzzy set in some ith domain may be interpreted as the concept of being near, similar, or approximately si (the core of the concept). This "soft" partition allows for varying degrees of membership, such that new data is ranked according to how similar it is to si. The membership function, µi,j, controls how we quantify similarity to the core, where j indexes the fuzzy set along the ith domain. Fuzzy sets found by CLIP are defined by Gaussian membership functions, with parameters for their center (i.e., core), ci,j, and sigma, σi,j. The value of si must belong to at least one fuzzy set in the state domain i with a degree exceeding ϵ. If none adequately describe si, then a new fuzzy set, µi,j, is centered on si, but σi,j depends on its neighbors, if any. If exactly one neighbor, µi,j ′ , exists, then its σi,j ′ is modified

such that κ controls how sigma is adjusted and σi,j = σi,j ′ (Fig. 1.2). Else, if µi,j has a left and a right neighbor, then apply (1) to both, and σi,j = Φ(σi,j ′ , σi,j ′′ ), where j ′ and j ′′ index the left and right neighbors, respectively. The Φ is a regulator function such that Φ(σi,j ′ , σi,j ′′ ) = 12 [σi,j ′ +σi,j ′′ ] where j ′ ̸= j ′′ and prevents malformed fuzzy sets by allowing a reasonable buffer between concepts to preserve their distinct semantic meaning. Fuzzy sets found on Table 1 using CLIP are shown in Fig. 2 with their parameters in Table 2.

3.2 Identifying Exemplars

The task of creating f.ℓ. rules is closely related to identifying exemplars, clusters, or prototypes within input-output data [Angelov and Gu, 2018]. In offline RL, although states have no associated output behavior before learning Q-values, we can still identify exemplars or "regions of interest" for f.ℓ. rules by the Evolving Clustering Method (ECM) [Kasabov and Song, 2002]. ECM quickly and dynamically estimates the number of clusters within data in a single pass and finds their current centers using a distance threshold (Dthr); Dthr affects cluster count, and subsequently, f.ℓ. rule count. Thus, Dthr can be adjusted to limit the growth of the knowledge base -a larger Dthr finds fewer candidates for f.ℓ. rules, but risks losing approximation power. Data is determined to belong to a cluster if its general Euclidean distance, defined as ||s - ˘s|| = (∑ni=1 |si - ˘ si|2 )12 /n 12 where s, ˘s ∈ S, is less than the Dthr. Clusters' centers obtained are candidates, X, to create f.ℓ. rules such that X = {x1, x2, . . . , xM}. ECM assists f.ℓ. rule generation by eliminating redundant states as f.ℓ. rules may grow linearly with respect to training data size (i.e., |D|) [Wang and Mendel, 1992b]. Applying ECM with Dthr = 0.2 on Table 1 simply aggregates states #2 and #8 to yield (1.265, 0.105, -0.235, 0.585) as there are only 8 rows.

3.3 Discovering Possible Fuzzy Logic Rules

The Wang-Mendel Method is well-established and widely used for f.ℓ. rule generation [Wang and Mendel, 1992b]. Given a set of training data, X, transform each x to its fuzzy representation through a Cartesian product of fuzzy sets, where each fuzzy set is for a specific input dimension. To determine which fuzzy sets, for 1 ≤ i ≤ n of x, we select the fuzzy set that xi attains the highest degree of membership:

where µi,j is a fuzzy set and |µi| is the count of fuzzy sets in dimension i. A f.ℓ. rule links compound fuzzy sets to a decision, but we will use 0 as the decision for versatility, as this is later learned through gradient descent. Thus, given candidate x, we make a f.ℓ. rule in the form: Rulek : (µ1,⋆, µ2,⋆, . . . , µn,⋆) ⇒ 0 where ⋆ satisfies (2) for 1 ≤ i ≤ n and Rulek means the kth f.ℓ. rule (k ≥ 1); rules with identical antecedents are eliminated to avoid redundancy.

From Table 1 and Fig. 2, we yield the f.ℓ. rules in Table 3. While these may be interpretable in our simple example, the f.ℓ. rules' premises grow linearly with respect to input dimensionality. If n = 142, then each f.ℓ. rule has 142 premises since CEW does not address this issue. Our proposed work in Section 4 extends CEW by fixing its shortcomings.

4 ProposedMethodology

Our model for maintaining ϵ-completeness (later defined) is introduced in Section 4.1, followed by FYD for simplifying f.ℓ. rules in Section 4.2. Finally, our NFN induces policies akin to a DNN but offers greater transparency.

4.1 Refining the Fuzzy Logic Rules

CLIP can generate high-quality fuzzy sets, but their interaction within premises may lead to subpar performance if the associated parameters are poorly chosen. In a small exploratory study, Hostetter and Chi reduced f.ℓ. rule count by generating f.ℓ. rules only for unique latent observations, using externally stored representations. This paper instead leverages the NFN's internal latent space to improve rule stability; to this end, we propose that the fuzzy representation of a state s, defined as {µ(s) | µ}, is of high quality if s can be reliably reconstructed from it.

Still, there is a significant caveat: na¨ıvely encoding to fuzzy representations and decoding will not work for fuzzy sets; pure decoding is insensitive to additional constraints such as maintaining semantics. Additionally, fuzzy sets may accidentally exceed the typical domain or violate ϵcompleteness if their parameters, such as centers and widths, are allowed unrestricted modification. The ϵ-completeness property guarantees input belongs to a fuzzy set (or f.ℓ. rule) with a degree ≥ ϵ, so NFNs have ample coverage of the input space to ensure a f.ℓ. rule is activated strongly enough to respond accordingly. Otherwise, catastrophic numerical issues stemming from (near) zero activation of f.ℓ. rules within a knowledge base can cause division by zero errors.

Let K be the number of rules discovered after Section 3.3. Then, f.ℓ. rules are created in the formMamdani-Rulek:

(µ1,⋆, µ2,⋆, . . . , µn,⋆) ⇒ (µ1,⋆, µ2,⋆, . . . , µn,⋆) where ⋆ again satisfies (2). Mamdani f.ℓ. rules contain fuzzy sets in their premises as well as their consequences. Naturally, they require a special form of NFN, and here, we use a Mamdani NFN with a center of sums defuzzification [Rutkowska, 2002]. We chose this approach, with Mamdani f.ℓ. control, as the knowledge base contained within truly embodies the intent of encoding and decoding (it's analogous, or a "f.ℓ. equivalent" to an approximate identity function that has a non-trivial mapping due to the non-linear internal workings of NFNs). Our remedy augments the target decoding by preserving activated rule strengths during fuzzy set tuning ( s µRule1 (s) . . . µRuleK (s) ) where µRulek (s) is the degree of activation of Rulek when given s. The Mamdani NFN auto-encoder's objective is to minimize the loss in reproducing s, as well as minimize the loss between µMamdani-Rulek (s) and µRulek (s) for all 1 ≤ k ≤ K rules. The reproduction of s is derived from the center of sums defuzzification of rules' consequences, which are fuzzy sets. If s can be reproduced from fuzzy sets in rules' consequences, then we argue that the fuzzy representation is of high quality. This promotes cooperative and aware fuzzy sets through their rule relations, resulting in improved reliability and stability in the selforganization process. In essence, both sets of rules cohabitate the final assembled NFN's knowledge base to help maintain consistency in the fuzzy hypercube mapping [Kosko, 1994] and avoid violating ϵ-completeness [Lee, 1990]. As observed in our experiments, adjusting individual premises is allowed if the compound premises' behavior remains consistent. If the need arises, preserving original compound rule activation strength can be omitted or updated if the system's dynamics demand it. Still, allowing compound rule activation to roam freely during Fuzzy RL tends to yield subpar performance or failure to learn. This is exacerbated if premise elimination occurs, which is introduced next in Section 4.2.

4.2 Simplifying the Knowledge Base

Fuzzy information granulation (i.e., f-granulation) theory is the magnum opus of Zadeh's legacy in soft computing [Zadeh, 1997]. This paradigm elegantly integrates fuzzy theory with numerous fields (e.g., probability, rough theory, graph theory) to leverage the full power of mathematics under a single umbrella. A compelling view in f-granulation reveals the relationship between f.ℓ. systems and graph theory. Although superficially simple, this has significant potential for interdisciplinary work between the fields. We leverage this viewpoint to discover f.ℓ. rules that have frequently occurring yet discernible premises from one another; this discernibility simplifies rules by removing redundancy, shrinking the number of premises contained in each rule, and enhancing the human readability of the knowledge while simultaneously not disrupting the rules' intended activation strengths. As such, we aptly call it the frequent-yet-discernible (FYD) method.

We retain premises with frequent activation to maintain ϵ-completeness. Fuzzy association analysis calculates fuzzy set frequency as the scalar cardinality, S [Chen et al., 2011]. The scalar cardinality of each premise term, µi,j , is computed by summing the membership degree of si to the fuzzy set µi,j across all states s in the dataset:

where rows and columns correspond to variables and possible terms, respectively.

A graph is built with vertices V and edges E, where V = {µi,j | (1 ≤ i ≤ n) ∧ (1 ≤ j ≤ |µi|)}, and E = {(µi,j, Rulek) | µi,j ∈ Rulek ∧ (1 ≤ k ≤ K) ∧ (1 ≤ i ≤ n) ∧ (1 ≤ j ≤ |µi|)}; put simply, vertices representing the input variables' terms are connected with directed edges to rules that involve them within their premises. To find discernible (i.e., unique) premises, it is easier if we first use metrics quantifying indiscernibility such as each vertex's closeness centrality, C [Freeman, 1978]; C typically calculates a social network's structural centrality (i.e., degree of centralization in the entire network). Vertex's closeness centrality is the inverse sum of distances to all other vertices (ignoring edge direction) [Cs´ardi and Nepusz, 2006]. In FYD, how reachable other premises are from mutual connections (via shared f.ℓ. rules) determines how often two premise terms co-occur. The possibility of redundant premise terms increases as this co-occurrence becomes greater.

C finds how indiscernible a premise term is from others by leveraging information regarding mutual connections (i.e., indiscernibility with respect to terms). So we also calculate direct usage, U, of each µi,j across all f.ℓ. rules with U = O K to further quantify premise indiscernibility (i.e., indiscernibility with respect to rules); the matrix O's entries correspond to each µi,j's out-degree (count of f.ℓ. rules it appears in) such that 0 ≤ i ≤ n and 0 ≤ j ≤ maxi |µi|. We experimentally found that FYD has more stable global or local feature elimination in f.ℓ. rules by incorporating both C and U.

We define indiscernibility, IND, of premise term(s) with C∧U, where ∧ is a t-norm (generalization of logical "and"). Since IND ∈ [0, 1], a premise term, µi,j, is more common as INDi,j → 1, suggesting it may be redundant, and viceversa. Here, we use the Hadamard product for the t-norm where IND = C ∧ U = C ⊙ U.

If discernibility is ¬IND (i.e., 1 - IND), then a higher value suggests a term is "poorly connected" or "uncommon". Assembling these components introduces the FYD formula:

where the t-norm (i.e., "logical and" ∧) may be defined in multiple manners, but for simplicity, we use the Hadamard product again. Matrix S and IND are normalized to bound them within [0, 1]. As FYDi,j → 1, the premise µi,j is more frequent-yet-discernible. Example calculations of C, O, and IND are provided based on the NFN built with fuzzy sets and f.ℓ. rules from Fig. 2 and Table 3, respectively.

Although some premise terms are considered discernible (e.g., 1-IND1,2 = 0.897), when we factor in activation frequency across D, then discernible premises (purely according to the graph) are no longer given high importance since they are infrequent (e.g., FYD1,2 = 0). So, the premise term, µ1,2, is unnecessary, as it is seldom activated in practice despite discernibility metrics allocating high importance to it because of its rare involvement in f.ℓ. rules. This highlights the glaring shortcoming of na¨ıvely simplifying rules solely on discernibility (e.g., LERS from rough set theory) as in the RSPOP family of NFNs [Ang and Quek, 2005; Das et al., 2016; Iyer et al., 2018] where the nature of premises' multi-valued degree of truth is totally disregarded. Similarly, relying only on activation frequency may disregard how often a premise term occurs across f.ℓ. rules. For example, FYD3,1 = 0 because although it occurs frequently, S3,1 = 4.603, it has the highest closeness centrality, C3,1 = 0.459, and is used across the most rules, O3,1 = 4. Our proposed FYD is the first work to identify this inadequacy of existing work in f.ℓ. rule simplification for offline Fuzzy RL, and requires no utilization of the intended outputs or targets for our f.ℓ. rules; this makes it appropriate for Fuzzy RL when there is no clear desired mapping yet.

Kneedle [Satopaa et al., 2011] finds an appropriate knee point in the sorted values from FYD. Then, any µi,j with a FYDi,j below this threshold is removed, and the f.ℓ. rules are updated accordingly. This performs a local (and, in the extreme -global) feature selection. It may also delete f.ℓ. rules if all their premises are deleted, as they are then nonessential. Only the maximal frequent compound premise of f.ℓ. rules are kept. Kneedle calculates 0.257 as the threshold from FYD(D), and retains only 5 from 11 terms: µ1,1, µ2,2, µ2,3, µ3,2, µ4,2. Rule 5 is non-essential, as all its premise terms have been cut. Rules 1, 3, 4, and 7 are marked for deletion as well, as their compound premises are subsets of larger FYD premises. Specifically, rules 3 and 7 are cut since they are proper subsets of rule 2, and their activation is sufficiently captured by rule 2. Similarly, rule 4 is redundant as it is a proper subset of rule 6, so they share overlapping activation. FYD finds only rules 2 and 6 are frequent-yetdiscernible from the 7 f.ℓ. rules in Table 3 given D.

A human may attach semantics to the "vague" symbols' precise definitions (Table 5) by analyzing the corresponding plotted fuzzy sets (Fig. 2). For example, rule 6 may be read as: "if the cart's position is slightly right of the middle, the cart's velocity is fast (positive), and the pole angular velocity is fast (positive), then the (Q-)values of pushing the cart left or right are and , respectively." Consequences are shown as blanks (i.e., ) since they change (via gradient descent) based on the NFN's performance, but per Section 3.3, we initialize all consequence values to start from 0.

Fewer premises may be retained than the maximal f.ℓ. rule premise identified here, but finding a minimal discriminant f.ℓ. rule premise is left for future research. The only step remaining is to train our simplified NFN with a learning algorithm. It is important to emphasize that an NFN, once built, is used similarly to a DNN but offers greater transparency in its decision-making. We showcase NFN's generalizability with Kumar et al.'s Conservative Q-Learning (offline RL) and Behavior Cloning [Torabi et al., 2018] (imitation learning).

5 Classic Control Experiments &Results

We evaluated on Cart Pole [Barto et al., 1983], Mountain Car [Moore, 1990], and Two-Link Arm [Sutton, 1995].

Experiment conditions. For baselines, DNNs were trained with 7 different strategies: 1. Conservative Q-Learning (CQL) [Kumar et al., 2020]; 2. Behavior Cloning (BC) [Torabi et al., 2018]; 3. Deep Q-Learning (DQL) (with no offline augmentation) [Mnih et al., 2015]; 4. Double DQL (DDQL) (with no offline augmentation) [Hasselt et al., 2016]; 5. Batch Constrained Q-Learning (BCQ) [Fujimoto et al., 2018]; 6. Neural Fitted Q Iteration (NFQ) [Riedmiller, 2005]; 7. Soft Actor Critic (SAC) [Christodoulou, 2019]. All were selected for their applicability to discrete control and offline learning potential. Additionally, we compare to NFNs self-organized using CLIP, ECM, and Wang-Mendel (CEW) [Hostetter et al., 2023b; Hostetter and Chi, 2023] (as in Section 3) and trained via CQL or BC. We evaluate our proposed FYD method (Section 4) against these baselines and train its NFNs via CQL or BC. We highlight CQL and BC to showcase NFNs' ability for offline RL or imitation learning, but it is compatible with others too (e.g., SAC). DNNs had two hidden layers (256 RELU neurons each), a linear output layer, and one output neuron per possible action. All methods were optimized by Adam [Kingma and Ba, 2014].

Policy induction. Shared parameters were identical: α = 1.0, γ = 0.99, learning rate η = 3 × 10-4 , and the batch size was 32. For FYD or CEW, CLIP used κ = 0.2, ϵ = 0.6, and ECM's distance threshold, Dthr, was 0.1. Training data was collected by a DNN using DQL and experience replay while solving the given environment online. Each condition was run 10 times across different seeds; each algorithm was shown the same data in the same order for each seed. During each run, the amount of data available for offline training gradually increased to show how conditions behave as more data is provided. Policies were evaluated online for 10 episodes using OpenAI Gym and mean performance was recorded.

Results. Fig. 3.a. shows FYD remains on par with themean performance of DNN or CEW with CQL or BC and in some cases, surpasses it (e.g., Cart Pole). Compared to the existing method of building NFNs called CEW, FYD would often obtain higher rewards and show greater stability. Since the two share the same processes for fuzzy set definition, exemplar identification, and rule generation (see Section 3), the performance difference can be attributed to our proposed changes in Section 4; the reduction of premises is shown in Fig. 3.b.

6 ITS Experiments &Results

A web-based ITS teaches ten probability principles (e.g., Bayes' Theorem). The ITS provides adaptive instructions, immediate feedback, and on-demand hints to enhance learning. Pedagogical decisions are whether a student should solve the next problem (Problem-Solving (PS)), study a workedout example (Worked-Example (WE)), or work collaboratively with the ITS on the next problem (Collaborative PS (CPS)). An additional level of interaction between student and ITS occurs during CPS - ITS can decide to tell the student the next step or elicit the student to solve the next step.

Experiment conditions. This study was homework for an undergraduate Computer Science class. Students were told to finish in a week and that they would be graded on demonstrated effort rather than learning performance. 225 students were randomly assigned: FYD (N=53), CEW (N=56), DNN (N=55) and ECLAIRE (N=61). ECLAIRE is an efficient, polynomial-time rule extraction to decompose DNNs into rule-based models [Zarlenga et al., 2021]. The tutor, general procedure, training materials, and questions were all the same during these studies. The training corpus provides the states, actions, and rewards for policy induction. Due to midterms and study length, 195 students finished, but 22 were excluded from the analysis due to perfect pretest scores. Final group sizes were FYD (N=39), CEW (N=38), DNN (N=44) and ECLAIRE (N=52). The students' completion rate between conditions was not significantly (sig.) different: χ2 (3) = 2.5443, p = 0.46734.

Procedure. Textbook: Students review probability principles. Pretest: Students' a priori knowledge is bench-marked with 8 single- and multiple-principle problems. ITS training: Students are trained with 12 problems (shown in the same order for each) with the assistance of an automated tutor (e.g., FYD, DNN). Posttest: Learning is evaluated with 12 problems -8 isomorphic to the pretest, with the remaining 4 as non-isomorphic multiple-principle problems. Tests were graded by 2 experienced graders in a double-blind manner.

State. 142 features that may impact student learning extracted from interaction logs are split into 5 groups: Autonomy (10): amount of work done by the student (e.g., steps done without help); Temporal Situation (29): time-related information (e.g., average time per step); Problem-Solving (35): current problem-solving context (e.g., problem difficulty); Performance (57): student's ability to solve (e.g., percentage of correct steps); Hints (11): student's hint usage.

Action. PS, WE, or CPS as previously described.

Reward. There is no immediate reward during tutoring, but the delayed reward is students' Normalized Learning Gain (NLG) -their learning gain irrespective of incoming competence [Abdelshiheed et al., 2024; Islam et al., 2024]. NLG is posttest-pretest √1-pretest , where 1 is the max score for posttest and pretest score is lower than 1.

Policy induction. A hierarchy of policies was created where one policy determines what action to take on the problem-level, and separate policies determine whether to elicit or tell the next step during a CPS for each problem. All pedagogical policies were induced offline with CQL using 2,421 students' interaction logs over 14 semesters of classroom studies; parameters were: α = 0.1, γ = 0.99, learning rate η = 3 × 10-4 , κ = 0.2, ϵ = 0.7, with a batch size of 32. ECLAIRE policies were produced from DNNs after CQL.

Results. Effect sizes are partial eta squared (η2 ) or Cohen's d. Pretests were not sig. different (F(3, 169) = 0.247, p = 0.781), indicating balanced incoming competence. Regardless, students' incoming competence was factored by adjusting for pretests with a one-way ANCOVA; due to each experimental condition's strength, there was also no statistically sig. difference in students' learning (F(3, 169) = 0.158, p

= 0.924), but time spent for ITS training was sig. different (F(3, 169) = 13.047, p ≤ 0.001, η2 = 0.188). TukeyKramer post hoc multiple comparisons found FYD had a sig. and large effect reducing the time to complete ITS training by 0.638, 0.719, 0.636 hours compared to ECLAIRE (d = 1.374), CEW (d = 1.301), and DNN (d = 1.102), respectively (p ≤ 0.001 each comparison).

Model complexity. The f.ℓ. rule count between FYD and CEW was comparable, t(18) = 0.316, p = 0.755, d = 0.15, but premises' count was sig. different with pronounced effect size in favor of FYD, t(5564) = 1720.551, p ≤ 0.001, d = 46.15. FYD had 286.8 (35.01) rules with 9.734 (3.993) premise terms, but CEW had 269.8 (92.991) rules, each with 142 premise terms. Although ECLAIRE had 2.1 (0.316) rules, their premises had 330.780 (236.032) terms. The term count in the rules' premises between FYD and ECLAIRE was sig. different with a pronounced effect (again, in favor of FYD), t(2907) = 72.978, p ≤ 0.001, d = 11.48. For qualitative analysis, a sampled rule from each FYD and ECLAIRE in Table 6 illustrates our rules' readability. For ECLAIRE -a state-of-the-art decompositional rule extraction method for DNNs -we select the smallest rule, which still contains 26 inequality relations within its premise. Our method's ability to construct short rules conditioned upon original features offers exciting possibilities for automatic knowledge acquisition and facilitation from agent to human.

7 RelatedWork

Hein et al. self-organize NFNs for offline RL by using fuzzy particle swarm RL in a simulation to learn its parameters, but must define sought-after f.ℓ. rule count, and assumes it is simple to model the system's dynamics. Incorporating other logic with RL has also been researched. Relational RL (RRL) combines RL with inductive logic programming (or relational learning) to produce interpretable and generalizable policies; these may be applied to planning tasks [Dˇzeroski et al., 2001].

Relational Deep RL leveraged DNNs with RRL to play StarCraft II [Zambaldi et al., 2018]. Alternatives such as neural logic RL represent induced policies with first-order logic [Jiang and Luo, 2019]. Deep Symbolic RL (DSRL) derives symbolic representation from unstructured data with a DNN [Garnelo et al., 2016]. Symbolic RL with Common Sense extends DSRL by generating symbolic representations prior to learning and decision-making algorithms [d'Avila Garcez et al., 2018]. Other approaches aim to mimic a DNN policy by extracting an interpretable model, but encounter limitations [Bastani et al., 2018; Hostetter et al., 2023a].

8 Conclusion

Self-organizing NFNs are interpretable and adaptable by leveraging data for their structure and parameters. Our framework yields effective NFNs with less rule complexity but no observable loss in performance. FYD shortened the ITS training time for students and exhibited statistically higher transparency concerning premise count. Our NFNs' rules are conditioned upon the original features expressed via a linguistic medium (i.e., constrained fuzzy partitions). Their knowledge can also be transferred to/from other NFNs, opening many exciting opportunities, such as in federated learning.

Acknowledgments

This research was supported by the NSF Grants: Integrated Data-driven Technologies for Individualized Instruction in STEM Learning Environments (1726550), CAREER: Improving Adaptive Decision Making in Interactive Learning Environments (1651909), and Generalizing Data-Driven Technologies to Improve Individualized STEM Instruction by Intelligent Tutors (2013502).