Taking STEPS Forward: Enhancing Online Peer-Counseling with Schema Therapy via Socratic Questioning

1 Introduction

Online mental health communities (OMHCs), such as 7Cups, 1 Beyond Blue, 2 and Patient, 3 have emerged as costeffective, scalable, and accessible avenues for those seeking mental health support amid the rising global mental health crisis [Yao et al., 2022; United Nations, 2015]. Peer counseling is essential in online mental health communities by providing relatable support and offering a safe space to share experiences and coping strategies. Through providing empathetic listening and emotional support [Salsabila et al., 2020;

1 https://www.7cups.com/community/

2 https://forums.beyondblue.org.au/

3 https://patient.info/forums

Syed et al., 2024; Ali et al., 2015], as well as sharing their own experiences and practical advice, the peer counselors are able to create a sense of connection with the support seekers [Wang et al., 2012; Joo et al., 2022].

However, to achieve lasting and positive behavioral and cognitive changes in the concerned individuals, deeper insights into their emotional and cognitive needs [Beck, 2020; Young et al., 2006] will be required. Unfortunately, online peer counselors often lack the extensive professional training in therapeutic counseling techniques required to effectively address these needs. As such, AI-powered tools such as CARE [Hsu et al., 2023] and Partner [Sharma et al., 2021], are now in use to help peer counselors engage with support-seekers, focusing on using motivational interviewing and empathy-based responses. We go beyond these basic approaches by asking: How can we design a therapy-based assistive tool to help the peer counselors effectively elicit cognitive and behavioral change in online support-seekers?

We developed STEPS (Schema Therapy-Enhanced PeerCounseling through Socratic Questioning), an AI-powered dialog assistant to address precisely this question. STEPS harnesses the generative capabilities of state-of-the-art large language models (LLMs) to emulate two key phases of Schema Therapy (ST)-based in-person counseling sessions, viz, (1) Schema Assessment where deep-seated maladaptive beliefs contributing to one's mental health problems are uncovered, and (2) Cognitive Change where these beliefs are reframed into healthier alternatives. Central to exploring these phases are Socratic questions-specific types of open-ended questions that encourage critical examination about one's beliefs and perspectives [Paul and Elder, 2019; Montazeri, 2022; Ang et al., 2023; Waltman et al., 2020]. Table 1 illustrates a STEPS-generated conversation snippet, highlighting various Socratic questions and their types (highlighted and labeled) in the corresponding counseling phases.

Section 2 summarizes the key components of STEPS, followed by an expert evaluation by licensed ST practitioners in Section 3. Supplementary details on implementation, LLM prompts, experiments, and evaluation are available in the appendices 4 along with a demonstration video. 5

4 Appendices are available in https://github.com/NUS-IDS/ems mentalhealth/tree/IJCAI STEPS

5 https://youtu.be/htY8CwqXzr0

2 Methods

Schema Therapy (ST) is an integrative therapy framework developed by Young [Young et al., 2006] for treating personality disorders and other mental health problems. ST-based counseling has been increasingly adopted for its effectiveness in addressing individual and relationship problems [Masley et al., 2012; Bakos et al., 2015; Taylor et al., 2017]. A key concept in ST is Early Maladaptive Schemas (EMS), negative and enduring cognitive patterns and beliefs formed in childhood experiences and affect one's perspectives, emotions, and behaviors [Young et al., 2006]. For example, individuals with the "Failure" schema often believe that they are just incompetent. 6 Identifying these EMS is central to Schema Assessment [Young et al., 2006]. Effective counseling support challenges these EMS and reframes them using structured techniques such as Socratic questioning to foster critical introspection and Cognitive Change [Young et al., 2006; Paul and Elder, 2019]. In STEPS, these phases are:

1. Schema Assessment that focuses on understanding the support seeker's problem by asking relevant questions that gather essential details on Schema Triggers (i.e. situations

6 All 18 EMS types are described in Appendix A. See footnote #4

that activate negative beliefs), Thoughts, Emotions, Behaviors, and Bodily Responses (e.g. "When you start to feel really bored, what does that feel like?"). The support seeker's beliefs are then elicited through probing questions from the problem (e.g. "What does this say about you?"). Next, EMS are identified from the support seeker's problem and stated beliefs. We extend a recent EMS prediction study [Gollapalli et al., 2023], which compared support seeker's beliefs with the Young Schema Questionnaire (YSQ) [Young and Brown, 2003], a psychometric tool of 232 statements on EMS, using various textual similarity, LLMs, and textual entailment-based techniques. Building on this approach, we compare the sentence similarity between the beliefs of the support seekers and the YSQ statements, selecting those EMS with scores above a threshold 7 for further validation through clarifying questions (e.g. the second question in Table 1).

7 Sentence transformers [Reimers and Gurevych, 2019] are used with a similarity threshold of θ = 0.4

2. Cognitive Change focuses on encouraging support seekers to critically evaluate their EMS and explore more balanced, healthier beliefs through questions that probe for alternative strategies, (e.g. "What other ways could you manage this belief?"), evidence-based inquiry (e.g. "What past experiences support your belief that you are a failure?"), and challenging their beliefs through alternative perspectives (e.g. "Could it be that other students also struggled with the test, and this doesn't mean you are a failure?").

Empathetic Socratic Question Generation: Socratic questioning is central to STEPS, where empathetic responses combined with specific open-ended questions are systematically employed throughout the counseling phases. STEPS uses recent LLMs to generate these Socratic questions. While LLM can generate empathetic responses and context-specific Socratic questions in isolated (one-step) contexts [Li et al., 2022; Priya et al., 2023; Ang et al., 2023], research has shown that their planning capabilities are insufficient for steering conversations in goal-oriented contexts such as negotiation or counseling. To address this shortcoming, a "dialog policy" is often used to guide the LLM on the appropriate response type [Jang et al., 2022; Deng et al., 2024; Gollapalli and Ng, 2025]. Given our counseling objectives and lack of training data, we directly incorporate the "dialog policy" suggested by ST framework to determine specific Socratic questions to generate within each phase. The precise prompts provided in Appendix C are used to generate the specific types of Socratic questions presented in Figure 1. STEPS is designed as an assistive tool in a human-in-the-loop framework, presenting suggestions for further editing before the final interaction with the support seeker. Part of our system interface illustrating this aspect is shown in Figure 2 with the full interface detailed in Appendix D. 4

3 Experiments and Results

Experimental Setup: We adopt the recent approach for conversation evaluation where two separate LLMs role-play the conversation actors [Abbasiantaeb et al., 2024]. That is, STEPS acts as a "peer counselor" and responds to LLMPatient simulating a "support seeker". We used 20 CounselChat forum posts [Bertagnolli, 2020] available from a prior study on ST [Gollapalli et al., 2023] and evaluate two configurations: (1) LLMST+phases, a LLM prompted to emulate a Schema therapist by incorporating Schema Assessment and Cognitive Change into its dialog and (2) STEPS, our proposed system. Our goal is to test if the existing SOTA LLMs with parameterized knowledge of ST are able to emulate the process designed to enable cognitive change in ST counseling given appropriate prompts, compared to STEPS. 8

8 We used the "gpt-4o-2024-05-13" model in experiments but based on initial comparisons, expect the results to apply to all LLMs

Evaluation Criteria and Metrics: We evaluate LLMST+phases and STEPS on their generated conversations along specific metrics of (1) Schema Exploration and Assessment (Assessment), and (2) Application of Cognitive Change Techniques (Cognitive) by collecting human ratings on a scale from 0 to 6. These metrics were established for evaluating a Schema therapist's proficiency in identifying EMS and applying cognitive strategies to reframe EMS into healthier beliefs [Young and Fosse, 2005] (Appendix E). Additionally, we evaluate if the conversations foster critical thinking, a key aspect of effective counseling [Overholser and Beale, 2023], by adapting metrics from the Organisation for Economic Cooperation and Development critical thinking rubrics [Vincent-Lancrin et al., 2019]. These metrics evaluate proficiency in: (3) Assumption-checking (Inquiring), (4) Identifying alternatives and hypotheticals (Imagining), (5) Justifying reasoning with logical premises (Doing), and (6) Assessing limitations of a perspective (Reflecting). 9 Following prior work [Gollapalli et al., 2023], we hired two certified Schema therapists and one Psychology graduate with experience in ST via Upwork. 9 They evaluated 30 generated dialogs at a rate of five samples per hour, with compensation of $20-$40 per hour. Six dialogs were rated independently by all evaluators, yielding an inter-rater agreement of 0.72 (p < .05) as measured by Kendall's τ [Kendall, 1938], indicating strong consensus among evaluators.

As shown in Table 2, STEPS achieved an overall average rating of ∼5, corresponding to a "very good" rating, and significantly outperformed the LLMST+phases by 3-4 points. This highlights STEPS's potential for enhancing online peer counseling via ST. Furthermore, the low variability in ratings, with standard deviations of ∼0.6 compared to the baseline's ∼1.4, highlights the more consistent performance in STEPS compared to that of LLMST+phases.

4 Conclusion

This paper presents STEPS, an AI-powered dialog assistant that integrates Schema Therapy principles and Socratic questioning with LLMs to enhance online peer counseling. Expert evaluation by counseling professionals shows that STEPS significantly outperforms the LLM baseline in identifying and addressing EMS and fostering critical thinking. However, in high-stakes domains like mental health support, it is critical that there are additional layers of oversight to ensure that the presented responses are safe, appropriate, and therapeutically sound. Before deployment in OMHCs, these findings must be carefully validated in real-world peer counseling environments. Additional safeguards, such as a controller to filter for safe LLM responses from STEPS, should also be in place.

with similar capabilities. Our prompts are included in Appendix C. 9 www.upwork.com

Ethics Statement

This research strictly adheres to the ACM Code of Ethics, emphasizing the ethical implications of AI usage in mental health contexts. STEPS is developed as an AI-assistive tool designed to support peer counselors in facilitating introspective conversations and promoting positive change, rather than serving as a replacement for professional therapy. The system operates within a human-in-the-loop framework, where all AI-generated suggestions from STEPS are reviewed and mediated by trained peer counselors. To ensure a thorough evaluation of STEPS, we engaged accredited Schema therapists through Upwork, offering fair compensation for their time (details provided in Section 3). The general ethical and privacy concerns for use of AI in health care are not applicable for our prototype system that has not been tested/deployed in real settings and all datasets are based on publicly-available mental health forum posts.

Acknowledgments

This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-GC-2019-001-2B) and the National University of Singapore, Institute of Data Science. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the National University of Singapore.