Interactive concept-map based summaries for SEND children

1 Introduction

The right to qualitative, equitable and inclusive education is a human right. Every student, irrespective of their origin or abilities, should be able to achieve their educational goals. Yet, significant differences and challenges persist, especially when it comes to children with special education needs and disabilities (SEND). SEND students with difficulties in Cognition and Learning often need support to automate processes such as writing, arithmetic and reading. Reading comprehension not only enables learners to attain primary and secondary education, but it is also a crucial life skill, serving as the foundation for many facets of daily life in the information age. As the years go by, children diagnosed with reading comprehension falls can improve at the decoding of single words at a functional level, but, in most cases, the overall process of reading remains often slow, painful and tiring. This has repercussions on attention capability and execution times and can lead them to drop out of school earlier than their peers [Mather et al., 2020].

∗Contact Author

Speech and language pathologists often make use of the construction of a conceptual map, a structured summary in the form of a graph [Novak and Ca˜nas, 2007], as a way to help these students process information. The active processing of information, stimulated by the use of their visual representation, results in non-mnemonic learning of concepts, making information stored in long-term memory. This has proven to be one of the most efficacious strategies to help these children make sense of written text at any stage of learning [Dexter and Hughes, 2011].

During Covid-19, the suspension of non-urgent care, including speech and language therapy, had a substantial impact on speech-language pathologist interventions, forcing the transition into online therapy practice. This has exposed significant gaps in the speech therapy system, such as the lack of IT resources or insufficient training, making it difficult to keep up with the same therapy standards at a distance. With schools closed and with a few speech-language centres offering an online alternative to traditional sessions, SEND students were thus left even more isolated [Tohidast et al., 2020].

Some AI models offering the possibility of automatically extracting concept maps from text have been implemented over the years [Falke, 2019; Hwang, 2003; Olney et al., 2011; de Aguiar et al., 2016; Qasim et al., 2013]. However, a tool specifically designed for clinical profiles by taking into account specific colour schemes and web accessibility guidelines is still lacking. Furthermore, the interactive process, fundamental to engaging in effective active learning, is completely missing in the existing tools. Finally, most existing methods automatically produce a canonical representation given an input text. However, different children with divergent levels of disability approach a collection of documents with dissimilar needs.

The current project collaboration proposal aims to bridge these gaps by implementing an online Artificial Intelligencepowered tool that could improve the reading comprehension skills of children with difficulties in written text comprehension. A new computational model for concept-map based document summarization (CM-DS) [Falke, 2019] will be implemented and embedded into an interactive and online web interface. The target language will be Italian in the first phase, given the particular use case, but we will keep multilingualism in mind during the design process. The tool will be tested and evaluated in a set of user studies by specialised speech therapists in Rome (Italy) over a treatment group and a control group to effectively verify its utility and effectiveness.

In this paper, we will outline the details of this collaboration, starting with the clear societal benefits it could bring and the UN's Sustainable Development Goals that we will tackle. The goals, the expected results, the methodology and the evaluation will be highlighted. Particular attention will be dedicated to the challenges, risks and ethical considerations.

2 Description of the Task and Problem Statement

The task of Concept Map-based Multi-Document Summarization (CM-MDS) defined by [Falke, 2019] is the process of automatically extracting concept-map based document summaries from a set of documents. Following the CM-MDS task [Falke, 2019], we define Concept-map-based Document Summarization (CM-DS) as the process of automatically extracting concept-map based document summaries from a single document. The amount of existing work in the area is limited both for single documents [Hwang, 2003; Leake, 2006; Olney et al., 2011; Kowata et al., ; de Aguiar et al., 2016; Villalon and Calvo, 2009], and for multiple documents [Rajaraman and Tan, 2002; Zubrinic et al., 2012; Qasim et al., 2013; Falke, 2019]. Moreover, most of the existing systems only produce a single representation for a given input text. In practice, however, different students with non-identical disabilities, approach a document with unlike information needs, incompatible levels of complexity and dissimilar background knowledge. Taking into account the differences in educational needs and accounting for those is fundamental when designing effective tools for inclusive, qualitative and nondiscriminatory education. Recent work in document summarization tasks proposed a model that creates personalized summaries based on users' feedback [Avinesh and Meyer, 2017; Ghodratnama et al., 2020]. To our knowledge, no research in the literature has focused on the interactive construction of conceptual maps based on students' specific clinical profiles. Moreover, a tool specifically designed and tested on a clinical group that enables the interactive creation of the concept map together online is still lacking.

In this research collaboration, the Sony Computer Science Laboratories Paris (France) aims to tackle this gap by implementing a new model for CM-DS and embedding it in an online and open-access web interface. Highly skilled speech therapists from Centro Ricerca e Cura (CRC), a language disorders rehabilitation centre in Rome (Italy), will test the tool and the underlying algorithm through a set of user studies performed over a treatment group and a control group.

More formally, expanding the formal definition of Concept Map specified by [Falke, 2019], given a document D, the nodes of concepts and relational edges (respectively, C and R) with labels (l(c), l(r)), and an optional seed concept S (representing the most important concept in the text), we want to create a personalised concept map G = (S, C, R) that: (I) represents the most important concepts C and relational edges R in D, (II) has no more concepts C and relational edges R than the limit λ set λ(c), λ(r), (III) is a connected graph, (IV) is suitable to the educational needs of each user. The output obtained will be different from an ontology in the sense that concept maps do not aim to provide a fixed vocabulary to represent the relations. The relational edges are formed using free text, which we believe is fundamental to enhancing the lexical richness of children with learning disabilities.

3 Target SDG(s) and clear societal benefits

The United Nation's 2030 Agenda for Sustainable Development sets out 17 interlinked sustainable development goals (SDGs) that define a plan for the pursuit of prosperity and equality. The SDGs focus on the principles of capturing universal contexts across the globe, being inclusive of all sectors of society and leaving no one behind, especially the most vulnerable and marginalised.

Our collaboration proposal largely focuses on SDG target 4: "Quality Education". In particular, our work is dedicated to ensuring a qualitative, equitable and inclusive education for children diagnosed with falls in written text comprehension skills. With this goal in mind, Sony CSL Paris wants to collaborate with speech therapists from CRC, a language disorders rehabilitation centre in Rome treating more than 700 hundred children a year. In line with target 4.1., the relevance of conceptual maps for a clinical group has already been ascertained by the literature and by CRC speech therapists which are therefore perfectly placed to measure the effectiveness of the technology that is developed.

Following a Human in the Loop approach, we want to build and test an AI system that would help children with reading comprehension deficits interactively create a personalised conceptual map. This guarantees that the learning effort required by the student hasn't been replaced by technology, but that it's been enhanced and supported by it.

Though our empirical evaluation is focused on a specific clinical group and a respective control group, the technology behind the construction of conceptual maps together online is useful beyond that scope. In line with target 4.a., our personalised and inclusive system can be used whenever expert tuition is not available to non-clinical users. This is particularly relevant in home-schooling contexts compelled by causes of force majeure or by a lack of proper educational staff.

Moreover, we are committed to providing our technology online and free of charge and to developing it under an opensource license so that others can collaborate in improving our technology in line with the indicator 4.3.1.

A lifetime of being under-resourced in learning leads to various hardships in adult life and society, which reduces the "personal resources" (aka psychological capital) for coping with various challenges such as self-efficacy, hope, optimism and general resilience [Costantini et al., 2020]. Adults with language disorders attain lower employment rates, earnings, and are employed in lower-skill positions compared to their counterparts without disabilities [Vogel et al., 2007; Stein et al., 2011]. Providing technology to empower young people with learning disorders allows them to climb towards closer parity in terms of income to their peers. This is in line with multiple SDG targets such as targets 10.1 and 10.2, concerning the inequality of income, but also targets 8.5 and 8.6, related to lifting people with disabilities out of low wage jobs and unemployment and, certainly, target 4.4., concerning learning appropriate skills to find decent jobs.

Moreover, children with language disorders risk growing up into adults who are held back from taking part in the global dialogue, when this involves reading comprehension and writing skills. Enabling participation on the global stage would help children affected by learning disorders to represent their personal context and challenges directly and, thus, help reach targets 10.3 and 10.2.

4 Goals

Our collaboration proposal aims at contributing towards equal rights and equal opportunity education by implementing an inclusive, multi-modal, online and open-access tool for enhancing reading comprehension skills thanks to interactively creating concept maps. The inclusiveness of the tool is ensured by tailoring the output of the tool according to user disabilities and by respecting web accessibility guidelines. The computational model produces user-specific concept maps based on the specific needs and learning goals of the children. To achieve this goal, this research collaboration will need to accomplish its objectives along three different fronts: (I) computational innovation, (II) clinical insights and practices, and (III) software development. From a computational point of view, it will need to implement a model for CM-DS with the inclusion of user-personalised extraction. This will include solving major sub-tasks, such as (1) Mention and Relations extraction, (2) Mention grouping, (3) Importance Estimation, (4) Visualisation, (5) Evaluation of the final result. From a clinical point of view, the clinicians will help define and utilise the tool in a set of user studies to test for its efficacy. This could help towards understanding whether an AIpowered tool could help to reduce the time of taking charge, increase the use of online therapies and generally the overall rehabilitation process for patients diagnosed with reading comprehension falls. Finally, from a software development point of view, we will need to develop and deploy a robust system that meets the needs of the clinicians, that is appropriately deployed online and that is developed according to specific web guidelines granting easier accessibility of the tool to every student, irrespective of their disabilities.

5 Expected Results and Long Term Impact on the SDGs

In long term, we imagine a future where tools like ours will enhance the completion rate and attainment of children and young people with learning disabilities (indicator 4.1.1). Thanks to the inclusiveness of our tool, we contribute by providing a personalised and, yet, a shared method to access learning. By virtue of its online access format, we make it usable in formal and informal contexts, either with a practitioner or in an auto-didactic context.

Furthermore, the interaction with the system, so successful in online learning platforms such as Duolingo, can incentivise students to pursue learning. In a nutshell, we want to make learning and education more equitably accessible and inclusive, with no discrimination against people in terms of their disability status, gender, race or any other quality that leads to marginalisation (indicator 4.5).

Our long-term impact goes beyond contributing toward a more accessible and inclusive education. Better education correlates with better employment outcomes. If children and young adults with learning disabilities have access to more qualitative education, then their employment and job satisfaction can improve in the long term. Employability and education together can directly contribute to achieving SDG 10: "Reduced Inequalities".

Last but not least, through this research collaboration, we hope to raise awareness about language disorders. Though there are various national and international organisations that target language disorders, many countries are not represented. Even in developed countries with relatively high awareness and support such as the United States, explicit knowledge about linguistic concepts or cognitive difficulties relevant to these disorders lacks and even shows, sometimes, misconceptions about the deficits involved. In developing countries this is even more crucial: paywalls to language disorder tools are significant barriers to quality education access. We think that the basic services offered by schools in rich and poor countries will need to include language disorders aware educational resources, even when they do not have teachers trained in teaching language disorders students. Let's just take the example of Dyslexia, which accounts for 80-90% of learning difficulties. This learning difficulty affects 20% of the world to varying degrees and the available resources worldwide are now insufficient [Mather et al., 2020]. Our single collaboration rooted in free and accessible open-source practices, based on SDG principles and powered by AI could provide a blueprint for global change, a flagship for the resourcing of tools for language disorders awareness.

6 Challenges

The main challenges posed by this collaboration proposal are clinical and computational. Concerning the clinical challenges, selecting a statistically significant sample for testing and evaluating the tool is a demanding task. Samples are expected to be chosen in such a way as to avoid presenting a biased view of the population. For the set of user studies, a heterogeneous sample will therefore need to be selected. As indicated in the Methods section, we plan to make our user studies more robust by splitting our users into a treatment group and a non-clinical one.

For the clinical group, the overall process of reading is painful and tiring. Their motivation to participate will need to be encouraged. We plan to tackle this through the calibration of the complexity of the stimuli, the use of positive feedback given by the system and enforcing an interactive process of learning, as indicated in the Methods section.

Ensuring motivation is fundamental especially when developing tools for online therapies. Tele-practice has several advantages, such as reducing travel and waiting room time, ensuring therapeutic continuity when it is not possible to go to the clinic, lessening the costs, increasing the frequency of exercise, and thus, potentially, increasing the effectiveness of the treatment. Nevertheless, the clinicians need to know how to calibrate the instrument and how to interpret the data logged in it. Over-complexity should be avoided in every case. We plan to tackle these challenges by involving all the stakeholders from the very beginning of this research collaboration, as indicated in the Ethical Considerations section.

Finally, given the limited amount of research in the domain of CM-DS, there is not a shared evaluation protocol or common evaluation metrics. As indicated in the Evaluation section, we evaluate the output of our systems with a mixture of manual and automatically implemented measures.

7 Methods

We aim at solving CM-DS as an abstract computational problem and through a pipeline approach that we believe can work across languages.

The participants of the users' studies will probably need to be a heterogeneous group. The treatment group could be composed of children in the age range of 9 to 13 years old, diagnosed with Learning Disorders. The non-clinical group could be composed of children in the same age group as the treatment group without a diagnosis but in need of support in developing appropriate study strategies.

Our contributions could be three-fold. We implement and evaluate a new pipeline for CM-DS and a clinically appropriate web interface. Finally, the benefits of conceptual map based summarization techniques have yet to be tested on children with language disabilities.

7.1 Model Pipeline

Following [Falke, 2019], we will use a pipeline approach, which mimics the best practice indicated for humans to build a concept map by [Novak and Ca˜nas, 2007]. Our pipeline would build from the one of [Falke, 2019] and could consist of four different sub-tasks, as illustrated in Fig. 2: (I) Mention and Relation Extraction, (II) Concept Grouping, (III) Importance Estimation, (IV) Visualisation.

In the first task, Mention and Relation Extraction, given a document, we could identify all mentions of concepts in the document and the relational edges among them [Falke, 2019]. This task could be considered an Open Information Extraction problem. Several OpenIE tools have been created to parse sentences and extract {subject, predicate, object} triples, as in [Angeli et al., 2015; Del Corro and Gemulla, 2013]. The second task, Concept Grouping, could consist of, given a set of triples, determining whether and which of the extracted nodes refer to the same concept [Falke, 2019]. A baseline approach is to group all mentions that are the same or all the morphological variants or synonyms of a particular term or to use WordNet 1 , together with a combination of lexical and vectors representation. To be more selective and create maps of appropriate size for the users, we could score triples and then use only a fixed subsection that scores highest. We could refer to this task as Importance Estimation. The highest concept will be the seed concept S. Finally, the generated concept map has to be stored against the input document and needs to be visualized. For this last sub-task, Visualisation, several libraries could be used, such as Cmap 2 .

7.2 Web Interface's Workflow

The web interface allows the subject to manually, or semiautomatically, construct the respective summary from the text, in the form of a concept map. The web interface is divided into two sections. On the left-hand side, there is the original text divided into sentences. The meaning and the most relevant entities evoked by the text could also be visualised by automatically generated images using popular algorithms such as Clip 3 . Moreover, the user can also listen to the audio version of the text generated with known API such as Google Text to Speech 4 . On the right-hand side, users can create the concept map. The concept map is not automatically displayed at the beginning. The construction of the concept map for the given text takes place in steps. The steps will depend on two criteria: the disabilities of the user and the readability of the text, as illustrated in Fig. 3. Two scenarios for both two targets will be possible. In the case of simple text, non-clinical groups will be requested to create a concept map from scratch, while the clinical group will be provided with the seed node, to avoid blank page anxiety. In the case of a difficult text, the non-clinical group will be provided with the seed node but they will need to continue themselves. The clinical group will already be provided with all the nodes extracted automatically by the AI algorithm. They will only need to select the correct label for the edge relationships among a set of alternatives provided by the underlying model. Learning, therefore, will follow the principles of intensity, specificity and repetition.

1 https://wordnet.princeton.edu/

2 https://github.com/ionstage/cmap

8 Evaluation Criteria

8.1 Evaluation of the Interface and the Model

A combination of existing automatic metrics and manual ones are used for evaluating the efficacy of the underlying computational model. The automatic metrics proposed by [Falke, 2019] could be computed to evaluate the model [Falke, 2019]. As a benchmark for evaluation, we could use the EDUC corpus [Falke, 2019]. Moreover, speech therapists evaluate the automatically created maps given by the underlying model for a set of documents over the ones created over the same texts during real-world therapies with children with similar clinical profiles. The user studies will evaluate also the usability and accessibility of the interface, as well as the satisfaction of those using the system (therapists and patients). Some of the dimensions that could be evaluated are: the degree of difficulty in using the system, how often the user would use the system, how much technical support is required by the user to use the system, how well the functions are integrated. The user experience of the system could then be analysed and information gathered on how easy or complicated, boring or motivating, interesting not interesting, creative or conventional the system is perceived to be. This feasibility study will have the goal of objectively and rationally uncovering the strengths and weaknesses of the tool.

3 https://huggingface.co/docs/transformers/modeldoc/clip

4 https://cloud.google.com/text-to-speech

8.2 Evaluation of Clinical Results

We intend to initially assess the written and oral text comprehension skills of the children who will use the platform through the use of standardized tests used in clinical assessment. The assessment protocol includes the administration of tests that assess the child's ability to understand a written text: the child reads the text independently and then answers multiple-choice questions, having the opportunity to consult the text. Sub-tests are also administered to specifically assess language skills, in particular, lexical and morpho-syntactic comprehension. The assessment is completed by a 'study test' in which the pupils study a text for 30 minutes and then identify several statements that best represent the content of the passage studied. They will later need to answer openended questions and true/false questions, without being able to consult the text. More precisely, we will use the following assessment protocol: BVL 4-12, battery for language assessment, Marini- Marotta, Giunti , 20: grammatical comprehension (test 10), lexical comprehension (test 9) Mt3 clinic, Cornoldi et al., Giunti, 2016: reading comprehension. New Guide to Text Comprehension, Volume 1: hierarchy to text, links, Amos 8-15: ability and motivation to study, CornoldiDe Beni, Erickson, 2005.

9 Mechanism for Collaboration and a Possible Roadmap

The Parties intend to cooperate to create an open-access webbased tool to help children with reading comprehension difficulties to interactively create a concept map and afterwards, have it tested by a selected sample. The Parties will negotiate a lightweight research agreement, in the same spirit as the DESCA model agreement for European projects. The Parties will define the specifications, features and usefulness of the tool, an agenda of cooperation and prospect on its eventual further use in the experiments for the target groups. Sony CSL Paris will act as Scientific Coordinator and oversee the software development and its adaptation to a functional webbased learning tool. CRC will use the tool in a set of user studies on a selected sample of children. During the experiments, the Parties will track progress and address issues that arise from their cooperation. The Parties will organize online meetings at least twice a month to discuss their progress. If deemed necessary, the Parties may organise meetings at their respective premises.

To ensure completion, the project collaboration will be divided into five different milestones. M1: the corpus will be selected, pre-processed and the readability measures will be implemented (from Month 0 to Month 1) M2: the CM-DS pipeline will be implemented (from Month 1 to Month 3). M3: the web-based interface will be developed (fromMonth 4 to Month 7). M4: the users' studies and evaluation will be conducted (from Month 7 to Month 8). M4: Finally, the results will be reported in a subsequent scientific paper (Month 8 to Month 9).

10 Risks

Given the variability of the type of material that can be inserted and processed within the system, there is a low risk that the clinician does not fully understand the output of the system. For this reason, it will be important to provide training for clinical staff (starting from the experimental phase) so that the tool is not used incorrectly, decreasing the effectiveness of the support provided.

Moreover, there is also a moderate risk that the tool would present an output that are too difficult for a certain functioning profile. This is also why, in the methods section, we decided to introduce parameters selection of the tool to beforehand inform the systems about who is using the tool and with which inputs so that appropriate exercises path could be selected.

Given that machine-learning models predict exactly what they have been trained to predict, the risk for a biased technology in this research collaboration is then moderate. The need to look for potential biases in the data will be particularly important when the data is presented to children in their learning age. This is why during the pre-processing phase, as explained in the methods section, a biased assessment will be done in the data to assess potential risks.

Finally, delays in important tasks will be communicated well in advance. This will provide ample opportunity to change the work to inform the two partners of all impending delays. Moreover, the spiral development model, adopted from the software engineering practices, will make sure that at end of each cycle, the previous one will already be tested. Delays caused by the pandemics are low, given that the two partners are based in two different countries and most of the work will be conducted online.

11 Ethic Considerations

Since this project aims to test its tools with children, the CRC partner, with decades worth of experience in this domain, will ensure that the strictest ethical standards are applied, especially for the safety and the privacy of all participants. The project will comply with the ethical standards of each of the partner's institutions, countries of origin, as well as the ethical guidelines of the European Union and the IJCAI conference. This includes the selection of age-appropriate teaching materials and data (e.g. in terms of a readability assessment and in terms of subject matter), as well as special attention to the privacy and protection of individual data. The project will correctly inform children and their parents or legal representatives about its goals and experiments so they can make an informed decision about consenting to participate in the study. A privacy impact assessment will be carried out beforehand to ensure privacy guidelines are implemented as part of the core design architecture.

The project will also follow the best practices that have been demonstrated to lead to positive learning experiences and feelings of well-being by children [Mai et al., 2011].

Moreover, the deployment of the tool is centred on the genuine needs of speech-language pathologists and children, rather than on commercial imperatives. This is why the tool and the algorithmic results of the research collaboration will be published under the GNU General Public License 2.0. which guarantees freedom to share and change free software. The final goal will be to make sure that the software is free for all its users.

Following a Human in the loop approach, all stakeholders will be involved starting from the design phase. Speechlanguage pathologists will ask for feedback as soon as a phase of the spiral development will be finished. Moreover, the final users, children from clinical and non-clinical groups will also taking part in the set of user studies.

Finally, we wish to emphasize that we strive for the project itself to be an inclusive collaboration experience. We are proud of the fact that the project already has women taking leadership roles, and we welcome researchers and teams from any background to join us in our dream of helping children in need.

Acknowledgements

We thank David Colliaux (Sony Computer Science Laboratories Paris) and In`es Blin (Sony Computer Science Laboratories Paris - Vrije University of Amsterdam) for valuable feedback and comments on the manuscript. We thank Pietro Gravino (Sony Computer Science Laboratories Paris) for valuable advice and comments. We thank Andrew Olney (Psychology Department, University of Memphis, Memphis, USA) for the interesting discussions. We thank Vittorio Loreto (Sony Computer Science Laboratories Paris - Sapienza University of Rome - Complexity Science Hub Vienna) for the constant support.