Get Out of the BAG! Silos in AI Ethics Education: Unsupervised Topic Modeling Analysis of Global AI Curricula (Extended Abstract)

1 Introduction

With the explosive expansion of Artificial Intelligence (AI), teaching ethics to developers is being stressed increasingly by educational, governmental, and industrial organizations. This work analyzes the patterns of teaching AI ethics and critically assesses their implications.

Teaching AI and AI ethics and research on how this is done have been there for more than half a century [Chand, 1974; Gehman, 1984; Martin et al., 1996; Applin, 2006; Ahmad, 2014], but the systematic assessment of the topics, developments, and trends in teaching AI ethics is a relatively recent endeavour. Previous research on a systematic analysis of teaching AI ethics suffered frommany limitations: 1) having a limited disciplinary scope [Saltz et al., 2019; Bielefeldt

The original paper is referenced as: Javed, R. T., Nasir, O., Borit, M., Vanh´ee, L., Zea, E., Gupta, S., Vinuesa, R. & Qadir, J. (2022). Get out of the BAG! Silos in AI Ethics Education: Unsupervised Topic Modeling Analysis of Global AI Curricula. Journal of Artificial Intelligence Research, 73, 933-965.

et al., 2019; Khademi and Hui, 2020; Towell, 2003] 2) having a limited geographical coverage [Hughes et al., 2020; Qadir and Suleman, 2018], 3) being biased towards Western cultures (e.g., [Moller and Crick, 2018], [Fiesler et al., 2020], [Garrett et al., 2020], [Raji et al., 2021], [Homkes and Strikwerda, 2009]); or including courses taught at only one single level (e.g., introductory level [Becker and Fitzpatrick, 2019]).

Our analysis is based on unsupervised topic modelling, where we use the computational lens of the topic model to understand the AI ethics curricula. We use topic modelling to automatically uncover hidden or latent thematic structures from a textual corpus. The uncovered topics are derived from groups of co-occurring words (i.e. words that frequently come up together, within and between documents) that are associated with a single subject (or theme), which is referred to as a topic [DiMaggio et al., 2013]. After identifying the hidden topics of AI ethics courses, we study the connection between these, the discipline of the department(s) giving the course, and core pedagogical concepts, i.e. Intended Learning Outcomes (ILOs) and their alignment with teaching modalities. In addition, we look at worldwide geographical trends, topic prevalence, and topic co-occurrence. Building upon this analysis, we distill a model of current pedagogical practices in the domain of teaching AI ethics. Our approach to analysing the use of pedagogical concepts in AI ethics courses is unique as, in contrast with previous research (e.g. [Fiesler et al., 2020; Saltz et al., 2019; Bielefeldt et al., 2019]), we anchor our study in well-recognized canons from pedagogy science, as explained in Section 2.3.

2 Methods

2.1 Data Collection

Our initial data was taken from a curated list of tech ethics curricula by [Fiesler, 2018]. At the time of our analysis (January 2021), this contained 259 courses. We filtered out all the syllabi that were not in English or that did not relate to AI. Thus, we retained only 123 courses. We decided to make the data more global and added 43 additional AI ethics courses. Hence, the final analysis used a corpus of 166 syllabi from around the world. Each course was related to a discipline, derived from the teaching department listed on Fiesler's list and/or the course's webpage. We clustered all the departments into four categories: "computer science", "humanities", and "law", while the label "multidisciplinary" was given to courses associated with at least two departments.

2.2 Topic Modeling

In order to understand better the syllabi in our dataset, topic modelling was performed via Latent Dirichlet Allocation (LDA) [Blei et al., 2003]. Our syllabi were in various formats: PDF, web pages (HTML), word documents, and text files. First, every syllabus format was turned into a text format. The text was then sanitized by removing stopwords and lemmatizing. In the end, lda mallet was used to extract topics from the corpus. We used topic coherence scores to find the best number of topics.

Topics were named based on the hybrid content analysis method from [Baden et al., 2020] by two experts in the ethics of AI and pedagogy, who assigned a label based on a close inspection of the 15 most probable words from each topic, the intertropical distance map, and content of the syllabi in the dataset.

2.3 Pedagogical Analysis

In this study, we used Bloom's taxonomy [Krathwohl, 2002] and Biggs' constructive alignment principle [Biggs and Tang, 2011] as the core of the pedagogical analysis. Bloom's taxonomy is a hierarchical model to classify educational learning objectives into levels of cognitive complexity: Remember, Understand, Apply, Analyze, Evaluate, and Create. In curricula design, the taxonomy is for devising the course ILOs. The prevalence of a given Bloom cognitive level within a syllabus was assessed from the frequency in this syllabus of lexicographic indicators (specific verbs) related to each level. Similarly, for assessing the prevalence of teaching activities, we used the teaching modalities corresponding to each Bloom cognitive level as identified in [Wong et al., 2019].

Biggs' constructive alignment principle states that the components in the teaching system, especially the teaching methods used and the assessment tasks, must be aligned with the learning activities assumed in the ILOs [Biggs and Tang, 2011]. In this study, we checked whether the words denoting teaching modalities are aligned with the words denoting a specific Bloom cognitive level (i.e. ILOs).

3 Key Results &Discussion

3.1 What is Taught and How

The 166 courses included in our analysis are taught at 105 universities around the world. We categorized the courses based on their level (i.e., undergraduate or graduate) and the department associated with the course. Our results indicate that the instances of the Law department teaching AI ethics are fewer compared to the instances of AI ethics courses taught by the Humanities and Computer Science departments. Moreover, the geographical analysis shows that the continent has an impact on the discipline of the department delivering the course. This difference is important because various disciplines might train for various types of professional groups (e.g., engineers, managers, scientists). A major difference among societies with regard to what department delivers AI ethics education can lead to different outcomes in terms of the operational capacity societies will have in the future for dealing with AI ethics matters and, therefore, how AI ethics is to be implemented differently by these societies. For example, a society where AI ethics education is delivered mainly by technical departments might foster the emergence of a techno-centric approach to solving AI ethics issues.

The LDA analysis uncovered 10 topics: five topics related to course administration and five to course content: T1 Media & society, T4 Philosophy, T6 Responsibility, T8 Data security, and T9 Applications (Figure 1). In order to focus the analysis on the actual course content and avoid being sidetracked by the administrative content, the administrative topics were hidden in the following analysis, and the remaining topics were normalized.

The heatmap displayed in Figure 2 shows the cooccurrence among topics since some topics tend to be used together in syllabi more frequently than others. This heatmap shows, given a dominant topic (set by the label of the row), the distribution of the prevalence of remaining topics. Analyzing the potential relations regarding the dominant topic and the prevalence of other topics, it is interesting to note that many of these relationships are asymmetrical.

Figure 3 answers the question: "What is the prevalence of the various topics depending on the discipline of the department associated with the syllabus?" This figure exhibits a form of discipline-based specialization.

Figure 4 answers the question: "What is the prevalence of the various Bloom cognitive levels for each given topic?" This analysis matches a-priori expectations one may have when relating a topic to a Bloom cognitive level.

Figure 5 answers the question: "What differentiates the syllabus from different departments in its formulation for each Bloom cognitive levels?" These results seem to confirm the stereotype that computer scientists are solution-oriented while those coming from humanities and law focus most on remembering and basic comprehension of facts and concepts.

Figure 6 answers the question: "What is the prevalence of the various topics in the syllabi of each continent?" To our knowledge, this analysis of what topics in AI ethics are taught across the globe is the first one of this kind.

As a means for assessing the pedagogical content of a syllabus, we also analysed the teaching modalities (i.e., what teaching activities are to be performed to achieve the ILOs), as mentioned in the syllabus text. The results indicate recurrent misalignments between ILOs and teaching modalities across continents and departments.

For assessing which of the topics uncovered by the LDA are best aligned with concrete ethical concerns, we compared the keywords linked to the identified LDA topics with the contents of the ACM ethics guidelines. As an evaluation criterion, for every topic, we summed the number of occurrences of each of its keywords within the ACM ethics guidelines. This analysis highlights imbalances between the main approaches for teaching AI ethics and the representativeness of the taught concepts within the ACM ethics guidelines (see Table 1).

3.2 In the BAG and Out

A model of how AI ethics is taught across the world can be derived from a meta-synthesis of the trends analysis results. This model, i.e. the BAG model (for Build, Assess, Govern), consists of three general approaches, each of these approaches relating specific Bloom cognitive levels, course content topics, and disciplines associated with a syllabus, as summarized in Table ??.

The BAG model shows that ethics education is still taught in isolated ways: specific disciplines, topics, and levels of ability. This is despite the general agreement that ethical and responsible AI design requires an interdisciplinary approach. Tomorrow's engineers are now trained with a focus on design but with limited insight into the legality or social desirability of their systems. Tomorrow's lawyers are now trained to see the risks and apply rules, with only minimal training on pragmatically weighing these risks over social benefits. Tomorrow's social scientists are now trained to see how AI can influence the dynamics of society, with only limited oversight on actual technical and engineering intricacies that can drive design decisions. This development in silos highlights an important issue that calls for action as a community if we want future generations to work together rather than against each other and to prevent society from becoming locked into undesirable path dependencies. Practitioners interested in renewing their practices could use the BAG model to identify where they position themselves relative to the three teaching approaches and try to cross topic bridges (e.g., Data security + Responsibility), Bloom cognitive level bridges (Remember + Apply), or discipline bridges (Humanities + Computer Science). As a matter of removing these silos and getting out of the BAGmodel, it seems relevant to strive for a form of holistic training, removing disciplinary, topic-oriented, and Bloom cognitive levels barriers

4 Conclusions

This study uses advanced statistical tools, such as a Latent Dirichlet Allocation (LDA) analysis and a meta-synthesis, to explore what is taught in AI courses around the world and how. It follows well-known canons from pedagogy science, i.e. Bloom's taxonomy and Biggs' constructive alignment principle. Unlike previous studies, our analysis covers a wide range of disciplines, regions (global level), and levels of education (undergraduate and graduate), and it uses automated statistical methods. The analysis indicates that there are numerous significant misalignments between the ILOs formulated for the courses included in our corpus and the teaching modalities put in place for training these ILOs. Furthermore, the analysis highlights the presence of three silos related to the pedagogy strategies for teaching AI ethics along a model which we call BAG (Build, Assess, Govern). The findings of this study highlight that current curricula perpetuate disciplinary mindsets and communities, which is suboptimal for the seamless design of systems that best serve society. This study suggests a solution: a generic-to-specific hybrid teaching and learning approach that connects different communities with a common understanding of the value of AI ethics.

Acknowledgments

LV acknowledges the support of the Knut and Alice Wallenberg Foundation (project number 570080103) and of the project AutogrAIde "A Student-Driven Interdisciplinary Hackathon on Whether and How to Automate Grading & Assessment" (project number 570002260, The R˚adet f¨or Artificiell Intelligens, Ume˚a University, Sweden). MB acknowledges the support of the project AFO-JIGG "Service design thinking to improve welfare and product quality in the Norwegian small-scale fishing fleet" (project number 302635, Research Council of Norway). JQ acknowledges the support of the project from Qatar University High-Impact Grant QUHI-CENG-23/24-127. SG acknowledges the funding provided by the German Federal Ministry for Education and Research (BMBF) for the project "digitainable".