Creating Dynamic Checklists via Bayesian Case-Based Reasoning: Towards Decent Working Conditions for All

1 Introduction

Checklists are extensively used in high-stakes decision making, such as in surgery or food inspections [Jelacic et al., 2020; Ho et al., 2018]. They are also used by government agencies in labour inspections, to survey individual organisations for non-compliance to working environment regulations [Dahl and Søberg, 2013; Karanikas and Hasan, 2022]. Such inspections are important to globally achieve UN's SDG 8, "protect labour rights and promote safe working environments for all workers", specifically SDG 8.8. Despite the inspection efforts, there are still 1.9 million registered deaths world-wide each year that are attributed to occupational health and safety risk factors [World Health Organization, 2021]. Our goal in this paper is to attack this problem by introducing dynamic checklists created via machine learning.

A conceptual view of a labour inspection checklist is shown in Figure 1. Here, a checklist consists of a subset of K of N items where each item corresponds to a specific regulation. Each item has a yes or no answer which indicates the inspected organisation's compliance to the corresponding regulation. Any non-compliant item found at an inspection is cited individually in a report which is sent to the inspected organisation, with an order to rectify the violations.

Each inspected organisation may be subjected to several hundred different regulations, which vary according to its size, location and industry. Prioritizing the correct regulations for inspections, in terms of risks to workers' health and safety, is very difficult. Assessing compliance to regulations is also time-consuming, so a checklist should ideally contain a small subset K of N possible items that are most likely to be found non-compliant at the inspection. Creating or using such checklists is difficult, partly because they are situation dependent [Ho et al., 2018; Catchpole and Russ, 2015].

In order to create optimized checklists efficiently, Flogard et al. [2021] propose to use machine learning (ML) to construct checklists and a new ML problem called the Checklist construction problem (CCP). They introduce a method called BCBR, which constructs checklists for labour inspections. BCBR uses Bayesian inference (BI) to construct features for cases used in case-based reasoning (CBR) [Aamodt and Plaza, 1994]. The input to BCBR is an organisation targeted for a labour inspection. BCBR then selects cases that contain a set of the K out of N possible unique items that are most likely to be found non-compliant at the given organisation. The output is an optimized checklist that consists of the selected K items.

Motivation. A shortcoming with BCBR is that it only creates static checklists, which can be inaccurate in many realworld situations where the context changes over time [Grigg, 2015]. For instance, new information obtained during an inspection could suggest that inquiries should be made into regulations that are not represented by the K items currently on the checklist on Figure 1. A possible solution for this

is to dynamically adapt the checklists during use. Figure 2 shows how a dynamic checklist can be adapted to a situation, inferred from how a user answers the checklist. This idea has been proposed and studied in clinical surgery trials [Kulp et al., 2021], where the current approaches for adapting checklists are based on process [Christov et al., 2016] or rulebased models [De Bie et al., 2017]. However, these models need to be built and maintained manually by domain experts, which is infeasible for creating the ideal high-detailed models needed for many complex tasks. Kulp et al. [2021] also discuss this limitation and highlight the need for a technical solution that can adapt checklists without relying on manually created models.

Scientific Contributions. There are two main contributions in this work. The first contribution is to establish ML as a means to create dynamic checklists. As far as we know, this is a difficult problem where ML has not been used before. A challenge here is to develop a method that is accurate and fast enough to make real-time dynamic adaptations of checklists.

The second contribution is a new method called Contextaware Bayesian Case-Based Reasoning (CBCBR), which is an extension of BCBR. The novelty, compared to BCBR, is a context-aware naive Bayesian inference model that enables dynamic adaptations to the constructed checklists during use. This means that CBCBR both creates new checklists and adapts them. The adaptations are done as recommendations of new items for the checklists. The recommendations are based on answers of the checklists, which can be considered a part of the temporal context of the situations where the checklists are used [Haruna et al., 2017]. CBCBR is also a fully transparent, online model that enables real-time creation and adaptation of checklists, which is crucial for their usability in real-world situations. Online learning models are known for their effectiveness in dynamic or non-stationary situations [Huleihel et al., 2021; Idrees et al., 2020], such as labour inspections.

Social Impact. Our work focuses on improving checklists used for labour inspections, which will have a direct impact on the promotion of decent work (see SDG 8). We show that CBCBR's dynamic checklists can improve task completion and increase the number of violations (non-compliance) found during inspections, which in turn will increase levels of compliance with working environment and labour rights and also reduce injuries (SDG indicator 8.8.1 and 8.8.2). More effective checklists could also reduce social dumping (SDG 8.5), human trafficking and forced labour (SDG 8.7).

2 RelatedWork

Checklist Construction. Until recently, checklists have been created manually by domain experts via approaches like the Delphi method [Morgan et al., 2007; Amaya et al., 2017]. However, creating checklists manually is difficult and time consuming [Hasson et al., 2000]. Instead of relying on human experts, Flogard et al. [2021] propose BCBR for constructing static checklists. In experiments, they show that labour inspection checklists constructed by BCBR outperform human domain experts and other ML methods. Zhang et al. [2021] propose an ML method for constructing checklists for medical diagnosis, assuming that a checklist is a binary M-of-N decision problem. They use an integer program to find an optimal checklist of N items that most accurately predict a diagnose, given that at least M items are checked. However, their method only creates static checklists and is also not usable for creating labour inspection checklists, as they are not binary M-of-N decision problems. An approach for generating checklist items for construction site inspections is proposed by Cai et al. [2020]. The approach is based on Natural Language Processing and generates new items from regulation document texts. Checklists are generated via SQL queries, by matching generated items with keywords specified by a user. The checklists are described as dynamic, but only in the sense that they are presented digitally, enabling users to manually customize them.

Context-Aware Recommender Systems. Research shows that dynamically elevating items on checklists can be effective for adapting them to context changes [De Bie et al., 2017]. CBCBR's recommendations can also be seen as a way to elevate context-relevant items and is inspired by contextaware recommender systems (CARS) and contextual modelling [Adomavicius and Tuzhilin, 2011].

Case-Based Reasoning. In recommender systems, CBR is used to address explainability and the long tail problem, which are known issues in collaborative filtering [Alshammari et al., 2017; Jorro-Aragoneses et al., 2020]. Reasoning in CBR (and CBCBR) is based on retrieving the best past cases to solve a given problem [Aamodt and Plaza, 1994]. To improve reasoning, Bayesian methods are used to infer missing case features or information in CBR systems [Nikpour and Aamodt, 2021; Kim et al., 2014]. For the same reason, CBCBR's cases are augmented with probability estimates to improve case retrieval. CBCBR also learns from answered checklists items, by using them for checklist adaptations or retaining them to create future checklists.

Summary. To our knowledge, there is no previous work where ML is used to dynamically adapt checklists. In our work, we show that checklists can be significantly improved by doing so, using answers to past items in a checklist to adapt future items in the same checklist. The approach is also generic and can likely be used in many other domains, since most checklists have some kind of answers recorded in them. Thus, our work can potentially be seen as a starting point for future ML research into dynamic context-aware checklists.

3 Definitions

Data Set and Cases. A data set D for variables Z = {E,X,L} is a tuple (d1, ..., dN) where a case dj ∈ D is an instantiation of Z [Darwiche, 2009]. A case can be defined as a tuple d = (e, x, l) where e denotes a single item of a checklist, x is the target organisation for the item and l ∈ {0, 1} denotes the binary answer to the item. A case in the data set can be viewed as a past experience where an item e has been applied to x to obtain the answer l. Any e, x and l are instantiations of the variables E, X and L, respectively.

Organisation. Every case in the data set contains a target organisation description x, that consists of multiple subfeatures. These sub-features are all categorical and describe the organisation's location and industry [Flogard et al., 2021]. For brevity we treat x and X as categorical in our work.

Item. Each case in the data set contains an answerable item e, used to survey the organisation x. There are N unique items in the data set, so an item e is a categorical value that may appear in multiple cases. Thus, E is also categorical.

Checklist. The item in each case of the data set belongs to a checklist y, which is a solution applied to the organisation x. A checklist consists of a set of items such that y = (e1 ∈ d1, e2 ∈ d2, ..., en ∈ dn). An item can only occur once per checklist, such that ei ̸= ej for every ei ∧ ej ∈ y. Thus, y consists of items frommultiple cases.

Answer. The label l ∈ {0, 1} of each case dj ∈ D is the recorded answer from applying the item e to the organisation x. A positive answer (non-compliance) is observed if l = 1.

The Checklist Construction Problem. Given a candidate target organisation xcnd , a model M needs to select K out of N unique items (e1, e2, ..., eK) for an initial candidate checklist ycnd . Each item ei ∈ ycnd needs to be selected so that it maximizes the probability for observing the answer li = 1 when applied to the targeted xcnd .

Dynamic Adaptation of Checklists. Let (l1, ..., li) be the observed answers to the items (e1, ..., ei) ∈ ycnd where i ≤ K. Given the target organisation xcnd and the answered items, adapt the checklist so that the posterior probability for observing a positive answer to any unanswered items on the changed checklist is maximized. The adaptation could be done by adding or removing items to the checklist in many ways. In this work adaptation is done by recommending any additional M of N items ˆ e /∈ ycnd that have higher posterior probability for observing ˆl = 1 (given (l1, ..., li)), compared to any existing items on ycnd . The M recommendations are appended to the existing K items of ycnd .

4 CBCBR Framework

The purpose of CBCBR is to create a dynamic checklist for any given xcnd , by retrieving past cases with items used in similar organisations, and with high estimated probabilities for non-compliance. Answering the checklist may change the estimates, triggering dynamic updates to the checklist.

An overview of CBCBR is shown in Figure 3 and consists of the following steps: (1) a naive Bayesian inference (NBI) model generates probability estimates (θbe ) for answers based on empirical distributions of the data set D. (2) A case base CB of augmented CBR cases cj is created by adding θbe as feature to the instances dj ∈ D. (3) Similarity based retrieval is used to retrieve K cases from CB, given a query q that contains the target organisation xcnd and a fixed target value for θbe in the cases. The retrieved cases contain the items for the initial candidate checklist ycnd , as illustrated by the white arrows on Figure 3. (4) The novel recommendation part for adapting ycnd is shown by the blue arrows in Figure 3. First, an item on ycnd is answered, which prompts CBCBR to refresh the CB with updated posterior θbe estimates (via NBI). CBCBR then retrieves any new cases from CB with sufficiently increased θbe , containing M recommended items which are appended to ycnd .

Further details for how θbe is calculated by the NBI model in steps 1 and 4 are covered in Section 4.1 and 4.2, respectively. The other details for step 2-4 are found in Section 4.3.

4.1 Naive Bayesian Inference for Checklist Construction

NBI is based on using empirical distributions of the data set D to estimate the probability for the event L = 1|x, e, expressed as the mean of a Beta distribution [Darwiche, 2009]:

where L = l, X = x and E = e denote the event where the outcomes l, x and e are observed. Both (D#(L = 1 ∧ X = x ∧ E = e) and (D#(L = 0 ∧ X = x ∧ E = e) are parameters for the Beta distribution, and ψL=l|x,e denotes the prior parameters. Equation 1 is used to calculate the probability estimates needed for selecting the optimal items when constructing the initial checklist ycnd .

4.2 Naive Bayesian Inference for Item Recommendations

After answering an item ei on the checklist ycnd , the obtained answer li holds evidence that can be used to update the posterior belief in any unobserved answer ˆl for a candidate item ˆ e. 1 This assumption can be used to update the probability estimates from Equation 1, by estimating new additional Beta distribution parameters with the following equation:

The parameters are made by counting cases in D. Each case in D is counted if it contains the given x, ˆl ∈ {0, 1}, ˆ e and if there exist at least one other case d = (x, ei, li) and a past checklist y in D such that both ei and ˆ e exists in that checklist. The posterior estimates can now be updated by inserting the parameters above into the Beta distribution from Equation 1:

where βL=ˆl|x,ˆ e = D#(L = ˆl ∧ X = x ∧ E = ˆ e) + ψL=ˆl|x,ˆ e . The θbe estimate in Equation 3 is calculated by summing the parameters for every ei ∈ ycnd that has an observed answer li. It should be noted that the equation above "naively" assumes that all applied items ei are mutually independent of each other given x. This is done to decrease the amount of θbe estimates based on low or zero case counts [Darwiche, 2009]. Equation 3 is an important technical contribution of this paper, since it enables online adaptations of checklists.

4.3 Case Base and Retrieval of Checklist Items

This section defines the details for the augmented CBR cases, case base and similarity based retrieval from Figure 3.

Creating Augmented CBR Cases and Case Base. Algorithm 1 shows the creation of a case base CB with augmented cases c. The κ feature is included to adjust for the case counts of the probability estimates when retrieving cases [Flogard et al., 2021]. The algorithm iterates through each case dj ∈ D and creates κ and θbe estimates for the case via Equation 1. Both θbe and κ are conditioned on the case features x and ej and added as features to each dj , to create cases c for CB.

Case Retrieval for the Candidate Checklist. To retrieve items ei for the candidate checklist ycnd , a query case q and a similarity function are used. The query consists of the target organisation xcnd and the desired values for both the probability estimates and the case count features. The similarity function assigns a score Sim(·, ·) ∈ [0, 1] to every pair (q, cj ∈ CB). The function is the same linear weight similarity function used by Flogard et al. [2021]. The function is applied to every cj ∈ CB and a set of unique items (e1, ..., eK) is then retrieved from the K cases with the highest similarity score, to create an initial checklist ycnd with K items.

1 In this setting, ˆ e is a potential candidate for recommendation.

Algorithm 1 Creating or updating CB with cases c containing new θbe estimates.

Input: D, ycnd ; //ycnd is only included when updating CB with new posterior θbe estimates after answering items.

Output: CB ← (); //Initialize CB.

for each dj ∈ D do

//Both xj and ej are found in the current case dj.

end if

c ← Join(dj , θ, κ); //merge dj , θ, κ to a single case.

CB ← Insert(c); //Add the new case c to CB.

end for

return CB; //The CB is now ready for any case retrievals.

Case Base Update and Retrieval for Recommendations. Algorithm 1 also shows how the case base is refreshed after answering items on the checklist. This procedure is done after an item ei ∈ ycnd has been applied to xcnd . First the existing CB is cleared for cases. Then updated probability estimates are calculated for each case dj ∈ D via Eq. 3, conditioned on xj , ej and every ei ∈ ycnd with an obtained li. The newly calculated θ and κ features are concatenated to each dj to create the updated cases c for CB.

The updated cases are then retrieved for recommendation. First, the function Sim(q, cj) is again applied to q and every cj ∈ CB. A filter is then applied to remove all cases with similarity scores less than any of the K cases that were selected for the initial ycnd . Duplicated items are also removed so that M cases with unique items remain. The size of M is usually very small (between 0 and 4) and is not known or fixed, but simply depends on how many eligible cases that remain after applying the filter. If there are M > 0 remaining cases, the items from these cases are recommended for the checklist.

5 Demonstration and Assessment of CBCBR

Data Set. We use the Checklist data set introduced by Flogard et al. [2021], which has 1 111 502 entries dj . The entries constitute 63 634 inspections conducted with 369 different unique checklists, which in turn contain N = 1 947 unique items in total. The checklists cover a wide range of industries and the items on each checklist typically have an inspection topic that specifically relates to a few intended target industries. Mengshoel et al. [2021] also use a similar data set.

Query. We use the configuration from Section 6.1. A recommendation is done after answering each item. The query is q = (xcnd , θ, κ) and contains the desired values of the retrieved cases, where θ is set to 100% and κ to 70. The target organisation for the inspection xcnd is a hotel in Oslo.

Constructed Checklist and Item Recommendations. Table 1 shows the constructed checklist ycnd for the target hotel xcnd where two of the items have been answered. The answers were randomly selected. The initial size of the checklist is K = 5 items, however two items have been added as recommendations based on the answered items. Further items may be added after more answers are given. The added items increases the checklist length, but allow the inspector to focus more on highly relevant risks in the inspected organisation.

Qualitative Assessment. The items in Table 1 cover a variety of topics such as working hours, overtime payment and violence. These are all common risks for the hotel industry, especially in a large city such as Oslo. The added items shown in the table are also relevant and closely related to the two answered items on the top of the checklist. Thus, the recommendations align well with the findings at the inspection. CBCBR also performed well when we, with domain experts, assessed a variety of other inputs (xcnd ) and lengths (K).

6 Experiment

In this section we conduct an experiment to measure the performance of CBCBR against BCBR and other baselines.

6.1 Experimental Setup

CBCBR and BCBR Configuration. Both CBCBR and BCBR generate the exact same initial checklists, but BCBR does not perform any additional recommendations or dynamic adaptations. The two methods use the same configuration as Flogard et al. [2021]. The NBI models for CBCBR and BCBR use fixed priors ψL=1|x,e = 1 and ψL=0|x,e = 5. For every query q, the target matching value for θbe is set to 100%. The target value for κ is set to 70. The NBI models are created and updated via MSSQL17 queries and stored as Python data frames. The similarity based retrieval is implemented via MyCBR [Bach et al., 2019] for both methods. For performance reasons, CBCBR recommendations are

made each time 5 items on the checklist have been answered.

Generating Checklist Answers for Predictive CBCBR Recommendations. To generate predictive recommendations for CBCBR in the experiment, we introduce a simulator. The approach is similar to user-behaviour simulations in recommender systems [Zhang and Balog, 2020], as the goal of the simulator is to perform a realistic walk-through of an initial checklist to obtain an answer li for each item ei ∈ ycnd . These answers are then used to generate realistic item recommendations in the experiment below. This is done as follows:

1. For a given target organisation xcnd , CBCBR (model M) first constructs an initial checklist ycnd with items ei. Each ei is retrieved from a case ci ∈ CB and has an unobserved answer li.
2. For each ei ∈ ycnd , a value for li needs to be generated by drawing from the set {0, 1}.
3. To do so, the parameters bi of a Bernoulli distribution B for li are estimated empirically from CB using the following expression: bi = CB#(L=1∧X=xcnd∧E=ei) CB#(X=xcnd∧E=ei) .
4. If ei is an item that has been recommended earlier in the simulation, the recommendation context is taken into account when calculating bi. This is done by applying Eq. 3 to CB (instead of D): bi = θbe (L = 1|xcnd , ei, ycnd ).
5. The value li ∈ {0, 1} is now drawn from a Bernoulli distribution, given by B(bi, 1 - bi). If 5 values have been generated since the last recommendation, then proceed to the next step. Otherwise, go to Step 2.
6. After 5 li-values are generated, CBCBR is updated with the new values in order to perform a recommendation. Any recommended items are then automatically appended to ycnd . After the recommendation, go to Step 2 if there are any unanswered items left in ycnd .

After the procedure above has been applied, the extended checklist is ready for evaluation.

Baselines for the Experiment. We use these following baselines to generate static checklists: Multi layer perceptron (NN), Random forest (RF), Naive Bayes inference (NBI), Decision tree (DT) and Logistic regression (LR).

Each baseline (M) above generates a checklist of K items as follows: (1) M is first trained on a training set DT of D, using l from each instance d = (e, x, l) as the target label. Each M is trained with the goal of correctly classifying the value of l, given x and e. (2) Given a validation set DCB of D and an input xcnd , M generates a prediction score ([0, 1]) for every N possible items. Predictions are only generated for items (e) with at least one corresponding instance (e, x, l) ∈ DCB where x = xcnd . (3) The K items with the highest scores are selected for the candidate checklist ycnd .

We also use the original, domain expert created checklists (ECL) from the Checklist data set as baseline.

Baselines Configuration. The NBI-baseline is calculated using Equation 1 and is implemented via MSSQL17. The other baselines are implemented via Sklearn, using the default configurations for most of them. For the NN and RF baselines we used GridsearchCV (Sklearn) for hyperparameter tuning. The optimal configuration for NN was 20 layers, logistic activation function and L2 penalty of 0.0001. The optimal configuration for RF was 50 estimators, bootstrapping sample size of 50% and minimum sample size of 10 for node splits.

Environment. The experiment is conducted on a fully upgraded Dell Precision 5560 with Intel i9 11950h and 64GB RAM, in a Python environment using Scikit-learn (Sklearn).

6.2 Evaluation of CBCBR's Dynamic Checklists

The goal of this experiment is to evaluate the performance of CBCBR against BCBR and other baselines.

Method and Data. The experiment is done on the data set D from Section 5. An 8-fold cross-validation is used where D is partitioned into training folds (DT ) and validation folds (DCB). DT is used to calculate any probability estimates needed to select or recommend items for a checklist ycnd . DCB is used for the case base (CB) and for performance evaluation. Each validation fold has 138 938 instances that constitutes 7 954 inspections. Each inspection contains a target organisation x that is used as input (xcnd ) to each checklist construction model M. Each model M then generates a candidate checklist ycnd for each given x as described in Section 4 and 6.1. The target length of ycnd is set to K = 15. 2

Statistics for all checklists created by each M are calculated as follows: For each generated ycnd , all items ei ∈ ycnd are considered as having predicted positive answers li = 1. The rest of the N possible items where e /∈ ycnd are considered as having predicted negative. The number of true positive (TP)/false positive (FP) answers and true negative (TN)/false negative (FN) answers of ycnd are estimated from empirical distributions of DCB, as described by Flogard et al. [2021]. This is done because every ycnd is a new checklist, so most items ei ∈ ycnd do not have an observed ground truth answer li. The estimates are used to calculate accuracy, precision and recall statistics for each ycnd via: Accycnd = TP+TN TP+FP+TN+FN , Precycnd = TP TP+FP and Recycnd = TP TP+FN . The final mean Acc, Prec and Rec for each validation fold are found by averaging each statistic over all 7954 generated ycnd . We also included Prec (gt), which is mean precision calculated by only using the subset of items on the checklists where ground truth answers are available. The statistics are comparable to mean average (MA), used to evaluate recommender systems [Natani and Watanabe, 2021].

Results and Discussion. The results are shown in Table 2. CBCBR has the highest Avg score and has higher scores than BCBR on most of the statistics. This is impressive as CBCBR's checklists are longer and should generally have lower Acc, Prec and Prec (gt), since only 17.7% of the instances in the data set have positive labels (l = 1) and longer checklists means more predicted positives. NN has the third highest Avg score, only marginally higher than NBI. It may be possible to improve the results for NN with more advanced parameter tuning methods, but this will unlikely be enough to match BCBR or CBCBR's performance. The domain experts' checklists (ECL) have the lowest score.

2 For CBCBR, each generated ycnd also includes recommended items. The initial length of CBCBR's checklists are K = 15, before adding any extra items. The code for the experiment is published at https://github.com/ntnu-ai-lab/cbcbr.

On average, the checklists constructed by CBCBR contain 9.13 violations (true positives) against 7.19 for BCBR. CBCBR's checklists also contain 18.1 items against 15.0 for BCBR. This means that CBCBR recommends 3.1 items on average, because CBCBR creates the same initial checklists as BCBR. There are 1.94 violations found among the 3.1 recommended items, which corresponds to a precision score of 0.63. Thus, CBCBR's recommendations have much higher precision than static checklists created by BCBR or other baselines in Table 2. Overall, the results suggest that using CBCBR's dynamic checklists for labour inspections will increase efficiency and the amount of violations to working environment regulations found at the inspected organisations.

Time-wise, the slowest method is NN with an average training time of 14 280 seconds, excluding hyper parameter optimization. The fastest method is NBI with 7.2 seconds training time. CBCBR is nearly as fast, as the combined time for model training and generating an initial checklist is 8.6 seconds. This is much faster than LR and DT, which are known for their low training times. It also takes 4.5 seconds to update CBCBR and recommend new items after answering a checklist. However, the cross-validation took more than 20 days to complete because CBCBR must be trained and updated individually for each validation case. Still, the short individual training and update times for each checklist means that CBCBR can be used as an online model and we believe that further speed-ups can be achieved via parallelization.

7 Conclusion

In this work we show that dynamic answer-based adaptations to checklists can significantly increase the number of violations found in labour inspections. We introduce CBCBR for real-time generation and adaption of checklists, which could be employed by labour inspection agencies to increase the efficiency of their inspections. This will likely increase national and global levels of compliance with labour rights and reduce injuries (see SDG indicator 8.8.1 and 8.8.2). We are currently testing the real-world performance of CBCBR in a trial.

This paper only explores one of many possible approaches for dynamically adapting checklists. Future work could investigate other designs as well. An example is to dynamically build checklists from ground-up, starting with one item and adding K more, one-by-one as answers are obtained. Another direction for future work is to explore approaches for creating checklist items, where Cai et al. [2020] could be a starting point. It could also be interesting to look into other ML methods for dynamically adapting checklists, such as RNNs. CBCBR's dynamic checklists could also be tested in other relevant tasks, such as food inspections, aviation or surgeries.