Pseudo-Labeling Enhanced by Privileged Information and Its Application to In Situ Sequencing Images

1 Introduction

In many real world applications, the type of available information for data samples is not direct and the existing indirect evidence cannot be exploited using methods developed for datasets with direct ground truth. Creative exploitation of this side information into a learning framework might significantly advance real world applications, especially in life sciences. These applications often have unique properties that require unique strategies. Nevertheless, these domain specific strategies could provide potential generic insights applicable to other fields. This work is motivated by a problem called barcode calling, in which we attempt to decode information from In Situ Sequencing (ISS) microscopy images.

In situ sequencing methods are rising in popularity across genomics due to their application in spatial transcriptomics and optical pooled profiling screens. Their ability to preserve the spatial information of biological measurements while genotyping single cells at scale without the need to physically harvest and dissociate cells, adds rich information to profiles of biological samples [St˚ahl et al., 2016; Vickovic et al., 2019; Asp et al., 2019]. The field is exploding with new and improved genomic perturbation and spatial barcoding technologies. These technologies create a series of images to be analyzed computationally for recovery of the experiment's encoded barcodes. Each barcode is a sequence of base letters that are encoded in a sequence of images (each letter is an object in each image). Each barcode is associated with a cell in the experiment.

Numerous attempts have been made to maximize the accurate recovery of barcodes over the past few years (see Section 2.1). Due to the simple structure of information encoded in these images and the lack of existent ground truth , traditionally barcode calling pipelines are formed by a series of expert-supervised multi-step image processing steps. However, they not only need expert optimization of parameters, but also are far from achieving maximal accurate detection of barcodes. Improvements would be extremely valuable, considering the cost and effort involved in performing extra experiments to compensate for missed signal and therefore statistical power of the downstream analysis. Recently, new methods [Chen et al., 2021; Andersson et al., 2021; Gataric et al., 2021] have attempted to model the physical process and find the optimum fit to an expected set of barcode solutions which is called an experiment's codebook. A codebook tells us the set of possible barcodes existing (at unknown locations) in a set of images. It thus provides an indirect evidence for the decoding solutions, which may be treated as side-information. It is highly labor-intensive to generate even a small amount of direct ground truth for this problem, because it would effectively require generating distinct datasets for each barcode, which is impractical except at a very small scale. Also, human observers cannot perform the task accurately on challenging instances, in light of the noise sources. The absence of such ground truth also limits strategies for evaluating performance and prevents consistent and fair comparison of new methodologies by well-defined metrics on benchmark datasets.

In this work, we introduce a novel approach for framing the barcode calling problem, which also contributes to the deep semi-supervised learning and object detection literature. The proposed framework takes advantage of special domain specific characteristics of this problem and dataset to form a novel learning strategy. We propose forming noisy labels by computationally cheap operations and then exploiting extra evidence available on data into the pseudo-labeling process in a semi-supervised object detection (SSOD) framework. This evidence, which could be in various forms and not limited to the examples we demonstrate here, are then used to adjust the initial decision boundary formed by noisy labels, during the self-training iterations. We name the overall strategy of incorporating privileged information in the pseudo-labeling process, Pseudo-Labeling Enhanced by Privileged Information, and the algorithm specifically developed for barcode calling (that relies on the application of this strategy into a SSOD framework) is called PLePI-ISS.

We can summarize our contributions as:

• We propose PLePI as a novel strategy for incorporating complementary extra available information (termed 'privileged information') for a problem into the pseudolabeling process. This strategy is shown to not only salvage a model suffering from noise overfitting but also enhance the regular pseudo-labeling process in a semisupervised learning framework.
• We frame the crucial problem of barcode calling in the spatial transcriptomics field as a semi-supervised object detection problem with noisy labels and available privileged information. We also provide a public benchmarking resource which can be used for evaluation of the novel methodologies addressing this problem. The introduced framework, PLePI-ISS, which is the first endto-end nonlinear framework to address the barcode calling problem, is evaluated in an out-of-sample fashion for the first time using the provided benchmark (whereas previous learning-based methods have only been evaluated within sample).

2 RelatedWorks

2.1 In Situ Transcriptomics Decoding Algorithms

Decoding barcodes from In-Situ Sequencing images or barcode calling is defined as reading sequences of cDNA copies of mRNA fragments in an image-based experiment. These fragments are spots scattered at various image locations. Each barcode is a length Nr sequence of four A, T, C, G nucleotides, or 'base letters', where Nr is typically 9-16. A reference library of barcodes or codebook (which is a subset of all possible combinations of length Nr sequences of base letters) is designed and known before each experiment begins.

Each letter in the sequence is captured by one round or 'cycle' of fluorescence microscopy images. Barcodes are formed by reading letters at a fixed location (or 'spot'), in a series of images taken at consecutive sequencing cycles. The color of each letter (or 'spot') corresponds to the specific intensity distribution of that spot across four base letter color channels. An example of a cycle image is shown in Supplementary Figure 1 1 , in which the top row shows a zoomed single spot to be decoded to one of the base letters, based on its intensity in the four color channels corresponding to A, T, G, or C. Despite the structural simplicity of these ISS images, many practical complications exist, such as background from the cellular matrix, variable focus quality in thicker sample mounts, imbalances of brightness across color channels, bleed-through of signal from one color channel to another or from one cycle of imaging to another.

Barcode calling algorithms are typically a multi-step expert-supervised process. Images are first corrected for general uneven microscopy patterns (due to uneven illumination or detection) and then aligned across channels and across rounds or cycles of imaging. The crucial step then is to recover the signal from noisy measurements. Signal here refers to the location of spots and their label. Previous literature has used various techniques to overcome this challenge by modeling and correcting the different sources of noise [Andersson et al., 2021; Gataric et al., 2021; Chen et al., 2021; Senel et al., 2022]. Various bottom-up or top-down processing approaches have been proposed and implemented as pipelines in the open-source Starfish [Axelrod et al., 2021] Python library. Bottom-up or spot-based approaches first determine the existence of a spot and then estimate the barcode in the spot across all rounds of imaging [Shah et al., 2016; Wang et al., 2018; Gyllborg et al., 2020]. On the other hand, top-down or pixel-based approaches look for the existence of a barcode in the experiment's codebook to determine the signal versus noise [Andersson et al., 2021; Chen et al., 2021; Lee et al., 2014; Moffitt et al., 2018]. Among recent pixelbased decoding approaches, BarDensr models the generative physical process linearly, considering various factors including the probe response function, phasing effect, cross-talk, background noise, and the codebook of barcodes. It estimates model parameters from data and solves a sparse non-negative regression problem by constrained optimization [Chen et al., 2021]. ISTDECO suggests optimization of a single model for simultaneous barcode localization and decoding for further denoising and barcode calling improvement [Andersson et al., 2021]. A more recent spot-based model called PoSTcode [Gataric et al., 2021] suggests probabilistic modeling of data by a matrix-variate Gaussian Mixture Model and estimates posterior probabilities of barcodes given the codebook and image measurements at spot locations. Without suitable ground truth, the field has not been able to compare methods against a unified benchmark.

2.2 Object DetectionWith Limited Annotations

Deep learning-based object detection in the presence of ground truth annotations has shown outstanding success. However, the labor-intensive and costly nature of collecting annotations in object detection, has motivated the recent attempts to study various learning strategies with limited supervision. Weakly supervised object detection (WSOD) studies the strategies to use image-level categorical annotations without location information to detect objects [Tang et al., 2018;

1 The source code and supplementary materials for this paper can be found at http://broad.io/PLePIISS

Dong et al., 2021; Lin et al., 2020; Shen et al., 2020]. Semi-supervised object detection (SSOD) leverages a small set of fully annotated images together with a large set of unannotated images for detecting objects in the unannotated data. SSOD attempts have mainly followed variations of the well-established strategies in semi-supervised image classification such as pseudo-labeling and consistency regularization [Jeong et al., 2019; Jeong et al., 2021; Sohn et al., 2020; Kuo et al., 2020]. However, due to the difficult nature of object detection relative to the image classification problem, the literature on SSOD is limited, and it lags behind semi-supervised object classification. Various other strategies have been also explored for label-scarce object detection; sparsely annotated object detection deals with partial annotations in each image [Wang et al., 2021a; Niitani et al., 2019; Zhang et al., 2020], single instance object detection takes annotations of a single instance per category [Li et al., 2022b], point-supervised object detection replaces box-supervision with single quasi-center point supervision for saving annotation costs [Chen et al., 2022]. These variations all aim to achieve a performance close to fully supervised object detection by using minimal/sparse levels of supervision. Our proposed strategy could benefit any categories of the work mentioned that involve pseudo-labeling where we have access to a source of privileged information.

3 PLePI

Here, we describe our proposed domain-free strategy for enhancing pseudo-labeling by a source of privileged information. Following the related works [Zhou et al., 2021; Xu et al., 2021; Li et al., 2022a; Zhou et al., 2022], we build on a teacher-student scheme, where both teacher and student share identical architecture, and the teacher's weights are updated iteratively by exponential moving average (EMA) of the student's weights. The teacher model makes decisions based on weakly augmented unlabeled data and generates labels for strongly augmented data inputted into the student model. This process, which is called input consistency regularization, ensures gradual improvement of the student and therefore the teacher in a dual iterative learning scheme. Without loss of generality, we describe the enhanced pseudolabeling algorithm for a single learning task and loss function.

Assuming sets of labeled (Dl) and unlabeled (Du) samples, the teacher model is first trained by the set of labeled data. The student model is trained by using both labeled and unlabeled sets to minimize a linear combination of, respectively, a supervised and an unsupervised loss; ℓs +λuℓu, where λu defines the contribution of unlabeled samples to the student's overall loss function. In each training iteration, the teacher model generates a set of class conditional probabilities PT for each sample in a mini-batch of samples. In a typical pseudo-labeling by confidence thresholding approach, we pseudo-label samples where their likelihood of belonging to any class k is higher than τc. These pseudo-labels are then fed to the unsupervised loss to update the student model using a cross-entropy or focal loss which minimizes the distance between the student's soft predictions and the teacher's generated pseudo-labels. We call the samples selected for pseudo-labeling, the set of confident samples OC = {o|max k∈K PT (yo = k) > τc}, where K refers to the full set of class labels. Here we define an additional mediocre set of samples with lower levels of confidence (τm ≤ τc), OM = {o|τm ≥ max k∈K PT (yo = k) ≥ τc}. When privileged information is available, the teacher fuses its own soft predictions (PT ) into each mediocre sample om with the evidence the privileged information provides (EPI) on the label of that sample. We model the probability of a sample label using both types of evidence as:

The choice of τm, τc, and their relative distance may vary depending on the specific problem and context. We assume privileged information could provide evidence or could be converted to the form of probabilities for a group of objects co-occurring in a mini-batch of samples. Hence, for a set of O = {oj}j of objects in a mini-batch B, privileged information could provide {PPI (OS)|OS ⊂ O}, where OS could be any possible subset of objects in B.

Then, we fuse the evidence provided by the current teacher's model (ET ) and privileged information (EPI), to form decisions for each object in the mediocre set, as also shown in the schematic overview of the strategy in Figure 1.

Thus, for every sample in the mediocre set, om ∈ OM, we estimate its label k∗ as:

As mentioned, K refers to the full set of class labels, however, we recommend limiting K to a set of top n candidate labels for each om according to PT to avoid unnecessary computational overload. The algorithm in compact form is presented in Supplementary B 1 .

4 PLePI-ISS

4.1 ISS Data-Structure

The images captured in each sample well of an experiment consist of Nf field of views, {If}Nf f=1 , where each If is a sequence of images captured at Nr rounds or cycles of the experiment; If = {If,r}Nr r=1 . Therefore, for each site of the experiment, there exist Nf ×Nr number of I ∈ RW×H×C images of height H, width W with C color channels. A barcode is formed by a sequence of Nr objects at a specific fixed location across If images.

We frame the barcode calling problem as an object detection problem on single-cycle images in the presence of noisy labels and privileged information.

4.2 Domain Specific Properties

Here we describe data properties that make our SSOD problem distinct and therefore provide the rationale for our design choices, which may not align with conventional SSOD techniques utilized for natural images.

Unique Structure

Our data has an unusual structure which is different from natural images that are the target of most SSOD designs. Compared to natural images, the objects' structure is much simpler which creates limitations on choices of augmentation and therefore regularization capacity which is required in a self-training learning strategy. Objects are very small spots in various forms densely scattered throughout each input image (Supplementary Figure 1 1 ). The label of each spot is shape independent and manifested by the arrangement of intensity distribution across color channels. Slight perturbations in these arrangements could create noise in predictions. On the other hand, this simple structure might allow extracting noisy labels by cheap and efficient image processing techniques. The size of objects (4-10 pixels in each dimension) allows using point-level annotation which is cheaper to extract compared to box-level annotation. Another unique property is the consistent location of objects across various cycles of the experiment (i.e. all objects corresponding to each barcode have similar location labels), which allows model regularization for the object detection task in lieu of intensity-based data augmentations.

Self-training and Noisy Labels

The basic idea behind self-training is that a model can improve itself by learning from its own confident predictions [McLachlan, 1975; Scudder, 1965]. The confident predictions are then added as labeled samples to the next training iteration. The initial model that is called a "pseudo-labeler" is typically trained using a source of confident labels. If a pseudo-labeler is fitted to noisy labels, the model would overfit to those labels and can't rely on its own confident predictions to improve its learned decision boundary [Arazo et al., 2020]. Strategies for alleviating noisy pseudo-labeling due to the confirmation bias in a poorly calibrated model are very limited [Arazo et al., 2020; Li et al., 2020; Rizve et al., 2021] and domain-specific. The fusion of privileged information to the model's decision could be a remedy to this problem in our application. By PLePI, a model learned from noisy labels corrects its formed decision boundary by adjusting pseudolabels in a mini-batch of data using a source of privileged information. We emphasize that, although the proposed strategy can improve the quality of the pseudo-labels generated in any self-training loop, it is particularly beneficial to a model suffering from noise overfitting which otherwise fails to benefit from unlabeled data through self-training iterations.

Codebook

As described in earlier sections, the experiment-specific codebook gives prior information on the expected set of solutions for the barcode calling problem. However, it doesn't provide any direct information on the label of each spot to be labeled by an object detector.

4.3 Privileged Information in ISS Barcode Calling

According to {PPI(OS)|OS ⊂ O}, evidence provided by privileged information is increased when we have more sample subsets that can provide information about each other in the form of a group (within a mini-batch of samples). Therefore, to maximize the amount of information the teacher has access to in each training iteration, we form a mini-batch by grouping images that can provide the maximum information about each other. In ISS domain application, this translates to tiles of images across various cycles of a single field of view.

Shared evidence on object locations across images. As mentioned earlier, the location of objects across images of various cycles in an experiment remains constant. Slight variations in object locations in these images are due to imaging noise and misalignments across cycles.

We can exploit this data property as an evidence on the location of a group of spots, which states that the probability for a group of spots at a fixed (x, y) location in various cycle's images is one if and only if the objects in the group have either all foreground or all background labels.

Codebook as an external source of evidence. Labels for each foreground object at a specific location across all cycles form a barcode sequence which should exist in the reference barcode library of barcodes (the experiment's codebook) in a noiseless scenario. However, due to experimental imaging noise, the formed barcode may not match any of the barcodes in the codebook, but it could still be assumed to remain in the neighborhood of its ground truth barcode in a Hamming space where all possible barcodes reside. Therefore, for an ordered set of object labels across cycles that form a barcode B, this evidence states P(B) = 1 only if B ∈ codebook and otherwise P(B) = 0.

4.4 PLePI-ISS: Application Into a Two-Stage SSOD

In the previous section, we proposed arranging images in each mini-batch so that object detection solutions across images become dependent and form a source of privileged information. We described our problem's domain-specific properties for the localization and classification tasks which are used as the sources of privileged information.

Given Nf ×Nr unlabeled images defined in 4.1 across all the fields of view and cycles in a well of an ISS experiment, noisy estimates for location and categories of objects are extracted for a subset of images by: 1. simple automated thresholding/argmax based approaches (cheap but low quality) and 2. a multi-step expert-supervised image processing pipeline (still noisy but higher quality). We later evaluate the effect of annotation noise level on our proposed model's performance.

We apply our learning strategy into a teacher-student based SSOD, build on the two-stage anchor-based Faster RCNN [Ren et al., 2015] object detector. Two-stage object detectors have specifically shown promising results in semisupervised object detection [Zhou et al., 2021; Wang et al., 2021b]. A teacher model is then initialized using noisy labels. The teacher attempts to correct the "overfitting to noisy labels effect" by updating its decisions according to the privileged information.

The multi-task loss function for the Faster R-CNN [Ren et al., 2015] two-stage object detector is a linear combination of object localization and classification losses. In the first stage, the region proposal network (RPN), is trained for binary classification of foreground versus background and bounding box regression of a set of predefined anchors. A region of interest (ROI) detection network is simultaneously trained for object classification and bounding box regression of a selected set of ROIs. In each training iteration, the RPN generates a set of objectness scores and bounding box deltas for each proposal anchor in the image. The objectness scores are used to assign an anchor to foreground versus background class and bounding box deltas are to estimate the adjustment needed to the anchor bounding box for each detected object.

Following the shared evidence on object locations across images (as described in 4.3), feature maps for extracting objectness score across different cycles of imaging in each tile of a single field of view that are arranged in a mini-batch, If = {If,r}Nr r=1 , can be considered as noisy perturbations of the true unknown feature map of locations. We minimize the inconsistency across these perturbed images, by enforcing all the objectness predicted labels (foreground versus background) for each anchor to be identical. The pseudo-labeling of each box on the feature map is done by the consensuspredicted labels across different perturbations. A confidence threshold for pseudo-labeling of RPN boxes to the foreground class is fixed and applied to consensus (median) objectness scores. Then, the median of estimated bounding box deltas for detected-as-foreground objects are calculated and considered as ground truth annotations for minimization of their corresponding cross-entropy and L1 loss functions of the RPN loss, LRPN (Supplementary figure 3 (c) 1 ).

Next, selected ROIs detected as foreground objects go to the second stage or the ROI detection network. We apply random rotations to the ROI feature maps in the student network. Each foreground anchor across all cycles' images forms a set of Nr objects, B, with the probability of a sequence formed by their independent object labels being

barcode is divided into two sets of confident and mediocre objects. Confident objects whose maximum class conditional probability is more than τc are pseudo-labeled and form the confident set of pseudo-labels ˆ YOC .

Incorporation of the codebook as an external source of evidence (described in 4.3) into equation 2 will result in pseudolabeling of mediocre objects so that the final barcode formed by the group is the most probable barcode that also exists in the codebook. The cross entropy between the teacher's estimated and updated pseudo-labels { ˆ YOC , ˆ YOM } and the student's soft predictions PS is then minimized by LROI to update the ROI detection network (Supplementary figure 3 (d) 1 ).

4.5 Barcode Calling Benchmarking

In-Situ-Sequencing Benchmark Resource

We created a benchmark resource for barcode calling by conducting an experiment testing a library of barcoded genetic reagents, which includes a plate of in-situ sequencing (ISS) images along with the corresponding next-generation sequencing (NGS) data as indirect ground truth. Note that, unlike the indirect evidence provided by the codebook, NGS data is not typically available in an ISS experiment. Here, it was specifically collected for this resource and for benchmarking purposes. The library contains 186 barcodes of nine letters represented in nine cycles of an imaging sequence. The resource (described in detail at Supplementary A 1 ) contains two six-well plates, each well containing about 100 images (called sites or fields of view). Each well contains about 500000 single cells and 20 million four-channel spots in ISS images of nine cycles. Not all cells have spots, and most cells have more than one spot. The results of per-spot barcode assignments based on the proposed and baseline methods are then converted to per-cell barcode assignments as described in the next section. The abundance of cell-level barcode assignments should approximate NGS-based barcode abundances. We use the latter, as the main reference to assess the quality of the former extracted by various decoding strategies.

Cell Calling as a Proxy for Barcode Calling Evaluation

Cell calling is defined as assigning a barcode to each cell, indicating the perturbation that the cell has received. There are usually multiple individual barcode spots detected in each cell. They ought to report identical sequences within a given cell (except when the experiment is configured otherwise), but that is not always the case due to confounding noise.

Our experiments are configured to typically have a single perturbation per cell and therefore any inconsistency in the detected barcodes is likely due to a computational decoding error. For each cell, we need to make a decision on the cell's perturbation (cell calling) based on the barcode calling evidence at hand. To map all methods' barcode calling results to cell-level barcode assignments, we use cell-segmentation image outputs from a classical image processing pipeline using CellProfiler, which uses cytoskeleton-stained cell images to identify cell borders. Next, we combine those segmentation masks with the barcodes' detection confidence (which most methods produce), by simply assigning the highestconfidence barcode sequence to each cell with Some proportion of cells assigned to no barcode.

5 Experiments

In the following, we evaluate PLePI in the context of stateof-the-art SSOD using the MS-COCO benchmark, with CLIP [Radford et al., 2021] as a hypothetical source of privileged information. Then using the introduced benchmarking resource in 4.5, we perform ablation experiments to evaluate the role of the main components in our proposed framework (PLePI-ISS) in cell calling performance. Finally, we compare recent top-performing related works on a subset of our benchmark resource.

5.1 PLePI: MS-COCOBenchmark for SSOD

Here we demonstrate PLePI on a popular SSOD benchmark as a proof of concept for incorporating independent evidence into the teacher's pseudo-labeling process. The privileged information here is provided by Contrastive Language-Image Pre-training (CLIP) [Radford et al., 2021] model. Details on extracting evidence using CLIP for mediocre samples and its fusion to the model's soft decisions are given in Supplementary C 1 . We follow standard evaluation protocols for SSOD, using COCO in the most label-scarce setting. In this setting, 1% of the labeled set are sampled and the remaining images are considered as unlabeled samples, which is then repeated in 5 different folds of data. We incorporated our privileged information-based enhancement of the pseudo-labeling process into PseCo [Li et al., 2022a] which shows superior results among other methods in this minimal ratio of labeled to unlabeled samples scenario. Table 1 shows that results improve by enhancing PseCo with PLePI to make the best use of mediocre samples according to the teacher's soft decisions (section 3).

5.2 PLePI-ISS: Ablation Studies for Cell Calling Evaluation

As we are presenting the first application of SSOD to the barcode calling problem, many variations of the model's many components might be evaluated. Here we focus on the most important questions related to the contribution of this work: in the context of SSOD, is the proposed framework able to: 1) take advantage of unlabeled data and improve the initial decision boundary formed by noisy estimates? 2) take advantage of privileged information to enhance the ISS SSOD pseudo-labeling strategy?

Learning/Evaluation Protocol

We follow burn-in strategy [Liu et al., 2021; Zhou et al., 2022] to initialize both the teacher and student models using labeled data. In our data, each mini-batch contains 9 image tiles of size 256×256, each containing up to 150 spots. We evaluate the ablation experiments by randomly selecting three sets of images as labeled set, unlabeled set and test set. Labeled set consists of images in a single site of the first experimental plate, containing 4356 images of size 256×256. Test set consists of images in a single site of the second experimental plate. Unlabeled set consists of two sites of the test set plate. We evaluate the performance of applying the trained model on the test set images when using labeled data and labeled plus unlabeled data.

Using barcode calling evaluation metrics (Section 4.5, Supplementary A.3 1 ), we report the rate of cell recovery and the goodness of the fit (as R2 scores) between the imagebased barcode abundance estimates and the NGS-based abundance values (Table 2). The table's first row is a baseline: direct application of the initialized teacher using labeled data to the test set, without taking advantage of unlabeled data.

Level of privileged information. As emphasized in the paper, our model relies on the privileged information to avoid noise overfitting and improve its initial decision boundary. We described two types of evidence in 4.3, on the location and label of objects in a mini-batch of samples. We found that the application of first evidence is required to prevent failure of the pseudo-labeling process. The second row of the table corresponds to the learning strategy using the evidence for object locations only, and the third row corresponds to the proposed framework when utilizing all the evidence at hand.

Level of annotation noise. As described previously, noisy annotations can be created by noisy estimations. These could come from the simple and efficient thresholding and argmax approach (termed 'low quality', LQ here) or a more costly expert-supervised parameter-tuned pipeline ('high quality', HQ). Note that both levels are considered noisy, but the goal is to evaluate the effect of using unlabeled samples and privileged information under these two annotation noise levels.

Barcode Calling Results. Values reported in Table 2 reveal that incorporating unlabeled samples together with any level of privileged information improves the baseline decision boundary formed by noisy labels. This benefit could be more influential under higher levels of label noise.

5.3 Comparison to Baseline SOTA ISS Decoders

Baseline Methods

Recent top-performing methods for decoding ISS images, as described in Section 2.1, find the best fit of the codebook to their real ISS data directly to estimate their proposed model's parameters. Although this strategy may cause over-detection of codebook barcodes, the lack of ground truth prevents validation of the detections in a more proper train-test evaluation scheme. The validation of their methodology usually relies on simplified simulated data with ground truth. BarDensr [Chen et al., 2021], reported its performance for barcode detection in simulated data using Receiver Operating Characteristic curve (ROC curve). ISTDECO [Andersson et al., 2021] suggests addition of extra 'non-targeted' barcodes to the codebook to allow calculating a false discovery rate (FDR), defined as the ratio of non-targeted barcode detection rate to the non-target percentage of the barcodes in the codebook. Again, no unified evaluation framework exists, which hinders a fair comparison of various methodologies.

Here, we chose four baseline methods to apply to our benchmarking ISS data. We began with the typical standard pipeline for spot-based decoding of ISS images in Starfish package [Axelrod et al., 2021; Ke et al., 2013]. As well, we test BarDensr [Chen et al., 2021] and ISTDECO [Andersson et al., 2021] which were developed in parallel based on modeling the physical process and non-negative matrix regression methods, and the most recent work PoSTcode [Gataric et al., 2021], which is based on probabilistic modeling of the data. The input to all the baseline methods is raw illumination corrected and aligned images by our in-house pipeline as described in Supplementary A 1 .

Evaluation Protocol

As described in 5.2, we suggest train-test schemes for evaluations to minimize the chances of false discoveries. However, not all baseline methods were developed to be performed in such a setting, so we chose one randomly selected site of the experiment as the 'inference' site for all methods. Using the available implementations for these baseline methods, we applied each method on the inference site and extracted spotlevel barcode assignments. Cell-level barcode assignments were then derived by confidence measures reported for each spot-level barcode assignment as described in 4.5. We note that all methods provided a barcode assignment confidence score by their defined metric except for BarDensr, for which we took the majority voting approach for spot-level to celllevel barcode assignments. As we aim to achieve the highest possible number of cells with a correct barcode assignment, the main evaluation metrics are the rate of cell recovery and the matches between the abundance of cell assignments and the NGS barcode abundance, which are reported in the first two columns of Table 3.

Our original codebook consists of 184 barcodes (referred as targeted barcodes) and we included 9 additional nonexistent false barcodes (referred to as trick barcodes) to the codebook for false discovery analysis. Trick barcodes are selected to have a large Hamming distance from each other and also with the targeted barcodes. Barcodes detected by each decoding method can fall into three categories: targeted, trick and not-targeted-nor-trick. Targeted barcode assignment rates are reported as Positive Predictive Value (PPV) at each cell and spot level. Incorrect barcode assignment rates for two categories of trick and not-targeted-nor-trick calls are reported as False Discovery Rates (FDR) and are denoted as FDRtrick and FDRother respectively (all metrics reported in the Table 3 are described in details in Supplementary A.3 1 ). We can see that both BarDensr and ISTDECO, which force all pixels to a barcode in the codebook, have higher FDRtrick rates. The standard ISS pipeline in Starfish has the highest overall false discovery rate, which is expected as it is the only method in the table that doesn't take advantage of the codebook in its barcode estimation process.

Although variable, we have not reported the hands-on time needed to tune parameters for each method due to the subjectivity of this task. We do report inference time for the application of each method on the test set - although our model has the minimum inference time, unlike others it requires a training phase.

According to the numbers reported in the table, none of the methods are outperforming the rest on every single performance metric. However, our proposed method overall shows a high NGS match, while 'salvaging' many cells, with a low barcode detection FDR rate, while performing in a more reliable train-test scheme.

6 Discussion

Alleviating the need for exhaustive human-provided labels, which may be tedious, subjective, or error-prone, is the main motivation behind semi, self, and unsupervised learning methods. These problems are explored extensively with welldefined objectives and benchmarks in the context of many common computer vision, language, and speech processing problems. However, these advancements are not always directly applicable to data from other fields of applied sciences including biology. These data can have their own unique properties, structure, and sources of noise. Furthermore, the ground truth is rarely direct, such as in the form of annotations from experts; instead, typically only indirect evidence in the form of domain or problem specific complementary information is available. These differences mandate a new set of assumptions and formulations. We have formulated the barcode calling problem as an object detection problem in the presence of scarce noisy labels, given a source of contextual privileged information, where a low false discovery rate is crucial. Motivated by this problem, we have proposed a general strategy for the enhancement of pseudo-labeling scheme by a source of privileged information, which is not domain specific and can advance the general field of SSOD.

Acknowledgments

Funding for this project was provided by Calico Life Sciences, LLC. We thank members of the labs of JT Neal, Paul Blainey, Frank Li, and Calvin Jan for their input on the project.