Private Semi-Supervised Federated Learning

1 Introduction

Recently, federated learning (FL) [McMahan et al., 2017] has received substantial research interests, as it provides a practical way of training machine learning models with distributed data sources while preserving data privacy. In the FL paradigm, each agent (participant) trains a local learning model with its owned data only, while a central server regularly communicates with all agents to generate a better global model through the aggregation of the local models. A key feature of FL is that no direct data exchange happens during the learning process, in contrast to centralized training.

However, existing FL studies assume either each agent owns sufficient training data [McMahan et al., 2017], or the agents collectively have sufficient data for the tasks of interest [Fan and Huang, 2021]. For instance, the image classification task, as the benchmark task of FL [McMahan et al., 2017; Zhao et al., 2018], requires each agent to prepare thousands of labeled training samples to fully train the deep-learning based models such as CNNs. In reality, the labeled data is scarce and distributed unevenly over only a few data sources. A typical scenario in mobile computing is that most end users create user data (e.g., tweets), while only a few users have the interests and time to annotate them (e.g., sentiments). The huge gap between lab scenarios with abundant labeled data and real situations with scarce and unevenly distributed labeled data severely limits the practicality and scalability of FL. It motivates us to consider the following question: How to make FL effective with scarce and skewly distributed labeled data as well as abundant unlabeled data?

Semi-supervised learning (SSL) [Chapelle et al., 2006] is an important machine learning topic that aims to leverage unlabeled data to enhance model capacity. The classical "manifold" assumption of centralized SSL [Zhou and Li, 2010] assumes that the data space is composed of multiple lowerdimensional manifolds, and data points lying on the same manifold should have the same label. A common practice [Sajjadi et al., 2016] to ensure this assumption is to firstly augment each data instance with label-invariant transformation to multiple variants, then tune the models to classify them as the same label. Recent studies [Laine and Aila, 2017; Zhang et al., 2018] further propose to perform soft data-label augmentations by mixing/blending both data features and la-

bels. However, it is not clear how to perform effective SSL in the FL context, as data exchange is prohibited.

To tackle this issue, we design an effective federated semisupervised learning framework (FedSSL) to fully leverage both labeled and unlabeled data sources while keeping the data locality as FL. We propose to learn a global generative model to establish a unified data space across all data sources, enabling each agent to generate labeled data instances for local model training. We jointly optimize the objective of training the local model F to estimate accurate labels of the generated samples and the objective of training a generator G to provide realistic data imputations conditioning on inferred labels by F . To prevent training divergence with mass unlabeled data, we further regularize the model with selfreconstruction and realism maximization targets.

In addition, we aim to prevent privacy leakage for the labeled data sources. We design a hybrid training strategy of sequential and parallel training steps on labeled and unlabeled data sources respectively. This design integrates the differential privacy (DP) scheme in FedSSL smoothly to prohibit excessive access to the labeled data. We show that our strategy can protect privacy with strict theoretical guarantee, and it causes only minimal degradation in model performance.

In conclusion, our contributions include:

• We study a critical but under-explored task of effectively performing semi-supervised learning with distributed and mixed types of data sources.
• Our learning framework prevents the violation of data locality that existed in all previous studies of SSL in FL.
• We design a mixed-data generation strategy to utilize both labeled and unlabeled data sources by establishing a unified data space without direct data exchange.
• We firstly propose a private SSL framework in FL which ensures strict privacy protections for labeled data sources.
• We outperform baselines by 5%-15% on vision and NLP tasks and prevent divergence with extremely scarce data.

2 RelatedWork

We briefly review recent related work in categories: 1) semisupervised learning (SSL), 2) federated learning (FL), and 3) settings involving both of them.

The SSL, e.g., [Chapelle et al., 2006], is an important machine learning topic which aims to utilize unlabeled data to improve task learning. Classical SSL methods include pseudo-labeling [Lee and others, 2013] and entropy minimization [Grandvalet and Bengio, 2004]. Data augmentation approaches, such as MixUp [Zhang et al., 2018], MixMatch [Berthelot et al., 2019], and FixMatch [Sohn et al., 2020], have also been developed and integrated into DL models. They interpolate pairs of data and label with random ratios to augment training data. However, existing approaches can only apply on centralized training paradigm, while our approach effectively applies on distributed data sources.

The FL has become a rapidly developing topic in the research community, e.g., [McMahan et al., 2017; Zhao et al., 2018; Li and others, 2019; Fan and Liu, 2020], as it provides a new way of learning models over a collection of distributed devices while keeping data locality. Recent studies focused on FL's robustness against non-IID data [Zhao et al., 2018; Sattler et al., 2019], few-shot data [Wu et al., 2020; Fan and Huang, 2021] and differential privacy [Wei and others, 2020; Xin et al., 2020]. However, these FL studies assume the agents in FL have plenty of labeled training data, while we focus on a practical scenario that labeled data is scarce.

Recently, several works made initial attempts to consider SSL in FL settings. Several surveys [Jin et al., 2020a; Jin et al., 2020b] discussed about applying existing SSL methods in FL without experimental proofs. Zhang et al. [2020] assumed that all labeled data is available at the server. Jeong et al. [2020] assumed that labeled data is available at every client. Itahara et al. [2020] assumed that all unlabeled data is shared across all agents. However, these studies violated the data locality property of FL. In contrast, we consider the scenario that labeled data or unlabeled data is kept at each client, and we prohibit sharing data across the agents.

3 Approach

We briefly review FL and SSL first, then formulate SSL objective in FL formally, and propose our FedSSL framework.

3.1 Review of Federated Learning

We consider a classical supervised FL system with K distributed agents. Each agent owns a local data source with which it trains a local learning model. A central server coordinates the agents by periodically collecting local models to fuse to a global model, then synchronizing back to all agents for next round of update.

Formally, let Xk be the data source of client k, nk be the number of data samples in Xk, n = ∑k nk be the total number of samples across all data sources. We consider a C-class data space D with label space Y = {0, ...,C-1}. Let F be the learning model with parameters w that maps data to label space F : D → Y, e.g., a CNN for image classification.

The global FL target is to minimize the joint training objective L over all data sources min w L(w) = ∑Kk=1 1K ℓk(Xk ; w) in which local objective ℓk is the task-specific local training objective, e.g., cross-entropy loss for classification.

A widely used FL strategy called FedAvg [McMahan et al., 2017] is to fuse the global model w with weighted average of local models such that w = ∑Kk=1 nk n wk.

3.2 SSLObjective in FL

We consider a distributed SSL scenario where each learning agent owns one of two possible types of data sources, as shown in Figure 1. The first type of source owns a few labeled data instances (with no unlabeled data), and we call the agents with such source as support agents (S-agents). The other type of source owns all unlabeled data instances, and we call the agents with such source as query agents (Q-agents). The mission of our study is to improve the collective capacity of both Q-agents and S-agents through SSL in FL paradigm.

We consider a system of NS S-agents and NQ Qagents. We denote the data collection for all S-agents as S = {X1,X2, ...,XNS }, with each labeled source Xi = {(xt, pt)}|Xi | t=1 . Here we use pt ∈ RC as one-hot encoding of the data label yt. We denote the data collection for all Qagents as Q = {U1, U2, ..., UNQ }, with each unlabeled data source as Uj = {ut}|Uj | t=1 .

The global FL target is the joint of all local objectives on all S- and Q-agents as follows

in which ℓs and ℓq are the task-specific loss functions on Sagent and Q-agent that we will explore later.

3.3 Local MixUp (L-MixUp) Operations

We introduce the local MixUp (L-MixUp) [Zhang et al., 2018; Berthelot et al., 2019] in classical SSL, for eliciting our design of global MixUp (G-MixUp) in the next section.

Pseudo Label. At Q-agent j with unlabeled data Uj, we can guess the pseudo labels given a trained model w. Following [Berthelot et al., 2019], we augment a data instance ut ∈ Uj to its K variants {ukt }Kk=1 with label-invariant operations, e.g., image rotation and cropping, and compute their mean probabilistic predictions ¯ pt = 1K ∑Kk=1 F (ukt ; w) ∈ RC .

We can estimate the pseudo label by applying label sharpening [Berthelot et al., 2019] to create low-entropy (sharp) label distribution ˆ pt, with likelihood of each category c as

in which Z > 1 amplifies the dominating classes.

Local MixUp (L-MixUp) Operations. We describe the LMixUp in details, which directly applies MixUp at each Sand Q-agent locally and individually to synthesize mixed data features and labels for local training.

Formally, for two data features x1 and x2 with label (or pseudo label) distributions p1 and p2, L-MixUp produces

where (x′ , p′ ) is the synthesized data, and α is the mixup hyper-parameter which controls the strength of interpolation between two inputs. L-MixUp augments the local data set and regularizes the model towards producing linearly-behaving boundaries between classes thus makes the model predict more accurately for unlabeled data [Zhang et al., 2018].

3.4 Global MixUp (G-MixUp) Operations

L-MixUp is limited as it can only operate locally, as data exchange among agents is prohibited in FL. This would lead to inferior performance in our challenging SSL setting of scarce and skewly distributed labeled data. To tackle this issue, we propose the global MixUp (G-MixUp) that operates across the agents to allow data imputation without data exchange.

G-MixUp utilizes a generative learning scheme to construct a global data space to generate and mix data of arbitrary class. We design a conditional generator G to synthesize a mixed data sample which conditions on both local datalabel pair as well as an additional sampled data class from the global label space. We will discuss the details with S-agent and Q-agent individually.

G-MixUp at an S-agent.

Let the conditional generator G accept a labeled data with true label (x1, p1), a randomly sampled class c2 of one-hot form p2, a blending factor λ as in (3), and a noise vector z ∈ RDn . The goal of G is to produce a synthesized data ˆx ← G((x1, p1), p2, z, λ) with a predicted label ˆ p = F (ˆx). Consistent with L-MixUp, the generated ˆx is supposed to be λ-likely as of x1's class, and (1 - λ)-likely as class c2 in appearance, i.e., p′ = λp1 + (1 - λ)p2, which we can evaluate with the cross-entropy loss. Formally, the joint training objective for F and G is

A better G produces more realistic imputations, and a better F produces more accurate estimation of the generated data labels. Hereby F and G improve each other to reach optimality. The global G can establish a global data space to facilitate effective global MixUp, thus improving SSL capacity.

G-MixUp at a Q-agent.

Next, we define G-MixUp at a Q-agent with unlabeled data. We firstly sample an unlabeled data u1, produce its pseudo label q1 with data augmentation and sharpening with (2). To perform mix-data generation, we then sample a new class c2 with its one-hot form p2, draw a noise vector z, and train G to produce a synthesized ˆx ← G((u1, q1), p2, z, λ) with a predicted label ˆ p = F (ˆx).

Ideally, the synthesized data ˆx is expected to be λ-likely as q1 and (1 - λ)-likely as p2 in appearance, i.e., p′ = λq1 + (1-λ)p2, which we can evaluate with the cross-entropy loss. Formally, the joint training objective for F and G is

We further regularize model training by ensuring the realism of generated samples and reconstructed real samples.

Realistic loss. Equations (4) and (5) imply that if we set the blending factor λ = 0, G will generate a data sample 100% of class c2. We can utilize a conditional discriminator D to encourage realism of the generated image with given class.

We extend to a general case of λ ∈ [0, 1], to encourage the realism of G output weighed by a factor g(λ) depending on

Algorithm 1: FedSSL algorithm.

Input: A set of NS support agents, and NQ query agents. Output: A global model F for SSL task. 1 Server executes: Initialize global F,G,D, let t ← 1 2 while t ≤maximum rounds T do 3 for each client i in support set S in parallel do 4 [D, G]ti ← G-Reg([D, G]t , Xi) 5 [F, G]ti ← G-MixUp([F, G]t , Xi) 6 for each client j in query set Q in parallel do 7 [D, G]tj ← G-Reg([D, G]t , Uj ) 8 [F, G]tj ← G-MixUp([F, G]t , Uj ) 9 [F, D, G]t+1 ← F edAvg({[F, D, G]tk∈S∪Q)} 10 The server sends [F,D,G]t+1 back to clients 11 t ← t + 1

blending factor λ. We design the dynamic realistic loss as

in which max{λ, 1-λ}-1 ∈ [-0.5, 0] and g(λ) ∈ [0.61, 1]. When blending from one source of either sampled image or noise (λ = 0, 1), we encourage more visually realism with large g(λ) = 1; when blending evenly from two sources (λ = 0.5), we tolerate the unrealism with a smaller g(λ) = 0.61. We adopt the training process of GAN [Goodfellow et al., 2014] to update D and G alternatively.

Reconstruction loss. When setting the blending factor λ = 1, Equations (4) and (5) imply that G should re-produce the sampled x1 (or u1). Thus we design an encoder-decoder structure of G to reconstruct the input, and we regularize its training with reconstruction loss term such that

FedSSL Algorithm Details. We show our proposed FedSSL with G-MixUp in Algorithm 1 and summarize as

Algorithm 2: Pseudo code for FedSSL-DP.

1 Let ϵtot ← 0, δ ← 10-5 . 2 for each client i in support set S in sequence do 3 if ϵtot > ϵ then 4 Out of privacy budget, break; 5 else 6 Perform DP update on G; 7 ϵtot ← ϵtot + αM(σ, δ); 8 Perform normal update on D, F ; 9 for each client j in query set Q in parallel do 10 Perform normal update on D, F,G;

follows. At local training stage of FL, we adopt a two-step optimization procedure to 1) regularize model by training D and G alternatively to minimize ℓreg = ℓreal +ℓrec , which we denote as G-Reg (line 4,7); 2) train F and G with G-MixUp to produce mixed data samples and improve accuracy (line 5,8). By federating the models (line 9), we unify G-MixUp operations on S-and Q-agents to leverage both true labels and pseudo labels to enhance SSL, and establish a unified global data space over all agents to facilitate data augmentation over entire data space with arbitrary classes, thus boost SSL.

4 Privacy

In the realistic setting as we consider, the small labeled data could be overly accessed by G during G-MixUp, which poses risks of information leakage (as the global G could reconstruct them at arbitrary agents). In the spirit of FL for privacy protection, it's critical for us to ensure privacy for GMixUp operations. In this section, we introduce (ϵ, δ)-DP to our framework and provide a practice algorithm which integrates noise injection mechanism into FedSSL to provide strict privacy guarantee for labeled data sources.

Definition 1. (ϵ, δ)-DP [Dwork and Roth, 2014]. A randomized function M : D → R, with domain D and range R, satisfies (ϵ, δ)-DP if for any two adjacent databases d, d′ ∈ D and for any subset of outputs S ⊆ R

A standard way to guarantee (ϵ, δ)-DP is by integrating a Gaussian Mechanism (GM) [Dwork and Roth, 2014] in the model learning process, by adding Gaussian noise N(0, σ2 ). Stochastic GM (SGM) [Abadi et al., 2016] is developed to work with SGD for training deep-learning models. SGM applies GM on sampled batches with a sampling ratio γ, and T update steps of SGM will imply a (O(√Tγϵ, δ)-DP by using Moment Accountant (MA)[Abadi et al., 2016] to track the overall privacy budget.

We propose a practical algorithm called FedSSL-DP, that seamlessly integrates SGM and MA into FedSSL to ensure (ϵ, δ)-DP , with pseudo code shown below.

Overall, we design a hybrid training strategy to train on S-agents and Q-agents efficiently. Specifically, we train on S-agents with DP sequentially first (line 2), i.e., train on Sagent i, then pass the updated model to S-agent i + 1. After looping over all S-agents, we perform the standard FL in parallel with Q-agents (line 9).

We first fix the tolerance term δ at a small value, e.g., 10-5 ; then choose a target privacy budget upper bound ϵ, e.g., 4, 8, or 16, and initialize accumulator ϵtot (line 1). Then we perform DP update on G with SGM, then calculate ϵ as privacy loss with MA αM(σ, δ) and accumulate to ϵtot (line 6-8). Once ϵtot reaches the budget ϵ, we stop accessing to labeled sources (line 3-4).

Lemma 1. Algorithm 2 guarantees (ϵ, δ)-DP for labeled data sources.

Proof. Algorithm 2 guarantees that the cumulative privacy loss for updating G during sequential S-agent updates ϵtot ≤ ϵ. Also, the parallel Q-agent update does not access the labeled data on S-agents. Therefore, G is (ϵ, δ)-DP with respect to labeled data sources.

5 Experiments and Discussions

We describe the datasets, parameter choices and models we experiment on. Then we analyse the performance with visual and textual tasks with ablation studies and visualizations.

5.1 Datasets and Splits

Following recent works [Berthelot et al., 2019; Zhu et al., 2020; Chen and et al., 2018] of evaluating FL and SSL, we describe three widely used benchmark datasets.

CIFAR-10 [Krizhevsky, 2009] is a common image recognition dataset with 50000 data instances of 10 categories (such as birds, cars, and horses). We try 3 settings with an increasing difficulty, by holding out 5000 (10%), 2500 (5%), and 500 (1%) as labeled instances, respectively, and keeping the rest as unlabeled instances.

MNIST [LeCun et al., 1998] is a digit recognition dataset with 60000 data instances for 10 digital classes. We try 3 settings with an increasing difficulty, by holding out about 300 (0.5%), 150 (0.25%), and 100 (0.17%) of total data as labeled instances respectively, and using the rest as unlabeled.

Sent140 [Caldas et al., 2018] is an FL benchmark for sentiment analysis as a 2-way classification task (positive and negative). We sample 60,000 sentences and try 3 settings with increasing difficulty by holding out 3000 (5%), 600 (1%), and 300 (0.5%) as labeled sentences, with the rest unlabeled. We tokenize each sentence to a max of 40 words.

5.2 FL Device Numbers

We try different numbers of S-agents (labeled sources) and Q-agents (unlabeled sources), denoted as NS and NQ respectively. We examine two different settings, i.e., (NS = 2,NQ = 6), and (NS = 3,NQ = 9), to represent the scenario of fewer S-agents and more Q-agents. We indeed try extremely challenging cases, e.g., 100 (0.17%) MNIST samples distributed to 3 S-agents so that each has about 34 labeled samples, for checking whether FedSSL is robust to extreme data-scarce scenarios.

5.3 Details of Our Methods and Baselines

We compare our proposed methods (FedSSL and FedSSLDP) with the most related baselines as follows:

FedSSL is our method which learns a global data space to generate mixed samples of arbitrary classes to better augment local training data, as in Alg. 1. FedSSL-DP additionally integrates Gaussian Mechanism (GM) into FedSSL to ensure DP of labeled data sources, as in Alg. 2.

Baselines. Supervise performs FL only at S-agents with labeled data. Q-agents (with unlabeled data) are not used. Pseudo performs additional SSL with pseudo labels (Sec.3.3) at Q-agents upon Supervise approach. MixMatch [Berthelot et al., 2019] is the commonly adopted centralized SSL approach, described as the L-MixUp in Section 3.3. We applied MixMatch on all agents to utilize both labeled and unlabeled data as a fair baseline. FixMatch [Sohn et al., 2020] is the augmentation of MixMatch. Its FL version - FedMatch [Jeong et al., 2020] - performs both L-MixUp and divergence minimization across all agents, which forms the strongest baseline.

5.4 Experimental Results Results on CIFAR-10.

We evaluate on CIFAR-10 with randomly sampled 10%, 5%, and 1% of the total training data as labeled data, which accounts for roughly 5000, 2500, and 500 total training samples, respectively. We uniformly distribute the labeled data to S-agents, and unlabeled data to Q-agents. Table 1 summarizes the results for 8 and 12 devices:

FedSSL consistently performs best in all settings of labeled data portion and client number. For 10% labeled data setting, FedSSL achieves the best accuracy of 0.855 and 0.854, outperforming the best baseline (MixMatch or FedMatch) by 4.2-5.3%, relatively. For 1% labeled data, the relative performance gain reaches 14.1%-19.2%. This shows the effectiveness of FedSSL especially in extreme data-scarce conditions.

Baselines are ineffective. Compared with Supervise, Pseudo labels could help improve performance by 3%, 6%, and 15% in absolute value for 10%, 5%, and 1% label data respectively. However, L-MixUp (FedMatch and MixMatch) could only further increase performance by less than 3% in absolute value, indicating the ineffectiveness of conventional SSL techniques, due to the skewly distributed data labels.

FedSSL-DP achieves comparable performance with FedSSL. For 10% and 5% labeled data cases, the performance gap is less than 2%. The biggest differences happen for 8 and 12 devices with 1% labeled data, which are about 2.5-2.8% in absolute value. DP affects the model accuracy more with less labels and larger device numbers.

FedSSL-DP outperforms the baselines significantly, indicating that our approach is effective in distributed learning while preserving the privacy of the labeled sources.

Results on MNIST.

We evaluate our methods on MNIST with 0.5%, 0.25% and 0.17% of the total data as labeled data, which accounts for just 300, 150 and 100 total training samples, respectively. We show the results in Table 2:

The baselines suffer from model divergence. In the extreme cases of training with 0.25% and 0.17% labeled data on 8 (NS=2, NQ=6) and 12 (NS=3, NQ=9) devices, all baselines failed to converge and predict randomly (denoted as '-'). Due to lack of a unified data space, each local model overfits to local data so that the federated global model collapses.

FedSSL and FedSSL-DP can prevent divergence. In contrast, our proposed FedSSL achieves reasonable accuracy of around 0.97. Thanks to the global data space, FedSSL augments local data and prevents overfitting to scarce labels (Sagents) and incorrect pseudo labels (Q-agents).

FedSSL-DP achieves similar accuracy with FedSSL, with performance gap generally under 1% in absolute value. Even for extreme data-scarce (0.17%) case, the gap is below 3%, indicating the usability of FedSSL-DP.

Results on Text Classification.

In Table 3, we show results on Sent140 dataset with 5%, 1% and 0.5% of the total tweets with sentiment labels, which accounts for roughly 3000, 600, and 300 sentences respectively. We implement blending operation of two sentences by a simple weighted sum of two words' BERT embeddings from two sentences at same positions. Table 3 shows the binary classification results. FedSSL consistently outperforms the baselines, leading the next best FedMatch by 1.8-5% relatively, while leading the weakest Supervise by 7-12% relatively.

5.5 Ablation Studies

Non-IID data partition. We consider a more challenging setting with non-IID partition of data classes. We adopt a round robin strategy of distributing non-overlapping MNIST

digital classes to each agent. As an example of (NS = 3,NQ = 9), we firstly distribute all labeled instances of digital classes [0, 3, 6, 9], [1, 4, 7], [2, 5, 8] to 3 S-agents respectively, then we distribute all unlabeled instances of [0, 9], [1], [2], ..., [8] to 9 Q-agents respectively. This creates non-overlapping partitions of digital classes within S-agent group and Q-agent group thus makes local training header.

We examine the performance of baselines and our models under this difficulty situation and observe in Table 4 that:

The non-IID settings bring about performance drop and divergence especially for scarce labeled data 0.25% and 0.17% with a large device number 8 and 12.

Only FedSSL could perform reasonably well in extreme settings (e.g., 0.17% labeled data), while all other baselines diverge due to collapsed local training on partial data classes.

General setting of partially labeled data. FedSSL can readily extend to a general setting in which each client has both labeled and unlabeled data, as our proposed G-MixUp can flexibly sample data pairs with true labels as Eq.(4) and/or with pseudo labels as Eq.(5). We allocate 10% of labeled images and the rest unlabeled images of CIFAR-10 uniformly to 4 clients. FedSSL outperforms the baseline FedMatch by 5.9% (85.1% v.s. 79.2%), as our G-MixUp can perform both local and global data imputation to better train the unified global model, while FedMatch can only perform local mixup.

Effects of lreal (Eq.6) and lrec (Eq.7). We evaluate FedSSL on non-iid MNIST with 3 ablation settings: no realistic loss lrec , no reconstruction loss lreal and without both. We find that both lrec and lreal are critical with better regularization especially for the extreme data-scarce scenario (0.17%), e.g., w/o both would yield 29.3% drop of accuracy.

6 Conclusion

We proposed a unified framework that makes FL effective in challenging SSL scenarios. We designed a generative learning strategy to establish a global data space across the agents while preserving data privacy with theoretical guarantee. Our approach outperforms the baselines significantly and works robustly in extreme data-scarce and non-IID cases.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (NSFC 62106156), Shenzhen Science and Technology Program (Project JCYJ20210324120011032), Guangdong Basic and Applied Basic Research Foundation (Project 2021B1515120008), and the Shenzhen Institute of Artificial Intelligence and Robotics for Society.