Meta-Learning via PAC-Bayesian with Data-Dependent Prior: Generalization Bounds from Local Entropy

1 Introduction

Meta-learning [Thrun and Pratt, 1998; Baxter, 2000], or learning-to-learn, extracts prior knowledge from past training tasks to accelerate future learning. This process enables knowledge transfer across tasks, leveraging previous experience to eliminate the need for training models from scratch. Recent advances in meta-learning have achieved impressive success across various fields, including computer vision [Hospedales et al., 2021], natural language processing [Lee et al., 2022], and reinforcement learning [Zhao et al., 2022]. Most current meta-learning methods [Finn et al., 2017;

∗Corresponding author

Zhu et al., 2023] presume access to a large pool of tasks, but this contrasts with the limited tasks typically available in realworld settings. This gap raises the risk of overfitting to the few tasks used during meta-training, potentially impairing performance on unseen target tasks. Hence, a critical challenge is how to effectively regularize the meta-learner to generalize well to unseen tasks.

Fortunately, the PAC-Bayesian theory [McAllester, 1999] provides a rigorous framework for understanding the generalization performance of learners, which incorporates prior knowledge and implicitly regularizes the learners. Recently, there has been a surge in studying meta-learning with PACBayesian theory [Amit and Meir, 2018; Guan et al., 2022; Rezazadeh, 2022]. However, these approaches still face several limitations: (1) They primarily focus on data-free priors, which tends to restrict achieving tighter bounds. Typically, generalization bounds balance empirical risk against model complexity, dominated by the gap between the prior and posterior. Priors that closely match the underlying task distribution can lead to tighter bounds. (2) Their optimization process is a complex and usually unstable procedure, fine-tuning posteriors that reflect task preferences without drifting too far from repetitive priors, and fitting the hyper-posterior to minimize the corresponding bound. Hence, the resulting algorithms rely on a complex nested optimization problem, incurring expensive computational costs. Here a question arises: are there priors to regularize meta-learning, achieving a tighter generalization guarantee and simplifying the optimization process?

To this end, we study the generalization property of metalearning through the PAC-Bayesian perspective with datadependent priors. First, we derive the relationship between the local entropy and the results in PAC-Bayesian learning theory, which helps us directly optimize the priors to adhere to a tighter upper bound without explicit posteriors. Although it is possible to acquire data-dependent priors by optimizing local entropy [Chaudhari et al., 2017; Chaudhari et al., 2019], such an approach is usually ineffective due to the restriction in PAC-Bayesian theory, which states that the prior cannot depend on the training samples. An intuitive solution to relax this restriction is the differential privacy (DP) mechanism [Dziugaite and Roy, 2018b]. DP enables the data-dependent priors to weakly depend on the data while ensuring they suffice for the valid generalized bounds. To this end, we extend the PAC-Bayesian analysis framework for meta-learning to the data-dependent priors under privacy constraints. We further refine our PAC-Bayesian bounds to directly relate to the local entropy of Bayesian learners, rather than explicit posteriors, thus getting around the need for complex nested optimization. Besides, we develop a novel meta-learning algorithm utilizing Stochastic Gradient Langevin Dynamics (SGLD) [Welling and Teh, 2011], which is straightforward to implement and simple to train. Experiments show that our approach yields competitive results compared to previous PAC-Bayesian metalearning methods.

Contributions. The main contributions of this paper can be summarized as follows:

• We study the generalization of meta-learning through the PAC-Bayesian perspective incorporating data-dependent priors (Section 3).
• We provide a new framework for PAC-Bayesian metalearning incorporating data-dependent priors. Our PACBayesian generalization bound is defined as a function of the local entropy, without the reliance on explicit posteriors for each task, which ultimately attains fast convergence ability (Section 3.2).
• With our bounds, we develop an efficient meta-learning algorithm grounded on SGLD, namely PAC-MLE. This algorithm eliminates the necessity of nested optimization, by effectively and implicitly sampling from the Gibbs optimal poster (Section 4).
• Numerical experiments on both classification and regression tasks demonstrate that our method outperforms previous theoretical work in terms of fast adaptation and generalization guarantee, and effectively impedes over-fitting (Section 5).

1.1 RelatedWork

In this subsection, we briefly overview the main related work. We provide a more detailed literature review in Appendix B.

Meta-learning [Thrun and Pratt, 1998] enables knowledge transfer across tasks, leveraging prior experience to eliminate the need for training models from scratch. Various methods include developing a shared embedding space across tasks [Vinyals et al., 2016; Xu et al., 2020], learning the initialization for quick adaptation [Finn et al., 2017; Rothfuss et al., 2019], or incorporating probabilistic modeling for uncertainty quantification [Finn et al., 2018; Chen and Chen, 2022]. Despite their ability to learn complex patterns, these methods often require many meta-training tasks and lack performance guarantees. To address this issue, we study the generalization of meta-learning through the PAC-Bayesian perspective and introduce a novel meta-learning method with generalization guarantees.

The PAC-Bayesian learning theory [McAllester, 1999] provides upper bounds on the generalization error that holds with an arbitrarily high probability. Further works [Pentina and Lampert, 2014; Amit and Meir, 2018; Liu et al., 2021; Guan et al., 2022; Rezazadeh, 2022] extend previous bounds to the scenario where priors are meta-learned. Liu et al. [2021] propose to use held-out data to construct data-dependent priors. However, these methods are considerably complex as they leave both the hyper-posterior and posterior unspecified. Rothfuss et al. [2021; 2023] derive the PAC-optimal hyper-posterior but overlook data-dependent priors, limiting the potential for achieving tighter bounds. In contrast, we directly optimize data-dependent priors to best meet generalization preferences using a simple local entropy, which greatly simplifies the learning process.

2 Preliminaries: PAC-Bayesian Framework

Background. In the common setting, a learning task τ is characterized by an unknown distribution D over space Z. We are given a set of m observations, S = {zj}mj=1 , with each zj ∼ D. In supervised learning, the observations zj = (xj, yj) comprise input features xj ∈ X and target labels yj ∈ Y. Given S, our goal is to find a hypothesis h ∈ H, which can minimize the expected error over D, L(h,D) = Ez∼D[ℓ(h, z)], where ℓ : H × Z → R+ is a loss function. Since D is unknown in general, we typically evaluates the empirical error instead, ̂ L(h, S) = 1m ∑mj=1 ℓ(h, zj ).

In the PAC-Bayesian framework, we are concerned with randomized predictors i.e., probability measures on the hypothesis space H. This allows us to reason about the uncertainty of h, resulting from the fact that only a finite number of data points are available for training. Our goal is to output a posterior Q := Q(S, P ) ∈ M(H), where P ∈ M(H) represents the prior knowledge andM(H) denotes the set of probability measures over H. Then, we can define the expected error of posterior Q as L(Q,D) = Eh∼QL(h,D) and its empirical counterpart as ˆ L(Q, S) = Eh∼Q ˆ L(h, S).

PAC-Bayesian bounds. In practice, L(Q,D) is typically unknown as well. Thus, it is common to minimize ˆ L(Q, S) instead. If there are no structural assumptions with additional constraints, this often results in overfitting and limited generalization performance. Hence, it is important to analyze the difference between the empirical estimate and its expected one. PAC-Bayesian theory helps us do so by bounding the unknown generalization error based on its empirical estimate:

Theorem 1 (McAllester's single-task PAC-Bayesian bound [McAllester, 1999]). Given a data distribution D, prior P ∈ M(H), for any confidence level δ ∈ (0, 1], the following inequality holds uniformly for all posteriors Q ∈ M over samples S ∼ Dm, with probability at least 1 - δ,

Generally, PAC-Bayesian bounds attempt to balance the trade-off between empirical error (fitting error) and the complexity term (deviation from prior), which promotes the choice of a simple hypothesis. The choice of prior distribution P not only impacts the tightness of PAC-Bayesian bounds, but also notably affects its performance. Priors that closely match the true data-generating process can lead to better bounds.

3 PAC-Bayesian Bounds for Meta-Learning

In this section, we present our main theoretical results. We begin with a general PAC-Bayesian bound for meta-learning and discuss its implications. We then introduce the local entropy and show its connection with the PAC-Bayesian framework. Based on this, we extend the general PAC-Bayesian metalearning bound to incorporate data-dependent priors. The overview of our proposed framework is illustrated in Fig. 2. The corresponding proofs can be found in Appendix F.

3.1 General PAC-BayesianMeta-Learning Bounds

In meta-learning settings, we aim to acquire such a prior P in a data-driven manner through a set of related learning task {τi}ni=1 , which can speed up the learning process on new target tasks τ ∼ T . All of these tasks τi := (Di, Si) share the same sample space Z = X × Y, hypothesis space H and employ the same loss function ℓ(h, z). However, they may differ regarding the (unknown) data distributions Di. For simplicity, we can assume that mi is constant across tasks, m = mi, ∀i. Moreover, every task τi ∼ T is considered to be drawn i.i.d. from an environment T .

To extract knowledge from the observed datasets, the goal of learning focus shifts from a singular hypothesis h ∈ H to the prior P ∈ M(H). Here, we assumes a hyper-prior P ∈ M(M(H)), i.e., a distribution over priors P. We combine the hyper-prior P with the datasets {Si}ni=1 from multiple tasks, and consequently output a hyper-posterior Q over priors which can then be used for new tasks. Accordingly, the quality of the hyper-posterior can be measured by the expected loss of learning new tasks using priors P drawn from Q, the so-called transfer error:

While the environment τ is unknown in practice, we can estimate it using the empirical multi-task error,

Building upon existing bounds, which affine-transformation steps at both the meta and task levels, we present a fast-rate bound for meta-learning,

Theorem 2 (Fast-rate PAC-Bayesian meta-learning bound). Let Q : Z m ×M→M(H) be a base learner, P be some fixed hyper-prior. Given any confidence level δ ∈ (0, 1], and λt, λe ∈ (0, 2), the following inequality holds uniformly for all hyper-posteriors Q, with probability 1 - δ,

Remark. Theorem 2 indicates that the transfer error is bounded by the empirical multi-task error under Q, plus the KL-divergence terms which serve as regularizers on the metalevel and task-level. This implies that regularization should be strong with limited data, but can vanish asymptotically as n,m→∞. Our fast-rate bound exhibits a convergence rate O(1/m + 1/n).

The PAC-Bayesian theory provides a framework for understanding meta-learning generalization, leading to algorithms with generalization guarantees through corresponding upper bounds. However, previous methods [Amit and Meir, 2018; Rezazadeh, 2022] often struggle with data-free priors. In practice, the mismatch between prior and posteriors often results in inefficiency estimates and slow convergence.

3.2 PAC-BayesianMeta-Learning Bounds with Data-dependent Priors

To overcome the drawbacks of data-free priors, a straightforward approach is to derive data-dependent priors P(Si) by reserving part of the training data specifically for this purpose [Thiemann et al., 2017; Liu et al., 2021; Dziugaite et al., 2021]. Although it is possible to apply priors that are data-dependent but independent from the training set, such approaches are ineffective due to the intrinsic slow convergence and potential risks of overfitting. An alternative is to employ differentially private (DP), [Dwork et al., 2015] mechanisms, which offer a compromise by ensuring priors are only weakly dependent on the data, thereby sufficient to the PAC-Bayesian bounds under the relaxed restrictions [Dziugaite and Roy, 2018a; Dziugaite and Roy, 2018b; Rivasplata et al., 2020].

In this way, the prior captures the underlying data distribution instead of being limited to the specific dataset it is trained on, maintaining resilience against data perturbations. This facilitates the somewhat loose but reliable bounds. Hence, we establish the PAC-Bayesian analysis framework for metalearning, wherein a data-dependent prior is incorporated.

Theorem 3 (PAC-Bayesian meta-learning bound with ϵ-DP data-dependent priors). Under the same settings of Theorem 2, the following inequality holds with probability at least 1 - δ,

Remark. Compared with the result in Theorem 2, the third term incurs an additional loss regarding private mechanisms. The bound Eq. (3) could significantly outperform the previous ones when ϵ ∈ O(m-1/2 ) are relatively mild and matches the rate at which the KL term decays [Dziugaite and Roy, 2018b].

3.3 PAC-BayesianMeta-Learning Bounds with Data-dependent Priors from Local Entropy

We will show that the priors that achieve a tighter generalization bound can be optimized exactly with a simple local entropy objective, greatly simplifying the learning pipeline. Before we look into the specific solution to data-dependent priors, let us first explore the posterior Q which can help us to minimize the generalization bound in Eq. (3).

Intuitively, Catoni [2007] indicates the closed-form solutions for such a minimization problem β ̂ L(Q, Si) + KL(Q||P ),

where Zβ(Si, P ) = ∫H P (h) exp(-β ̂ L(h, Si))dh is the partition function, and P, Si,m, δ are defined same in Eq. (3). The corresponding distribution Q⋆i , denoted as Q⋆i = Pexp(-β ̂ L(h,Si)) 1 , is also known as the optimal Gibbs posterior [Catoni, 2007; Lever et al., 2013].

Then, plugging Q⋆i back to Eq. (4), we have

Note that Eq. (5) establishes the connections between the PAC-Bayesian framework and the partition function w.r.t. the prior P. This suggests that the minimum of upper bounds essentially relies on the extent to which the prior can reflect the data. Recall that the hypothesis h is parameterized by w such that ̂ L(h, Si) := ̂ L(w, Si). Like previous works [Amit and Meir, 2018], assume that w has prior P(w) = N(w|v, σ2 ) and rewrite the right hand side (RHS) of Eq. (5),

1 Formally, a distribution like Pexp(-βg) is called a Gibbs distribution with energy function g and inverse temperature β.

It is worth noting that Eq. (7), known as local entropy [Chaudhari et al., 2017], is frequently used to measure flatness since it can measure the amount of "good" parameters w around given v. The resulting algorithm [Chaudhari et al., 2019; Pittorino et al., 2021] prioritizes exploring flat regions, minimizing negative local entropy rather than the original loss function ̂ L(w, Si), to enhance generalization performance. Thus, the connection between the generalization bound and local entropy can be reformulated as,

Lemma 4 ([Dziugaite and Roy, 2018a]). Assume some constant σ2 > 0. Then, the weights v that maximize the local entropy Fβ(P, Si) in Eq. (7), also minimize β ̂ L(Q⋆i , Si) + KL(Q⋆i ∥P), where Q⋆i = Pexp(-β ̂ L(w,Si)) and P = N (v, σ2 ).

Lemma 4 indicates that maximizing local entropy allows us to optimize the prior within the PAC-Bayesian bound. In other words, the acquired v can serve as the mean of the datadependent prior P(Si). Recall that Theorem 3 holds for any base learner Q(Si, P) with data-dependent priors for optimal performance. By Eq. (5), we can reformulate Theorem 3 in terms of local entropy, which yields

Corollary 5. Under the same settings of Theorem 3, assume an optimal Gibbs posterior Q⋆i as the base learner, the following inequality holds with probability at least 1 - δ,

where λ = (1-λt/2)(1-λe/2), α = λe(1-λe/2), β = mλt, constant C(δ, λ, α, β, ϵ) is deferred to proofs in Appendix.

Remark. Corollary 5 provides a tighter bound than previous works, where bounds hold for any Q ∈ M(H). It is worth mentioning that Eq. (8) eliminates the explicit dependence on Q(Si, P), and turns the nested optimization problem into a standard stochastic optimization problem. This bound consists of the expected local entropy under the hyper-posterior Q and the KL-divergence term, serving as meta-level regularization. As the number of training tasks n increases, the emphasis on the KL term diminishes, aligning with the general understanding that more data reduces the need for regularization, eventually becoming negligible as n, m → ∞.

In practice, achieving strong DP can be computationally intractable. However, Dziugaite and Roy [2018b] relax the privacy requirement and show that convergence in distribution to a DP mechanism is adequate for ensuring generalization.

This insight permits the use of SGLD [Welling and Teh, 2011]), which under certain conditions, is known to weakly converge to its target distribution. Consequently, we can formulate an optimization method for Meta-learning with PAC-Bayesian guarantee that incorporates data-dependent priors derived from local entropy.

4 PAC-BayesianMeta-Learning Algorithm

In this section, we discuss how to translate our theoretical findings into a practical algorithm. We begin with the objective function from the upper bound in Corollary 5 and proceed to detail the derivation of low-variance gradient estimators.

For simplicity, we focus on priors Pvi (w) = N (w|vi, σ2 i ), where σ2 i is a predefined constant. Then, we limit the space of hyper-prior and hyper-posteriors as a family of isotropic Gaussian distributions: P(v) := N(0, σ2P ) and Qν(v) = N(v|ν, σ2Q), where σ2P , σ2Q > 0 are some predefined constants.

The objective function for the RHS of Eq. (8) reads as

To reduce variance, we use the reparameterization trick [Kingma and Welling, 2013], and rewrite vi := f(ν, ε) = ν+σiε with ε ∼ N(0, 1). Hence, the corresponding gradients follow:

Gradient Estimation. Then we can compute the derivative of Fβ(Pvi , Si) w.r.t. vi by using methods such as REINFORCE [Williams, 1992]:

where Q(Si, Pvi ) ∝ N(w|vi, σ2 i ) exp(-β ̂ L(w, Si)) is the optimal Gibbs posterior.

Eq. (11) does not have closed-form solutions when ̂ L(w, Si) is not a squared loss, due to the high-dimensional integrations for the neural network. In practice, we draw samples wi ∼ Q(Si, Pvi ) through Monte Carlo sampling methods, i.e., SGLD, which gives

where µki = αwk i +(1-α)µi k-1 , µ1i = w1 i , and {wk i }Kk=1 are the (approximately) i.i.d. samples from Q(Si, Pvi ), α ∈ (0, 1) denotes the average weight. With the help of injected noise in each update step, SGLD can explore regions beyond local optima and converge asymptotically to the global minimum for non-convex objectives [Raginsky et al., 2017]. It is also worth noting that the sequence of {wk i }k>K converges to Gibbs distribution Q(SiPvi ) with sufficiently large K, under the conditions that ∑∞k η′k →∞and ∑∞k η′k 2 → 0.

Algorithm 1 PAC-MLE algorithm: meta-training phase

• 1: Require: hyper-prior P, datasets {Si}ni=1 , learning rate η, η′ , weight average α, initialize ν 0 ∼ P 2: for t = 1, . . . , T do 3: sample |It| tasks uniformly at random, It ⊆ [n] 4: ηt ←φη(ηt-1) ▷ learning rate decay 5: for i = 1, . . . , |It| do 6: vi ∼ Qν t , w0 i , µ0i ← vi 7: for k = 1, . . . ,K do 8: Sample a mini-batch S′ i ⊂ Si 9: η′k ← φη′ (η′k-1 ) ▷ learning rate decay 10: ∇F (wk-1 i ) ←-β 1 |Si | ∑ ∇ ̂ L(wk-1 i )-(wk-1 i - vi)/σ2 i 11: wk i ←wk-1 i + η′k∇F(wk-1 i ) + √2η′k ζ k,i 12: µki ←αµk-1 i + (1 -α)wk i ▷moving average 13: end for 14: end for 15: ν t ← ν t-1+ηt 1 |It |λβ ∑i (vt-1 i -µKi )/σ2 i -ηt 1 σ2P ( 1 nα + 1 λβ )ν t-1 + √2ηtξ t 16: end for

It is computationally expensive to learn meta-information from all the tasks when the number n of tasks is large. Thus, we can even use mini-batching on the task level. For each iteration t, we can sample a mini-batch of tasks, which are indexed by It ⊆ [n]. Subsequently, the update rule w.r.t. ν is expressed as follows:

where ηt is learning rate at t, and ξ t ∼ N(0, 1) is isotropic Gaussian noise injected during the update.

Algorithm. The Pseudocode of the proposed meta-learning algorithm is summarized in Algorithm 1. At the beginning of each round t, we sample a subset of tasks It from the task sets. During the task update phase, each sampled task runs SGLD for K steps using its data, followed by updating its state concerning priors with learning rate η′t . In the subsequent meta-parameter update phase, we aggregate the updates of the sampled tasks and then update the meta-information by Eq. (13) with learning rate ηt. Here, φη(·) and φη′ (·) denote the learning rate decay function ( e.g., cosine decay).

It is worth mentioning that previous works [Amit and Meir, 2018; Liu et al., 2021; Rezazadeh, 2022] adopt a similar variational form of our theoretical bound, in which they set factorized Gaussian distribution Qθ = N(θ, I2d) and Qϕi = N(µi, σ2 i ) as the hyper-posterior and the posterior of task τi, respectively. In addition, they utilize P = N(0, I2d) for the hyper-prior distribution and Pθ = N(µP , σ2P ) for the prior distribution, where (µP , σ2P ) ∼ Qθ. They then optimize the empirical multi-task error in addition to the regularization terms (e.g., Theorem 2) w.r.t. parameters θ ∈ R2d and ϕi ∈ R2d using SGD. In contrast, our algorithm avoids such a complex optimization process, reducing the required optimization parameters by a quarter. Instead, we employ SGLD to simulate the distribution.

Hence, one key advantage of our approach, compared to previous works, is its ability to simplify the nested optimization problem into a simpler stochastic optimization problem. This simplification not only enhances the stability of meta-learning, but also improves its scalability. Specifically, explicit computation of the task posteriors Q(Si, P) is unnecessary; instead, mini-batching over tasks can be employed. On the other hand, our approach incorporates principled meta-level regularization in the form of the hyper-prior P. This regularization effectively addresses overfitting to the meta-training tasks. Consequently, our approach enables successful meta-learning with a limited number of tasks, contrasting with the majority of popular meta-learners [Finn et al., 2017; Finn et al., 2018; Nichol et al., 2018] that heavily rely on numerous tasks to achieve generalization at the meta-level.

5 Numerical Experiments

In this section, we validate our theoretical results and demonstrate the efficiency of the algorithm through numerical experiments, including tasks related to few-shot regression and image classification. We defer complete setup details and further experiments to Appendix C.

5.1 Toy Example

Setup. This subsection considers a simple regression problem on a synthetic dataset. Each task involves regressing from the input to the output of a sine wave, with varying amplitude and phase [Finn et al., 2017; Finn et al., 2018]. To solve each regression task, we employ a fully connected network with 2 hidden layers, each with 64 hidden units. The goal is to recover the underlying periodic structure of unseen sine waves from a few observations and previous tasks. We set the same experiment setup as that in [Finn et al., 2017]. We can evaluate the quality of the structure recovered by measuring the root mean squared errors (RMSE) over the novel tasks.

Results. The Fig. 3 illustrates the qualitative performance of some baselines. MAML nearly recovers the underlying function but needs more rounds r and is slower than our method. MLAP, hindered by a small m, fails to capture underlying patterns, highlighting a gap between theory and practice The meta-knowledge extracted by our method captures more comprehensive task environment information and can be easily adapted to new tasks.

5.2 Few-shot Regression

Setup. In this subsection, we consider regression experiments in one synthetic and four real-world meta-learning environments. The synthetic environment involves regression on 2-dimensional mixture of Cauchy distributions plus a random Gaussian Processes function. We utilize datasets corresponding to various calibration sessions of the Swiss Free Electron Laser (SwissFEL) [Milne et al., 2017]. Additionally, we employ data from the PhysioNet 2012 challenge [Silva et al., 2012], consisting of time series of electronic health measurements from patients in intensive care. Specifically, this includes the Glasgow Coma Scale (GCS) and the hematocrit value (HCT). In this context, each task corresponds to a different patient. Finally, we make use of the Intel Berkeley Research Lab temperature sensor dataset (Berkeley-Sensor) [Madden, 2004], where the tasks involve auto-regressive prediction of temperature measurements from sensors placed in various locations throughout the building.

Results. Table 1 shows that our method achieves comparable results across five regression environments. Specifically, it can obtain the best test errors in two regression environments and competitive errors in the remaining three. This demonstrates that the introduced meta-learning framework is not only theoretical sound, but also yields competitive empirical performance in practice.

5.3 Image Classification

Setup. In this subsection, we conduct classification experiments on augmented MNIST dataset, and compare our bounds with previous works. Each task from the same environment is constructed by a limited number of pixel swaps or a random permutation of the labels to ensure task-relatedness. We followed previous works [Amit and Meir, 2018; Liu et al., 2021] on all experimental details During the meta-training phase, we choose 10 training tasks, each consisting of 60,000 training examples; while in the meta-test phase, each task is constructed with reduced training samples, specifically 2,000. We utilize a four-layer fully-connected network for shuffled pixels experiments, and a four-layer convolutional network for permuted labels experiments.

Results. Table 2 shows the comparison of various bounds for both shuffled pixels and permuted labels experiments. The performance of our classic bounds is significantly better than the existing bounds, in terms of test bounds and test errors over the novel tasks. This demonstrates that our framework is not only theoretically sound, but also achieves a competitive generalization performance guarantee over all tasks from different environments. Meanwhile, the prediction performance of different methods gets worse with the increase in the number of pixel swaps. This also indicates the success of meta-learning is significantly influenced by the task-relatedness of the environment.

Next, we investigate the impact of different numbers of training tasks n and sample sizes per task m on the quality of the learned prior. The Fig. 5 (see Appendix C.3) plots the average test error of learning new tasks based on the number of training tasks and sample sizes per task in two distinct environments: the permuted labels environment and the permuted pixels environment where 100, 200, and 300 pixel swaps. The results demonstrate that the increased number of tasks (n) and sample sizes (m) used for learning the prior leads to improved performance on novel tasks. As anticipated, this effect is particularly discernible when the number of meta-training tasks is limited and diminishes as the value of n increases.

6 Conclusion

In this paper, we have introduced a simple framework for meta-learning with generalization guarantees, grounded on a PAC-Bayesian bound that incorporates data-dependent priors. Rather than coercing the data-dependent priors into standard meta-learning settings, we identify a connection between local entropy and PAC-Bayesian theory that allows us to optimize priors for tighter bounds, with a simple local entropy, without the explicit reliance on posteriors. The resulting algorithm is stable, performant, and computationally lightweight, eliminating the need for complex nest optimization. Experimental results demonstrate that our method outperforms previous approaches in terms of fast adaptation, generalization performance, and combating meta-overfitting.

Acknowledgments

Shaogao Lv's research was partially supported by the National Natural Science Foundation of China (No. 12371291, No. 72342019). This work was partially supported by the National Key Research and Development Program of China (No. 2022ZD0115301, No. 2018AAA0100204, No. 2021ZD0112900), the National Natural Science Foundation of China (No. 62206139), the Major Key Project of PCL (PCL2023A09), a key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. JCYJ20200109113403826), and an Open Research Project of Zhejiang Lab (NO. 2022RC0AB04).