Credit Assignment and Fine-Tuning Enhanced Reinforcement Learning for Collaborative Spatial Crowdsourcing

1 Introduction

With the popularity of smart mobile devices, spatial crowdsourcing (SC) has emerged as a promising computing paradigm [Li et al., 2022c; Li et al., 2022b], where workers perform spatial tasks assigned by the SC platform for payments [Wu et al., 2024b; Mei et al., 2024]. The increasing complexity and diverse requirements of spatial tasks have driven the evolution of a more sophisticated model within SC, namely Collaborative Spatial Crowdsourcing (CSC). In this paradigm, multiple workers can collaboratively complete tasks, leveraging a collective workforce to accelerate task completion. Such collaborative approaches are prevalent in various real-world applications, including home renovation, Assigned Optional

* Corresponding authors.

Unmatched Workers

Matched Workers

furniture assembly, and venue arrangement [Cheng et al., 2016; Cheng et al., 2019; Zhao et al., 2024], etc.

To illustrate CSC, consider the example shown in Fig.1, which involves four workers (A-D) and two tasks (X, Y). The initial configuration assigns workers A and B to tasks X and Y respectively, while workers C and D remain unallocated. The scenario is characterized by worker heterogeneity: worker C is exclusively qualified for task Y, while worker D possesses the versatility to perform either task. Task completion times directly impact payment structures, and the assignment of worker D to either task X or Y yields different compensation outcomes. This study aims to optimize the platform's revenue through strategic task allocation. Notably, as the scale of workers and tasks expands in real-world applications, the computational complexity of CSC increases exponentially.

Existing research on CSC has explored diverse approaches, ranging from heuristic algorithms [Cheng et al., 2019; Zhao et al., 2024] to learning-based methods [Zhao et al., 2023]. Cheng et al. (2019) introduced a collaborative framework where multiple workers jointly handle space tasks to maximize overall collaboration quality. Zhao et al.(2024) developed an equilibrium-based approach that combines simulated annealing with Nash equilibrium refinement to optimize total rewards while maintaining priority-aware fairness in task assignments. Zhao et al. (2023) advanced the field by incorporating mutual information and attention mechanisms for group preference modeling, alongside tree decomposition and curriculum learning strategies to enhance task allocation efficiency. Building on this progress, Zhan et al.(2024) integrated graph neural networks to assess worker trustworthiness and implemented a specialized Tabu search algorithm for worker assignment.

Although multi-agent reinforcement learning (MARL) has been widely applied in task allocation problems, its application to CSC faces significant challenges due to delayed and sparse reward feedback. The credit assignment of reward in CSC is particularly complex due to the temporal nature of task completion, where payments are only generated upon task completion, yet workers join and contribute at different timestamps throughout the execution process. This temporal misalignment complicates reward allocation, as simple time-based distribution methods fail to capture the nuanced dynamics of worker participation. Furthermore, workers' heterogeneous contributions to the same task create additional complexity in fair reward distribution - certain workers may provide critical contributions despite shorter participation periods. For instance, strategically assigning a worker to a nearly-completed task rather than a newly published one can accelerate task completion and worker availability, creating a cascade effect that enhances overall platform efficiency. In such cases, although the worker's time investment may be brief, their strategic contribution significantly impacts the platform's revenue, warranting higher rewards despite shorter participation duration.

Another critical challenge in CSC stems from the inherent heterogeneity across tasks. As each task presents unique characteristics and requirements, the learning experiences derived from different tasks may contain significant noise and variability. This non-stationary data distribution makes it challenging to effectively learn knowledge across tasks, as parameters trained on one task may not generalize well to others due to task-specific patterns and biases. Such diversity in task characteristics creates substantial learning interference, potentially compromising the stability and robustness of the learned policies.

To address the aforementioned challenges, this paper presents CAFE (Credit Assignment and Fine-tuning Enhanced), a novel framework for MARL that incorporates two key components: (i) a credit attribution mechanism that accurately evaluates individual worker contributions, and (ii) a robust parameter fine-tuning approach that effectively mitigates task-level sample noise. The key contributions of our work are as follows:

• A Causality Reward Redistribution Methods for MARL. We propose a multi-agent reinforcement learning approach to address the CSC problem, and subsequently introduce a causal-perspective reward redistribution scheme that leverages Bayesian surprise to quantify individual agent contributions.
• A Bi-Level Meta-Optimization Approach. We propose a bi-level optimization framework leveraging implicit gradients, which concurrently optimizes RL objectives and reward redistribution parameters by strategically integrating historical learning trajectories.
• Efficient Parameters Fine-Tuning for Diverse Data. To address the non-stationary data distribution challenge in real-world datasets, we propose a parameters fine-tuning approach built upon implicit learning that enables rapid adaptation without deviating from the original optimization objective.
• Extensive Empirical Studies. We evaluate the effectiveness of our proposed approaches on real-world datasets. Extensive experiments demonstrate the effectiveness of our methods.

2 RelatedWorks

2.1 Reinforcement Learning in Spatial Crowdsourcing

Reinforcement learning (RL) is increasingly applied in crowdsourcing to optimize task assignment and scheduling through adaptive policy learning in dynamic environments [Li et al., 2025; Wu et al., 2024a]. A common approach is to model task assignment as a Markov Decision Process. For instance, Zong et al. (2022) developed an end-to-end multiagent system for pickup and delivery problems. In air-ground spatial crowdsourcing, Wang et al. (2023) introduced a multicenter attention-based graph convolutional network for communication. In contrast, Ye et al. (2023) proposed a heterogeneous multi-agent framework exploring both individual and collaborative environments. Jiang et al. (2023) designed a fairness-aware concurrent dispatch system for instant delivery services. For ride-hailing order dispatching, Zhang et al. proposed an offline deep reinforcement learning framework with a method for managing dynamic nondeterministic action spaces [Zhang et al., 2024] Beyond direct application in decision-making, RL enhances established matching methods. It can determine the sliding window size of decision time steps [Li et al., 2022c] and optimize existing algorithms based on game theory [Li et al., 2023], large-scale search [Li et al., 2022a], and combinatorial optimization [Tong et al., 2021].

2.2 Credit Assignment in Reinforcement Learning

Delayed rewards present a fundamental challenge in reinforcement learning, wherein rewards manifest several timesteps after an action, making it difficult for agents to establish clear associations between actions and their consequent outcomes. Reward reshaping has emerged as a widely adopted approach to address this temporal credit assignment problem. Zhu et al. (2023) investigated episodic reinforcement learning with trajectory feedback, introducing an adaptive reward redistribution method grounded in bi-level optimization. In a complementary approach, Ren et al. (2021) developed a randomized return decomposition technique to transform long-horizon delayed reward problems into more manageable shorter sequences, thereby enabling effective training through mini-batch gradient descent. Advancing the field further, Ma et al. (2024) introduced a novel dual-agent framework comprising a policy agent for optimal behavior learning and a reward agent for generating auxiliary reward signals, thus achieving reward design without dependence on expert knowledge or hand-crafted functions. Qu et al. (2024) demonstrated that large models can be leveraged to encapsulate valuable decision-making knowledge, offering a promising avenue for reward redistribution. This approach of utilizing large models for processing state features has shown considerable promise in effectively addressing the temporal credit assignment challenge. In addition to credit assignment methods, many studies address the reward distribution problem from other perspectives, such as enhancing agent training in reinforcement learning by combining the advantages of group strategies and individual strategies [Wang et al., 2022].

3 ProblemFormulation

Generally, we define the CSC problem on a road network represented by a graph G = (N,E), where n ∈ N represents road intersections, and eij ∈ E denotes roads connecting two intersections. Next, we define other entities related to the CSC problem.

Worker. A worker w ∈ W is described by the tuple w = (l,K), where l denotes the worker's location and K represents the set of skills. Furthermore, to simplify the problem, we assume that all workers have the same basic attributes, such as movement speed, work speed, etc.

Spatial Task. A spatial task τ ∈ T is described by the tuple τ = (tc , te , td , l, k, q, ρ). The task's temporal aspects are represented by tc , te , and td , indicating the creation time, expected completion time, and deadline, respectively. Additionally, l is the location of the task, k represents the skill required to complete the task, each task has exactly one skill requirement, and q represents the task quantity, which is proportional to the workload completed by a worker per unit of time. Furthermore, ρ signifies the payment obtained upon successful completion of the task.

Considering the actual accomplishment of the task, the real benefit ρ of the task is expressed as follows:

while tf is the finish time of the task, ρmaxis the maximum payment that the task publisher can offer, ζ is the penalty rate, as shown in Fig.2.

CSC Problem. Given a stream of tasks T and a set of works W, CSC problem aims to find the optimal assignment M to maximize the global revenue P .

4 Methodology

4.1 Markov Decision Process

We establish a multi-agent network system based on Markov Decision Process (MDP). This system is represented by the tuple (N, S,O,A,R,P, γ), where P represents the state transition probability matrix and γ is the discount factor. Next, we elaborate on the meanings of other components in this MDP tuple.

Agent n ∈ N: Each worker w is modeled as an agent n, and all agents are assumed to share an identical network structure and parameters.

State s ∈ S: The global state st is represented by a vector (T ,W), which includes both worker and spatial task realtime information.

Observation o ∈ O: The observation ont is represented as a triplet (Tnear , Wnear , Widle, Toptinal , wself ), which captures the conditions of nearby workers, surrounding tasks, idle workers, surrounding optional tasks, and the agent's own real-time information.

Action a ∈ A: The action is defined as {τ0, τ1, τ2, . . . , τx}, where τ0 represents the option of not selecting any task and τx represents the task index of the surrounding optional tasks.

Reward r ∈ R: Reward serves as the critical factor in evaluating the quality of agent actions and directly influences the model's training outcomes. In the methodology section, we propose a novel reward redistribution approach.

4.2 Neural Network Architecture

Our proposed method, CAFE, builds upon the Actor-Critic framework by introducing a novel neural network architecture. This architecture features a base module that seamlessly integrates with established multi-agent actor-critic approaches such as MAPPO [Yu et al., 2022], IPPO [De Witt et al., 2020], and MADDPG [Lowe et al., 2017] as shown in Fig.3. In the following sections, we present a detailed description of the key components and their interconnections within our proposed architecture.

Actor: The actor network employs four Multi-Head Attention (MHA) [Vaswani, 2017] layers to process the observation, followed by a Multi-Layer Perceptron (MLP) to integrate the outputs of the MHA layers, and finally uses a softmax function to output the action.

Critic: The Critic network has a similar architecture to the actor network, with the key difference being that the Critic network outputs the evaluation of the current state.

4.3 Reward Redistribution

To comprehensively evaluate how individual workers' actions affect overall revenue, we propose a reward redistribution mechanism that quantifies each worker's contribution.

To systematically analyze agent contributions in MARL, we first develop a structural causal model (SCM) that captures the underlying causal relationships of the process (Fig.4). Within this model, directional arrows represent inherent causal dependencies between components. To quantify how each agent's action ait causally impacts the subsequent state st+1, we employ conditional mutual information (CMI):

To enhance computational efficiency while preserving the essential causal relationships, we make two key approximations. First, we replace the global state st with agent ni's local observation oit . Second, we consider only the actions of nearby agents a-i t,o instead of the complete joint action set a-i t . These simplifications are theoretically justified by two established principles: independent causal mechanisms [Peters et al., 2017] and spatiotemporally bounded interactions [Sch¨olkopf et al., 2021] between autonomous entities [Du et al., 2024]. This leads to our simplified CMI formulation:

Since our primary objective is maximizing platform revenue, we focus on how agents' actions influence future task payments within their local observation range. We implement this through a Variational Autoencoder (VAE) [Liu et al., 2023] architecture, where zρ encodes the revenue information of nearby tasks. This allows us to reformulate CMI in terms of zρ and ait , yielding:

The final expression employs Bayesian surprise [Li et al., 2024; Mazzaglia et al., 2022] to compute CMI. Our VAE implementation, illustrated in Fig.5, consists of:

With the corresponding loss function:

Relying solely on an action's state-impact assessment as the agent's reward is suboptimal. Traditional reward mechanisms typically incentivize actions that dramatically alter the environment, which may diverge from the goal of maximizing platform revenue. To mitigate this misalignment, we introduce a regularization term into the reward function:

While ϕ1 and ϕ2 are hyperparameters. In the next subsection. we will introduce the method for determining the specific values of hyperparameters ϕ.

4.4 Implicitly Learning

To optimize the hyperparameters in the reward function, we propose a novel bi-level optimization method based on implicit gradients [Rajeswaran et al., 2019] to tune the reward function's hyperparameters. This approach enables the concurrent optimization of hyperparameters during the reinforcement learning process.

The bi-level optimization method consists of two levels: an inner-level that optimizes the reinforcement learning model and an outer-level that optimizes the reward function's hyperparameters. The loss function can be expressed as:

while J is the target function of the actor's network. And we aim to ensure that the rewards after redistribution still maintain the same long-term return as the undivided rewards, thereby maintaining the invariance of the optimal policy. To achieve this, we use LΛ to regulate the long-term reward, which is expressed as:

η is a hyperparameter used to control the magnitude of the reward, and r(ϕ) is the reward function of ϕ.

We define F(ϕ) as the outer-level loss function, and its gradient is:

while dLΛ/dϕ is straightforward to compute, dJ(θ∗ , ϕ)/dϕ is more challenging to calculate. We can derive dJ(θ∗ , ϕ)/dϕ using the chain rule as:

The key to calculating dJ(θ∗ , ϕ)/dϕ is to solve for ∂θ ∂ϕ . First, consider the situation within the inner level, where the inner loss function has been optimized, meaning that θ attains a reasonable parameter value. In this case, the gradient of θ is close to 0:

Thus, in order to get ∂J/∂θ, we differentiate both sides of the above equation with respect to ϕ:

Next, apply the chain rule to solve the resulting equation:

Rearranging the above equation, we can get:

Here, ∂ 2J/ (∂θi∂θj) represents the Hessian matrix. We calculate ∂θ/∂ϕ by combining the inverse of the Hessian matrix with the secondary gradient ∂ 2J/ (∂θ∂ϕ). Therefore, dJ/dϕ is computed as:

Computing second-order derivatives requires substantial computational resources, particularly for neural networks with numerous parameters, which significantly increases the computational complexity of secondary gradient calculations. Yosinski et al. demonstrated that the initial layers of neural networks capture transferable knowledge, while the later layers specialize in task-specific features. Building upon these findings, we propose replacing the complete network θ with only its later layers θl , focusing our optimization efforts on these task-specific components. This approach substantially reduces the computational overhead by limiting calculations to a subset of the network parameters. So the gradient of ϕ can be represented as:

4.5 Swift Parameter Refinement

In real-world crowdsourcing scenarios, learning from largescale, complicated, and noisy data presents significant challenges. When task data distributions are non-stationary, a Reward Redistribution network with fixed parameters may not generalize effectively across diverse conditions. However, by leveraging the inherent correlations between tasks, rapid parameter adaptation allows the network to capture trajectoryspecific patterns better and improve performance across varying contexts. Therefore, in this subsection, we propose a method for rapidly adapting parameters to better handle variations in real-world scenarios.

Firstly, we approximate Equ.6 using the Taylor series expansion to the second-order term[Wu et al., 2024c]:

While ˆ ϕ is the best possible initialization obtained through optimal multi-trajectory optimization. And then we take the derivative of the above expression with respect to ϕ:

Consider the concept of implicit gradients for optimizing the network parameters. If ϕ∗ is the optimal neural network parameter for the dataset, then the gradient of the loss function at ϕ∗ is zero, i.e.:

This leads to the following expression:

Therefore, we can observe the optimal parameters ϕ∗ in a single step as follows:

4.6 Model Training

The Independent Proximal Policy Optimization (IPPO) algorithm [De Witt et al., 2020] is utilized as an example. The pseudocode is presented in Algorithm 1.

In IPPO, the objective function of the actor network and the loss function of the critic network respectively are as follows:

Algorithm 1 Model Training and Fine-Tuning Algorithm

• Input: actor network θ and critic network ϕ Output: θ,ϕ 1: Initialize policy network θ, value network φ, hyperparameters ϕ1 and ϕ2. 2: Initialize replay buffer D 3: for episode← 1, . . . ,E do 4: Use θ to execute actions and collect experience, then store transitions (st, {ont }, {ant }, {rn t }, st+1) in the data buffer D 5: if parameter ϕ has been sufficiently trained then 6: Fine-tune ϕ by Equ.(19) 7: end if 8: for iteration ← 1 to M do 9: Sample mini-batch from D 10: for each agent n ∈ N do 11: Update actor network θ and critic network ψ 12: Update hyper parameter ϕ by Equ.(14) 13: end for 14: end for 15: end for 16: return θ, ϕ

where ψ = πθ(at|ot) πθold (at|ot) is the probability ratio between the current policy and the old policy and ϵ is a hyperparameter that controls the clipping range. Vϕ(ot) is the current estimate of the value function and V target is the target value.

5 Experimental Evaluation

In this section, we evaluate CAFE on some Simulation experiments to answer the following questions: Q1: Is the credit assignment performance of CAFE superior to that of stateof-the-art frameworks in delayed reward settings? Q2: Is the fine-tuning method for CAFE accurate and effective in facilitating reinforcement learning? Q3: Is the performance of CAFE superior to that of state-of-the-art collaborative spatial crowdsourcing methods?

5.1 Experimental Setup

Datasets. The road network data was extracted fromOpenStreetMap for two cities: Chengdu (17,378 nodes, 26,961 edges) and Haikou (9,667 nodes, 14,458 edges). For spatial tasks, orders were extracted from DiDi, comprising 7,065,937 orders for Chengdu and 14,160,170 orders for Haikou, where the pick-up location of the order corresponds to the location of the spatial tasks. The task skill model was developed based on Point-of-Interest (POI) types from AutoNavi Maps, classified into 8 distinct categories. Each task derives skill requirements from nearby POIs. Worker skills were modeled according to a normal distribution (µ = 2.5, σ2 = 1), while worker initial locations were also initialized by DiDi order.

Implementation Details. The Adam optimizer [Kingma, 2014] was employed for training all models, with a learning rate of 1 × 10-3 . And the hyperparameter update rate is maintained at 1 × 10-3 . In the simulation experiments, worker speed was set to 10 m/s, with an acceptable task range of 5 km. All experimental procedures are executed on a computational system operating Windows 10 with Python 3.8, equipped with Intel Core i9-13900K CPU @ 5.80 GHz, NVIDIA GeForce RTX 4080 super GPU, and 64 GB RAM.

Baselines of Reward Redistribution. We use the following algorithm to compare CAFE's performance in multi-agent reinforcement learning with delayed rewards: Individual Reward (IR), uses the remaining task working time as the reward; ICES [Li et al., 2024], uses individual contributions as intrinsic exploration scaffolds to motivate exploration by assessing each agent's contribution from a global perspective; ReLara [Ma et al., 2024], a dual-agent reward shaping framework composed of two synergistic agents, a policy agent to learn the optimal behavior and a reward agent to generate reward. CA-noFE, derived from CAFE but without the fine-tuning component.

Baselines of Task Assignment. We use the following algorithm to compare CAFE's ability for task assignment in the context of the simulated spatial crowdsourcing experiment: Greedy, assigns workers greedily to the nearest tasks that match the required skills; IPPO [De Witt et al., 2020], utilizes the same network as CAFE, but with individual reward training; PAU [Zhao et al., 2024], achieves task assignment through the formation of worker coalitions, modeling the CSC problem as an exact potential game; CA-noFE, derived from CAFE but without the fine-tuning component.

5.2 Comparison of Reward Redistribution

In this subsection, we evaluate the performance of different reward redistribution methods under identical crowdsourcing environments. Using platform revenue as the evaluation metric, we compare the performance variations of models trained with different methods across various training batches. The experimental results are presented in Fig.7. Notably, in the training of CAFE, we first train the model using the CA-noFE method to reduce computational requirements. Once the model training stabilizes, we then introduce the fine-tuning method to improve the reinforcement learning training.

During model training, the IR method progressed slowly due to its simple reward assignment approach. In contrast, ICES and ReLara utilized more advanced reward allocation policies, resulting in faster training. Although CAFE initially showed significant performance fluctuations and required more iterations, it eventually outperformed the other methods. This improvement is attributed to CAFE's secondorder gradient optimization, which takes longer to converge but enables more precise reward allocation. The results confirm CAFE's effectiveness in credit assignment, allowing for more nuanced reward distribution to agents. Compared to the non-fine-tuned CA-noFE method, CAFE enhances training dynamics and generates higher platform revenue, highlighting the importance of its fine-tuning methodology.

5.3 Performance Analysis and Evaluation

We evaluated the proposed algorithms by conducting experiments with different numbers of workers and tasks, using the platform's total revenue as the main metric. All results are averages of five independent runs.

Effect of the number of Tasks. Table 1 compares the revenue performance of various algorithms across different numbers of tasks for two datasets. The experiments were conducted with a fixed number of workers (Nw = 200), and the results highlight the revenue trends as Nw increases from 50 to 350. The CAFE algorithm consistently achieves the highest revenue across both datasets, demonstrating its superiority in optimizing task allocation. The results demonstrate that the CAFE outperforms all other methods across different task numbers and datasets, particularly excelling in large-scale task scenarios. This confirms its effectiveness in optimizing task allocation policies.

Effect of the number of workers. Table 2 compares the revenue performance of various algorithms across different numbers of workers, with a fixed number of tasks (Nτ = 200). The results demonstrate that CAFE consistently outperforms other methods, achieving the highest revenues across most configurations, particularly a sNw increase. Notably, CAFE exhibits superior scalability and stability, with lower variance in performance compared to alternatives such as CA-noFE and PAU. While Greedy performs the worst due to its inability to optimize task-worker assignments effectively, IPPO provides moderate results but fails to match the advanced methods. These findings highlight the robustness of CAFE in leveraging increased worker availability to maximize platform revenue.

The experimental results conclusively demonstrate CAFE's superior performance compared to state-of-the-art collaborative spatial crowdsourcing methods, as evidenced by its higher platform revenue and improved task assignment efficiency. The fine-tuning approach effectively captures each task's unique characteristics, enabling precise reward allocation and enhancing training dynamics.

6 Conclusion

This paper introduces a novel credit assignment method to address delayed reward challenges in cooperative spatial crowdsourcing using multi-agent reinforcement learning. By leveraging causality and Bayesian theories, we analyze agents' contributions and develop a meta-optimization approach for training reward functions. We further propose a rapid parameter adjustment technique to mitigate non-stationary data distributions across spatial tasks. Extensive experiments on two datasets validate the superior performance of our method compared to existing approaches.

Acknowledgments

This work is supported by the following grants: NSFC Grants 62372416, 61972362, 62036010, and 62325602; HNSF Grant 242300421215.