Laravel Job Batching and Chaining: Coordinating Complex Async Workflows
#laravel #queues #async #job-batching #backend

Laravel Job Batching and Chaining: Coordinating Complex Async Workflows

4 min read Mohamed Said Mohamed Said

Why Simple Queued Jobs Are Not Enough

A single dispatch(new ProcessInvoice($id)) gets you far, but real SaaS workflows are rarely one step. You need to import a CSV, validate each row, enrich records via an external API, then notify the user — and you need to know when all of it finishes, or which part failed.

Laravel's Bus::batch() and job chaining solve exactly this, but the nuances around failure handling, nested batches, and state propagation trip up even experienced engineers.


Job Chaining: Sequential Guarantees

Chaining enforces order. Each job runs only if the previous one succeeded.

use Illuminate\Support\Facades\Bus;

Bus::chain([
    new ValidateImport($importId),
    new EnrichRecords($importId),
    new NotifyUser($importId),
])->onQueue('imports')->dispatch();

If EnrichRecords throws, NotifyUser never runs. The chain is stored in the job payload itself — no extra database row. That simplicity is also a limitation: you cannot inspect chain progress from outside.

Passing State Between Chained Jobs

Avoid coupling jobs through shared mutable state in the database when you can pass identifiers instead. Each job re-queries what it needs:

class EnrichRecords implements ShouldQueue
{
    public function __construct(private readonly int $importId) {}

    public function handle(ImportRepository $repo): void
    {
        $import = $repo->findOrFail($this->importId);
        // enrich and persist
    }
}

This keeps jobs idempotent and safe to retry.


Job Batching: Fan-Out with a Finish Line

Batches let you dispatch many jobs in parallel and react when they all complete — or when any fails.

use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;
use Throwable;

$batch = Bus::batch(
    $importRows->map(fn ($row) => new ProcessRow($row))->all()
)
->then(fn (Batch $batch) => NotifyImportComplete::dispatch($batch->id))
->catch(fn (Batch $batch, Throwable $e) => ImportFailed::dispatch($batch->id, $e->getMessage()))
->finally(fn (Batch $batch) => Import::markFinished($batch->id))
->onQueue('imports')
->dispatch();

$importId = $batch->id; // store for status polling

then fires once when all jobs succeed. catch fires on the first failure. finally always fires. These callbacks are serialized closures stored in the job_batches table — keep them small and side-effect-free.

Allowing Partial Failure

By default, one failed job cancels the batch. For bulk operations where partial success is acceptable:

Bus::batch($jobs)
    ->allowFailures()
    ->then(fn (Batch $b) => $this->summarize($b))
    ->dispatch();

Inside then you can inspect $batch->failedJobs to report which rows failed without aborting the whole import.


Combining Batches and Chains

The real power emerges when you nest them. Run a batch of parallel jobs, then chain a sequential step after all of them finish:

Bus::chain([
    new PrepareImport($importId),
    Bus::batch(
        $rows->map(fn ($r) => new ProcessRow($r))->all()
    )->allowFailures(),
    new FinalizeImport($importId),
])->dispatch();

The chain pauses at the batch step until the batch resolves, then continues to FinalizeImport. This pattern handles fan-out/fan-in without any custom orchestration code.


Pruning the job_batches Table

Batches accumulate rows. Add the prune command to your scheduler:

// routes/console.php
Schedule::command('queue:prune-batches --hours=48 --unfinished=72')
    ->daily();

--unfinished prunes batches that never completed — important for catching leaked batches from deploy-time failures.


Monitoring Batch Progress

Expose a lightweight status endpoint for the frontend:

public function status(string $batchId): JsonResponse
{
    $batch = Bus::findBatch($batchId);

    return response()->json([
        'progress'  => $batch->progress(),
        'finished'  => $batch->finished(),
        'failed'    => $batch->failedJobs,
        'cancelled' => $batch->cancelled(),
    ]);
}

$batch->progress() returns an integer 0–100. Poll this from a Livewire component or Alpine.js interval for a real-time progress bar without WebSockets.


Key Takeaways

  • Use chains for sequential steps where each depends on the previous succeeding.
  • Use batches for parallel fan-out where you need a collective finish line.
  • Nest a batch inside a chain to combine both patterns cleanly.
  • Call allowFailures() on bulk operations; inspect failedJobs in then.
  • Keep batch callbacks minimal — they are serialized closures, not service-container-aware by default.
  • Schedule queue:prune-batches to prevent unbounded table growth.

Found this useful?

Frequently Asked Questions

3 questions
Q01 Can a batch callback dispatch another batch or chain?
Yes. The `then`, `catch`, and `finally` callbacks run in a queue worker context, so you can dispatch new jobs or batches from them. Keep the callbacks thin — offload heavy logic to a dedicated job dispatched from within the callback.
Q02 What happens to a batch when a worker restarts mid-processing?
Each job in the batch is an independent queue message. If a worker restarts, unprocessed jobs remain in the queue and are picked up by the next available worker. The batch row tracks counts atomically, so progress is not lost. Failed jobs increment `failed_jobs` on the batch row.
Q03 Is there a limit to how many jobs I can put in a single batch?
There is no hard framework limit, but very large batches (tens of thousands of jobs) can cause slow dispatch because all job records are inserted in one transaction. Chunk your batch dispatch in groups of a few hundred using `Bus::batch($chunk)->dispatch()` inside a loop, each as a separate batch, then coordinate them with an outer chain or a custom aggregator job.

Continue reading

More Articles

View all