Job Batching with Laravel Horizon: Reliable Async Workflows at Scale

Why Job Batching Deserves More Attention

Laravel's Bus::batch() API has been available since Laravel 8, yet most teams still reach for simple dispatch() calls or manual counters to coordinate parallel work. Paired with Horizon's real-time supervision, batching gives you a first-class primitive for fan-out/fan-in patterns — think bulk imports, report generation, or multi-tenant data migrations — without building your own orchestration layer.

Anatomy of a Batch

A batch is a collection of jobs that share a lifecycle. Laravel tracks completion, failure counts, and pending jobs in the job_batches table.

use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;

$batch = Bus::batch([
    new ProcessChunk($chunk1),
    new ProcessChunk($chunk2),
    new ProcessChunk($chunk3),
])
->name('nightly-import')
->allowFailures()          // keep running even if some jobs fail
->onProgress(function (Batch $batch) {
    logger()->info('Batch progress', [
        'id'        => $batch->id,
        'pending'   => $batch->pendingJobs,
        'failed'    => $batch->failedJobs,
        'progress'  => $batch->progress(),
    ]);
})
->then(function (Batch $batch) {
    // All jobs succeeded
    ImportCompleted::dispatch($batch->id);
})
->catch(function (Batch $batch, \Throwable $e) {
    // At least one job failed (called once per failure when allowFailures is on)
    ImportFailed::dispatch($batch->id, $e->getMessage());
})
->finally(function (Batch $batch) {
    // Always runs — success or failure
    ImportFinished::dispatch($batch->id);
})
->dispatch();

Key distinction: ->catch() fires for each failed job when allowFailures() is set. Without it, the first failure cancels the batch and ->catch() fires once.

Making Jobs Batchable

Add the Batchable trait and guard against cancelled batches:

use Illuminate\Bus\Batchable;
use Illuminate\Contracts\Queue\ShouldQueue;

class ProcessChunk implements ShouldQueue
{
    use Batchable;

    public int $tries = 3;
    public int $backoff = 10;

    public function __construct(private readonly array $rows) {}

    public function handle(): void
    {
        if ($this->batch()?->cancelled()) {
            return; // bail early — another job may have triggered cancellation
        }

        foreach ($this->rows as $row) {
            // process row...
        }
    }
}

Horizon Configuration for Batch Workloads

Batches benefit from dedicated queues so they don't starve interactive jobs.

// config/horizon.php
'environments' => [
    'production' => [
        'supervisor-default' => [
            'connection' => 'redis',
            'queue'      => ['high', 'default'],
            'processes'  => 5,
        ],
        'supervisor-batch' => [
            'connection' => 'redis',
            'queue'      => ['batch'],
            'processes'  => 20,   // scale independently
            'timeout'    => 300,
        ],
    ],
],

Dispatch batch jobs onto the dedicated queue:

Bus::batch($jobs)
    ->onQueue('batch')
    ->dispatch();

Pruning and Observability

Batch records accumulate. Schedule pruning and expose batch status via an API or Filament panel:

// routes/api.php
Route::get('/imports/{batchId}', function (string $batchId) {
    $batch = Bus::findBatch($batchId);
    abort_unless($batch, 404);

    return response()->json([
        'progress'    => $batch->progress(),
        'pending'     => $batch->pendingJobs,
        'failed'      => $batch->failedJobs,
        'finished_at' => $batch->finishedAt,
    ]);
});

In app/Console/Kernel.php (or a scheduled command in Laravel 11+):

$schedule->command('queue:prune-batches --hours=48')->daily();

Nested Batches and Dynamic Fan-Out

You can add jobs to a running batch from within a job — useful when the total work isn't known upfront:

public function handle(): void
{
    $subJobs = $this->discoverMoreWork();

    if ($subJobs) {
        $this->batch()->add($subJobs);
    }
}

Laravel increments pendingJobs atomically, so progress tracking stays accurate.

Takeaways

• Use allowFailures() for resilient fan-out; omit it when partial success is unacceptable.
• Always check $this->batch()?->cancelled() at the top of handle() to avoid wasted work.
• Isolate batch queues in Horizon so throughput scales independently of interactive queues.
• Prune job_batches on a schedule — unbounded growth will hurt query performance.
• Expose batch progress via a lightweight endpoint or admin panel for operational visibility.
• Dynamic batch()->add() enables adaptive fan-out when the total job count is unknown at dispatch time.