Laravel Horizon: Queue Metrics, Supervisor Tuning, and Reliable Job Throughput

Laravel Horizon: Beyond the Pretty Dashboard

Most teams install Horizon, glance at the dashboard, and call it done. The real value is in understanding how Horizon's supervisor model maps to Redis queue semantics, how to read throughput metrics meaningfully, and how to prevent silent job loss under load.

How Horizon's Supervisor Model Works

Horizon runs a master process that spawns one or more supervisors. Each supervisor manages a pool of worker processes for a specific queue (or set of queues). Workers are forked PHP processes — they boot the application once and then loop over jobs.

The key config lives in config/horizon.php:

'environments' => [
    'production' => [
        'supervisor-default' => [
            'connection' => 'redis',
            'queue' => ['critical', 'default', 'low'],
            'balance' => 'auto',
            'autoScalingStrategy' => 'time',
            'minProcesses' => 2,
            'maxProcesses' => 20,
            'balanceCooldown' => 3,
            'tries' => 3,
            'timeout' => 90,
            'memory' => 256,
        ],
    ],
],

A few non-obvious details:

• balance => 'auto' uses the autoScalingStrategy to decide how to distribute workers. time scales based on wait time; size scales based on queue depth. For latency-sensitive queues, prefer time.
• balanceCooldown prevents thrashing. Three seconds is aggressive — consider 10–30 seconds for stable workloads.
• timeout must be shorter than your Redis BLPOP timeout and shorter than any upstream HTTP timeout in the job. A job that exceeds timeout is killed with SIGKILL, not gracefully.

Separating Queues by Supervisor

Running critical, default, and low in a single supervisor means a burst on low can starve critical. Split them:

'supervisor-critical' => [
    'queue' => ['critical'],
    'balance' => 'simple',
    'minProcesses' => 5,
    'maxProcesses' => 5, // fixed — always ready
    'tries' => 1,
    'timeout' => 30,
],
'supervisor-default' => [
    'queue' => ['default'],
    'balance' => 'auto',
    'autoScalingStrategy' => 'time',
    'minProcesses' => 2,
    'maxProcesses' => 15,
    'tries' => 3,
    'timeout' => 60,
],
'supervisor-low' => [
    'queue' => ['low'],
    'balance' => 'auto',
    'minProcesses' => 1,
    'maxProcesses' => 5,
    'tries' => 5,
    'timeout' => 120,
],

Fixed process counts on critical eliminate cold-start latency. Auto-scaling on default and low handles burst without wasting memory at idle.

Reading Horizon Metrics Correctly

Horizon stores metrics in Redis under horizon: keys. The dashboard shows throughput (jobs/minute) and runtime (average execution time). Two traps:

1. Throughput is a rolling average — a spike followed by silence looks healthy. Export raw metrics to your APM (Datadog, New Relic) via the Horizon::routeMailNotificationsTo and snapshot approach, or query Redis directly.
2. Runtime outliers are hidden — the average masks P99 slowness. Instrument your jobs:

public function handle(): void
{
    $start = hrtime(true);

    // ... job logic ...

    $ms = (hrtime(true) - $start) / 1e6;
    logger()->channel('metrics')->info('job.runtime', [
        'job' => static::class,
        'ms' => $ms,
    ]);
}

Ship these logs to a log aggregator and build P95/P99 dashboards there.

Graceful Failure Handling

Horizon respects $tries, $backoff, and $failOnTimeout. Use them deliberately:

class SendWebhookJob implements ShouldQueue
{
    public int $tries = 5;
    public bool $failOnTimeout = true;
    public int $timeout = 20;

    public function backoff(): array
    {
        return [10, 30, 60, 120, 300]; // exponential-ish
    }

    public function failed(Throwable $e): void
    {
        WebhookDelivery::markFailed($this->webhookId, $e->getMessage());
        // notify, alert, compensate
    }
}

Set $failOnTimeout = true so a hung job doesn't silently retry forever — it fails fast and triggers failed().

Deployment Without Dropping Jobs

Horizon workers are long-lived. On deploy:

php artisan horizon:terminate

This sends SIGTERM to the master, which propagates to workers. Workers finish their current job, then exit. Your process supervisor (Supervisor, systemd) restarts Horizon with the new code. Combine with a zero-downtime deploy tool (Envoyer, Deployer) to ensure the terminate fires after the new release is in place.

Takeaways

• Split high-priority queues into dedicated supervisors with fixed process counts.
• Use balance => 'auto' with autoScalingStrategy => 'time' for latency-sensitive work.
• Set failOnTimeout = true and define backoff() arrays to avoid thundering-herd retries.
• Export raw job runtime to your APM — dashboard averages hide P99 pain.
• Always horizon:terminate on deploy; never horizon:restart alone in production.