Laravel Horizon: Queue Metrics, Supervisor Tuning, and Graceful Scaling in Production
#laravel #horizon #queues #production #redis

Laravel Horizon: Queue Metrics, Supervisor Tuning, and Graceful Scaling in Production

3 min read Mohamed Said Mohamed Said

Laravel Horizon in Production: Beyond the Pretty Dashboard

Horizon ships with a slick UI, but most teams stop there. The real value is in its supervisor model, metric snapshots, and the knobs that determine whether your queue survives a traffic spike or silently drops work.

How Horizon Supervisors Actually Work

Horizon manages one or more supervisor processes, each of which spawns a pool of queue:work child processes. The key insight: supervisors are not queues. A single supervisor can watch multiple queues with a priority order.

// config/horizon.php
'environments' => [
    'production' => [
        'supervisor-default' => [
            'connection' => 'redis',
            'queue' => ['critical', 'default', 'low'],
            'balance' => 'auto',
            'autoScalingStrategy' => 'time',
            'maxProcesses' => 20,
            'minProcesses' => 2,
            'maxTime' => 0,
            'maxJobs' => 500,
            'memory' => 256,
            'tries' => 3,
            'timeout' => 90,
            'nice' => 0,
        ],
    ],
],

balance => 'auto' tells Horizon to redistribute workers across queues based on wait time. autoScalingStrategy => 'time' scales toward minimising wait time rather than raw queue size — better for latency-sensitive work.

Choosing the Right Balance Strategy

| Strategy | Scales on | Best for | |---|---|---| | simple | Queue size ratio | Predictable, uniform jobs | | auto | Wait time | Mixed job durations | | false | Fixed processes | Isolated, critical queues |

For a SaaS with a critical queue (password resets, webhooks) and a low queue (report generation), use two supervisors with balance => false on critical and auto on low:

'supervisor-critical' => [
    'queue' => ['critical'],
    'balance' => false,
    'minProcesses' => 3,
    'maxProcesses' => 3,
],
'supervisor-bulk' => [
    'queue' => ['default', 'low'],
    'balance' => 'auto',
    'minProcesses' => 1,
    'maxProcesses' => 15,
    'autoScalingStrategy' => 'time',
],

This guarantees critical work always has dedicated workers, while bulk work scales elastically.

Reading Horizon Metrics Programmatically

Horizon stores throughput and runtime snapshots in Redis. You can query them directly:

use Laravel\Horizon\Contracts\MetricsRepository;

$metrics = app(MetricsRepository::class);

$throughput = $metrics->throughputForQueue('default'); // jobs/min
$runtime = $metrics->runtimeForQueue('default');       // ms average

Pipe these into your observability stack (Datadog, Prometheus via a custom exporter) rather than relying solely on the Horizon UI. Alert when throughputForQueue drops to zero during business hours — that's a dead worker, not an empty queue.

Graceful Termination and Long-Running Jobs

Horizon sends SIGTERM to workers during deploys. Workers finish their current job, then exit. The danger: a job that runs for 180 seconds will block the deploy for 180 seconds per worker.

Mitigate with maxTime (seconds a worker runs before self-terminating after its current job) and by designing long jobs to be interruptible:

public function handle(): void
{
    foreach ($this->chunks() as $chunk) {
        if ($this->job->isDeleted() || $this->job->isReleased()) {
            return;
        }

        $this->processChunk($chunk);

        // Respect the terminate signal between chunks
        if (app('queue.worker')->shouldQuit) {
            $this->release(5);
            return;
        }
    }
}

Accessing app('queue.worker')->shouldQuit lets a job voluntarily release itself back to the queue when a shutdown is pending, preventing message loss.

Memory Leaks and maxJobs

PHP workers accumulate memory over time — Eloquent model event listeners, static caches, and package bootstrapping all contribute. maxJobs => 500 forces a worker restart after 500 jobs, which is a pragmatic ceiling. Pair it with memory => 256 (MB) as a hard cap:

'maxJobs' => 500,
'memory' => 256,

Monitor actual worker memory with horizon:snapshot on a schedule:

// app/Console/Kernel.php
$schedule->command('horizon:snapshot')->everyFiveMinutes();

Without snapshots, the Horizon metrics graphs are empty.

Takeaways

  • Use two supervisors to isolate critical queues from bulk work; never mix them with auto balance.
  • Set autoScalingStrategy => 'time' for latency-sensitive queues; size can over-provision on bursty workloads.
  • Export Horizon metrics to your APM — don't rely on the UI for alerting.
  • Design long jobs to check shouldQuit between chunks so deploys don't stall.
  • Schedule horizon:snapshot every five minutes or your throughput graphs will be blank.

Found this useful?

Frequently Asked Questions

3 questions
Q01 What is the difference between `balance => 'auto'` and `balance => 'simple'` in Horizon?
`simple` distributes workers proportionally to queue size. `auto` targets wait time, so it shifts workers toward queues where jobs are waiting longest regardless of raw count — better when jobs have variable durations.
Q02 How do I prevent a long-running job from blocking a Horizon deploy?
Check `app('queue.worker')->shouldQuit` between processing units inside your job. When it is true, call `$this->release()` to put the job back on the queue and return early, allowing the worker to exit cleanly.
Q03 Why are my Horizon throughput graphs empty?
Horizon only records metric snapshots when `horizon:snapshot` runs. Schedule it every five minutes in your console kernel; without it, the metrics repository has no data to display.

Continue reading

More Articles

View all