Laravel Horizon: Queue Metrics, Supervisor Tuning, and Graceful Scaling in Production
#laravel #horizon #queues #production #performance

Laravel Horizon: Queue Metrics, Supervisor Tuning, and Graceful Scaling in Production

3 min read Mohamed Said Mohamed Said

Beyond the Pretty Dashboard

Most teams install Horizon, glance at the dashboard, and call it done. That's leaving serious reliability and throughput on the table. Horizon's real value is in its supervisor model, its Redis-backed metrics, and the levers it exposes for graceful scaling — none of which are obvious from the UI alone.


Supervisor Configuration That Actually Matters

Horizon's config/horizon.php environments block is where production behaviour is defined. The defaults are conservative; here's a production-grade starting point:

'production' => [
    'supervisor-default' => [
        'connection'  => 'redis',
        'queue'       => ['critical', 'default', 'low'],
        'balance'     => 'auto',
        'minProcesses' => 2,
        'maxProcesses' => 20,
        'balanceMaxShift' => 3,   // max workers added/removed per cycle
        'balanceCooldown' => 3,   // seconds between rebalance attempts
        'tries'       => 3,
        'timeout'     => 90,
        'nice'        => 0,
    ],
],

balance Modes

| Mode | Behaviour | |---|---| | simple | Splits maxProcesses evenly across queues | | auto | Scales per-queue based on workload ratio | | false | Fixed process count, no autoscaling |

auto is almost always correct for mixed-priority workloads. The balanceMaxShift cap prevents Horizon from spawning 15 workers in one cycle and overwhelming your database connection pool.


Reading Horizon Metrics Programmatically

The dashboard shows throughput and wait times, but you can query the same data in code for alerting or custom dashboards:

use Laravel\Horizon\Contracts\MetricsRepository;

$metrics = app(MetricsRepository::class);

// Throughput (jobs/minute) for a queue
$throughput = $metrics->throughputForQueue('critical');

// Average runtime in milliseconds
$runtime = $metrics->runtimeForQueue('critical');

// Snapshot history (last 24 two-minute windows)
$snapshots = $metrics->snapshotsForQueue('critical');

Wire these into a scheduled command that pushes to your APM or fires a Slack alert when runtimeForQueue exceeds your SLA threshold. Horizon stores snapshots in Redis sorted sets, so reads are O(log n) and cheap.


Graceful Worker Termination

Horizon sends SIGTERM to workers when you deploy. Workers finish their current job and exit — but only if timeout is set correctly. A job that runs longer than timeout is killed mid-execution.

Rule: timeout in horizon.php must be less than the PHP process timeout (max_execution_time) and less than the queue connection's retry_after.

// config/queue.php
'redis' => [
    'driver'       => 'redis',
    'retry_after'  => 120, // seconds before job is re-queued
    ...
],

// config/horizon.php supervisor
'timeout' => 90, // must be < retry_after

If timeout >= retry_after, a slow job gets re-queued while still running, causing duplicate execution.


Scaling Horizon Itself: Multiple Supervisors

For heterogeneous workloads, use multiple named supervisors rather than one catch-all:

'production' => [
    'supervisor-heavy' => [
        'queue'        => ['pdf-generation', 'video-processing'],
        'balance'      => 'auto',
        'minProcesses' => 1,
        'maxProcesses' => 5,
        'timeout'      => 300,
    ],
    'supervisor-fast' => [
        'queue'        => ['notifications', 'webhooks'],
        'balance'      => 'auto',
        'minProcesses' => 5,
        'maxProcesses' => 30,
        'timeout'      => 30,
    ],
],

This prevents a backlog of slow PDF jobs from starving your notification queue. Each supervisor manages its own process pool independently.


Deployment Without Dropping Jobs

# In your deploy script — order matters
php artisan horizon:pause   # stop accepting new jobs
sleep 5                      # let in-flight jobs finish
php artisan horizon:terminate
# deploy code
php artisan horizon:publish  # re-publish assets if updated
sudo supervisord restart horizon

horizon:pause sets a Redis flag; workers check it between jobs and idle rather than pick up new work. This gives you a clean drain window without SIGKILL.


Key Takeaways

  • Set balanceMaxShift and balanceCooldown to prevent thundering-herd autoscaling.
  • Always keep timeout < retry_after to avoid duplicate job execution.
  • Use multiple named supervisors to isolate slow queues from fast ones.
  • Query MetricsRepository programmatically for SLA alerting, not just the UI.
  • Drain workers with horizon:pause before horizon:terminate on every deploy.

Found this useful?

Frequently Asked Questions

3 questions
Q01 Why do my jobs run twice after deploying with Horizon?
This is almost always a timeout/retry_after mismatch. If your supervisor `timeout` equals or exceeds the queue connection's `retry_after`, Redis re-queues the job before the worker finishes it. Set `timeout` to at least 30 seconds less than `retry_after`.
Q02 When should I use multiple Horizon supervisors instead of one?
Use separate supervisors when queues have significantly different job durations or throughput requirements. A single supervisor with `balance=auto` will still share its process pool across all queues, meaning slow jobs can starve fast ones. Separate supervisors give each queue class its own dedicated pool with independent scaling limits.
Q03 Can I access Horizon metrics outside the dashboard for custom alerting?
Yes. Bind `Laravel\Horizon\Contracts\MetricsRepository` from the container and call `throughputForQueue`, `runtimeForQueue`, or `snapshotsForQueue`. These read from Redis sorted sets and are safe to call in a scheduled command or health-check endpoint.

Continue reading

More Articles

View all