Laravel Horizon, Queues, and Background Processing at Scale

Laravel’s queue system is one of its most powerful features, but running it at scale requires understanding the moving parts. Here’s what we’ve learned processing millions of jobs across production Laravel applications.

Choosing Your Queue Driver

Redis: The best default choice. Fast, supports Horizon, handles retries and delays natively. Use Redis for everything until you have a specific reason not to.

Amazon SQS: Better for distributed systems where workers aren’t always running. SQS handles message durability, visibility timeouts, and dead letter queues. The tradeoff is slightly higher latency and no Horizon support.

Database: Only for development or very low-volume applications. It creates table locks and doesn’t scale past a few hundred jobs per minute.

Horizon: The Missing Dashboard

Laravel Horizon provides a beautiful dashboard for monitoring your Redis queues. It shows throughput, wait times, failed jobs, and worker status in real time. But Horizon is more than a dashboard — it manages your worker processes with automatic scaling.

We configure Horizon with multiple worker groups: a high-priority queue for user-facing jobs (emails, notifications), a default queue for standard processing, and a low-priority queue for batch operations (reports, data exports).

Job Design Patterns

Keep jobs small and focused. One job should do one thing. If a job needs to process 1000 items, dispatch 1000 small jobs instead of one large job. Small jobs are easier to retry, monitor, and scale.

Idempotent jobs. Design every job to be safely retryable. Use unique job IDs to prevent duplicate processing. Check if the work has already been done before starting.

Proper error handling. Distinguish between retriable errors (API timeout, temporary network failure) and permanent errors (invalid data, missing resources). Only retry retriable errors with exponential backoff.

Supervisor Configuration

Never run queue workers in the foreground. Use Supervisor to manage worker processes with automatic restarts. Configure max requests per worker to prevent memory leaks. We typically set auto-restart after 1000 jobs or 2 hours, whichever comes first.

Monitoring and Alerting

Set up alerts for: queue size growing faster than processing rate, failed job rate exceeding threshold, job wait time exceeding SLA, and worker process crashes. Horizon’s metrics API integrates with Prometheus for time-series monitoring.

Scaling Strategies

Start with 2-4 workers on a single server. When that’s not enough, scale horizontally: add more worker servers pointing to the same Redis instance. For bursty workloads, use Horizon’s auto-scaling or deploy workers to auto-scaling groups on AWS.

The key insight: queue workers are stateless. You can add and remove workers at any time without affecting in-progress jobs. This makes horizontal scaling straightforward.