Ship AI with Laravel: Failover, Queues, and Middleware for AI Agents

Why AI Agents Break in Production

Development is forgiving. Production is not. A single AI provider outage at 2 AM, a sudden spike of concurrent users, or a mysterious bad response with no audit trail — these are the real problems that ship with your AI features if you don't plan for them.

This tutorial covers three infrastructure patterns that close those gaps: provider failover, queue-based processing, and agent middleware.

1. Provider Failover

The simplest change with the biggest impact. Instead of locking your agent to a single provider, pass an array of providers to the SDK. If OpenAI fails, the SDK automatically retries with Anthropic, then Gemini — same agent, same tools, same instructions.

$agent->withProviders([
    'openai',
    'anthropic',
    'gemini',
]);

Start by setting this up at the call site, then move the failover chain into your config file so you can adjust it without touching application code.

To stay informed when a provider drops, listen for the AgentFailedOver event:

Event::listen(AgentFailedOver::class, function ($event) {
    // send alert, log to monitoring, etc.
});

Your customers never see an error. You get an alert the moment something goes wrong.

2. Background Queue Processing

Not every AI call needs to block the HTTP request. The TicketClassifier from an earlier episode is a good candidate: swap prompt() for queue() and the classification runs through Laravel's queue system in the background.

// Before
$agent->prompt($ticket->body);

// After
$agent->queue($ticket->body);

The customer receives an instant confirmation. The AI work — including the full failover chain — happens asynchronously on a queue worker. This pattern keeps response times fast under load and makes your AI features far more resilient to bursts of traffic.

3. Agent Middleware

The SDK lets you intercept every prompt and response flowing through an agent, exactly like HTTP middleware. Three middleware layers are worth building for any production agent:

Logging Middleware

Capture each prompt alongside response metadata: token counts, provider used, and duration. This is your audit trail for debugging and compliance.

Rate Limiting Middleware

Enforce per-user limits — for example, ten prompts per minute — before any other work happens.

Cost Tracking Middleware

Use token counts to calculate spend per user, per agent, and per day. This is the foundation for billing, budgeting, and anomaly detection.

Attach all three to the agent in a deliberate order:

$agent->withMiddleware([
    RateLimitMiddleware::class,   // reject early
    LoggingMiddleware::class,     // log only accepted requests
    CostTrackingMiddleware::class,
]);

Rate limiting runs first so you never waste resources logging or tracking a request you're about to reject.

Key Takeaways

• Pass an array of providers to enable automatic failover with no changes to agent logic.
• Move the failover chain into config so it can be updated without a code deploy.
• Listen for AgentFailedOver to get real-time alerts when a provider drops.
• Use queue() instead of prompt() to move AI work off the request lifecycle.
• Build logging, rate limiting, and cost tracking as reusable middleware.
• Order middleware deliberately: rate limit before logging, log before cost tracking.

The full source code for this series is available on GitHub: https://github.com/harris21/ship-ai-with-laravel

Watch the video tutorial and read the original article at https://laravel-news.com/ship-ai-with-laravel-failover-queues-and-middleware-for-ai-agents