Production AI Agents in Laravel: Streaming, Token Budgets, and Structured Output Contracts

Production AI Agents in Laravel: Streaming, Token Budgets, and Structured Output Contracts

Most tutorials stop at "call the API, dump the response." Production agents are a different beast. You need to stream tokens to the browser without blocking a PHP worker for 30 seconds, enforce hard token budgets so a runaway prompt doesn't drain your quota, and guarantee that the model returns data your application can actually parse. This article covers all three.

Streaming Responses Over SSE

Laravel's StreamedResponse pairs naturally with OpenAI's streaming API. The key is flushing output incrementally without buffering the entire completion.

use Illuminate\Support\Facades\Route;
use Symfony\Component\HttpFoundation\StreamedResponse;
use OpenAI\Laravel\Facades\OpenAI;

Route::get('/chat/stream', function () {
    return new StreamedResponse(function () {
        $stream = OpenAI::chat()->createStreamed([
            'model' => 'gpt-4o',
            'messages' => [['role' => 'user', 'content' => request('prompt')]],
        ]);

        foreach ($stream as $response) {
            $delta = $response->choices[0]->delta->content ?? '';
            if ($delta !== '') {
                echo 'data: ' . json_encode(['token' => $delta]) . "\n\n";
                ob_flush();
                flush();
            }
        }

        echo "data: [DONE]\n\n";
    }, 200, [
        'Content-Type' => 'text/event-stream',
        'Cache-Control' => 'no-cache',
        'X-Accel-Buffering' => 'no', // critical for nginx
    ]);
});

X-Accel-Buffering: no is the header most people forget. Without it, nginx will buffer the entire response before forwarding it to the client, defeating the purpose of streaming entirely.

Enforcing Token Budgets

Token overruns are a billing and latency problem. Enforce budgets at two layers: before the request (prompt token estimation) and inside the request (max_tokens).

final class TokenBudget
{
    public function __construct(
        private readonly int $maxPromptTokens = 3_000,
        private readonly int $maxCompletionTokens = 1_000,
    ) {}

    public function assertPromptFits(string $prompt): void
    {
        // ~4 chars per token is a safe heuristic for English text
        $estimated = (int) ceil(mb_strlen($prompt) / 4);

        if ($estimated > $this->maxPromptTokens) {
            throw new \OverflowException(
                "Prompt exceeds budget: ~{$estimated} tokens (max {$this->maxPromptTokens})"
            );
        }
    }

    public function completionLimit(): int
    {
        return $this->maxCompletionTokens;
    }
}

Bind this as a singleton scoped to the current tenant or user plan:

$this->app->scoped(TokenBudget::class, function () {
    $plan = auth()->user()?->plan ?? 'free';
    return match ($plan) {
        'pro'  => new TokenBudget(8_000, 2_000),
        default => new TokenBudget(3_000, 500),
    };
});

Using scoped rather than singleton ensures the budget resets per request, which matters under Octane.

Structured Output Contracts

Asking a model to "return JSON" is not a contract. OpenAI's response_format with json_schema mode (available on gpt-4o and later) lets you enforce a schema server-side. Pair it with a DTO and a Pest assertion.

$response = OpenAI::chat()->create([
    'model' => 'gpt-4o',
    'messages' => [
        ['role' => 'system', 'content' => 'Extract the invoice fields.'],
        ['role' => 'user', 'content' => $rawText],
    ],
    'response_format' => [
        'type' => 'json_schema',
        'json_schema' => [
            'name' => 'invoice',
            'strict' => true,
            'schema' => [
                'type' => 'object',
                'properties' => [
                    'vendor'  => ['type' => 'string'],
                    'amount'  => ['type' => 'number'],
                    'due_date'=> ['type' => 'string', 'format' => 'date'],
                ],
                'required' => ['vendor', 'amount', 'due_date'],
                'additionalProperties' => false,
            ],
        ],
    ],
]);

$data = json_decode($response->choices[0]->message->content, true, flags: JSON_THROW_ON_ERROR);
$invoice = InvoiceData::from($data); // Spatie Data DTO

With strict: true, the model will refuse to produce output that violates the schema rather than hallucinating extra fields. Validate the DTO immediately after hydration — never trust the model's output downstream without a type check.

Putting It Together in a Job

For non-interactive agents, run the completion inside a queued job with a timeout that matches your token budget:

class ExtractInvoiceJob implements ShouldQueue
{
    public int $timeout = 60;
    public int $tries = 2;

    public function handle(TokenBudget $budget, InvoiceExtractor $extractor): void
    {
        $budget->assertPromptFits($this->rawText);
        $invoice = $extractor->extract($this->rawText, $budget->completionLimit());
        InvoiceExtracted::dispatch($invoice);
    }
}

Set $timeout conservatively. A 500-token completion at peak load can still take 20+ seconds.

Takeaways

• Add X-Accel-Buffering: no to every SSE response or nginx will swallow your stream.
• Use scoped() for per-request token budgets under Octane, not singleton().
• OpenAI's json_schema response format with strict: true is a real contract, not a prompt suggestion.
• Validate and hydrate into a typed DTO immediately — never pass raw model output into business logic.
• Set explicit job $timeout values that reflect your worst-case token budget, not a generic default.