Streaming AI Responses in Laravel: Token Budgets, Structured Output, and Agent Contracts

The Problem With Naive LLM Integration

Most Laravel + LLM tutorials show a single chat() call and a dd($response->content). That works in a demo. In production you face three hard problems: responses block the HTTP worker until the model finishes, uncapped prompts silently drain your budget, and free-form JSON from the model breaks your downstream code without warning.

This article tackles all three with concrete patterns.

Streaming Responses to the Browser

OpenAI's streaming API sends server-sent events (SSE). Laravel's StreamedResponse lets you forward them without buffering the entire completion.

use Illuminate\Http\Response;
use OpenAI\Laravel\Facades\OpenAI;

Route::get('/chat', function () {
    return response()->stream(function () {
        $stream = OpenAI::chat()->createStreamed([
            'model' => 'gpt-4o',
            'messages' => [['role' => 'user', 'content' => request('prompt')]],
        ]);

        foreach ($stream as $response) {
            $delta = $response->choices[0]->delta->content ?? '';
            if ($delta !== '') {
                echo "data: " . json_encode(['token' => $delta]) . "\n\n";
                ob_flush();
                flush();
            }
        }

        echo "data: [DONE]\n\n";
    }, 200, [
        'Content-Type' => 'text/event-stream',
        'X-Accel-Buffering' => 'no', // critical for nginx
        'Cache-Control' => 'no-cache',
    ]);
});

The X-Accel-Buffering: no header is the most commonly forgotten detail when running behind nginx — without it, nginx buffers the entire stream before forwarding.

Enforcing Token Budgets

Token costs compound fast when users craft adversarial prompts. Enforce budgets at two layers.

1. Hard limit via max_tokens

$payload = [
    'model' => 'gpt-4o',
    'max_tokens' => config('ai.max_completion_tokens', 512),
    'messages' => $messages,
];

2. Prompt token pre-check

Count tokens before sending using tiktoken-php or a simple heuristic, and reject early:

use Yethee\Tiktoken\EncoderProvider;

final class TokenBudgetGuard
{
    private const MODEL_LIMIT = 8_192;
    private const RESERVED_FOR_COMPLETION = 512;

    public function __construct(private EncoderProvider $provider) {}

    public function assertFits(array $messages, string $model = 'gpt-4o'): void
    {
        $encoder = $this->provider->getForModel($model);
        $tokens = array_sum(
            array_map(fn ($m) => count($encoder->encode($m['content'])), $messages)
        );

        $budget = self::MODEL_LIMIT - self::RESERVED_FOR_COMPLETION;

        if ($tokens > $budget) {
            throw new TokenBudgetExceededException($tokens, $budget);
        }
    }
}

Bind this as a singleton and inject it into your agent service. Throw early — never let an oversized prompt reach the API.

Structured Output Contracts

OpenAI's response_format with json_schema mode guarantees the model returns JSON matching your schema. Pair that with a typed PHP DTO and you get end-to-end type safety.

readonly class ProductSuggestion
{
    public function __construct(
        public string $name,
        public string $reason,
        public int $confidencePercent,
    ) {}

    public static function fromArray(array $data): self
    {
        return new self(
            name: $data['name'],
            reason: $data['reason'],
            confidencePercent: $data['confidence_percent'],
        );
    }
}

$response = OpenAI::chat()->create([
    'model' => 'gpt-4o-2024-08-06', // structured output requires this or later
    'messages' => $messages,
    'response_format' => [
        'type' => 'json_schema',
        'json_schema' => [
            'name' => 'product_suggestion',
            'strict' => true,
            'schema' => [
                'type' => 'object',
                'properties' => [
                    'name' => ['type' => 'string'],
                    'reason' => ['type' => 'string'],
                    'confidence_percent' => ['type' => 'integer'],
                ],
                'required' => ['name', 'reason', 'confidence_percent'],
                'additionalProperties' => false,
            ],
        ],
    ],
]);

$suggestion = ProductSuggestion::fromArray(
    json_decode($response->choices[0]->message->content, true, flags: JSON_THROW_ON_ERROR)
);

With strict: true the model will refuse to emit keys not in your schema. Validation failures become model refusals, not silent bad data.

Wiring It Together in a Job

For non-interactive workloads, push the agent call to a queued job and store the result:

class RunProductSuggestionAgent implements ShouldQueue
{
    use Dispatchable, Queueable;

    public int $tries = 2;
    public int $timeout = 60;

    public function __construct(private int $productId) {}

    public function handle(TokenBudgetGuard $guard, ProductRepository $repo): void
    {
        $product = $repo->findOrFail($this->productId);
        $messages = MessageBuilder::forProduct($product);

        $guard->assertFits($messages);

        // ... call OpenAI, hydrate DTO, persist
    }
}

Set $timeout explicitly — the default 60 s is often too short for large completions and too long to leave zombie workers hanging.

Key Takeaways

• Stream via response()->stream() and set X-Accel-Buffering: no for nginx.
• Pre-check prompt token counts before hitting the API; throw early.
• Use max_tokens as a hard ceiling on every request.
• Lock structured output with json_schema + strict: true and hydrate into readonly DTOs.
• Push long-running completions to queued jobs with explicit $timeout and $tries.