Streaming AI Responses in Laravel: Token Budgets, Structured Output, and Production Contracts

The Problem With Naive AI Integration

Most Laravel AI tutorials end at Http::post() and json_decode(). In production you face three harder problems: responses that arrive token-by-token (streaming), models that hallucinate structure (unstructured output), and long-lived Octane workers that silently carry state between requests. This article tackles all three with concrete, opinionated patterns.

1. Streaming Completions Without Blocking the Worker

OpenAI's streaming API sends text/event-stream chunks. Laravel's HTTP client wraps Guzzle, so you can consume the stream lazily:

use Illuminate\Support\Facades\Http;

function streamCompletion(string $prompt): \Generator
{
    $response = Http::withToken(config('services.openai.key'))
        ->withOptions(['stream' => true])
        ->post('https://api.openai.com/v1/chat/completions', [
            'model'      => 'gpt-4o',
            'stream'     => true,
            'max_tokens' => 512,
            'messages'   => [['role' => 'user', 'content' => $prompt]],
        ]);

    $body = $response->toPsrResponse()->getBody();

    while (! $body->eof()) {
        $line = trim($body->read(4096));
        if (str_starts_with($line, 'data: ') && $line !== 'data: [DONE]') {
            $chunk = json_decode(substr($line, 6), true);
            yield $chunk['choices'][0]['delta']['content'] ?? '';
        }
    }
}

Return this generator from a StreamedResponse so Nginx flushes each chunk immediately:

Route::get('/stream', function () {
    return response()->stream(function () {
        foreach (streamCompletion('Explain CQRS in one paragraph') as $token) {
            echo "data: {$token}\n\n";
            ob_flush();
            flush();
        }
    }, 200, ['Content-Type' => 'text/event-stream', 'X-Accel-Buffering' => 'no']);
});

X-Accel-Buffering: no is mandatory when Nginx sits in front — without it the proxy buffers the entire response.

2. Enforcing Token Budgets

max_tokens is a ceiling, not a guarantee. A budget-aware wrapper counts tokens before the call and aborts early if the prompt itself is too large:

final class TokenBudget
{
    public function __construct(
        private readonly int $maxPromptTokens = 3_000,
        private readonly int $maxCompletionTokens = 512,
    ) {}

    /** Rough estimate: 1 token ≈ 4 chars for English prose */
    public function promptFits(string $prompt): bool
    {
        return (int) ceil(mb_strlen($prompt) / 4) <= $this->maxPromptTokens;
    }

    public function completionLimit(): int
    {
        return $this->maxCompletionTokens;
    }
}

Bind it as a singleton in AppServiceProvider and inject it wherever you build prompts. This prevents runaway costs when user-supplied context is large.

3. Structured Output Contracts with JSON Schema

OpenAI's response_format with json_schema mode guarantees the model returns valid JSON matching your schema — or it refuses rather than hallucinating:

$schema = [
    'type'       => 'object',
    'properties' => [
        'summary'    => ['type' => 'string'],
        'confidence' => ['type' => 'number', 'minimum' => 0, 'maximum' => 1],
        'tags'       => ['type' => 'array', 'items' => ['type' => 'string']],
    ],
    'required'             => ['summary', 'confidence', 'tags'],
    'additionalProperties' => false,
];

$result = Http::withToken(config('services.openai.key'))
    ->post('https://api.openai.com/v1/chat/completions', [
        'model'           => 'gpt-4o-2024-08-06',
        'max_tokens'      => 256,
        'response_format' => [
            'type'        => 'json_schema',
            'json_schema' => ['name' => 'analysis', 'strict' => true, 'schema' => $schema],
        ],
        'messages' => [['role' => 'user', 'content' => "Analyse: {$text}"]],
    ])->json('choices.0.message.content');

$dto = AnalysisResult::fromArray(json_decode($result, true));

Map the validated JSON straight into a typed DTO — no defensive isset() chains needed.

4. Octane Safety: No Static State, No Singleton Leakage

Octane workers are long-lived. Any static property or singleton that accumulates per-request data will bleed across users. For AI work:

• Never store conversation history in a singleton. Use the session or a database-backed Conversation model.
• Bind AI client wrappers as scoped() (reset per request) rather than singleton().
• Use defer() for logging token usage so it runs after the response is sent.

// AppServiceProvider
$this->app->scoped(AiClient::class, fn () => new AiClient(
    apiKey: config('services.openai.key'),
));

Takeaways

• Stream via Http::withOptions(['stream' => true]) and yield chunks through a StreamedResponse.
• Set X-Accel-Buffering: no when Nginx proxies the stream.
• Estimate prompt token size before the API call to enforce hard cost budgets.
• Use OpenAI's json_schema response format to get guaranteed-valid structured output.
• Register AI clients as scoped() bindings in Octane to prevent cross-request state leakage.
• Map structured responses directly into typed DTOs — skip defensive null-checking.