Prompt drift across environments
When prompts live only in application code without versioning, staging and production diverge quietly. Pin prompts like dependencies: semantic versions, changelogs, and automated gates that run eval harnesses on pull requests.
If you cache completions, version the cache keys with prompt hash and model identifier—otherwise you will serve stale answers after a hotfix.
Streaming UX without a cancellation story
Streaming improves perceived latency, but without cooperative cancellation you burn tokens and confuse users who navigate away. Thread abort signals through your gateway and provider clients, and propagate UI state so partial responses never commit side effects.
Policy belongs outside the model
Put irreversible actions behind deterministic checks: schema validation, allow-listed tools, and human approvals for high-risk classes. LLMs propose; services decide.
Log structured traces with redaction defaults—not raw transcripts—so security teams can approve retention policies without blocking launches.
Unit economics at scale
Batch where you can, cache where it is safe, and cap concurrency per tenant. Pair dashboards for tokens and latency with budgets that trip circuit breakers before finance does.
Finally, rehearse degraded modes: provider outage, rate limits, and model deprecation should each have a tested fallback path—not a postmortem first.