Open Source: Your Stack, Your Responsibility

Netflix brought the stack inside

Netflix describes an in-house LLM serving platform built within its existing production environment. It did not simply place a model on GPU infrastructure and call the job done.

The model sits inside a much larger serving system: routing, A/B test logic, feature fetching, post-processing, logging, model packaging, Triton, vLLM, API frontends, deployment strategy, health checks, autoscaling, versioning, metrics, rollout and rollback.

The model still matters enormously, but it is only one component of the production system.

What users actually encounter is the whole assembly: the package, route, API, runtime, constraints, scheduler, deployment state and the evidence the platform retains around what happened.

That complete assembly is what we think of as the behaving system.

The model did not change. The system did.

The value of Netflix's account is in the detail.

One example concerns compatibility between Triton and vLLM. Netflix describes a case where the Triton vLLM backend could fail to load because it expected an API surface that had changed in vLLM. Triton 25.09 was attempting to import a module removed in vLLM 0.11.2, so compatible versions had to be pinned together.

Nothing about the underlying model needed to change for the production system to behave differently.

The response_format example is even more revealing.

Netflix reused Triton's OpenAI-compatible frontend and found that response_format could be accepted by the API schema but silently dropped before reaching vLLM.

A caller could therefore ask for JSON, have the request accepted successfully, and still receive malformed output because the guided-decoding instruction never reached the inference engine.

Same model. Same endpoint. Different wrapper behavior. Different result.

The surrounding deployment also has to remain coordinated with the interfaces consumers expect. A deployment strategy that works when input and output schemas remain stable may no longer be safe when those schemas change while older consumers continue sending requests in the previous format.

Even observability has to survive the assembly. Netflix found that useful metrics existed inside both vLLM and Triton, yet Triton's built-in bridge surfaced only 9 of 40+ vLLM metrics. The platform therefore had to merge signals from both systems to create a more complete operational view.

None of these are arguments against open source.

They are the normal facts of running complex software in production.

They are also why “we run it ourselves” is not the same as “we know exactly what is operating.”

Production pressure changes the system

Some of the most interesting problems in Netflix's post appeared only under realistic production pressure.

Constrained decoding is a useful example. Netflix pushes some output constraints inside the decode loop so that the model generates compliant output directly, rather than producing invalid output and repairing it afterward.

That mechanism depends on token history. The system has to know what has already been generated in order to determine which tokens remain valid next.

Under realistic runtime conditions, however, vLLM can dynamically rebatch requests or preempt partially completed work under memory pressure. When a request later resumes, its token history may no longer match the state the constraint mechanism expected.

Netflix therefore had to detect the change and rebuild the constraint state.

This matters because the behavior users receive can be shaped by much more than the weights of the model itself. Batching, preemption, cache state, schema translation, runtime versions, constraints and rollout strategy can all become part of the observable system.

Not every stack change will alter behavior, and not every behavioral movement is necessarily harmful.

The point is more fundamental: architectural control alone cannot tell an organization whether behavior remained equivalent.

That has to be measured.

Most conventional assurance artifacts see only part of this picture. Benchmarks measure selected behaviors under controlled conditions. Model cards describe a defined release. Deployment checklists verify known configuration points. Observability platforms tell us whether infrastructure is healthy.

None of those necessarily tells us whether the complete behaving system is still producing the behavioral state the organization originally accepted.

Production is where those interactions become real.

The question is whether the organization can see them clearly enough.

Hosted or self-hosted, the continuity question remains

The Netflix example matters because it balances the hosted-model story.

With hosted systems, responsibility for much of the serving architecture sits outside the enterprise. Providers can change models, routing, inference settings or surrounding infrastructure while customers continue consuming the same named endpoint.

With self-hosted AI, more of that responsibility moves inside the organization. The enterprise itself can change vLLM, Triton, tokenizers, API wrappers, system instructions, constrained decoding, routing, model packages, metrics or deployment strategy.

The underlying governance question is the same in both cases:

Is the AI system behaving today still the one we tested and approved?

The behaving system needs evidence

An organization does not really govern a model name. It governs the system its users encounter.

That system may contain a model at its center, but it also contains the route, runtime, wrappers, tools, prompts, constraints, permissions, memory, monitoring and deployment state around it.

Open source gives enterprises greater control over many of those layers. It also makes them more accountable for understanding what those layers become after an update, rollout, compatibility fix or production-pressure intervention.

Without evidence, ownership can become reassuring but incomplete.

The organization may control the infrastructure while remaining unable to demonstrate whether the behavior produced by that infrastructure remained consistent with the state it originally approved.

Owning the infrastructure does not prove the behavior.
It makes you the person who has to prove it.

From control to evidence

Moving from control to evidence does not mean replacing vLLM, Triton, Kubernetes, model registries, observability platforms, deployment systems, cybersecurity controls or engineering judgment.

Those technologies answer important questions about whether infrastructure is available, correctly configured, secure and operating as designed.

Toriel asks a different question across the resulting system:

Is the behaving system still consistent with the behavior the customer designated as its reference?

For hosted AI, the provider can change what sits behind the endpoint. For self-hosted AI, the enterprise can change the stack beneath itself.

Either way, users encounter behavior rather than an architecture diagram.

Open source gives organizations control over far more of the machine. It lets teams build, patch, fork, tune, optimize and replace components of their AI systems.

Evidence is what makes that control governable.