Laplace 3B cuts per‑sequence memory and raises concurrent throughput on Trainium2

Using Laplace 3B on Trainium2 lowers per‑session memory needs and can increase concurrency, which may reduce serving cost and operational risk for teams that run long‑context or agentic applications.

What actually changed

i14 reworked attention away from the usual softmax KV cache and delivered Laplace 3B, a linear‑attention model whose recurrent state remains 29.5 MB per sequence regardless of context length. That removes the growth in KV cache size that normally happens as conversation or document context expands. i14 worked with AWS to run the model on Trainium2 hardware via TorchNeuron Native while keeping its Gated Linear Attention (GLA) design and the Muon optimizer.

On a single trn2.48xlarge instance the team reported training at about 120,000 tokens per second. For inference the setup produced 11,544 output tokens per second while serving 1,280 concurrent sequences. Those numbers show the model operating with high concurrent sequence throughput on Trainium2.

Who it affects

This change matters most to teams running interactive or agentic applications where context length grows over the lifetime of a session. The source points out that KV caches that grow with context can reduce concurrency, increase serving cost, and force agentic apps to drop history. Any service where per‑session memory is a limiting factor — conversational SaaS, assistants that preserve long histories, or multi‑agent systems — will see the most direct operational effect.

Teams that currently run transformers with standard softmax attention and large KV caches will be the ones deciding whether to experiment with Laplace 3B. Pure research groups or teams that do not face context‑length or concurrency limits will feel less pressure to change.

What it costs or what it replaces

What it replaces: Laplace 3B replaces the standard softmax attention/KV cache inference pattern with a linear attention approach that keeps a fixed, small recurrent state per sequence. That removes the principle driver of KV cache growth during long sessions.

What it runs on: the reported experiments used AWS Trainium2 via TorchNeuron Native on a trn2.48xlarge instance.

What it costs: the announcement does not state pricing for running Laplace 3B or for the trn2.48xlarge instances. You must consult AWS Trainium2 pricing or your cloud reseller to convert the reported throughput and concurrency numbers into dollars per token or dollars per concurrent session for your workload.

Operational trade‑offs: the source preserves the model’s GLA design and Muon optimizer, which suggests switching may be largely a model‑and‑runtime change rather than a complete reimplementation of training tooling. The concrete cost impact depends on how your current stack maps to Trainium2 instance pricing and on whether the higher concurrency reported translates to your request patterns.

What we don't know

• Exact Trainium2 trn2.48xlarge pricing and the resulting cost per token or per concurrent session for these runs.
• How model quality (response accuracy, alignment, hallucination rates) compares to equivalent softmax‑attention models at similar compute.
• Latency characteristics at smaller batch sizes or different request patterns than the reported runs.
• Compatibility and integration effort with existing toolchains (Hugging Face transformers, existing inference servers) beyond TorchNeuron Native.
• Licensing or distribution terms for Laplace 3B and any pretrained weights.

What to do next

1. If you operate conversational or agentic services with long histories, run a pilot: deploy Laplace 3B on a single trn2.48xlarge instance via TorchNeuron Native and compare per‑session memory, max concurrent sessions and tokens/s against your current model.
2. Convert the pilot throughput into cost estimates by checking AWS Trainium2 pricing for trn2.48xlarge and calculating dollars per token and dollars per active session for your traffic patterns — the announcement does not provide pricing.
3. While testing, measure output quality against your production model and verify toolchain compatibility; delay a full migration until latency, accuracy and integration outcomes meet your acceptance criteria.