Chatterbox TTS pipeline · 2026
Fitting a 48 GB voice model into 5 GB
Turning thousands of real-estate listings into AI voiceovers. The speech model’s memory grew with input length, so long listings crashed or hallucinated, and it needed 48 GB-class GPUs to survive.
- 16–80 → 5 GBVRAM, any input length
- −87%infra cost
- 2,655+listings, zero OOM
- Constraint
- S3Gen held the whole sequence in memory: 16 GB at baseline, 48–80 GB on long inputs. Position overflows at token boundaries also produced garbled “hallucinated” audio.
- Decision
- Rewrote S3Gen to run in bounded-memory chunks and patched the position and token-boundary handling, so memory stays flat regardless of length.
- Tradeoff
- Chunking adds stitching logic at the boundaries, but it gives memory a hard ceiling, and that ceiling is what moved inference from 48 GB accelerators to an RTX 3070.