Next-Concept Modeling (NCM)
NCM is a two-stage, coarse-to-fine framework for parallel text generation. A multiscale residual VQ-VAE tokenizer encodes text as a hierarchy of discrete codes, and a transformer generates that hierarchy one level at a time.
Instead of producing one token per step like autoregressive models, or refining the entire sequence repeatedly like diffusion models, NCM predicts all codes within a level in parallel. Generation therefore takes only a handful of forward passes, moving from a coarse sketch of the output to the final text.
Across language modeling, translation, summarization, and reasoning benchmarks, NCM matches or beats strong diffusion baselines while using significantly less compute.
// Current focus
In progressI am currently working on solving two bottlenecks in NCM: tokenizer reconstruction, where small code-level errors can change output tokens, and the fixed-size output schedule, which spends the same number of levels on every request. My work explores higher-fidelity reconstruction and input-dependent level schedules.