Preface:

As large model technology evolves rapidly, the open source ecosystem emerges as a pivotal force driving industry innovation.

Recently, ByteDance's Seed team unexpectedly announced the open sourcing of the Seed-OSS series of large language models, entering the market with a medium-sized model boasting 36 billion parameters. Leveraging cutting-edge technologies such as native 512K ultra-long context and programmable [thinking budget], it has surpassed the best scores of open source models in seven public benchmark tests.

Author | Fang Wensan

Image Source | Network

Dual Breakthroughs in Ultra-Long Context and Controllable Reasoning

The Seed-OSS series stands out with two groundbreaking features that redefine the capabilities of open source large models: native 512K ultra-long context and programmable [thinking budget] mechanisms.

The former addresses the breadth of information processing, while the latter enables precise control over the reasoning process.

The native 512K context window is Seed-OSS's [ace in the hole].

This capability is intrinsically built during pre-training, rather than through post-processing techniques. It supports a sequence length of 512K tokens, equivalent to processing 900,000 Chinese characters at once, roughly the length of the entire "Three-Body" trilogy.

This capacity is four times that of current mainstream open source models (like DeepSeek V3.1), making it adept at handling complex scenarios such as comprehensive financial report analysis, lengthy legal contract reviews, and large code base understanding.

In the RULER-128K benchmark for long document understanding, Seed-OSS-36B-Instruct scored 94.6, significantly outperforming Qwen3-32B's 77.5 points, with a leading margin of 17.1 percentage points.

This data underscores the practical impact of its ultra-long context.

When tackling real-world tasks involving documents exceeding 128K tokens, the model maintains information coherence, avoiding context truncation-induced information loss, crucial for scenarios requiring in-depth long text logical relationship analysis.

The [thinking budget] mechanism showcases Seed-OSS's refined control over the reasoning process.

Users can limit the model's intermediate reasoning steps via the [thinking_budget] parameter, measured in tokens. The recommended settings are multiples of 512 (e.g., 0, 512, 1K, 2K).

Its underlying mechanism employs a dynamic programming algorithm, where the model continuously evaluates the remaining budget during reasoning and prioritizes resource allocation to key logical nodes.

This mechanism allows the model to dynamically adjust its reasoning strategy based on task difficulty: for simple tasks like IFEval, increasing the budget has minimal impact on performance, and setting a budget of 0 (instant response mode) ensures quick responses and cost reduction;

However, for challenging mathematical reasoning tasks like AIME24 or code generation tasks like LiveCodeBench, increasing the budget from 512 to 4K boosts accuracy by 6.3% and 4.7%, respectively.

For instance, in code generation, a higher budget prompts the model to automatically add function dependency verification steps, significantly enhancing code reliability.

Top Scores in Seven Open Source Model Performance Tests

Seed-OSS-36B-Instruct achieved the highest scores among open source models in seven public benchmark tests, covering general knowledge, mathematical reasoning, code generation, and long document understanding, demonstrating its robust strength with medium-sized parameters through hard data.

In the MMLU-Pro benchmark for assessing general knowledge and multi-domain capabilities, Seed-OSS-36B-Instruct scored 82.7, 0.8 points higher than the second-best open source model, Qwen3-30B-A3B.

This score indicates that even without relying on ultra-large parameters, through optimized training data and network structure, the model excels in cross-domain knowledge mastery.

Complex mathematical reasoning serves as the [litmus test] for large models. Seed-OSS led Qwen3-30B-A3B by 4.0 points with a score of 91.7 on the AIME24 benchmark, showcasing its robust capabilities in tackling advanced mathematical problems.

This is attributed to both the data augmentation strategy and the ample reasoning space provided by the [thinking budget] mechanism.

The model can perform formula derivation, step decomposition, and self-verification within the budget, significantly reducing computational errors.

In code generation, Seed-OSS-36B-Instruct scored 67.4 on LiveCodeBench v6, 3.6 points higher than OAI-OSS-20B;

It achieved a HumanEval pass rate of 76.8% and an MBPP of 80.6%, both setting new records for open source models.

This is closely tied to its temporal data augmentation strategy, where by learning the code evolution process in Git commit records, the model gains a deeper understanding of code logic and development specifications.

In the SWE-Bench Verified benchmark for evaluating software engineering tasks, the model scored 56, 1.2 points higher than OpenHands, proving its practicality in solving real software engineering problems.

In the AgentBench benchmark for agent tasks, Seed-OSS also ranked first among open source models, validating its applicability in complex scenarios such as multi-step interaction and tool usage.

In terms of multilingual capabilities, Seed-OSS scored an average of 4.3 points higher than Llama 3-65B in the XTREME evaluation covering 90 languages, thanks to its 155K subword multilingual tokenizer and cross-language contrastive learning strategy.

In logical reasoning, it scored 87.7 on the BBH benchmark, exceeding Qwen3-30B-A3B's 81.2 points, demonstrating strong logical chain construction capabilities.

Remarkably, these achievements were attained using only 12T tokens of training data, compared to the over 15T tokens used by many models of similar scale.

This underscores that the Seed-OSS team achieved a [less but better] performance breakthrough through more efficient training strategies and data processing methods, providing fresh insights into cost optimization in large model training.

Innovation from Network Design to Training Strategies

The Seed-OSS series' outstanding performance is not a coincidence but the culmination of systematic optimization of the large model technology architecture.

From network structure design to training strategy selection, every detail embodies the seamless integration of engineering and academic innovation.

In terms of network structure, Seed-OSS-36B employs a dense Transformer architecture with 36 billion parameters, comprising 64 layers and a hidden dimension of 5120.

Its core innovation lies in the attention mechanism design, utilizing Grouped Query Attention (GQA) with 80 query heads and 8 key-value heads.

Compared to traditional multi-head attention, GQA significantly reduces memory usage and computation during reasoning while maintaining model performance by allowing multiple query heads to share key-value heads.

This optimization enables a single 80GB VRAM GPU to run a half-precision model, substantially lowering deployment costs.

Position encoding technology is crucial for supporting the 512K ultra-long context. Seed-OSS adopts Rotational Position Encoding (RoPE) but increases the base frequency parameter from the conventional 1×10 to 1×10⁶.

This seemingly simple adjustment enables the model to more accurately capture the relative positional relationships in long sequences, fundamentally resolving the context continuity issue in long text processing.

When processing contract texts as long as 1,600 pages, the context continuity error rate of Seed-OSS-36B-Instruct is 42% lower than models of the same scale, invaluable in professional scenarios such as legal document review and financial report analysis.

In terms of training strategies, high-quality corpora of 12T tokens are used, undergoing rigorous deduplication, toxicity filtering, and copyright cleaning to ensure data quality.

The training framework combines PyTorch 2.3's and Megatron-LM's mixed parallelism technology, utilizing 1024 A100 GPUs for continuous training over 60 days. For precision control, bf16 forward computation and fp32 master weights are used, with gradient clipping set to 1.0 and the learning rate reduced to 1×10⁻⁵ through cosine annealing.

For multilingual alignment, cross-language contrastive learning of Chinese and English corpora boosted the score for the Chinese-English mixed test in MMLU-Pro by 3.2 points;

For code generation tasks, temporal training data constructed using Git commit records improved the HumanEval score by 2.1 points;

In mathematical reasoning training, 15% of deliberately incorrect derivation processes were mixed in to force the model to learn to identify logical flaws, ultimately enhancing AIME24 accuracy by 6.3%.

Regarding reasoning optimization, Seed-OSS supports 4-bit and 8-bit quantization (including GPTQ and AWQ methods) and offers dual backend reasoning scripts for vLLM and Transformers.

Through vLLM backend optimization, a single 80GB VRAM GPU can achieve a generation speed of 32 tokens per second, fully meeting the demands of real-time scenarios like live caption generation.

The innovative [thinking budget] mechanism allows users to control the depth of reasoning through token-level switches, striking a flexible balance between performance and cost.

The Seed team has previously open-sourced projects such as the Seed-Coder code generation model, the BAGEL multimodal model, and the Seed Diffusion language model. Together with the Seed-OSS series, they form an extensive open source matrix spanning multiple domains.

From a technical perspective, Seed-OSS's success validates the value of two major directions.

① Fine-grained optimization of medium-sized models. Through network structure innovation, training strategy enhancement, and reasoning mechanism design, a 36 billion parameter model can rival larger models in specific scenarios.

② [Controllability] has emerged as a core indicator for large model practical application. The [thinking budget] mechanism returns control over performance and cost to users. This [human-centered] design approach may become a standard feature of future large models.

Conclusion:

From an option to a standard, open source is reshaping the competitive landscape of large models. The advent of the Seed-OSS series represents not only a technological breakthrough but also an exploration of industry innovation models.

As the technological dividend benefits more entities through open source, and innovation costs are significantly reduced due to sharing mechanisms, the golden age of large models truly begins.

References:

Fit Theory: "A 36B model can read 900,000-word contexts? Deciphering ByteDance's first open source large language model

Quantum Bit: "ByteDance suddenly opens source Seed-OSS, with a 512K context crushing the mainstream by four times the length! Reasoning capabilities set a new record

Intelligent Things: "ByteDance opens source reasoning model for the first time, winning seven first places in a row"