The DeepSeek Series: A Technical Overview

Motivation & Overview

The authors set out to answer an important question: Given a fixed compute budget for pre-training, how do we choose the scale of the model and how much training data to use? Prior studies (e.g. Chinchilla vs. GPT-3) differed on the ratio between these two factors. DeepSeek-LLM addresses that by measuring scale in a different way. Earlier work measured scale in terms of how many parameters were in the model, DeepSeek-LLM instead measured scale as non-embedding FLOPs/token1 They then found they could predict computation with:

$$ C = M \times D $$

where $C$ is the compute budget, $M$ is non-embedding FLOPs/token, and $D$ is data size.

This more granular representation helps them predict how a 7B or 67B model might train on 2T tokens of bilingual data.

Training Instability

A central concern they grapple with is training instability (sudden irrecoverable divergences in the training process), which can often manifest in large-scale language models—especially those with mixture-of-experts or very long contexts.

By carefully tuning learning rates, batch sizes, and other hyperparameters 2, DeepSeek-LLM demonstrates that stable large-scale training is achievable, but it requires meticulous design of the architecture of the transformer model together with the infrastructure of the High Performance Computing (HPC) data center used to train it. This interwoven design of both architecture and infrastructure is called HPC Co-Design.

Data Quality & Model Scale

A point the authors make is about how data quality shifts the optimal ratio—i.e., higher-quality data can justify a bigger model for the same number of tokens. You can intuit this by imagining two scenarios:

• Scenario A: You have a 100-billion-token corpus full of duplicates, spammy text, or incomplete sentences. The model might not glean much new knowledge because the data is partly redundant or low-value.
• Scenario B: You have a carefully curated 100-billion-token corpus with broad coverage of code, math, multi-lingual dialogues, factual text, etc. Each token is more “information-rich,” so the model can “afford” to use more parameters without hitting diminishing returns prematurely.

In other words, when data is denser in useful information, scaling the model further pays off because each parameter can learn from richer signals.

Key Takeaways

• Hyperparameter Scaling: They propose simple power-law fits to pick batch size and learning rate as compute $C$ grows.
• Bilingual Data: They train two base sizes (7B, 67B) on 2T tokens covering English/Chinese, then do Supervised Fine Tuning (SFT) and a simpler preference-based alignment called Direct Preference Optimization (DPO).
• Results: The resulting DeepSeek-LLM67B “Outperforms LLaMA-2 70B” on math/coding tasks, illustrating how HPC co-designed approaches can keep training stable while efficiently pushing scale.

The seeds planted here - scaling laws and infrastructure for extremely large training - will reappear in subsequent works.

Expanding the Model While Reducing Memory

Where DeepSeek-LLM mostly explored high-level scale tradeoffs, DeepSeek-V2 dives into specifics of Transformer architecture overhead. Two big obstacles in large LLMs are:

1. Attention KV Cache: Storing Key/Value vectors for thousands of tokens is memory-intensive.
2. Feed-Forward Computation: Typically the largest consumption of FLOPs in a Transformer.

To tame both, they propose:

1. Multi-Head Latent Attention (MLA): compresses Key/Value vectors to reduce memory.
2. DeepSeekMoE: a sparse Mixture-of-Experts approach that activates a fraction of the feed-forward capacity per token.

Multi-Head Latent Attention (MLA)

In standard attention, each token's Q/K/V can be as large as $d_{model}$  times the number of heads. MLA folds them into smaller “latent” vectors:

$$ \quad \mathbf{c}_{t}^{KV} = W^{DKV}\mathbf{h}_t, \quad \mathbf{k}_{t}^{C} = W^{UK}\mathbf{c}_t^{KV}, \quad \mathbf{v}_{t}^{C} = W^{UV}\mathbf{c}_t^{KV}, \quad $$

Where $c_{t}^{KV}$ is the compressed latent vector for keys and values. $W^{DKV}$ is the down-projection matrix, and $W^{UK}, W^{UV}$ are the up-projection matrices for keys and values, respectively. In simpler terms:

1. Replaces the standard QKV computation by using low rank factorization to turn one matrix of dim (in, out) into two matrices of (in, rank) and (rank, out)
2. Project the compressed KV latent vector for each head to get the full K and V head corresponding to each Q head
3. Cache the compressed KV latent vector instead of each of the KV heads in full, and compute the KV heads on the fly from the latent vector.

DeepSeekMoE: Sparsely Activated FFNs

Next, they adopt a Mixture-of-Experts (MoE) in the feed-forward blocks:

• Shared Experts handle universal patterns for every token.
• Routed Experts handle specialized sub-problems, chosen dynamically via gating.
• Auxiliary Loss ensures balanced usage so no expert collapses (i.e. is never used).

They further limit cross-device routing with a “device-limited routing” scheme - instead of allowing any token to access any expert, DeepSeekMoE selects a limited number of devices ($M$) per token, and performs expert selection only within these devices. The basic process is as follows:

• Identify top $M$ devices that contain experts with the highest affinity to the token
• Perform top $K_r$ expert selection within these $M$ devices
• Assign the selected experts to process the token

Without device-limited routing, MoE models can generate excessive communication overhead which is incompatible with the hardware limitations imposed on the DeepSeek team. In addition, MoE models typically risk uneven expert utilization, where some experts are overused while others remain inactive. To prevent this, DeepSeekMoE introduces three balancing loss functions:

• Expert-level Balance Loss ($L_{ExpBal}$):
• Ensures uniform distribution of tokens across experts to prevent expert collapse
• Uses a loss function based on softmax scores of token-expert affinity
• Device-level Balance Loss ($L_{DevBal}$):
• Ensures workload is evenly distributed across devices
• Communication Balance Loss ($L_{CommBal}$):
• Balances incoming and outgoing token routing to each device

Training & Outcomes

DeepSeek-V2, with ~236B total params (21B activated), is pre-trained on 8.1T tokens. They do Supervised Fine Tuning (SFT) on 1.5M instruction samples, then reinforcement learning (RL) for alignment. The end result:

• Inference and training are both faster and cheaper (MLA + sparse experts)
• They remain stable at scale

This paper is really when iteration gains due to HPC Co-Design start to become apparent. By designing the model architecture with the training infrastructure in mind, and implementing a training regime that considers the realities of the hardware (e.g. low interconnect speeds on H800s), the team was able to lay the foundation for their most notable breakthrough.

Scaling MoE to 671B While Preserving Efficiency

Building on V2, DeepSeek-V3 further extends sparse models to 671B parameters (37B activated), training on 14.8T tokens in under 2.8M H800 GPU hours. The authors credit extensive HPC co-design:

Lastly, we emphasize again the economical training costs of DeepSeek-V3, summarized in Table 1, achieved through our optimized co-design of algorithms, frameworks, and hardware.

-- DeepSeek-V3 Tech. Report, p.5

The major novelties are:

1. Refined MLA
2. Refined DeepSeekMoE
3. Co-Designed Training & Inference Frameworks

Refined MLA

Multi-Head Latent Attention was introduced in V2 to reduce KV cache overhead. In V3, it is further refined with several new features:

• Dynamic Low-Rank Projection: Instead of a static compression dimension, MLA adjusts how strongly it compresses Key/Value vectors depending on sequence length. For shorter sequences, less compression preserves fidelity; for extremely long sequences (32K–128K tokens), deeper compression manages memory growth.
• Adaptive Query Compression: Where V2 used a fixed $d_c$ dimension, V3 employs an adaptive scaling of the query up/down at different layer depths. Early layers use higher-dimensional queries for expressiveness; deeper layers more aggressively compress to save activation memory.
• Improved RoPE Handling: V2 only partially decoupled keys, but V3 extends the concept for more stable 128K context. They track a “decoupled shared key” that reduces numerical drift in extremely long generations.
• Joint KV Storage: V2 stored compressed keys and values separately. V3 merges them into a shared compressed representation to further reduce memory traffic during multi-node inference.
• Layer-Wise Adaptive Cache: Instead of caching all past tokens for all layers, V3 prunes older KV entries at deeper layers. This helps keep memory usage in check when dealing with 128K context windows.

Together, these MLA refinements ensure that while DeepSeek-V3 can attend across very long sequences, the memory overhead remains manageable.

Refined DeepSeekMoE: Auxiliary-Loss-Free, Higher Capacity

On the MoE side, DeepSeek-V3 drops the auxiliary-loss approach from V2. Instead of an explicit penalty term, each expert acquires a dynamic bias $b_i$. If an expert is overloaded at a step, $b_i$ decreases; if underloaded, $b_i$ increases. The gating decision then adds $b_i$ to the token's affinity:

$$ s'_{i,t} = s_{i,t} + b_i $$

Key Improvements:

• No Token Dropping: V2 occasionally dropped tokens if certain experts got overloaded, but the new bias-based method keeps everything.
• More Activated Experts: They raise the number of routed experts from 6 to 8 per token, improving representational power.
• Higher Stability: By removing auxiliary losses, they avoid potential interference with the main training objective, focusing purely on the intrinsico gating signals plus bias adjustments.

Hence, the final feed-forward module is a combination of a small set of shared experts plus up to 8 specialized experts chosen adaptively.

Co-Designed Frameworks: FP8, DualPipe, and PTX Optimizations

Scaling an MoE model to 671B demanded HPC-level solutions for training and inference. The authors emphasize:

Through the co-design of algorithms, frameworks, and hardware, we overcome the communication bottleneck in cross-node MoE training, achieving near-full computation- communication overlap.

-- DeepSeek-V3 Tech. Report, p.5

FP8 Mixed Precision

They adopt an FP8 data format for General Matrix Multiplications (GEMMs), halving memory. The risk is reduced numeric range so they offset it with:

• Block-wise scaling (e.g., 1x128 or 128x128 tiles).
• Periodic “promotion” to FP32 after short accumulation intervals to avoid overflow/underflow.

DualPipe Parallelism

They propose DualPipe to overlap forward/backward computation with the MoE all-to-all dispatch. It rearranges pipeline stages to ensure that network communication (particularly across InfiniBand) is hidden behind local matrix multiplications.

PTX-Level & Warp Specialization

To fully exploit InifiniBand(IB) and NVLink:

• They tune warp-level instructions in PTX (a level lower than CUDA), auto-tuning the chunk size for all-to-all dispatch.
• Dynamically partition Streaming Microcontrollers into communication vs. compute tasks so that token dispatch never stalls local GEMM.

As a result, training costs were cut to 2.8M H800 GPU hours per run - low for a 14.8T token corpus.

Outcomes

The resulting DeepSeek-V3 excels at code, math, and some multilingual tasks, outperforming other open-source LLMs of similar scale. Deep HPC co-design (FP8, DualPipe, PTX-level optimization) plus refined MLA/MoE implementation achieve extreme scale with stable training.

Emergent Reasoning Behaviors Through RL-Only

All prior DeepSeek releases used SFT (plus occasional RL). By contrast, DeepSeek-R1-Zero tries an extreme: no supervised warmup, just RL from the base model. They adopt Group Relative Policy Optimization (GRPO), which:

1. Samples a group of old-policy outputs ${o_1, ..., o_G}$
2. Scores each with a reward (in this case, rule-based)
3. Normalizes the advantage $A_i$ by group mean/stdev
4. Optimizes a clipped PPO-like objective

The reward function for the R1 models is rule-based - a simple weighted sum between 2 components

• Accuracy Reward - if the task has an objective correct answer (e.g. a math problem, coding task, etc.), correctness is verified using mathematical equation solvers for step-by-step proof checking, and code execution & test cases for code correctness verification
• Format Reward - the model is rewarded for following a structured reasoning process using explicit reasoning markers <think></think> and <answer></answer>

The relative advantage $A_i$ for a given output is calculated as:

$$ A_i = \frac{r_i - mean(\{r_1, r_2, ..., r_G\})}{std(\{r_1, r_2, ..., r_G\})} $$

where $r_i$ is the reward calculated for the given output. The model's policy is updated to favor responses with higher rewards while constraining changes using a clipping function which ensures that the new policy remains close to the old.

In so many words: the authors created a testing/verification harness around the model which they exercised using reinforcement learning, and gently guided the model using simple Accuracy and Format rewards. In doing so, emergent reasoning behaviors were observed:

• Self-verification where the model double-checks its own answers
• Extended chain-of-thought where the model learns to explain its reasoning more thoroughly
• Exploratory reasoning - the model tries different approaches before converging on an answer
• Reflection - the model starts questioning its own solutions and adjusting reasoning paths dynamically

R1-Zero is probably the most interesting outcome of the R1 paper for researchers because it learned complex chain-of-thought patterns from raw reward signals alone. However, the model exhibited notable issues:

• Readability Problems: Because it never saw any human-curated language style, its outputs were sometimes jumbled or mix multiple languages.
• Instability in Non-Reasoning Tasks: Lacking SFT data for general conversation, R1-Zero would produce valid solutions for math or code but be awkward on simpler Q&A or safety prompts.
• Limited Domain: Rule-based rewards worked well for verifiable tasks (math/coding), but handling creative/writing tasks demanded broader coverage.

Hence, the authors concluded that while “pure RL” yields strong reasoning in verifiable tasks, the model’s overall user-friendliness was lacking. This led them to DeepSeek-R1: an alignment pipeline combining small cold-start data, RL, rejection sampling, and more RL, to “fill in the gaps” from R1-Zero’s deficits.

Refined Reasoning Through SFT + RL

DeepSeek-R1 addresses R1-Zero's limitations by injecting a small amount of supervised data before RL and weaving in additional alignment steps.