Deep Dive into Kimi K3 Architecture: From Kimi Linear to 2.8T, LatentMoE, NoPE, and Attention Residuals

Detailed architecture analysis of Kimi K3: scaled-up production version of Kimi Linear (48B to 2.8T), adds LatentMoE (similar to Nemotron 3 Ultra), uses NoPE everywhere (first frontier model to do so), introduces attention residuals for improved residual paths. Also supports native multimodal.

• Moonshot
• Kimi K3
• LatentMoE
• LLM架构
• NoPE
• 注意力残差