ZICQ
中 Log in / Sign up
ZICQ Info LLMs & Foundation Models #Kimi K3 #Large Language Model #Inference Optimization #Cloud Deployment #Cost Analysis

Reddit User Benchmarks Kimi K3 Model: 2.8 Trillion Parameters, 92 Tokens/s Inference Speed

Avatar of Mr.Xu

By Mr.Xu Community Post

Published: · 19 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:A Reddit user shared detailed experiences of hosting the Kimi K3 large language model in the cloud, which boasts 2.8 trillion parameters. Using 8 B300 accelerators, the inference speed reached 92 tokens/s, with a cost of $190 per million tokens. The user also benchmarked Unsloth's Dynamic GGUF, which, while cheaper, showed significantly lower performance. This sharing provides valuable insights for AI developers on large model deployment and cost optimization.


Key Deployment Details

  • Model and Hardware Configuration: The Kimi K3 model, with 2.8 trillion parameters, was deployed on the cloud platform Modal using 8 B300 accelerators.
  • Performance Metrics:
    • Cold boot time is approximately 27 minutes, loading 1.56 TB of data.
    • Time to First Token (TTFT) ranges from 0.92 to 1.02 seconds.
    • Inference speed is steady at 92 tokens/s, with an average of 83 tokens/s (based on 4 prompts).
  • Cost Analysis:
    • The cost is $190 per million output tokens, with a single run requiring about $36 of GPU time.
    • When kept warm, the daily cost amounts to $1,363.

Comparison Scheme

The user also benchmarked Unsloth's Dynamic GGUF, which uses 1-bit UD-IQ1_S (594 GB) and runs on 8 A100-80GB via llama.cpp.

  • Performance Metrics:
    • Inference speed is around 9 tokens/s.
    • TTFT ranges from 7 to 60 seconds.
  • Cost Analysis:
    • The cost is $620 per million tokens, which is 3.3 times more expensive than the Kimi K3 scheme.

Although the 1-bit scheme showed good arithmetic and prose coherence, its lower speed and higher cost make it less favorable for high-performance scenarios.

Technical Highlights

  • Efficient Inference: Kimi K3 achieves high-throughput inference through multi-accelerator parallel computing.
  • Cost Optimization: By appropriately configuring hardware and software, a balance between inference speed and cost is achieved.
  • Multi-Scheme Comparison: Detailed comparisons of different deployment schemes provide practical references for AI developers.

Industry Impact and Recommendations

  • Implications for AI Developers: When choosing a large model deployment scheme, it is crucial to consider inference speed, cost, and hardware configuration.
  • Impact on Cloud Service Providers: High-performance accelerators and optimized software stacks are key to attracting AI developers.
  • Future Outlook: As large model applications become more widespread, optimizing deployment schemes and cost control will become important research directions in the AI field.

Source: Reddit r/LocalLLaMA (2026-08-23)

— END —

Tags: #Kimi K3 #Large Language Model #Inference Optimization #Cloud Deployment #Cost Analysis

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…