ZICQ
中 Log in / Sign up
Newsroom Open Source AI #Individual Developer #Open-Source Model #Large Language Model #LoRA #Gemmeh

Individual Developer Open-Sources 1.1B Parameter LLM 'Gemmeh' Trained for $200

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 4 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:A developer has shared the complete process of training a 1.1 billion parameter language model (LLM) named 'Gemmeh' from scratch. Based on the Gemma3 architecture, the model was pretrained on 20 billion tokens and fine-tuned using LoRA, with a total cost of around $200. The developer has open-sourced the model weights, code, and a demo website, providing a practical case for individuals to build and understand large language models at a low cost.


Open-Sourcing the 1.1B Parameter LLM 'Gemmeh'

Project Background

A Reddit user @SevereTilt shared their personal project on the LocalLLaMA community – training a 1.1 billion parameter language model (LLM) named 'Gemmeh' from scratch. The primary goal of the project was to learn the process of building large language models and to enhance the developer's skills in machine learning.

Technical Details

  • Architecture: Gemmeh is based on the Gemma3 architecture but includes modifications such as reducing the context length to 4096 tokens and training a tokenizer with a 32k vocabulary using SentencePiece.
  • Training Data: The model was pretrained on the fineweb-edu dataset and fine-tuned using the OpenHermes dataset with LoRA for efficient fine-tuning.
  • Training Process:
    • Pretraining: Conducted on vast.ai using 3090 and H100 GPUs, training on 20 billion tokens, resulting in a validation perplexity of 10.93.
    • Fine-Tuning: Performed on a 3060 GPU with LoRA over 52 hours, achieving a validation perplexity of 2.71.
  • Model Performance: While the model performs well on certain tasks, it still lags behind commercial models. For example, in poem generation, the model evolved from simple vocabulary to more complex expressions over time.

Open-Source Contributions

The developer has open-sourced the following:

Project Significance

  1. Low-Cost Practical Example: The project demonstrates how individuals can build and train large language models at a low cost, providing valuable insights for AI education and research.
  2. Open-Source Contribution: By open-sourcing the model weights and code, the developer promotes the dissemination of AI technology and fosters community collaboration.
  3. Technical Exploration: The developer experimented with various technical solutions, such as running the model on WearOS devices and using lm-eval for benchmarking, offering new perspectives on the diverse applications of AI.

Recommendations for Developers

For those looking to replicate or improve upon this project, consider the following:

  • Data Quality: Selecting high-quality training data is crucial for model performance.
  • Training Resources: Plan training resources wisely to avoid unnecessary computational costs.
  • Model Evaluation: Evaluate the model on more tasks and datasets to fully understand its capabilities and limitations.

Conclusion

Although Gemmeh may not match the performance of top commercial models, its open-source nature and low-cost characteristics make it a powerful tool for learning and exploring large language models. The developer has not only enhanced their technical skills through this project but also contributed valuable experience and resources to the AI community.


Source: Reddit r/LocalLLaMA (2026-08-10)

— END —

Tags: #Individual Developer #Open-Source Model #Large Language Model #LoRA #Gemmeh

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…