Struna: Open-Source Rhythm Game Generates Music-Driven Scenes with Local AI Models

Summary:Struna is an open-source rhythm game developed by independent creator MrFelliks that allows users to upload their own MP3 files and experience real-time music-driven interactions in the browser. The game leverages four lightweight neural networks—BeatNet for beat tracking, Basic Pitch for note transcription, Spleeter for vocal separation, and SwiftF0 for pitch tracking—all running locally via WebAssembly for efficient on-device processing. Struna's architecture ensures data privacy by keeping al

---

Technical Mechanism Analysis

Struna's core architecture is built on four lightweight neural networks, which are executed locally in the browser via WebAssembly (wasm) to ensure data privacy and efficient processing:

1. BeatNet: Responsible for beat and downbeat tracking, identifying every beat and the start of each bar in the music. This provides the temporal grid for placing obstacles, jumps, and kicks in the game.
2. Basic Pitch (provided by Spotify): Performs note transcription, converting sustained notes into bars or lines for the player to swing on, and solos into strings of rings at the pitch of each note.
3. Spleeter: Handles vocal separation, extracting the voice from the mix to prevent it from confusing subsequent models.
4. SwiftF0: Tracks pitch, capturing the pitch of the singer's voice and the start time of each syllable every 16 milliseconds, enabling interactions with the melody.

These models have a combined size of about 22MB and are cached after the first load. If a device cannot run them in time, the game falls back to regular DSP methods, ensuring compatibility across various devices.

Engineering Trade-offs and Performance

Struna's design demonstrates significant engineering trade-offs in the following areas:

• Performance and Resource Consumption: By using lightweight models and WebAssembly, Struna maintains high performance while minimizing resource consumption, allowing it to run smoothly on mobile and low-end devices.
• Data Privacy: All models run locally, eliminating the risk of user data being uploaded to servers, thus protecting user privacy.
• Scalability: The game supports user-uploaded MP3 files and real-time AI-generated interactions, offering high customizability and interactivity.

In testing, Struna performs well on mainstream browsers and mobile devices, processing music and generating corresponding game scenes in real-time. However, BeatNet sometimes misjudges fast songs as half-speed, indicating room for improvement in model accuracy.

Developer Deployment and Recommendations

Struna's open-source nature makes it an excellent case study for developers exploring the intersection of AI and gaming. Here are some recommendations:

• Model Optimization: Developers can experiment with more advanced models or optimize existing ones to improve the accuracy of beat tracking and pitch detection.
• Multi-platform Support: Further optimize the code to support more platforms, such as VR devices and gaming consoles, to expand the user base.
• Community Collaboration: Encourage the community to contribute new models and features, such as support for more music genres or new interaction mechanisms.

Conclusion

Struna showcases the innovative application of AI in gaming, seamlessly blending music and interaction through local AI models. Its open-source nature provides a flexible platform for developers, inspiring more AI-driven creative projects.

---

Source: Reddit r/LocalLLaMA (2026-10-24)

— END —

Tags: #Open-Source AI Follow #Gaming AI Follow #Local Inference Follow #WebAssembly Follow #Music Interaction Follow