Reddit User Shares AI Algorithm Research: Optimizing Decision-Making for a Stochastic Single-Player Merge Game
Summary:Reddit user CaiwenGong shares progress on an AI for a single-player stochastic merge puzzle game, which resembles 2048 but with a larger action space, stack constraints, and a previewed random event. The user details the game rules, current AI architecture, training methods, and challenges faced, while seeking advice on related algorithms, papers, or open-source projects. This research highlights the potential of AI in complex decision-making scenarios and provides insights for similar game AI d
Research Background and Objectives
Reddit user CaiwenGong is developing an AI for a single-player stochastic merge puzzle game, which shares similarities with 2048 but has the following characteristics:
- Larger Action Space: Each action involves selecting a pair of different columns, resulting in 30 possible actions.
- Stack Constraints: The game board consists of 6 vertical stacks, each with a maximum height of 7.
- Random Event Preview: Every fourth player action is followed by a preview of six randomly generated elements, which the player must consider when making decisions.
The AI aims to maximize the number of 9s in a single game and the total number of 9s generated within 30 minutes. Since the game allows restarts after death, it is closer to a continuous average-reward/throughput problem than a traditional episodic score problem.
AI Architecture and Training Methods
State Representation
The current AI state representation includes:
- 6x7x9 one-hot encoded game board
- Four-action cycle phase
- Six preview values (if known) and a preview-present flag
- Current random tile value cap
- Maximum number of empty columns reached in the current cycle and in each of the previous three cycles
There are a total of 394 features.
Network Structure
- Policy/Value Network: Employs a column-permutation equivariant design, with each column processed by a shared encoder.
- Action Scoring: An ordered source/destination pair head scores the 30 possible actions.
- Value Prediction: Predicts the future count of 9s, the normalized distance to the next 9, and the short-term death risk.
Training Process
The training process follows an expert iteration/reanalysis approach:
- Use beam search and policy/value-guided PUCT to generate long games.
- Save complete episodes, root visit distributions, 9-event positions, deaths, and optional root action values.
- Train on column-permutation augmented data.
- Give extra policy weight to states after the first 9, states containing 7/8 tiles, high-scoring episodes, and states with human-like long-game structures.
- Use the updated network to generate new PUCT trajectories and repeat.
Current Results and Challenges
Under the assumed high-value-biased random distribution, the AI's performance is as follows:
- An earlier explicit-chance PUCT model scored 81 total 9s in 16 episodes (mean 5.06, maximum 11, 2,365 actions).
- Search distillation later produced a game with 13 total 9s in 272 actions, which remains the single-game maximum.
- Adding four-cycle empty-column history resulted in 59 total 9s in 1,675 actions over 12 new episodes, versus 47 in 1,537 actions for its no-history teacher on the same seeds.
While the AI shows promise, it still lags behind human performance in the timed mode.
Issues Encountered
- Q-head Learning Failure: Achieved low offline MAE but worsened closed-loop search.
- Joint Fine-Tuning of Q Shared Encoder: Degraded existing policy and value estimates.
- Increasing Root Minimum Visits: Increasing from 2 to 3 reduced performance.
- Increasing Simulations: Increasing from 128 to 192 did not improve the paired sample.
Future Directions
The author is considering the following approaches:
- 2048-style afterstate TD/N-tuple value learning
- Gumbel MuZero/root sequential halving
- Persistent tree reuse
- Multi-horizon or distributional values
- Average-reward training
- Regime-switching drop model
- Frozen base network plus residual adapters
Key Questions
The author poses several questions, including whether there are established algorithms or open-source projects for this type of game, whether an N-tuple afterstate value network plus expectimax would be a better fit, and how to allocate 128 simulations across 30 root actions and stochastic preview outcomes. The author also seeks advice on training an afterstate value without the all-action extrapolation failure and formulating the 30-minute objective as an average-reward continuing MDP or a fixed-horizon problem with automatic resets.
Conclusion
This research demonstrates the potential of AI in complex decision-making scenarios and provides insights for similar game AI development. The author seeks further guidance on algorithms, training methods, and potential improvements to enhance AI performance in this domain.
— END —Source: Reddit r/MachineLearning (2026-08-11)
Tags: #AI Games #Reinforcement Learning #Intelligent Agents #Decision Optimization #PUCT Algorithm
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments