ZICQ
中 Log in / Sign up
Newsroom Agentic #AI Games #Reinforcement Learning #Intelligent Agents #Decision Optimization #PUCT Algorithm

Reddit User Shares AI Algorithm Research: Optimizing Decision-Making for a Stochastic Single-Player Merge Game

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 6 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Reddit user CaiwenGong shares progress on an AI for a single-player stochastic merge puzzle game, which resembles 2048 but with a larger action space, stack constraints, and a previewed random event. The user details the game rules, current AI architecture, training methods, and challenges faced, while seeking advice on related algorithms, papers, or open-source projects. This research highlights the potential of AI in complex decision-making scenarios and provides insights for similar game AI d


Research Background and Objectives

Reddit user CaiwenGong is developing an AI for a single-player stochastic merge puzzle game, which shares similarities with 2048 but has the following characteristics:

  • Larger Action Space: Each action involves selecting a pair of different columns, resulting in 30 possible actions.
  • Stack Constraints: The game board consists of 6 vertical stacks, each with a maximum height of 7.
  • Random Event Preview: Every fourth player action is followed by a preview of six randomly generated elements, which the player must consider when making decisions.

The AI aims to maximize the number of 9s in a single game and the total number of 9s generated within 30 minutes. Since the game allows restarts after death, it is closer to a continuous average-reward/throughput problem than a traditional episodic score problem.

AI Architecture and Training Methods

State Representation

The current AI state representation includes:

  • 6x7x9 one-hot encoded game board
  • Four-action cycle phase
  • Six preview values (if known) and a preview-present flag
  • Current random tile value cap
  • Maximum number of empty columns reached in the current cycle and in each of the previous three cycles

There are a total of 394 features.

Network Structure

  • Policy/Value Network: Employs a column-permutation equivariant design, with each column processed by a shared encoder.
  • Action Scoring: An ordered source/destination pair head scores the 30 possible actions.
  • Value Prediction: Predicts the future count of 9s, the normalized distance to the next 9, and the short-term death risk.

Training Process

The training process follows an expert iteration/reanalysis approach:

  1. Use beam search and policy/value-guided PUCT to generate long games.
  2. Save complete episodes, root visit distributions, 9-event positions, deaths, and optional root action values.
  3. Train on column-permutation augmented data.
  4. Give extra policy weight to states after the first 9, states containing 7/8 tiles, high-scoring episodes, and states with human-like long-game structures.
  5. Use the updated network to generate new PUCT trajectories and repeat.

Current Results and Challenges

Under the assumed high-value-biased random distribution, the AI's performance is as follows:

  • An earlier explicit-chance PUCT model scored 81 total 9s in 16 episodes (mean 5.06, maximum 11, 2,365 actions).
  • Search distillation later produced a game with 13 total 9s in 272 actions, which remains the single-game maximum.
  • Adding four-cycle empty-column history resulted in 59 total 9s in 1,675 actions over 12 new episodes, versus 47 in 1,537 actions for its no-history teacher on the same seeds.

While the AI shows promise, it still lags behind human performance in the timed mode.

Issues Encountered

  • Q-head Learning Failure: Achieved low offline MAE but worsened closed-loop search.
  • Joint Fine-Tuning of Q Shared Encoder: Degraded existing policy and value estimates.
  • Increasing Root Minimum Visits: Increasing from 2 to 3 reduced performance.
  • Increasing Simulations: Increasing from 128 to 192 did not improve the paired sample.

Future Directions

The author is considering the following approaches:

  • 2048-style afterstate TD/N-tuple value learning
  • Gumbel MuZero/root sequential halving
  • Persistent tree reuse
  • Multi-horizon or distributional values
  • Average-reward training
  • Regime-switching drop model
  • Frozen base network plus residual adapters

Key Questions

The author poses several questions, including whether there are established algorithms or open-source projects for this type of game, whether an N-tuple afterstate value network plus expectimax would be a better fit, and how to allocate 128 simulations across 30 root actions and stochastic preview outcomes. The author also seeks advice on training an afterstate value without the all-action extrapolation failure and formulating the 30-minute objective as an average-reward continuing MDP or a fixed-horizon problem with automatic resets.

Conclusion

This research demonstrates the potential of AI in complex decision-making scenarios and provides insights for similar game AI development. The author seeks further guidance on algorithms, training methods, and potential improvements to enhance AI performance in this domain.


Source: Reddit r/MachineLearning (2026-08-11)

— END —

Tags: #AI Games #Reinforcement Learning #Intelligent Agents #Decision Optimization #PUCT Algorithm

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…