Many recent studies in procedural content generation (PCG) are based on machine learning. One of the promising approaches is generative models, which have shown impressive results in generating new pictures and videos from existing ones. However, it is usually costly to collect sufficient content for training on PCG. To address this issue, we consider reinforcement learning (RL), which does not need to collect training data in advance but learns from its interaction with an environment. In this work, RL agents are trained to generate stages, which we define as series of events in turn-based role playing games. It is a challenging task since several events in a stage are usually highly correlated to each other. We first formulate the stage generation problem into a Markov decision process. A hand-crafted evaluation function, which simulates players’ enjoyment, is defined to evaluate generated stages. Two RL algorithms are selected in the experiments, which are deep Q-network for discrete action space and deep deterministic policy gradient for continuous action space. The generated stages from both models receive evaluation values indicating good quality. To solve the delayed reward problem and further improve the quality of the stages, we employ virtual simulations (VS) to give rewards to intermediate actions and get stages with higher average scores. In addition, we introduce noise to avoid generating similar stages while trying to keep the quality as high as possible. The proposed methods succeed in generating good and diverse stages.
Modeling human players' behaviors in games is a key challenge for making natural computer players, evaluating games, and generating content. To achieve better humancomputer interaction, researchers have tried various methods to create human-like AI. In chess and Go, supervised learning with deep neural networks is known as one of the most effective ways to predict human moves. However, for many other games (e.g., Shogi), it is hard to collect a similar amount of game records, resulting in poor move-matching accuracy of the supervised learning.We propose a method to compensate for the weakness of the supervised learning policy by Blending it with an AlphaZerolike reinforcement learning policy. Experiments on Shogi showed that the Blend method significantly improved the move-matching accuracy over supervised learning models. Experiments on chess and Go with a limited number of game records also showed similar results. The Blend method was effective with both medium and large numbers of games, particularly the medium case. We confirmed the robustness of the Blend model to the parameter and discussed the mechanism why the move-matching accuracy improves. In addition, we showed that the Blend model performed better than existing work that tried to improve the move-matching accuracy.
Procedural content generation (PCG) is essential in game development, automating content creation to meet various criteria such as playability, diversity, and quality. This paper leverages reinforcement learning (RL) for PCG to generate Super Mario Bros levels. We formulate the problem into a Markov decision process (MDP), with rewards defined using player enjoyment-based evaluation functions. Challenges in level representation and difficulty assessment are addressed by conditional generative adversarial networks (CGAN) and human-like AI agents that mimic aspects of human input inaccuracies. This ensures that the generated levels are appropriately challenging from human perspectives. Furthermore, we enhance content quality through virtual simulation (VS), which assigns rewards to intermediate actions to address a credit assignment problem (CAP). We also ensure diversity through a diversity-aware greedy policy (DAGP), which chooses not-bad-but-distant actions based on Q-values. These processes ensure the production of diverse and high-quality Super Mario levels. Human subject evaluations revealed that levels generated from our approach exhibit natural connection, appropriate difficulty, non-monotony, and diversity, highlighting the effectiveness of our proposed methods. The novelty of our work lies in the innovative solutions we propose to address challenges encountered in employing PCGRL in Super Mario Bros, contributing to the field of PCG for game development.
The AlphaZero algorithm achieved superhuman levels of play in chess, shogi, and Go, by learning without domain-specific knowledge except for game rules. This paper targets stochastic games and investigates whether AlphaZero can learn theoretical values and optimal play. Since the theoretical values of stochastic games are expected win rates, not a simple win, loss, or draw, it is worth investigating the ability of AlphaZero to approximate expected win rates of positions. This paper also thoroughly studies how AlphaZero is influenced by hyper-parameters and some implementation details. The analyses are mainly based on AlphaZero learning with lookup tables. Deep neural networks (DNNs) like the ones in the original AlphaZero are also experimented and compared. The tested stochastic games include reduced and strongly-solved variants of Chinese dark chess and EinStein würfelt nicht!. The experiments show that AlphaZero can learn policies that play almost optimally against the optimal player and can learn values accurately. In more detail, such good results are achieved by different hyper-parameter settings in a wide range, though it is observed that games on larger scales tend to have a little narrower range of proper hyper-parameters. In addition, the results of learning with DNNs are similar to lookup tables.