A computer’s victory in a video or board game can make headlines. For researchers, however, the important question is what the result reveals about how the machine learned and made decisions.
Additionally, different types of video games can help identify the limitations of the current AI systems being studied.
Clear Guidelines Allow for Quantification of Success
Video games typically have specific rules governing available actions and specific criteria for success. Therefore, when a computer generates a move and receives some form of feedback (for example, a score or a win/loss result), a researcher has a practical method to evaluate the effectiveness of various approaches to achieve similar success.
One example of such research was published in 2015 in the journal Nature, titled “Human-level control through deep reinforcement learning”. This research project tested a reinforcement-learning approach known as a deep Q-network (DQN) across 49 different Atari video games. The DQN received input from pixelated images taken from each game’s graphics along with game scores.
In essence, this study attempted to find out if it were possible for one particular type of solution to be effective across multiple visual challenges. While one approach may prove effective in one environment (or even multiple environments), it may fail in another. Additionally, although the individual games used in this study provided a convenient means to evaluate performance, they also created environments that can be repeated and evaluated again.
Self-Play Generates Experience
Another advantage of using games to train AI systems is that experience can be generated internally through self-play. In self-play, a computer trains by playing against itself. Each match provides feedback that can improve the strategy employed by the computer on subsequent matches.
For instance, AlphaZero demonstrated this concept in research published in 2018. Using the same general process for training, AlphaZero mastered three games: chess, shogi, and Go.
AlphaZero’s training required no pre-existing collection of expert human games. The combination of a learned evaluation function and a search through potential moves separated knowledge of legality from knowledge of usefulness. Specifically, the rules of a game define which moves are permissible; however, experience is needed to evaluate which permissible moves are most promising. Thus, video games represent a mechanism for researchers to study learning itself. Researchers are able to observe how performance improves over time due to additional experience rather than assuming that a programmer must specify every good decision in advance.
Hidden Information Changes the Problem
A chess player can view the entire board and see all pieces. A poker player cannot see opponents’ private cards. Regardless of how much computation is performed, missing information creates a problem that cannot be solved by simply performing more calculations; each decision must account for multiple possible scenarios.
When facing a bet in poker, a player must consider which hands an opponent could hold. The guide to poker ranges on GipsyTeam describes considering groups of potential hands rather than guessing one exact holding. That distinction helps explain why poker tests reasoning under uncertainty as well as searching ahead.
Similarly, an AI player must also take into consideration what information an opponent can infer based upon the AI player’s actions. For example, if an AI player consistently bets when holding a very strong hand but never bets when holding a weak hand, then opponents can exploit this consistent behavior. Therefore, choosing between alternative actions may include intentionally creating variation in behavior.
Pluribus, developed by Carnegie Mellon University and Facebook AI, demonstrated significant strength in six-player Texas Hold’em poker in 2019. Pluribus faced an additional challenge: it had to simultaneously compete with five other opponents versus competing with only one opponent.
Winning Takes More Than Just One Great Hand
In one Pluribus study, the system played against five professional players at once for 10,000 hands. The opponents came from a pool of 13 professionals. Carnegie Mellon University’s report describes both experiments. In one, Pluribus played five professionals at a time, selected from a pool of 13, for a total of 10,000 hands. In another, Darren Elias and Chris Ferguson each separately played 5,000 hands against five copies of Pluribus.
In addition to looking at the results themselves, it is helpful to consider the circumstances under which these scores were achieved. For example, time limits and the amount of processing power allowed per player can influence what a test measures. By providing enough information about the circumstances (in addition to the results), other researchers will have a much clearer idea of whether a later comparison is testing a better method for solving a similar problem, or simply trying to solve the problem in a way that is completely different than before.
New Situations Reveal the Limits
Achieving success in one type of environment does not prove that a program will achieve similar success in different environments. A program may learn specifics about its training data that do not transfer into new data. This problem is known as overfitting.
To evaluate how likely programs are to overfit, researchers created the CoinRun research environment. The environment creates random levels of platform games and provides opportunities for a program to train on them and then test on additional random levels. The primary goal is to determine if a program can perform similarly in new layout designs versus in the previously seen design elements. The researchers found that overfitting persisted even after large amounts of training data were provided.
This is an important reminder whenever evaluating the achievement of any form of game playing. Capabilities like planning, learning from feedback, and taking action with incomplete information are all valuable; however, each result pertains only to the specific tests performed. No single game achievement demonstrates capability at unrelated real world tasks. Games are well suited to allow researchers to ask very specific questions and receive detailed responses regarding those questions.
