Mastering the technique, tactical comprehension, and team play that make up modern sophisticated multiplayer video games indicates a vital challenge for artificial intelligence research. In one of the latest researches, now released in the journal Science, new developments are put forth within reinforcement learning, having the outcome of human-level performance in Quake III Arena: Capture the Flag. This is a complicated, multi-agent environment and one of the canonical 3D first-person multiplayer titles. The agents positively cooperate with both human and artificial teammates, and depict high performance even when received training with reaction times which can be compared to human players. 

Further, we illustrate how these strategies have managed to scale beyond research Capture the Flag environments to the complete game of Quake III arena. 

Billions of people make up the population of the planet, every one with their own individual objectives and actions, but still have the capacity of coming together through teams, organizations and societies in impressive displays of collective intelligence. This is a scenario we refer to as multi-agent learning: several individual agents must act autonomously, yet go about learning to interact and cooperate with other agents. This is a massively tough issue – as with co-adapting agents the world is consistently changing. 

To look into this issue, 3D first-person multiplayer video games are observed. These titles signify the most famous genre of video game, and have captured the imagination of millions of players due to their immersive gameplay, in addition to the hurdles they pose with regards to strategy, tactics, hand-eye coordination, and teamplay. The hurdles for our agents is to go about learning directly from raw pixels to generate actions. This complexity makes first person multiplayer games a fruitful and active sphere of research with the Artificial Intelligence community. 

The game that was concentrated on in this research is Quake III Arena (which was aesthetically modded, though all in-game mechanisms stay the same). Quake III Arena set down the foundations for several modern first-person video games, and has attracted a long-standing competitive e-sports scenario. We undertake training of agents that go about learning and act as individuals, but which must be capable of playing on teams with and against any other agents, human or artificial. 

The rules of Capture the Flag are easy to understand, however, the dynamics are complicated. Two teams of individual players contest on a provided map with the objective of capturing the rival team’s flag while safeguarding their own. To obtain tactical advantage they can tag the rival team members to put them back to their spawn points. The team with the most flag captures upon five minutes wins. 

From a multi-agent viewpoint, Capture the Flag needs players to both positively cooperate with their teammates as well as contest with the rival team, while staying robust to any playing style they might face. 

To make things even more fascinating, we think of a variant of CTF in which the map layout alters from match to match. As a result, our agents are compelled to obtain general techniques over memorising the map layout. Also, to level up the playing field, the learning agents experience the domain of CTF in an identical way to humans: they look at a stream of pixel images and issue actions via an emulated game controller. 

The agents must go about learning from scratch how to view, act, cooperate, and compete in not before seen environments, all from a singular reinforcement signal per match; whether their team had victory or not. This is a big challenge as a learning issue, and it solution is on the basis of three general concepts for reinforcement learning: 

• Instead of training a singular agent, we go about training a population of agents, which undertake learning by playing with and against one another, furnishing a diversity of teammates and opponents. 
• Every agent in the population goes about learning its proprietary reward signal, which enables agents to produce their own internal objectives, like capturing a flag. A two-tier optimisation process streamlines agent’s internal rewards directly for victory, and leverages reinforcement learning on the internal rewards to go about learning the agent’s policies. 
• Agents function at two timescales: quick and slow, which enhances their capability to leverage memory and produce consistent action sequences. 

The outcome agent, referred to as the “For the Win (FTW)” Agent, goes about learning to play capture the flag to a very high standard of quality. Critically, the learned agent policies are robust to the size of the maps, the number of teammates, and the other competitors on their team.  

A tournament was run consisting of 40 human players, in which humans and agents were arbitrarily matched up in games – both as rivals and as teammates. 

The FTW agents go about learning to become  a lot more stronger than the strong baseline techniques, and supersede the win-rate of the human agents. As a matter of fact, in a survey amongst participants they were rated more collaborative than human agents. 

Moving beyond just performance assessment, it is critical to comprehend the emergent intricacy in the behaviours and internal representations of these agents. 

To comprehend how agents represent game state, we take up activation patterns of the agent’s neural networks plotted on a plane. Dots in the image below indicate scenarios during play with nearby dots indicating similar activation patterns. These dots are coloured going by the high-level CTF game state in which the agent identifies itself. In which room is the agent present? What are the status of the flags? What teammates and rivals can be viewed? We observe clusters of the similar colour, signifying that the agent indicates similar high-level game states in a likewise fashion. 

This is a viewpoint in how the agents represent the game world. In the plot on the top, neural activation patterns at a provided time are plotted based on how similar they are to each other. The nearer two points are in space, the more similar their activation patterns. They then receive colouring based on the game scenario at the time – same colour, similar situation, we observe that these neural activation patterns are organised, and make up clusters of colour, signifying that agents are indicating meaningful aspects of gameplay in a stereotyped, organised manner. The agents that have received training even demonstrate some artificial neurons which code directly for specific situations. 

The agents are never informed anything with regards to the rules of the game, yet learn about basic game theories and basically generate an intuition for CTF. As a matter of fact, we can identify specific neurons that code directly for some of the most critical game states, like a neuron that activates when the agent’s flag is taken, or a neuron that undergoes activation, when an agent’s teammate is holding a flag. The paper furnishes further analysis covering the agent’s utilization of  memory and visual attention. 

How did the agent feature as good performance as they did. To start with, it was observed that the agents had very quick reaction times and were really precise taggers, which might describe their performance (tagging is a tactical behaviour that puts opponents back to their beginning point.) Humans are relatively slow to process and act on sensory input, owing to our reduced biological signalling. Here’s an instance of a reaction time test you can look for yourself. Therefore, the agent’s superior performance might be an outcome of their quicker visual processing and motor control. But, by artificially minimizing this precision and reaction duration, we observed that this was only one aspect in their success. In a subsequent study, further agents were trained which have an inherent delay of a quarter of a second (267 ms) – that is, agents possess a 267ms lab before observation of the world – compared with reported reaction durations of human agent video game players. These response-delayed agents still outpaced human competitors with robust humans only winning 21% of the time. 

The win percentages of human players against response-delayed agents are minimal, signifying that even with human-comparable reaction delays, agents outpace human players. Additionally, observing the average number of game events by humans and response-delayed agents, we observe comparable figures of tagging events, demonstrating that these agents do not possess benefits in this regard. 

Via unsupervised learning we set up the prototypical behaviours of agents and humans to find out that agents as a matter of fact learn human-like behaviours, like following teammates and camping in the rival’s base. 

These behaviours prop up in the course of training, via reinforcement learning and population-level evolution, with behaviours, like teammate following – falling out of favour as agents go about learning to cooperate in a mor complementary fashion. 

While this blog post concentrates on Capture the Flag, the research contributions are general and many are thrilled to observe how other develop upon these strategies in differing complicated environments. Ever since preliminary publication of these outcomes, success has been identified in extension of these methods to the complete game of Quake III Arena, which consists of professionally contested maps, increased multiplayer game modes, on top of Capture the Flag, and more gadgets and pickups. Preliminary results demonstrate that agents can play several game modes and several maps competitively, and are beginning to challenge the abilities of our human researchers in test matches. Indeed, ideas put forth in this research, like population-based multi-agent RL make up a foundation of the AlphaStar agent in our research on StarCraft. 

Generally, this work illustrates the potential of multi-agent training to advance the development of artificial intelligence: exploitation of the natural curriculum furnished by multi-agent training, and forcing the production of robust agents that can even collaborate with human beings.