Humans are competent at finding solutions to a broad array of challenging issues, from low-level motor control all the way through to high-level cognitive functions. Deepmind has endeavoured to develop artificial agents that can accomplish a similar level of performance and generality. Much like a human being, their agents learn autonomously to accomplish effective techniques that have the outcome of the biggest longer-term rewards. This paradigm of learning through trial-and-error, only from rewards or punishments is referred to as reinforcement learning (RL). Also, much like a human, their agents build and go about learning their own knowledge straight from raw inputs, like vision, with no hand-engineered features or domain heuristics.  

This is accomplished by deep learning of neural networks. DeepMind has pioneered the combo of these strategies – deep reinforcement learning – to develop the first artificial agents to accomplish human-like performance levels across many challenging fields. 

The agents must make value judgments on an ongoing basis in order to choose good actions over bad. This know-how is indicated by a Q-network that estimates the cumulative reward that an agent can look forward to obtain after performing a specific action. DeepMind, in recent history, put forth the first broadly successful algorithm for deep reinforcement learning. The primary idea was to leverage deep neural networks to indicate the Q-network, and to train this Q-network to forecast cumulative reward. Prior efforts to bring together reinforcement learning with neural networks had mostly met with failure owing to unstable learning. To tackle these instabilities, the Deep Q-Networks (DQN) algorithms records all of the agent’s experiences and then arbitrarily samples and replays them to furnish diverse and decorrelated training information. DQN was applied to go about learning to play games on the classic Atari 2600 console. At every time-step the agent looks at the raw pixels on the monitor, a reward signal corresponding to the game scoreboard, and chooses a joystick direction. In a Nature paper, separate DQN agents obtained training for 50 different Atari titles, without any previous know-how of the game rules. 

Surprisingly, DQN accomplished human-like performance in nearly 50% of the 50 games to which it was applied, almost 25 games to be precise; surpassing any prior methodology by quite a margin. The DQN source code and Atari 2600 emulator are freeware for anyone who desires to play around and experiment for themselves. 

The DQN algorithm has been subsequently enhanced in several ways, through further stabilisation of the learning dynamics, prioritization of the replayed experiences, normalising, aggregating, and re-scaling the outputs. Bringing together a lot of these enhancements together had the outcome of a 300% enhancement in mean score across Atari titles, human-like performance has now been accomplished in nearly all of the Atari titles. It is even possible to go about training a singular neural network to learn about several Atari titles. A massively distributed reinforcement learning system has also been developed, referred to as Gorilla, that leverages the Google Cloud Platform to hasten training times by an order of magnitude, this framework has had application to recommender systems within Google. 

But deep Q-networks are just one method to identify a solution to the deep reinforcement learning issue. An even more practical and efficient strategy was put forth on the basis of asynchronous reinforcement learning. This strategy exploits the multithreading functionalities of conventional CPUs. The concept is to execute several instances of the agent in parallel, but leveraging a shared model. This furnishes a feasible alternative to experience replay, as parallelization also diversifies and decorrelates the information. The asynchronous actor-critic algorithm, A3C, brings together a deep Q-network with a deep policy network for choosing actions. It accomplishes bleeding-edge results, leveraging a fraction of the training time of DQN and a fraction of the resource consumption of Gorilla. By constructing unique approaches to intrinsic motivation and temporally abstract planning, they have also accomplished revolutionary results in the infamously challenging Atari title, Montezuma’s Revenge. 

While Atari titles illustrate a broad range of diversity, they are restricted to 2D sprite-based video games. Due to this, DeepMind has put forth Labyrinth: a challenging suite of 3D navigation and puzzle-solving environments. Once more, the agent just observes pixel-based inputs from its immediate field-of-view, and must find out the map to isolate and exploit rewards. 

Surprisingly, the A3C algorithm accomplishes human-like performance, out-of-the-box, on several Labyrinth activities. An alternative strategy on the basis of episodic memory has also proved its efficacy. Labyrinth has currently been released as an open-source project. 

They have also developed a plethora of deep reinforcement learning methodologies for ongoing control issues like robotic manipulation and locomotion. The Deterministic Policy Gradients Algorithm (DPG) furnishes a continuous analogue to DQN, exploiting the differentiability of the Q-network to find solutions to a broad variety of continuous control activities. Asynchronous reinforcement learning also features good performance in these fields, and when augmented with a hierarchal control technique, can find solutions to challenging issues like ant soccer and a 54-dimensional humanoid slalom, with no know-how beforehand of the dynamics. 

The game of Go is one of the most challenging classic games. Regardless of years of effort, methods leveraged before just accomplished amateur level performance. They generated a deep reinforcement learning algorithm that goes about learning both a value network (which forecasts the winner) and a policy network (which chooses actions) through games of self-play. The program AlphaGo brought together these deep neural networks with a bleeding-edge tree search. 6 years ago, in October 2015, AlphaGo became the first program to defeat a professional human player. In March of that year, AlphaGo beat Lee Sedol by 4 to 1, in a matchup that was looked at by an estimated 200 million viewers. 

Independently, they have also generated game theoretic strategies to deep reinforcement learning, culminating in a super-human poker player for heads-up limit Texas Hold’em. 

Ranging from Atari to Labyrinth, from manipulation through locomotion, to poker and even the game of Go, the deep reinforcement learning agents have illustrated remarkable progress on a broad array of challenging activities. The goal is to persist in enhancing the capabilities of the agents, and to leverage them to make a positive impact on society, in critical domains such as healthcare.