Disciplining children to clean up after themselves can be quite challenging. As a matter of fact, even some adults find this quite difficult. However, we encounter an even bigger challenge in attempting to get our artificial intelligence agents to perform the same. Success is dependent on the mastery of various core visuo-motor abilities, approaching an object, clutching it and lifting it, opening a box and depositing objects inside of it. To make matter even more complex, these abilities must be applied in the correct sequence. 

Control tasks, such as cleaning up a table or stacking up objects, need an agent to decide how, when and where to coordinate the nine joints of its simulated arms and fingers to shift accurately and accomplish its goal. The sheer variety of potential combinations of movements at any set time, combined with the requirement to carry out a protracted sequence of accurate and correct actions make up a serious exploration issue – making this is an especially fascinating sphere with regards to reinforcement learning research. 

Strategies such as reward shaping, apprenticeship learning or learning through demonstrations can assist with the exploration issue. Although, these strategies are dependent on a significant amount of know-how with regards to the task – the issue of learning complicated control problems from the ground up with minimal knowledge beforehand remains an open challenge. 

A new research paper puts forth a new learning paradigm referred to as ‘Scheduled Auxiliary Control (SAC-X)’ which intends to surpass this exploration problem. SAC-X has its basis on the concept to go about learning complicated tasks from the ground up, an agent has to go about learning to explore a grouping of essential skills first. Just like a baby gradually develops coordination and balance prior to her crawling or walking – furnishing an agent with internal (auxiliary) objectives correlating to simplistic skills enhances the probability it can comprehend and execute more advanced tasks. 

The SAC-X strategy was demonstrated on various simulated and real robot tasks leveraging an array of tasks which included stacking issues with differing objects and cleaning up a playground, which consists of shifting objects into a box. The auxiliary activities are defined adhering to a general principle, they compel the agent to explore its sensor space. For instance, activation of a touch sensor in its fingers, sensing a force in its wrist, maximization of a joint angle in its proprioceptive sensors or forcing a movement of an object in its visual camera sensors. Every task is connected with a simplistic reward or one if the objective is accomplished, and zero otherwise. 

The first thing the agent undergoes learning with regard to is activation of its touch sensors within the fingers and to move both objects. 

The simulated agent  ultimately masters the complicated activity of stacking objects. 

The agent can then determine autonomously with regards to its present intention, which objective to go after next. This could be auxiliary activity or an externally defined targeted activity. Critically, the agent can identify and learn from reward signals for all other activities that it is not presently following by making comprehensive utilization of replay-driven off-policy learning. For instance, when picking up or shifting an object, the agent could incidentally stack it, causing the observation of rewards for ‘stacking’. As a sequence of simplistic tasks can have the outcome of observation of an uncommon external reward, the capability to schedule intentions is critical. It can develop a personalized learning syllabus on the basis of all the tangential know-how it has gathered. This happens to be an efficient fashion of exploiting know-how in such a broad domain, and is especially useful when there are only minimal external reward signals available. The agent determines which intention to adhere to through a scheduling module. This scheduler is enhanced in the course of training through a meta-learning algorithm that attempts to maximize progress on the primary activity, which has the outcome of considerably enhanced data-efficiency. 

The assessments demonstrate that SAC-X is capable to find solutions to all the activities we set it from scratch – leveraging the same underlying grouping of auxiliary activities. Thrillingly, SAC-X is also to successfully go about learning a pick-up and a placing activity from scratch directly on an actual robot arm.  

SAC-X is considered to be a critical step towards learning of control activities from scratch, when just the cumulative objective is specified. SAC-X facilitates you in defining auxiliary activities randomly, they can have their basis on general insights, such as deliberately activating sensors as indicated here, but could ultimately integrate any activity an analyst believes is critical. In that regard, SAC-X is a general reinforcement learning strategy that is widely applicable in general sparse reinforcement learning settings beyond control and robotics.