This blog post by AICoreSpot, we put forth a new long-range memory framework, the Compressive Transformer, combined with a new benchmark for book-level language modelling, PG19. We furnish the conceptual utilities required to comprehend this new research in the context of latest advancements in the realm of memory models and language modelling. 

Over the course of our lives, we develop memories that are stored over a diverse array of timescales, ranging from minutes to decades, and for the lucky few, even centuries. When we are reading a novel, we can remember characters who came into the story several chapters ago, or in an earlier entry in a book series, and reason with regards to their motivations and probable actions in the present context. We can even set the book down in a busy week, and pick back up from where we were last reading without having to go over the plotline again.  

We do not accomplish such feats by recording every aspect of sensory feedback and input we gather about the world throughout our lifespan. Our brains choose, filter, and integrate input stimulus on the basis of aspects of relevance, surprise, perceived danger, and repetition. To put it in different words, we leverage a kind of compression mechanism for lifelong experiences to a group of salient memories which assist us in comprehending the past, and better forecast the future. A primary objective of artificial intelligence scientists is identifying ways to facilitate implementation of such abilities in computing systems and benchmarks which need complicated reasoning over protracted timeframes. 

Memory frameworks for artificial neural networks have progressed significantly in the previous twenty years. In this blog post by AICoreSpot, we look to historical advancements to look into why this is such a tough task and consider how natural language modelling could provide an effective means of developing improved long range memory frameworks. We reflect on the requirement for improved compressive memory architectures, and sparse memory access mechanisms, to work towards the objective of integrating lifelong reasoning in our computing frameworks. 

One of the forerunners and most broadly leveraged memory architectures in the current day is a recurrent neural network (RNN) referred to as the Long Short-Term Memory (LSTM). The LSTM goes about maintaining a concise memory in the shape of a vector of numbers, which it accesses and alters with gated read, write, and forget operations. It was originally produced on a suite of synthetic activities that consisted of learning logical functions on a stream of bits. But it has since become a widely-used model of sequential data: from identifying handwritten notes to forecasting the early onset of kidney injury.  

One drawback of the LSTM, and of several recent RNNs, is their capacity. They are developed so that every unit of memory can influence each other unit in memory with a learnable weight. But this has the outcome of an inefficient system in a computational sense, the number of learnable parameters in the framework expands quadratically with the memory size. For instance, an LSTM with a memory of 64KB size has the outcome of parameters of 8GB size. Bypassing this memory capacity hurdle has been an active sphere of research. 

Scientists at DeepMind have put forth a unique architecture, the Differentiable Neural Computer (DNC) which goes about augmenting an LSTM with a much bigger memory matrix to tackle these drawbacks. The DNC leverages an attention operation to go about reading from this memory matrix. In visual attention, our eyes focus on pertinent objects in a visual scene – for instance, one might usually use more time observing a friend’s face when they are having an emotionally-loaded conversation that on observing their shoes. Here, memory frameworks can attend to specific events/information from history. This attention function needs a static number of parameters, independent of the memory size, and therefore the memory capacity of the framework can be considerably enhanced. 

Along with the DNC, recurrent neural networks with an extra attention mechanism were demonstrating promise in the fields of translation and question answering. These frameworks were capable of reasoning over the passage of time leveraging dual memory structures: a small and compact LSTM memory and a large exterior memory. But, in recent times, scientists at Google Brain Team put forth the Transformer which eradicates the LSTM, and just leverages attention to communicate data across time. 

The Transformer was initially demonstrated to considerably outperform recurrent neural networks with regards to machine translation. But it has since then had application in a plethora of applications within natural language processing, from question answering, document summarization, sentiment classification and the modelling of natural language – an activity that seen especially thrilling advancements over the previous year. 

Discovering machine learning tasks which both drive the production of improved memory architectures and push us further in the path of artificial general intelligences is a challenge, to say the least. Statistical language modelling is one such activity that many believe could be worthwhile for both causes. Language models function by sequentially forecasting the next word in a stream of text. They can be leveraged to model current texts and also to produce novel texts. As they improve at modelling history, their forecasting becomes more precise, and the texts they produce become more authentic. 

In Claude Shannon’s groundbreaking article “A Mathematical Theory of Communication”, put out all the way back in 1948, which discovered the domain of information theory, he spoke about primitive language models and demonstrated how adding extra context enhances the quality and realism of text that is produced. He performs this by introducing the most simplistic framework of English text, which has nil contextual modelling, a character level model which treats every character independently. Through sampling of characters with their comparative frequencies, (8% of the time for ‘a’, 1.5% for ‘b’ etc.) we come up with a string that is somewhat nonsensical. 

XFOML RXKHRJFFJUJ ZLPWCFWKCYJ FFJEYVKCQSGHYD QPAAMKBZAACIBZLHJQD. 

But, he remarks at the enhancement within sample quality if one rather models the probability of words independently. Now, the modelled context is nearly 7x bigger (the average amount of characters in a word.) 

REPRESENTING AND SPEEDILY IS AN GOOD APT OR COME CAN DIFFERENT NATURAL HERE HE THE A IN CAME THE TO OF TO EXPERT GRAY COME TO FURNISHES THE LINE MESSAGE HAD BE THESE. 

Through modelling the probability of word pairings, a further 2 fold in context length, even more authentic text emerges. 

THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED 

To put it in different words, an appreciation in the length of the context has the outcome of an enhancement in the quality of text that is produced. Shannon comments on the quality of his generated samples and theorizes that natural text samples may crop up from an adequately complicated statistical model, “The particular sequence of ten words “attack on an English writer that the character of this” is not at all unreasonable. It appears then that an adequately complicated stochastic procedure will provide a satisfactory representation of a discrete source. 

One critique of language modelling as an activity for long-range reasoning is that models can gather a major portion of their forecasting from the local context. Natural language models have conventionally overlooked the broader context, concentrating mainly on the shorter term. For instance, in 2017, Dailuk et al. discovered their neural language model uncommonly attends past the preceding five words. But in the previous year, big transformer models have been demonstrated to leverage hundreds of words of context to produce ever more authentic text with a longer range of coherence. A demonstration from OpenAI’s GPT-2 a 1.5B parameter transformer, suggests that the model is capable of producing authentic text and retaining key entities. (For e.g. Dr. Jorge Perez and unicorns) across several paragraphs: 

The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science. 

Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. 

Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. 

Pérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez. 

Pérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns. 

While examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.” 

Dr. Pérez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America. 

While their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilization. According to Pérez, “In South America, such incidents seem to be quite common.” 

However, Pérez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution, or at least a change in social organization,” said the scientist. 

Such samples would probably shock Shannon, 70 years added to his preliminary language model experiments. But the real advantage of powerful neural language models, and their relevance to the objective of artificial general intelligence, is their capability to go about transferring knowledge to a suite of activities. In the procedure of learning how to model text, neural language frameworks appear to develop a knowledge-base of associations, and a broad array of skills.  

For example, scientists at OpenAI demonstrated that GPT-2 can have application to natural language processing tasks like question answering, paraphrasing, or sentiment analysis with shockingly good performance – particularly for a model that has never overtly received training to execute such tasks. When large transformer language models are tuned on specific tasks like question answering, the outcome performance is considerably improved in contrast to models that were developed and received training just for question answering. Google’s prominent natural language framework, BERT, accomplishes bleeding-edge performance on a broad array of NLP benchmarks, and is currently a part of Google search. And more lately, it was demonstrated that GPT-2 can go about learning to play basic chess by training it on strings of game moves. 

A famous long-range language model benchmark is WikiText-103, which consists of English language wiki pages, and was produced by scientists at Salesforce AI. Articles are approximately 3600 words on average, which, at the time of development, far surpassed the memory window of cutting edge models. 

But scientists at Google lately demonstrated that a Transformer variant referred to as the TransformerXL – which maintains a memory of historical network activations and lately obtained bleeding-edge outcomes on WikiText-103 – can leverage contexts spanning over one thousand unique words. This puts forth the question: will models, sooner, rather than later, saturate these benchmarks?  

To support increasing interest in long-range sequence models, DeepMind have released a new language modelling benchmark, PG-19 which is obtained from books in the Project Gutenberg reserve. 

Books furnish rich context for the production of long-range memory models. A subset of nearly 28,000 books was chosen from Project Gutenberg put out before 1919. Unlike previous language modelling dataset releases, there was application of very minimal pre-processing to the text. For instance, the vocabulary size was not restricted and numbers were not censored, to prevent the filtering of useful data. 

PG-19 is two-fold the size of previous language modelling benchmarks, like the Billion Word Benchmark and consists of text that is over ten-fold longer in context that the previous long-range language model benchmark. WikiText-103.  

Along with a new benchmark, DeepMind put forth a long-range memory model referred to as the Compressive Transformer. This draws inspiration from the role of sleep in the formation of consolidated episodic memories. Sleep is critical for memory, and it’s general knowledge that sleep functions to compress and consolidate memories, therefore enhancing reasoning capabilities for memory activities. In the Compressive Transformer, granular memories like episodic memories are gathered online as the model passes over a sequence of inputs, over time, they are ultimately connected. 

The Compressive Transformer leverages attention to choose data from history, like the Transformer. It upkeeps a short-term memory of historical activations, in the same fashion as the recently-proposed TransformerXL. Where the TransformerXL gets rid of historical activations when they get older, the Compressive Transformer rather compacts them into compressed memory. The compression is executed by a neural network directed by an auxiliary loss that directs it to keep around task-relevant data. It can go about learning to filer irrelevant memories, as well as bring together memories so that the critical data is maintained and can be recovered over a longer period of time. 

It was discovered that the Compressive Transformer has cutting-edge performance in the modelling of natural language for two broadly-leveraged long-range benchmarks. WikiText-103 and Enwik8 contrasted to released results that do not leverage extra sources of training information. It is also demonstrated that it can be leveraged effectively to model speech, manages uncommon words extremely well, and can be leveraged within a reinforcement learning agent to find a solution to a memory task. 

The Compressive Transformer generates the biggest performance gain in modelling long-context book text from the PG-19 benchmark. The model’s conditional samples can be leveraged to write book-like extracts. Below is depicted a sample that is input a paragraph of text to be leveraged as context, taken from “The Patrol of the Sun Dance” by Ralph Connor which the model has not observed prior. 

Along the tote-road, which ran parallel to the steel, a man, dark of skin, slight but wiry, came running, his hard panting, his streaming face, his open mouth proclaiming his exhaustion. At a little trail that led to the left he paused, noted its course toward the flaunting flag, turned into it, then struggled up the rocky hillside till he came to the wooden shack, with a deep porch running round it, and surrounded by a rustic fence which enclosed a garden whose neatness illustrated a characteristic of the British soldier. The runner passed in through the gate and up the little gravel walk and began to ascend the steps… 

Continuation by the Compressive Transformer 

 …As though by mutual consent, all the three members of the Superintendent Force were standing on the veranda watching his progress, a man came strolling slowly down the path, stopping often to gaze curiously about him, then stopping once more when he saw the four standing on the veranda. His face was dark, his eyes prominent, and the lines that furrowed his forehead showed marks of many woes, yet his step was alert and he began to mount the steps with steady strides.  

The man’s attention was diverted by the unexpected appearance of the two policemen; a third man came slipping up, grumbling under his breath, and stopped long enough to look doubtfully around him, then slipped away again. In the meantime, the four on the veranda continued to watch the approach of the new arrival. In spite of himself, the Superintendent Force became conscious of a curious feeling–uneasiness that was almost fear–in the presence of that strange man. He would not allow himself to admit the fact, yet he felt it in his bones. But to the watchers, the plain, everyday features of that stranger and his coming, seemed only just what the Seven White Shee owed him–their weight, their hurry, their blast… 

The Compressive Transformer is capable to generate narrative in a plethora of styles, from multi-character dialogue, to first-person diary entries, or third-person prose. Even though the model does not possess comprehension of language that’s based in the practical world, or the events that occur in it, by gathering longer-range correlations, we witness the emergence of text with increased coherence. 

As we look to develop agents that function over days, weeks, or even years, it will not be practical to compute over all raw input information at every timestep. Even with the present growth in computational capability, we will be required to develop compressive and sparse architecture for memory to develop representations and reason about behaviours. 

Models which are capable to gather relevant correlations across the timespan of days, months, or years’ worth of experience are just around the corner. The pathway to more capable reasoning over time will emerge from improved selective attention of history, and more efficient mechanisms to compress it. As we look into ideas and concepts in this sphere, we require tasks and datasets that span longer and longer time intervals. The PG-19 dataset can facilitate researchers shift in this direction, putting forth textual information in the longest form that we conventionally consume as humans: complete length books. The hope is that its release will compel interest in new frameworks that compress history in the pursuit of forecasting the future and act efficiently in the present.