Humans have the ability to forget unnecessary information to make space for new patterns that can be significant while making decisions. Facebook AI is moving the needle with Expire-Span to make this a reality for machine learning models. As machine learning models learn new patterns, it keeps on collecting new information, making them computation intensive. As a workaround, researchers embraced the compression technique, where less relevant data is compressed. But, this resulted in blurry visions of memory for tasks that require models to look a long way back to enhance accuracy.
To eliminate this challenge, Facebook AI introduced a novel approach — Expire-Span — to set the expiration time of data. According to the researchers, Expire-Span is a first-of-its-kind operation that enables neural networks to forget at scale. It allows machine learning-based systems to make space for more information while reducing the computational requirements.
For instance, if a machine learning model is tasked to find a yellow door, it stores all the patterns collected while iterating to find the right path. Even after finding the correct patterns, it remembers other unnecessary details that might not help it achieve its goal in the future. This is where Facebook AI’s Expire-Span approach is making the grounds in achieving human-like abilities by deleting nonessential data.
Also Read: Brain Storage Scheme Can Solve Artificial Networks’ Memory Woes
“Expire-Span calculates the information’s expiration value for each hidden state each time a new piece of information is presented, and determines how long that information is preserved as a memory,” mentions the researchers. Facebook AI’s Expire-Span determines the span based on context learned from data and influenced by its surrounding memories. Besides, the span size can be adjusted when needed at a later stage to retain information for a longer period.
Facebook AI researchers evaluated the performance of models that are equipped with Expire-Span against the state-of-the-art models. The Expire-Span models required less computation while delivering comparable performance. “The impressive scalability and efficiency of Expire-Span has exciting implications for one day achieving a wide-range of difficult, human-like AI capabilities that otherwise would not be possible,” wrote the researchers.
The research is in its early stages, and Facebook AI is committed to further enhancing the capabilities of the approach. Nevertheless, the researchers believe that Expire-Span can go beyond research and help in real-world applications.
Read the complete research paper here.
