An AI has reconstructed the frame sequence of a video based on the brain activity of mice that viewed the images.
An artificially intelligent system has put the frames in a black-and-white video in almost perfect order based on the brain activity of mice that viewed the video. The system is developed by AI researcher Mackenzie Mathis from the Technical University of Lausanne and her colleagues. The results have been published in Nature.
The researchers used data on the brain activity of about fifty mice. This brain activity was measured while the mice watched a 30-second video ten times. They trained an AI to link the data from the first nine viewing sessions to the video. The video consists of 600 frames, and shows a man running to a car and opening the trunk.
READ ALSO
We don’t need miracles to save the world
If we focus on revolutionary solutions such as carbon capture and geoengineering …
Mathis and her team then measured whether their AI could predict the order of the frames in the video based on the brain activity the mice showed when they viewed the video for the tenth time. In 95 percent of cases, the AI was able to correctly predict which frame the mice were looking at within one second.
Visual sensations
Other AI systems that reconstruct images from brain signals work better if they are trained on brain measurements from the same mouse on which they make their predictions. To test whether this also applied to their AI, the researchers trained the system on brain measurements of individual mice. After that training, the AI predicted the viewed frames of film with an accuracy of only 50 to 75 percent. ‘Training this AI on data from multiple animals actually makes the predictions better. So you don’t have to train the AI on data from specific individuals,’ says Mathis.
By making connections between brain activity patterns and visual input, the system could eventually be used to find ways to induce visual sensations in people with visual impairments, says Mathis. “You can imagine a scenario where you can help someone with a visual impairment see the world in interesting ways, by inducing neural activity that gives them that visual sensation,” she says.