Sunday, July 14, 2024
HomeNewsMeta develops AI models for realistic sound experience in VR 

Meta develops AI models for realistic sound experience in VR 

The AI models are designed to push the industry faster towards the immersive reality experience, said Meta in a statement.

Meta has developed three new AI models – Visual-Acoustic Matching, Visually-Informed Dereverberation, and VisualVoice – to make the sound more realistic in mixed and virtual reality (VR) experiences. 

The three AI models focus on human speech and sound in the video. They are designed to push the industry faster toward the immersive reality, the company said in a statement.

The AI models were built in collaboration with the University of Texas at Austin. The company is also making the audio-visual understanding models open to developers.

Read More: Microsoft Uses AI To Improve Audio And Video Quality On Microsoft Teams

Acoustics play a role in how sound will be experienced in the metaverse. According to Meta’s AI researchers and audio specialists, AI will be core to delivering realistic sound quality.

AViTAR, the self-supervised Visual-Acoustic Matching model, adjusts audio to match the space of a target image.

Despite the lack of acoustically mismatched audio and unlabelled data, the self-supervised training objective learns acoustic matching from in-the-wild web videos, said Meta. VisualVoice learns by learning visual and auditory cues from unlabelled videos to achieve audio-visual speech separation.

VisualVoice generalizes well to the challenging real-world videos of diverse scenarios, said Meta AI researchers.

For instance, consider attending a group meeting in the metaverse with colleagues worldwide. However, instead of people having fewer conversations and talking over one another, the acoustics and reverberation would adjust accordingly as they join smaller groups and moved around the virtual spaces. 

Subscribe to our newsletter

Subscribe and never miss out on such trending AI-related articles.

We will never sell your data

Join our WhatsApp Channel and Discord Server to be a part of an engaging community.

Sahil Pawar
Sahil Pawar
I am a graduate with a bachelor's degree in statistics, mathematics, and physics. I have been working as a content writer for almost 3 years and have written for a plethora of domains. Besides, I have a vested interest in fashion and music.


Please enter your comment!
Please enter your name here

Most Popular