Microsoft Research Asia has introduced a new artificial intelligence model for creating realistic speaking faces called VASA. article published. The VASA-1 model can produce realistic videos based on a single static image and a speech audio clip.
VASA-1’s results are impressive and surpass all previous tools that use generative AI to produce realistic deepfakes. A particular highlight of the model is its overall ability to mimic natural facial expressions, a wide range of emotions, and the ability to lip-sync with very little artificiality. Still, the researchers admit that the model, like all other models, still struggles with non-rigid elements like hair.
Microsoft says the technical cornerstone of the model is an innovative holistic facial dynamics and head motion rendering model that operates on an expressive and resolved facial latent space. VASA-1 can also deliver real-time efficiency.
The method produces 512×512 video frames at 45 FPS in offline batch processing mode and can support up to 40 fps in online streaming mode with a prior latency of only 170 ms when evaluated on a desktop with a single NVIDIA RTX 4090 GPU.
One of the pros of VASA-1 is the easy of use. Watch this real-time demonstration: pic.twitter.com/QvHnpHVx8e
— Martin Hodás (@Hody_MH11) April 18, 2024
The vehicle, based on the new model, is described as very easy to use and offers the ability to control optional signals as a condition. This means the user can adjust the main eye gaze direction, head distance, and emotional offsets. VASA-1 can also process non-realistic input such as art and bring images to life.
The model can also make photos sing, rap or speak in languages other than English. A funny clip of Mona Lisa rapping was even released for this.
Rapping Mona Lisa. Not sure I wanted to see this… pic.twitter.com/1B8sgm5qQ9
— Martin Hodás (@Hody_MH11) April 18, 2024
Microsoft acknowledges the possibility of abuse. In this sense, the company does not intend to publish an online demo, API, product, additional implementation details or any related offerings until it is confident that the technology will be used responsibly and in accordance with appropriate regulations.
