Microsoft: New AI Patent Enables Real-Time Speech Modification and Lip Synchronization

09:05

Microsoft has patented an artificial intelligence technology designed to modify what people say in video recordings and adjust their lip movements accordingly, potentially opening new possibilities for real-time translation and audiovisual editing.

The technology, described in a patent application titled Enhancing Video Streams Using Artificial Intelligence, could transform the way audiences consume digital content. However, its potential applications also raise important questions about the reliability of audiovisual information in an era increasingly shaped by AI-generated media.

An AI System That Can Rewrite Speech on Screen

The patented system is designed to intervene simultaneously in a video's audio and visual components. By processing both elements, the technology could replace a speaker's original words while modifying their lip movements to match the newly generated speech.

The process is reportedly expected to operate with a delay ranging from one to ten seconds, allowing modifications to take place during an ongoing video stream.

Beyond changing spoken words, the technology could also improve the overall quality of recorded conversations. Its potential functions include removing verbal hesitations, correcting pronunciation mistakes, filtering offensive expressions and reducing unwanted background noise.

Such capabilities could make digital presentations, interviews and online communications more polished without requiring extensive post-production work.

Personalized Content and Real-Time Translation

Another notable aspect of Microsoft's patented technology is its potential to tailor spoken content to individual viewers.

The system could take into account different user preferences, including language, age and geographical location. It might also adjust informal expressions or unfamiliar vocabulary to make conversations more accessible to specific audiences.

Real-time translation represents another particularly promising application. Instead of simply adding subtitles or replacing the original soundtrack, the technology could generate translated speech while synchronizing the speaker's lip movements with the new language.

This approach could significantly change the experience of watching international conferences, online meetings, interviews and audiovisual productions.

It could also disrupt traditional approaches to dubbing and translation by reducing the technical barriers associated with adapting content for audiences speaking different languages.

Deepfake Concerns and the Challenge of Authenticity

Despite its potential benefits, the technology raises concerns about the growing difficulty of distinguishing authentic recordings from AI-modified content.

The ability to alter someone's statements while preserving the appearance of a natural conversation could create opportunities for misleading audiences. Viewers might hear words that were never actually spoken by the person appearing on screen, without immediately recognizing that the original recording has been modified.

This possibility adds another dimension to the ongoing debate surrounding deepfakes, digital manipulation and misinformation.

One feature outlined in the patent is the preservation of an original, unmodified version of the video stream. This could provide a reference against which modified content might be compared.

The technology is also described as potentially compatible with several devices, including smartphones, tablets, cameras and smart glasses.

A Patent Does Not Mean an Imminent Product Launch

Although the patent offers insight into Microsoft's exploration of advanced AI applications, its registration does not necessarily mean that the company intends to release a commercial product.

Microsoft has not announced a deployment schedule for the technology. Furthermore, the patent description does not establish how viewers would be informed that a video has been altered or how the authenticity of modified content would be verified.

For now, the project illustrates both the opportunities and the risks associated with increasingly sophisticated AI systems capable of transforming audiovisual communication.

As these technologies advance, transparency and reliable authentication mechanisms are likely to become essential to maintaining public trust in digital media.