Awkward "AI Face-swapping" Techniques in Film and Television

08/07 2024 371

Since the AI-generated content controversy surrounding the novel-writing platform Fanxiaoshuo, AI has begun to infiltrate the realm of short film production.

China's first AI-generated original fantasy short film, "The Miraculous Mirror of Mountains and Seas," garnered 600,000 views for its concept trailer alone, and its first three episodes surpassed 10 million views on Kuaishou, even making its debut on the big screen.

While it still retains a distinct AI feel in terms of overall texture, movements, and aesthetics, the emergence of "The Miraculous Mirror of Mountains and Seas" signifies that low-budget video creations by small teams now have greater potential for commercialization.

However, some AI-generated short films still struggle to gain audience acceptance.

For instance, AI-generated versions of Jackie Chan and the show "The Outcast" have been met with mixed reactions, with some viewers preferring even the earlier seasons of "iPartment" over these AI-enhanced works due to their lack of naturalness.

Nonetheless, AI short films have arrived, driven by the need for cost-effectiveness and storytelling capabilities.

I. The Arrival of AI-Infused Short Films

In addition to "The Miraculous Mirror of Mountains and Seas: Breaking Waves," Douyin (the Chinese version of TikTok) also premiered the AI sci-fi short film "Sanxingdui: Revelations of the Future" on July 8, which has garnered over 38 million views to date.

Earlier this year, several companies attempted to export short films using "AI dubbing + face-swapping" technology, as reported by various media outlets.

As AI video technology advances rapidly, the market now boasts an array of AIGC-focused shorts and short films beyond "The Miraculous Mirror of Mountains and Seas" and "Sanxingdui: Revelations of the Future."

In June, CCTV's Film and Television Drama Documentary Center launched three AI short films on its platform: "Hero," "Love Never Ends," and "Fantasy Shop."

Image Source: Screenshot from "Sanxingdui: Revelations of the Future"

In the same field, Alibaba recently introduced a large model for film and television makeup and costumes.

Sci-fi blockbusters are notoriously expensive in terms of post-production special effects and pre-production processes, making AI a valuable tool for saving time and effort in scriptwriting, concept designing, and other preliminary stages.

Chen Kun, the director of "The Miraculous Mirror of Mountains and Seas: Breaking Waves," mentioned that even renowned directors like Zhang Yimou struggle to complete many CG effects due to budgetary constraints, highlighting a significant pain point in the traditional industry.

Therefore, both long and short video platforms, as well as internet giants aiming to expand their AI portfolios, are actively pursuing cost-effectiveness and efficiency gains through AI.

However, AI short films are still in their infancy, and current AI technology has yet to achieve a high level of control and precision in character consistency, scene coherence, character performance, and interactive movements.

This inevitably leads viewers to scrutinize AI short films, with most finding the lack of narrative coherence in complex storylines particularly awkward.

These shortcomings are inherently linked to the technology. While AI excels in data processing and pattern recognition, it falls short in creative generation and emotional expression.

A notable example is the facial expression issues in AI-generated content, as seen in the nearly 20-minute animation segment in "The Outcast," which was converted from live-action footage using AI technology.

Similarly, "The Outcast" and AI-generated Jackie Chan received criticism due to viewers' high expectations and their status as pioneering AI-infused film and television projects.

If the overall effect fails to meet viewers' expectations of naturalness, those who consume such content may feel dissatisfied.

AI's potential to empower short films is undeniable, whether for overseas expansion or content innovation. However, besides the lack of naturalness, AI short film creators also face challenges in premium content development, including insufficient innovation and content depth.

Especially in the context of AI short film exports, enhancing narrative creativity is more crucial than technological advancements alone.

II. Low-Cost Export with Numerous Hurdles

According to research by the Short Film Study Room, the production cost of exporting short films is relatively high, averaging $130,000 to $200,000 per film, which is 2-3 times more expensive than domestic productions (equivalent to RMB 400,000 to 600,000).

AI face-swapping significantly reduces these costs. Currently, the basic cost of AI face-swapping stands at approximately RMB 50,000 to 100,000 per film, with additional charges for lip-sync translation, bringing the total package price to approximately RMB 200,000 per film.

Assuming an overseas short film costs $150,000 to produce, domestically produced short films (costing around RMB 500,000) plus RMB 200,000 for AI face-swapping can save at least several hundred thousand yuan overall.

While human actors may not be replaced overnight, AI's impact could lead to a reduction in other job opportunities, such as replacing live actors with image licensing.

However, this solution is only temporary, and AI face-swapped short films are still far from being mature products ready for widespread overseas markets. As mentioned earlier, AI face-swapping risks facial distortions, requiring caution.

Furthermore, to enhance overseas viewers' willingness to pay, AI short films must adapt their overall narrative styles to different cultural contexts, avoiding the rigid transplantation of domestic storytelling approaches.

Exporting AI short films at this stage carries inherent risks, as domestic AI short film blockbusters have yet to emerge, making the overseas journey even more challenging.

Take "The Miraculous Mirror of Mountains and Seas: Breaking Waves" as an example. Among 220 Kuaishou short films analyzed by netizens, it ranked 18th in terms of average views per episode, surpassing approximately 92.07% of other films.

With 50 million views, "The Miraculous Mirror of Mountains and Seas: Breaking Waves" falls within the top 15% bracket, indicating its success among short films on short video platforms but still a long way from becoming a blockbuster.

Fortunately, domestic AI giants remain rational, focusing on technological advancements despite the allure of overseas expansion. Even as they aim to develop film and television content with AI, they prioritize technological development over immediate results.

Text generation models like ChatGPT 4, Bing, Wenxin Yiyan, and iFLYTEK Spark have achieved similar results in short film creation but still require extensive AI training.

In conclusion, with technological advancements, AI-generated film and television content will inevitably become mainstream, albeit gradually.

References:

Short Film Study Room: Short Film Exports and the AI Face-swapping Boom: Making Millions Daily?

Xinshang: Douyin and Kuaishou's AI Short Films Are Not Yet a New Frontier

Digital Life Cazike: After Analyzing 227 Short Films with AI, I Finally See the Future of Kuaishou's AI Short Films

Guixinren: He Used Three AIs to Create a Viral AI Short Film in 10 Days: "You Can Do It Too!"

Solemnly declare: the copyright of this article belongs to the original author. The reprinted article is only for the purpose of spreading more information. If the author's information is marked incorrectly, please contact us immediately to modify or delete it. Thank you.