Reflections on the Tokyo AI Film Festival in Japan, what effects can ChatGPT Sora not achieve?

The Japan AI Film Festival (AIFJ) was grandly held at the Tokyo Innovation Base (TIB) in Yurakucho, Tokyo, providing audiences with a rare experience to witness the innovative energy sparked by the fusion of artificial intelligence and film art. This year's film festival screened over 70 short films created using AI technology, spanning diverse forms including narrative films, animations, documentaries, and experimental films. Here, we introduce a noteworthy work that received an official special mention at the festival, To The Bones. This short film is also available on YouTube, where director Claudio Sa explains in detail on his YouTube channel how he used AI tools like ChatGPT Sora, Midjourney, and Runway in combination with traditional Photoshop and Premiere to create a 3D animation.

Introduction to To The Bones

To The Bones is directed by Claudio Sa, who is the founder and animation director of Pistachio Studios and has over a decade of experience in animation production. This short film is his attempt to create an animated film using ChatGPT Sora and other AI tools combined with traditional software like Photoshop and Premiere.

To The Bones Story Introduction

A weary traveler encounters a little dwarf, who gives him a sparkling red gemstone. After carrying the gemstone, he develops insatiable desires and ultimately pays the price. This film is inspired by a fairy tale by Jordan Peterson.

To The Bones 30-Day AI Animation Production Challenge

To The Bones director challenged himself to create a 3D animated film within 30 days on a very limited budget using AI tools. The director did not want the film to look like it was generated by artificial intelligence, hoping to create a work that is more handcrafted, coherent, and story-focused through initial sketch storyboards and his filmmaking style. This 6-minute short film utilized tools including OpenAI's newly launched Sora ChatGPT Open Art platform, Runway, as well as traditional software like Photoshop, Premiere Pro, and After Effects.

ChatGPT + other AI creation tools for making animated videos

First, Claudio used Photoshop to draw a sketch of the protagonist's shape, and generated the background and clothing using Midjourney. Claudio uploaded the hand-drawn protagonist and story draft to ChatGPT, giving the AI some text prompts (prompt) to make the plot details more specific. After realizing the story lacked tension, Claudio added other characters and let ChatGPT perform its function to simplify the script work and add more plot.

ChatGPT Sora 360 feature allows characters to turn naturally.

Opening Sora's green screen feature allows for different backgrounds to be replaced. Claudio uses the 360-degree rotation feature to allow characters to turn. In animated films, if characters cannot move their bodies naturally, they can appear stiff and strange. Many films at the film festival could not overcome this difficulty, but Claudio, based on years of animation production experience, lets AI complete part of the work. ChatGPT turns the story draft into a script, uses ElevenLabs for character voiceovers, and then adds as many vivid frames as possible to make the storyline and visuals more natural and engaging.

Things that OpenAI ChatGPT Sora cannot do

Sora can be understood as a high-level food processor like a pressure cooker or an oven. If food is processed and then put inside, the final product will turn into a delicious dish more quickly. The success of To The Bones lies in the use of AI for both “pre-production” and “post-production”. OpenArt generates storylines, and Claudio brings the scenes back to Photoshop to process each frame. He particularly recommends Runway and OpenArt for generating drafts and images, while also training OpenAI to generate more images. However, the biggest challenge in making animations with AI is “narrative coherence”. For animated films with plots, it is essential to connect the story and scenes one after another. Creating an animated film is not difficult, but telling a compelling story relies on strong visual effects and plot twists. At this point, only those with a background in film production and scriptwriting can solve these technical challenges. Many plot twists are still beyond the predictive capabilities of existing AI. This brings to mind the recent announcement from Japan about solving a case that had been unsolved for over twenty years. A housewife was murdered at home, and the police had no clues regarding the motive or the perpetrator. Even with DNA science, they could not find the murderer. The ultimate motive for the crime turned out to be the killer's simple jealousy over the woman receiving the affection of a man they admired. This complexity of human brain function and artistic talent cannot be completely replaced by artificial intelligence. To have a story that twists and turns while still being logical requires the human brain and human thought processes, which machines will never be able to fully predict. For now, Hollywood screenwriters still have work.

Reference:

To The Bones

I Made a Film With AI. This Is What Happened

To The Bones Tools Used:

ChatGPT Sora by OpenAI

AI Assistant Kling

Script Runway

AI-generated voiceover Elevenlabs

This article is a review of the AI Film Festival in Tokyo, Japan. What effects can ChatGPT Sora not achieve? It first appeared in Chain News ABMedia.