AI Cinematography Research: Where Is My Soul?
Timeline
10 Weeks
My Role
Research
Cinematography
Editing
Team
2 researchers
Process
Research
Storyboarding
Filming
Editing
Tools
Runway AI
DaVinci Resolve
Overview
I was tasked with producing and filming a 2 - 10 minute short film that reckons with artificial intelligence. Over the course of 10 weeks I conducted research which included analyzing and discussing previous texts, experimenting with video generation technology, filming content, and editing.
I primarily worked with Runway AI which is a video generation technology that uses artificial intelligence to assist users in generating, editing, and transforming content, such as videos and images. I primarily worked with Runway AI's automated text-to-video generation and video-to-video generation.
After initial brainstorming, me and my research partner Kamden Cykler settled on creating a short film that analyzes the presence of "soul" in artificial intelligence and machine learning.
This research group was led by PhD Student Brett Halperin with guidance from Associate Professor Daniela Rosner.
“Where Is My Soul?”
Synopsis
Our film introduces a person asking an AI questions about the limitations of AI and the existence of its soul, getting frustrated at the answers. Finally, the person asks the AI to show them its soul through music. The AI shows them two versions of the world. The first is a clearly artificial and unsettling view of a person sitting in front of a window with repetitive music. The second version is interspersed with clips of the real world. The film ends with the person asking the AI if what they saw was real and leaves the audience wondering if what they saw was generated or genuine.
Analysis
The diegesis that we built in our film appears to be the same as our world, but towards the end of the film, it becomes unclear what parts of the world were generated by the AI and what parts are “real.” We used framing in our film by positioning our character silhouetted in a window frame. Our use of color was also intentional. We shot our film in low lighting, at night and in the rain. This cast our film in shades of blues, blacks and grays. This enhances the ominous and thought-provoking tone of our film. We used several fade-ins from black, and mostly hard cuts. We developed mise-en-scene with acting, color and low-key lighting. At the beginning of the film, we used a blank, high-key lit screen, reminiscent of a typing platform along with the traditional typing signifiers of blinking cursors and keyboard noises to show distinct differences between the online AI-controlled areas and the low-lit real world. As the film progressed, we combined framing shots with high-key lighting to show the merging and uncertainty of the boundaries between what was AI generated and what was real.
AI Usages
During pre-production of our film, not much AI techniques were used as we were both learning and exploring Runway AI. During production, we used Runway AI to generate a 13 second video clip. To do this, we screen-shotted a frame of the footage that we had filmed previously and entered this into Runway’s image+text to video feature. We prompted Runway with the exact text: “add lots of movement, person sitting at piano, in front of a window, make it creepy, and scary, and disturbing.” After viewing a few outcomes and adjusting the wording to reach this prompt, we settled on a result. We intentionally asked Runway to make the clip surreal and unsettling because we had noticed during our explorations that Runway often generates clips that don’t look exactly the way you intend. This allowed us to generate a clip that served our purpose without spending too much time waiting for the output that we wanted. We used Runway’s “extend from video” feature to increase our clip to 13 seconds.
During post-production, we used AI to generate a section of the sound for our film. To do this we asked Chat GPT to write a version of the song “Where Is My Mind?” by The Pixies. Chat GPT produced altered lyrics as well as simple chords. Chat GPT wrote ABC notation for these chords, and we were able to convert this notation into a MIDI file. This was after a failed attempt to get ChatGPT to write a more complex song with both melodic and harmonic components. The final song consisted of two repeating notes with no pattern variation. Because of this, we were unable to use an AI-generated song for both versions of the world shown by our in-video AI “character” because real AI couldn’t create something with enough emotional body, including variation, as we wanted to portray our story.
We chose to only use AI to generate a small portion of our film because we wanted to juxtapose real art with the artificial “soulless” art that AI creates. One maneuver that we used with Runway was generating consecutive clips. After generating one clip that we liked, clicking the ‘expand clip’ option to add another four seconds to the clip was effective to keep characters the same and maintain the mood. As described above, we also prompted Runway to add motion and zoom to the clip with the camera control options.
Reflection
Bias & Ethics
One thing that was surprising was both times we asked Runway to create a video, it did not create a white male even without gender or ethnicity specified. The first ask was a text to video feature with the prompt “a person playing a guitar.” Runway provided an Asian man playing a guitar. The second video was created from a screenshot of our raw footage - a silhouette of a person of indeterminate gender playing the piano. Runway provided a Hispanic and then a Black woman (they morphed into each other). This was of note, as there have been a lot of news articles about AI presenting predominantly white, conservative men when asked to present a “person” or using pejorative or basic stereotypes when asked to create people exemplifying a trait not tied to an ethnicity or race.
Ethically, the production of AI-generated videos felt like cheating. Cheating artists out of opportunities and cheating the system by passing off the art as something you can claim ownership over. Can someone claim to have created something AI-generated?
Lessons Learned
Overall, there is still a long way for AI to go. Whether it was the length of videos, human appendages, camera motion or more, there are many ways that AI video generation can be improved. The biggest learning curve of this project was the current limitations of the machine learning software. The most important lesson to learn was that AI is really good at specific tasks and figuring out how to incorporate those specific tasks in one’s work.