Promptspiration Installment 3
Creative Fusion with image + video + music + text
In my post last Thursday, I made a joke about how it often feels like “a year’s worth of advancements in Generative AI happened last week”...well, this last week somehow felt more like several years.
Given there are more than enough articles flying around right now about the situation with OpenAI, I’m going to take this moment to focus on something that I find to be fun and joyful – experimenting with making short videos!
I decided to play around with combining Generative AI from Firefly, Runway, MusicLM, and ChatGPT to create 3 short videos complete with their own original soundtracks. I created each video by following these 5 steps:
Generate an image in Firefly text to image
Add motion to it using Runway
Create a soundtrack in MusicLM
Stitch these all together in iMovie
Come up with a fun name using my Viral Headliner custom GPT
The first video was inspired by my recent family trip to the California Academy of Sciences in which we spent a very long time staring at a lot of amazing fish. So, in honor of them, I present to you a piece I call The Mesmerizing World of Neon Seascape Wonders.
I started with the following prompt in Firefly to generate an image:
ocean fish, including an angel fish, clown fish, dragon fish, and other salt water creature, swimming in the vast ocean - psychedelic and hyper realistic
I then used the new Motion Brush feature in Runway to focus on making the big fish in the foreground move.
Finally, I created my soundtrack in MusicLM with the following prompt:
soundtrack to an underwater scene in a cartoon movie, mellow and slow with bubble sounds, slow synth vibes
And then used my custom GPT to come up with the name!
My next video was inspired by fall time in Canada, which is what I always think of when I hear, “Thanksgiving” (because in Canada this happens in October). This one is titled Autumn's Paintbrush: Fall's Fiery Hues.
I started with the following prompt in Firefly:
A forest of whiteoaks in the fall, colourful leaves in the trees and on the ground, with a gentle rolling mist swirling near the forest floor.
I then used this prompt in Runway:
subtle mist swirling around the forest floor ebbing in and out while the leaves rustle in the trees and on the ground
Followed by this one in MusicLM:
a rustic, heartfelt, and authentic sound, blending elements of folk, Americana, and indie genres
And then used my custom GPT to come up with the name!
While making the first two videos, my toddler decided to sit beside me and became absolutely enamored by what I was creating. I asked her what she wanted my last video to be about and she said “space”. The result is Galactic Odyssey - a Cosmic Kaleidoscope!
My initial Firefly prompt was:
Rocketship travelling through space with galaxies, planets, and stars all around
I then input the following into my Runway prompt:
a rocketship flying through space, passing by planets and galaxies
Followed by this as the inspiration for MusicLM:
classical and orchestral pieces to convey the grandeur, mystery, and sometimes the eeriness of space and the future
And then used my custom GPT to come up with the name!