APPLE FINAL CUT PRO: Apple Final Cut Pro X FCPX Debates FCP Legacy FCP Tutorials

Re: Catalina issues?

COW Forums : Color Grading

Respond to this post   •   Return to posts index   •   Read entire thread

David Lawrence
Re: Catalina issues?
on Mar 27, 2020 at 3:30:32 am
Last Edited By David Lawrence on Mar 27, 2020 at 3:36:40 am

[Oliver Peters] "Cool. David, what app(s) were you using to edit and composite these?"

Hi Oliver,

Here's an overview of both projects -

tl;dr version:

Mistika VR and Z CAM Wonderstitch for stitching
Mocha Pro for rig removal and roto
Neat Video plugin for noise reduction
Boris FX Continuum VR plugins for stereo compositing
Cinema 4D + Octane Render for 3D environment/character modeling
Poser Pro for character modeling
Adobe apps - Premiere and After Effects for keying, compositing, editing and finishing

Longer "sheltered in place and I'm bored" version:

Project 1) Wittgenstein's Garden

Stereoscopic 360 documentation of video artist Doug Hall's immersive sound sculpture at the Oliver Ranch Foundation in Geyserville, CA.

Doug's producer, my longtime friend and occasional collaborator, Starr Sutherland, invited me and my buddy Gary Yost to bring Gary's VR camera to the Oliver Ranch to shoot documentation.

Because the site is inaccessible to the general public, 360 VR is the perfect medium for conveying a sense of the environment and experience of the piece.

Gary's camera - a Z CAM V1 prototype (the only one of its kind in the world at that time) was in China needing repairs. So we borrowed its big brother, the Z CAM V1 Pro for our shoot in May of 2018.

I used Mistika VR for stitching, After Effects for compositing, Mocha Pro AE for rig removal, and Premiere Pro for the edit. The audio for the piece is second-order ambisonic with headlocked stereo ambience, mixed by Jim McKee, who was the lead sound designer on the piece itself.

By July of 2018 we had a three-minute demo for a curator we were talking to about possibly exhibiting in conjunction with a bigger show of Doug's work. The show didn't happen so we put the project on hold.

I returned to it this past February finishing rig removal and recutting video to Jim's new 9-minute sound mix.

Doug loves the finished piece and we hope to do a small exhibition of it at his San Francisco and New York galleries using Oculus Go headsets.

Project 2) Inside A Circle of Dreams - The Residents in 360

Fresh off Oliver Ranch, Starr started thinking about other 360 projects and invited us to bring the camera to small book release party The Residents were doing for Litquake in October 2018.
Starr's idea was - let's put the VR camera in front of the band and see what we get. No pressure or expectations, just experiment and have fun.

We shot with Gary's new production ready Z CAM V1. What's cool about this camera is its compact size and 31mm (~50% of average human eyes) interaxial lens distances means you can place this camera as close as 12-18 inches from your subject and still get perfect, very comfortable to view, stereo stitching.

We shot the whole set, using a Go.dingo rover to move the camera between songs.

I got the footage and went to work using these tools:

• stitching - Z CAM Wonderstitch - One nice thing about having a dual boot Hackintosh is I can run Windows 10 apps like Wonderstitch using full CUDA processing with the 1080Ti. (I've since replaced it with a Radeon VII for better stability and compatibility in macOS)
• stereo 360 rig removal - Mocha Pro AE plugin
• optical flow fixes - Mistika VR
• noise reduction - Neat Video plugin
• final compositing and mastering - After Effects

After I finished the first song, an intense, aggressive piece called DIE! DIE! DIE!, I sent it to Gary. He called me immediately totally excited. He said "you guys need to come over to my studio NOW and see this in the Vive Pro! It's unfreaking believable!"

Starr, Homer Flynn (president of The Cryptic Corporation) and I went to Gary's to look at what we had and yes, it was amazeballs. The band looked incredible. One of the artifacts of shooting close with the V1 is it tends to create an exaggerated hypo stereo effect where super close subjects look unnaturally big. Usually we want to avoid this but especially for this band, it looks great and makes you feel like they're playing right in front of you.

While the band looked awesome, the rest of the 360 sphere wasn't very interesting since it was just the audience and venue, mostly in the dark. What to do about that? Well, The Residents know and work with a lot of visual artists so what if we could get a couple of them to make some art/animation to *replace* the audience and clubspace?

We chose three songs from the set - DIE! DIE! DIE!, Teddybear (a cover of the Elvis tune), and MonkeyMan, a lyrical, 7-minute guitar-driven dreamscape. The idea was to have variety and a package that might be released as a VR EP. Then we started reaching out to artists, eventually finding three very talented, very different artists who all graciously volunteered to do artwork/animation for the project:

DIE! DIE! DIE! - Nick denBoer a super talented, wacky, 3D animator from Toronto Canada who's worked with bands like Flying Lotus and Deadmau5. Nick was a long time Residents fan and jumped at the chance to do a project with them.
Teddybear - Bill Domonkos, a fine artist, stereographer, and long time collaborator with The Residents.
MonkeyMan - Stéphane Blanquet, an internationally renowned artist/illustrator and another frequent collaborator with The Residents.

Each artist had a unique style and workflow:

Nick used Cinema 4D and Octane Render to craft a fully-realized 3D modeled, coliseum world where eyeball-headed victims meet their demise to a cheering crowd. The scenarios are comedic, ala road runner cartoons. Nick's 3D world blends seamlessly with the 3D video and at one point, the lead singer walks into it.

I did the roto for this sequence using Mocha Pro. It's 900 frames of organic, non-planer shapes with motion blur, in the dark, in 360 stereo. Very painful, but so worth it for the final effect. I'm a Mocha beta tester and worked directly with the Boris FX product manager and team fighting thru bugs and figuring out workarounds. Gotta say - the folks at Boris FX are the *best*. Their outstanding tools and support were integral to the success of both projects.

Bill took a totally different approach with Teddybear. Instead of removing the audience and venue, he decided to *keep* everything in the video and mess with it. He's a master stereographer and years ago figured out a way to use Poser to export 3D characters and objects so that they can be viewed in stereo 3D with a stereoscopic viewer.

His workflow is pure genius - he clones the model he's working on then positions the two models side by side in the scene and views them through slightly different camera views, fusing the stereo image in his head and adjusting the camera angle to get the right disparity. It works! I'm blown away by his technique and creativity.

I helped him choose a new iMac 5K and HTC Vive for the project. We developed a workflow to integrate his 3D models into the video using the Boris FX Continuum VR plugins and the native Adobe Immersive plugins. Using the Vive, he could dial in stereo z-plane settings viewing in real time as he worked with the plugins.


I could easily write an article on collaborating with Blanquet on MonkeyMan. (Tim, if you're reading this, hit me up!)

The man's a mad genius and his work is both deeply disturbing and beautifully sublime. He goes there. Here's video of one of his exhibitions to give you an idea of the range of his talent:

When we got word Blanquet was interested in the project I was beyond thrilled. This is the rare kind of project I love the most. But of course, there have to be challenges:

1) He doesn't speak English
2) He's in Paris (+8/9 hours time difference)
3) He's physically challenged
4) He's busy and has limited availability

Over email and Skype, Blanquet and his manager Antoine (who speaks fluent English) proposed sending us his "pantin" (puppets) to use as we wish with the video. You'll see them in the video above - they're the black sculptures attached to the wall. They look like bizarre Balinese shadow puppets and he's made over 400 of them, each one unique. He also has a series of "silhouette" paintings that reference their style.

Weeks go by and then one day a box arrives at my studio. It's filled with puppets. They are exquisite - with incredibly fine details that look like they would break if you breathe on them too hard.

Okay, we have puppets and now we needed a puppeteer. We found the perfect one with Christine Marie, an innovative shadow puppet genius who incorporates anaglyph 3D in her award winning shadow puppet performances.

We brought Christine Marie to the Bay Area to puppeteer the puppets in front of greenscreen in my studio. We didn't need shadows, we wanted movement of the puppets themselves. After two days shooting with a Blackmagic Pocket 4K camera, I had a hard drive full of BRAW puppet greenscreen footage.

I spent the next couple months working with this footage to clean it up, isolate and generate keys, and make loops of various motion sequences and gestures. This gave me a palette of movement for each puppet that I could work with in the piece. Here's a video contact sheet I made for Blanquet to give you an idea what they look like:

At this point, it was August 2019. I had animated assets and was finally ready to actually begin making the piece.

I won't go into actual details of making the piece here because that really deserves its own post. I will say that in the many years of projects I've worked on, I think this one pushed me the hardest and farthest, both technically and creatively, and with it's successful completion is one of the most satisfying. There was no script, no storyboard, and the only direction I had from Blanquet was this sketch:

I look at the finished piece now and honestly have no idea where it came from. Fishing in deep water is fun and mysterious. Like The Residents, all three videos are awesome and unique. There's nothing quite like them in either the world of of 360 or music videos.

As far as what happens next, like so many other projects (maybe even yours), the current health crisis puts both Oliver Ranch and The Residents VR in limbo for now. Our launch, publicity, and distribution plans are all on hold and will need to be rethought for the new world we come back to once things return to normal.

And it's not like there was much audience for VR 360 to begin with. There are many extraordinary cinematographic VR pieces out there and their audience is a tiny fraction of what's needed for a sustainable industry. This medium is still in its infancy. It'll be years before what I see as the biggest obstacle - the awkward VR headset itself - is refined to a point where there's a critical mass of audience and the chance of turning this medium into a viable business.

If any of you guys have an Oculus Quest, Go, or Rift, let me know your Oculus ID and I'll share private Oculus Media Studio links that will let you save them to view in your headset. We're not putting this stuff out on YouTube or Samsung VR because we really believe they have to be seen on a VR headset to be fully and properly experienced. It just won't work on your tablet of phone.

So that's what I've been up to the past year and a half. The SF Bay Area was the first in the country with a shelter in place order and we're doing a good job flattening the curve. For me, it's not all that different then being locked in the studio working on VR art projects. ☺

p.s. - Gary *loves* FCPX and uses it for all of his VR post. I tease him about it. He got a new Mac Pro and after some initial bumps, finally has it set up and working well. It really screams.

David Lawrence

facebook: /dlawrence
twitter: @dhl

Posts IndexRead Thread 

Current Message Thread:

© 2020 All Rights Reserved