ADOBE PREMIERE PRO: Tutorials Forum Articles Creative Cloud Debate

4k edit: storage solutions

COW Forums : Adobe Premiere Pro

<< PREVIOUS   •   FAQ   •   VIEW ALL   •   PRINT   •   NEXT >>
Gabriel Sanchez
4k edit: storage solutions
on Mar 18, 2015 at 11:16:12 am

Hi all!
I´m updating my workstation for the next upcoming proyect, a 100min. documentary film for theaters shot in 4k (Raw .mxf from sony FS700).
The production company told me yesterday that i´ll need to handle 120 TB of 4k footage. Wow! That´s too much for my machine capability. And i´m not counting storage need if i make 1080p proxies, although i´m planning in doing the job with raw material in PPro CC2014 which i´ve tested and worked fine with no need of proxies.
My PC only have 3 free slots for upgrading my drive setup, and using them only will rise my internal storage capability to 6 TB more, 18 tb in total.
So how could i handle this huge archive? I can only think in external Raid devices, but usb 3.0 connection could be slow for footage drive working with 4k video, what do you think? Is there any solution for this crazy storage needs?
Regards


Return to posts index

Tero Ahlfors
Re: 4k edit: storage solutions
on Mar 18, 2015 at 11:58:42 am

[Gabriel Sanchez] "Is there any solution for this crazy storage needs? "

Of course there is but do you have the money for it? A NAS would be a good start.

[Gabriel Sanchez] "but usb 3.0 connection could be slow for footage drive working with 4k video,"

A small USB 3 RAID should handle compressed 4K like the MXF files from Sony cameras. When you're dealing with uncompressed 4K (film scans, DPX/TIFF sequences) you'll need some super hardcore drive setup to even play the files back in realtime.


Return to posts index

David Roth Weiss
Re: 4k edit: storage solutions
on Mar 18, 2015 at 12:12:54 pm

He does not require a NAS unless he plans to share the storage with other users, and he would most likely require expensive 10GB connectivity to handle the 4K material. See my initial response for direct attached storage, which is s much less expensive solution because it does not require an expensive server.

David Roth Weiss
Director/Editor/Colorist
David Weiss Productions


David is a Creative COW contributing editor and a forum host of the Apple Final Cut Pro forum.


Return to posts index


Tero Ahlfors
Re: 4k edit: storage solutions
on Mar 18, 2015 at 1:11:14 pm

[David Roth Weiss] "He does not require a NAS unless he plans to share the storage with other users"

Quite true, I just thought that if one is getting a 120 TB storage it wouldn't be just for one user. It will be pretty goshdarn expensive either way.


Return to posts index

David Roth Weiss
Re: 4k edit: storage solutions
on Mar 18, 2015 at 2:58:53 pm

Expensive indeed, but much more expensive when adding a server to the equation.

David Roth Weiss
Director/Editor/Colorist
David Weiss Productions


David is a Creative COW contributing editor and a forum host of the Apple Final Cut Pro forum.


Return to posts index

David Roth Weiss
Re: 4k edit: storage solutions
on Mar 18, 2015 at 12:06:06 pm

To accommodate a project of120GB plus overhead (you can't fill the drives completely), renders, etc. you are going to require at least two 16-drive SAS enclosures, most likely filled with the new 6tb SATA drives, and an Areca or ATTO SAS controller card. Since it sounds as if storage is not your area of expertise, you should definteky call one of the many storage companies advertising here, as you should not try a build it yourself solution.

David Roth Weiss
Director/Editor/Colorist
David Weiss Productions


David is a Creative COW contributing editor and a forum host of the Apple Final Cut Pro forum.


Return to posts index


Gabriel Sanchez
Re: 4k edit: storage solutions
on Mar 18, 2015 at 7:23:07 pm

Thanks all for your valuable advices.
Definitely looking for direct attached storage will be my next step, and those 16-drive SAS enclosures sound good. Also I´ve been reading about the ATTO SAS controller which is used for several storage solutions, looks awsome.
I noticed that will need extra space for renders, media cache, animations, etc.. but even i´ll need double space for the footage backups...
Also, i´ll need extra storage for the final file, (¿DPX?, don´t know as i´ve never worked in 4k and never delivered a file for theatres), so here comes another question: how much storage space will i need for a 100 min. DPX file? And how could i deliver such huge file out of my computer?
Regards


Return to posts index

David Roth Weiss
Re: 4k edit: storage solutions
on Mar 18, 2015 at 7:30:50 pm

I prefer Areca cards these days... And, for delivery, you deliver on hard drives these days - FireWire 800, eSATA, USB 3.

David Roth Weiss
Director/Editor/Colorist
David Weiss Productions


David is a Creative COW contributing editor and a forum host of the Apple Final Cut Pro forum.


Return to posts index

Tero Ahlfors
Re: 4k edit: storage solutions
on Mar 19, 2015 at 5:15:07 am

[Gabriel Sanchez] "a file for theatres"

http://en.wikipedia.org/wiki/Digital_Cinema_Package


Return to posts index


walter biscardi
Re: 4k edit: storage solutions
on Mar 18, 2015 at 4:20:56 pm
Last Edited By walter biscardi on Mar 18, 2015 at 4:21:34 pm

[Gabriel Sanchez] "The production company told me yesterday that i´ll need to handle 120 TB of 4k footage. Wow! That´s too much for my machine capability. And i´m not counting storage need if i make 1080p proxies, although i´m planning in doing the job with raw material in PPro CC2014 which i´ve tested and worked fine with no need of proxies.
My PC only have 3 free slots for upgrading my drive setup, and using them only will rise my internal storage capability to 6 TB more, 18 tb in total."


You'll need a serious external RAID, probably multiple 16 bay models to get that much storage. You won't be connecting via USB 3, it'll be a more robust connection like SAS or possibly even 10gig E which we use here. Look at Studio Network Solutions, Small Tree, G-Technology and other large volume RAID suppliers. In short, it's rather expensive to set up a reliable external RAID for what you're looking for and remember if they tell you 120TB for just the raw media, you'll want to double that amount to ensure you have enough for renders, graphics, animations, audio, and whatever else is associated with the project.

We run approx. 100TB here all via 10gig E through a system from Small Tree, though we're about to install a new system from Studio Network Solutions which will add another 48 - 64TB to our configuration.

Walter Biscardi, Jr.
Editor, Colorist, Director, Writer, Consultant, Author, Chef.
HD Post and Production
Biscardi Creative Media

Craft and Career Advice & Training from real Working Creative Professionals

Blog Twitter Facebook


Return to posts index

Alex Udell
Re: 4k edit: storage solutions
on Mar 18, 2015 at 10:25:11 pm

hey walter...

I've been eyeballing SNS Sharebrowser. Have you had a chance to try it?

Alex Udell
Editing, Motion Graphics, and Visual FX


Return to posts index

walter biscardi
Re: 4k edit: storage solutions
on Mar 18, 2015 at 10:45:33 pm

We've had the online demo with the team at SNS and it's quite powerful. We're scheduled to have a system installed at our shop soon, so once we have it online I can report back.

Walter Biscardi, Jr.
Editor, Colorist, Director, Writer, Consultant, Author, Chef.
HD Post and Production
Biscardi Creative Media

Craft and Career Advice & Training from real Working Creative Professionals

Blog Twitter Facebook


Return to posts index


Ht Davis
Re: 4k edit: storage solutions
on Mar 30, 2015 at 10:32:36 am

They're talking to you in total storage for raw footage, not in total storage for working footage. Big difference. Working footage should be internal. RAW should not if it is too big to store internal. You will need some SAS.

You will need ram. You will need proxies, you will need to edit in your output resolution, but interpret your proxies to that for your previews. Use a server farm to build your 240tb for footage and have them catalogue it to that location (this is a logistics bomb).
You will need a fiber based connection of 10-20gb/s. You will also need to be able to run the video out to proxies.

How to:
1. set servers up with AE and AME, set up watch folders\scripts for auto copying and proxy creation
2. Set up your internal system with the max it will hold with 1-2tb drives in a RAID formation and use this for cacheing
3. Add a SAS with an exterior shell casing that can handle up to 30th RAID for holding working files
4. Make a disk image big enough to hold the proxies, project and previews, store it on the External SAS and begin building the folder structure on that. Every night, back up the changes to servers by imaging, and every two weeks have changes burn to discs (look into acronis software solutions). This way, you can move the external SAS and the disk image anywhere, and your entire project is contained. You only need to relink the original media and render effects when you cross system platforms.

When you finish with the video itself, you'll need to move that entire project to the servers to output your final work. You'll probably need more like 300-400tb. However, you'll be able to work on your main editing system while the servers are exporting the final file, so you'll still be able to work. You may want to look into compressor and an apple server setup... ...this would give you the ability to not only share storage but processing cores for compression of the final output. That final output will be placed on a disc or a hard drive for playing in theaters, and it will be compressed formatting. You get better quality and compression in Compressor.


Return to posts index

Gabriel Sanchez
Re: 4k edit: storage solutions
on Mar 30, 2015 at 4:10:48 pm

When they told me i´ll need 120 TB for storage they meant this will be in the case they shot with Odyssey attached to the FS700, which produces CinemaDNG files in Raw mode. The cheapest solution is to record in 4k ProRes HQ 4:2:2 through the Odyssey, which will produce about 5 times or more smaller files, but then we won´t have a Digital Negative.
So finally we´ll be shooting with the sony AXS-R5 instead, producing Sony Raw mxf files, which are 2,5 times smaller than the Odyssey´s Cinema DNGs, and very much easier to handle in PPro.
This means that the storage needed will be about 48-50 TB, not 120.

I´ve talked to a people expert in storage solutions, and they suggested a 64Tb NAS for de Digital Negative backup, that can be moved to another location with all the original files plus the exported DPX sequence or Cineform file of the final edit for another people to make the DCP. We think about a NAS because it can be connected to any other machine with no specific card installed.
By the way, thanks Walter for making me notice what i haven´t (forget to think about the backup, whitch obviously doubles the storage needs!)

For the work, i´ve been told (as David said) to use one Direct Attached Storage cabin of 16x4Tb Raid5, through an Areca Fiber Card (PCI Express), with all the original footage, and other DAS same storage capacity (or maybe half the storage needed for original footage) through another Areca Card for the proyect assets, animations, exports, previews, etc.. Will this be enough?

Also, i´ll have an internal 6x3TB Raid0 for media cache, i have no more free slots for making a raid with more disks (other slots in my mobo are taken by a bunch of SSD drives).

Sony Raw MXFs play smooth in PPro, although i haven´t made any upgrades to my workstation yet, so i think i wont make proxies and going to work directly with the raw files. Do you think i could be wrong? I´ll work with Speedgrade, Red Giant Mojo, and eventually using Twitch and some Optical Flares.
Anyway i can always convert all the raw files in Cineform files or DNxHR HQ to make the edit if i encounter any problem working with the original Raw files.

I´m planning to put another graphics card for the 4k monitoring (Dell P2715Q UHD external monitor through a GTX980) and use my older GTX780Ti for MPE in GPU mode and for program interface in an extended desktop of two 1080p monitors. As far as i know PPro CC2014 won´t use both for MPE playback, but he´ll use them both for encoding/exporting, which would be of help.

Also i plan to upgrade my RAM from 32GB to 64GB.

What do you think? Will it be enough?
Thank you all very much for your inputs.


Return to posts index

Ht Davis
Re: 4k edit: storage solutions
on Apr 3, 2015 at 10:35:08 am

Heh... ...Must be nice... ...Yeah, that will do it. If you have 2xgfx and CUDA capable windows system, you should fly right through. Just remember to get 2gb gfx cards or higher for 4k throughput and make sure their data path is wide enough for the playback of the video. Alternatively, render out a preview first thing. That will give you iframe playback speed in premiere but let you edit away at similar to native quality. Ram to 64 is probably a bit much, but you might be glad you have it when multitasking your edits.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]