FORUMS: list search recent posts

4k 3D Feature - Have to rethink shared storage

COW Forums : Storage & Archiving

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Andrew Scholotiuk
4k 3D Feature - Have to rethink shared storage
on Nov 19, 2014 at 5:28:04 pm

We are filming a low budget 3D Feature in January with a tight delivery. I won't get into the whole workflow here but my simple question is this. Is there anyway to have a top of the line MacPro act as a iSCSI server as well as the editor? The main reason is that we need the MacPro to have full bandwidth to the files and would lose to much if we had to connect to it via 10Gb ethernet. I'd like to use a 32TB Promise R8 via Thunder2.

Camera records 4K CinemaDNG at 16GB/min

But if we could setup this sever, we could connect a iMac to it for the off line cut (ProRes 4K LT 2D) and the MacPro doing some basic composite work in AE and stereo correction. Then when we go to online and colour grade, we turn off the server and just operate on the MacPro with direct storage.

The simplest way to do this would be to setup file sharing with OSX, but I know the SMB connection isn't fast enough to handle this type of footage.

Many Thanks!

Andrew Scholotiuk
12pt Media
40 Below & Falling 3D


Return to posts index

Nate Cooper
Re: 4k 3D Feature - Have to rethink shared storage
on Nov 19, 2014 at 7:43:37 pm

I'd be surprised if this works. I think you'd be asking too much of the processors. AE compositing and stereoscopic work will eat up the resources of the Mac Pro, especially with multiple streams of 4K DNG.

You'd likely need a dedicated server if you want to do sharing. That said, keep us posted on your results.

Best,

Nate Cooper
http://www.promax.com
nate.cooper@promax.com


Return to posts index

Bob Zelin
Re: 4k 3D Feature - Have to rethink shared storage
on Nov 20, 2014 at 1:13:18 am

Hi Andrew -
I hate these kinds of posts (no offense to you personally). It's basically saying "we have no money, but we want to do the impossible". Let me tell you about real life, and neither Nate or myself represent AVID. When you look at the unlimited budget big Hollywood Features, they use AVID. And AVID can only do up to DNxHD220 which is 30 MB/sec (equivalent to Apple ProRes422HQ). And when they need to CONFORM to the "big stuff" that you want to do, they conform on a Quantel Pablo Rio, or Clipster, with NO SHARED STORAGE, but they EDIT on AVID Media Composer at LOWER RESOLUTIONS (usually DNxHD36, which is about 14 MB/sec, which any shared storage system can do). And this is if you are George Lucas, etc. with big bucks.

Now we come to you.
Your Promise R8 will do about 800 - 900 MB/sec. This is plenty for
even RED Epic Dragon 6K footage at 8:1 compression. But you want SHARED STORAGE. How many clients ? Exactly what is your question - how can you do it without spending ANY MONEY, and just get free advice from Creative Cow ? I can assure you that Nate at ProMax has a solution for you, and so do I. And so does Facilis. And so does EditShare. And so does GB Labs, and so does Small Tree. And so does Apace Systems. And so does Archion. And so does AVID. And so does Neil at LumaForge. BUT YOU ARE GOING TO PAY US. It doesn't need to be me. And it doesn't need to be Nate. But it needs to be SOMEONE that has experience in doing this.

Do your research, and you will find the answer to your problem. What you wont' find is a free solution, that will cost you $1000 bucks, and some questions on Creative Cow (although all the advertisers on Creative Cow can help you).

10GbE Ethernet will give you all the bandwidth you need.

bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index


Andrew Scholotiuk
Re: 4k 3D Feature - Have to rethink shared storage
on Nov 20, 2014 at 1:31:21 am

Hey Bob,

I was hoping to hear from you. I completely understand your frustration, I've been reading on these posts since small tree started with their bare bones iSCSI solution. And on this project alone we have been researching post workflows for the last 3 months. I'm not looking for a free solution, we are low budget not no-budget. I'd like to keep the system under $14K. I am in talks with SNS & Small Tree on their products which are great. But as I mentioned I'm trying to do something a little different. I need 2 systems with offline editing capabilities, and one system with a huge pipeline for the DNG. I'm not against buying a dedicated server, my only concern is that the Online Suite it won't be able to access the media fast enough.

Perhaps, I'll rephrase the question.
1. If I wanted to share storage between 3 computers, where the MacPro 12core is connected to the R8 can it handle acting as the server and editing at the same time?
2. SNS looks like it has the software to accommodate this, I'm a little confuses by what software packages I'll need.
3. Am I correct to assume that if I connect 2 computers together via 10GbE that the typical SMB protocol will have less then ideal transfer speeds/lag time. I hear the new SMB with Mavericks has had a massive improvement however.

Your time and Advice is greatly appreciated.


Return to posts index

Bob Zelin
Re: 4k 3D Feature - Have to rethink shared storage
on Nov 21, 2014 at 1:04:41 pm

Perhaps, I'll rephrase the question.
1. If I wanted to share storage between 3 computers, where the MacPro 12core is connected to the R8 can it handle acting as the server and editing at the same time?

REPLY - NO. Your server must be a dedicated server. You can't edit on the server while it is acting as a file server. Everything will choke. But what is the big deal ? A used Mac Pro from ebay for $1000 bucks ?

2. SNS looks like it has the software to accommodate this, I'm a little confuses by what software packages I'll need.

REPLY - don't be confused. SNS gives you a complete TURNKEY system, with everything you need. The small SNS Prodigy is about $5000. Is that too much money ? I thought you had $14,000 ! You don't need to think, just call SNS, Small Tree, or any of the other brands you see discussed here on Cow, and these companies will do it all for you. You don't need to be confused. Everything will be handled for you, as long as you hand over your money to them.

3. Am I correct to assume that if I connect 2 computers together via 10GbE that the typical SMB protocol will have less then ideal transfer speeds/lag time. I hear the new SMB with Mavericks has had a massive improvement however.

REPLY - all the companies you mentioned offer 10GbE connection from server to client. SMB, AFP, and NFS will all work, and yes, SMB on a Mac with Mavericks or Yosemite works quite wonderfully. The most important thing is that you look at the ads on Creative Cow, look at the forums, and buy a solution that you know is going to work. Stop thinking so much, and pick up the phone, and say "I have $14,000, what can you give me". People will bend over backwards to help you.
Doing 4K over a 10G connection is nothing.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Simon Blackledge
Re: 4k 3D Feature - Have to rethink shared storage
on Nov 28, 2014 at 7:38:30 pm

Your thunderbolt Promise will choke

It'll take time to get unto speed every time you hit play if shared out



Transcode all your rushes to 1080 prores proxy and put on a smaller pegasus and cut off that locally.
Theres no reason for the proxy to be on the same machine as the master rushes. (which you'll have a second copy of on a similar raid incase the promise fails )

When you hit render in anything on the Mac Pro it'll take all the cores.. and anything connected to share will choke.

fyi I am getting 750>800MBs over 10Gig between macs running smb with a few tweaks here and there. Which is plenty. It's not 10gig. It's whats either end thats going to cause you issues.

Remember - No one will thank you when it starts to fall over mid post production.

s



Return to posts index


Andrew Scholotiuk
Re: 4k 3D Feature - Have to rethink shared storage
on Nov 28, 2014 at 7:54:03 pm

Thanks Simon,
That was our original thought and looking like the best option. Shared storage for 4K just isn't in our price range, to bad we weren't just using 1080p because there are some great options out there from small tree and others.

Question regarding your 10Gig network. Can you describe this in more detail? Do you connect to a media server or are you connecting directly from system to system?

Thanks,
Andrew


Return to posts index

Simon Blackledge
Re: 4k 3D Feature - Have to rethink shared storage
on Nov 28, 2014 at 8:32:36 pm

Initial tests were direct connected a MacBook Pro with a 10Gig Promise Sanlink to a new Mac Pro with Areca TB2 raid and another promise sanlink

After that ran well a 3rd sanlink was placed on an iMac and the Areca connected to that. The iMac runs as the server. SSD / 32 Gig ram etc..

The sanlinks have 2 ports. so with 3 machines you can create a triangle connection.

After we have run this for a while I'll probably throw a 12 port 10Gig switch in for more machines.

VFX are connoted to the iMac over 1Gig so they don't bog the 10Gig down.

Happily grading multi channel 1080 EXRs in Resolve over 10Gig. Plenty for us.



Return to posts index

Andrew Scholotiuk
Re: 4k 3D Feature - Have to rethink shared storage
on Dec 11, 2014 at 10:47:16 pm

Just wanted to followup, I found a great article on something that I was trying to achieve and showing why it is not reliable.

https://forums.creativecow.net/thread/197/859967


Return to posts index


Thomas Mathai
Re: 4k 3D Feature - Have to rethink shared storage
on Jan 30, 2015 at 11:27:57 pm

>>And when they need to CONFORM to the "big stuff" that you want to do, they conform on a Quantel Pablo Rio, or Clipster, with NO SHARED STORAGE,<<<

That's funny, I've worked on many DIs where the conform was done WITH SHARED STORAGE. Maybe we were doing it wrong ;)



Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2020 CreativeCOW.net All Rights Reserved
[TOP]