On-set DIT SAN/Finalshare?
Conceptualizing a solution for my on-set DIT needs for faster storage accessable via more than one machine, to divvy up the workload for RAW cinema workflows.
1) camera originals securely ingested as fast as possible.
2) processing of footage to proxy formats via Resolve
3) LTO 5 backups (eventually)
I currently run a hackintosh as my main system, which hosts a RAID 5 via ARECA 1880 card, so fast processing and access on THAT machine is fine for now. The issue is just moving the files in and out. Currently it's faster for me to make the individual MASTER footage copies via ingest over TB and my MBP connected to 2 eSATA drives. I then use AFP over cat5 to finder copy "working" files to my RAID on the Hack.
What I'm wondering is if it makes sense (speed wise) to continue to host the RAID on the Hack, and utilize AFP with an installed 10GBe card in the Hack to make it act as a small SAN with one of the new Netgear 10GBe 8-port switches, and a couple of other Macs with TB connections hosting PCI break-out boxes for 10GBe cards in each? An iMac could be a great ingest station for a Codex TB dock, networked over AFP to the RAID hosted on the Hack. And LATER, I could theoretically host another breakout with LTO 5 either hosted by another machine (mac mini? - not TB2 AFAIK) or connected to the iMac as well for backups from the RAID data.
Thanks for humoring me, any worthwhile pointers, suggestions, or potential pitfalls appreciated.
so exactly what is your question? How can you build this yourself without hiring someone that does this for a living to help you ?
Because the basic premise of what you want to do is all correct.
I need some plumbing done in my condo - we are re-doing a couple of bathrooms. If you could come over and do this for me, I would be happy to help you get everything working. I really don't know a damn thing about plumbing.
Rescue 1, Inc.
Is it possible for you to reply non-sarcastically Mr. Zellen?
I can appreciate the social aspect of the COW and information access and sharing, but for the life of me I don't understand why you routinely respond the way you do.
It's obvious I'm seeking more information other than just if it's conceptually possible, and that it's indeed implied I'm going to be doing this myself if not impossible on it's face. I've seen you link to Final Share information in the past, which has resulted in near step-by-step instructions, albeit for GBe connections. I just don't want to build some grand vision and realize it's not possible or advisable because of my "question" in the initial post....
"....any worthwhile pointers, suggestions, or potential pitfalls appreciated." ? <- There, I put a question mark at the end this time.
Hi Chris -
I will move this forward, until management at Creative Cow throws this thread off this forum.
I would LOVE to call up AVID, and say "can you tell me how you are getting two Ethernet cables to work without doing LACP" - but they won't tell me. I would love to call up Facilis and EditShare, and ask them "can you tell me exactly how you are getting AVID Project Sharing to work with your shared storage system so wonderfully, because I cant' figure it out. And I would love to call up Small Tree, and ProMax, and Tiger Technology, and countless others and ask them detailed questions, so that I can do exactly what they are doing, so that they don't get any money. But I think it's pretty obvious that they won't tell me - this is what they do for a living - they will discuss these subjects, but won't say exactly how they are doing it. After all - ALL of these companies are using off the shelf hardware to build their systems, and if we knew all the secrets, why on earth would be give them any money, when we could do it ourselves.
Final Share is the product that I build. Now, I could tell you the details of exactly how to get it to work, but then, I won't get any money ! So I believe that it might be you who may be sarcastic in this instance "can you tell me how to build a Final Share system with these off the shelf parts (which by the way, you have already figured out correctly), so that I don't have to pay you any money. My sarcastic reply to you was not so sarcastic. You want a shared storage system - I need some plumbing done (I am not kidding) - you pay for my plumbing, and I will tell you how to build the shared storage system. Why should you get something for free, and then I get nothing, and still have to pay my plumber ? Shouldn't I be saying to you "are you kidding me ? You want me to tell you what ?".
Sorry Chris, but I have to eat too. Do some research on the company websites whose products you have mentioned, and you will figure it out. I wonder when the producers you work for will ask you "can you show my nephew how to use RedCine-X, and Shot Put Pro, and tell me exactly what equipment you are using, so that I can use my nephew instead of hiring you from now on". We will see how you respond to that.
Rescue 1, Inc.
Bravo, Mr Zelin!!!
Chris, please read Bob's reply carefully and stop being a DickHead ... not only are you trying to pick the brains of a highly experienced and talented engineer for free and rob him of his livelihood, you quietly slip in that you're happily using a Hackintosh on set for mission critical data back-up of production dailies.
Have you checked recently with a little company in Cupertino on how they feel about you using their patented IP in a an illegal way? (read the EULA that comes with OSX) ... have you also explained to the producers of the shows that you're DITing that you're knowingly using illegal and unsupported systems and that if you loose any of their precious production content they won't have a leg to stand on?
Get real, dude.
high performance workflow
Best thread ever.
Amazing! Only on here can someone ask an innocent question, get answered with sarcasm due only to my own naivete, and in my following reply acknowledging the sarcastic reply that I answer in kind, I'M THE ONE called a "dickhead" publicly.
Nothing in my original post was intended as "Tell me how to do this so I don't have to pay your services," though that appears the way it was received, because I didn't realize that "Final Share" was anything Mr. Zellen (or anyone else for that matter) was personally invested in as a material participant, and for that matter didn't realize it was anything more than branding a system from a bunch of off-the-shelf components, in the same way I can buy the same hardware RAID enclosure and populate it with drives myself that any other number of vendors sell as turnkey systems and save myself money as well as gain experience doing new things.
I posted in a public forum for any insight or answers from ANYONE, and in no way was I "trying to pick the brains of a highly experienced and talented engineer for free and rob him of his livelihood..." Ridiculous assumption.
The idea of a public forum is to ask questions of people with differing skill sets and experience for information, and to share information if you choose. HOW you choose to do it if at all is where this thread went all wrong. A simple denial/inform that what I was trying to do was some sort of proprietary system would have sufficed, but that's not what I was presented. Instead I was met with sarcasm, which I've unfortunately experienced before on the COW on several occasions, including from Mr Zellen. I don't know why people feel the need to reply this way from the start, but I've gotten to the point where I may choose to call someone on the way they interact, by response in like manner.
For my naivete of relationships involved I'm publicly called a "dickhead," by someone who's only interaction with me is reading a couple of paragraphs, and the sarcasm I received in response to an innocent inquiry is actually applauded... whatever.
I am on set right now assisting with Phantom footage. There is literally a mountain of data.
If you are serious about this, I'd look in to one of the companies that is offering all in one systems, where the storage is a host machine that controls metadata, houses the raid of whatever size, and also serves the data out to client machines via a number of available protocols.
Be prepared to spend money, and if I were you, I would not be afraid of thunderbolt 2.
The client machines sign in to the host, and the host will a mange the software needed and the required connections (I am over simplifying this, but that's the overall workflow).
If not, you can try and cobble something together with a bunch of disparate parts, but eventually you will run in to trouble, and depending on your level of busy, it might stop working at an inopportune time.
Since I sell 10Gb stuff and would get some money anyway, I'll answer! :)
If you are copying files around, using 10Gb is a pretty safe bet provided everything uses flow control and the local disks are all fast enough. (If you are copying from a sata drive to your Areca, you're only going to see the SATA drives performance).
If you want people to play out video on all those attached systems, you might be disappointed. I haven't looked at area in a while to know what their latency looks like, but the load difference between one system prefetching and playing video is huge compared to 2 or 3 systems trying to play video. It's not sequential anymore and there are lots more metadata actions due to the shared protocol the clients are using. (Plus you have to tune the server for the higher network activity).
There's tons of stuff online discussing how to tune for TCP performance over NFS/AFP/Samba.
CTO, Small Tree Communications