I started this because I didn't want to resurrect an old thread. I know that it's recommended to render to a local disk array. Makes sense...but as we talk around the studio here to decide if a SAN is going to work for our work flow, we had this question:
Can we pull media from the san in the rendering process?
Example, if we worked with reference files, or a video file on the san, can I use that in a layer in an AE project? Or would it be recommended to pull that file to local storage first? Of course when it came to rendering the final file, we would target the local drive....
You're simply asking about something that can be done both ways. (Speaking in terms of capability of a "SAN" network)
About AE, I don't know specific requirements, however, in a world where perhaps Final Cut exists and you have properly implemented your hardware and network configurations, yes, you can share media files off the same storage where the project files reside.
Again though, this may not be the case with your situation, based on limitations of your actual application in use, and software/hardware installed.
If you are truly in a SAN, where you have multiple reads, and multiple writes going on at one time with many users, then you likely have some traffic cop software overlying on the network that will prevent people from reading and writing over each others files, and using files which are locked for projects, or media files which someone else is accessing. Another thing that might affect what you want to do, is the File locking, or Volume locking capabilities of your installation.
Again, this depends on what your set up is capable of in it's current form, what room it has to upgrade or expand, etc etc etc ...
Can you expand your current network description so we might all understand just how you're set up now, etc etc... ???
Does this help any?
Kind of...I was talking more about everyone's favorite topic around here....SAN over gig E.
I know that rendering to the shared volume will bog down the system....so you usually render files from the NLE or AE to a local drive or drives. The question is can you use files on the shared volume in a graphics situation. I'd just like to not have to move all my source files to a local drive before I import them to AE....was trying to find out if this is possible.
Did I explain better?
We use an Apace vStor over Gig-E and we render constantly to the shared storage via After Effects, Digital Fusion, ProCoder, and our editing system (VelocityQ) with no problems whatsoever.
We load media into AE and Fusion directly from the shared storage. Again, no problems. Rendering on compositing and encoding apps is no different than editing or rendering from the non-linear edit system. We can typically get about 10-12 real-time channels of DVCPro50 SD video in real-time across our system across 4 edit stations. Occasionally (like once a week), a video clip might stutter, or a clip might freeze if 3 or 4 edit systems are going at the same time and using multiple clips. But this is rare and it's never caused any issues goint back to tape or exporting to digital files.
We've never had a render fail or cause any issues using this setup. I would think a SAN, with it's faster drive throughput, would easily be able to handle rendering across mutliple stations.
Magnetic Image, Inc.
On our system, it is unfortunately more complicated. I promote a very simple SAN, with simple link aggregation. We have found that when you render using the SAN volume, it bogs down the SAN volume, and causes major issues. So many people (including many XSAN people) say "render locally - drag your rendered files to the shared volume, so everyone can share".
We had two clients that this was totally unacceptable to. So we did this - instead of one big mama link aggregate port (which makes the system so simple - so that anyone can just plug in) - we set up seperate subnets, and created multiple bonds, so that a heavy duty After Effects user had a dedicated bond (2 ports) just to his workstation. Now he can do 4 hour renders, without bogging down the entire system. I HATE doing this, because the purpose of my article, and our system was to be VERY SIMPLE, that required no thought, and not training. When someone insists on doing massive renders on our system, it requires setting up subnets, which makes it more complicated for non technical clients. The very appeal of our system (other than cost) is that unlike XSAN, it requires NO training. However, with multiple subnets, it does require training, and "paying attention". So can we do it - yes - do I advise this - NO. Render locally, and drag your rendered files to the shared volume - just like Mark Raudonis does with his 100 seat XSAN system.
Just out of curosity, what is it about rendering that causes it to bog down the SAN?
Interesting that we don't see slowdowns on Gig-E. Of course we rarely do renders that take longer than an hour or two, but they don't seem to cause any problems on the setup we have.
Of course we're primarily running SD stuff through it, but I just did a complicated HD project in After Effects and was able to render, play-out after render etc. in real-time as long as I used either the Avid DNX (or whatever it's called) or the CanopusHQ HD codecs.
Magnetic Image, Inc.
Yeah, I got that....don't render TO the San.....but maybe I'd be more clear if I put it this way. Can I render FROM the San? As in, can I have source files on the San (for AE graphics), but render my finished outputs to a local drive (and then of course drag them to the San)?
Hey David, we have a fairly new ethernet based (iSCSI) SAN that has been doing this fine (rendering to a local file from SAN based source material). We've had real-time playback on other SAN based computers during AE renders.
In my experience, rendering from AE is not very demanding on your SOURCE footage location -- nowhere near as demanding as real-time playback through an NLE like Final Cut. AE tends to pull in footage a lot slower than real-time when it renders, so it doesn't need to suck a ton of bandwidth. It's more processor and RAM (client) intensive than anything, and having a fast destination hard drive doesn't hurt.
That SAN only has a few active clients at a time (three), so I can't say how well things would scale to a larger environment or a strictly AFP based environment instead of iSCSI. We're also dealing mostly in SD on that SAN. So, no idea if this will be good for you, but I know it's okay here.
Great...thanks, that does answer my questions.
That would be about the same number of users on our system...and exactly how we would want to pull footage.
Leaving the source files on the SAN and rendering to a local drive will save us a lot of time in moving media around. It won't take long to just drag the finished renders to the SAN drive compared to having to move the source file also.
What causes a bog down of a SAN?
Well that's like asking what's making it snow outside..... okay ....
Its generally never just one thing, it's usually several things at once. For example, if you design the network to support so much bandwidth, but your storage doesn't go that fast - that could cause slowness. This could be the opposite as well, the storage could be super fast (MB/s), but the network bandwidth is not that fast, therefore causing a bottleneck.
It's also common for some devices to be sitting behind a Shared server, but the devices were not designed to be hit with large IO/s, but a bunch of small ones - like in a Point of Sale system .... depending on what the storage was designed to do, this could be a bottleck.
The RAID cards, they can cause slowness. The processors can cause slowness, the memory can cause slowness, the fiber channel connections can cause slowness, the communication between client / sever / storage and back, can be long, causing slowness (also referred to as latency)
Answering the question for you, is just going to generalize this. Most SAN's that are slow, are having problems because of something in their environment, be it to many users, to many servers hitting one storage, etc etc.... it would require looking (reviewing logs, kernal i/o's, etc) over the configuration to determine what's going on.
Some of the Storage SANs are performing well with SD environments from what I've seen around. The problem is starting to creep up when people try to do something larger then SD format. Formats which require larger size IO/s, and require "back and forth" movement in a certain time period, otherwise the frames drop....
It's good for people like me when all other people think about are bandwidth requirements. Since there's more to it then that when your working with larger IO/s, I could care less about bandwidth in most cases, since that's not typically an issue. 90% of the time, it's Raid Cards, and Storage Devices which are causing bottlenecks for editing in real time.
For example, I know it's entirely possible, probable, and works very well to edit up to 2 x Pro Res HQ streams on a single Gigabit wire
(about 60MB-70MB/sec needed for this (using pro res @ 30-35MB/sec as the basis)) ---- if you multiply this by 4 people, now you need to push 240MB/s just to support 4 users running 2 streams each ... So I know if I have 400MB/s of network bandwidth, the network will not be a bottleneck. However, what I also need to know, is also how quickly each of those streams need to move their peice of data before the next piece of data. This is usually done in miliseconds, and moves faster or slower depending on ALL the pieces of the hardware / network solution.
If you just live in the MB/sec world, and you need to be realizing a world where you have Real Time needs, well, that "real time" requirement changes how you need to design and implement a Real Time Shared Solution to work without the hiccups we all hear about and hear so many people failing at some point or another. (Fiber Channel SAN, or otherwise...)
Did that help at all ???
Yes...that helped. I was just curious why a basic Gig-E setup like ours doesn't seem to take any sort of performance hit from rendering from AE and Digital Fusion while also editing from one or two NLE workstations. And both the compositing apps are pulling source footage from the shared storage as well. I just did a project that had over 100 layers and used multiple video source clips per frame and it didn't seem to tax the system at all.
It wouldn't seem like a compositing app would use more resources than a NLE system does, unless we're talking about projects that are simultaneously loading dozens of source files for each frame being rendered, which I guess is possible.
I was just curious about it.
Magnetic Image, Inc.
I've harped on this before and it still makes ZERO sense to me. There's nothing about rendering from After FX, Final Cut Studio or a number of 3D render stations that should require anywhere near the network bandwidth or drive array speed that capturing footage or playing back footage in real-time needs...even for modest codecs.
Rendering in After FX will only write a few frames a second at most (and most likely it will take a few seconds per frame instead)...that's only a few megs per second even at 1080 uncompressed. Similarly a 3D station will save out one frame (a few megs) every few minutes. I simply have not seen a satisfactory answer why you can conceivably have 3 or 4 people reading and writing to an array+network capable of handling 200-400+ megs/second without issues and suddenly somebody needs to render out an After FX animation of a couple megs a second and everything is out of wack. Even rendering out of FCP you are still not rendering/writing in real-time...why can't you render to the SAN?
Is it that the RAID can't handle writing a small file like a single HD frame (or small bits of a larger quicktime) intermittently while also serving a constant flow of 34-40 megs/sec to two other places simultaneously?
There are people here saying they have no problem with it - and why should they? I don't understand which systems can't handle it and which can - what's the defining factor? I get the feeling nobody here knows.
The fact is that people want to read and write to the same SAN weather it's FCP, AE, a 3D render or an audio mix....copying over a PDF script...whatever! People want to organize ongoing productions in ONE place for simplicity, organizational and archiving purposes.
What is it about writing out a small amount of data per second or intermittently (single HD frames or small bits of a larger quicktime) that interrupts an otherwise capable system's throughput?
Sabertooth Productions, Inc.