Need to setup my first File Server
I own a small VFX/design company that is steadily growing and badly need to setup a file storage system as the size of my projects and number of systems is growing. Currently I'm using 2 PowerPC G5 Macs, and 1 8-core Mac Pro that runs both OS X and Windows XP Pro X64. I am going to be setting up a RAID on the Mac Pro, one for each OS, and want to setup a file server that I can store the data on. I am also interested in setting up a backup that will mirror the file server.
I do not want to break the bank on this but I need to at least set something up with 4-6TB of Hard drive space to start with with gigabit ethernet. I will not be running files off of the file server but use but more as an initial backup (mirror) of my workstations and a place to put my completed jobs. It may also be used in the future as the controller for a startup renderfarm that I would like to get going this year.
I have been told that linux server would be the best bet with a hardware firewall so the file server can also act as a gateway for the internet.
Any info will be appreciated as I have never set up a file server before.
you want my advice - hire a professional to help you. A backup system for your data is a lot different than a shared storage enviornment. You can look for any drive that advertises itself as "network attached storage", and follow their instructions for installation. I assume that you have a simple network setup at your place of business.
If you are a PC and Mac guy, how do you expect to figure out how to configure a linux server ? Hire someone to help you. There are lots of solutions out there, at a lot of price points. If you have no money, and only want a backup system, just get a NAS drive. When you want true shared video storage, well, thats another story, and that's gonna cost more than $1500.
Another consideration is that external drives and network based storage appliances below $1000 have a pretty dismal failure rate. In the last year alone we've had half a dozen external drives fail (some less than a year old), and one network attached unit flake out so badly it had to be reformatted and only some of the data could be salvaged (using Hiren Boot CD and Knoppix).
So you don't want to skimp on your backup system...or if you do, backup onto two different drives so you have a fail-safe in case one of them fails.
Our shared video/audio storage is Raid 5, which is capable of recovery if one drive in the raid fails, and we have a LaCie 4GB network drive that was $2000 from B&H Photo that incrementally backs up everything nightly.
If we're doing a project that has a really tight deadline, we'll even backup those project files and video/audio files to one of our direct attached external units just as a precaution. Call me anal, but I don't like re-doing something I've spent a month working on!
Just think how long it would take you to rebuild the stuff you're going to be storing on these, not to mention the embarrassment and heartache it would cause with clients, employees etc.
Magnetic Image, Inc.
Hi Bob, et al
Thanks for the quick responses.
Hire a professional what? Tech guy? Browse google again? Pull out the phone book and and point? Sorry for the sarcasm, can you be a bit more specific? I haven't had much luck when hiring people for this sort of thing because it's tough to know who is good.
I use primarily Mac, windows only for 64-bit apps when needed. I know guys who have linux systems setup that would help me do this, I just wanted a second opinion before I spent any money to see what options there are for me. I've been meaning to learn linux anyway (not that I'm thrilled about it) as I want to try out some of the high-end software I use to see if it fares better stability wise, such as Maya and Realflow. Considering I taught myself pretty much everything I do, I'm not too daunted about wrapping my head around linux, but have been warned about it's differences from other OSs.
The main thing I was told was by several people was that it is not easy to setup a file server system where mac and windows can simultaneously read and write to the disks because their methods are different and something like samba needs to be used which is crap and doesn't work well. Then they bring up linux and it's stability, etc, etc. All hearsay to me which is why I'm posting this question. If it was common that things just worked out of the box, I would probably bite the bullet and just go buy an Xserve or 2, even old ones, plug in some wires, configure a few things and get back to work.
I don't need shared video storage right now (although it would be nice) but I am definitely looking into something more than a single drive. I already have 4 firewire 800 drives and have had numerous 400 drives over the years fail which is why I want to move over to something more reliable as I fear my largest drive here may be reaching the end of it's life.
As far as cost, I'm not scraping the bottom of the barrel here, I just don't want to drop $5,0000 or $10,000 initially if I don't have to and instead build something sturdy I can expand upon. So if I could get a setup for $2,000 - $2,500 and add to it as I need to, that would be ideal. Or build it in stages where this month I get the main node, next month increase it's RAM, month after that I get the secondary backup, month after add more drives, hardware firewall, etc. And then add in extra nodes or Raid cards / disk arrays as needed.
I assume that for a file server that I would need (that is really more like a backup and gateway at this point) I wouldn't need the latest and greatest processors, RAM, or SAS drives, and same goes for a second level backup. Just a sturdy system with enough RAM that is configured properly with 7200 RPM drives in a RAID 5 or 6 that is mirrored to another backup system and a bunch of gigabit ethernet ports.
And I had a question too about archiving but maybe I should leave that for another post.
I think what Bob's trying to say... in an uncharacteristically NICE way... is that this stuff get VERY complicated very quickly. The minute you go beyond two connected computers you get into a whole
different level of IT considerations. Yes, you want to spend as little as possible and yes, your CURRENT needs are modest. But, if you're anticipating any future growth, and if reliability is required, my advice would be to NOT cheap out.
"Who to hire?" Well, you've just communicated with him... MR. Z.
On the question of who to hire. Good IT companies usually have very knowledgable people when it comes to backup, moving files etc. We use a local company in our mid-sized town (150,000) and they're very knowledgable. But they also realize what we do is very different than a corporate network...so they listen to my needs and my recommendations (based on a lot on my own research and knowledge), and they've been indispensible in helping to setup our shared video storage and backup.
I could've done all of it on my own. But at a huge price in terms of time and headaches. It's worth it to pay them their $100/hour fee for the really complicated stuff. Often, they'll come in and will tell me, "this isn't that complicated," and they'll walk me through a configuration so that if it needs to be done again, I can just do it on my own. It saves them from having to come to our place for a task that takes 15 minutes and frees them up for their bigger jobs, and it makes me happy.
All we did was ask around town for recommendations and this company's name popped up repeatedly. They like working with us so much (because we're knowledgeable) they ended up becoming one of our clients too!
As for spending $2-2,500...you're going to still be in the "cheap" range when it comes to drives and configurations (if you build it yourself and if that includes switches etc.)
Our 4TB backup LaCie NAS drive was $3000 I believe, the 2TB LaCie NAS we use for projects was $2000 and the 4TB Apace vSTor for shared video storage (12 DVCPro50 real-time streams over GigE) was $9000. We then spent another $1000 on a good managed switch, $150 on a rack to hold it all, $300 on new upgraded cabling, and about $1,500 to pay the IT company to install it all (it took them an entire day to install, cable and setup the hardware, then another day configuring and testing stuff with the help of Apace via GoToMeeting.com.)
That's $17,000. And guess what the weakest part of the chain is?? The 2 LaCie NAS drives we bought. I didn't research them thorougly and they just use 4 spanned drives rather than a raid. I foolishly assumed they would be RAID 5 or 6. They don't even have the capability of doing either. They also don't have the ability to adjust for jumbo frames (MTU), which could speed up throughput.
They work, but they only give us about 20-25MB sec throughput and we notice it occasionally while editing. A project file will pause, a photoshop file will take a bit longer to open or save etc. So I should've spent the extra $1500-$2000 on RAID based storage for at least the project drive.
Even if you subtract the shared storage and the 2TB unit, we still would've spent $6,000 for the backup alone. Sure we could've gone with a cheaper switch, but the one we chose was already in the middle of the pack. 4 months after installation, it went south and had to be replaced (under full warranty thankfully.) So again...if you try to do it "on the cheap," just prepare yourself for failures. They WILL happen, usually when you're on deadline.
You can probably configure something for $2,500 that's pretty solid, but I'd do a lot of research on the components to make sure they're reliable.
Magnetic Image, Inc.
Thanks for the info guys.
First off I do understand the complicated nature of these things but I cannot afford to spend $10,000 - $20,000 on a file server at the moment. Going the way of RAID seems like a good way because I can always add to the server when I need more space. And I don't mean a RAID setup like in one of those big lacie drives, but a controller card + individual SATA drives that get striped.
Using those shared storage drives I don't like the idea of either, especially Lacie. I had 2 Lacie F400 drives die on me and lost a ton of work. I actually still have them here so maybe one day I can restore some of the audio recordings on them. I currently use G-tech drives for backing up and storing uncompressed HDV footage and captured audio and the firewire 800 makes it fast enough to keep those on those drives while working on my workstation and loading the files into AE, Shake and FInal Cut.
From the other guys I know in the film, tv and game industries, they all suggest RAID setups using internal arrays and/or external chassis with a workstation which is where I'm leaning towards now for my workstation. I was assuming this could be done with a file server since SATA drives are very cheap compared to those network drives (SATA 7200 RPM around $120 for 1TB) and are very fast compared to SAS arrays for this type of work from benchmarks I have seen, at least on the Mac side. Is there something I'm missing about this when it comes to a file server because I could buy almost 20 SATA drives for $2,000 + add the cost of controller cards and chassis to house them, + the actual server + hardware firewall + wires and maybe a new power backup. I don't see how that should reach $10,000 or more even if I spent $2,000 - $3,000 on the actual server and $2,000 on drives (which I wouldn't need that many anyway).
I live in a small town in southern westchester NY, so no real good places here. I have access to New York City companies for IT and maybe in westchester there might be a good one, but given the size of the NYC there are so many companies it's hard to know who is good and who is not and large comapnies who are probably reliable might cost a lot of money or not do small setups like mine. I do agree that spending $100/hour isn't steep if it only takes a day especially since this is a small network for myself. Any larger jobs I work on I hire remote freelancers so it's basically for the 3 machines I use here and no one else.
Incase I have been misled, or the people I know don't understand networking enough, I wouldn't mind using a Mac (desktop or Xserve) to be my file server. I could even use my old mac that is a Dual proc G5 (about 7 years old) as long as it would work between mac and windows. I could just buy a fresh disk as a boot drive, it already has 4 GB of RAM I think and I could chuck in some RAID cards and drives. If the hardware would be too outdated (no PCI-E in that one) I could even buy a lower model Xserve or the cheapest desktop apple offers.
Btw, Bob, if you do this sort of work, where are you located? Do you have a website / contact info?
[Darren D'Agostino] "I could buy almost 20 SATA drives for $2,000 + add the cost of controller cards and chassis to house them, + the actual server + hardware firewall + wires and maybe a new power backup. I don't see how that should reach $10,000 or more even if I spent $2,000 - $3,000 on the actual server and $2,000 on drives (which I wouldn't need that many anyway)."
If this is simply for backups, as you say it is, why would you want 20 SATA disks striped together?
I don't want 20 drives striped together. My point was the cost between SATA drives and network storage drives.
The file server would mainly be so I can easily share my project files between my 3 machines and also the boot of windows on my workstation and contain all files on each machine which is why I said it would be "like" a backup. I eventually would want an actual backup that was a mirror of the file server and a way to archive.
Check out freenas.org - that is without question the best method for a DIY NAS.