Xsan: Best practices for admin and backup

	Re: Xsan: Best practices for admin and backup by Dave Klee on Sep 1, 2009 at 11:55:31 pm Hey Arturo, and welcome to the fun-filled world of Xsan administration! This is a REALLY broad question and there are a lot of ways to go, so I'm just going to spit out some good ideas and bad ideas that I know about. Hopefully a few of them are new and/or helpful to you. Good Ideas: -Rely on a well-maintained DNS. DNS is one key to a happy Xsan. Use host files if you must, but you'll be happier in the long run with DNS. Regardless, ALWAYS use Static IPs. -Backup everything in the /Library/Filesystems/Xsan/config/ folder on your metadata controllers on a regular basis. These have come in handy for me. Doesn't hurt to backup the same folder on a client every now and then. -Target important data on your Xsan for regular, scheduled backups using some good software. I like ChronoSync, but there are a lot of decent software options. I backup Final Cut Pro project files and any kind of "Digital Master" file that can't easily be replaced (like files acquired from tapeless cameras like the EX1 or HVX-200). As for a backup location, I like to use disk devices like the DroboPro with some redundancy built in, and plug them into one of my servers using Firewire 800. Lots of good hardware options out there -- doesn't need to be fast. -Format any Xsan backup destinations as case-sensitive file systems. Xsan is case-sensitive, and it helps for backups to be. -Make a clone of your metadata controller's system hard drive from time to time. I make a clone before every server upgrade, just in case it doesn't go well. Carbon Copy Cloner is a great software tool. -Make sure to keep both a primary and backup metadata controller healthy and on identical software versions. -Keep all edit suites (clients) on the SAN on the exact same versions of software. Using your backup metadata controller as a Software Update server is one way to help manage software deployment. -Keep your Xsan volume usage at or below 75%. Xsan, like all filesystems, is prone to fragmentation when it gets full. However, Xsan fragmentation is pretty ugly, and the tools to fix it are imperfect. -Keep your Xsan volumes mounted on your metadata controllers if you want to modify file permissions or attempt to use Spotlight (for Xsan 2). Bad Ideas: -Don't mix metadata with user data in the same storage pool. -Don't try to mix Intel-based and PPC-based metadata controllers (for a primary and backup pair). -Don't upgrade a client (edit suite) to a newer version of Xsan or Mac OS X before upgrading your servers. Your servers (metadata controllers) always get upgraded first -- even for very minor (dot) releases. -Don't try to backup your entire SAN onto LTO/DLT tape unless you really need to. (Some might disagree with me here, and it really does depend on your environment and needs. Nonetheless, backing up a whole Xsan can be very expensive and somewhat complicated.) Find your most important data and back it up, instead. There's very little on our Xsan that can't be lost right now -- it's either backed up on disk (DroboPro or older SCSI devices) or on video tape (original field footage and master tapes). -Don't try to re-share your Xsan volume over AFP (or any other protocol). Apple generally says it's a bad idea (though there is improved AFP support in Xsan 2), but I just don't like the concept. I think of a SAN as a sports car, and file sharing is like pulling a trailer uphill. Don't ask your GTO to pull a trailer uphill -- it's not what it's meant for. Let the SAN go fast and work out other solutions for accessibility. (Someday, I hope we will be in a situation where one centralized repository will serve both the needs of a SAN and file sharing, but I haven't seen a perfect solution yet.) -Don't mess with the contents of the /Library/Filesystems/Xsan/config/ directory unless you really know what you're doing. -Don't change root passwords on your metadata controllers if you don't have to. It's a pain. -Don't change IP addresses on your metadata controllers if you don't have to. Also possible, but a pain. -Don't take for granted that expanding your Xsan volume in the future will be easy. It can be done, but adding new storage to an existing volume in a way that makes sense can be a challenge. -Don't plan on Spotlight working well on your Xsan. I have yet to meet anyone who raves about how well Spotlight works in Xsan 2. -Don't rely on the cvcp command to copy media files. cvcp doesn't copy extended attributes. Copy media files to/from an Xsan using something else -- I use plain old cp a lot and ChronoSync. I think two things are the most important: Know a talented person who can get you out of a bind when necessary (Jordan comes to mind), and carefully PLAN anything involving Xsan. You mentioned cvgather, and it is pretty helpful. Running it regularly wouldn't hurt -- and allows you to back up a lot of stuff in addition to your config folder. In general, though, I don't think you need to need to use the terminal as much with Xsan 2 as you used to. cvgather, cvfsck, snmetadump and cvadmin are all pretty helpful every now and then, and snfsdefrag can be useful(ish). But, I think most of your best backup practices involve common sense backups of important data with a GUI. I use these command line tools a lot before and after system changes and updates, but not every day. Above all, good luck! Dave Respond to this post   •   Return to posts index

	Re: Xsan: Best practices for admin and backup by Dave Klee on Sep 2, 2009 at 4:59:16 pm Hey Arturo... yeah, unfortunately, I haven't had re-shares work well -- regardless of what server is actually doing the sharing (the backup would be a good choice if you really needed to do it, though). The problem is that Xsan is (in my experience) not great at prioritizing file sharing workload over actual real-time video throughput. So, if you have a few clients connected over AFP/NFS/Whatever all trying to copy files to and from the SAN, but you also have edit suites working with fibre channel in real time, you might get dropped frames and overall bad performance. Xsan simply sees all requests for files as equally important, and even though YOU don't really care if a file copy over AFP takes a few extra seconds -- when compared to a real-time request to an edit suite over fibre channel that you want to be taken care of RIGHT AWAY -- Xsan won't prioritize the fibre channel traffic over file sharing. Also, Xsan used to be really bad with essentially giving AFP clients pretty much full bandwidth all the time -- if a file sharing client was connected over AFP, it just sucked as much bandwidth as possible, even when it wasn't DOING anything. What I do is create a "Bridge" volume that people use to copy files to and from the Xsan. Another option is Final Cut Server. This is a longer description from a couple months ago: http://forums.creativecow.net/readpost/180/856757 Now, I haven't tried AFP reshares much in Xsan 2, so it's possible it works better now. But, I'm still skeptical that a single system can serve file-sharing needs to lots of clients and still be an effective SAN providing real-time response to edit suites. If you or someone has made it work well, I'd love to hear about it. One suggestion that Matt Geller had in his book about Final Cut Server could apply to general file sharing if you needed to reshare an Xsan. He said that you should setup a dedicated server to host Final Cut Server, and when you connect that server to your Xsan, you could unplug one of the fibre channel cables (so only a single fibre channel cable connected Final Cut Server to the Xsan). This moderates the bandwidth that Final Cut Server uses down to whatever a single cable can provide. You could try applying this idea to an Xsan reshare; setup a dedicated server to take care of the AFP/NFS reshare of your Xsan, and only connect it to your fibre channel switch with a single cable. I haven't done it and don't recommend it, but if you really need a reshare, it's worth a shot. Hope that helps -- the Xsan reshare is a complicated issue, and I know I thought it would work when I started out a few years ago. It just hasn't made me happy. Let me know if you have questions, and have fun! Respond to this post   •   Return to posts index

	Re: Xsan: Best practices for admin and backup by Mark Raudonis on Sep 3, 2009 at 3:41:51 pm Dave, I agree with everything you've said EXCEPT your comments about AFP Resharing. Contrary to your experience, we've been successfully using AFP to reshare volumes for over five years now. In fact, this feature was one of the primary motives for us switching from Avid Unity. Here's how it works for us. The secret to successful AFP sharing is LIMITED BANDWIDTH. We work in a classic "off-line to on-line" workflow, so most of our clients (editors) are using the "Off-line RT" codec. This is an extremely compressed image that runs at approx 1/10th the bandwidth of SD DV video. Therefore, we can have 20- 30 people ALL using AFP to tap into the volume and still not bog down either the X-SAN or our ethernet switch. We also use a couple of dedicated X-Serves attached as fibre clients to facilitate this sharing. With three X-SANs under our roof feeding almost a hundred users, I can tell you that AFP works for us. You need to set it up properly, and understand it's limitations. Can you reshare uncompressed HD over ethernet? No. Will it work for multiple streams of off-line RT or maybe even DV? Yes, absolutely. Just because this limitation may not suit your workflow, doesn't mean it's not perfect for someone else... like ME! The people connected via AFP are NOT heavy users. They're story department, music department, producers, etc. They're primarily viewing media or doing "stringouts", not intricate layered editing. That kind of "light use" works quite well. In conclusion, don't beat up on AFP because it's not as capable as a fibre connection. Look at AFP as an extra "bonus" capability that X-SAN can provide that many other SAN's cannot. Mark Respond to this post   •   Return to posts index

	Re: Xsan: Best practices for admin and backup by Dave Klee on Sep 3, 2009 at 4:18:44 pm Hey Mark, this is great info! Thanks for "sharing." (Bad joke.) A couple questions for you. First, how are you regulating the bandwidth? Do those AFP servers have a single-cable attached to the fibre channel switch so they don't "steal" too much from your edit suites? Is the connection negotiated down to 1Gb? Use software like Vmeter SQM? Or do you just rely on the fact that those small files don't suck too much bandwidth, and the thing pretty much regulates itself? Second, do you have people doing full-resolution file copies over AFP, or only the online / offline workflow? For example, if editors want to get some large files off the SAN for use on a non-Xsan system (or copy large files back to the system), do they regularly copy those big files over AFP? Thanks again for the info -- it's great to hear from someone who's made the AFP reshare WORK. Respond to this post   •   Return to posts index

	Re: Xsan: Best practices for admin and backup by Mark Raudonis on Sep 4, 2009 at 1:12:52 am Dave, All of our fibre connections are Duplex, meaning there are TWO fibre strands per cable. I've never just connected a single strand. I don't even know if that will work. The fibre cards have TWO spaces for two fibre cables. We only use one of those spots per computer. That's on all of our clients... either hi rez or low rez off-line. Our switches are a mixture of 2 gig and 4 gig. We have the raid storage and our "heavy user" systems attached via the 4 gig switch. We do NOT use any bandwidth limiting software. By definition, the streams that they're pulling are low. We do make it clear to our users that full rez material will NOT work via AFP, so don't even try it. We HAVE done very large MEDIA copies from SAN to SAN via AFP. Those large transfers are always done at night or on the weekend to limit conflicts. Usually no problem at that time. Try it during a busy morning... and no go. Hope this helps. mark Respond to this post   •   Return to posts index

	Re: Xsan: Best practices for admin and backup by Dave Klee on Sep 4, 2009 at 8:00:36 pm Hey Mark, thanks again -- excellent info. Sounds like you've got a very interesting place. You're definitely right about the duplex fiber optical cable -- one of those strands is "send" and one is "receive," so they only work together. I meant to ask about the entire Fibre Channel SFP connector (from either copper or optical), and that's good to know that you only use one fibre channel SFP connection on every client. Based on your experience with limited bandwidth AFP shares, I'm still hesitant to have AFP clients try to copy full-resolution (HD) files to and from the Xsan while edit suites are in session. But, I'll never again say that the AFP reshare doesn't work well -- clearly it does for the right people in the right workflow. Thanks for setting the record straight! Respond to this post   •   Return to posts index