Re: XSAN over 100 TB by Jordan Woods on Oct 29, 2009 at 5:12:23 pm
It won't matter what storage you have tied into your XSAN, unless you have some weird LUN sizes, but if you are under normal conditions with "regular" LUN sizes you should be fine no matter what fibre solution you work with. The size of the XSAN volume is a matter of its filesystem not the RAID.
With 2TB enterprise drives just around the corner (they're here, but not readily deployed) you will see 100TB+ XSANs with minimal amount of money needed to get there.
Here is my issue with 100TB+ XSAN volumes... How are you going to back up the data? If LTO, how fast is your recovery? (answer: sloooooowwwwww) So, that being said, there are plenty of discussions going on for the correct way to put together 100TB+ volumes, it just depends on your circumstances. If you are a broadcast facility, then you might need two identical 100TB volumes, ready to back each other up if failure occurs. Or if you are a post house, you might opt for a fibre near line or LTO, just be prepared to be down for at least a week for data recovery.
If your budget just gets you to 100TB without backup and you will build that "later," I'd scale down the XSAN plans to fit money for a backup solution. It is too important nowadays with so much tapeless work going on. (I think I went on a tangent, but I felt it is an important topic for those reading and planning an XSAN)
-Jordan
Senior Systems Engineer
Active Storage Labs
Los Angeles, CA
Re: XSAN over 100 TB by Dave Klee on Oct 29, 2009 at 11:07:28 pm
Hey Robert, I've worked a little with a university who has a 120TB Xsan 2, divided into two separate volumes for two different purposes (live playback of video footage on-air for television, along with a general video production SAN for editing). It's all built on the previous model of Promise RAID (750GB SATA drives).
Getting it running was a pain -- both from a Fibre Channel management and RAID hardware management perspective. There were quite a few issues with the older Promise stuff (controller and drive failures out of the box), which were all eventually resolved. Since then, the system has been fairly stable. There were also some challenges in Fibre Channel switch management for so many devices and ports.
It also sounds a little touchy -- it's hard to know the right mix of RAID firmware, switch firmware, and software versions for the OS and Xsan to make everything happy. (True for Xsan in general, but sounds like it is amplified with a big system like this.) A RAID firmware update needed to be rolled back after it didn't work in the environment. Propagating updates across all RAIDs was a challenge, partly due to Promise's web-based management tool that left something to be desired. (I hear they have been updated, but the core management tools are still command line and web based.)
Both volumes get really good performance (I believe one is 90TB, the other 30TB). Once running, they're very stable. The smaller volume has a real-time mirror with another Xsan volume in another building (because it needs immediate fail-over redundancy for live playback on-air). The larger Xsan volume has more limited backup.
Jordan brings up a GREAT point about backups -- especially dealing with tapeless files. Also, making changes (firmware and software updates) to a large, mission critical Xsan that is being used in production can be very challenging. Best to have solid redundancy plans that fit your needs -- whatever they are.
Also, considering easier to manage RAID hardware might save you some headaches. I'm not sure how well Active Storage scales to the 100TB+ level, but at the 30TB level, management is really a piece of cake. It helps me sleep better.
Re: XSAN over 100 TB by Mark Raudonis on Nov 1, 2009 at 7:30:47 pm
We're in the 100+ Tb category, but it's split into three separate SANs... primarily for reasons of convenience, security, and operation. Our offices are spread over two buildings across the street from each other. The cost of running a fibre link across the boulevard is in the six figure range... primarily for "pole rental" to the utility company. Don't even get me started about bureaucracy!
Our storage is a mix of brand new Active Storage units and old as dirt Apple X-Raids. All three SANs have this mix. No Promise boxes at all. We're quite happy with how it's performed so far, and I don't see the 100 Tb benchmark as an issue at all.
Jordan's point about back up is well taken. At a certain point of "mission criticalness", Raid 5 just doesn't cut it. However, we take a pragmatic approach to backup. With our workflow, the entire 100+ Tb is NOT mission critical. We do keep the original media as the ultimate fail safe. If we know that a specific show is nearing deadline, and any interruption of work would ruin everyone's day, we will copy that media local to one of the internal drives on a local client. With up to six TB easily held locally, there's very few projects that we can't back up this way.
I agree that if you're trying to keep the entire system backed up onto LTO it would be impractical. That's why we don't even bother. For someone that needs that kind of "mission critical" uptime, there are plenty of mirroring solutions out there. You just have to be willing to pay for it. If you follow any of Bob Z's rants... you'll know that most people on these boards have NO IDEA how much that would cost.
So... in SAN terms, bigger IS better, but you've got to be realistic about the cost of backing up a large system. Breaking down your critical needs to a much smaller size is the most practical way to approach this issue.
Re: XSAN over 100 TB by Bob Zelin on Nov 2, 2009 at 2:08:36 am
and not to get off the subject (and maybe I am missing something) but isn't the Cache-A Prime (and 800 Gig LTO4A backup system for 8 grand) a little silly, when the reality of 100TB+ systems becomming more commonplace. Once we see the reliablity of 2TB enterprise drives, this just means 3 16 bay chassis - which means that by 2011, "everyone" will have 100TB+. What will an 800 Gig LTO4A tape do for you then ?
Re: XSAN over 100 TB by Jordan Woods on Nov 2, 2009 at 5:04:20 pm
I would never use the Cache-A LTO box for a true backup to a live massive XSAN, however I would use it for an "on-the-fly" backup to my tapeless workflow, or at least that is how I see it being used. If I was shooting a bunch of P2 cards, I could use the Cache-A box to have a tape backup of all my P2 cards as they are loaded into the system. There are a ton of mirror based ingest systems that use a Cache-A style backup for ingest data. Meaning... it is ingesting to a set of mirrored drives, checksum'd the data, then dumped to LTO simultaneous to the massive XSAN.
