Anybody use Isilon storage or seen it in action? Super expensive and I've not heard much good news about it.
I've got an Isilon 6000x that I'm about ready to retire. 4 nodes, around 20TB total.
I've talked about it before... but I'll do it again...
First, the good stuff:
Next, the bad stuff:
Who should buy it?
All in all, my experience has been kind of negative, but honestly, my company should not have bought it in the first place. Isilon is great for people who have huge amounts of data and can benefit from having it consolidated in one place, with a purchasing strategy that will take advantage of the increasing rewards of adding nodes. If you're not working with half a petabyte, I'd look elsewhere. Four nodes of Isilon does not make good financial sense.
Isilon IS expensive, but it is also some cool tech, but few and far between are the people who actually can take advantage of it. Also it seems like it's being retrofitted for more typical IT environments with additions of features like iSCSI, Hadoop, VM integration, etc. If I had tons money and I had to pick ONE storage platform for the rest of my life, I might pick Isilon. Then again, I'm not sure I could live with their support.
Thanks for the detailed reply David.
You mentioned you had no issue with bandwidth. Can you describe the workflows the Isilon volume supported? Any live ingest?
Yeah, live ingest is no problem for us, because we're using ProRes and DVCProHD.
But you should be able to upgrade to 10Gb if you needed uncompressed or something.
4x nodes, 2x 1Gb ports on each, for 8 total.
2x playout servers
2x capture servers
Each server is connected to a separate node (or sometimes I double up if I want to free up a node for edit).
For one event I captured 7x live feeds DVCProHD, while also allowing editors access. I posted it here somewhere in the forums, you can look up that report.
I use NFS as my protocol and use Softron Movie Recorder for capture. No problems with it. The only problems I had were with some wonky NFS permissions messing up my files, but once that was sorted it was smooth sailing.
Pretty cool, but in theory you should be able to do the same with a much cheaper system. Maybe if I had to capture like 16 channels while editing multichannel something like this might be worth it...
Interesting you mention 16 channels of ingest. That's how many we send to each volume for a total of 32. We do require growing file playback also.
You mention connecting clients TO a node. Don't all the node and client connections go to a switch?
I agree that the cost and the typical IT targeting are drawbacks. I put them in the same category as NetApp.
Yes they connect through a switch, but you can access the cluster through a specific IP address allocated to a single gigabit port. Of course you can do bonding etc., but it's pretty easy to just segregate your traffic, which also means that file transfers won't slam into a recording process.
Are you doing 32 channels of uncompressed or compressed?
I've read your post and was wondering why you are looking at an Isilon for the kind of work you want to do.
Are there any IT features that you require that makes you want this solution?
There are heaps of vendors offering scalable (not auto balancing though) storage that does it in one 3 HU box at a fraction of the price.
Editshare, Facilis, GBlabs, Smalltree, etc.
They all are well suited for many streams of compressed (and even uncompressed HD).
Looks like a waste of money to go for Isilon unless you need all the other features.
The big selling point for Isilon is easy expandability and increased resiliency the bigger it gets.
Editshare can't be expanded once a volume is created and Smalltree is very limited to how large it can grow. Can't comment on Facillis. Isn't that a block level SAN system?
We have high hopes for GBLabs and will be testing a system. While expandable and redundant, it is far more complicated to achieve.
The IT market Isilon targets is both good and bad. Bad that it expends resources to cater to that huge market, but good in that helps keep the company in the black. A small media only focused company, like GBLabs, does limit it's market and therefore it's sustainability.
every company that you mentioned, that you feel is "limited in expandability" makes no sense.
Most vendors are using SAS host adaptor cards from ATTO or Areca (EditShare uses LSI Logic). These cards (like the ATTO R680) can run 128 physical volumes, or 8 16bay chassis. If you put 4TB drives inside the chassis, this means that you can have 512 TB (1/2 pedabyte) on a single server. Need a full pedabyte, put in a second server, and connect both servers (via 10gig card) to a switch, and now all of your users have a Pedabyte of storage, with only 2 servers.
I cannot even imagine what your application is. All the companies talked about on Creative Cow can do exactly what you want. You don't need Isilon (not unless you want to feel like a "big boy"). I work for huge corporations, and no one is using Isilon (well, I don't work for Apple, but the iCloud site in North Carolina is all Isilon chassis).
By expandability I mean expanding existing volumes without reformatting or loosing capacity when adding a larger chassis. I think the industry term for this is "scale out". This is one of the issues we face with Xsan. Deploying new volumes makes sense to a point, but we're at a point where we have enough volumes and need each volume to grow.
Scale out is where nobody can really touch Isilon. It was designed with that forefront along with chassis redundancy. GBLabs can be built in a way that lets you add disparate chassis sizes as well as chassis redundancy, but is requires a rather complex deployment. Islion even allows replacement of an older chassis without interruption.
Performance is really the question as far as realtime "do or die" ingest goes. EMC claims many media installations but many times it's the consumer streaming side which is the case with MLB. David had no complains so that is really the first good report I heard first hand.
One can certainly build a faster small system with GBLabs as each Isilon node is only capable of 740MB/sec while GBLabs is 1400MB. Even worse - Isilon is a round robin write, much like Xsan storage pools, so no request could exceed 740MB while GBLabs will scale to a point.
Cost of course is the main issue with Isilon per Terabyte. The minimum system is a 3 node cluster in which case you loose 1/3 of capacity. Add to that the fact it has to grow in 3 node blocks, so with 3TB drives that's an expensive big 200TB block. The nice thing about it is it gets more efficient and resilient the more nodes are added. The exact opposite of Xsan/StorNext.
David pointed out Coraid http://www.coraid.com/products/high_performance_nas which may have the best of GBLabs and Isilon with some compromises.
We're really in the investigation stage but see the future being NAS.
Just some corrections on Isilon -
1) you can grow to any number of nodes (not a multiple of 3), but three is the minimum per NODE TYPE. Meaning, you need a minimum of 3 NL nodes for an NL cluster. But you can add just one if you like to get a 4 node NL cluster.
2) On performance, I don't know any specific numbers to their current offerings (mine is 4 years old now), but I know you can add performance by adding S or X Series devices, and you can also add a "performance accelerator" node.
3) The cost isn't just the "per tb" costs, but also the per year costs. Their support costs are unbelievable. My cluster is now unsupported and ready to be put to rest as soon as I can replace it with something cheaper.
(Anyone wanna buy a 20TB Isilon cluster? Didn't think so...)
[Bob Zelin] "I don't work for Apple"
Which is why, in the long run, they're doomed.
Had about a week of training on EqualLogic arrays (somewhat similar to Isilon), and it's just a different animal from DAS, NAS, SAN boxes that I was used to.
Like David said, expandability: not just the petabytes, but how it's done. Add a chassis full of drives (to the iSCSI subnet), sign in to management, add it to the pool. You're done. You could still do a few things but the main idea is that its capacity is instantly available. No RAID expansion, no adding controllers and HBAs, no restarting the system.
Resiliency. If your primary chassis goes up in smoke, sign in to management (yes, it's transparently available through surviving members of the group), add or order a spare chassis, keep working as if nothing happened. Try that with any regular DAS, SAN or NAS.
There's a bunch of other things that enterprises often need: automatic snapshots, 1- and 2-way replication, tiering. All that costs a pretty penny, too, of course - five to twenty times the cost per GB vs. "regular Joe" DAS and NAS boxes.
EqualLogic is a block (iSCSI) rather than a NAS solution, so a very different beast to Isilon. It does offer some limited scale out, but like similar 'scale-out' iSCSI such as HP LeftHand once you have a couple of dozen nodes management and performance start to become 'challenging'.
Isilon is not for people who buy just on cost per GB, but rather it does three things really well: scales capacity (up to >20PB in a single files system), scales performance (many many GB/s) and is very easy to manage. Very little, if anything else, does all three well. For that capability there is a cost, so not for everyone, but from a pure NAS perspective it is perhaps the coolest tech out there.
We do ProRes LT for the mass ingest.
ProRes SQ for everything else.
If you're interested in Isilon, I'd say that Coraid is also worth a look. Again it's IT focused, but their ZX series gear is pretty slick for large media stuff.
You get dual heads for increased bandwidth and failover, ZFS filesystem, NFS mounts, and you can grow on the fly by simply adding shelves of storage or even individual drives. Also cool is that they don't charge an arm and a leg for drives or support, so it's really cost effective to grow.
There's a lot of options here for growth, you can add SSDs for faster read/write cache, SAS if you need IO, or just cheap sata drives for more capacity.
Also it's based on Solaris and so you don't have to deal with all the proprietary commands and horrid support of EMC/Isilon.
It's definitely not for the smaller shops, but for larger operations it might be fitting. The cost savings start to kick in at around 200TB.
Thanks for the tip David!