FORUMS: list search recent posts

10gb Experiences?

COW Forums : SAN - Storage Area Networks

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Sean ONeil
10gb Experiences?
on Jan 31, 2009 at 12:21:33 pm

Not sure if they recently lowered prices or not, but I checked today and was pretty shocked. A dual-port 10gbe adapter from Small Tree for $1k is very reasonable. Two of those cards in a Mac Pro could serve four stations without the need for a 10gbe switch. I think I'm going to go for it next month.

I've had bad experiences with 1gb on Mac equipment, so I'm a little worried. Is anyone actually doing this with high-end formats (like 4:4:4) and not having issues with dropped frames, laggy playback, etc.? Could one editor be capturing an HDCam SR tape while 2-3 others are editing uncompressed video without a hitch? I'd be using a CalDigit RAID card and 16 regular SATA disks set up for RAID-5.

I'm just hoping that 10gb is so much extra bandwidth I won't have any of the same issues I had with 1gb.

Sean


Return to posts index

Bob Zelin
Re: 10gb Experiences?
on Feb 1, 2009 at 12:04:26 am

I understand that even with 10G ethernet, you get a max thruput of 180Mb/sec - which is fine for a single stream of 10 bit uncompressed HD-SDI, but nothing beyond this. I don't know what the bandwidth requirement is for 4:4:4. I think 10 bit 1080i is about 120Mb/sec.


Bob Zelin




Return to posts index

Sean ONeil
Re: 10gb Experiences?
on Feb 1, 2009 at 1:39:09 am

[Bob Zelin] "I understand that even with 10G ethernet, you get a max thruput of 180Mb/sec - which is fine for a single stream of 10 bit uncompressed HD-SDI, but nothing beyond this. I don't know what the bandwidth requirement is for 4:4:4. I think 10 bit 1080i is about 120Mb/sec."

Ouch. That worries me. It tells me is that there's still no TCP offloading for OS X (where the card does the heavy lifting) so the CPU is doing all the work and it can't handle 1/4th of a 10gb stream, let alone four of them.

Thanks for the important info. I think I'm going to wait a few months, maybe when Snow Leopard is out.

Sean


Return to posts index


Shane Sokolosky
Re: 10gb Experiences?
on Feb 3, 2009 at 10:25:33 am

Hey Sean,

10Gb is like crazy fast, your CalDigit RAID card and 16 regular SATA disks set up for RAID-5 couldn't deliver even half the speed that you can fit down a 10Gb pipe (which I think that's what Bob Zelin was referring to about 180Mb/sec maybe more but..), to really take advantage of 10Gb you'd need some serious storage, what your asking about above (Could one editor be capturing an HDCam SR tape while 2-3 others are editing uncompressed video without a hitch?) can easily be obtained by 4Gb Fibre Channel at a fraction of the cost of 10Gb. The only time I could see that you might want to dabble in 10Gb or infiniband (10Gbps on copper wire and 30Gbps switches) is if you were trying to like setup a RAW 4K SAN with multiple editors or like 2k stereoscopic video for real time 3D editing on the fly with more than one user or something like that. Even 8Gb fiber is becoming more and more popular and soon you'll have more 8Gb Fibre RAID Controllers choices out there.

So What your talking about above would require more than the 16 SATA Drives,

You also mentioned that "I'm just hoping that 10gb is so much extra bandwidth I won't have any of the same issues I had with 1gb."

If you tried to use "Gigabit Ethernet" and some kind of Internet Protocol (IP) based file sharing service such as SMB, Apple File Sharing (AFP), or something like that then there's no way to pull off HD like your talking about (If your looking at some of Bob Zelin's solutions with gigabit ethernet, then sharing is possible but not at that bandwidth that you'd need.)

If you actually tried 1Gb fiber channel or Ethernet cables using iSCSI (both SCSI Protocols) you'd of probably came closer but still not quite enough unless you trunk the ports (and by trunk I mean probably 3).

Technically....
(Could one editor be capturing an HDCam SR tape while 2-3 others are editing uncompressed video without a hitch?)

If your talking about 3 people editing HDCAM SR then..
To do something like this you would need a SAN that could do about 720MB/s sustained and that would be cutting it low, realistically you'd want to more so I'd rate it at 800MB/s to be somewhat on the safe side meaning more reliable (Differences in storage manufactures makes a world of difference here).

If your talking about One person capturing HDCAM SR (I'd estimate @ true dual link 240MB/s) and 2 people editing SD uncompressed (25 Mb/s each) (which I doubt but I just want to point out the formula) Then you would need a SAN that could handle about 300MB/s

So first of all..the combined total bandwidth of the storage should be able to do the speeds listed above

Second of all.. they should be fibre channel because when your sharing storage a fibre fabric can easily handle this.


Hope this helps and isn't to crazy confusing. ;)

BTW last time I checked on 10Gb pricing an 8 port 10Gb switch was about the same price as 2 x 16TB 4Gb Fibre Channel RAIDs (25k+)
*update after just checking it now (IBM) it's down to 1 x 16TB 4Gb Fiber channel RAID.


Shane Sokolosky

Consultant / Systems Engineer

XSAN for Video Apple Certified Technician
Apple Consultants Network - Storage Area Networks
Apple Developer Connection

SANtech.TV
Office: 714-639-3767
Mobile: 714-599-1611

shanesky@santech.tv
http://www.SANtech.tv


Return to posts index

Sean ONeil
Re: 10gb Experiences?
on Feb 3, 2009 at 4:46:55 pm

[Shane Sokolosky] "10Gb is like crazy fast, your CalDigit RAID card and 16 regular SATA disks set up for RAID-5 couldn't deliver even half the speed that you can fit down a 10Gb pipe (which I think that's what Bob Zelin was referring to about 180Mb/sec maybe more but..), to really take advantage of 10Gb you'd need some serious storage,"

So you think 16 disks aren't enough, or that the CalDigit card cannot handle that kind of speed? When these larger disk sizes come out every other week, my understanding is that they also have the added benefit of being faster. The new 2TB Western Digital drives are like 90MB/s sustained. You don't think 16 of those could go over 800MB/s? What about 32 of them using two CalDigit cards? Or what about using 15k RPM SAS disks?

[Shane Sokolosky] "can easily be obtained by 4Gb Fibre Channel at a fraction of the cost of 10Gb."

I'm gonna have to go ahead and disagree with you on that one. Fibre requires expensive dedicated storage (as opposed to just using a server with a CalDigit card) as well as licenses for Xsan or Metasan. 10gb with file sharing does not require those things at all. I know 10gbe switches are still astronomical, but my proposed setup does not require a switch.


[Shane Sokolosky] "
If you tried to use "Gigabit Ethernet" and some kind of Internet Protocol (IP) based file sharing service such as SMB, Apple File Sharing (AFP), or something like that then there's no way to pull off HD like your talking about (If your looking at some of Bob Zelin's solutions with gigabit ethernet, then sharing is possible but not at that bandwidth that you'd need.)"


I guess that was my question, so thanks for the info. I was worried about that. However I believe there is a way for it to work well, but one would need 10gbe drivers that support TCP offloading or at least segmentation offloading. I don't think that option is available on Mac OS X, but I've heard from several people that Snow Leopard is going to have drastically improved TCP features and performance. So maybe it can happen down the road.

[Shane Sokolosky] "If you actually tried 1Gb fiber channel or Ethernet cables using iSCSI (both SCSI Protocols) you'd of probably came closer but still not quite enough unless you trunk the ports (and by trunk I mean probably 3)."

Yes I've been down that road. In the case of iSCSI I agree with you. Fibre Channel makes more sense at that point since iSCSI still requires a SAN metadata system.

[Shane Sokolosky] "BTW last time I checked on 10Gb pricing an 8 port 10Gb switch was about the same price as 2 x 16TB 4Gb Fibre Channel RAIDs (25k+)"

Right, but I wouldn't use a switch. Just four ports in the server, one port for each client. The dual-port cards from Small Tree are very reasonable. And I would only need CX4 since all our computers are in the same machine room (we use Cat5 extenders to send DVI and USB to the edit bays).

[Shane Sokolosky] "Hope this helps and isn't to crazy confusing. ;)"

Thank you!

Sean


Return to posts index

Bob Zelin
Re: 10gb Experiences?
on Feb 5, 2009 at 4:26:32 am

Sean writes -
So you think 16 disks aren't enough, or that the CalDigit card cannot handle that kind of speed? When these larger disk sizes come out every other week, my understanding is that they also have the added benefit of being faster. The new 2TB Western Digital drives are like 90MB/s sustained. You don't think 16 of those could go over 800MB/s? What about 32 of them using two CalDigit cards?


Sean is correct. 8 7200 RPM SATA drives with a modern SAS/SATA Raid controller are doing just over 600Mb/sec, so 16 drives - forgetaboutit. 180Mb/sec is the limitation of ethernet protocol. This is why AoE and alternate methods are appealing. you can do faster than 10G ethernet, and ethernet aint' gonna speed up !
Don't worry - this is day 1 of 10G ethernet. Belden and all the others want 10G ethernet (10GX) to succeed, and obsolete fibre.
It will happen, and probably soon. Running copper cable is very appealing to every industry - not just ours.

bob Zelin




Return to posts index


Sean ONeil
Re: 10gb Experiences?
on Feb 5, 2009 at 5:35:11 am

Bob, to quote someone just highlight the portion of their message and then hit "q" on your keyboard :).

Sean


Return to posts index

Shane Sokolosky
Re: 10gb Experiences?
on Feb 5, 2009 at 9:16:39 am

Well if the drives are fast enough, 10Gb Ethernet is not the bottle neck.

Editshare can do it. I dont see why your saying 180MB/s is the max throughput for 10Gb Etherenet.

Shane Sokolosky

Consultant / Systems Engineer

XSAN for Video Apple Certified Technician
Apple Consultants Network - Storage Area Networks
Apple Developer Connection

SANtech.TV
Office: 714-639-3767
Mobile: 714-599-1611

shanesky@santech.tv
http://www.SANtech.tv


Return to posts index

Shane Sokolosky
Re: 10gb Experiences?
on Feb 5, 2009 at 10:24:25 am

[Bob Zelin] "Belden and all the others want 10G ethernet (10GX) to succeed, and obsolete fibre.
It will happen, and probably soon. Running copper cable is very appealing to every industry - not just ours."


10Gb Ethernet is still going to use fiber optical cables if you want to go any kind of distance with it.[Bob Zelin]
I could say you could run fibre over copper also same difference.

"Sean is correct. 8 7200 RPM SATA drives with a modern SAS/SATA Raid controller are doing just over 600Mb/sec, so 16 drives - forgetaboutit"

Yea the drives are pretty fast, I think that's an overestimate (maybe a burst rate of a RAID 0 / JBOD, but sustained I agree it should be plenty fast enough. Don't know what I was sayin, late night maybe?
(as a side note, a box that get's 600MB/s only gets 180MB/s even with a 10Gb pipe in a sharing environment isn't that fast, some poeple ask why fibre channel? well this is the answer).

So I think the confusion sets in when talking about connections and different types of networks. The term "Fibre" can be used to reference a cable and a type of network protocol. The term "Ethernet" can also be loosely to describe both as well.

If we start with the storage and look at how it works and why it's fast, it communicates with the machine it's directly attached to via SCSI commands.

If we look at "Fiber" referring to the Fibre Channel Protocol which transports SCSI commands, it makes sense that your going to get the most out of your storage in a network like this with the greatest of ease and most likely all the bandwidth your storage has to over, so if we see Sean's drives as capable of doing what he wants then this would be the way to go, no overhead and simple.

If we look at "Ethernet" referring to the TCP/IP Protocol which was made to communicate with packets which can be received out of order, then why would you want your SCSI commands from your storage to go this translation and ask for the highest amount of speed? There are otehr file sharing protocol that we were talking about such as AFP, SMB, etc. these are based on the TCP/IP Protocol with modifications.

It doesn't make sense why you would want to make a SAN even more complicated than it should be.

The argument over connection types are invalid, as you can setup TCP/IP over fiber and also you can use iSCSI over ethernet, hence the fibre protocol over ethernet, and the "ethernet" protocol over fibre.

Both are standards devised by different groups for different reasons.

So I guess I could careless which kind of cable is being used, but I would care about which protocol is being used, especially in a case where I'm trying to eek out as much performance as I can from the storage. If I didn't care about performance then why get fast storage that's not going to be taken advantage of, kind of like Sean's storage it's plenty fast to capture that one stream of 4:4:4 uncompressed (220MB/s) but it can do 600Mb/s! according to the spec?

Fiber Channel FTW!









Shane Sokolosky

Consultant / Systems Engineer

XSAN for Video Apple Certified Technician
Apple Consultants Network - Storage Area Networks
Apple Developer Connection

SANtech.TV
Office: 714-639-3767
Mobile: 714-599-1611

shanesky@santech.tv
http://www.SANtech.tv


Return to posts index


Shane Sokolosky
To Aviod Further confusion
on Feb 5, 2009 at 10:39:16 am

What you and Sean are setting up is called a NAS system, meaning Network attached storage.

Not a SAN.

The difference being that in a NAS you have your storage behind a server, where your server is doing the sharing. In a SAN your storage is directly attached to all your clients, hence the reason for SAN management software.

Shane Sokolosky

Consultant / Systems Engineer

XSAN for Video Apple Certified Technician
Apple Consultants Network - Storage Area Networks
Apple Developer Connection

SANtech.TV
Office: 714-639-3767
Mobile: 714-599-1611

shanesky@santech.tv
http://www.SANtech.tv


Return to posts index

Sean ONeil
Re: To Aviod Further confusion
on Feb 5, 2009 at 7:40:18 pm

Shane,

That's correct, we were not talking about a SAN. If I were going to do a SAN system today I personally would choose fibre channel rather than 10gbe with iSCSI or AoE. I agree with you there. But we were discussing AFP file-sharing, which is not possible over Fibre Channel. There is a protocol for "IP over FC" as you mentioned but it doesn't exist in the Mac OSX kernel and surely never will. If it did exist, I'd be using it right now.

Yes AFP is less efficient than block-level SAN. But it's also a lot less complicated, less expensive, and makes it much easier to use your own storage devices that can easily be upgraded down the road. It might make more sense for smaller places.

The reason Bob was only getting 180MB/s is surely the limitations of AFP on OSX Leopard. If one were to use NFS sharing over 10gbe on Linux or Solaris, it would no doubt be 3 times faster at the very least. Based on what I've read, AFP is inefficient and the TCP stack in Leopard sucks. But I've also heard that Snow Leopard is going to drastically improve this. We shall see. I think people at large facilities will still be using SANs for years to come. But when 10gbe becomes more common I think it'll be more than enough to satisfy the needs of small shops.

As far as copper vs. fiber interconnects, it's just a matter of distance. All the Macs at my place are next to each other in a machine room, so if I were to get 10gbe adapters I'd only need the copper CX4 ones.

Sean


Return to posts index

Bob Zelin
Re: To Aviod Further confusion
on Feb 5, 2009 at 11:42:35 pm

Shane -
do you know why people discuss this so much. Because people don't want to spend any money. People want cheap. And they want easy. Do you know why they want easy ? Becuase they don't want to hire people like us. They want a cheap system, that will run without administration. This is why XSAN is a failure (relatively speaking). No small company wants to deal with it, unless they employ you or Mark Raudonis.

CX4 can only run 15 meters. CAT5a promises a normal 100 meter run for 10Gig ethernet. I have NEVER EVER run cables (including audio and video) more than 200' at ANY facility I ever worked at in NY or Florida, so the idea of fragile, expensive Fibre (along with all the stuff that goes along with it) means nothing to me. However, I like staying employed, so COST is the #1 factor - performance (like enough to handle DPX files) is very unimportant.

If you observe the AV industry, CAT5e and CAT6 has taken off because they don't need expensive double shielded coax cable for hi def, and they can use "telco installers" to do the wiring, instead of video professionals. Labor is very very very important in decision making for corporations, and the ability for "Joe Schmo" to install Belden 10GX for 10G transmission is A LOT MORE APPEALING than the trained expensive guy that knows how to terminate a fibre cable. 10 Gig will win - end of story. I know fibre is teriffic.

But that's just my world.

Bob Zelin




Return to posts index


Sean ONeil
Re: 10gb Experiences?
on Feb 5, 2009 at 6:52:16 pm

[Shane Sokolosky] "you can setup TCP/IP over fiber"

I would do this in a heartbeat if I could. But my understanding is that Solaris is the only OS that supports this.



Sean


Return to posts index

Shane Sokolosky
Re: 10gb Experiences?
on Feb 6, 2009 at 3:49:47 am

I still think it's less expensive and less complicated to go with fibre, although your initial investment was not fiber, but you could get a 4Gb fibre RAID for the same amount you paid for the Cal Digit.

The only components that you'd need after that to have a 3 seat SAN would be :

3 Seats of MetaSAN
3 Apple HBA's (copper cables included)
and a Qlogic SANbox 1400 switch

and only costs

about $6,800

From a tech perspective 10Gb E is cool and high tech, but from Bob's perspective I don't see ethernet being any easier. this is a REAL solution for you if you REALLY wanted to do what you originally were trying for with 3 people editing HD. I also bet if you wrote an instruction manual on how to install this stuff it's be shorter than Bob's tutorial here on the cow.

If we don't want to put 7k towards a solution to take full advantage of a HDCAM SR Deck and 3 editing stations then we might want to rethink a strategy altogether.

Shane Sokolosky

Consultant / Systems Engineer

XSAN for Video Apple Certified Technician
Apple Consultants Network - Storage Area Networks
Apple Developer Connection

SANtech.TV
Office: 714-639-3767
Mobile: 714-599-1611

shanesky@santech.tv
http://www.SANtech.tv


Return to posts index

Sean ONeil
Re: 10gb Experiences?
on Feb 6, 2009 at 5:23:41 am

[Shane Sokolosky] "4Gb fibre RAID for the same amount you paid for the Cal Digit"

The CalDigit RAID card costs $470. I would provide storage I already own.

[Shane Sokolosky] "3 Seats of MetaSAN
3 Apple HBA's (copper cables included)
and a Qlogic SANbox 1400 switch

and only costs

about $6,800"


Plus the FC storage, which would be over $15k for the speed we're talking about. So like $22k total

[Shane Sokolosky] "From a tech perspective 10Gb E is cool and high tech"

That's pretty much all I intended this discussion to be about.

[Shane Sokolosky] "Bob's perspective I don't see ethernet being any easier."

I'm totally baffled by how you could possible think that!?! I used MetaSAN for two years. I still have four licenses of version 2. It's vastly more complicated than sharing storage over AFP. And you need a dedicated extra network for metadata. And then of course there's configuring the FC switch. I'm not knocking it or calling it overly difficult. But to say it's EASIER than AFP file sharing - that's crazy.

[Shane Sokolosky] "this is a REAL solution for you if you REALLY wanted to do what you originally were trying for with 3 people editing HD"

I totally agree. But I'm not shopping to address an immediate need. Sorry if I wasn't clear. 10gbe adapters suddenly became within the price range to where I would buy a setup like that now just for the luxury of it - if it worked well. It doesn't, so I'm not.

Sean


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]