10Gbe over fibre speed
We have just had new server installed for after effects and resolve work. The server is a dell power edge r720, raid5 array, 5 drives. It is connected with a fibre cable through pcie. Now when we were thinking of getting this, the idea was to reach speeds of about 250MB/s read and write. Right now we are getting 280 write and 180 read.
As you've probably noticed i'm no pro in these questions. I just want to know if anything can be done to speed it up, so i know what to ask from the company that fitted it.
All the best
whose 10G cards are you using ?
Rescue 1, Inc.
The card is a singleport Solarflare card.
What kind of storage controller does your r270 have? Also what OS is it running?
AFAIK some of the Dell Perc stuff is usually rebranded LSI kit... but not all! Its not impossible that the bottleneck is on the storage side of things... although with modern kit it really shouldn't be.
What I'd do:
- Benchmark the storage at the server end (if its linux use Bonnie++)
- If you are confident this issue is on the network side of things, use something like iperf for diagnostic and go from there.
The trouble is that there are lots of possible causes here!
At the end of the day if you've paid somebody for setup, get them back! Sounds like they need to come back and finish the job ;)
indiestor.com - "Avid project sharing, shared!"
ok, hang on Stephan -
so let me get this straight - you have a single port Solarflare card in the Dell Server and you have a single port Solarflare card in your client computer ? Or are you going into a switch ? All these little details make a huge difference !
Rescue 1, Inc.
Sorry i'm not too knowledgeable about this stuff so not sure which details to provide. Basically a Solarflare pcie card is in the macpro that i'm using. The fibre cable (with a few extensions, it's not a single wire) is going straight into the switch.
Thanks for the reply, don't have any experience troubleshooting this but at least now i've got a direction to work in. Thanks! Will write how I get on!
ok Stephan, you have now made yourself clear.
You have a Dell server, with a single port Solarflare card going into the 10G port of your switch. You now connect your client computers via 1gig Ethernet to this switch.
You are NOT going to get more than about 100MB/sec R/W speed over 1GbE. Just because you are using a 10G card in your server, does not mean that this will give you faster speed over regular 1GbE to the client computers. The theoretical bandwidth (line speed) of 1GbE is 125MB/sec, but no one gets this speed. How you were measuring even faster rates than this is beyond me. If you want faster speeds, you get a full 10G switch, and 10G cards for your client computers.
Rescue 1, Inc.
Crickey, it's sometimes is hard to explain things over the internet. Sorry, i'm not a native English speaker.
The macpro that I am using for grading has a solarflare pcie card, through which it is connected to the switch. Then another fibre cable runs out of the switch and goes into a slot into a the Dell r720xd Server (just found out that it's xd rather than just dellr720 like it says on the front of it).
Including the picture of the switch and the ports on the back panel of the server. According to the server manual the ports are two 10/100/1000 Mbps NIC and two 100/1Gbps/10Gbps SFP+/10GbE T connectors
Hope this explains a bit better =)
you have some "unknown" 10G card in the Dell server. this connects to the Netgear GS752TXS switch (one of the 4 10G SFP+ ports on the switch). You take another 10G SFP+ on the switch and connect to the Solarflare PCIe card in your Mac Pro. Now it's clear.
The Mac needs the Solarflare tuning file for increased speeds. Solarflare has one, I can assure you. What I don't know is if there are any tuning requirements on the Dell Win server side. I never setup a PC as the server with a 10G port - I have only used Mac's as the server.
Contact me privately if you want more info on the Solarflare.
Rescue 1, Inc.
I hope someone comes and backs me up since I have not posted in what seems far too long. However, to set the stage, perhaps my friends can reassure you to some, most, or all of my advice im going to give you. I do come with references in any case.
Im going to talk about what I know. Looks like Bob has your back also.
10Gb Ethernet is very capable of reaching the speeds you'll want to reach while working with many if not all of your Adobe, and similar "class" products.
It is capable of reaching speeds on a Mac and Windows of over 800MB sustainable and reliable in a single direction. Taking and doing that in two directions at once and is called BI Directional. (so you see numbers of between 1.6 to 1.9GBps for total flexibility... Thats what is called Near Line rate...
Dont forget to measure all your current overhead. You need to make sure you tie up all your bottlenecks in your network. Especially a Shared Network, which is why its a Standard IT scenario to run your business of graphics and pictures you rely on to keep going. You will need more stuff. Start building a diagram of your network.
So measure each client, all the connected online and offline connected also. Even though it is off, stuff is still sending little pings to make sure its actually still there on the network.
Make sure you know what each machine is doing, and how it's user is working. They will interact with the work flow. You will want all your usual ACL perms etc.
Now, its always more than my customer could ever chew off the first time too.
Were all here to give advice and help.
I have witnessed Ethernet vs Fibre Channel battles since much of the early 2000s, when I startedworking for SGI (Silicon Graphics Inc.). Ethernet will ALWAYS win the war. In all areas of Simplicity, Performance Scalability, Excellent Community-based Development and much more.
Ethernet can even win a place in Bob Zellins heart for useful gear. So Take our advice. Please.
The price differences are still knee jerking from super high to super low on configurables.
Everyone also has their soft spots.
Your Dell Servers are great for serving files. I KNOW those specific ones, and they are not the best at the highest performance without many in a cluster working in tandem. (which opens more performance and engineering needs to address also). (Have you ever tried running HELIOS on those?) Windows, as well as all their operating versions, and Mac, and Linux too, require tuning to have your performance sustainability match that of your networking loads, cpu, memory usage and more.
What tests have you run on your network? What tools did you use? Have you checked out Promax Windows specific platform scalable solutions?
For Mac Specific Workflows in all areas of High Performance Networking and Shared Storage, by far a wonderful product, did you check Small Tree perhaps? HAVE YOU CALLED THEM ALL YET?
HELIOS would be another company to talk to directly. Adobe, perhaps too. A lot of your places have personal consultants.
Hopefully you continue to post your progress, etc.
Sales and Marketing Consultant.
High Performance Computing Consultant
Digital Media, Film, and Video workflow Specialty.