FORUMS: list search recent posts

OS X 10GbE Read Terrible, Write Amazing

COW Forums : SAN - Storage Area Networks

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Chris Cantey
OS X 10GbE Read Terrible, Write Amazing
on Jul 9, 2015 at 3:54:41 am
Last Edited By Chris Cantey on Jul 9, 2015 at 7:54:19 am

Does that make sense to anyone else?

We've been using an ATTO FastFrame NS12 in a 2010 Mac Pro for a while. BlackMagic Disk Speed Test has always shown read and write to our AFP server at over 600 MB/s.

This week we switched to a new ATTO ThunderLink NS2102 attached to a new 12-core Mac Pro. Read speed is now 20 MB/s (yes, 20) but write speed is actually better at over 800 MB/s.

We've tried tweaking TCP settings per ATTO and alternatively per several AFP-related tuning suggestions. Turning off delayed ACK gets us as high as 40 MB/s

We actually get nearly-full 10 GB/s in both directions when doing IPerf tests with the server as well as over SCP.

Gigabit runs fine from the same machine (read and write).

FWIW, we have been using a 1500 MTU on the old card and the new device, as well as the server and switches. 9000 doesn't seem to change the result.

I will note that our 2010 Mac Pro was still running Mountain Lion and our new system is on Yosemite. And of course this is actually a thunderbolt device.

The throughput is there but something seems to have changed between OSs, Macs, or ATTO devices. But why the high write and oddly low read?

Has anyone else seen anything like this with 10GbE on Yosemite/Thunderbolt/new Mac Pro?


Return to posts index

Bob Zelin
Re: OS X 10GbE Read Terrible, Write Amazing
on Jul 9, 2015 at 11:47:35 am

you give no information in your post. What is the other computers card. What is your server. What is your switch. How is your switch configured ? And what company do you work for ? In US or UK ?

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Chris Cantey
Re: OS X 10GbE Read Terrible, Write Amazing
on Jul 9, 2015 at 5:20:36 pm
Last Edited By Chris Cantey on Jul 9, 2015 at 5:45:44 pm

Hi Bob,

I did mention the other card, right at the top: ATTO FastFrame NS12

I work for a company called wonders, we're in the U.S.

The switch is a Dell Powerconnect 8132F. Currently the workstations are connected via fiber with SFP+ LC SR modules. The servers connect with SFP+ copper. What else would be helpful to know about its config?

Our sever is a custom NAS using Netatalk for AFP.

I should clarify that I truly am just looking to see if anyone has encountered this one before. The hope is that someone has, they have some possible solutions, and they are available for hire (consulting, possibly remote assist).


Return to posts index


Bob Zelin
Re: OS X 10GbE Read Terrible, Write Amazing
on Jul 9, 2015 at 11:44:13 pm

Hi Chris -
I wish I could help you - believe me, I want your money, but I am
not qualified to help you.

1) I have no idea what Netatalk even is.
2) I am unfamiliar with the 8132F switch

I can tell you that if the switch is configured for Jumbo Frames (not MTU 1500) and the card is configured for Jumbo Frames, you should be getting incredible performance. I use AFP, and SMB all the time. NFS has crappy performance, but AFP and SMB work wonderfully with these products, using Win 7, Win 8.1 Pro and OS X.
I have also used Linux servers, using these cards on clients, and they also have wonderful performance.

If I had the knowledge of the Powerconnect 8132F, I would remote in, to check your switch configuration. But I am not qualified to do so. I also do not know what Netatalk is, so I am once again incompetent to assist you with this.

As I often mention - you are suffering thru the growing pains of a do it yourself system. Had you purchased an existing proven system, the company you had purchased from would have done all this legwork for you, and you would not be suffering. This does not mean that you do not have a wonderful system - but every piece of hardware has its variables and it's "tweeks" and without suffering thru this, you never know what is going to happen. On the gear that I work on, I go thru the same "growing pains", but I eventually figure things out (usually with the help of other manufacturers). I am afraid that I just don't know how to help you with this.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Chris Cantey
Re: OS X 10GbE Read Terrible, Write Amazing
on Jul 10, 2015 at 12:50:43 am

Thanks! And thanks for being frank.

Netatalk is the open source AFP server software that most of the off-the-shelf / custom built *nix based system are using.

Yes indeed these are the pains of custom solutions. We've had a lot of luck with them over the years so we keep at it. Overall, the pains of this custom build have been small compared to the gains. This one has been especially successful until this last turn with switching out the ATTO card for a Thunderbolt solution.

Thanks again for the input.

If anyone else has seen anything like this let me know. The excellent write and terrible read is what has been perplexing. Especially since we're only seeing it after moving off of the PCI-based 10GbE card and/or to Yosemite. And only with the AFP protocol (vs raw network speed).


Return to posts index

Chris Cantey
Re: OS X 10GbE Read Terrible, Write Amazing
on Jul 10, 2015 at 6:52:16 am

Ok, here is the/a solution, for the later Googlers.

Ultimately we reached 1025 MB/s read and the same 850 MB/s write. We just had to tune the client sysctl.

We used the following:

net.inet.tcp.doautorcvbuf=0
net.inet.tcp.doautosndbuf=0
net.inet.tcp.sendspace=4194304
net.inet.tcp.recvspace=4194304
net.inet.tcp.maxseg_unacked=8
net.inet.tcp.delayed_ack=0
net.inet.tcp.win_scale_factor=7
net.inet.tcp.rfc1323=1


For our 2013 Mac Pro with ATTO Thunderlink NS 2201, these are what did the trick. The critical difference between this and what is often suggested (as well as what ATTO includes by default with their driver install) is that we did not set the kern.ipc.maxsockbuf to 8388608. We just left it off. It's odd because the default is exactly this value, but hard setting it caused the read speeds to drop back to 20-40 MB/s.

The other settings seemed to have a smaller effect on the speed but they did inch it up. For example, maxseg_unacked=8 vs 32, gave about 80 more MB/s on the read. And the same for delayed_ack=0 vs 2 as well as the win_scale_factor=7 vs 3. And rfc1323 needed to be on, though this is the default.

The send and recv space definitely had an effect vs the default but these (or around these) are the commonly offered 10 gbe tuning values we've seen.


No switch changes were necessary and nothing on the server end. Jumbo frames were not necessary to reach these speeds. Our MTU is still 1500 end to end.


Return to posts index


Bob Zelin
Re: OS X 10GbE Read Terrible, Write Amazing
on Jul 10, 2015 at 12:00:23 pm

you are very generous Chris. This is great info, and I am sure that most of "the players" are gobbling this up right now (that read Creative Cow, which is most of them).

I owe you one.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Simon Blackledge
Re: OS X 10GbE Read Terrible, Write Amazing
on Aug 6, 2015 at 1:41:22 pm

net.inet.tcp.rfc1323=1

I don't have this setting in any of my machines sysctl.conf is it an Atto thing ?

s



Return to posts index

Olivier Jean
Re: OS X 10GbE Read Terrible, Write Amazing
on Oct 29, 2015 at 11:54:37 am

Great info Chris,
and thank you for sharing your findings.

I'm setting up a Mac Pro Based NAS Graphic Server/ Video server
Mac Pro Cylinder 10.10.5
Atto Thunderlink Fibre 16GB to an Infortrend 16 Bay ( R/W 1.1GB/S )
SANLink2 10GBase-T peer to peer to clients ..

Protocol used is AFP,

I've used a couple of systcl.conf, your settings works quite well within our scenario,
I've managed to get some really nice speed out of the system however under AFP connection the read and the write speed is ramping up where as under SMB connection the speed ( 750MB ) kicks in instantaneously !!
I can't work out for the life of me this issue any suggestions are welcome ...

I have not seen Speed ramping issue on any system i have previously configured ..





Regards
Olivier Jean
Video Sales Consultant
Apple Certified Trainer Final Cut Pro 7
Mac Centre
Sydney Australia


Return to posts index


Steve Chandler
Re: OS X 10GbE Read Terrible, Write Amazing
on Dec 9, 2015 at 11:47:59 pm

"Never a good idea" until you actually need to change them!

Thanks a lot for sharing this, Chris. We are experiencing similar issues with a 10GbE XSAN, albeit much less pronounced that yours. We're averaging around 600 MB read, which sounds great - until you realize there is a 2-3 second "ramp-up" in speeds. Looking at a graph, our actual sustained read is closer to 900, but the ramp-up screws with our numbers. Something is obviously not right, and I want to fix it before it actually becoms a problem for us. I tried coming up with my own sysctl numbers that failed spectacularly - but included the kern.ipc.maxsockbuf parameter. My hope is that that was the issue, and combined with your updated numbers we will see a change. I'll report back here.


Return to posts index

Elvin Jasarevic
Re: OS X 10GbE Read Terrible, Write Amazing
on Jul 25, 2015 at 4:48:15 pm

Read in general should be always better than Write and changing the TCP/IP settings/hacking the system is never a good thing.

Having a software driver is ALWAYS BETTER than having no software installed, but in this case ATTO have done bad job.

Speak to ATTO and give them yellow card!!


elvin@ddp.jp
http://www.ddpsan.asia


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2019 CreativeCOW.net All Rights Reserved
[TOP]