FORUMS: list search recent posts

Troubleshooting 10GbE sluggishness over Thunderbolt

COW Forums : NAS - Network Attached Storage

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Troy Williams
Troubleshooting 10GbE sluggishness over Thunderbolt
on Nov 14, 2017 at 1:25:12 am
Last Edited By Troy Williams on Nov 14, 2017 at 1:53:08 am

In preparation for rolling out a new high speed network storage system at my facility, I'm getting 10GbE deployed at all of our workstations. However, I seem to be getting sluggish performance on 10GbE cards connected via Thunderbolt. I'd like to know if anyone is encountering similar experiences, or if anyone has some troubleshooting suggestions.

To test, I'm using two SmallTree P2E10G-2-SR 10GbE network cards and I'm testing them in multiple configurations. A TwinAx DAC cable connects them directly to each other (no switch), jumbo frames are enabled (MTU 9000), and SmallTree driver 3.3.15 is installed. The command-line program "IPerf" is used to measure raw throughput from Computer A to Computer B.

--Config 1:
Computer A: MacPro5 (macOS 10.11.6) with 10GbE card installed directly to the PCIe bus.
Computer B: MacPro5 (macOS 10.11.6) with 10GbE card installed directly to the PCIe bus.
Result: 10Gb/s (full line speed)

--Config 2:
Computer A: MacPro5 (macOS 10.11.6) with 10GbE card installed directly to the PCIe bus.
Computer B: iMac 2015 (macOS 10.11.6) with 10GbE card connected via Thunderbolt-2 to a Sonnet Echo Express SEL
Result: 6.3Gb/s

--Config 3:
Computer A: MacPro5 (macOS 10.11.6) with 10GbE card installed directly to the PCIe bus.
Computer B: MacBook Pro 2012 (macOS 10.12.6) with 10GbE card connected via Thunderbolt-1 to a Sonnet Echo Express SEL
Result: 2.5Gb/s

I've also swapped the Echo Express SEL with an Echo Express SE II, as well as with an AKiTiO Thunder2 PCIe Box. Results are the same. I've also tried downgrading to driver 3.3.11 with no improvement. I also have additional 10GbE cards and have swapped them out in case the problem was the cards themselves, but results were the same.

All signs are indicating bandwidth starvation of the Thunderbolt bus, especially given the fact that TBolt-1 is even slower than the TBolt-2 results, but there's nothing else connected except the 10GbE card. I've seen people report getting better results over Thunderbolt before, so I can't help but wonder if I'm overlooking something somewhere.

Suggestions and troubleshooting tips welcomed. Thank you!


Return to posts index

Bob Zelin
Re: Troubleshooting 10GbE sluggishness over Thunderbolt
on Nov 14, 2017 at 1:48:30 pm

I have absolutely no idea of what on earth you are doing, or trying to accomplish.

you write -
"a new high speed network storage system at my facility"
what high speed network storage system at my facility ? A Small Tree, a Facilis, or are you building it yourself ?

you are using Small Tree cards. Have you called Small Tree, who offers excellent support ?

And why on earth are you using Twinax cables, and why did you purchase Small Tree cards with Twinax connectors (SFP+). In 2017, the world uses Cat 6 and Cat6A cabling for Ethernet connection. Is your switch going to be 3 meters away from all of your editing systems ? Exactly what switch are you planning on implementing ? A Cisco 10GbE SFP+ switch. So you will add 10G SFP+ transceivers to every one of these cards ? Isn't this a little ridiculous in 2017 ?

In your test results, you specify your iPerf results, but you do NOT specify what you are testing. I don't give a crap about line speed from iPerf. You have to test against a RAID array to verify that you will have speeds that will be acceptable for professional video editing. SO - are you testing the performance of the single internal boot drive of these computers ? Well, that makes no sense.

In Configuration # 1, you say that you are getting 10Gb/sec.
In Configuration # 2, you say that you are getting 6.3 Gb/sec
In Configuration # 3, you say that you are getting 2.5 Gb/sec

10 Gb/sec is 1250 MB/sec. You will never achieve this speed, not even with Thunderbolt 3. Typical performance with properly working computers is 800 MB/sec. With Thunderbolt 3, you will see a MAXIMUM of 1100 MB/sec in real life - and that's with a new iMac or MacBook Pro with a Promise SanLink3 thunderbolt 3 to 10G adaptor.

6.3 Gb/sec is 787 MB/sec is EXCELLENT for Thunderbolt 2 performance. You will be lucky to maintain this performance. RED 6K Media (Red Epic Dragon with 5:1 compression) only requires 143 MB/sec. 6.3 Gb/sec is nothing to complain about.

2.5 Gb/sec is 312 MB/sec. Gee - your other computers are working properly, MAYBE ITS THE COMPUTER ? Maybe you are testing with SMB, and did not disable SMB signing. Maybe you should install a REAL RAID ARRAY to test against. Even with that said, you are still doing 3 times the rate of a 1G connection, so your editing systems will still work.

I am not sure what this "micro management" testing is all about. You buy a system from a manufacturer, and you make it work. If you have problems, you call that manufacturer. If you are going to build this system yourself (which I bet money that you are), you will fail.

I am looking at your #3 Config test results again. You are using a MacBook Pro with Thunderbolt 1. You should be jumping up and down that you are getting 312 MB/sec.

This is my advice to you. Stop this nonsense. Buy a system from SOME company that does this for a living. They will help you - all of them will - Small Tree, Facilis, EditShare, DDP, GB Labs, ProMax, Studio Network Solutions, etc. Your job is to provide your facility with a WORKING SYSTEM that everyone can successfully edit with. And buy a normal switch like a Netgear XS series, not the Cisco that you intend to use, so it's easy to trouble shoot.
No one wants to see your iPerf results. You need to do tests in real world situations against a RAID array.
Even if you were to achieve 1250 MB/sec per system (which you will never do with 10GbE), 3 computers running at that speed would saturate the bandwidth of a 16 bay SATA RAID array, and your system would choke. I am sure that you are not running an all SSD array for your final system.

With all of that said, your system is running fine, except for #3, and that's thunderbolt 1 in an old MacBook Pro laptop, so no one cares.


Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Troy Williams
Re: Troubleshooting 10GbE sluggishness over Thunderbolt
on Nov 15, 2017 at 10:29:51 am

Hi Bob. Always a joy to read your rather spicy flavor of advice.

I intentionally did not go into detail on the storage system because I did not wish to invite critique, nor are such details relevant at this time. The system in question is a small proof-of-concept in a lab to identify problems before going full scale -- and yes, in fact, it is all SSD. The final system will also utilize Tiger Spaces. It's going quite well except for this odd issue with Thunderbolt. That said, I'm well aware of your loathing and scorn for those who wish to pursue the DIY path. Objection noted and lovingly ignored.

I'll provide a little further detail though -- The MacPro5,1 can access the test array at 1.1GB/s over NFS, according to AJA System Test. The fact that I am not seeing the same results over Thunderbolt 2 indicated a problem, which meant taking steps to isolate the problem.

While 6.3Gb/s is "good enough" performance, I have seen multiple screenshots here from people reporting better over Thunderbolt 2. There is a bottleneck somewhere in the config, and though the performance is "good enough", that bottleneck could result in other issues down the road.

The configurations listed in my previous message were strictly troubleshooting configurations with the network adapters direct connected to each other. No switch and no array, though for what it's worth I did also test with a switch -- specifically, a Dell N4032 with jumbo frames enabled.

The network performance tool I used (iPerf) generates raw TCP packets and does not read/write to storage at all. One system runs iPerf in transmit mode, the other runs in receive mode. Hard drive throughput is not a factor at either end, nor is protocol overhead as is inherent to NFS, AFP or SMB (the latter of which I'm fully aware is hindered in OSX 10.11 and up).

As for why I don't contact SmallTree directly, the cards were acquired secondhand from a studio no longer in business. I don't know if SmallTree will support secondhand product. I will be ordering a Sonnet Twin 10GbE SFP+ as a control reference for the sake of determining if I somehow have flawed cards, but that would surprise me greatly.

Lastly, I *do* do IT for a living. Please do not make assumptions. You don't know me, my qualifications, my experience, my plans, or the situation. Also, your entire post seemed to serve little purpose but to promote your turn-key solution friends and to sow fear about even the mere thought of pursuing a DIY path. Though I respect your knowledge and experience, you seem to make this forum an astoundingly hostile environment for people who might wish to learn something about data storage.


Return to posts index


Bob Zelin
Re: Troubleshooting 10GbE sluggishness over Thunderbolt
on Nov 17, 2017 at 2:27:18 pm

sorry for the delay - I actually have to work once in a while -
you write -

" The MacPro5,1 can access the test array at 1.1GB/s over NFS, according to AJA System Test. The fact that I am not seeing the same results over Thunderbolt 2 indicated a problem, which meant taking steps to isolate the problem."

REPLY - wow, I never get good results with NFS. I am surprised that you want to proceed with NFS and not SMB.
I only use NFS as a last resort on Mac's.


" While 6.3Gb/s is "good enough" performance, I have seen multiple screenshots here from people reporting better over Thunderbolt 2. There is a bottleneck somewhere in the config, and though the performance is "good enough", that bottleneck could result in other issues down the road."

REPLY - I have seen, that based on the generation of the computer, the results of Thunderbolt 2 (using the same damn thunderbolt to 10G adaptor) varies widely from Mac to Mac. This is very discouraging, and I don't have an answer for it. I have seen Thunderbolt 2 Mac Book Pro's dramatically outperform a Mac Pro 6,1 on the same network with the same Tbolt adaptor. I don't know why.



The configurations listed in my previous message were strictly troubleshooting configurations with the network adapters direct connected to each other. No switch and no array, though for what it's worth I did also test with a switch -- specifically, a Dell N4032 with jumbo frames enabled.

REPLY - you will find quite a jump when you enable an MTU of 9000. Many people "poo poo" jumbo frames but for 10G performance, it makes a big deal.


The network performance tool I used (iPerf) generates raw TCP packets and does not read/write to storage at all. One system runs iPerf in transmit mode, the other runs in receive mode. Hard drive throughput is not a factor at either end, nor is protocol overhead as is inherent to NFS, AFP or SMB (the latter of which I'm fully aware is hindered in OSX 10.11 and up).

REPLY - while SMB is hindered in 10.11 and up, you can disable SMB signing. You can turn of DS_Store. NFS, I don't know - I don't get great results. The LumaForge guys are the only ones who are getting really great performance with NFS.


Lastly, I *do* do IT for a living. Please do not make assumptions. You don't know me, my qualifications, my experience, my plans, or the situation. Also, your entire post seemed to serve little purpose but to promote your turn-key solution friends and to sow fear about even the mere thought of pursuing a DIY path. Though I respect your knowledge and experience, you seem to make this forum an astoundingly hostile environment for people who might wish to learn something about data storage.

REPLY - I figured that if you were already dealing with Tiger Spaces, you could simply get a nice Tiger Store or Tiger Serve system, that will work out of the box, without all the fuss you are going thru.
https://www.tiger-technology.com/appliances/

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Troy Williams
Re: Troubleshooting 10GbE sluggishness over Thunderbolt
on Nov 17, 2017 at 3:20:09 pm

[Bob Zelin] "REPLY - I have seen, that based on the generation of the computer, the results of Thunderbolt 2 (using the same damn thunderbolt to 10G adaptor) varies widely from Mac to Mac. This is very discouraging, and I don't have an answer for it. I have seen Thunderbolt 2 Mac Book Pro's dramatically outperform a Mac Pro 6,1 on the same network with the same Tbolt adaptor. I don't know why."

I chewed on this for a day and came up with a hypothesis. I'm suspecting that the PCIe to Thunderbolt adapter is falling back to PCIe 1.1 4x rather than using PCIe 2.0 4x. The former tops out at 8Gb/s, whereas the latter is 16Gb/s, which would perfectly explain the bandwidth starvation. On the Mac Pro it's connected on an 8x lane which even at PCIe 1.1 spec lets it achieve full bandwidth. I might drop it into a 4x slot and test how it does then. The card should support PCIe 2.0 though, as do the TBolt chassis, so it's puzzling. I'm tempted to order an ATTO card just to test how that performs over TBolt 2.

Also, a sidenote regarding the Macbook Pro with TBolt 1 -- I discovered that cold booting with the TBolt unit plugged in makes a difference. If hot plugged, it tops out at 2.5Gb/s, as previously stated. If cold plugged, I get 5.0Gb/s. I still think I should see better, but for TBolt1 I wont complain.

[Bob Zelin] "REPLY - you will find quite a jump when you enable an MTU of 9000. Many people "poo poo" jumbo frames but for 10G performance, it makes a big deal."

I understand the reasons why they do. It can affect smaller packet traffic such as basic internet on the same network. The solution is simple -- don't put them on the same network.

And yes, when testing the two direct connected workstations, Jumbo Packets (MTU 9000) were enabled at both ends.

[Bob Zelin] "REPLY - I figured that if you were already dealing with Tiger Spaces, you could simply get a nice Tiger Store or Tiger Serve system, that will work out of the box, without all the fuss you are going thru."

Almost did, until Bernard said "NTFS". An NTFS filesystem for a POSIX house? No thank you. Dealt with the headaches of that before, don't want a repeat.

And it's not fuss -- this is my idea of fun. I'd be doing this in my free time if I wasn't being paid to.


Return to posts index

Bob Zelin
Re: Troubleshooting 10GbE sluggishness over Thunderbolt
on Nov 17, 2017 at 11:33:26 pm

you still have not explained why you are going to use NFS. What are your client computers ?
Bob

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index


Troy Williams
Re: Troubleshooting 10GbE sluggishness over Thunderbolt
on Nov 17, 2017 at 11:43:43 pm

[Bob Zelin] "you still have not explained why you are going to use NFS. What are your client computers ?"

That was just for testing. What I'll be "going with" is whatever method Tiger's client software uses. Tiger Spaces will be using NFS between it and the data repository unless I have reason to use SMB. Doesn't matter to me either way. Client computers are all Mac.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]