FORUMS: list search recent posts

SFP+ Cable Issues, Qbiquit and Mellanox

COW Forums : NAS - Network Attached Storage

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Kevin Patrick
SFP+ Cable Issues, Qbiquit and Mellanox
on Nov 29, 2018 at 7:59:39 pm

I'm having issues with performance between an iMac Pro and a QNAP NAS.

QNAP checked my configurations, found no issues.
QNAP ran some performance checks interally and determined the HW setup is fast.
QNAP ran some low level network test and confimed the slow performance I'm getting.
QNAP elevated the issue.

I then talked to Ubiquiti, where they asked about the SFP+ cable I used. When I checked, I noticed that while it said it would work with Ubiquiti products, but no Mellanox. They did have another cable that would supposedly work with both.

QNAP is using Mellanox. Though to me, not work means nothing should work. My set up works, just slow.

So, just wondered if anyone here came across these kind of cabling issues.


Return to posts index

Steve Grappone
Re: SFP+ Cable Issues, Qbiquit and Mellanox
on Nov 29, 2018 at 9:05:44 pm

Sometimes it can be a simple MTU settings. I typically do not use 1500 and use Jumbo frames 9000 MTU.

I’d run iperf and post the results.


Return to posts index

Alex Gardiner
Re: SFP+ Cable Issues, Qbiquit and Mellanox
on Nov 29, 2018 at 10:03:45 pm

Yep, check MTU - the mellanox driver defaults to MTU 9000 in some environments, seen that before.

Ifconfig or ethtool on the linux side if poss, see what it's saying :-)

alex@indiestor.com


Return to posts index


Alex Gardiner
Re: SFP+ Cable Issues, Qbiquit and Mellanox
on Nov 30, 2018 at 9:22:05 am

EDIT: I woke up and realised I was thinking about the Myricom driver - this *does* set the interface to jumbo frames by default. Mellanox probably won't.

Either way check you don't have a mismatch, it'll cause headaches.

alex@indiestor.com


Return to posts index

Kevin Patrick
Re: SFP+ Cable Issues, Qbiquit and Mellanox
on Nov 30, 2018 at 1:04:03 pm

Both the iMac and the QNAP server needed to be changed to Jumbo frames, 9000 MTU. However, makeing the changes had no effect on performance.


Return to posts index

Steve Grappone
Re: SFP+ Cable Issues, Qbiquit and Mellanox
on Nov 30, 2018 at 1:34:13 pm

Hi Kevin,

Well that was unfortunate. Typically MTU settings are the issue. Is there a switch involved? If so send me the make and model and remember that MTU settings do need to be made on the switch.

Also I’d run iperf on the both directions server to client and then client to server. Iperf version 2 seems to be best and also supported under Mellanox so if you open a case with them they’ll want to see the version results.

The point of this test is to see if there is performance lost on both send and receive as some systems are setup to recieve and others to send.

Iperf works most OS’s too. Pretty simple to use

Server: iperf -s
Client: iperf -c server_ip -P 4

The capital P means how many processes to run, in this case we’ll run 4 I’d also do 3 and just the default. Lowercase p would to specify the port if the default is different.


Return to posts index


Kevin Patrick
Re: SFP+ Cable Issues, Qbiquit and Mellanox
on Nov 30, 2018 at 3:57:29 pm

Steve,

On a long call with QNAP, their tech guy did run iPerf. (something I'm not familiar with)

The iPerf results were basically the same as all the other basic tests. (Blackmagic and AJA)

Though I'm not sure if he ran it from the server to the iMac Pro. He's esculating it to QNAP HQ. Any advice on how to run iPerf on the server side would be appreciated.

For reference:
~ 560 MB/sec Write (many test sources confirm)
~ 350 MB/sec Read (many test sources confirm)
~ 630 MB/sec Read (testing the HW only within the server, so this is basically max perf)

It's odd that the read speeds are less (a lot too) than the write, as QNAP marketing posts the opposite.

I realize their is always overhead that keeps any real world results from getting anywhere near what the marketing guy says is theoretical maximum. But the read speeds are about half what they should be. Sadly read speed are rather important when editing.


Return to posts index

Steve Grappone
Re: SFP+ Cable Issues, Qbiquit and Mellanox
on Nov 30, 2018 at 4:29:34 pm

Hmmm. Well the writes being faster actually doesn’t surprise me too much. I think qnap uses ZFS which is a copy on write file system. Means it uses ram and or fast SSDs to cache the info then it rights it to the actual disk.

Do you have anything else plugged in to the iMac that uses TB? If so try testing without any other TB device connected.

Do you have another system with TB that you could also test with?


Return to posts index

Bob Zelin
Re: SFP+ Cable Issues, Qbiquit and Mellanox
on Nov 30, 2018 at 10:00:03 pm

wow -
lots of opinions, without asking many questions.

# 1 - exactly what QNAP model do you have ?

# 2 - QNAP checked your configuration ? There are a million configurations. Are you in a single static volume with all 8, 12 or 16 drives, or do you have a storage pool. And if you have a pool, is it thick volume or thin volume ?
If you have a 2 bay, 4 bay or 6 bay QNAP - forget it - you are wasting your time. I don't care what the QNAP salespeople or your dealer told you. You MUST have a minimum of EIGHT 7200 RPM drives spinning in a single RAID group to make this work with speeds fast enough for video editing.

#3 - how are you connecting to the iMac Pro ? The iMac Pro has a built in 10Gbase-T port. You should have a
10GbaseT card for your QNAP. All modern QNAP's have the ability to accept a PCIe card. The QNAP QXG-10G1T is only $89. It's the exact same Aquantia AQC-107 chip that is in your iMac Pro. Perfectly compatible.

#4 - you said SFP+ cable issues ! So with your iMac Pro that has a free 10G port, you purchased a thunderbolt 3 to SFP+ adaptor. Well that is wrong. Now, for the moment, I am going to ASSUME that you do not have one of the larger industrial QNAP products like the TVS-EC1680U-SAS-RP, which is a 16 bay with a dual port SFP+ card that comes standard. I am going to ASSUME that you have purchased a cheap QNAP TS-832X, or any Annapurna ARM based CPU. These QNAP products are GARBAGE and no matter what you do - no matter what QNAP told you - you are NEVER EVER EVER going to get good performance from them. If you don't have an Intel or AMD based QNAP with EIGHT drives, you are going to fail.

You could have simply asked me before buying, and purchase a TVS-873e for about $1250, and a QNAP
QXG-10G1T, plugged this into your iMac Pro, and you would be getting 1000 MB/sec READ (500 MB/sec write).

Now, let me assume again that you purchase a QNAP Product that their own press says "is great for video editing" -
like the TVS-453BT3 - well, they are lying. This product is NOT suitable for professional video editing.
QNAP makes TONS of products that are wonderful (and so does Synology, and so does Netgear) - and at the same time, they make "budget" products that are ok for running Plex server, and your home videos from your iPhone, but are NOT SUITABLE for professional video editing.

AND ON TOP OF ALL OF THIS - if you did not disable SMB signing on your iMac Pro and you are connecting via SMB, you are going to get crappy speeds.

Well - until you tell me the EXACT model that you purchased, and the exact model of SFP+ adaptor you have (and I sure hope its not an Akitio) - then I can give you further analysis of what your problems are. But like I said - if you have a 2 bay, 4 bay or 6 bay product - or if you have an Annapurna based QNAP product - it's never going to do what you want it to do.

You buy the right product and it will be great. And QNAP makes some great stuff (and they make some not so great stuff).

OH - and take that Ubiquiti switch out of the equation until you get this resolved. It's just adding more to the confusion of diagnosing all of this.

Bob Zelin




I'm having issues with performance between an iMac Pro and a QNAP NAS.

QNAP checked my configurations, found no issues.
QNAP ran some performance checks interally and determined the HW setup is fast.
QNAP ran some low level network test and confimed the slow performance I'm getting.
QNAP elevated the issue.

I then talked to Ubiquiti, where they asked about the SFP+ cable I used. When I checked, I noticed that while it said it would work with Ubiquiti products, but no Mellanox. They did have another cable that would supposedly work with both.

QNAP is using Mellanox. Though to me, not work means nothing should work. My set up works, just slow.

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index


Kevin Patrick
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 1, 2018 at 2:25:21 pm

Hi Bob. Thanks for replying.

Given the volume of help you provide here on the Cow, you probably don't recall helping me out a few weeks ago when I first set up our QNAP. (see my configuration below) I started this post to specifically discuss the SFP+ cable.

Interesting that the QXG-10G1T card you recomend has the same controller as my iMac Pro. I'd gladly replace the pre-installed card if this either addressed the performance
issue, or simply made the system happier.

While we probably can't rule out the Ubiquiti switch as being an issue, it does allow write speeds much higher than read speeds. Could a switch even tell the difference?

QNAP checked to make sure that I had the correct MTU/Jumbo frames set correctly on both the iMac Pro and the server, which I did.

At this point, QNAP and Ubiquiti are esculating this performance issue. I hope to hear back from QNAP Monday.

I did order another SFP+ cable. This new version (from the same vendor) does state that it's compatible with both Ubiquiti and the Mellanox controller. I don't have a high level of confidence this will address the issue. I'd guess this is a works or doesn't work issue, as opposed to a write is faster than read issue.

TS-873U-RP
Dual Port 10 Gbe SFP+ (preinstalled)
- I’m guessing it’s the LAN-10G2SF-MLX
- Mellanox MT27520
8 x Hitachi 8 TB Drives, HUH728080ALE600
One Static Volume
(no Storage Pool)

10Gtek Cable from QNAP to Switch
This is the cable in question, as 10Gtek says this is compatible with Ubiquiti but not compatible with the Mellanox chip in the QNAP 10Gbe card.

US‑16‑XG UniFi 10Gbe Switch

iMac Pro connected via RJ-45 Cat 6 to Switch

SMB:
I followed Apple’s published article about disabling SMB signing. It appears this is off by default on iMac Pro models, since I didn’t have to change anything.

Per Apple: (I made no change to this file, it was already set to no)
Open the /etc/nsmb.conf file.
Set the signing_required value to “no,” like this:
[default]
signing_required=no
Save the /etc/nsmb.conf file.
Disconnect and then reconnect any mounted SMB shares to make the changes take effect.

MTU Jumbo Frame:
iMac Pro (sys prefs)
- Configure Manually
- 10GbaseT
- full-duplex
- Jumbo (9000)
- (checked) AVB/EAV Mode
QNAP
- Jumbo Frame 9000


Return to posts index

Bob Zelin
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 1, 2018 at 2:57:11 pm

you are going to order the QNAP QXG-10G1T right now. End of story. You will spend the $89

https://www.bhphotovideo.com/c/product/1402317-REG/qnap_qxg_10g1t_single_po...

plug the iMac Pro directly into the QXG-10G1T. Assign static IP address for the QNAP like
192.168.2.3
assign the iMac Pro a static IP address like 192.168.2.11
Jumbo Frames MTU9000 on both.

Now do AJA Speed Test or Blackmagic Disk Speed Test. Now what do you get ?

Bob Zelin

ps - I use 10GTek Twinax cables all the time. They work great.

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Kevin Patrick
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 1, 2018 at 3:00:50 pm

Done.

I get the card Monday, probably late.

I'll post back with results.

Thanks Bob


Return to posts index


Kevin Patrick
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 3, 2018 at 3:38:35 pm

Bob,

While waiting for the new card (coming today) I heard back from QNAP. Apparently, they have the same UniFi switch I have and their test results are the same as mine. In their words, "not good."

They suggested connect direct and retest. (same suggestion as yours, they just have a higher latency response time)

Though I do have to connect up to other devices (both wired and wireless) to this server. So I will still need a 10 Gbe switch. I've nothing but success with Netgear, so I'm wondering what you opinion is of Netgear's GS110MX switch?


Return to posts index

Kevin Patrick
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 3, 2018 at 9:56:51 pm

I changed out two things.

First I tried the new 10Gtek SFP+ cable. This cable is supposed to be compatible with the Mellanox driver in the QNAP network card. No change at all.

Next I replaced the network card in the server with the QXG-10G1T card.

I connected this card to the RJ-45 port of the UniFi 10 Gbe switch. No change at all.

Finally, I connected the iMac Pro directly to the new network card in the server.

I set the IP address manually on the server and the iMac Pro. However, QNAP complained about not having any DNS server address. I rebooted anyway, since with my limited networking knowledge, I didn't think this should matter for this test.

No change at all.

Of everything I've tried I simply cannot get the performance numbers to change.

However, when I went to check the Jumbo Frame settings the server I discovered that the only Jumbo Frame option for this new network card was 1500. No other option was available.

(insert you favorite frustration expletive here)

Am I missing something thats preventing me from changing the Jumbo Frames settings?


Return to posts index

Bob Zelin
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 3, 2018 at 10:43:43 pm

well Kevin -
I guess it must be time for you to finally call me to help you.

Bob Zelin
CEL (407) 257 9157

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index


Bob Zelin
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 4, 2018 at 7:34:49 pm

I am holding my breath waiting for your call Kevin. Hurray - I am turning blue.
I will fix your problem.

Bob Zelin
CEL (407) 257 9157

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Kevin Patrick
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 4, 2018 at 11:56:55 pm

Bob,

I have a call scheduled with QNAP tomorrow. I'll let you know how that goes.

This was scheduled after they got back to me regarding the Jumbo Frame issue. They told me to update FW and that would fix the Jumbo Frame setting issue. It didn't.


Return to posts index

Bob Zelin
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 5, 2018 at 7:38:58 pm

did QNAP help you ?
If they did not, I can help you.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index


Kevin Patrick
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 5, 2018 at 8:41:40 pm

I had to reschedule the call for tomorrow.

Kevin


Return to posts index

Kevin Patrick
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 6, 2018 at 6:33:51 pm

Hi Bob,

Executive summary of QNAP call, "It's broke. We'll get back to you."

After installing the RJ-45 card both you and QNAP suggested, I ran into the problem of not being able to change the Jumbo Frame setting on the card. 1500 and only 1500 was selectable. The point of today's QNAP support session was to fix this. Which they didn't. It's been esculated. Again. So, the direct connect performance was the same as the performance going throught the UniFi switch.

You've recomended this card, so I'm guessing you've had success with this card. How can it be that I can't?

I suppose it's possible that QNAP broke the Jumbo Frame setting in the current release.

I'd like to take you up on your offer, but I don't want to waste your time.

I'm not sure which is going to give out now. My desk or my forehead.


Return to posts index

Bob Zelin
Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 6, 2018 at 9:50:27 pm

It's not broken. They are loading in the wrong firmware version.
Exactly what is it going to take for you to contact me privately ? Do you want me to send you money ?

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index


Bob Zelin
Problem Solved - Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 7, 2018 at 5:56:22 pm

Hello Creative Cow readers -
I hope that I have kept you entertained.

Kevin Patrick contacted me. There were 2 problems in his system causing issues, that resulted in his slow speeds.

1) the latest QNAP firmware is not working correctly with his QNAP TS-873U-RP. After several tests with different firmware, we finally installed QNAP QTS 4.3.4.0569 from May 1, 2018, and now he is able to select MTU 9000
on the QNAP 10G card.

2) the Ubiquiti 16 port 10G switch is unsuitable for these applications. Once we were able to select MTU 9000 on both the iMac Pro and the QNAP 10G port, we were still getting poor speeds. Because I am unfamiliar with all the menus in the Ubiquiti (for Jumbo Frame selection) - we simply bypassed the Ubiquiti US series switch and plugged the iMac Pro directly into the 10G port of the QNAP, and now we are getting 500 MB/sec WRITE and 1000 MB/sec READ.
I will continue to research the Ubiquiti 10G switch, but for the moment, I still stress the use of Netgear XS series 10G switches, or the QNAP QSW-1208-8C 10G switch.

The end ! It's all working.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Salvador Garza
Re: Problem Solved - Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 7, 2018 at 10:31:53 pm

I have a couple of Ubiquiti products in my home office. You'll need either a cloud key or controller software running on a computer. Assuming you've adopted the switch and you have access to the controller's web GUI, you can enable jumbo frames here:
Select "Devices" on the sidebar, select the switch, this will open up the properties tab, go to config > services, enable jumbo frames and apply. This will enable Jumbo Frames for all ports on the switch.

Salvador Garza
http://salvadorgarza.com


Return to posts index

Kevin Patrick
Re: Problem Solved - Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 7, 2018 at 11:09:33 pm

[Salvador Garza] "config > services, enable jumbo frames and apply"

I have no idea how I missed that, as I went looking for it. I even chatted with Ubiquiti tech support, asking them specfically if I need to do anything to improve the performance through their switch. I guess "nope" means, "Hey! you've got to enable Jumbo Frames, which is over here ... "

There doesn't seem to be a setting for how many frames. Perhaps enable Jumbo Frames means 9000?

Either way, thanks Salvador.

Though, with Bob's help, the iMac Pro is now directe connected to the QNAP server and the performance is great. I guess I'll just have to use the UniFi 10 Gbe switch to maintain heat levels in the rack.


Return to posts index


Salvador Garza
Re: Problem Solved - Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 7, 2018 at 11:13:48 pm

If you ever do get around testing the switch with Jumbo Frames enabled and the QNAP NAS, I'd be interested in the results. I'm about to install a TS-1685 in my home office and would be nice to keep it all Ubiquiti. Love the idea of one management interface for all my network components, however performance needs to be on par.

Salvador Garza
http://salvadorgarza.com


Return to posts index

Kevin Patrick
Re: Problem Solved - Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 7, 2018 at 11:23:28 pm

Well, not sure I will. I'm now in the mode of ... It's not broke.

Besides, I'm still all Unifi. Router, 48 Port Switch, 5 Access Points and a Cloud Key in a pear tree.


Return to posts index

Kevin Patrick
Re: Problem Solved - Re: SFP+ Cable Issues, Ubiquiti and Mellanox
on Dec 7, 2018 at 11:25:22 pm

Here's a not quite correct timeline of my QNAP adventures.

Nov 16 - Hey QNAP, why so slow?
Nov 16 - Hey Kevin, set your Jumbo Frames to 9000
Nov 17 - Hey QNAP, Jumbo Frames set, why still so slow?
Nov 17 - Hey Kevin, reboot
Nov 17 - Hey QNAP, I’m not stupid, both rebooted
Nov 17 - Hey Kevin, let’s schedule a call
Nov 29 - Hey Kevin, I don’t know why slow. Hang tight
Dec 03 - Hey Kevin, it’s your Unifi Switch. (ha ha)
Dec 03 - Hey Kevin, buy more QNAP stuff, RJ-45
Dec 04 - Hey Kevin, it’s Bob, call me
Dec 04 - Hey QNAP, why new card so slow?
Dec 04 - Hey Kevin, cause I waited to tell you about new FW!
Dec 04 - Hey QNAP, new FW can’t set Jumbo Frames
Dec 05 - Hey Kevin, it’s Bob again, call me
Dec 06 - Hey Kevin, let me into your system (again)
Dec 06 - Hey Kevin, new FW is broke
Dec 06 - Hey Kevin, it’s Bob. Again, Please call me.
Dec 07 - Hey Kevin, it’s Bob. Let’s look at your problem
Dec 07 - Hey Kevin, Bob again. All fixed.
Dec 07 - Hey Kevin, QNAP here, confirmed our FW is broke, hang tight

Next time?

Call Bob.

Thanks Bob.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2018 CreativeCOW.net All Rights Reserved
[TOP]