Trouble Configuring 10GbE Network
We're moving 4 of our Premiere editors onto their own dedicated 10GbE network and I'm having two serious issues that I'm trying to troubleshoot. First off our setup is a late 2014 Mac Mini server (OS 10.10.1) sharing out a LaCie 8Big 48TB raid 5 over afp/ smb with a SanLink2 10GBase-T adapter running into a Netgear Prosafe XS708E switch. Most clients are connecting to the switch through SanLink2's and we have one older Mac Pro with a small tree 10GbE card.
Initially when the first client connects read/ write speeds are great, around 600-700 (which is almost what the Mini is getting directly to the LaCie), but after a client is connected for about 10 minutes or so the read/ write speeds will totally tank, sometime dropping as low as 10-20MB/ sec and then jumping up and back down from there. I noticed that r/w speeds onto the LaCie from the Mini during this time also drops almost as low.
The second issue is that once more than one client connects, both of their speeds fluctuate wildly as if the switch is having trouble directing the traffic and then eventually speeds tank per issue #1. I know that speeds should drop per client since there's only so much throughput to the raid but it seems that something else is going on
I'm not sure what the deal is, flow control and jumbo frames are on everywhere and I've created the sysctl.conf files per many contributors suggestions on COW, but the network seems to get way too congested for having so few clients. Plus the fact that the Mac Mini r/w speeds straight to the raid drop make me think it could be a Thunderbolt bus issue? Any input anyone has would be great. I know 10GbE requires a different kind of fine tuning than fibre channel (which is what all of our other editors are on), I'm just not sure what that tuning entails.
Thanks for any help!
John McNeil Studio
720 Channing Way
Berkeley, CA 94710
oh come on Jonathan, you are not a systems administrator.
Your Mac Mini is not fast enough to handle all this bandwidth. You have the SanLink2 and the Lacie 8BIG on the same x4 Lane Buss of the Mac Mini. Mac Mini can't handle the bandwidth that you are trying to force thru it. Its fine at 20 - 30 MB/sec, but when you get into multiple 10G client speeds, it's choking. You said that you have most of the clients on SanLink2 boxes, and the older Mac Pro on a Small Tree 10G card - you can't push that kind of bandwidth.
And your Lacie can't do more than 800 - 900 MB/sec (it's thunderbolt and only 8 drives, AND it's on the same buss and the SanLink 2 that connects to your XS708E). And do you really know that your XS708E is running at Jumbo Frames ? There ain't no setting for it on the switch. Should have spent the couple of extra bucks for the XS712T !
Get off your Mac Mini (make it a client computer) and switch to a Mac Pro Cylinder 6,1 - put the SanLink2 on one buss, and the Lacie on another buss, and you will see instant improved performance.
and exactly what are your sysctl.conf files, and are you connecting via smb or afp ?
You kids think that this stuff just plugs in and works - you have now learned that it doesn't.
Rescue 1, Inc.
Thanks for you response Bob (however loaded). Swapping out the Mini for a Pro and putting the Sanlink and LaCie on separate buses was the next step I took in troubleshooting, but even though speeds are slightly better I'm still largely getting the same results with more than one client connected. I've tried over both afp and smb without much difference, and jumbo packets are pinging to the server so the switch is definitely sending them.
The part that boggles my mind though is that even when I connect clients over gigabit ethernet to the XS708e the same problem occurs: one gigabit client connected gives r/w of 100-110MB/s, but as soon as a second connects both of their speeds get choked up and they end of fighting over about 100MB/s of total bandwidth (one will be around 30-40MB/s with the other around 60-70MB/s). I think you can agree this is well below what the Mac Pro, Sanlink, and LaCie are all capable of, I'm just not exactly sure why it's going on. sysctl.conf settings are below:
Any helpful response is appreciated. And yes, this was definitely not my choice for a shared storage solution but I'm the guy who has to try and make it to work.
the tuning file that you list was posted by Steve Modica some months ago. The 4000000 send and receive space is slightly wrong, but close enough. I build systems like this all the time - I just did a system for Good Great Grand in NY City with a Maxx Digital 8 bay thunderbolt array, and it's working just fine - using the same promise SanLink2 that you are using.
I have seen the XS708E be problematic, which is why I now refuse to use the 708E and only use as a base model the XS712T, where I know that I can actually set things. Update your 708E firmware, and RESET the damm switch (I blame the switch at this moment, without knowing anything else). If you are using a Mac Pro 6,1 and have the Lacie on one buss, and the Promise on another buss (with your tuning file), you should have no issues. I assume you have tested the speed of your Lacie array, and that it is fast - correct ?
Are your clients 10GbE ? Promise boxes on your client computers ?
I have only seen the loading issue that you are seeing (30 - 40 MB/sec) when I use a Mac Mini as the server, not the Mac Pro 6,1.
I have done numerous installs like this with almost identical hardware (712T switch) and have not seen the loading like you are seeing UNLESS I use the Mac Mini as the server.
so when you play back your video on your clients, you are seeing stuttering video ? What editing software are you using ?
Rescue 1, Inc.
Thanks again for the response Bob,
All of our clients are using Premiere CC 2014 and sometimes After Effects running through 10GbE Sanlink2's. We updated the switch firmware which looks like it helped read speeds a little but writing from multiple clients still chokes up pretty badly. I had our 3 editors test out their Premiere workflow yesterday and things seemed functional for the most part. Today we had a couple additional editors throughout the studio connect over gigabit to transfer some files as well as use After Effects and that seemed to result in more stuttering playback and hangs in Premiere, so maybe we're just limited to a very few number of clients?
We managed to isolate the issue to the storage by running r/w speed tests directly to the server's local drive over the 10Gig network. Since the network had no trouble maintaining consistent speeds during these tests that leaves the connection to LaCie as the only possible bottleneck. After reaching out to their support the answer I received was basically that the 8Big was not intended or designed for file sharing at this capacity and that they don't support it.
At this point I'm wondering if setting up something like SNS's iSCSI server/client target/initiator software might help manage the random read/writes coming from multiple clients better? I don't have any experience with it so any input anybody has on it would be very useful. Other than that, it sounds like replacing the RAID array with something like a Maxx Digital is the only other thing I think we could do.
I will contact you.
Rescue 1, Inc.
How did you solve your issue in the end ??
Video Sales Consultant
Apple Certified Trainer Final Cut Pro 7
After troubleshooting/ testing every other component in the signal chain (NIC's, client machines, switches, cables, server, software, etc.) we ended up replacing the LaCie 8Big with a Maxx Digital 48TB ThunderRAID 2 and haven't had any issues since. The RAID controller on the LaCie for whatever reason just can't negotiate traffic properly from multiple clients. In the words of LaCie's customer support the 8Big units (and probably their others too) weren't designed to do this and therefore they don't offer support for this kind of networked setup. Thanks again to Ryan and Bob over at Maxx Digital for their help getting us a better rig.
Final working signal chain ended up being:
Thunderraid2 -> Mac Pro server -> Promise SANlink2 -> Netgear XS712T -> Promise SANlink2's -> 4 - iMac/ Mac Pro Premiere CC 2014 clients
Our company has a Synology Rackstation RS3614xs, I was tempted to make a move from internal raid 10's to everyone (3 mac pro edit suites) editing directly from the NAS in a raid 6 setup. Its currently using 8 bays on WD 6TB Red NAS drives.
Is it my understanding that most NAS systems will bottleneck on random access read/writes because a NAS isn't optimised for such use? Or is it just some NAS systems, where they didn't build the product with multiple editing video streams in mind?
Mac Pro 2.4Ghz 8 core, 24GB RAM, GTX 670
There are many NAS systems suitable for video editing but I would not recommend the 8Big as the storage component in a NAS. As I understand it with most video you'll primarily be concerned with sequential read/ write speeds rather than random r/w you'd see with a mail server or database.
If your needs dictate a shared storage solution make sure your throughput requirements are met by both the Synology RAID hardware and your network. A NAS will also require more tuning and/ or support hours than editing off of internal RAIDs so it's well worth it to find an expert you can reach out to if you run into issues. We currently have 2-4 Premiere editors at a time working successfully in H.264 off of a Maxx Digital Thunderraid2 and Mac Pro server on a dedicated 10GbE network.
How would you rate this setup in terms of its simplicity to administer? Using the same products and tuning file, is this relatively set and forget for someone with an average handle on networking?
I see Bobs condescending comment that "You kids think that this stuff just plugs in and works - you have now learned that it doesn't."
But it seems to me from wading through these forums that if you are capable of setting up a robust 1G network, using the right equipment in the right configuration should not require a terribly steep learning curve to adopt a 10GbE workflow
What would you say to that?
Hi Adam -
if you can setup a robust 1G network, then you can setup a robust 10G network. It is not difficult to learn (or no more difficutl to learn) how to setup a Synology, or QNAP, or other home made system, compared to learning Davinci Resolve, or Adobe Premiere, etc. The only difference here is that most of the readers of Creative Cow are editors, many who are small business owners, who have put the effort out to learn AVID Media Composer, or Adobe Premier, or Maxon Cinema 4D or Adobe After Effects, or Davinci Resolve. And they are involved with production as well for their clients. But they are on tight budgets, and so they would rather DIE than to hire someone that knows how to do this. That is where my condescending comments come from. I assure you that YOU can learn how to do this, no different that you learned FCP 7 or Adobe Premiere, etc. but you are busy, and you (or Jonathan) just say "why can't I just plug this stuff in, like a firewire drive, and have it just work !". Well, you can't, that's why. All of these systems are stable once they are SET UP correctly. NONE of these systems (except XSAN) require constant administration. So if you A) buy a system, and set it up correctly (which happens when you read all the manuals, suffer thru the installation, and finally "get it") - then you will have a nice stable system that requires not constant administration, or B) you hire someone to do this for you, and then you have a nice stable system that works day in, and day out, and requires no administration.
Let me tell you something, Adam. I am not a plumber. I had a toilet installed. The flapper failed, and I hired a plumber to change the flapper. So now it's 3 months later, and it's making noise, and occationaly not letting me flush. If I screw around with it, the water comes back in the tank, but I am sick of this. Why can't my toilet work 100% of the time, without having me hire another plumber, which will cost me MORE MONEY when it should have been done right the first time, so I don't have any of this aggravation ? Same with these 10G networks. You can do it - and it might work - kind of - but if you hire a professional, or buy one of the systems that you see advertised on Creative Cow, from a pro company, you wil have a nice working system, with no aggravation, and you will be happy. But if you say "no, I am going to buy a cheap QNAP/Synology, and set it up myself" - then you might have aggravation. NOW, you can sit there, and tear thru web URL's trying to find out why things are not working like they should. And you can post on Creative Cow, and say "I just bought this NAS from B&H Photo for 1500 bucks, and it seemed like it was working, but now everything is locking up". Simple answer - study the manuals and YouTube videos, or HIRE A PROFESSIONAL - and if you cheap out (like I did with the toilet plumber), then you get a running toilet. It flushed, but it pisses you off.
Rescue 1, Inc.
To echo Bob yes if you can setup a robust 1Gb network than setting up a 10Gb network is not much of a jump. The issue for us in this case - and Bob can attest to this - was the LaCie hardware. The RAID controller was a piece of junk. As DAS the LaCie we used worked ok but over any network 1G or 10G it was awful.
All in all to echo Bob once more you want to invest in solid gear from the get-go. If you know what you're doing network-wise or find someone who does you should be fine. As soon as we replaced our LaCie with a Thunderraid2 from Maxx Digital our problems disappeared.