FORUMS: list search recent posts

Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....

COW Forums : SAN - Storage Area Networks

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Neil Smith
Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 4, 2014 at 8:58:26 am

Been testing Thunderbolt 2 Bridge networking between two new Mac Pros ... a couple of 6 core nMPs with D500s arrived today and I upgraded them both to 64 GBs RAM each - so identical machines with max RAM.

Attached an ARECA Thunderbolt 2 8 x bay RAID 32 TBs to one - this unit is still in beta so expect the driver to be tweaked and these prelim speeds to go up ... but still impressive speeds from eight spindles.



Attached a Pegasus 2 R8 8 x bay 24 TB to the other one:



Tested them both in DAS mode.

Then connected a Thunderbolt cable between the two nMPs and set up a fixed IP Thunderbolt 2 Bridge between them and then tested the Tbolt 2 RAID attached to the other machine.

Tested the Tbolt Bridge speed both with only one RAID at a time going and then with both of them going together.

Used Blackmagic Disk Speed test for all the testing ...the DAS mode Tbolt 2 RAIDS came in around where I expected them but Tbolt Bridge testing was very inconsistent and throughput erratic and non consistent .... get the feeling that the SMB IP stack has not been optimized for Tbolt2 bridging or else I'm doing something totally wrong.

Here's some screen grabs that illustrate the DAS mode speeds and the Tbolt 2 Bridge speeds ... we'll be demoing the two nMPS with Xsan and the Tbolt2 Bridging networking at our X Pro monthly meeting on Sat Jan 18th, if anyone wants to come along and see the topology in action - details on our website at http://www.lumforge.com

1) 6 core nMP in DAS mode attached to ARECA Thunderbolt 2 RAID:



2) 6 core nMP in DAS mode attached to Promise P2R8 RAID:



3) 6 core nMP attached through Thunderbolt2 Bridge to ARECA RAID - one BMD test only :



4) 6 core nMP attached through Thunderbolt2 Bridge to P2R8 RAID - one BMD test only:



5) both nMPs attached through Tbolt2 Bridge to other RAID - two BMD tests running at the same time:



But please note, the Thunderbolt2 Bridge dual speed tests were all over the place ... anywhere from 40MB/s to 800 MB/s ... the throughput rates never settled down ... seemed like there was a quota of IP bandwidth available that was somehow parceled out to each nMP depending on disk caching or packet density ... never achieved a steady state data flow.

Any explanations on reason for erratic double Thunderbolt Bridge transfer speeds? Was rather hoping that you could have two editors sharing projects from their own DAS Tbolt2 RAID with no SAN in the middle.

We'll be demoing above configurations and announcing prices for our DAS, SAN and hybrid Thunderbolt2 solutions at the Jan 18th event if you want to place orders ... ARECA Thunderbolt 2 RAIDs and new MAGMA Tbolt 2 expansion chassis will start shipping in Feb.

Cheers,
Neil

Neil Smith
CEO
LumaForge LLC
faster storage faster finishing
323-850-3550
http://www.lumaforge.com


Return to posts index

Bob Zelin
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 4, 2014 at 1:22:34 pm

we observed the eratic speeds with Thunderbolt 1 bridging. I was hoping that this would disappear with Thunderbolt 2. While I have not done a Tbolt2 to Tbolt 2 bridge network test, like you have, I was hoping to be able to provide a super low cost NAS solution for 2 - 3 editors that want to share, but without having consistant speeds, this is pointless. I look forward to doing my own tests, but thank you so much for publishing these results. While 10GbE is a low cost alternative to a fibre system, as you know, it's STILL not cheap enough for some people, and I was hoping that Tbolt2 bridging was the answer. I guess, not yet !

Bob Zelin

Bob Zelin
Rescue 1, Inc.
maxavid@cfl.rr.com


Return to posts index

Neil Smith
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 4, 2014 at 4:36:46 pm

Thanks for feedback, Bob ... for straightforward data transfer between two new Mac Pros then Thunderbolt 2 Bridging is very useful ... I transferred half a terabyte of files between the ARECA and P2R8 in under ten minutes which was a lot quicker than having to copy to a transfer drive and then copy again.

But for a couple of editors trying to work constantly off HD or 4K files in realtime it might be a pain even though we were getting 500 MB/s to 600 MB/s in both directions at the same time at peak moments - even had a moment of 800 MB/s in both directions at one point!

Am going to try with FPC X 10.1 running on both nMPs and see if it's useable ... if it is, you could at least have an editor and an assistant working on the same show with different Libraries.

What's the root cause of the IP stack being so erratic and inconsistent? ... maybe the SMB IP stack needs some collision detection code or 'Jumbo Frames' option written for it. The other thing I tested was putting the cables on the same and different Tbolt buses but that didn't seem to make much difference ... as you know there are six Tbolt 2 ports but only three buses ... was wondering if the internal Tbolt2 switch was adding to the inconsistent data flow?

Will start testing the new MAGMA Tbolt2 to PCI expansion chassis next week with 10 GbE, 8 Gb/s FC and 6 Gb/s SAS cards to see what kind of throughput we get ... at the moment we're capped by the 800 MB/s limit of Tbolt1 but Tbolt2 should take that up to over 1200 MB/s.

Interesting times for sure in the Apple world ... I love the quietness of nMPs ... even with two of them on the desk right in font of you you can hardly hear them purr ... but have to say, when you have six Thunderbolt cables plugged into the I/O ports it really is fiddly to take them in and out ... and with the slightest bit of tension they pop out ... lost a couple of renders yesterday when the RAIDs dismounted unintentionally.

Neil

Neil Smith
CEO
LumaForge LLC
faster storage faster finishing
323-850-3550
http://www.lumaforge.com


Return to posts index


Chris Murphy
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 6:13:31 am

Maybe one of the network gurus knows whether Apple's gigabit ethernet hardware uses TCP offload. In any case the physical and link layers have been dealt with in hardware for some time. In data centers much of TCP/IP is also crunched, or at least "pre-digested" by the ethernet card. Otherwise the (general purpose) CPU has to do a whole lot more work.

It's a totally different implementation, but you can kinda see an ethernet card as the network equivalent of what a GPU does with graphics. If we didn't have a GPU the CPU would have to do all of the graphics rendering and it would be very very costly if even today's CPUs could keep up (they can't). It's not comparable in that the whole network stack in software with a general purpose CPU is doable, but my expectation is that this is just super expensive CPU wise, and why it's erratic is because it's subject to being pre-empted by the kernel for other time sensitive tasks vying for CPU time.


Return to posts index

Alex Gerulaitis
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 4, 2014 at 8:13:06 pm

Neil,

Thanks so much for doing the tests - very interesting.

[Neil Smith] "Any explanations on reason for erratic double Thunderbolt Bridge transfer speeds?"

Don't know but can only guess the bridging / device driver are not optimized for such high speeds. I've seen similar behavior with 10GbE links (speeds way under 10Gbs ceiling) and that was mostly attributable to NIC drivers and OS TCP/IP stack not optimized for sequential transfers.

Any chance of getting a little more granular speed measurements - perhaps with iPerf/jPerf?

Would be interesting to get bandwidth-over-time graphs similar to ones in this image, along with testing various packet sizes, and measure network performance separately from storage.

There's a good chance this will pinpoint the bottleneck.

P.S. Clicking on the images in your post results in "404 Not Found" - wanted to see the top two in full glory and couldn't. Any chance of fixing it... in post?


Return to posts index

Neil Smith
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 4, 2014 at 10:23:45 pm
Last Edited By Neil Smith on Jan 4, 2014 at 10:28:57 pm

We can fix everything in post, Alex!

Will try and do some more detailed testing but I'm off to CES on Monday for the week and up to my neck in transcoding footage for UHDTVs ... apparently 4K is the next "big thing".

Just did a 'quick and dirty test' on running FCP X 10.1 on both nMPs with each one accessing the Tbolt 2 RAID on the other nMP playing 4K ProRes4444 files across the Thunderbridge link ... either one would play well or the other would play well but not both at the same time ... strange me thinks.

1) 2 x nMPs playing 4K ProRes4444 files over Thunderbridge link with 4K monitoring in background:



2) Close-up of single Thunderbolt cable connecting 2 x nMPs:



So I disconnected the two nMPS and put a Thunderbolt 2 MBP laptop in the middle and mounted both RAIDs ... ran the BMD system test and got reasonable speeds to each RAID separately:

3) MBP Thunderbolt 2 laptop connected to 2 x nMPs:



N.B. the numbers show in the BMD tests varied greatly when testing Thunderbridge connectivity ... pls don't take them as gospel .. besides which, ARECA is busy tweaking their drivers based on real-life testing ... but still think the real is issue is how Apple implemented the SMB IP stack in Mavericks.

4) MBP connected to ARECA Tbolt2 RAID through thunder bridge link to nMP:



5) MBP connected to PROMISE Tbolt2 RAID through ThunderLink bridge to nMP:



6) FCP X 10.1 playing 4K ProRes4444 files from both Tbolt2 RAIDs with dropped frames:



So, it looks like though there is plenty of bandwidth available in Tbolt 2 (theoretically 20 Gb/s) we're not going to be able to utilize IP over Tbolt2 until some smart cookie comes up with some nifty software to optimize sustained throughput and reduce packet contention ... over to Quantum, Alex.

Neil Smith
CEO
LumaForge LLC
faster storage faster finishing
323-850-3550
http://www.lumaforge.com


Return to posts index


Alex Gerulaitis
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 5, 2014 at 1:40:29 am

Thanks Neil,

I still think isolating network performance from storage and measuring is separately might help zero in on the bottleneck.

Let me know if you'd like me to hop over and do some iPerf tests on your setup.


Return to posts index

Neil Smith
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 5, 2014 at 3:10:15 pm

Yes, agree on importance of separating Tbolt 2 networking from Tbolt 2 drive performance to get a better understanding of where the real bottleneck is .. like you, I suspect that the underlying issue is in how Apple implemented the IP stack in Mavericks and maybe the internal Tbolt 2 bus switch in the nMPs.

And yes, be great to have you come over and put iPerf though its paces and see what we find .... I'm off to CES tomorrow for the week (much joy) so maybe the week after when I'm back you can come over to The Lot and we'll roll up our sleeves and see what we can suss out.

Will also have a 16 x bay Tbolt 2 RAID to test by then which should saturate the Tbolt2 bandwidth even more so ... it was still good to see that even on 8 bay arrays I was getting peak I/O of over 800 MB/s Read/Write using the BMD speed test between two 6 core nMPs ... which means that if we can find a way to smooth out the IP traffic then utilizing IP over Thunderbolt Bridge will be a viable way to connect a small group of Tbolt2 editors together.

For basic file transfer between the two nMPs, Tbolt2 bridging works very well - transferred half a terabyte of 4K files from one RAID to the other in under ten minutes ... but for editorial work where we need a consistent real-time playback off the timeline there still needs some optimization done.

See you in a week's time, assuming I survive CES ... it's going to be interesting to see where all this 4K content we're producing on these spiffing nMPs is going to end up ... if 4K UHDTV delivery into the home takes off, then the consumer market for 4K content will be more significant than the DCI cinema 4K opportunity.

One's thing for sure, if we do move to 4K workflows then the demand for storage and bandwidth is only going to grow rapidly.

Cheers,
Neil

Neil Smith
CEO
LumaForge LLC
faster storage faster finishing
323-850-3550
http://www.lumaforge.com


Return to posts index

Neil Sadwelkar
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 19, 2014 at 4:36:34 am

One aside.

How come you have a Quantel colourist panel working with FCP? Or is it just incidentally there?

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index


Neil Smith
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 19, 2014 at 6:47:55 am

Well spotted! ... and yes, it's not attached to the nMPs but to a PABLO RIO PC ... the NEO panel just happens to be in the middle of the desk in front of the HD SDI monitor I'm using for color grading. I have the two new Mac Pros either side of it due to space limitations.

One of the things I'm testing in the Workflow Integration Lab are the different options for connecting the PC world to the Mac Thunderbolt world using devices like the MAGMA Tbolt 2 PCIe expansion chassis with 10 GbE Myricom cards.

One way to share files and data between the PC world and Mac world is to use Xsan or metaSAN which work very well but require a FC switch in the topology ... 10GbE is an efficient means to do file transfer between Macs and PCs but you end up having duplicate data on both NTFS and HFS+ sides.

What I'm ideally looking for is a way to color correct in the PABLO RIO PC world and then render out directly to ProRes Quicktimes in the Mac world without having to copy the DPX files over to the Apple RAIDs.

All suggestions welcome.

Neil

Neil Smith
CEO
LumaForge LLC
fast data
323-850-3550
http://www.lumaforge.com


Return to posts index

Neil Sadwelkar
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 19, 2014 at 6:35:58 pm

[Neil Smith] "What I'm ideally looking for is a way to color correct in the PABLO RIO PC world and then render out directly to ProRes Quicktimes in the Mac world without having to copy the DPX files over to the Apple RAIDs."

But there's no way - on a Win PC - to render out as ProRes from within any app, like Rio, for instance. You could probably export as SStP and then use Resolve on Mac to convert to ProRes.

Or, consider exporting to DPX on Rio and then GlueTools to use DPX sequences directly inside FCP. With the kind of bandwidth you have from the Promise or Areca over TBolt 2, you'll be able to play back Hd or 2k res DPX-QTs in real time.

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index

Sergei Yakovlev
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Apr 7, 2014 at 9:37:57 pm

Intel has just announced Thunderbolt Networking for PCs at NAB:

http://blogs.intel.com/technology/2014/04/intel-nab-2014-4k-video-drives-th...

Really exciting news. I hope Apple fixes its implementation in time for Intel’s PC drivers release.


Return to posts index


Chris Murphy
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 6:00:41 am

re: 2) 6 core nMP in DAS mode attached to Promise P2R8 RAID, 1077MB/s writes vs 817MB/s reads. I see the scren shots, but it seems like reversed numbers. Anyone have an explanation?

As for why IP over Thunderbolt is erratic, that's a matter of how Apple's implementing it. If they're emulating ethernet or something like InfiniBand/RDMA in software it's probably a huge CPU and memory hog. It's a PCIe bus, so imagine taking a plain PCIe cable from computer to computer. First I'd kinda expect that to fry one or both logic boards, but aside from that, there's no mechanism for them to communicate anything this way so that has to be coded somehow. That comes well before SMB, and if it is SMB being used, then it sounds like they're emulating ethernet in software. Very expensive to do that. It's not like these ethernet cards have junk on them, or generic purpose chips. They're specialized and the 10gigE ones have heat sinks on some of those chips.

Can you repeat this bridge test and take a screen shot of Activity Monitor set to All Processes with the %CPU column clicked on? Or even better would be to open Terminal and use:

top -s10 -ocpu

Wait at least 10 seconds for it to update, the initial display is not sorted. Then take a screen shot. BTW I see the screen shots above but when I click on them I get a 404 error, so I can't see them bigger than they are inline. Any ideas?


Return to posts index

Neil Smith
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 9:20:00 am
Last Edited By Neil Smith on Jan 6, 2014 at 9:38:58 am

I think you're onto something, Chris and we're getting closer to an explanation of the erratic behavior of 'IP over Thunderbolt' .. just found an insightful article by ILJITSCH VAN BEIJNUM on 'Ars Technica' written back in October 2013 where he highlights the "choppy" Tbolt Bridge throughput issue:

" .... The Thunderbolt network interface also indicates that it supports TCP segmentation offloading for both IPv4 and IPv6 (TSO4 and TSO6), but presumably, there's no actual network hardware in the Thunderbolt interface that could perform this function. The idea behind TSO is that the network software creates one large packet or segment, and the networking hardware splits that packet into pieces that conform to the MTU limit. This allows gigabit-scale networks to operate without using excessive amounts of CPU time. What seems to be happening here is that the system maintains an outward appearance of using the standard MTU size so nothing unexpected happens, but then simply transmits the large TCP segment over Thunderbolt without bothering with the promised segmentation. ...."

Here's the link to the full article - worth reading for the detailed analysis that Iljitsch provides on the inconsistent throughput of IP over Tbolt:

http://arstechnica.com/apple/2013/10/os-x-10-9-brings-fast-but-choppy-thund...

Presumably the same issues that Iljitsch identified with Tbolt 1 bridge networking apply equally, if not more so, to Tbolt 2 bridging.

I'll repeat the testing between the two nMPs and see what I can measure.

The other thing I was testing this evening was to try and set up a Compressor 4.1 distributed render farm using Thunderbolt bridging between a Tbolt 2 MBP and the two nMPs ... didn't have much success but if we could get it to work it would be a useful way to edit offline on a MBP with proxies and then connect to a nMP and utilize all the available CPU cores for online conform and deliverables.

Anyone else tried Compressor 4.1 over Tbolt bridging yet?

Neil

Neil Smith
CEO
LumaForge LLC
faster storage faster finishing
323-850-3550
http://www.lumaforge.com


Return to posts index

Neil Smith
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 9:46:39 am

Apologies guys, just seen that Bob Z referenced the same Ars Technica article in an earlier thread below ... should have read that before I posted.

Have sent an email to Iljitsch asking him if he'd care to join in our discussion.

Neil

Neil Smith
CEO
LumaForge LLC
faster storage faster finishing
323-850-3550
http://www.lumaforge.com


Return to posts index


Neil Smith
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 11:40:48 am

Just got a reply from Iljitsch ... it would appear that's he's having some trouble signing up to the Cow.

Below is a snippet from his email reply ... as soon as he gets signed up he'll join in the thread ... agree with his comment on the linked to blog about the need for 10GbE port on the back of the nMP ... was thinking the same thing yesterday while plugging in a gig E cable for Xsan config.

I'm waiting for a Tbolt2 to PCIe expansion box to come in next week ... when it does, I'll put a 10GbE Myricom card in it and connect to a 5,1 Mac Pro and Win 7 PC and see how that goes.

From Iljitsch email:

" .... I tried to sign up for an account but I didn't get an email.

What I wanted to add to the discussion is seeing if the TSO can be turned off (using ifconfig?) so many small packets are used rather than fewer big ones. Perhaps this will help. I don't think CPU utilization is an issue here as even on those < 3 GHz dual core machines I was able to get impressive speeds some of the time.

And I'm interested to hear what you guys think about this blog post:

http://www.muada.com/2014/01-03-the-mac-pro-needs-10-gigabit-ethernet.html ..."

Neil

Neil Smith
CEO
LumaForge LLC
faster storage faster finishing
323-850-3550
http://www.lumaforge.com


Return to posts index

Iljitsch van Beijnum
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 11:44:40 am

Thanks for the invite!

It's very disappointing that this issue persists with the latest and greatest hardware. I was hoping it was caused by an immature Thunderbolt implementation in my 2011 MacBook Air, but apparently not.

One thing I wanted to try was disable TCP segmentation offloading (TSO) to see if that would help but unfortunately the ifconfig en2 -mediaopt tso command doesn't work.

I don't think CPU usage is an issue here, as even between my sub-3 GHz laptops with only dual core CPUs the speeds can be good, they're just not consistent.

It's too bad that this doesn't work as it should, as other ways to do > 1 Gbps networking on the Mac Pro are cumbersome and expensive, as I wrote in a blog post the other day:

http://www.muada.com/2014/01-03-the-mac-pro-needs-10-gigabit-ethernet.html


Return to posts index

Chris Murphy
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 5:27:05 pm

TSO is typically done in hardware, hence "offloading", meaning the CPU doesn't do it. In this Thunderbolt IP bridge arrangement, there's no hardware to do the TSO in this case, it's done by the CPU no matter what. So it'd only be a choice of where the segmentation is being done, and it stands to reason they may have more efficient segmentation code at a lower level, so they simply don't allow it being disabled.


Return to posts index

Iljitsch van Beijnum
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 5:34:15 pm

Right. I just wanted to see if it made a difference just in case there was some bug in that code or some kind of unexpected interaction that is triggered by the very large segments, because those don't happen in "normal" networks.


Return to posts index

Alex Gerulaitis
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 5:35:56 pm

Would artificially setting the packet size below MTU size help figure if it's TSO that's the problem? E.g. doing a disk or iperf benchmark with block size at 1K?

Just wild guessing that this might help limit or eliminate segmentation and if we see less choppy bandwidth (although perhaps much lower due to high latencies) - that will pinpoint probable cause?


Return to posts index

Iljitsch van Beijnum
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 7:08:02 pm

Probably not. The way TSO works in network interface cards is that the software hands over a big packet and then the NIC slices it into MTU-sized chunks. In that case, lowering the MTU makes no difference at all to the sending TCP stack, which is in charge of setting the transmit rate.

But I suspect what's happening with Thunderbolt networking is that the segmentation step is simply skipped. In that case the MTU value is completely meaningless.


Return to posts index

Alex Gerulaitis
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 7:26:09 pm

[Iljitsch van Beijnum] " lowering the MTU makes no difference at all to the sending TCP stack, which is in charge of setting the transmit rate"

I meant lowering the block size (below MTU) before it gets sent to TCP. Eliminate TSO overhead.


Return to posts index

Chris Murphy
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 10:15:29 pm

I'd think TSO overhead would be less than the layers that are normally done on hardware that have to be done in software with this implementation. The lower block size translating into smaller packet sizes, even if they're close to the same size, means more packet metadata needs to be generated, transmitted, and parsed. Also, between network and application layer, the page size is 4KB on x86 so pushing around 1K block sizes may actually produce worse results as moving data at less than page size is probably not so well optimized. But I still think it's a valid test to see if it has any effect at all.


Return to posts index

Chris Murphy
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 7:33:10 pm

Umm, pretty sure MTU still applies no matter what. This isn't a case where we can skip physical and link layers by bonding two network layers together and just share large blocks. That the physical layer is virtualized doesn't mean we can break the laws of ethernet and share 64KB packet sizes at that layer. I don't think.


Return to posts index

Chris Murphy
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 6, 2014 at 7:12:17 pm
Last Edited By Chris Murphy on Jan 6, 2014 at 7:13:30 pm

Does this virtual NIC support jumbo frames? Maybe I missed that part.

We know that Thunderbolt 2 can produce a sustained high transfer rate greater than 1000MB/s. And we know a hardware 10gigE NIC can produce a sustained high transfer rate also. So the problem seems to be somewhere in the virtual NIC implementation which is in the kernel.

Linux has been doing TSO in paravirtualized NICs for a while, and getting well more bandwidth between virtual machines than is needed for the use case in this thread. So it seems it could work here too. But it's also a matter of optimization.


Return to posts index

Jack Zahran
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 7, 2014 at 1:04:37 am

The nMP has three (3) separate thunderbolt buses. Did you attach each of the high bandwidth devices to separate buses?



Return to posts index

Neil Smith
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 7, 2014 at 3:59:15 am

Valid point, Jack ... but I already knew about the bus config and kept the Thunderbolt Bridge cable on Bus 2 and the Tbolt2 RAIDs on BUS 1 and the monitors on Bus 0.

I even tried with the TB2 Bridge and the TB2 DAS on the same bus to see if the nMP 3 x bus config was acting as some kind of internal switch which was adding "choppiness" when transferring packets from one bus to the other but that had no impact on overall consistency.

Also, has anyone managed to get Compressor 4.1 to work over Tbolt2 bridging yet? ... that would be a sweet way to set up a nMP render farm if it works ... but just can't seem to get it working between 2 x nMPs and a TB2 MBP.

Very impressed with the performance of the ARECA TB2 8 x bay RAID ... consistently getting over 1000 MB/s WRITE and 1100 MB/s READ speeds in RAID 5 ... they're going to get me their 16 x bay Tbolt2 enclosure next week to test ... think we should see speeds up around 1500 MB/s if Areca engineers manage to work their magic with their drivers.

Am at CES in Vegas now and it definitely looks like 4K is the next BIG THING (well according to the TV vendors anyway) .... Apple is well positioned to capitalize on 4K workflow with the nMP and FCPX 10.1 and Logic Pro X plus Resolve 10.1 which are all optimized for the new 64 architecture and dual GPUs ... really hope that Apple put some engineering resource into fixing the data flow over TB2 bridging.

Neil

Neil Smith
CEO
LumaForge LLC
fast data
323-850-3550
http://www.lumaforge.com


Return to posts index

shawn koppenhoefer
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 9, 2014 at 3:28:40 pm

Are your tests with full drives or <50% full drives.

I'm interested in reading whether your measures are so good because you're only using the outer partition on the platters,.. or conversely, what improvement you might get if you DID ensure you were doing so.

Thanks for the sharing of your experience (and pics!).
I can't WAIT to get my hands on an ARC-8050v2 TB2 device (and would love them forever if they think to include a lowly usb3 justincase interface as on their 5026 device)(dare I wish for a micro-SAS hole too?).

cheers from Switzerland,
Shawn Koppenhoefer


Return to posts index

Neil Smith
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 10, 2014 at 7:06:39 pm

Testing was done on mainly empty RAIDs but then also with about 6 TBs of 4K footage ... when I get back from CES will fill them up to 80% and see what happens.

ARECA will be delivering a 16 bay Tbolt 2 enclosure next week so I'll put that through its paces ... have a feeling that we'll see sustained throughput go up to around 1500 MB/s Read&Write - the more spinning spindles the better.

We'll publish prices for the ARECA 8 bay Tbolt2 RAID next week and they should start shipping units around the end of the month.

We'll be demoing the Tbolt2 RAIDs in action with the nMPs at Larry Jordan's FCP X 10.1 Training Day in Burbank on Tuesday Jan 14th and at Michael Horton's Jan LACPUG meeting on 22nd:

http://www.larryjordan.biz/powerup-4k-in-fcpx/

http://www.lafcpug.org/user_schedule.html

Saw something of interest on the Intel booth yesterday at CES ... Lacie had a 1TB flash drive attached to a Tbolt2 PC and they were getting around 1000 MB/s R&W on the BMD speed test ... nice small compact unit that will make a nifty shuttle drive from on-set back to post ... plug it into a nMP and Bob's your uncle ... transfer a terabyte of data in under 20 minutes ... will test as soon as they ship me one.

CES is all about 4K/UHDTV this year ... UHDTV panels all over the place ... all they need now is some engaging 4K content and we should see adoption rates start to ramp up.

One really cool thing I did see on the Display Port booth was 3 x 4K TVs attached to a flight simulator! ... they had some young dude flying Spitfires across the Kent country side ... with three 50 inch UHDTV panels and 60 KHz refresh rate, man oh man, was it immersive ... know what Santa needs to bring me for next Christmas :-)


3x4k_flight_sim.jpg


3x4k_flight_sim_cu.jpg

Neil

Neil Smith
CEO
LumaForge LLC
fast data
323-850-3550
http://www.lumaforge.com


Return to posts index

Eli Berg
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 17, 2014 at 1:45:36 am

So, I guess my question, is back to your original post. I am doing a job where I will be editing on set. But I have an assistant downloading the footage.
We both will be using Thunderbolt 2 compatible computers. What is the best RAID to put between us. He will literally just be ingesting and organizing. I will then grab and color and cut down and output to a separate drive.
Would the Pegasus work as the bridge between or no? I just need simultaneous access to files he has already organize.


Return to posts index

Neil Smith
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 19, 2014 at 6:12:26 am
Last Edited By Neil Smith on Jan 19, 2014 at 6:27:19 am

Eli,

If I understand your question correctly, I think you're suggesting something that is not a good thing to do ... i.e., you want to connect two Macs at the same time to one Thunderbolt 2 RAID enclosure. The issue you're facing isn't so much a Tbolt connectivity problem, it's more to do with maintaining the integrity of the Directory Structure on the RAID file system.

I'm assuming you'd have the RAID formatted with HFS+ which is designed to only have one Mac writing and reading to it at a time ... if you connect two Macs to the same RAID which one is the master in charge of maintaining the Directory Tree Structure? I haven't tried it personally but I suspect that you will find that the Directory gets corrupted pretty quickly and neither machine will be able to Read or Write to the drive. Data is not written directly to the physical drive but to the logical layer that manages the directory of where all the bits and bytes are written to ... in this case HFS+ which is not designed to have multiple hosts writing to the same physical drive at the same time.

The safer option is to attach the RAID to the assistant's MBP and then for you to use IP over Thunderbolt bridging to connect directly to his MBP and then mount the RAID on your desktop ... that way you can work on the files on the RAID and his MBP will be maintaining the Directory Structure.

As long as you alway unmount the RAID from his machine first, you could then also directly attach the RAID to your machine safely.

Hope that makes sense.

Neil

Neil Smith
CEO
LumaForge LLC
fast data
323-850-3550
http://www.lumaforge.com


Return to posts index

Eli Berg
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 19, 2014 at 8:22:05 pm

Would it be better to build a server? That way we could both pull off the shared drives? Would that be fast enough?

Have you done anything with the Mac Mini Servers from sonnet? Seems like an okay option


Return to posts index

Bob Zelin
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 27, 2014 at 2:29:59 pm

Hi Eli -
I know exactly what you want to do. You want to build a tiny shared storage system (2 users) that are using Thunderbolt 2, instead of Ethernet as the network connection. You want to plug in a drive array (Promise Pegasus, etc.) into a Mac Mini, and use Thunderbolt as the network connection. Will it work ?

The answer is NO. We tried it. At this moment, even if you used a BRAND NEW Mac Pro cylinder as the server (which is thunderbolt 2, not thunderbolt 1), and connected your two Tbolt 2 workstations to the new Mac Pro (your instant Thunderbolt "switch") - this STILL would not work - Thunderbolt 2 is not going to give you the consistant speeds that you will need to playback from the same drive array, that you can currently get from a conventional 1GbE network (least not a 10GbE network, which will work flawlessly).

Much of this conversation was started by an editor in San Francisco Greg Leutenbauer, who was determined to make a Thunderbolt bridging network system work, instead of a conventional network. Well, we tried it. It ain't happening yet.

SO - can you use a Mac Mini with a Thunderbolt 1 drive array, a Thunderbolt to 10GbE adaptor, perhaps a little switch (if you need more than 2 connections) and go to your two Mac Book Pro workstations to have shared storage that absolutely works - YES YOU CAN. But not with just Thunderbolt (at this moment, not even Thunderbolt 2). And of course, for the record, the Mac Mini is Thunderbolt 1, not 2.

bob Zelin

Bob Zelin
Rescue 1, Inc.
maxavid@cfl.rr.com


Return to posts index

Justin Hammons
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Jan 12, 2015 at 6:47:19 pm

Hey Bob! You seem to be the expert on here! Have a quick question about setting up our editing studio! We're both video editing and design on both workstations and we do a lot with shared resources (primarily for the design side of the business...video editing tends to be more self contained).

We currently have 1 Mac Pro, 1 Mac Mini and an Older Mac Pro (not the new one) and a 20 TB Pegasus Drive. We're replacing the old Mac Pro with a new one and picking up a new 20 TB Pegasus.

So we'll have two Mac Pros, Two 20 TB Raids and a Mac Mini.

So from what I can gather from the threads, the best networking scenario would be to attach a Pegasus drive to each of the Mac Pro's via thunderbolt and then connect each of the Mac Pros to each other via ethernet network...

What would have been ideal would have been to stack the raids onto the Mac Mini and connect the Mac Pros to the Mini through thunderbolt...but that seems to be problematic.

Any suggestions. Is there another piece of hardware that we need to effectively use the Raid Systems as one central server and have each of the Mac Pros draw off of that shared storage?


Return to posts index

Johannes Obst
Re: Two new Mac Pros, two Thunderbolt 2 RAIDs, one Thunderbolt Bridge ....
on Mar 5, 2014 at 10:34:50 am

Hi Neil,

did you already receive the 16 bay RAID from Areca?


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]