FORUMS: list search recent posts

Fast read, Wicked Slow Write on RAID 6 Array

COW Forums : RAID Set-Up

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Chad Self
Fast read, Wicked Slow Write on RAID 6 Array
on Oct 2, 2013 at 8:09:28 pm

Hey there. Since nothing else I've tried is working, I'll throw it at you guys!

Here's what you need to know about my config:
• Early '09 Mac Pro QuadCore
• Highpoint Tech RocketRAID 3522 (SATA)
• 8TB array in RAID 6 (two parity drives, total of 6TB storage)
• all 6 disks are Seagate Barracuda 7200 SATAs

I had a single disk go kaput last week. "No big deal," I thought, "just pop in a replacement drive and keep going."

NOPE

Somehow I lost the partition map and while I was able to get all the important data off to other drives, and create a new partition, my array is acting like a firewire drive. (Remember those? I'm having to use some for actual editing now!)

Disk testing is remarkably inconsistent.

My AJA Disk tests (using 1920x1080 8-bit video but in varying file sizes, reported in MB/s) go like this:

File Size Sweep
MB MB/sec
Read Write
128.0 357.4 376.4
256.0 355.4 482.5
512.0 395.0 515.5
1024.0 437.5 508.7
2048.0 9.9 376.5 (Yeah, you read that right)
4096.0 12.9 347.6
8192.0 62.9 529.2
16384.0 0.0 0.0 (Can't do either on 16GB file)


Any ideas on why the array tanks at files larger than 1GB?

And why the write is so much better than the read?

The HPT webGUI says all individual drives are fine. It's not the most reliable thing ever, but not sure what else to do.

Thanks for your help!
______________________________________________________________________________

I feel more like I did when I came here than I do now.


Return to posts index

Alex Gerulaitis
Re: Fast read, Wicked Slow Write on RAID 6 Array
on Oct 2, 2013 at 8:32:56 pm

Have you tried a different benchmark?

Tried testing individual drives and/or this array configured in RAID0?

The usual suspects here are the drives, the controller, and to a lesser extent, the chassis, the OS (and the file system), and the benchmark tool. To zero in on the problem, I'd start by eliminating the first two possible culprits: testing the drives individually and/or in a different RAID level, and I'd certainly try a different controller. There's a reason G-Tech switched away to ATTO after using Hipoint for years on their G-Speed eS Pro drives.


Return to posts index

Chad Self
Re: Fast read, Wicked Slow Write on RAID 6 Array
on Oct 2, 2013 at 8:39:44 pm

Thanks for the quick response.
Other than using the SMART utility in the HPT GUI, how would I test the individual drives?

This box is already pretty dated. I've been waiting to see what Apple was planning for the new MP before upgrading —and what I know doesn't seem to help me any :) — so I hate to drop a bunch of coin on a new controller card if I might be moving to something like a ProMax ONE.

Cheers,
-C

I feel more like I did when I came here than I do now.


Return to posts index


Alex Gerulaitis
Re: Fast read, Wicked Slow Write on RAID 6 Array
on Oct 2, 2013 at 8:56:51 pm

[Chad Self] "Other than using the SMART utility in the HPT GUI, how would I test the individual drives?"

Break the RAID set, set it up as JBOD (i.e. no RAID at all) - most controllers support that. This way, all drives will show up individually to the OS.

The other way is to take drives out of the array, connect them individually to onboard SATA or a different system.

Either scenario results in losing whatever was on the array, so tread carefully.

[Chad Self] "might be moving to something like a ProMax ONE"

I keep hearing about it. It's Windows only, right? What attracts you to One? The CPU power and the ability to have a bunch of drives internally?

[Chad Self] " I've been waiting to see what Apple was planning for the new MP before upgrading"

The new MP leaves three storage connectivity options: Thunderbolt, 10GbE (again, via TB-to-10GbE), and whatever you can put in a TB expansion chassis - which includes a RAID controller. So that investment doesn't have to be written off, and has advantages over either pure TB or 10GbE: expansion, cost and proven performance.

All that aside, try first a different benchmark, e.g. simply copy a large file from one location on the array to another one (or your boot drive) and time it. If this confirms AJA results, I'd try the other tests I mentioned.


Return to posts index

Chad Self
Re: Fast read, Wicked Slow Write on RAID 6 Array
on Oct 3, 2013 at 4:01:43 am

Thanks for the input. The AJA tests seemed to confirm the weird problem I was having trying to get all the data off the array so that I could re-partition. It too two and a half days to get 4TB off the array to a G-Tech FW800 drive.

I separated the drives out (creating JBODs from each of the 8 drives to make it easier) and ran tests again. No drive was super different than the others. All within 5% of each other.

So now I'm rebuilding the array with a full-on foreground init. (You were right to suggest this as I had only done a quick init the last time). I've also increased the block size to 128K (previous blocks were 56K). Once it's rebuilt, I'll test again.

You had a some questions about ProMax.
I started looking at them about 6mo ago in frustration over the lack of a Pro offering from Apple. As much as it pains me to move back towards Windows (left after XP, excited at the prospect of a serious Unix for creatives) I'm really frustrated by Apple's hyper focus on phones to the detriment of content creators. Lack of serious processing power, poor video card choices, delayed updates were all getting to me.

Since I left FCP with FCP X (or as I call it iMovie Pro) for Premier Pro, I'm not quite so tied to the OS as I once was. I will say, I like OS X better than Windows 7, nevermind 8. Walter Bascardi did a nice writeup on his blog about an early ProMax ONE. That's one of the things that got me looking in their direction. Walter is as platform agnostic as I would like to be. :)

The truth of the matter is that ProMax builds a helluva machine, but they aren't cheap.
Of course neither are MacPros historically. I've spent near-enough to 10K on my QuadCore 2.6 with a Quadro Card in 2009. The spec'ed machine and dual 20" monitors ran close to $7K. I build the RAID myself using a RocketRAID 3522 (now much out of favor thanks to the lousy support HIghPoint provides) and what is now a Provio 8bay MiniSAS EdiBox with what I thought were Seagate server drives for about $3K. That seems like a lot, until I started spc'ing something new. MacPro 12cores started at $5K and topped out past $9K the last time I was able to spec one.

As far as ProMax goes, I've spec'ed a ONE Plus OnSet, which specs Dual Xeon 2.4s (16 Cores), BluRay burner and basic stuff. That OnSet Machine cost is right at $7K. Adding 32GB RAM, a Quadro K5000 video card (after effects loves CUDA cores) an internal 16TB raid array (which I could stripe and use the current array as a daily backup), plus an internal LTO tape backup (for disaster recovery and long-term storage), and dual HP ZR2440W screens nearly doubles that. (ouch) As a company we were working out the cashflow details when Apple dropped that Mac Pro teaser.

Since the release of the teaser for the Pro, I've been thinking I might swap out the Quadro for a couple (or more) GTX cards instead. Might actually be cheaper. Or I might get a Mac Pro suite. They are an Apple Reseller as well. I'll be interested to see how the MP's AMD cards work with Adobe CC, since CC seems geared towards NVIDIA at present.

I will also go on record to say that the ProMax guys are really good people with whom to talk over your workflow issues and come up with a solution. Ask for Josh.

I feel more like I did when I came here than I do now.


Return to posts index

Chad Self
Re: Fast read, Wicked Slow Write on RAID 6 Array
on Oct 2, 2013 at 8:33:47 pm

Just another bit of info.
I have an old G-RAID (striped) FW800 1TB drive. Just ran the same battery of tests:


File Size Sweep
MB MB/sec
Read Write
128.0 82.9 31.7
256.0 77.3 31.8
512.0 81.8 31.8
1024.0 81.5 31.8
2048.0 82.5 31.8
4096.0 82.7 31.8
8192.0 82.8 31.7
16384.0 83.0 31.8



Pretty consistent.

I feel more like I did when I came here than I do now.


Return to posts index


EricBowen
Re: Fast read, Wicked Slow Write on RAID 6 Array
on Oct 2, 2013 at 11:27:58 pm

I would start with a fresh raid set after you move the data off. Make sure you Full initialize when you create the new Raid 6 array. Don't use the quick initialization. It never hurts to test the drives individually as Alex suggests. Keep in mind the Desktop drives are not recommended for parity raids. They can be ok for Raid 10 but not what I would suggest for Parity. I would also suggest writing zeroes to all the drives before you create the new raid set. That should zero out block issues on a drive that may occur in the raid index area.

Eric-ADK
Tech Manager
support@adkvideoediting.com


Return to posts index

Chad Self
Re: Fast read, Wicked Slow Write on RAID 6 Array
on Oct 4, 2013 at 2:22:15 am

The array rebuilt, but then disk utility bombed when zeroing out the array (after 6 hours, with 15 minutes left. Ugh.)

So I've tried a different tack. I've used two drives In a RAID 0 (ill use it for a scratch disk) and then setup the remaining drives as a RAID 50 (where I'll be keeping the media) the striped array is formatted and ready to go. The fitty is still building. We'll see how it goes.

Interestingly, the striped array still writes faster than it reads. And it reads slower than the RAID 6 on smaller files. Here's the test results on the striped array

File Size Sweep
MB MB/sec
Read Write
128.0 171.2 349.2
256.0 145.0 204.0
512.0 161.1 227.4
1024.0 184.9 216.5
2048.0 192.1 212.4
4096.0 200.4 207.6
8192.0 201.7 208.2
16384.0 202.2 203.5



I feel more like I did when I came here than I do now.


Return to posts index

EricBowen
Re: Fast read, Wicked Slow Write on RAID 6 Array
on Oct 4, 2013 at 11:51:59 pm

What driver and firmware are you running? What do you have the Controller cache mode set to? Do you have an option to enable disk cache in the management?

Eric-ADK
Tech Manager
support@adkvideoediting.com


Return to posts index


Chad Self
Re: Fast read, Wicked Slow Write on RAID 6 Array
on Oct 5, 2013 at 1:47:41 am

Eric, Thanks for the reply. I just updated the firmware last week thinking that was the problem , but I didn't think to check the driver. I'll try that as soon as the RAID 50 is finished zeroing out.

The management console gives me two options for disk cache. "Write Through" and "Write Back". I've chosen "Write Back".

Incidentally, I finally got some replacement drives in today and noticed that the new drives were missing a jumper that the pulled drives had in place. Upon checking, I see the jumper says "limit to 1.5Gb/s Operation." If the jumper is open (new drives hd jumper removed) the setting is 3Gb/s.I pulled all the jumpers off and rebuilt by RAID 50. It's zeroing out now....and yes, I'm an idiot for not noticing this when I built the array in 2010.

The RAID 0 is operating on one of the old drives and one of the new ones, but I built it before I noticed the whole jumper thing, so I'll have to work on that later, as I'm actually trying to get work done, and I'm using it as my scratch drive. Fortunately, RAID 0's take no time to build and only a couple hours to zero out (and I may not need to zero it out again?)

I can't find any "enable disk cache" in the WebGUI for the HPT, but maybe it's a system thing? I note that the disk test lets me disable system cache when it runs (it's disabled by default). So here are tests on the scratch disk done with and without the system cache. Note that even with the cache enabled, the read speeds tank at the 2GB file size. (I will allow that it's possible that PremierPro was doing something in the background, but I note that the non-cached test is remarkably consistent by comparison.) Maybe I just run out of cache? I've only got 12GB RAM installed. Oh hell, I don't know. I give up.)


AJA Test with system cache enabled.

File Size Sweep
MB MB/sec
Read Write
128.0 2409.4 312.7
256.0 2350.8 181.3
512.0 2472.6 200.1
1024.0 2506.9 208.1
2048.0 124.8 193.2
4096.0 116.2 198.8
8192.0 169.5 204.0
16384.0 192.0 206.2




AJA Test with system cache DISabled.

File Size Sweep
MB MB/sec
Read Write
128.0 270.0 370.5
256.0 180.3 216.1
512.0 184.4 222.0
1024.0 197.7 221.2
2048.0 197.5 213.1
4096.0 186.8 208.6
8192.0 199.7 209.4
16384.0 201.3 208.1



I feel more like I did when I came here than I do now.


Return to posts index

Chad Self
Re: Fast read, Wicked Slow Write on RAID 6 Array
on Oct 5, 2013 at 3:49:37 am

OK, The 50 is zeroed out. Here is the cache enabled test.



File Size Sweep
MB MB/sec
Read Write
128.0 2506.7 167.3
256.0 2530.8 172.6
512.0 2450.1 188.7
1024.0 2589.2 205.2
2048.0 2555.2 218.5
4096.0 341.6 217.6
8192.0 352.4 214.0
16384.0 331.7 222.9



File system cache DISabled


File Size Sweep
MB MB/sec
Read Write
128.0 170.4 265.8
256.0 240.1 222.8
512.0 288.7 214.7
1024.0 314.9 218.1
2048.0 295.7 211.3
4096.0 330.8 217.2
8192.0 322.5 213.1
16384.0 326.1 205.1



I don't know if it's totally fixed, but it seems to be working. I'll be updating the driver next.
Sorry if I'm posting too much info, It helps ME keep track of where I've been. :)
-C

I feel more like I did when I came here than I do now.


Return to posts index

EricBowen
Re: Fast read, Wicked Slow Write on RAID 6 Array
on Oct 5, 2013 at 4:18:41 pm

That looks normal depending on Highpoints caching model. Sounds like you do have it sorted now. Drivers do update caching models so I would do that next.

Eric-ADK
Tech Manager
support@adkvideoediting.com


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]