FORUMS: list search recent posts

ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days

COW Forums : Storage

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Reinis Traidas
ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 27, 2014 at 10:42:01 am
Last Edited By Reinis Traidas on Jan 27, 2014 at 10:45:05 am

I've recently built a low-cost storage solution for video editing and grading using a Sans Digital TR8X-B 8-bay SAS enclosure, coupled with the ATTO R680 controller and 8 WD RED drives. Yes, I know these are not uber server class SAS drives, but hey, one step at a time.

After setting up the 8 drives in RAID6 in a single volume using ATTOs config tool, all was fine. The initialize took some 16 hours to finish though, but after that was done, disk speed performace was great (averaging 760MB/s read and write).

After a couple of days, the ATTO config tool says the Group is now degraded and specifically says Drive #8 is degraded. The LED on the enclosure for #8 doesn't light up too. Despite, everything seems to work normal. Also, the little color icon next to the drive is green (as in, not RED, meaning it's not dead?)

What am I doing wrong? I have set the NVRAM settings for the channel to Device Wait Time 10 and Wait Count 15 (as per the instructions in another thread).

Any tips on what could be causing this?


Return to posts index

Reinis Traidas
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 27, 2014 at 1:33:40 pm

I've initiated a rebuilt on the "degraded drive", will see if that makes the error dissapear.


Return to posts index

Chris Murphy
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 28, 2014 at 6:18:14 am

Check the array log with ATTOExpressSASRaid log utility. Optionally post the log somewhere like pastebin and provide a link.


Return to posts index


Ron Amborn
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 29, 2014 at 9:24:48 pm

Reinis
Do you have the latest driver and fw etc? It is also good that you are doing Raid 6 since you are using NON Enterprise drives. You will see more failures that way. This is said from experience and not just what drive manufactures say.

Sincerely,
Ron Amborn President
Maxx Entertainment Digital
21562 Newland Street
Huntington Beach , Ca 92646
Direct 714-374-4944
Cell 714-713-4492 Fax 714-374-3404
ron@maxxdigital.com
http://www.maxxdigital.com




Return to posts index

Reinis Traidas
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 29, 2014 at 9:33:52 pm
Last Edited By Reinis Traidas on Jan 29, 2014 at 9:36:00 pm

Hello there,

I did a rebuild of the group and it is now back to normal. I also updated my computers BIOS to the latest, as well as flashed the R680 to the latest FW. I will see how the RAID performs and if I get further parity errors or not.

I do want to point out, though, to whoever might google and stumble upon this is - my experience with the Atto R680 in the HP Z820. After installing the card, all my fans in the Z820 went nuts and kept running at very high (and loud) speeds right after turning the thing on. They just wouldn't slow down. I couldn't find any info on this, but after updating the BIOS to J63 v03.65 and after flashing the R680 to the latest, the fans went quieter. I don't know if this is the R680 that is just running hot or the fan daemon on the card itself (I read that some Mac Pro's have had heat/fan issues with it).

Anyway, I am happy with the performance of the R680 with the Sans Digital enclosure and WD RED drives. It's giving 680-780MB/s easily and greatly decreasing render times for .tiff sequences. Before, rendering 15 minutes of Prores 444 footage to XYZ Tiffs would take about 50 minutes if done on a single internal SATA drive. With the RAID6 group, that time went down to about 15 minutes, so practically real time, which is great.

I'm probaby getting a "hot spare" HDD just to keep on the shelf incase any of the drives fail.


Return to posts index

Ron Amborn
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 29, 2014 at 9:52:40 pm

Reinis
Send me a private email and I will connect you with an ATTO SE directly. I just can not post his info on line openly. My email is ron@maxxdigital.com.

Sincerely,
Ron Amborn President
Maxx Entertainment Digital
21562 Newland Street
Huntington Beach , Ca 92646
Direct 714-374-4944
Cell 714-713-4492 Fax 714-374-3404
ron@maxxdigital.com
http://www.maxxdigital.com




Return to posts index


Ron Amborn
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 30, 2014 at 5:52:29 pm

"I would always suggest flashing the ATTO adapters to their latest firmware, as well as using their latest drivers and config tool. That is the first thing to do upon purchasing an ATTO card.
ATTO are constantly improving and adding functionality, and fixing hiccups in previous versions of code.

The BIOS update of the HP Machine is what makes the fans go faster -- it is not anything that ATTO does. The fan speed can be controlled by adjusting the BIOS settings; however, the ATTO R680 runs hot, and it is recommended to keep the fans (at least in the PCI area) going as fast as possible, while still keeping a quiet studio. The ATTO fan controller daemon does not interact with any PC workstation -- only Mac Pro towers (not the new mac pro or any imacs / macbook pros). The WD Red drives are more meant for NAS systems that use software RAID, not hardware RAID. For the R680, it is recommended to use drives that are meant to be used for hardware RAID, because not only do they last longer and are faster, they will prevent problems down the road and give you a longer warranty. Yes they are more expensive, but if you want to use spinning disks with a hardware RAID controller, HW RAID drives (Like the RE series, Seagate Constellation Hitachi Ultrastar, etc). are worth it. Sans Digital does not work closely with ATTO to make sure everything is tested in order to work together.
Maxx Digital's arrays last longer and have fewer failures, offer higher drive counts, and also support dual path configurations for high availability setups at a fraction of the cost of an array from someone like Dell or HP; plus Maxx Digital works closely with ATTO to make sure everything just works out of the box."

Sincerely,
Ron Amborn President
Maxx Entertainment Digital
21562 Newland Street
Huntington Beach , Ca 92646
Direct 714-374-4944
Cell 714-713-4492 Fax 714-374-3404
ron@maxxdigital.com
http://www.maxxdigital.com




Return to posts index

Chris Murphy
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 30, 2014 at 6:33:06 pm

Certainly the firmware should be updated as a first step before creating the array. After the array is created, I'm skeptical of hardware RAID firmware updates. They should be safe, but... if they aren't, it's a mess. So it's better to assume it'll blow up the raid, and therefore make sure the array is suitably backed up and you're prepared for a complete restore should the upgrade go wrongly. And postpone the firmware upgrade to a time when you can afford the rebuild time, like maybe a Friday after lunch: test the raid that afternoon and if there are unacceptable regressions or it blows up, you have the weekend for an unattended restore to happen.

I'd also make sure the file system(s) on the array are all unmounted, and I'd follow the manufacturer's instructions for firmware upgrades exactly.


Return to posts index

Steve Katz
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 30, 2014 at 7:51:09 pm

Frankly, sounds like a thermal problem. We use the R680s quite frequently and in a very well (properly cooled) enclosure they don't degrade over time; in one that's not well cooled, they can.

If you'd like to contact me off line for suggestions, I'll be happy to try to help!

Steve Katz
JMR ELECTRONICS INC.
stevek@jmr.com


Return to posts index


Alex Gerulaitis
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 30, 2014 at 8:08:41 pm

Second Chris's idea to check the logs. Has the rebuild finished?


Return to posts index

Reinis Traidas
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 30, 2014 at 8:29:44 pm

Yes, the rebuilt went fine, all drives are OK now, flashed the controller to the latest FW, updated the BIOS on the HP Z820 and all seems well now. We'll see how long the RED drives last.

Performance is good, averaging 740-820MB/s on the desktop.


Return to posts index

Chris Murphy
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Jan 31, 2014 at 3:17:01 am

For it to become degraded in 3 days isn't normal. If the firmware update explicitly fixes a problem that's the direct result of the array going degraded, then great. But the only way you'd know this is if you or someone checked the array log, understands why the array went degraded, and is familiar with the changelog for the firmware update. Otherwise, I'm skeptical it's just magically going to be fixed with a firmware update.

While the Reds aren't enterprise drives, they do have the proper error time out that enables bad sector "repair" (via overwrite) on any read (including read only scrubs).


Return to posts index


Tim Jones
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Feb 4, 2014 at 9:14:38 pm
Last Edited By Tim Jones on Feb 4, 2014 at 9:15:22 pm

[Reinis Traidas] "After a couple of days, the ATTO config tool says the Group is now degraded and specifically says Drive #8 is degraded. The LED on the enclosure for #8 doesn't light up too. Despite, everything seems to work normal. Also, the little color icon next to the drive is green (as in, not RED, meaning it's not dead?)"

Reinis - I'm a bit late to the party here, but We have seen issues with heat in the SANS Digital TR4 and TR8 chassis. In each case, loading the chassis with the full compliment of drives almost always resulted in the last drive in the case (closest to the power supply) to drop off. To solve this permanently, we now only load 3 or 7 drives or we run the units with the doors off and additional fans blowing into the fronts of the chassis. Keep an eye on drive 8. I suspect you'll be revisiting this sooner or later.

We've reported it to SANS Digital, but nothing has been heard since before last year's NAB. The chassis are just too inexpensive for our short term lab projects to stop using ( we have 4 4s and 3 8s).

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.productionbackup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

Reinis Traidas
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Feb 4, 2014 at 11:48:19 pm

Thanks for that note, I will keep observing my TR8 unit to see if heat becomes an issue. So far it has been running well.


Return to posts index

Reinis Traidas
Re: ATTO R680 + Sans Digital TR8X-B RAID degraded after 3 days
on Feb 25, 2014 at 12:23:26 pm

This indeed seems to be the case.

I sent all diangnostic logs to ATTO and their reply was that the drive may be faulty.

I have kept the chassis door open since the last time the group got degraded - and guess what - it's still up and running fine.

So currently it appears that the TRX8 is suffering from heat related issues with the lowest drive trays - 7 and 8. Keeping the door open and/or aiding ventilation from the front of the unit seems to help/solve this issue.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]