FORUMS: list search recent posts

Large number of copy and verification errors in LTO using PreRoll Post

COW Forums : Archiving and Back-Up

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Joe Bender
Large number of copy and verification errors in LTO using PreRoll Post
on May 7, 2016 at 8:14:20 pm

Hi All,

Following your responses to my questions in a previous thread, I decided to go with a Quantum LTO-7 drive (using LTO-6 tapes until the v7 media cost comes down) and PreRoll Post to back up my footage for a shoot in which we're generating ~8TB a day of 4K ProRes. The footage is getting offloaded to a large RAID6 volume and, in theory at least, backed up to LTO-6 tapes.

However, we've been having a number of problems — first, we're getting a large number of copy and verification errors in the PreRoll Post backup sessions. I'm new to LTO backup, so am confused. Is it normal to have copy and verification errors when writing to LTO? We're talking in the range of 1-3 copy errors and 2-6 checksum errors per tape, consistently across now a half dozen attempts.

Second, when attempting spanned backups (e.x. ~4.5TB, so 2 tapes), PreRoll Post gets to the end of the first tape, reports the copy and checksum errors, and asks for another tape to complete the backup. However, I can't get it to eject the first tape. The communication log in PRP says "resource busy, try 'diskutil unmount,'" but diskutil unmount returns a 'disk in use' message and won't unmount the tape. The only way out is to cancel the backup, restart PRP, and then unmount the tape, but of course at that point there's no way to resume the backup. In each backup attempt we've formatted a fresh HP LTO-6 Ultrium tape in the PreRoll Post settings menu before starting the backup.

Any thoughts on why these issues might be occurring and what I can do to avoid them? I urgently need to find a solution, as the footage needs to be backed up ASAP...

Your help, as always, will be hugely appreciated.

Many thanks in advance,

Joe

My system:
Brand new Quantum LTO-7 drive in a rack mount enclosure, operating over SAS through a HighPoint RocketStor 6328. This is running into a 6-core late 2013 Mac Pro with 64GB RAM running OSX 10.11.4. We're pulling data from a 12-bay rackmount RAID over 10GbE with read speeds in the 900MB/s range.


Return to posts index

Neil Sadwelkar
Re: Large number of copy and verification errors in LTO using PreRoll Post
on May 19, 2016 at 4:45:57 pm

Just to ensure that the tape drive and everything else in the system is functioning normally could you try doing a backup with something else? Like Bru-PE for instance. Its free for 30 days, and if a copy goes through without any error you can rule out your drive or system being the problem.

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index

Tom Ervin
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Jun 2, 2016 at 4:04:34 pm

Hi Joe,
Were you able to resolve your issues? I have had numerous errors with both YoYottaLTFS and PreRollPost and an mTape LTO-7 drive. I've run three tests, reformatting the same LTO-7 tape for each test. Is this 'verboten'? Curious...

Let us know what transpires. We're all in the same boat. Good luck.

--
Tom Ervin


Return to posts index


Joe Bender
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Jun 3, 2016 at 3:13:46 pm

Hi Tom,

Wish I had better news — I've been working through this problem for almost a month now without any conclusive answers. I've tried swapping out the SAS and Thunderbolt cables, the computer, the data source, and the software. I've downgraded and upgraded the firmware on the RocketStor and the LTO drive, and I've spent hours on the phone, email, and chat with the various tech support departments, and I still haven't been able to resolve the issue — my backups only complete successfully about 40% of the time (out of dozens of attempts). At this point the folks I've been talking to seem to think it's likely a hardware issue, but I can't afford to buy an alternate controller card or drive to test.

So that's my current status. Frustration level: high. I was warned that setting up an LTO system isn't the easiest thing to do, but frankly this is getting ridiculous. I've spent thousands of dollars on this hardware/software and no one can suggest anything other than "must be one of the other vendors' fault."

Happy to fill you in on more things we've tried, and I'd be grateful to hear if you have any better luck.

Best,

Joe


Return to posts index

Dima Toukan
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Jun 7, 2016 at 11:32:50 am

I've been having the same problems and have tried several tapes and different files but constantly have error files yet don't know what exactly the errors are. I've emailed Imagine Products and hope they will be able to help resolve this because it was quite an investment. I'm using mlogic LTO 6 and PreRoll Post.

Please keep us posted on any positive developments


Return to posts index

Joe Bender
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Jul 8, 2016 at 4:49:19 pm

Hi all,

Just a quick update on my situation — I've now been working with tech support at Imagine Products, YoYotta, HighPoint, and Quantum for over two months, and no one seems to be able to shed any light on why these errors continue to happen. I've forged ahead with backing up my data as much as possible, but am getting an approximately 40% error rate. Meaning 40% of my backups fail with file verification errors. In addition, I don't seem to be able to reformat and reuse the tapes, so this is driving my cost through the roof.

Does anyone have any idea what else I can do to resolve this situation? This is a crazy outcome for an investment of over $5K in hardware and software.

For folks who have implemented successful LTO systems, what's your setup and workflow?

Any input will be a huge help.

Thanks again for your thoughts.


Return to posts index


Jeff Lehman
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Aug 23, 2016 at 6:45:52 pm

I am having the exact same issue!

mLogic Thunderbolt LTO-7 drive
PreRollPost for Mac
Source drive is a G-SPEED Studio XL

The backups fail all the time. And you cannot span tapes (like PreRollPost claims you can). I called Imagine Products about the tape spanning issue and they told me to "stick to backing up 1 tape at a time".

Even then, I'm getting tons of errors and failures. Rough going when it takes 10-12 hours per tape just to find out that you have to do it again.

I've been able to reformat tapes, but I worry that reformatting and retrying the backup over and over again could be compromising the integrity of the tape.

Had anyone made any progress with this issue?
Has anyone made any progress on this?


Return to posts index

Neil Sadwelkar
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Aug 23, 2016 at 7:23:35 pm

Two different setups...

Joe Bender
Quantum LTO-7 rack mount, SAS HighPoint RocketStor 6328, 6-core late 2013 Mac Pro, 64GB RAM, OSX 10.11.4.
Source: 12-bay rackmount RAID over 10GbE, read speeds 900MB/s range.

Jeff Lehman
mLogic Thunderbolt LTO-7 drive, PreRollPost for Mac
Source: G-SPEED Studio XL

Hardware spec totally different. Common software PreRoll Post among others.

I own PreRoll Post, as well as Bru-PE. Use Bru-PE almost always. PreRoll Post, tried ran into similar issues, gave up.

My setup
Tandberg LTO-6 connected via SAS to a Atto H680 SAS card inside a Sonnet PCIe expander.
Use MacBook Pro/Mac mini/MacBook Air and Bru-PE MacOSX is 10.10 and 10.9.

Not one single backup failure in 4 years. Barring mains outages where even the UPS ran out.

Here are my thoughts.

Joe, I think your GigE card and/or SAS card is suspect. Is the SAS card recommended by the LTO drive maker? Have you tried backing up to LTO via some other drive not the NAS RAID.

Jeff, you too, Have you tried backing up to LTO via some other drive not the G-SPEED Studio XL.

And for both, have you tried backing up with Bru-PE? Its free for 30 days so you can try. If it backs up successfully, then your entire hardwares setup is doing just fine.

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index

Jeff Lehman
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Aug 23, 2016 at 7:49:43 pm

Looking into BRU-PE. It does not use LTFS, correct?

It does look like a simplified version of PreRollPost, but I like the functionality of LTFS.


Return to posts index


Joe Bender
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Jan 30, 2017 at 9:15:36 pm

Hi All,

Just want to check back and share the outcome of this struggle. After months of wrestling with hardware support from HighPoint, Quantum, etc. and many hours of software support for both PreRoll Post and YoYotta, I was never able to get either of those programs to produce a successful backup.

Then, after months of frustration, I learned a minor detail that clarified the whole situation: my files are often well over 100GB each (long takes in 4K ProRes), and it appears that LTFS just doesn't handle very large files well. Bizarre, considering it's designed for archive, but it appears to be the case. I switched over to BRU, which uses its own proprietary file system and (as I understand) writes/verifies smaller blocks of data, and finally was able to create verified backups of my footage.

Moral of the story: think twice before using LTFS and LTFS-based software to archive very large files. I don't know what the size threshold is, but I consistently got verify errors with files in the ~100GB range and above.

So, thanks everyone for your help with this, and happy archiving,

Joe


Return to posts index

James Vorley
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Jan 31, 2017 at 2:02:26 am

BRU is superb.

It's supposed to support single archives up to an exabyte in size.

I've been in your position before trying to troubleshoot issues between software, computer, HBA and LTO drive. It can be impossible to get to the bottom of an issue with each supplier blaming it on the other.

The cool thing about TOLIS is they supply both archiving hardware and software and know what works with what.

LTFS seems like a good idea long term due to the open source nature but tbh I would rather trust proven proprietary technology to safeguard my data.

As an aside, we had an issue with the way BRU was reading catalogs which resulted in BRU requesting the wrong volume of a spanned tape set. This was quickly patched by TOLIS once we flagged the issue. We've also seen occasionally slower restore/verify times. TOLIS have suggested this is due to memory fragmentation on our Mac.

Anyhow the CRC verification method used by BRU is great and helps me sleep at night!


Return to posts index

Dan Montgomery
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Aug 24, 2016 at 2:15:13 pm

Sorry to hear you're struggling with this. We have had a handful of folks running into it but have never been able to replicate the issue in-house which tends to say it's hardware or data related.

That said, one recent report included the error log which showed that LTFS lost connection to the drive, thus resulting in cascading copy and verification failures. In that case it clearly was not an application issue; it was something in between the app and the LTO device (LTFS, OSXFuse, SAS/TB drivers, etc).

BTW, IBM just released another LTFS update about a month ago to "fix certain problems" with LTO-7 devices. So it's a good idea to keep up to date with their drivers, especially if you're constantly updating OSX.

One thing we can do is make the application more fault tolerant and we're working on that. We already added a "reverify" feature that attempts to read and gather the checksum a second time when a comparison error is reported. Of course that doesn't help if the 'connection' to the drive is lost.

Spanning does work in PRP, else we wouldn't have put it in. But if you're having issues it's safer from a time investment to try to do just one tape at a time until you figure out what's failing. What we've noticed in the 5+ years we've been working with LTO and Sony ODA is that people tend to throw all manner of files at it, and some just aren't LTFS happy. Our original assumptions that people would only archive field footage or finished products proved false.

Offload with Confidence...


Return to posts index


Jeff Lehman
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Aug 25, 2016 at 3:52:59 pm

Update:

I changed 2 settings and have since gone 4/4 successful archives.

1: Default Link Choice -- I know this should resolve itself within the prep process, but I decided to change it from "Cancel" to "Remove links from backup"

2: Turned off ALL options under the "Video" tab. Since 99% of the files I'm offloading are video files, I thought maybe the creation of thousands of proxy files and saving those to my computer was causing slow-down/communication interruptions. And I don't really want/need them anyway.

Since these 2 changes, I have had no issues, but I will report back if the problem pops up again.

While I have you here, Dan - in regards to spanning: do I need to format all tapes BEFORE starting the multi-tape offload? And are there certain settings that need to be enabled to ensure smooth spanning? e.g. Auto Eject

Thanks.

Jeff


Return to posts index

Dan Montgomery
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Aug 25, 2016 at 6:22:30 pm

Jeff,

Glad to hear you figured out what was causing your failed copies. Yes, it's usually a good idea to set the Symbolic Link treatment option to something other than Cancel.

Likewise you're correct that determining video types is time consuming and can also cause trip ups with files this option may think are videos but turn out to not be handled. This option is needed when you're simultaneously making proxies using ProxyMill, in which case you generally know the content and quality of the video files you're feeding the archive.

We recommend pre-formatting tapes ahead of a spanning session, but the software will prompt to pause and do that if need be.

Offload with Confidence...


Return to posts index

Bobby Cooper
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Aug 29, 2016 at 9:30:26 pm

My other post may have been dumped/moderated and never posted...

Great to hear IBM has an update to 2.2.1.1 for their LTFS for Mac. Do you have the link available? I have a Quantum LTO-7 tape drive, but Quantum is not offering the updated version and IBM will not provide me with a download link to it since I do not have a contract with IBM.

Thanks!


Return to posts index


Bobby Cooper
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Aug 29, 2016 at 8:56:50 pm

That is really exciting that IBM released an update to LTFS 2.2.0.1. Can you please pass along a link if possible to 2.2.1.1?

I tried IMB directly since Quantum does not have an updated version posted yet... Still waiting on a workable download link to be forwarded to me from IBM Fix Central Dept (they say no contract, no support even though I have the Quantum LTO-7 drive that uses their LTFS software to operate).

Thanks!


Return to posts index

Dan Montgomery
Re: Large number of copy and verification errors in LTO using PreRoll Post
on Aug 30, 2016 at 3:38:54 pm

Email us at support@ imagineproducts.com and I can send you a link from our website to download it.

Offload with Confidence...


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]