FORUMS: list search recent posts

PreRoll Post - abort verify and release hard drive?

COW Forums : Archiving and Back-Up

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Neil Sadwelkar
PreRoll Post - abort verify and release hard drive?
on Oct 31, 2014 at 1:17:23 pm

In Bru-PE, one can abort verify (not recommended), in a pinch, then unmount the drive and still verify the tape independent of the drive.

In PreRoll Post, what are the consequences of aborting verify? Is the tape still secure? The 'Details' button shows a window with a list of files showing 'copied' against their names.
Can one verify the tape independent of the drive?

In a shoot situation one often has to hand over the drive back as soon as possible. After a 6 hour backup, waiting for a nearly equal time to verify and holding the drive back is painful.

Possible?

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index

Martin Greenwood
Re: PreRoll Post - abort verify and release hard drive?
on Oct 31, 2014 at 3:14:28 pm

Here at YoYotta we have also had customer requests to delay verification. It's not ideal, but the situation does arise in the real world.
So now in our YoYottaID LTFS software once the copy completes you can unmount the source media and the verify will continue. Hopefully the source media won't be erased until the verify is completed !
Also you can abort the verify and then start it again later without the source drive.
You can reverify against the original camera checksums at any time in the future using a free version of our software.

Martin Greenwood.

CTO

yoyotta.com


Return to posts index

Neil Sadwelkar
Re: PreRoll Post - abort verify and release hard drive?
on Oct 31, 2014 at 5:28:20 pm

I began an archive in PreRoll Post and after the copying was done, and verify was under way, I hit cancel. Got a warning that the archive would not be complete, but I continued. PreRoll Post went into beach ball mode for about 10 mins. I gave up and force quit it.

Now, the tape shows up on the desktop, and all contents seem to be fine. I checked only folder sizes, of course. But there seems to be no way for getting this LTO tape into the PreRoll Post database. So I can't verify it, nor use PreRoll Post for retrieve. Maybe I need to dig deeper to find a way. The manual doesn't seem to have any info on this unusual mode I forced the software into.

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index


Dan Montgomery
Re: PreRoll Post - abort verify and release hard drive?
on Nov 1, 2014 at 12:33:26 am

PreRollPost is an LTFS application. In LTFS, after copies complete the tape is recued to read each file (retrieve them) to compare MD5s to those of the source files. If you have a lot of files this step obviously can take some time to complete.

Once that's done, the index partition on the tape is updated to reflect the data that's been written to the tape. During this timeframe PRP also updates it's SQLite database file with the new information. Again, this step may take some time if there are tens of thousands of files involved. Once all these steps are complete, you'll get a 'Done' notification (or error information if any checksum problems are encountered, etc.).

So, if you completed the copy but quit the app and ejected the tape during the next phase your data should be intact. It's just the indices and second checksum that didn't get a chance to finish.

Depending on the severity of tape damage you'll be prompted to do different recovery steps when the tape is next mounted. PRP will first attempt to repair the tape--getting the index on the tape to match what's found in the data partition. Sometimes you may have to use implicit LTFS commands which are provided to you in a simple 'Tape Recovery Utility' interface under PRP's application menu (above Quit). These recovery tools are designed to be used in sequence--usually the 'Full Recovery' is all that's needed to get the tape's partitions back into sync.

The next option is named Deep Recovery and it's commonly needed when the tape is improperly ejected, power lost during the write process, etc. It should only be used if the Full Recovery is unsuccessful because this process can take hours to complete.

The last option is Read Only Mount. It's the last resort and generally only needed in cases of actual physical damage to the tape or it's memory, etc.

Any of these recovery options should never be interrupted once started. Otherwise you run the risk of the tape becoming permanently unmountable.

So, again, in your case it's probably a simple matter of running the Full Recovery to get the tape's index back in good shape.

To put this tape, or any other LTFS tape, into PreRollPost's database: Go to Settings>Local Indices. Click on the lower left button to 'Index Generic LTFS Tape'. This will copy the tape's index information into PRP's database and it will then be Retrievable/Searchable in that window. (To remove any tape or session from the retrieve view, select it and press Delete on your keyboard. To add it back, use the steps just described.)

Hope this helps. It's generally not a good idea to stop an archive in progress if possible.

Offload with Confidence...


Return to posts index

Tim Jones
Re: PreRoll Post - abort verify and release hard drive?
on Nov 15, 2014 at 6:32:29 pm

Ugh - that's one of the biggest foibles of LTFS - if something happens during the actual tape write, you can end up with a tape that can't be recovered - meaning that all of the data on the tape is now lost.

With BRU-based applications, you can power off the tape drive during the write operation and everything that was written to the tape(s) up too that point can be fully recovered. No further recovery steps required.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

Dan Montgomery
Re: PreRoll Post - abort verify and release hard drive?
on Nov 15, 2014 at 6:54:03 pm

Who said anything about losing data or an unmountable tape? Any data written to an LTFS tape is intact (just like the magical BRU software), it's just the index that may need syncing.

Why do I always envision windmills whenever I see a post by Tim Jones? ;-\

Offload with Confidence...


Return to posts index


Tim Jones
Re: PreRoll Post - abort verify and release hard drive?
on Nov 15, 2014 at 7:51:05 pm
Last Edited By Tim Jones on Nov 15, 2014 at 8:10:51 pm

Dan, that's a cheap shot (but I expect no less) and I'm sorry that you have a Don Quixote complex, but the proof is in the tasting of the pudding.

Insert and format an LTFS tape
Mount it
Start copying 300GB to it
Half way through, turn off the tape drive
Wait while Finder goes a bit wonky trying to warn you that the disk was improperly ejected
Turn the tape drive back on
Try to mount the LTFS volume
Ta Da - no windmill, just a failed tape.
Run "ltfsck --deep" and wait a few hours and maybe, just maybe, you'll get access to some of the files written to the tape.

Insert a blank tape
Start a BRU backup of the same 300GB
Half way through, turn off the tape drive.
Restart tape drive
Insert that tape
Restore every file up to the point of the power off

Otherwise, if LTFS was so good why does it need the many recovery mechanisms that you describe?

Dan wrote: "The next option is named Deep Recovery and it's commonly needed when the tape is improperly ejected, power lost during the write process, etc. It should only be used if the Full Recovery is unsuccessful because this process can take hours to complete. "

and then: The last option is Read Only Mount. It's the last resort and generally only needed in cases of actual physical damage to the tape or it's memory, etc.

Any of these recovery options should never be interrupted once started. Otherwise you run the risk of the tape becoming permanently unmountable.


You'll never find any such warnings and caveats with a BRU-based archive.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

Dan Montgomery
Re: PreRoll Post - abort verify and release hard drive?
on Nov 15, 2014 at 8:11:18 pm

Yeah, I agree, a cheap shot in response to another ("ugh"). I apologize. However, this is a technical forum Tim, not a sales one, so I figured you deserved it.

And do you really want to start comparing what BRU can do to your tapes? We damaged a tape just trying to use BRU! Those who live in glass houses...

Offload with Confidence...


Return to posts index

Tim Jones
Re: PreRoll Post - abort verify and release hard drive?
on Nov 15, 2014 at 8:20:16 pm

[Dan Montgomery] "And do you really want to start comparing what BRU can do to your tapes? We damaged a tape just trying to use BRU! Those who live in glass houses...
"


And, there's another cheap shot... Please, don't allude to an issue and NOT describe what truly happened. If you have such a tape, I'd love to see it. What did you do to the tape using BRU? Please, since you've opened this gate, don't perform a hit and run.

1) there's is no way for software to "damage" a tape as you allude
2) the only thing that BRU might do to an LTFS tape is to overwrite the index on partition 0 making the LTFS tape unrecoverable (for others reading, we warn if you try to overwrite an LTFS tape).

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index


Dan Montgomery
Re: PreRoll Post - abort verify and release hard drive?
on Nov 15, 2014 at 9:34:53 pm

It's okay Tim, we still love you. My Dad thought color TV was a fad too.

BTW, how's your tests of PreRollPost going? We noticed your technician downloaded twice last week...

Offload with Confidence...


Return to posts index

Tim Jones
Re: PreRoll Post - abort verify and release hard drive?
on Nov 15, 2014 at 11:44:02 pm
Last Edited By Tim Jones on Nov 15, 2014 at 11:44:36 pm

You didn't expound on your statement about the damaged BRU tape (and continued to throw childish comments).

We're looking at what you're doing in the same manner in which you guys looked at BRU.

And, since you mentioned it, we're looking at it because a customer created a series of LTFS tapes to send to another shop who was looking at PRP and LTFS. The second shop can't import any of these LTFS tapes. We had our customer send us two of the tapes and it seems that PRP won't import a tape that's been written to multiple times. The error that both we and the original PRP user are getting is that PRP can't import tapes with multiple indexes. Since every non-continguous write to an LTFS tape creates a new index, you guys should look at this issue. Currently, PRP will only catalog non-native LTFS tapes that have been written in a single atomic operation and have the index write rule set to on unmount. If the index update is set to time or the tape has multiple writes in separate session, you guys won't parse it and simply toss up an error message that the end user doesn't understand.

There's nothing wrong with the tapes and the other shop's user can mount them properly from the command line.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

Dan Montgomery
Re: PreRoll Post - abort verify and release hard drive?
on Nov 1, 2014 at 12:40:22 am

Oh. And to verify the contents of the tape against the source files, you can retrieve them from the tape to a disk. Then run ShotSum to compare the two.

With ShotSum ($99) you can check any files or folders at any time. You can also find all the instances of the same files across several disks or network, etc. (regardless how they're named).

ShotSum, like ShotPut Pro, has an option to save the MD5 checksum values to a text file with the files. If you do this, at any time in the future you can check the files integrity by just dropping this .md5 file into ShotSum (or ShotPut's checksum utility) and the software will follow the embedded links to check the associated files against the md5 checksums of the originals. This of course has the advantage that you may move or copy the files and always be able to verify they're exactly matching the originals.

Offload with Confidence...


Return to posts index


Neil Sadwelkar
Re: PreRoll Post - abort verify and release hard drive?
on Nov 4, 2014 at 2:54:30 am

Thanks Dan. I'll try that.

About the number of files, we use LTO-5 and LTO-6 to backup camera originals. So, if its from an Arri Alexa shooting ArriRaw, there are 1,440 files per minute (at 24fps). Each is 7 MB in size. So, for every TB there are 142,000 files. Plus sound files, xml files.
On an LTO-5 you can easily have 180,000 or more files, and on an LTO-6 over 320,000 files.

Sony F55/F65 or Red Raw produces much less files.

If one is backing up a Mac home folder containing address books, certain documents, iPhoto Library then too the backup quickly swells up to hundreds of thousands of files as many Mac document and data formats are packages. So, in PRP, just the 'Prepare backup' takes ages.

Which could be healthy, because I did my routine evening walk of 3km in the time Prepare backup got done with my RAID containing old iPhoto Libraries. This needs to be speeded up if possible.

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index

Dan Montgomery
Re: PreRoll Post - abort verify and release hard drive?
on Nov 4, 2014 at 3:45:51 pm

Who needs that iPhone Health app? lol.

Lots of files are simply going to take time. But there are a few things you can do. One is in the Settings>Video turn OFF the "Determine video type" feature. This setting creates thumbs for each known video file type, and also extracts the metadata for known types (timecode, frame rate, codec, etc.) so it takes processing time.

Alternatively, you can leave the 'Determine type' ON and set the advanced OPTIONS for it to ignore certain file extensions or small files, etc.

Also, on the Prepare Backup window, check the box to "Automatically begin backup after indexing is finished". This eliminates the need for your presence to hit the Begin button once indexing is ready (so you can keep on walking).

PS: We routinely test with hundreds of thousands of files. However, there is no need to write an entire tape at once. You may add to it in several smaller sessions. After all, it's not a WORM drive.

Offload with Confidence...


Return to posts index

Tim Jones
Re: PreRoll Post - abort verify and release hard drive?
on Nov 15, 2014 at 6:26:55 pm
Last Edited By Tim Jones on Nov 15, 2014 at 6:36:30 pm

And that's a perfect example of what make's BRU's Anytime Verify different from any other tools out there. Because BRU records the 32bit CRC values with every 2K of data in the archive stream (far more dependable that a file-based checksum), you are able to fully verify and audit a BRU tape at any time and even on a completely different operating system.

So, even if the source disk IS erased before you run BRU's verify, you can still perform the verify and audit of the tape's contents. It's always been this way with BRU - since 1985,

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]