FORUMS: list search recent posts

Checksums for entire AVCHD or Canon XF folders?

COW Forums : Archiving and Back-Up

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Seth Goldin
Checksums for entire AVCHD or Canon XF folders?
on Feb 28, 2014 at 4:12:59 pm

Hi all,

I'm looking to improve my post-production workflows, particularly at the point of ingestion.

I work with a variety of cameras and formats, including codecs not neatly wrapped in QuickTime wrappers, like AVCHD and Canon XF.

I know that ingestion applications like Adobe Prelude or ShotPut Pro offer MD5 checksum functions for individual clips, but it seems like that would only be useful for footage that has clips individually wrapped with all the metadata, not folders that have critical metadata sprawled all over the place, like AVCHD or Canon XF.

So, what are the best practices to verify the integrity of ingestion of codecs like AVCHD or Canon XF?

Thanks,
-- Seth Goldin.
Freethink Media
Washington, DC
@sethgoldin


Return to posts index

Tim Jones
Re: Checksums for entire AVCHD or Canon XF folders?
on Feb 28, 2014 at 5:48:05 pm
Last Edited By Tim Jones on Feb 28, 2014 at 5:49:33 pm

md5 theClipFolder

Retraction - I just discovered that the OS X md5 utility won't operate on a folder...



Return to posts index

Chris Murphy
Re: Checksums for entire AVCHD or Canon XF folders?
on Mar 3, 2014 at 11:01:18 pm

find -type f -name *.jpg -exec md5 {} +

Instead of jpg put in the file type you're after.


Return to posts index


Tim Jones
Re: Checksums for entire AVCHD or Canon XF folders?
on Mar 3, 2014 at 11:22:37 pm

It appears that the only way to get this to do what Seth is looking for is to use 'shasum' to create a checksum list for the original media folder, copy that file into the destination folder after the copy, and then run the 'shasum -c' option to compare those sums against the files in the copy folder.

mount the card (/Volumes/CanonCard for this example)
cd ino the mount point
find . -type f -exec shasum {} >> checksums.sha \;
copy the clips to the new location
copy the checksums.sha file to that same folder
cd into the new folder
shasum -c checksums.sha | grep -v "OK$"


That should only print out the files that don't match.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.productionbackup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

Tim Jones
Re: Checksums for entire AVCHD or Canon XF folders?
on Mar 3, 2014 at 11:25:47 pm

Another option would be to use the "diff" utility:

Mount the camera card as before
copy the clips to the new folder
diff -q /Volumes/CameraCard /Volumes/Array/NewFolder

that will only print the names of the files that differ.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.productionbackup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

Chris Murphy
Re: Checksums for entire AVCHD or Canon XF folders?
on Mar 4, 2014 at 12:36:08 am

If the idea is to avoid most storage stack related file corruption and make it easy to use, the solutions are: a.) use enterprise drives, since they have an order of magnitude fewer unrecoverable errors and corruptions than consumer drives; and/or b.) use a file system that checksums every block and uses data redundancy; any detected corruption causes the replicated data to be used instead. Examples are ZFS and Btrfs. Of course, this means non-local storage.

It also does nothing for ingestion. There are no file system mechanisms with exFAT, NTFS, or HFSJ/HFSX to detect or correct data corruption. So you're at the mercy of the media being used by the camera, the card reader, its drivers, the local computer and its drives, and the network. Ideally, there'd be ingestion right at the storage stack itself to limit the opportunity for corruption to occur.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]