I've been working on multiple projects for one client over several months. I created a bunch of the footage, but a lot of footage/GFX came from outside sources. To try to manage the elements, I was keeping all of the source footage on one 4TB RAID 5. As crunch time approached segments were added, additional shooter/editors were brought in, and many files were duplicated and transferred to additional hard drives. So the next step is the tedious process of comparing dives and files and weeding out the duplicates so I can archive (DLT).
I found that I can export a batch list and paste that into an Excel spread sheet and it places everything into an appropriate column. Then, by sorting, I can reorder the source files into the correct rows. This has given me a hard copy of all sources to reference for the weeding process. I will still need to manually delete duplicate files, but that why they call it work and not play:-)
I was wondering how others might be approaching this type of library/archiving exercise.
I have the same issues and I'm looking into Mac Magna's File Finder (http://macmagna.com/filefinder/index.htm) as a way to find duplicate files and as a way to manage multiple hard drives. What's your DLT setup?
I do this for different reasons (to always have multiple copies of all critical data), but the process is the same. On Windows machines I use a program called GoodSync and on Macs I use Tri-Backup to pretty much automate the process of two-way drive synchronization. The programs compare two data sources (individual files, folders, entire drives, etc.) and synchronize them. For example, for each "work" drive I read/write critical data to/from, I keep two synched copies that I don't work from (backups only). If drive A fails, it's replaced by drive B, or, if drives A and B fail, I pull drive C off the shelf ... zero data loss or downtime.
If you go the automated route, it's important to understand the software well before diving in. For example, many synchronization programs have an option to propagate deletions ... meaning if a file has been deleted from drive A, it will be deleted from drive B upon synchronization. A very useful feature that saves lots of time from tedious manual tasks, but it can also be dangerous if one isn't careful, which is why it's generally optional.