FORUMS: list search recent posts

Archiving Strategy

COW Forums : Corporate Video

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
James Fuller
Archiving Strategy
on Sep 30, 2010 at 3:50:08 pm

I'd like to get some ideas about the most cost-effective and reliable archive solution for my workplace.

We average around 2 hours of footage per day need to keep it all safe. Everything is shot on P2 (1080p). We need to archive all footage as clients may approach at anytime to purchase footage, we might need to re-purpose, etc. I'm guessing the best strategy would be to store the raw MXF files? We'd also need to have an easily accessible, very small version to show footage to interested parties on the fly, remotely, etc. (I've heard this referred to as a mezzanine? True?)

Losing the footage is not an option, but as always, money is a concern. So - pay top-dollar for very high-end hard drives? With at least one backup? Two? Is BluRay being used for this? If HDs are the solution - suggestions on brands and recommended maintenance?

I've also heard about LTO tape, but am not sure how expensive or feasible this is?

I can't really imagine outsourcing this, but if anyone has ideas, I would love to hear them.

Thanks in advance!


Return to posts index

Mark Suszko
Re: Archiving Strategy
on Sep 30, 2010 at 4:47:23 pm

If it were me, I'd go with BluRay and FTP for sending to clients and internal archiving to a spare BluRay and hard drives. I would much rather mail out a cheap "disposable" BluRay dub than ship hard drives around. For those that can't afford a BD player there is the option of SD dubs on DVD or FTP-ing sample files.


Return to posts index

Noah Kadner
Re: Archiving Strategy
on Sep 30, 2010 at 6:51:32 pm

I dunno I think BD-Players are exceedingly rare. I'd go with cheap hard drives/Flash drives that can be charged for if not returned instead- at least for the client dubs. Every modern computer has a USB port...

Noah

Unlock the secrets of 24p, HD and Final Cut Studio with Call Box Training. Featuring the Canon 5D Mark II and 7D.


Return to posts index


Walter Soyka
Re: Archiving Strategy
on Sep 30, 2010 at 6:52:45 pm

Personally, I trust neither hard drives nor BD-Rs on shelves. Since acquisition has gone tapeless, a lot of production companies are looking to data tape backup for their archives.

I'm looking at LTO solutions from Cache-A and TOLIS Group.

Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog - What I'm thinking when my workstation's thinking
Creative Cow Forum Host: Live & Stage Events


Return to posts index

Mark Suszko
Re: Archiving Strategy
on Sep 30, 2010 at 8:32:56 pm

We had nothing but trouble the one time we tried to use DLT.
Your safest bet is storing in multiple formats and multiple locations.

As to BluRay skepticism, now that RedBox rents BluRay in the same box as SD DVD movies, (so does netflix) that tells me that more people own a BD player than we might think. Here's where I imitate a stuck record needle and repeat my line that BluRay is a format that has player options in PC drives, in Sony game consoles, as well as dedicated set top players, which means it is one of the very few HD formats these days that can go from computer room to newsroom to boardroom to breakroom to rec room. There is an armada of container ships full of cheap BD players and recorders anchored in Asian ports ready to sail for Christmas delivery, so I'm anticipating a drop in prices for the machines.

When we first internally started discussing moving from tape dubs to DVD for distro of a lot of our stuff, I pointed out that we could send every client of ours a brand new free DVD player along with his first DVD dub, and we'd STILL be saving money versus the shipping and stock costs of umatic and betacamSP dubs.

A DVD player at my corner Walgreens is now $30. BD players are a bit costlier than that, sure, but I think the analogy still applies. You can get a robotic mass dubber for BluRay disks for about a grand and kick them out with inkjet full-bleed printing at dozens per hour. What are you going to do with hard drives that compares to that, and are you really going to "throw-away" $75 USB drives on one-time dubs, or are you going to expect to get them back? How many of them do you think WILL get sent back? Right. And can you buy stacks of those hard drives in bulk like you can with blank DVD or BD media? No.

So, figure a dub at your internal costs for the time and effort, plus $75 for the drive and then some fees for fedex. Or spend $2.50 for a Verbatim brand BluRay blank, figure a buck for amortizing the dubber/ inkjet printer expense, a buck for a nice case, and ten bucks for shipping. I think just a couple of dub runs would pay for the "free" BD player or drive for those clients that don't have one.

FTP is cheaper still, but maddeningly slow for huge files. Until that improves, heck yes I think BD is the way to go: generate a self-storing permanent, high-capacity and durable shelf archive, ships easily and cheaply, fast and easy to duplicate.


If only I could get Steve Jobs to see things my way.


Return to posts index

Noah Kadner
Re: Archiving Strategy
on Sep 30, 2010 at 8:40:39 pm

Are we talking about Blu-ray for data or for just video clips? If it's just viewing copies I'd share HD H.264 clips with Drop Box and forget all the hassle of physical media anything.

It's less useless from an editing perspective as a Blu-ray disc and you waste a lot less of the environment with shipping, burning discs, etc. If we're talking about data the Blu-ray player you're talking about at a store is not going to be useful. And I would never expect to tell a client to drop a new BD drive into their machine just to access what- 50GB of data per disc. In short- yeah I see little utility for BD discs in a pro workflow.

Hard drives or file sharing are much more universal and just as cost-effective, and in some cases much more so. Besides- who wants to wait for a disc to arrive overnight and then spend hours on the phone getting the client to figure out how to play your BD disc- all the while blaming you for the tech issues.

Noah

Unlock the secrets of 24p, HD and Final Cut Studio with Call Box Training. Featuring the Canon 5D Mark II and 7D.


Return to posts index


Mark Suszko
Re: Archiving Strategy
on Sep 30, 2010 at 9:05:37 pm

It takes the same time for fedex to ship a hard drive as it does a bluray disk.

If you send by USPS weight becomes a cost factor; your drive in a padded shipping box will cost more than a BD, the BD will survive many drops and punting kicks along the way, the drive... maybe, maybe not.

No one answer is best for every and all situations, I will agree. And in cases where the content is more than a BD can carry, then FTP or shipping a drive/drive set is the imperfect answer for now.

But if you check out the discussions in the Digital Delivery forum, FTP is not a perfect panacea either. Upload time can be a bear if they demand uncompressed. The number of standards options for an FTP are mind-boggling as well. I would submit that for the great unwashed masses of non-technical users, putting something that looks like a DVD into a slot and hitting "play" is more goof-proof than sending hard drives ("Did you format this as FAT 32 or NTFS journalled? "What's THAT mean?" ) or FTP's ("Is this in Pro Res? Doesn't matter, it's a quicktime").

There is no one perfect answer, yet, is all I'm saying. But I think BR is the closest parallel we have to that era when most people you worked with had a Beta SP deck and knew how to use it. Format fragmentation is exponential at this point. I'm for settling on a common standard, even if it is not the best standard.

And I'm typing this while listening to music via MP-3, just to illustate that last point.


Return to posts index

Stephen Mann
Re: Archiving Strategy
on Sep 30, 2010 at 9:20:32 pm

Always save the highest-resolution that you can - basically, whatever file the camera makes. For dollars per Gb you simply can't beat hard-disk archives. If replacement is not an option, then make two (or even three). You can get 1Tb disks for less than $100 today.

Tape may have a higher density, but you are forever linked to a specific tape drive and format. Tough luck if your hardware fails and the supplier is either out of business or they don't support your format any longer. I have data diskettes from the 1980's that are still readable by any PC with a diskette drive.

Here's what I do. Once a year I buy enough new HDD to contain all of my archives. This last time, a few 1Tb drives did it. I then copy all of my media to the new drives. This does two things. First, I now have another copy on the old disks. (The old disks will eventually be retired as they become third-gen copies.) Second, by copying the files, data errors that a disk on the shelf can acumulate over time get error-corrected. (A DVD or BD DVD do not have the robust error-correcting that hard-disk does). Also, if a media file is so corrupted that it can't be recovered, then I should have another copy on an older archive drive.

Next year, I'll do it again.

Steve Mann
MannMade Digital Video
http://www.mmdv.com


Return to posts index

Walter Soyka
Re: Archiving Strategy
on Sep 30, 2010 at 9:40:07 pm

[Stephen Mann] "Tape may have a higher density, but you are forever linked to a specific tape drive and format. Tough luck if your hardware fails and the supplier is either out of business or they don't support your format any longer. I have data diskettes from the 1980's that are still readable by any PC with a diskette drive."

These problems are not unique to tape.

LTO is LTO, and you can buy the drives from multiple manufacturers. Backwards-compatibility is guaranteed across 2 generations (i.e., an LTO-5 drive can read LTO-3 and LTO-4).

The format on tape can be linked to a specific manufacturer, but again, this is not a problem unique to tape -- all my project files (Final Cut Pro, After Effects, Cinema 4D, etc.) are proprietary, too. The risk of a failed

Tape vendors are aware that you are buying their system because you care about keeping your data around, so they work to prove that they are solutions for the long term. TOLIS Group has tools that allow you to migrate from LTO4 (today) to LTO7 (doesn't exist yet), and they are willing to escrow their source code with you in case something catastrophic happens to them. Cache-A uses standard UNIX .tar archives on tape.

[Stephen Mann] "Here's what I do. Once a year I buy enough new HDD to contain all of my archives. This last time, a few 1Tb drives did it. I then copy all of my media to the new drives. This does two things. First, I now have another copy on the old disks. (The old disks will eventually be retired as they become third-gen copies.) Second, by copying the files, data errors that a disk on the shelf can acumulate over time get error-corrected. (A DVD or BD DVD do not have the robust error-correcting that hard-disk does). Also, if a media file is so corrupted that it can't be recovered, then I should have another copy on an older archive drive."

This sounds like a great system, provided you have the time to actually keep up with it.

Tape is very good at something that hard drives are not -- sitting on a shelf for a couple years, and then actually still working.

Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog - What I'm thinking when my workstation's thinking
Creative Cow Forum Host: Live & Stage Events


Return to posts index


Cory Petkovsek
Re: Archiving Strategy
on Sep 30, 2010 at 10:02:59 pm

Harddrive archives are very viable. They do not have error correction however; see below for details. Only use them in duplicate or more.

Data is stored in sectors on the disk, and indexed in a file allocation table (a catalog) at the beginning of the disk. There are usually two copies of this table.

Operating systems utilities for scanning the disk only do two things:

1) Verify the consistency of the file allocation table, with itself and with the backup if there is one for the filesystem being used.

2) Scan each sector to make sure it is readable, and possibly writable. (takes a long time and is usually not a default option)

If a sector has gone bad under your data, there is no way for you to know about it except by attempting unsuccessfully to read it, or by doing one of these scans. The drive does have a mechanism to seamlessly write to another sector when it detects a bad one. Thus you'll often never know unless you attempt to read and can't.

If the file allocation table index becomes corrupt the operating system disk utilities can often fix it by it's own cross references. However if it is not repairable then the data out on the disk is not recoverable by the IT lay person. There are no borders of files.

Imagine the library of congress, except we remove the covers from every book. Oh and the library is NOT sorted by section/subject, but random; And many of the books are split up; 50 pages on one shelf, 200 on another, 50 on another, 10 on another.

Now the shelves are filled with millions of bare pages, books randomly situated and not contiguous. Only the library catalog contains the exact beginning and end of each section of each book, which makes the system function.

If the library catalog gets corrupt; yes the data is there, but the only way to recover it is by manually scanning through the pages to recognize the title pages, and you hope that the whole thing is contiguous.

Many file types have magic numbers at the beginning of their data streams which acts like a title page. If it was split up in sections, not even an IT master can do anything for you. This is a terrible way to recover data. I've done it; it sucks.

"Error correction" specifically refers to a mechanism such as parity, CRC or ECC as are built into other technologies like server RAM or RAID systems. Standard harddrives do not have this technology. With their reliability and the above mentioned disk scanning utilities, it may seem this way, but its not the case. Error correction is like running the verify at the end of a DVD burn; except it happens in smaller blocks and it can fix it!

Finally, with moving parts, harddrives eventually will fail when the electronic motors give out. Thus plan for it failing eventually and you won't be disappointed. If you only use the drive once in a while it may last 50 years; who knows? If the motors give out, there is a high likelihood that the data is still readable on the platters of the disk. However you have to take it to a specialty shop for recovery and will spend $1500-2000 for a data dump, with no guarantee.



All in all, two copies of your archive data on two spindles (or disks or tapes), at two locations will reduce the probability of unrecoverable data to something inconsequential.


Cory

--
Cory Petkovsek
Corporate Video
http://www.CorporateVideoSD.com


Return to posts index

Stephen Mann
Re: Archiving Strategy
on Oct 1, 2010 at 5:29:34 am

Ummm. No.

FAT hasn't been used for a long time. NTFS has a much more robust indexing mechanism that is a lot less at-risk of losing a piece of the chain as FAT was. (When did you last have to run chkdsk /f to try to recover a lost chain?)

The drive hardware does have error correction built in. No magnetic recording medium is error free and error correction is used to hide those from normal operation. Further, the act of reading and writing (copying) adds another level of error detection. If there are bad sectors on the hard disk, your copy will fail.

I've been archiving on hard disks for ten years, and I haven't had one failure yet. (I do have a bunch of 10Gb drives to recycle.)

Steve Mann
MannMade Digital Video
http://www.mmdv.com


Return to posts index

Cory Petkovsek
Re: Archiving Strategy
on Oct 1, 2010 at 6:39:20 am

Steve
Ummm. No.
FAT hasn't been used for a long time.


"File allocation table" is a concept. Yes FAT12/FAT16/FAT32 are three implementations, one widely used today (FAT32 is often used on flash disks). Since you mentioned NTFS, that also uses the same concept. NTFS uses a relational database called the Master File Table (MFT) to store it's file allocations of the data store. Like FAT it has two copies and utilities like chkdsk run many consistency checks within the database and between the two copies, among other things. So FAT, MFT, same concept, but these details aren't really necessary to answer the op's question.

The drive hardware does have error correction built in. No magnetic recording medium is error free and error correction is used to hide those from normal operation. Further, the act of reading and writing (copying) adds another level of error detection. If there are bad sectors on the hard disk, your copy will fail.


In the IT industry, "error correction" is a term often used to describe something that verifies either with the actual source or with a checksum of some sort. Consumer level memory does not have error correction for instance. I checked my own knowledge and found that you are indeed correct. Consumer harddrives do have ECC built into them. For the Op, ECC will only verify during reading or writing, but won't help when the spindle fails.

As I mentioned to the Op, hard drive archival is a viable option, but with it's own caveats.

Cory

--
Cory Petkovsek
Corporate Video
http://www.CorporateVideoSD.com


Return to posts index


Walter Soyka
Re: Archiving Strategy
on Oct 1, 2010 at 1:13:28 pm

[Cory Petkovsek] "Finally, with moving parts, harddrives eventually will fail when the electronic motors give out. Thus plan for it failing eventually and you won't be disappointed. If you only use the drive once in a while it may last 50 years; who knows? If the motors give out, there is a high likelihood that the data is still readable on the platters of the disk. However you have to take it to a specialty shop for recovery and will spend $1500-2000 for a data dump, with no guarantee."

This is why I'm leary of disk drive-based backups. They are an integrated package: the controller, the motor, and the read/write mechanism are all bundled with the actual storage medium. A failure of any one of them means data loss, and as you noted, repair is not trivial.

Tape separates the storage medium from the control and access mechanism, so it's more robust over a longer period of time. If the tape drive dies, you replace the tape drive, and your tape library will remain accessible.

Cory, I completely agree with you that duplication is necessary for archiving. But if you're going to consider a hard drive-based archive, I think that Steve's systematic and frequent refreshing of the archive is also critical.

Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog - What I'm thinking when my workstation's thinking
Creative Cow Forum Host: Live & Stage Events


Return to posts index

Cory Petkovsek
Re: Archiving Strategy
on Sep 30, 2010 at 9:35:39 pm

I have an IT background of 15 years; 9 as a consultant. Backup is a regular part of my business. It saved my butt last night when I accidentally over wrote a design file!

The original op had two parts of the question: 1) How to archive masters 2) How to make information readily available to clients. I would recommend treating them differently.

1) How to archive mass amounts of data
I would archive the master camera files, and all your work project files. Whatever you retained during the project and can't regenerate (transcodes I'd probably delete).

Optical storage and tapes are the cheapest storage available, with tapes being the largest investment, but cheapest GB/$ ratio.

Starting with tapeless acquisition and archiving to tapes is.... well amusing.

Anyway, tapes are susceptible to tape errors and data loss. Bluray discs are susceptible to scratches. Neither can hold up to heat or mishandling. Still either is a solid storage solution with proper care.

Harddrives with spindles have a 100% failure rate. You can use them (I do), as long as you always have multiple copies. At any given time I have copies of my data on 2-4 different spindles. RAID is not backup, it is used for fault tollerance which is a different issue (ask if you want details).

SSD drives can fail; I have had one do so. They are not perfect either. Poor GB/$ ratio.

Large flash sticks could work. Not much room for sharpies; not designed for stacking/cataloging; And the GB/$ ratio isn't there.

Regardless of the hardware, I recommend at least dual storage. When the archive is made, make two copies. One is stored on site, the other off site: at your home in a safe (if client data is sensitive), or locked up at another environment controlled facility.

You should target at least $750 for a single midrange backup drive, or at least $1500 for an autoloader. Exabyte (now Tandberg) make some fairly high GB per $ ratios for lower end data storage. There are many good manufacturers and tape technologies.

http://www.tandbergdata.com/us/index.cfm/products/tape-drives/

They have a VXA-320 which holds 160GB per tape (320 w/ compression, but not in video!), or you can get their autoloader which holds 10 tapes (useful for backing up your server, run by automated backup software).

LTO tapes are the top end; they have up to 1.5TB tapes! But a single drive w/o autoloader is $2500; $10k with.

DLT, SLR and DAT are other technologies. When researching consider data path from the machine to the drive (USB/Firewire/SCSI, etc), the cost of media, the GB/$ ratio, the transfer speeds, and of course the cost of the drive.

Finally, you'll need tape backup software. Many of these drives will come with a free version of backup software. Some of that may be fine. Windows NT, 2000, XP, 2003 used to come with backup software that could run tape hardware. They removed it in vista.

Backup Exec is industry standard (as is Arc Serve and some others) for automated backups, but it may be overkill for what you need unless you get an autoloader or want automated backups of your network. Don't get Backup Exec System Recovery; that's a very different and confusingly named product.

There are other cheaper, simpler and probably free tape backup software programs. Just note it's not as widespread as CD/DVD burning software because it's not a consumer level technology.


Bluray discs on the other hand have low investment cost. $100-200 for a bluray burner for your desktop or laptop, burning software, a spindle of discs and a sharpie. Just make sure to get discs that don't have any longevity/deterioration problems as some DVDs did.

I currently work on one system and backup to my server in two locations on different spindles. For now it works, but when I'm ready to move into archives I'll switch to Bluray. Already some of my projects are 100-150GB which will take multiple discs. At some point I may switch to LTO or VXA tapes, but only when my the bluray plan is taking too long or too much management.


2) How to show clients
This is a very different issue that needs it's own strategy.

Setup batch processing to compress out smaller versions of your files and store in a public/easily accessible place. Scripting with ffmpeg is an easy way to do this, as long as it can read your mxfs.

Then you can send them any BD/DVD/Flash disc or have it online.

Cory

--
Cory Petkovsek
Corporate Video
http://www.CorporateVideoSD.com


Return to posts index

James Fuller
Re: Archiving Strategy
on Oct 5, 2010 at 10:05:57 pm

Thanks to all for the feedback. As usual it seems there is no one correct answer - but all of these thoughts are helpful as I determine the most appropriate strategy.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]