Storage and Archiving
by Andrew Freiband
on
Mar 10, 2008 at 4:57:58 pm
I'd like to hear some more members' practices in archiving and storage. With a large and growing library of short (under 30 min each) projects, I'm working toward moving off of Beta SP because maintaining analog SD equipment is becoming impractical as well as incompatible with a transition to HD. Projects are coming off in 10 bit uncompressed, and currently live on hard disk. Is that a good enough home for the next 20 years? What are people choosing for this problem in their own work? An archive, it seems to me, should not be a diminished-quality version of the piece, so any of the compressed tape formats don't sound like good choices. Let's hear what you've got.
Re: Storage and Archiving by Cale Mooth on Mar 10, 2008 at 6:58:20 pm
It's going to come down to how you input your source video. For example, if you're going SDI out of the camera straight into the computer, then 10-bit uncompressed is the way to go with archiving. If you're shooting on say HDV or DVCPRO tape and then import that into the computer, I would question whether a 10-bit uncompressed archive is necessary. With the digital formats(DV, HDV, XDCAM, etc.) you can go from camera to tape to computer, edit, then go back to tape and not lose any image quality since all you're doing at that point is moving data around. You're not recompressing.
Long-term harddrive based storage just feels too risky at the moment. I have drives that are 10 years old and younger that have broken down mechanically just from sitting on the shelf. I have tape that is over 20 years old that is still holding its information.
If you're capturing uncompressed footage maybe consider a DLT based archive solution? But if you're shooting in a particular digital format, I would archive to that same format using tape simply for the reliability and the lower cost.
Re: Storage and Archiving by Andrew Freiband on Mar 10, 2008 at 11:28:42 pm
Yeah- my archiving issue is twofold: half is generated as 10-bit uncompressed, and the other half is a hodge-podge of camera-derived formats. The archive should really be highest common denominator, otherwise it's not a very efficient archive... Besides, call me a skeptic, but I don't love the long-term chances of any of our current crop of proprietary HD camera formats. Then again, I have drives that are ONE year old and have failed... I know this is a wide open problem; I'm curious as to how the rest of us (you) are coping.
Re: Storage and Archiving by Bob Zelin on Mar 17, 2008 at 11:14:16 pm
Perhaps Blu Ray DVD will be the "answer" - but I don't know anyone doing this for archiving purposes as of now. I do know MANY companies that are using disk drives to create their Beta archives, and in my opinion, they are JUST NUTS, because as you know, there won't be a computer in the world that will read these drives 10 years from now. Try to read old MAC OS AVID SCSI drives, try reading Iomega Zip disks.
Newer products like Apple XServe RAID's are on their last legs, so imagine an inexpensive FW800 drive (like a Lacie drive) 10 years from now - pretty funny - huh ?
Re: Storage and Archiving by Bob Cole on Mar 22, 2008 at 12:44:20 am
LTO tape looks like the best bet, and Quantum has a terrific, MXF-aware device, but it is around $7k, and that is probably more than most people want to spend for something they hope they never need. The bottom line is reliability, and tape is more reliable than hard drives. BluRay lacks capacity.
Due to cost, I suspect most of us will use hard drives for source media, BluRay for finished shows, and live with the fact that the hard drives with our source media won't last forever.
Maybe it's time for a reality check. I've diligently kept shelfloads of tapes, and what happens when I need to use some old material??? I go back to the edited master, because it already has the "good bits," and it's faster than searching the shelves for the right camera tape.
Bob C
MacPro 2 x 3GHz dualcore; 10 GB 667MHz
Kona LHe
Sony HDV Z1
Sony HDV M25U
HD-Connect MI
Betacam UVW1800
DVCPro AJ-D650
Re: Storage and Archiving by Andrew Freiband on Mar 22, 2008 at 3:56:20 pm
I'm a newcomer to the LTO option, though I've heard it bandied about in archiving discussions before. The issue there seems to me that a new version is released about every 2 years, since LTO-1 in 2000. But the format's own stated goal is only to be able to read from cartridges that were written in one of the two prior versions. I don't know how it has been in actual practice, but LTO's manufacturers are content to say that LTO-4 drives, the most current version, should NOT necessarily be able to read LTO-1 tapes, which means data written only as long ago as 2002 isn't accessible anymore, without a hefty data transfer and an expensive hardware update.
Is this just manufacturers setting expectations lower than actual performance? And if not, if LTO is this limited in retrieval ability, how do we use to archive our work without hoarding machines now and putting them in cold storage?
Re: Storage and Archiving by Bob Cole on Mar 22, 2008 at 4:43:01 pm
Great point about the format technology itself becoming obsolete. I don't see this as a major problem; for one thing, you can always keep transferring your digital tapes to new digital tapes. And if you miss a couple of generations, it won't be hard to find machines to play your old data tapes. You can still get 3/4" video decks to play your old videotapes. Given the size of the computer industry, old tape drives will be vastly easier to find than old 3/4" decks.
To clarify, the selling point of tape is that "the stuff sticks" -- if you have a machine capable of reading the tape, the data will be there for decades. Supposedly, this is not reliably true of hard drives.
I suspect that the "camera tape archive" concept is going away anyhow. Where we once religiously preserved all that was shot (even though most of us only went back as far as the masters), now we're going to archive only what was actually captured to the computer that edited the show. In some cases, that will be the whole camera tape.
MacPro 2 x 3GHz dualcore; 10 GB 667MHz
Kona LHe
Sony HDV Z1
Sony HDV M25U
HD-Connect MI
Betacam UVW1800
DVCPro AJ-D650
Re: Storage and Archiving by Bob Zelin on Mar 22, 2008 at 7:02:11 pm
your analysis is correct. Todays drives won't run on anything 10 years from now. FW800 drives are ok for temporary storage (and they are cheap), and Blu Ray is good for archiving finished shows, but there is no great solution, and there may never be a great solution, because the buss structure becomes obsolete.
I will give you a simple stupid example on my own backup on this little computer I have here. I have a "backup" internal drive, but it's an "old" HP xw4100. Starting with the xw4200 and newer (4400, etc.) they no longer use ATA interface connectors for the drives, but SATA. So how will I get this info onto another computer if it dies. I have backup CD's of the important info, but of course, CD's are small. Same analogy for video media. FW800 in 10 years will be considered a joke, just like Iomega ZIP drives are considered a joke today. I dont' know what the long term solution is.
Re: Storage and Archiving by Ramona Howard on Mar 25, 2008 at 4:27:05 pm
Unfortunately there isn't a single answer to this problem.
Short term archiving can certainly be done with drives (in a raid configuration of course). Long term we have seen done on everything from CD to film.
Drives certainly are also used but in a larger scale than a single firewire or SATA drive. Drive failure is certain so it is factored into the solution.
The solution is usually going to be what best fits your pocketbook because there are numerous ones to choose from and there will always be newer and better ones that come along.
The problem is like a squirrel and his stash of nuts. Constantly shuffling and re-assessing.
Our new Bravo solution, a system that currently holds SAS or SATA drives but that can adapt easily to future needs was not originally designed for this (archiving) but we are finding out that many are looking at it solely for this purpose.
It is a 2 SixPak canister unit but each canister can hold 1 to 6 drives and each canister can be a different type of drive. i.e one canister can hold inexpensive, slow SATA (Raid of course) while the other a newer faster drive. The benefit is in the shuffling. Going from Pak to Pak is as fast as the content can come off the disks. It's all over the buss, no outside protocols to deal with.
Tie in an LTO, DVD or CD Rom or yes even a Zip Drive to the system and now you have an all in one way to shuffle away :)
Re: Storage and Archiving by Eric Peters on Jun 19, 2008 at 12:20:01 am
We are currently looking into the cheap storage/archiving realm, trying to come up with a cheap solution for storing a digital backup of all footage shot for an XDCamHD project (there will eventually be ~75 TB of footage at full resolution). We are currently thinking of just loading up bare 1TB SATA drives and then throwing them into a JBOD chassis when filled up. The chassis would then sit idle with the drives in it until a time where we needed to go to this emergency backup (essentially we are looking this as a form of "fire insurance", keeping the JBODs in a different building from the master tapes).
The question is... Has anyone tried something similar to this? We are looking at one of the chassis that RaidAge makes: http://www.raidage.com/products/mage316u40pcie.aspx# Anyone have any experience with any of their products?
Re: Storage and Archiving by Bob Cole on Jun 19, 2008 at 12:24:25 am
Repeating myself, but I think we need to look beyond our own industry when it comes to archiving. From what little I know, Quantum tapes like the LTO-3 are what the "big world" of computing uses. Hard drives are highly problematic.
Bob C
MacPro 2 x 3GHz dualcore; 10 GB 667MHz
Kona LHe
Sony HDV Z1
Sony HDV M25U
HD-Connect MI
Betacam UVW1800
DVCPro AJ-D650
Re: Storage and Archiving by Eric Peters on Jun 19, 2008 at 12:51:03 am
In general, I would agree, but in the case I am talking about, it is a very temporary "fire insurance" type backup that would get blown away off the drives as soon as the project is finished. As such, the cost of bare drives is pretty hard to beat for any other backup storage solutions out there.
Re: Storage and Archiving by Ramona Howard on Jun 19, 2008 at 5:24:13 am
Eric,
I would still go with a Raided solution, even if it is temp. The loss of any data can be painful and one that does not need to happen. OK so you loose one drive in the mix if you go with Raid5, at the cost of drives today it is a wise decision. If you can't afford to do that then I would simply look at LTO for achieve.
If you can go back to original material, then why archive this to begin with? If you are talking about archiving any work in progress, then RAID will be your friend but in that scenario, the EDLs are far more important than the content and these should be looked at the most valueable assets (as long as you can go back to original content).
75TB is nothing these days for a large facility and I guarantee they are running it in Raid so if your going this route please consider that.
Just because parts are "cheap" the solution may not be. Choose wisely, otherwise you are simply playing Russian Roulette with your livelihood.
