archiving P2 cards
by Gerret
on
Oct 13, 2007 at 11:50:32 am
I've been shooting with my HVX200 for a year and have backed up all media to pairs of OWC 500 GB drives, but I've continued to worry about how to archive long-term. Quantum has a drive for such purposes but it costs about $6k, too much for me. I've tried to research LTO as an alternative, since it seems well established in the IT departments of the corporate world, but I haven't been able to find much information, especially for Macs.
So my current answer is to use dual layer DVD. At 8.5 GB each one will hold the contents of our 8 GB cards. When we move up to 16 or 32 GB cards, the drive and Roxio's Toast will allow recording across multiple DVDs. At around $2/disk it's cheaper than archiving to a drive, and may even be more stable... though none of us will know until it's too late.
Barry Green was kind enough to point out the importance of write-protecting cards before moving them into a Mac, and I've continued that protection by locking the files once they're on my drives. If you don't, you'll find that copying on the Finder level will regularly yield errors.
So far I've shot about 500 cards in the last year and lost only two: once in erasing before transferring and the other, who knows. I love the quality of the footage, but file management has been much more work than I ever anticipated. If you have any experience with any of the P2 workflow on a Mac--especially archiving-- and can recommend strategies I'd welcome them.
Re: archiving P2 cards by Tim Wilson on Oct 13, 2007 at 3:09:44 pm
Maybe it's just me, but I'd never consider a DVD backup. For anything really, but certainly not something as valuable as footage.
I have a couple of reasons for saying that:
--The burn process can be flaky. Are you willing to verify every file after you burn?
--Recordable optical media is notoriously unstable. I've had disks fall apart after a couple of plays. I've only got a handful of self-burned disks that are still error-free after a couple of years.
--I know you didn't mention blu-ray Gerret, but has anybody here burned some? I haven't but it seems like they'd take a year to finish.
--Anyway, compare the unpredictability of optical media to hard drives. I just found a couple of Firewire/USB drives that I'd completely forgotten about. Took 'em along around the world while I used 'em, then in a box with no special protection for moves to 5 different houses in two states -- and they came through like champs. Two years later, I'm still using them.
These are cheap, off the shelf consumer drives from 10 years ago. Newer, reliable, pro drives seem like the way to go for me...even if they're not deep archive drives.
Not that any of this is specific to P2, just general storage thoughts.....
Re: archiving P2 cards by N.Adam.Smith on Oct 14, 2007 at 6:59:52 am
I went with an Exabyte VXA2 Packet Tape Drive for P2 and FCP project backups, as I like the fact that it verifies as it writes, and of course magnetic tape is still the best solution for long-term storage.
I initially copy from the camera to an external firewire drive (with verify), then I use P2CMS to copy (with verify) to internal HD. From there I back up to my tape archive and I'm set.
--
Video Photographer / Avid Editor / Final Cut Neophyte
Re: archiving P2 cards by Gerret on Oct 14, 2007 at 12:08:07 pm
Thanks Tim and N. Adam,
I've heard some criticism of DVDs for archive, but they're probably the best compromise for now. While working on our projects--often for many months--I keep two hard drives of the MXF files. The DVDs are a 3rd, and are then given to the client with the warning to back them up. And yes, I do verify all burns.
I'd love to burn to Blue Ray, but I haven't found one yet. Currently it takes me an hour per DVD on my LaCie to burn/verify an 8 GB group of files.
The Exabyte looked good, but their site says it's for Windows only, and we'll all Mac.
Re: archiving P2 cards by Gerret on Oct 14, 2007 at 3:55:02 pm
Since I wrote earlier I've found a firewire Blu Ray drive at OWC, and--according to the review they they link to--burning a 25 GB disk takes 45 minutes. Compared to my 1 hr per burn/verify 8 GB that's hard to imagine.
The real question is whether you, Tim, are right about DVDs being too unstable to serve as an archive of footage. Anyone out there know?
Re: archiving P2 cards by Tim Wilson on Oct 14, 2007 at 4:22:30 pm
[Gerret]"burning a 25 GB disk takes 45 minutes. Compared to my 1 hr per burn/verify 8 GB that's hard to imagine."
Well there you go. I just assumed that it would take the same amount of time per gig.[Gerret]
"The real question is whether you, Tim, are right"
Never much question about that. :-)
Mine was strictly an anecdotal observation, nothing scientific.
I continue to use DVDs, even CDs, all the time as transportable storage. They're still the best media for that job. They're cheap, they hold a lot, and they're disposable. I'm just not using them for anything I want to keep around for long.
Re: archiving P2 cards by DevilDodo on Oct 15, 2007 at 11:30:43 pm
I've been doing a little research into archive solutions with regard to P2 media.
After looking into LTO tapes, Hard drives, DVDs and Blu-ray there seems to be no clear winner. Here's how it breaks down (all prices in NZ dollars):
LTO3:
- Most expensive setup cost (the cost of the drive itself) at around $2000. In terms of tapes, it works out to about $0.19 /GB
- Large storage space (200GB or 400GB compressed) on the one I looked at.
- Approx. 30+ year lifetime.
- Write speed is quite slow when compared to DVD. Read speed is quite slow (due to the fact it has to do it sequentially. IE it's not random access).
HDD:
- Relatively cheap, around $0.48 /GB with no setup costs.
- Storage space varies, but significantly larger than DVD or Blu-ray.
- Unreliable. 3-5 year lifetime.
- Harder to store.
- Very fast write speed, random access.
DVD+R:
- Cheapest of all options at $0.18 /GB.
- Comparatively very small storage space.
- Fast write speed compared to Blu-ray and LTO.
- 10-30 year lifetime.
- Less reliable. Prone to damage.
Blu-ray:
- Most expensive at $1.41 /GB. Setup cost of around $1200 for burner.
- 50GB at dual layer. A lot larger than DVD, but still small when compared to LTO or HDD.
- Slow write speed (around an hour per 25GB depending on burner).
- Apparently very robust. 100+ year lifetime.
Take from that what you will. This information was taken from various places on the net, so is by no means definitive.
A few thoughts of my own: To me, Blu-ray seems like a very viable solution. The 100+ year lifetime may sound a bit ridiculous, but I've spoken to several people who have said that they are incredibly robust and that really the only way to destroy the information is to physically snap the disc.
Of course, a huge advantage to the Blu-ray solution for those of us in the video profession is that buying the bruner gives you the ability to master high def Blu-ray discs (with the right software, of course). Which is a huge plus, espcially if working with the HVX.
And prices are coming down on the burners fast. I compiled this information only a couple of months ago, but recently I've seen a burner for $800 that claims to write 25GB in 25 minutes!
My research into HDD storage has been widely varied. Ultimately I've concluded that it is complete luck as to when or whether you HDD will crash on you. I've obviously had a lot of bad luck - in the year we've been in business we've lost a total of six external drives. Completely randomly. Thus, I personally have vowed to never use them again as a storage system.
At the moment we've resorted to using DVDs to archive until we make a decision. It's been working fine so far... Though it is a pain having to manually split the 16GB folders into 4GB segments...
Re: archiving P2 cards by n.adam.smith on Oct 16, 2007 at 4:22:57 am
[DevilDodo]"DVD+R:
- Cheapest of all options at $0.18 /GB.
- Comparatively very small storage space.
- Fast write speed compared to Blu-ray and LTO.
- 10-30 year lifetime."
Hmm.. last info I heard was about 6 months ago and that research was showing that the dyes used to form the "pits" in burned DVDs can begin to fade in as little as 5 years.
--
Video Photographer / Avid Editor / Final Cut Neophyte
Re: archiving P2 cards by DevilDodo on Oct 16, 2007 at 9:04:20 am
Yeah, I've actually heard similar things with regards to CD-R's failing after about 5 years.
As I said, this info is not at all definitive... And worse, is probably mostly speculative. Hell, DVDs have barely been around long enough to be able to live up to its claim of 10-30 years... And we'll have to wait 100 years to find out if Blu-ray lives up to its claim!
I think it also depends on the type of DVD-R you get as well. I believe there are specialist archive-grade DVDs around. Though my pricing was based on a standard spindle of 50 DVD-Rs.
Also, one thing I forgot to mention was that I've been warned off Dual-layered DVDs... apparently they're even more unreliable and more prone to data loss than DVDs. Also, I've just realised that I had "+R" in my original post... My pricing was actually based on "-R"... not sure if that makes any difference...?
Re: archiving P2 cards by Arthur Aldrich on Oct 16, 2007 at 9:30:37 am
I have been using blu-ray for about 1 year now as one method of archiving p2 cards.
I bought a Panasonic SW5582 internal and Toast 8. The drive cost me about $1000, but now I have seen them for less than $500.
The burn time is 45 mins for a 25GB disc, and about 90 mins for a 50GB. These speeds are of course optimal, and not average.
A 50GB disc may take 2 hours to burn and another 2 hours to verify.
The 50GB discs are down to $30 per disc. Maybe less if you can buy in bulk.
You can use Toast to burn data across multiple discs, but the size of the discs must match. So if you want to burn 60GB of data, you will need to use 3 25GB discs or 2 50GB discs.
I also have a Quantum 600a dlt drive with the mxf support. I like this format for projects that are larger than 50GB. Each tape holds 300GB.
Re: archiving P2 cards by Gerret on Oct 16, 2007 at 11:54:46 pm
Thanks to all, especially Devil & Arthur. Quantum and Blue Ray seem the best P2 archiving solution, though I'm still concerned that Blu Ray could be the next Betamax. On the other hand, who cares if HD-DVD wins the big battle, so long as Blu Ray remains for data storage.
Anyone else know about the dependability of DL DVDs. I've got a lot of time invested in backing up MXF files to them, and am hopeful they're secure.
I wonder, Arthur, if the Quantum solution is worth the high expense?
Re: archiving P2 cards by n.adam.smith on Oct 16, 2007 at 1:37:49 pm
[DevilDodo]"Also, I've just realised that I had "+R" in my original post... My pricing was actually based on "-R"... not sure if that makes any difference...?"
No idea about longevity, but I heard and been directed to a few articles that suggest +R is a better format, burning faster and with fewer errors. Haven't experienced it, but it was interesting enough info to make me wish I could switch over. Too bad none of our (older) Macs at work will read a +R disc...
--
Video Photographer / Avid Editor / Final Cut Neophyte
Re: archiving P2 cards by mattjgerard on Oct 22, 2007 at 3:44:50 pm
I guess no one has ventured to ask, How long do we WANT to archive this stuff?
I am now using a device from weibetech, its a trayless SATA drivebay for Hard drives. I buy 300 Gig Seagate SATA drives for about $64 each, and slip one into the drive bay,( no trays, really nice) and archive my whole project as it sits on my xraid. If I need to acess somethign from that project, I just slip the drive in again, copy what I need off the drive and take it out again. Fast, no special unaccessable formats (retrospect) and. I keep them in anti static bags, and stored well. The only problem I've had with drives are LaCie's, don't get me started on those.
Been working well so far, and with our deadlines, we can't wait to unarchive stuff off Tape, heck who has time to spend 4 hours burning projects to DVD's? I surely don't. Plus I hate having to unarchive the whole project just to get a couple of files.
Re: archiving P2 cards by Shane Ross on Oct 22, 2007 at 4:14:37 pm
[mattjgerard]"I guess no one has ventured to ask, How long do we WANT to archive this stuff?"
10, 20....30 years? A while. I have stuff I shot back in 1987 that I keep around...and transferred to DVCAM from 3/4" Umatic. I found a use for it.
Hard drives...3-4 years tops for archival. Drive types and connections change so fast. SCSI, IDE, SATA...in a span of what, 10 years?
Also, how long will DVDs last? Blu_ray? No one knows? I do know that a few of my early CD burns are failing...but that was early technology. Blu_ray is early technology too...
Re: archiving P2 cards by mattjgerard on Oct 22, 2007 at 4:43:22 pm
Yes drive connections do change, and that is why I still have a syquest EZ135 drive sitting in my drawer!!1
For 90% of our clients, we will backup their show to a selects reel, to tape. Our biggest client, this works great for. they aren't interested in keeping the OK shots, they just want to archive the best of the best. After the show is done, we make a selects reel to Tape(usually DVCPRO50), we archive thier show to a HD, sit it on the shelf. I have asked around the industry, and no one know the shelf life of a HD, just sitting there. I don't know why, but I feel more comfortable with a HD than with DVD's or CD's. Maybe I have just had so many of them go bad on me.
The neverending debate, will, I fear, never end. But, its good to hear pros/cons and other's ideas. My boss is a real fanatic about ease of use, and convienence. About the only way i could get him to use packet tape BU's is to get an auto loader library system. But he just dropped 60G's on a HPX2000 setup, so that might have to wait until next year....
Re: archiving P2 cards by Steve Sherrick on Nov 18, 2007 at 8:11:25 pm
Budget and time are a factor here. Here is my list of factors for archiving.
1. Reliability - How safe is this precious data? How long will it last?
2. Read/Write times - If I'm doing the archiving on set, what is the speed at which I can perform these archives? In a post environment, you may be able to setup stations for doing this that do not have time stringent limitations.
3. Cost - What's the price that I have to pay to get the reliability and speed I need.
4. Client expectations/willingness to pay for it
I have to weigh all of these things carefully, and determine the best price/performance for the given situation. For me, I'll be backing up P2 and Redcode Raw files so I am looking at robust tools. Losing even a single take is unacceptable. Tape backup is tried and true. Hard drives are not all equal, and same goes with optical media. Brand does matter, but even then how much trust do you put into it?
Somebody has to pay for the expenses involved with archiving because it takes time and materials and both cost money. Can we be expected to backup to several media formats indefinitely? In my opinion, the client has to help out with these costs and they have to determine how long they expect us to archive their projects.
Here is what I intend to do on future projects.
1. Once the aquisition format is determined (for this example let's use P2), we determine how many P2 cards will be needed on set, usually determined by how many the production can afford. Or if they are using a Firestore, how many will fit into the budget.
2. Determine how much footage on average will be aquired in a given day. For this example, let's say the client is shooting in 720P 24N and the camera package incudes two 8GB P2 cards for a total of about 40 minutes of footage. It is estimated that about 3 hours of footage will be aquired that day. So about nine P2 cards will be used during the course of the day. So at a minimum, you will need to be archiving seven P2 cards on set, with the other 2 potentially being backed up later.
3. How will we back up those P2 cards? In this case, the content is aquired on a P2, then at some point that P2 is brought over to the backup station and needs to get archived before it is needed again. Once it is properly archived, it needs to be formatted. There is the rub. Once that card is formatted and new material is shot, we can't get it back, so whatever we just archived has to in fact be a replica of what was on that card. So what do you put your faith in? Hard drive? DVD? Tape (DLT, LTO)?
That is why a bulletproof setup has to be in place. Even if we had ten P2 cards on set and we could bring them back into post and back them up in several places, eventually those P2 cards go back out the door and get formatted again. There will be no chance to compare the data later on if a question comes up about the integrity of the backup. This is the challenge to file based aquisition. With the Red cameras, there is talk of treating the compact flash media just like you would a film reel or a tape. It gets put on the shelf as a master. At $200 for an 8GB CF card, that's a pretty expensive master, but could be justified in some situations (feature film). But at $899 for a P2 card, it's just cost prohibitive.
So when it comes time to archive this project for the client, what are we keeping on the shelf? Let's assume we have decided to backup the P2 cards to a SDLT600a. The client shot 200GB of material and we archived that to a single SDLT tape which has a capacity of 300GB. That tape then get's put on the shelf as our production master. The material had been transferred over to Firewire800 drives for editing, and in this case FCP wrapped the files in a Quicktime blanket, creating new files. Through the course of the project, new files are imported such as audio, graphics, music, etc. Now we have a finished piece, and we output to a DVCProHD master for shelf and delivery. What now to do with the material sitting on the drives. Our 200GB of material has blossomed to 280gb of material with all the additional content. We could backup once again to the SDLT which could fit this onto a single tape. Not a bad solution. We could fit it onto a bare hard drive using the weibtech or other similar technologies. Also, a pretty good option. Perhaps we don't want to add the quicktimes to the backup because we already have the original MXF files backed up to tape. You still have 80gb left to backup. Maybe Blue Ray is the answer.
Okay, once that is all worked out, now who is paying for the backup? Who is paying to have it sit on a shelf taking up space? You already have the original files, and the final tape master. You're covered in that regard. Okay, the client is talking about coming back in a few months to revisit this edit and repurpose it. It will be much easier to just restore the project from hard drive or disc backup. But again, is there a price to be paid here?
There are a lot of things that need to be thought through with file based aquisition. As I mentioned earlier, you have to weigh price/performance. For those clients that want to pay fees for having everything archived and stored on a shelf, there is revenue potential there. But storing files on several formats indefinitely for no charge, seems like bad business. Not saying anyone is doing this, but I'm sure it's the type of thing that crosses our minds. We want to be absolutely certain that these files are safe, so we go above and beyond to do so in some situations. But as we can see, this is where the time we saved from having to digitize footage can be quickly lost if we are not smart. I am still researching all of the current archive solutions to see what will work best in my situation and I encourage everyone else to do the same. It's critical that we have the right kind of tools in place. Otherwise, we may be wishing we could go back to the old days. I see the future as bright, but sometimes it's like the wild, wild west and you just have to get all of your facts lined up so that you make smart purchasing decisions.
Re: archiving P2 cards by Kit Lammers on Feb 28, 2008 at 6:32:32 pm
I realize that this thread is old, but I wanted to add what we are planning to do:
We were saving our footage to DVDs but that is not the best way to do it and a cataloging nightmare!
So we have purchased a Quantum LTO-3 Tabletop Drive, we didn't want to have to maintain a large amount of on-line media and I didn't want to worry about it crashing. The tapes hold about 400GB or 800GB compressed and are about $50 each - not too bad.
Now we also purchased the Network Tape Gateway system to add a keyword search and proxy to our archive. I actually have not recieved it yet, but I recommend you take a look at: http://www.pictron.com/products_ntg.html
Since we are a smaller station without a large budget this still cost us around $17K, but will save us money when we are not looking through hundreds of old DVDs and reshooting footage.
Re: archiving P2 cards by Kit Lammers on Jul 17, 2008 at 9:43:17 pm
The Pictron Software did not work out and I was not happy with the performance. So I have decided to go with an Apple Xserv and a Vtrak 12TB Raid with Final Cut Server to manager the media. We will still backup to the LTO3 tape.
