Archiving: Keep a great lossless copy or not?
I will certainly show my ignorance at some point but I have reached a point in my reading where I was wondering if anyone would comment on something I am pondering.... Some of this is about archiving and some is about exporting so I was not sure where to put it....
Why would one archive a big file when they could just archive their project file and output the big file again later?
So here is my situation....
I am using Premiere CS4(I know old but that's what I got)
I am exporting using Adobe Media Encoder CS4
I need to accomplish:
Archive my work
Create a final deliverable for the web(Youtube in particular)
I am getting ready to export my creation and have read a ton about different codecs, etc. on this site and others... I have done some tests as well...
I came across the idea from several sources that proposed this workflow for final export/archiving:
1. Creating a high quality export file (using quicktime or avi container) with (prores, png, DNxHD, or something else I am missing) [these are virtually lossless or lossless]
2. Using some other software(Sorenson Squeeze,etc.) to crunch that high quality file from #1 down for the web.
3. Save the file from #1 for archiving
My question is:
Why would you even save that big file unless you plan on making several copies of it to different formats? The only thing you would be saving would be the time it would take to re-render everything right? In my case I will probably only make one copy for youtube and I need to archive something. That would be it, then on to the next project.
Wouldn't I be better off just making a quality export like #1, crunching it down, and then just saving that crunched down file and my project files?
Why would I want to save such a big file like #1 listed above....?
From my testing, I did a quicktime,png sequence export from Premiere CS4 to> Adobe Media Encoder CS4 to> the final file. My result was a .mov file that was 28.95 GB in size. This is only 5 minutes and the total video will range from about 25-35 minutes. We hope to make a vid a month so archiving is a concern.
I guess I just feel like I am missing something for my particular setup....
It seems many recommend a lossless export from Premiere because perhaps they don't trust its compression capabilities? I plan to test but that is the premise I assume for my workflow.
I am thinking:
1) Make a lossless export of my work (Quicktime,PNG codec or whatever like listed above)
2) Squeeze a youtube vid out using this lossless copy with third party software
3) Delete the huge lossless file(#1)
4) Archive the project files, original footage, and youtube vid
Since I don't need to make several copies of this all the time, wouldn't this be enough to archive?
Do I even need to worry about step #1...
Ask yourself these questions -
Based on your honest answers, consider these costs to archive the work:
Do the numbers balance (i.e.: is the disk or tape more cost effective than recreating the work)?
I recommend that you save everything - from originals through to rendered finals - unless you KNOW that it's a one shot job and you'll never need to revisit the subject (not just the job). You never know when today's cast off clips might become a money and time saving b-roll in tomorrow's project.
CTO - TOLIS Group, Inc.
BRU ... because it's the RESTORE that matters!
Tim - very well said.
Tim made a great point that storage is cheap compared to labor and cost of creation/acquisition.
But I want to address your other point about project files and re-exporting.
It's a bad idea.
You alluded to it right at the beginning: "I am using Premiere CS4(I know old but that's what I got)"
Imagine what you'll be using 10 years from now. Will it be able to open your project without screwing it up? Will the project file corrupt? Will the files all reconnect (that's the biggest problem I've run into even a few months later). If you later take up Creative Cloud and then decide to end your subscription, you will be locked out of your project files. You will want masters that don't require an app to play back.
I've always operated under the idea of the 'tape on a shelf' or 'negative in a vault', which is why I eventually adopted LTO tape. No matter what system you used to edit, if you printed an HDCAM tape, it can be played back in any HDCAM deck. This is a godsend when you realize that your intern accidentally deleted all your files, but you still have a master tape.
The other question is codec, and that's a complicated one. Never save a master as a web-quality codec. You want to save the highest quality that you can. But, I've had feature length ProResHQ files develop errors in playback (glitches or 'hits' in the video), and some even become completely corrupted and won't open. I'm not sure exactly why this happens. I switched to DPX for masters because it is frame based - if one frame corrupts, it doesn't screw up the entire movie. yes, DPX exports take up a LOT of space, and slower systems can't play them back in realtime. But as Tim said, storage is cheap and we're only talking about the final export, not all the source media. For sources, i would just keep the original formats they were shot in, along with any files that were used in the timeline (basically, a media manage).
since film is going the way of the dodo, many production houses and networks require source footage and masters to be archived on LTO and not hard drives.
i hope that helps
Production Workflow Designer / Consultant / Colorist / DIT
Many reasons to keep a high-quality master, versus just the "YouTube" quality. What if you need to re-edit later, fix a mistake, use parts of the clip in another project? Now all you have is the nasty, heavily-compressed YouTube clip to work with, then when you export AGAIN and compress AGAIN it looks that much worse.
As someone said, if it's a one-shot deal, been delivered and the customer signed off on it, then don't keep any copy, that is up to you. No one says you MUST keep a "lossless archival copy", that is up to you. For most of the weddings and events I deliver, once I know the customer is happy with the edit, all I keep is a copy of the DVD they got. If I have some really good clips that I might want to use for a future demo, then I will keep the original cameras clips or a high-quality master of them. Just depends.
You didn't say if you were on the PC or Mac. The huge files you saw were from choosing the wrong codec, basically "uncompressed" as a series of still frame images.
Rather, you will want to use something "visually lossless" for mastering. On Mac, that would be ProRes for instance. On the PC, something like the free Lagarith or UT codecs will provide a good balance between great quality and reasonable file sizes. Or even the "Matrox" codec which is MPEG-2 I-Frame with 4:2:2 color and is smaller yet.
Safe Harbor Computers
[Eric Hansen] "I've had feature length ProResHQ files develop errors in playback (glitches or 'hits' in the video), and some even become completely corrupted and won't open. I'm not sure exactly why this happens. I switched to DPX for masters because it is frame based"
I'm so glad I saw this thread tonight.
Today a client with ancient (3/4") videotapes told me she had been advised to transfer them to
"DV-50" for archiving. My immediate reaction was, why archive to a videotape format which will be more and more difficult to play back? (btw, I'm fairly sure she meant DVCPro 50 when she referred to DV-50.)
I was going to advise transfer to ProRes 422, onto hard drives and LTO tape. But your comment above, about corruption in ProRes files has me concerned. What do you use to create DPX files, and what assurance is there that DPX will be around, 10, 20, 30 years from now?
What digital format is "The Best" for archiving? Is there any consensus on this?
I don't think my problem was with ProRes. In reality, I think it could happen with any codec. Since I wasn't able to figure out the source of the error, just that I experienced it with multiple random files spread out over years. It reminded me of the glitches I would see on old BetaSP tapes.
DV50 is just another codec. It just so happens that it's also a tape-based codec, but I don't think that really matters. I think any playback device that can do ProRes will also be able to do DV/DVCPRO in the future. It was one of the top codecs.
With DPX, the only reason that I prefer it is because any possible data corruption (not tape or disc or drive), will be limited to a single frame. It's pretty easy to rebuild a single missing frame in After Effects or similar.
There's never a guarantee in digital. I wouldn't have guessed 10 years ago that DV would still be playable today, and it still is. I have no reason to believe that it won't be playable 10 years from now, but I wouldn't be surprised if other codecs go away.
The easiest way to create DPX now is with Resolve Lite - import your file and export it as DPX. You can also export as TIFF, which I've seen occasionally for DCP delivery. You will need to keep separate audio files that are synced.
Production Workflow Designer / Consultant / Colorist / DIT
This is what we get for tolerating proprietary formats.
Some project choose to record the final to film for archiving because they're solidly convinced in 30 or whatever years there will be no way to deal with the original digital data at all, or it will be prohibitively expensive, and therefore why bother paying to store all of that digital originally captured material in the meantime? The original material is a lot bigger than the final digital deliverable. And film, well that's comparatively cheap to archive with a freezer. And it's somewhat self-describing in that its encoding isn't totally obscuring the content.
Back in the old plate making days, quite a few artists would break the plates after all members of an edition were made. The only thing that mattered were the edition prints. So another valid tactic would be to render out the "best practical" deliverable file you'd ever need and obliterate the rest. And even that's not as aggressive as destroying the plates, because the digital final such as it is can be identically duplicated, unlike edition prints.
A 2TB consumer-grade disk costs around $99
A 2TB enterprise-grade disk costs around $160
If you have an LTO-5 or LTO-6 tape drive, a 1.5TB LTO-5 tape costs around $30, 2.5TB LTO-6 around $70
Although this is somewhat esoteric information, the rate of unrecoverable reads for the above options decreases as you descend the list. Consumer SATA shouls have less than 1 unrecoverable read error (bad sector) in 12TB read. Enterprise SATA should be less than 1 in 120TB read. Enterprise SAS/FC, should be less than 1 in 1.2PB read. And LTO tape should be less than 1 in 12PB read. So that's in the realm of 1200x the likelihood of losing a sector of data on a consumer SATA hard drive, than on LTO tape, or three orders magnitude greater. And their shelf life is longer also.