Archiving EX footage to XDCAM disks
Obviously, one of the problems with using SxS cards to record to is, where do you put your footage for long-term storage? In our shop, we also use Sony 355 cameras that record onto the XDCAM-HD disks. Is anyone archiving their EX footage initially recorded onto SxS cards onto the XDCAM-HD disks? If so, what is your work flow for this?
With firmware upgrade to expand the data capacity of the XDCAM recorder, the EX can be saved as a data file. Otherwise the limit is 500MB I believe.
Of course there's always 25GB and 50GB Blu-ray discs but obviously that's not quite as secure as XDCAM disc.
We're in the process of starting just that. We have a PDW-U1 on the iMac used to organize and catalog our footage. We started with DVD while awaiting the firmware upgrade but now will be using the unlocked data capabilities of the XDCAM discs as a (hopefully) more durable backup medium. We keep our stuff online so this will just be for archival backup.
[Craig Seeman] "there's always 25GB and 50GB Blu-ray discs but obviously that's not quite as secure as XDCAM disc"
Not as secure, true, but much cheaper! (By half)
We archive to BluRAY 50 gig (double sided) DVDs.
You are referring to camera original files (media) right? What are you archiving finished, edited productions to? In the past I rented HDCAM decks and archived to tape (expensive and stupid) and also mastered a downscaled version to DVCAM tape. Can we record back to an EX via SDI a finished master? I guess people are archiving to drives now but how reliable are they and will they play on demand in ten years? XDCAM discs seem a good idea but at a cost.
[Michael Slowe] "expensive and stupid"
One man's ceiling is another man's floor.
I think you're confusing "archiving" and "Mastering". Most people refer to "archiving" as the process of
storing the "ORIGINAL CAMERA MEDIA". In the old days (tape based) you simply boxed up the camera original tapes and sent them to storage. With tapeless, file based media, it's a whole new world of options. I work in broadcast, and when the networks want to take delivery of ALL original production elements, you can't just hand them a firewire drive. They want something thats a bit more stable. Blu Ray data discs fit that bill. XDCAM discs which with their built in "caddy" are quite robust, and work well also, but they're alot more expensive per gig of storage. When you're talking about five thousand hours of original media, the price diffrential adds up quickly, so we use BLURAY DVDs.
As for the final edited masters... the de facto standard here in Hollywood has been Sony's HDCAMsr for most projects. Yes it's expensive, but that's the cost of doing business here. HDCAM is a slightly cheaper format, and still a good choice. Bottom line, most final projects are printed to tape and saved that way.
On the subject of archiving and XDCAM EX I think there a few points worth mentioning.
Archive itself often means different things to different people. For example:
Do you need the camera masters or master of that local cable spot or corporate video to last 10 years or 100? Often time's it may the former is sufficient. On the other hand you may hope that the documentary or feature films you've done may last beyond your lifetime. Ultimately most sources really only have a limited and declining utility and value over time.
Media includes the players. This is a BIG and often overlooked one for me. Having done 2" and reel to reel 1/2" video restoration, the difficulty in finding a player just a decade or two later may make the media difficult to use. Things I've burned on good quality CD media 15 years ago can still be read in any modern computer. Data DVD readers exist in nearly all modern computers as well.
What about Blu-ray and XDCAM? While the XDCAM disc may have a greater chance of survival I can't say the same for the players. 20 years from now they may be as scarce as 2" or 1" machines. 3/4" players are certainly not commonly found. Overtime format specific players are discarded. Manufacturing stops. Parts for servicing become scarce.
What will you be able to play your medium on in 10 - 20 years? It's hard to say with Blu-ray and XDCAM. I believe Blu-ray readers will be more common than XDCAM disc readers but I'm not sure. Blu-ray readers/burners are still rare compared to DVD as built in to off the shelf computers. Certainly they're far more common as 3rd party purchases for those venturing into data backup or video burning then even the PDW-U1. I don't think Blu-ray will reach the ubiquity of DVD though.
Solid State. My own guess is that something like SDHC (solid state) will eventually be the medium for both recording and archival although I think we have a few years of improvements to go before that hits mark though. It's certainly noteworthy than many desktops come with built in card readers. Even newer Mac laptops have SD card readers built in. Multi format card readers are abundant and inexpensive. Proprietary slots (PCMCIA which P2 uses) are becoming scarce and even Apple has reduced the laptop models have have Express slots (used by SxS). I think "common solid state" will be the future.
In short, the the media last 50-100 years (or even 20) it doesn't have much value if you can't easily find a player. I think a good "archival" quality Blu-ray disc carefully handled since it doesn't have the case that XDCAM disk has, will be easier to retrieve than XDCAM Disc.
I've been working in video for about 30 years and my subjective opinion is "informed" by my experience with a wide range of formats and "proprietary" no matter how long the media archival has been a serious drawback.
It is with some irony that the 2" preservation project I worked on was re-archived to D2 rather than BetaSP. Guess which decks are easier to find now.
[Craig Seeman] "Guess which decks are easier to find now"
That would be Beta SP, Craig! D2 has mostly disappeared.
And yes, I have several 3/4" machines standing by in our machine room "just in case". Occasionally, they do get used.
Long term, I'd favor XDCAM discs over BLURAY. I agree that the future of consumer Blu ray DVD may be in doubt, but the professional XDCAM format is a different story. I see increasing adoption of that format industry wide, across many different genres of production; news, sports, reality, etc.
A PDW-U1 is such a simple machine (few moving parts) that it's quite possible to keep one in good working order for a long time. If consumer Bluray doesn't take off, then the drives to play it will become quite rare. In a professional environment, having a PDW-U1 around is a no brainer.
Mark, I think you miss my point, or maybe just disagree.
A facility may certainly keep decks and other playback devices alive for some time. Accessibility dwindles and it dwindles rapidly. I can only "prove" my point based on past history. Nearly every major and even widely used format became scarce inside of a decade or so. You may keep your PDW-U1 going but in 15 years they will be scarce. I know way too many graphic artist who have their work or Syquest or Jazz or Zip disks. Sure you can go on a hunt and find one around but I wouldn't call that secure archival.
Beta, BetaSP, DigiBeta, has had an unusually long lifespan and that seems to be the lone reliable tape format to do that. The amount of NEW material shot on Bata formats is dwindling and so well the decks and their parts. It's inevitable as summer becomes fall.
CD seems to one of the other rare media that has endured and I suspect DVD readers may be around for several more years too.
I am astonished how fast things change even in just 4 or 5 years now. Not only does media change faster but it continues to accelerate. We are headed to a time when codecs may endure longer than media types. I do think solid state (not XDCAM disc) will prevail and I do think solid state readers (much like DVD readers can also read CD) will be either backwards compatible or will be multi format.
[Craig Seeman] "or maybe just disagree."
I agree to a point. I'm saying the PDW-U1 is cheap enough and durable enough to keep with your archive. If, as you point out, the playback device is as crucial as the medium, then the PDW-U1 is a good candidate for a long term option. It's independent of the computer, USB connected, and has very few moving parts to maintain.
[Mark Raudonis] "It's independent of the computer, USB connected, and has very few moving parts to maintain. "
One would hope it has a long life expectancy but even things with few moving parts have a way of dying over time. Also you can't hand the XDCAM discs off to a facility that doesn't have access to a PDW-U1 or other XDCAM disc reader. You can hand a Blu-ray disc and they can get a Blu-ray disc reader for under $300. I honestly can't determine the life on those either though.
On example some are facing is that if you buy a new Mac with Snow Leopard (you have no choice now) you will find the PDW-U1 kext (extension) is broken. That's a perfect example of the problem with depending on proprietary technology.
In answer to an earlier comment following my earlier post, I am concerned with archiving finished productions rather than camera media. It's really the same question though - tape, disc (moving), drive (moving) or static card. To archive back to, say, an S X S card in the camera (possible through SDI?) would be an expensive option at current card prices but might be the safest option, what do others think about this?
Basically your options are Blu-ray or XDCAM disc (as data). The above discussion should give you pause for thought as both have long term pitfalls. If the master is small there's always DL-DVD.
There's LTO which we didn't bring up but that has its on benefits and detractions (as long as there are readers, you're ok).
We didn't discus the issue of codec. You can go back to XDCAM MP4 quite easily in FCP. Some would use EX .MOV others would go to Apple ProRes. MP4 can go back to MOV or MXF or used as MP4 in the future so it's sort of the "universal." At this point Apple makes the ProRes decoder freely available crossplatform so would not be needed but QuickTime would be. There are third party plugins (Calibrated Software) for EX .MOV but if Sony/Apple's EX FCP codec and FCP itself go away that can become a challenge to resurrect.
Personally I think the codecs will endure. It's the media and the players that will be a problem as time marches on.
Solid state isn't there yet. I've heard SDHC manufactures say data retention is at around 10 years. That' will change but right now any of the optical discs mentioned above (if of good archival quality) would outlast that.
[Michael Slowe] "I am concerned with archiving finished productions rather than camera media."
Go to video tape. Specifically: HDcam or HDCAM sr. This will give you the widest range of professional options for future use. Sony has a very good record of making future formats play nice with old legacy formats. For example, the Digibeta decks that we have can play back just about every form of Beta ever made. Our HDCAMsr decks can playback the lower quality HDCAM format. For masters, I say you want to be on tape.
It took me a year before i bought an EX3 as i was mulling over the archive conundrum. I looked at all the archive options and found i would be spending more time dumping footage onto discs than i really wanted with most options. Finally i decided on LTO4 which so far seems to be very good value with an easy workflow. With LTO4 the tapes are 800GB and can be set up to automatically backup my entire edit or i can just drag and drop.
1 x 50gb xdcam disc = £45
1x 800gb LTO4 tape = £30
I would have to agree with Craig on this one. Devices come and go very rapidly. The facility that I'm currently working at recognized this and bought into the Sony Petasite, which is a hybrid based tape storage solution with hard disk on the front end to expedite access. The thought behind this being lesser proprietary file based archival device is preferred.
Some will say that the this is a potentially obsolete device as well, but here's the catch...
When the time comes to find a new device, it becomes a matter of data migration, which is simpler for a single point access device. In other words, we won't have to hire someone to endlessly put a huge stack of BlueRay disc into a player to migrate our archives.
I think this discussion is fragmenting into two different subjects.
In house archival and deliverables archival (I'm not talking about finished masters here, JUST raw footage and perhaps the edit project).
In this respect, there is little practical choice for deliverable archives these days but a blueray solution (whether common Blueray DVD or proprietary Sony Blueray). Either of these usually represent short term requirements, usually to appease a hiring entity requirement that all media from a project be provided at the end of the job. In this case, I think the client drives that decision (what are THEY asking for?) and it is up to us, as producers, to provide this. Matters little what the best solution might be, only what is required to fulfill the contract.
Of course, you COULD go to tape with all of your raw footage, but who wants to do that?
8317 Front Beach Rd, Suite 23
Panama City Beach, Fl
phone 850-234-2773 ext. 128
skype username - vidmarv
Good arguments Marvin.
Delineating between in house and deliverable archive is important. I'd venture to say that in the longer term (you mentioned shorter term) there is less of a difference.
In the case of the 2" preservation project, the original entity once had many 2" machines and eventually had NONE nearly 20 years later so the process of finding a facility with a working 2" machine was expensive I'm sure. They wen't with D2 as the format to archive to so I must imagine that process must have happened again.
Mark made mention of the PDW-U1 having few moving parts. Devices can still fail at a point where finding new parts or a replacement device difficult. I've seen this with Syquest, Jazz, Zip drives although none of those are really like an XDCAM device. I have seen CD and DVD readers whose lasers age and fail over time. The ability to focus declines. I am concerned that a laser based XDCAM disc reader may eventually suffer from declining or failed lasers as well. Certainly there's the risk that the appropriate extension/drivers may not be maintained as Operating Systems update. Look at Sony's handling (lack of) for the .kext and Snow Leopard on Mac.
I can certainly envision a day where one may be faced with failing laser, lack of driver for recent operating systems, becoming a major road block. This is why I again reiterate we must look at both media and player availability into the future when we think of long term archival.
Marvin brings up the important point of the ease of moving the files to a newer media as part of the process.
Lucian, what is (or would be) your workflow to "reconstitute" an offline project?
One thing optical discs afford is the "random access" and fast copy times the LTO doesn't have.
On the other hand LTO has very low cost of use for large amounts of data and its apparently a format that will remain in use (or at least backwards compatible) as advancements are made.
This has been a good discussion, honestly, a LOT more than I expected. In our particular shop, we've already committed to the large-form XDCAM-HD format. We have the cameras and decks that handle those nifty plastic-case enclosed XDCAM disks. As I'm typing this, I have a Mac Book Pro sitting in front of me. It has a SxS card reader on the left side, and I have a PDW-U1 hooked up to the USB port on the right side. It just seems like it would be SO EASY to plug in a SxS card on one side, and transfer those shots over to a disk in the U1. Then I can put the disk on a shelf, to access whenever I need those shots.
Craig, you did bring up a scary thought which had previously escaped me....what if software support and development of the PDW-U1 stops, but the computer OS continues to the point that the U1 will no longer "talk" to the computer? Crap....I haven't even thought of that....a drive that still functions perfectly, but at some point, the computer and drive may not communicate with each other.
I guess it's just a gamble. While Blu-Ray presently looks promising for long term storage, there really is no guaranty that Blu-Ray will out last XDCAM, or vice-versa.
[Tim Allison] "Craig, you did bring up a scary thought which had previously escaped me....what if software support and development of the PDW-U1 stops, but the computer OS continues to the point that the U1 will no longer "talk" to the computer? Crap....I haven't even thought of that....a drive that still functions perfectly, but at some point, the computer and drive may not communicate with each other.
It seems Snow Leopard broke PDW-U1 compatibility. All new Macs are Snow Leopard. When I upgraded my MacPro to Snow Leopard the OS reported that the PDW-U1 extension is not compatible. Sony is apparently MONTHS from fixing this. I certainly can NOT TRUST Sony to diligently maintain OS compatibility. Yes it is quite possible that there may come a time a perfectly good PDW-U1 will not function under a current OS and you may not be able to install the old OS with old components/extensions/drivers. Sony's handling of SxS driver, the PDW-U1 extension, the XDCAM Transfer software does NOT give me "consumer" or "professional" confidence in Sony's long term support. Sure it's my very subjective opinion but the "breaking" of the above are quite real. Only the SxS driver has been fixed to date.
On the other hand it would be fairly easy to hook up an external Blu-ray burner/reader with USB, Firewire, eSata and get access to the data on the disc. There are several manufacturers that make such devices and if one drops from the market I can look to another manufacturer.
I am not a fan of proprietary devices especially when the come from a single manufacturer. Granted 2", 1", D2 had support from the "major players" and all those machines are not easy to come by in working order. Syquest and Iomega had drives/readers that were proprietary but quite popular. Many computers had Zip drives built in. So even wide use doesn't mean continued manufacturing support.
It's just a hunch but I suspect solid state (SD and CF) will be as standard and backward compatible as CD and DVD. Readers will certainly advance (Blu-ray can read and burn CD and DVD) but there is interest from the manufacturers in maintaining backward compatibility and "instant" support as OSs advance. That's why I think solid state will be the future.
Again subjective but I am NOT yet confident in the long term support for XDCAM disc. I don't think Beta is quite direct comparison in part because Beta tape formats endured and dominated for a long time yet I think I do not think XDCAM disc will (certainly not 10 years for example).
I have mixed feelings about Blu-ray since it is not "built in" to computers like DVD is but there are many manufacturers making burner/readers though.
Sorry if any if this is redundant but I keep clarifying my thoughts as I go along.
[Craig Seeman] "Again subjective but I am NOT yet confident in the long term support for XDCAM disc."
Geez Craig, I think you're a "glass is half empty" kind of guy!
Nothing is for certain in this life. That is for certain.
We can argue semantics, but I'd say that Xdcam is a good bet for the long term. Sony has proved throughout my career to consistently have the technology and corporate will to create formats that
[Mark Raudonis] "Geez Craig, I think you're a "glass is half empty" kind of guy! "
It depends on who previously drank out of the glass and what happened to them. I was senior engineer at a facility that bought close to 200 BetaSX machines. Not my decision of course. Zip drives were once so popular that they were standard in many computers, even Macs.
Beta, BetaSP, DigiBeta has had a very long life. I do not see any evidence as the codec and media fields broaden immensely, that XDCAM will have the same longevity as Beta. I see MPEG-2 (which is one common thing on XDCAM discs) living a long time even after it's supplanted by other codecs. The ability to read old codecs will endure IMHO. Media is changing fast and solid state will become the norm and I can see Sony toying with this as well. When it comes to media I think ubiquity beats priority over time especially when the ubiquity means easier storage and retrieval. It'll take 3-5 years (at most IMHO).
To put it another way, codecs will change but older widely used codecs will be retrievable for a very long time. Media will change but unless the media is/was ubiquitous, readers will become scarce due to manufacturing and hardware maintenance issues. When I mean ubiquitous I mean massively so and if it's "consumer" as well as "professional" the better, CD, DVD for example. SDHC is heading in that direction as well.
All this makes me feel that the very best archival medium is the one that endured for 100 years - FILM.
Film didn't endure too well either. Many films decayed and are gone forever. Others cost a fortune to be painstakingly restored. Of course in almost all these cases we have only the masters (the Final Cut) and none of the source material, the "out takes" as it were.
As a Marx Brothers fan this breaks my heart
The Marx Brothers
Humor Risk (1921)
The House That Shadows Built
which exist but is in poor condition
Also they did a radio series (not film obviously)
Flywheel, Shyster and Flywheel in which all the recordings were lost but the scripts found in the 1980s
Hate to think of the workflow from BPAV to film for all my raw footage.
8317 Front Beach Rd, Suite 23
Panama City Beach, Fl
phone 850-234-2773 ext. 128
skype username - vidmarv
No Marvin, I referred to acquisition in film, I take your point about archiving digital material. Craig, not sure that properly cared for film hasn't lasted. Obviously the old flammable nitrate stock had to be rescued and transferred and some of the early colour neg suffered (particularly 'Seven Brides For Seven Brothers') but generally stuff seems OK and it can all still be played and will be for many, many years yet. Nothing very mysterious about the playing - shine a light through it and keep it moving!
Filem decays in many ways. Even newer films. That's why you see restoration of films even from the 1980s. It's part of the chemical process (chemicals used at the time).
The "modern" archival is that data itself can be preserved even though media has a "life span" The idea is that you should be able to copy 10 year old data or 100 year old data (depends on the media used at the time) and the data remains true. It's not restoration but simply moving to a "more modern" (reliable, longer lasting) media. Even data can have issues and that's why things such as error correction are important. Whether LTO, Optical Disc, Solid State, there's a chemical (and sometimes physical depending...) process of decay. There's no "forever" yet that I know of and even longevity testing theories and measurements can be hotly debated.
The reason why I think Solid State will be the next winner (and who knows about some yet unknown successor) is not only what's being worked on around the media (see WORM mentioned previously) but that I personally believe PLAYERS will be around with backwards compatibility for a LONG TIME. That's because such devices are cost effective (inexpensive) to keep manufacturing.
Tape players are toughest to keep going given parts and age and decline as new players types replaced older ones. Optical Disk (and I mean CD and DVD, not XDCAM) are wide used in "consumer" as well as professional gear and even today's Blu-ray player and handle DVDs and CDs from the past. I do see Blu-ray as the beginning of the decline because, while third party devices are common, they are not generally built into computers as data storage/retrieval devices. And they are a "mechanism" device which can have failure over time (lasers age and fail).
With solid state certainly contacts can fail and chemicals age too but I think they have a potential for much longer life especially once "archived" they frequency of use drops and aging can be slowed. retrieval devices are wide spread and computer makers often build them in to computers and I think that's on the upswing as it seems every consumer recording device seems to have SD and now SDHC. It takes almost nothing to add support for the oodles of other types like CF or microSD or what have you. If 50 years from now, it'll cost pennies to add old formats to what ever new fangled Solid State card format of the day, then manufactures will probably do it. In other words that reader will handle the "GoogolplexSplotchHD" card they'll probably throw in an SDHC slot in it's 1000 card format reader.
Absolutely right Craig, I was being a bit provocative referring to film, it's just that I have nostalgic feelings for it being very old!
Solid state is now with us and will be refined again and again and your suggestion that there will be ways of accessing data from these cards with computer software of various sorts for many years to come seems sensible.
Tim et. al.,
I am a user of BOTH XDCam HD (optical) and XDCam EX (SxS flash card). I must say that the XDCam optical format is extremely robust. I've found myself wearing more "cargo" pants these days, because it's easy to carry a spare disc when I work in the field. I've accidentally WASHED a disc or two and didn't lose a single sector of data! You can't say that with tape!
I live in Arizona, and it can reach temperatures of well over 150 degrees in vehicles. The discs have never had an issue with heat either. They take a licking and keep on ticking, to borrow a phrase.
In 2006, I was able to receive a pre-release version of the EX1, original model camera. I began using it frequently as a second camera for our XDCam HD (optical) clients. The question became "how do we provide the media to our clients?" The answer became clear, when some of our clients DEMANDED that all of the files be identical. We started using Clip Browser and the Main Concept codec package to transcode all of our files from MP4 to MXF. I would then use an F70 deck or a PDW-U1 drive, the latter being faster, to copy the files onto an optical disc. Further, we backed up the source footage onto external HDD's. This in reality created FOUR copies of the media DL DVD, 2x HDD, and optical disc. (We've had a few issues throughout the years with HDD's, so I didn't particularly care for that method. Thus, we backed up to TWO HDD's and DL DVD. It wasn't pretty, but it worked.)
In the last 24-18 months, we've began to use the Convergent Design nanoFlash as a secondary recording for ALL of our shoots, with EVERY camera we shoot. Our F800 has a nanoFlash, so does our F350 and EX1. With the advent of the "User Data" folder, we began storing ORIGINAL files in that space from both EX cameras and the nF. This has worked out extremely well. Typically, we STILL convert the files for our clients into the MXF wrapper, and write those to optical disc too... but we will keep the "originals" on optical disc for archive purposes. In effect, this creates MULTIPLE COPIES of the source material, albeit in different states, both file type and frame size; Optical disc transcodes, Optical disc originals, and HDD (typically at client request).
No one has a crystal ball... therefore it is hard to "estimate" what will and will not be around years from today. If one really needs to make a 3/4" or MII dub, or utilize source footage there ARE decks out there to do it.
As previously stated in this thread... films are being restored from the 80's. Remember, 3/4" and beta became prominent in that time frame. I don't know if ANY format is fool proof for the extreme long haul. All one can do is protect themselves for the daily work pitfalls. Hopefully that will carry you through for the next 10 years or so. I'm four years into all of the XD formats and going very strong. I haven't lost a single clip of either type since we started in 2006. I had tapes that broke, stretched, etc. in shorter time frames.
I'm convinced that the XDCam HD and EX systems and formats are robust and here for a while. Sony is committed to both formats. These have proven to be more stable, and secure with a more rapid turn around time than any format we've used in the past.
As for tape, I don't think most XDCam HD or XDCam EX users or clients are WILLING to spend the necessary funds to PURCHASE an HDCam SR deck. Further, the media is still quite expensive. These are some of the reasons that Sony created the optical and flash workflows.
The optical media prices will drop, as the discs are becoming more widely accepted for other uses. Many "other" industries/users have become familiar with the relatively inexpensive PDW-U1 drive and are utilizing it for non-traditional uses. These include P2 users, Red users, nanoFlash users, still photographers, and medical / legal offices. Due to the rigidity of the "cartridge" in which the disc is housed, many industries have began a transition into using XDCam optical discs for their long-term data storage needs. The footprint is small and they are reliable.
I suggest you examine your own personal needs, listen to the experiences of others and decide what works within your own facility.
Picture This Productions
Sony ICE Team
Adobe Premiere Pro CS3 HD/Matrox