FORUMS: list search recent posts

How to best archive decades of video, audio?

COW Forums : Archiving and Back-Up

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Neil Orman
How to best archive decades of video, audio?
on Jul 1, 2015 at 4:06:34 pm

I recently started working for a 30-year old think tank that wants me to archive decades of Institute events (video and audio) recorded on a wide range of antiquated media. What's the best method of archiving that media? This is something we probably won't access often (although we may from time to time) but mainly wanted it preserved.
Thank you very much for any feedback!
Neil


Return to posts index

Tim Jones
Re: How to best archive decades of video, audio?
on Jul 1, 2015 at 4:56:22 pm

Of course, my response is going to be BRU and LTO tape :).

Seriously though, LTO technology is fast, big, stable, and very cost effective.

With a capacity of 2.5TB per tape, your cost is around 2.4 cents per GB (< $40 per tape for certified media). But the big savings comes in the maintenance of the stored data over time. Vaulting LTO tapes requires simple, basic environmental support (if a human can be comfortable in the environment, so can an LTO tape). They also require no power, so the result is the "greenest" storage medium around, even topping paper since humidity is not really an issue with LTO tape as it is with paper documents.

Your next decision is the format that you intend to use for the actual data. There are many different formats - including MP2, MP4, JPEG, MXF, ProRes, AVCHD, combinations of these, and on and on... You should pick one that best suits your long term plans. In whatever format you choose, always archive the highest resolution for the media involved. You can always transcode down, but the other direction is not so successful.

Marketing follows :) -

Finally, the way that you record the data onto the tape is also very important. There are a few choices out there, but our BRU format has been proven for over 30 years in business, financial, medical, research, education, and media and entertainment. Every BRU archive is completely verified down to 2K blocks with no special efforts by the user and no external sidecar hash files that can disappear leaving you with no method for verification. Because the checksum for your data is written directly into the data stream, you can also re-verify and audit your archives at any time - in 2 weeks, 6 months, or 20 years. You don't need to know about the original filesystem or even have access to the original computer type or operating system.

Which ever solution you choose, always ask that question - "How do I verify and audit the data on the tapes in 20 years?" With BRU, it's easy.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

Neil Orman
Re: How to best archive decades of video, audio?
on Jul 1, 2015 at 5:28:16 pm

This is extremely helpful, much appreciated Tim! If we went with LTO tape, which sounds good, or whatever format really, what's the best approach in terms of how the media is physically transferred? We have media on a huge range of old formats like BETA and even older video formats, as well as audio on old formats too. Are there firms that will do this for you? Do you or anyone else have any advice on the best approach there?


Return to posts index


Tim Jones
Re: How to best archive decades of video, audio?
on Jul 1, 2015 at 8:23:10 pm

For the transfer, simple tools that connect your standard line outs from the analog source systems to a capture system are pretty widely available. For example, using an Aja IO Express will give you a very good, all in one box for various analog sources and allow you to send the signal to your PC or Mac system.

https://www.aja.com/en/products/io-express

There are other options as well, but I've actually use Aja's equipment with great success.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

Neil Orman
Re: How to best archive decades of video, audio?
on Jul 1, 2015 at 9:44:20 pm

Thank you Tim. I don't think we even have some of the old analog systems corresponding to some of this media. In that case what's the best option, in terms of trying to do it ourselves versus outsourcing this? And if we outsourced do you or anyone have any guidance on the best approach to finding someone?
Much appreciated,
Neil


Return to posts index

Bob Cole
Re: How to best archive decades of video, audio?
on Jul 9, 2015 at 7:27:10 pm

re service company for the digitizing: In the Washington, DC area, Video & Film Solutions is probably a good bet. They do transfers for the National Archives, so they have a lot of equipment and experience.

You may also discover that old tape decks are readily and inexpensively available, as long as you have an I/O solution such as the AJA device Tim specified.

The other part of archiving is logging, so that after the immense job of digitizing, you can actually access the relevant material. Ideally you'd obtain text descriptions and thumbnail images of each shot. As the librarians say, "a misfiled book is a lost book." By its nature, motion imagery is much more difficult to catalog, access and scan than are stills or words.

This forum is a wealth of information on all aspects of the task.

Good luck!

Bob C


Return to posts index


Neil Orman
Re: How to best archive decades of video, audio?
on Jul 9, 2015 at 9:46:43 pm

Thanks so much, Bob, for the recommendation of Video & Film Solutions, as well as the great guidance here.


Return to posts index

Neil Sadwelkar
Re: How to best archive decades of video, audio?
on Jul 17, 2015 at 4:52:57 pm

A word on the tape size and capacity.

If you're using Bru-PE (and this probably applies to other software too), don't budget for a capacity of 2.5 TB per LTO-6 tape. Depending on the kind of media you're backing up, you'll get approximately 2.2 TB of hard disk capacity per tape with Bru-PE.

This, coupled with the fact that you may not be able to split some folders or files, (so some tapes may need to be closed at 2 TB or less), budget for 2 TB per tape to allow for this 'wastage'.

Certain type of files means if you are backing up a large number of files (like .ari files from an Alexa ArriRaw Shoot) or DPX files from a DI backup, there will be more 'wastage' compared to a small number of large files like for a Red R3D shoot or Sony F65 Raw shoot.

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index

Tim Jones
Re: How to best archive decades of video, audio?
on Jul 17, 2015 at 5:28:12 pm

[Neil Sadwelkar] "This, coupled with the fact that you may not be able to split some folders or files, (so some tapes may need to be closed at 2 TB or less), budget for 2 TB per tape to allow for this 'wastage'.
"


This brings up a question that I've never been able to get a reasonable answer for:

Why are folks so worried about splitting files over tape boundaries?

The BRU format handles this automatically and has successfully for 30+ years now. I understand that other formats like some tar versions and LTFS don't offer this feature, but that's not a reason to have to manage your backup sessions to that level of minutiae. With BRU solutions, you just toss the 48TB of data over and we'll deal with the heavy lifting and prompt as new tapes are needed (or automatically change them if you have a library).

I really would appreciate input on this to better understand the concerns involved with splitting files across tapes.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index


Neil Sadwelkar
Re: How to best archive decades of video, audio?
on Jul 18, 2015 at 3:25:02 am
Last Edited By Neil Sadwelkar on Jul 18, 2015 at 3:51:40 am

Tim,

If someone gave me 48 TB to backup, on one contiguous drive, I would definitely do the spanning thing instead of micro-managing the tape backup.

How we get data is one of two ways.
During shooting each shoot day creates, 400 GB- 1.5 TB of data. This is sent to us on drives of 3 or 4 TB either daily or couple of times a week. We backup data and send back the drive and the tape to be kept at different locations. So, if the data is less than one tape's worth, we either don't start backup unless we have at least 2 TB, or just backup anyway depending on schedule. So, data is received in about 2 TB increments.

Later, if at all the production has any issue with the hard drives, we are called upon to restore specific day's data. For this, there has to be some way for the human at the other end to pull out the specified tape to send back. And that's labelled so its easy. A spanned data set would mean we have to ship all tapes in that set, right?

Second, sometime we need to backup an entire film's data of, say 52 TB sent all at once. This 52 TB is spread over, say, 6 RAIDs of 10 TB each, or often 19 drives of 3 TB each. So, while its possible to span each RAID of 10 TB, spanning drives of 3 TB into 'mini-sets' of 2 tapes is also wasteful. The drives themselves have anything like 2.3 to 2.8 TB data on them, usually stored day-wise.

So, backing up day-wise and letting space go 'in the end' of each tape is the easiest solution for labelling, storage and restore.

We often need to restore specific shots form these tapes, or specific sounds. These are specified to us by file name and date of shoot. So tracking back the tape is easy. Particularly since the tape may be stored at a faraway location.

I know a lot of this can be fixed by using barcodes instead of human readable labels, but most of the LTO storage locations are private offices or homes and don't necessarily have that facility.

But one question. Supposing I did backup 48 TB into one set of, say 21 tapes. And send off the tapes and drives.

Later, if someone asks me to restore a specific file or folder, can I restore with only the concerned tape/s? Or do I need the entire set of 21? This is significant because backup tapes are often shipped out to faraway locations, so shipping back would be required.

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index

Tim Jones
Re: How to best archive decades of video, audio?
on Jul 19, 2015 at 6:28:36 pm

[Neil Sadwelkar] "So, backing up day-wise and letting space go 'in the end' of each tape is the easiest solution for labelling, storage and restore.
"


This, I can totally understand and agree that it's a great workflow.

[Neil Sadwelkar] "But one question. Supposing I did backup 48 TB into one set of, say 21 tapes. And send off the tapes and drives.

Later, if someone asks me to restore a specific file or folder, can I restore with only the concerned tape/s? Or do I need the entire set of 21? This is significant because backup tapes are often shipped out to faraway locations, so shipping back would be required.
"


We're getting a bit specific to BRU, but in our case, absolutely. If what you're needing to restore is on tape 3 of a 21 tape set, you ONLY need tape 3 with a BRU archive set. In fact, in BRU PE you can select what you want to restore in total and we'll tell you which exact tapes you'll need to restore the selected data so that you know ahead of time what to pull back in.



Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]