LTO Archive vs. drive based "near line" storage
This forum has always been an LTO tape backup forum.
I cannot help to acknowledge that as the price of inexpensive NAS products from Synology, QNAP, and Netgear
continue to drop in price, the reality that NEAR LINE STORAGE (secondary shared storage with cheaper products) is becoming a viable alternative to LTO tape backup.
Not that drives are safe, and not that anything currently can replace LTO tape for safety (until large SSD's become available at normal prices), but for real world "our RAID blew up, and it's going to take a week to restore from these 24 LTO tapes" - the reality of inexpensive NAS products is becoming more "do - able" for many companies, because you can access this data instantly, and don't have to make excuses to a client that your main storage blew up, and you can't do your session today (or tomorrow) because it's going to take time to recover the LTO tapes.
Just my observation.
Rescue 1, Inc.
Indeed you should have an near line backup of your current projects along with new material (graded, rendered, processed footage).
Using a NAS for this makes sense. But this should be in addition, not as an alternative to LTO backup.
LTFS tape can be part of your workflow from the very start of the project.
There is the raw footage, the majority of which is not being used by the project. Plus all completed projects.
That is where LTFS tape fits in perfectly.
With 4K+, high dynamic range, high frame rate, multi camera capture being commonplace, along with longer takes even a small project can have multi TB of material.
However after editorial the majority of this isn't needed, but still needs to be kept.
A lot of our users have projects with hundred of TB's of footage. Most of this never touches online storage.
With LTO-7 tapes the restore speed is comparable to NAS speeds.
So I would suggest online storage (direct attached or SAN) plus near line NAS backup.
Then archive from the NAS to LTFS tape as archiving direct from "live" online storage is not a good idea.
Here is a suggested workflow.
When footage is created as soon as possible make two copies on LTFS tape.
If there is a lot of material, then at the same time transcode it to a proxy format for editorial.
After editorial restore the original footage from tape (if you didn't keep it online).
Link rather than copy material into your project.
Create new material outside the project. That way you keep the project small in size.
Backup the projects and new material to NAS.
The original footage doesn't need to go to NAS, because you have two tape copies.
Backup projects and new material from the NAS to tape.
On completion remove projects from online. If there is space then keep then on the NAS for a bit longer.
While it's true that recovery from a storage failure needs to occur as fast as possible, in the event of one client's work and meeting their emergency needs, would you really be restoring a complete 60TB or more data, or just the data for that client's project? Hopefully, your normal edit and storage workflow would keep your clients' projects separated on your local storage. In that case, even if you were restoring 10TB of data from LTO-6 tapes, you only need to restore 3 or 4 tapes, not all 24.
While I can't speak to other products, BRU PE and BRU Server will both only require the tapes that contain the specific data you're looking to restore - we don't need all 24 tapes to restore the data written onto 3-4 tapes. That's 8-10 or so hours, not a full week. Even if you were copying the data from a backup disk array to a new working volume, it's still going to take hours to copy the data disk to disk since you would most likely not want to work from your backup copies as live data.
Having disk-based backups for small-event situations (oops, I overwrite the RAW with the color-corrected clip), LTO tape is far more cost-effective than disk unless you are buying consumer storage products (there's a reason that an EVO, GraniteSTOR, or NEXIS cost far more than a Buffalo NAS). Remember, once you've purchased your LTO-6 configuration, each LTO-6 tape costs less than $30 on average. Can you really purchase 2.5TB of enterprise grade disk for $30?
CTO - TOLIS Group, Inc.
BRU ... because it's the RESTORE that matters!
While a project is ongoing, then of course, a disk based storage (or RAID) is inevitable. But some projects remain semi-ongoing over long periods of time. There, a near-line storage is preferable to LTO backup, as project media on drives is easier to bring back to rework on.
About the NAS solutions you've mentioned, my issue with them is that the core storage within a NAS remains at the mercy of the 'server', which is a casing with some kind of a Linux based server built in.
Should that box fail, the data on the drives is inaccessible unless you get another one of the exact same box. So, instead of a 8-bay Thunderbolt QNAP, I prefer a G-Tech 8-bay Thunderbolt RAID connected to a MacPro, or even a Mac mini plugged into a switch over 10GigE.
The RAID connected to a Mac is then shared and the Mac is a 'server'. In case the 'server' fails, the storage itself is accessible over any other Mac or PC.
For projects that I handle I make a backup to LTO of all shoot 'footage' as the shoot progresses. And after the edit is locked, we back up consolidated footage to fewer LTOs. Sometimes just one or two LTO-6 can hold the entire used footage. Bringing that back is just hours away in the event of storage failure. We use Bru-PE for footage backup, and like Tim mentioned, we don't need the entire lot of LTOs to restore just a few files.
FCP Editor, Edit systems consultant
Hi Bob –
Thanks for bringing up this topic – it is really a great discussion to have, considering the sheer amount of content being shot, stored and worked on in today’s production pipelines. There are many practical reasons why LTO is still the most reliable, cost-effective backup/archive solution available to the professional media marketplace. At StorageDNA, we’ve always treated LTO as more than just an archive solution and with the introduction of LTO-7 into the marketplace this year, it is now time to consider it an option as extended nearline storage.
We firmly believe that with LTO-7, which provides nearly 6TB capacity per cartridge and up to 300MB/sec drive speed, you can certainly do much more with LTO than with previous generations. The key to enabling functionality beyond traditional archiving is two-fold. First, you need a system architecture that is capable of leveraging the speeds of LTO-7 and beyond. Second, you need a highly intelligent software that not only drives the hardware to its full potential, but also incorporates unique workflow enhancing capabilities.
We feel that DNAevolution does both. Please check out the latest iteration of our solution at: http://www.storagedna.com/extended-nearline/