FORUMS: list search recent posts

LTO Archive vs Near line storage

COW Forums : Archiving and Back-Up

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Eric Strand
LTO Archive vs Near line storage
on Nov 14, 2018 at 5:00:50 pm
Last Edited By Eric Strand on Nov 14, 2018 at 5:15:10 pm

While googling, I stumbled upon this thread that Bob Zelin posted in 2016 and was wondering if the different companies had a 2018 update on this topic?

https://forums.creativecow.net/docs/forums/post.php?forumid=330&postid=2240...

Thanks!

@ericstrand11


Return to posts index

Bob Zelin
Re: LTO Archive vs Near line storage
on Nov 15, 2018 at 4:22:41 am

an update on what topic. You are not clear.

The wonderful companies that make LTO software today are Imagine Products, YoYotta, and Hedge.
Thunderbolt LTO drives can be purchased from Magstor, and MLogic.

Today, LTO 8 is the "latest thing" - but LTO 8 cannot read an LTO 5 or LTO 6 tape. It will only read LTO 7 and LTO 8.

As for near line storage, QNAP and Synology are the most popular products.

If you have further questions - please ask SPECIFIC questions, and I will answer them.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Eric Strand
Re: LTO Archive vs Near line storage
on Nov 15, 2018 at 3:05:09 pm
Last Edited By Eric Strand on Nov 15, 2018 at 3:10:15 pm

Thanks for your reply Bob; I meant on your exact initial post:

"I cannot help to acknowledge that as the price of inexpensive NAS products from Synology, QNAP, and Netgear
continue to drop in price, the reality that NEAR LINE STORAGE (secondary shared storage with cheaper products) is becoming a viable alternative to LTO tape backup."

What is your and other execs thoughts two years later/the current state of near line as archive?

@ericstrand11


Return to posts index


Bob Zelin
Re: LTO Archive vs Near line storage
on Nov 15, 2018 at 5:18:41 pm

near line is great.
systems from QNAP and Synology are dirt cheap, and drives are super reliable today. When you create a RAID 6 configuration, this allows for 2 drives to fail. So your backup "nearline" solution is RAID 6, and your master system is RAID 6, so you can have FOUR drives fail, and you will still have your data intact. Of course, you should change your defective drive the second that it happens.

So will a RAID array like near line storage last for 30 years ? Of course not. But neither will LTO. I don't care if it's "theoretical" for 30 years. No one will write drivers for products like the Cache-A for example. So what do you do with all of your LTO 4 and LTO 5 (and now LTO 6) tapes. You create the LTO 6 archive. You keep it in a storage vault.
Your place burns down, and today you can only get LTO 8 drives, which will not read an LTO 4, or LTO 5 or LTO 6.
So someone will come back and say "well, that is ridiculous, you can still find LTO 5 and LTO 6 drives out there" -
well what about 10 years from now - or even 5 years from now. Can you find an LTO 4 except on eBay ?

Everything becomes obsolete. Nothing is going to last for 30 years. The appeal of cloud site storage is that "someone else does it for you". So all the big cloud companies are continuously migrating their technology to new technology, as it approaches obsolescence.

This is why near line storage is appealing. You keep your near line system with it's 5 year warrantee drives for 5 - 6 years, and in 5 - 6 years, you buy a NEW SYSTEM (this time with 20 TB drives - because 14 TB drives are out right now for $499 a piece, and in 5 years, I assure you 20 TB drives will be available) - and you migrate your data to the new larger Near line system.

If you say - "well, I don't want to keep buying new stuff every 5 - 6 years - I want it to last for 30 years" - this is my advice to you. Open up a sandwich shop, near video production facilities. Because that meat slicer and refrigerator may in fact last for 30 years. But NOTHING that is "technology" is going to keep going for 30 years, or 20 years, and if by some miracle it lasts for 10 years today - well, it's a Miracle.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Eric Strand
Re: LTO Archive vs Near line storage
on Nov 15, 2018 at 8:22:52 pm

Thanks Bob, along the lines of what I thought you may say. Small production facilities, as always, are continually fighting the budget vs new technology battle.

Also interesting to note that although 20TB drives will be available in the future, the file sizes that cameras generate continues to grow. Although one could look at new Blackmagic Raw with it's lower data rates and say that camera and codec manufacturers will also keep innovating and giving us better quality for similar or smaller file sizes.

Thanks again for your insights.

@ericstrand11


Return to posts index

Jerzy Zbyslaw
Re: LTO Archive vs Near line storage
on Nov 16, 2018 at 11:33:08 am

[Eric Strand] ""I cannot help to acknowledge that as the price of inexpensive NAS products from Synology, QNAP, and Netgear continue to drop in price, the reality that NEAR LINE STORAGE (secondary shared storage with cheaper products) is becoming a viable alternative to LTO tape backup.""

Yes, for the amount of data they store they are perfectly viable, the answer may swing back towards LTO drives, auto loaders and tape libraries if suddenly you need to shoot in 8K and find you may need five to ten times as much storage as before. Realistically though, people will most likely have a mix of both in having a NAS for stuff they are working on and tape for stuff that's current but inactive at the moment.

[Bob Zelin] "So what do you do with all of your LTO 4 and LTO 5 (and now LTO 6) tapes. You create the LTO 6 archive. You keep it in a storage vault. Your place burns down, and today you can only get LTO 8 drives, which will not read an LTO 4, or LTO 5 or LTO 6. So someone will come back and say "well, that is ridiculous, you can still find LTO 5 and LTO 6 drives out there" - well what about 10 years from now - or even 5 years from now. Can you find an LTO 4 except on eBay ?"

Dell still sell two different models of brand new LTO4 drives so that covers you down to LTO2 tapes https://pilot.search.dell.com/lto%20drive , all higher versions (LTO5+) are also available everywhere else so I don't see this issue as being a problem at the moment although it may change in the future as you suggest, but I'm guessing that people will still be building old versions of LTO drives well into the future because if they are still going to be priced at four grand apiece the reply to any query is probably going to be "how many of them do you want to buy?", I don't know who would still be sitting on LTO1 tapes today as they are only 100GB so I presume a used LTO2 or LTO3 off Ebay would suffice to get the data off those onto something a bit more modern, LTO2 was introduced in 2003 so people have had plenty of time to migrate LTO1 data which was introduced in 2000.

[Bob Zelin] "This is why near line storage is appealing. You keep your near line system with it's 5 year warrantee drives for 5 - 6 years, and in 5 - 6 years, you buy a NEW SYSTEM (this time with 20 TB drives - because 14 TB drives are out right now for $499 a piece, and in 5 years, I assure you 20 TB drives will be available) - and you migrate your data to the new larger Near line system."

In theory yes, but the cost of the hard drives on a $ per TB basis since they consolidated to only two main manufacturers is not declining that rapidly, if say a 6TB drive cost $300 and they introduced 8TB and 10TB hard drives they would come in at $400 and $500 respectively, gone are the days when people went crazy buying 1 TB drives and I more wisely would buy two 750GB ones for the exact same price. Hard drive prices have to decline by 50% every 2.0-2.5 years to keep pace with LTO storage density roughly doubling each new generation for pretty much the same price. You can have an 8 bay NAS with 14TB drives or even 20TB drives you mentioned but then if say you need to back up this data a second NAS is going to be expensive whereas I've seen a bare LTO8 drive for sale at just under USD $3K and LTO7 type M8 tapes are just under AUD $100 (USD $72.50) and they store 9 TB uncompressed. And then as I mentioned previously if storage requirements blow out for 4K or 8K then this question becomes even more acute if you read the reduser forums at http://www.reduser.net/forum/forum.php on this topic or even on the Reddit Datahoarder forums https://www.reddit.com/r/DataHoarder/ where LTO technology is starting to be discussed more and more where people have a need to backup their stuff in the 50TB - 100TB+ range.


Return to posts index


Bob Zelin
Re: LTO Archive vs Near line storage
on Nov 16, 2018 at 1:58:37 pm

Jerezy writes -
In theory yes, but the cost of the hard drives on a $ per TB basis since they consolidated to only two main manufacturers is not declining that rapidly, if say a 6TB drive cost $300 and they introduced 8TB and 10TB hard drives they would come in at $400 and $500 respectively, gone are the days when people went crazy buying 1 TB drives and I more wisely would buy two 750GB ones for the exact same price. Hard drive prices have to decline by 50% every 2.0-2.5 years to keep pace with LTO storage density roughly doubling each new generation for pretty much the same price

REPLY - well, in November 2018, a Seagate Ironwolf 7200 RRPM 6TB drive costs $184. And to keep going with your model, a 10 TB costs $299, not $500. For $500 you get the new 14 TB drives, and those prices of course will come down within 6 months.

Nearline also offers the ability of INSTANT ACCESS. No one has to restore a nearline archive. And if you are not using the LTFS tape format, good luck trying to restore any LTO tape (like LTO1, LTO 2, LTO 3) that has been done in a proprietary format, whose manufacturer (like Cache-A) has gone out of business.

One of the downfalls of LTO, which has been greatly simplified by wonderful companies like Hedge, YoYotta and Imagine Products, is that the RESTORE process needs to be easy. The minute that any technology product becomes someone's career (oh - we are experts in LTO archive) - then this technology is useless. In 2018, people want stuff to be easy, and not require an IT professional to assist them with it, or have an IT staff person just to restore an archive.

The observance of Zip drives AIT tape, DLT tape, DAT tape, and early LTO makes all of this very nerve racking.
And the recent release of LTO 8 tape does not make people feel more secure - only because it's not very backward compatible. Will LTO 10 read an LTO 8 tape when it comes out.

But do drives fail - OF COURSE THEY DO. And I see on these very forums, some poor slob with his old Rorke Data array, or Apple Xserve RAID array, or Sonnet Port Multiplier array, or Medea RAID 3 array, saying "I can't read my data anymore"
So drives are no miracle. But they certainly last 5 - 6 years, and if you simply are tired of making continuing investments into our business, then you should NOT be in this business anymore.

I look at guys that have spend a fortune in cameras, only to have new wonderful cameras from companies like Blackmagic come out, that run circles around their old expensive "old gear". I often bring up the $80,000 Media Composers, and $100,000 AVID Symphonies. You know what you can do with these ? Put them in the trash bins.
And this applies to the original $400,000 Davinci Resolve systems, Ampex ADO's, Chryon's, GVG Kaleidoscopes, Quantel everything, and countless other deadly expensive products that have basically put countless people out of business because of their massive investments, and not getting a good ROI on them.

A current LTO 8 system from MLogic is about $6400 and that includes Hedge LTFS software. I love this product, and I love Hedge software. Will this give the purchaser of this product life beyond 2023 ? And the REAL question is -
will Thunderbolt 3 even exist in 2023 ? (the MLogic LTO 8 drive is Thunderbolt 3 interface only). For clarity - 2023 is 5 years from now - about the life of a nearline NAS.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index

Eric Strand
Re: LTO Archive vs Near line storage
on Nov 16, 2018 at 3:41:57 pm

I love Hedge, but am slightly wary of their Canister LTFS product simply because they are a smaller company. Even though LTFS is open source, where will technology and companies be in 5 and then 10 years? What happens if a newer technology comes out meant to replace LTFS?

@ericstrand11


Return to posts index

Bob Zelin
Re: LTO Archive vs Near line storage
on Nov 16, 2018 at 8:57:52 pm

Eric writes -
I love Hedge, but am slightly wary of their Canister LTFS product simply because they are a smaller company. Even though LTFS is open source, where will technology and companies be in 5 and then 10 years? What happens if a newer technology comes out meant to replace LTFS?

reply -
really? You do know that YoYotta is a tiny company. So exactly who do you want to rely on?
Lets pick Tolis Group (who does not support the LTFS format) -
look at this recent post on Creative Cow -
https://forums.creativecow.net/readpost/330/2897

what do you think about that ?

How long was Cache-A in business ? ProMax bought them. Does ProMax support Cache-A products now ?

OH - I know - you can get Archiware P5
http://p5.archiware.com/products/p5-backup

it's about $5000 for the software - and now you buy the LTO -

how about this -
https://www.gblabs.com/products/EasyLTO/

It's about $10,000

how about this -
https://storagedna.com
I understand that their SUPPORT CONTRACT is about $4000 per year (perhaps I am wrong about this) -

so you tell me, Eric - what is the right decision ?

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com


Return to posts index


Eric Strand
Re: LTO Archive vs Near line storage
on Nov 20, 2018 at 3:22:18 am
Last Edited By Eric Strand on Nov 20, 2018 at 3:08:32 pm

I understand your point and fully agree with you; Cache-A is a great example. Additionally I learned about YoYotta from your articles.

There is no right answer. I thought your original post from 2016 was interesting and decided to start a conversation given the downward trends in pricing. We live in a time of amazing, affordable technology, as your articles post NAB often highlight.

@ericstrand11


Return to posts index

Tim Gerhard
Re: LTO Archive vs Near line storage
on Nov 21, 2018 at 6:28:05 pm

I'm the LTO guy so I'll stick to the LTO talking point, but I don't think it should be disk storage vs LTO, you should have both.

LTFS is just the way to go now when using tape. The old tarball way of making 1x file that is proprietary to a single version of software is a thing of the past like Casch-A.

I see Hedge has the canister app now, which I intend on running through the ringer when I find some time. Imagine Products and YoYotta have been in the LTFS game much longer than Hedge and have much more experience. When pondering over the single deck LTFS type support, these are all small companies, but you need to take their location / time zone into account. Imagine Products are in Indiana USA, YoYotta is in the UK, and Hedge is in Amsterdam or something? (Their website has no contact us or about us section, NOR support page).

StorageDNA is probably the largest company for LTFS single deck, with their myltoDNA product, which is built off of Imagine Products myLTO, but it has the "HyperTape" version. It isn't $4000 for the support, that is for the library version I would guess.

I wouldn't worry too much about Thunderbolt going out of style, and you can always just remove the drive from the enclosure later as they're all SAS drives.

I sell Imagine Products and YoYotta with our MagStor line of LTO Thunderbolt & SAS solutions, and both have proven to provide top notch support.

I can't speak for Hedge or Tolis group as I have no experience, but it seems like most of this forum is complaining about Tolis group most of the time, which could just be unwarranted because their ticket wasn't responded to immediately.

Tim Gerhard
MagStor Inc.
614-505-6333
tgerhard@magstor.com


Return to posts index

Neil Sadwelkar
Re: LTO Archive vs Near line storage
on Dec 1, 2018 at 4:18:14 am

[Tim Gerhard] "but you need to take their location / time zone into account. Imagine Products are in Indiana USA, YoYotta is in the UK, and Hedge is in Amsterdam or something?"

This statement is for those currently in the USA. If you happen to be in India like me, then all are far away, with the US being the farthest, with a solid 9.5-12.5 hour time difference making any telephonic support a night long affair for any one side.

Regarding Hedge, I own 10 licenses of Hedge and just bought 3 of Canister. I've found Hedge support to be excellent and usually get a response within a few hours.
Similarly Tolis and Bru. I have three licenses, and I've not had too many issues that needed issues, and for the few that have, I usually get an e-mail response within a day or two.
And I have one license of Yoyotta ID LTFS. For which support from Martin has been excellent. Even though I've contacted him, not with issues with the software, but my lack of understanding how to use it and he's been patient too.

I also bought a license of Imagine Preroll Post some years ago, but ran into a database problem when just starting a large feature rushes backup. A problem that didn't get resolved so I stopped using that software and eventually abandoned it.

I have 4 LTO drives of my own and use two others at client establishments. And I backup to LTO about half a Petabyte annually.

When dealing with LTO, hardware support is sometimes more crucial than software support, as I've found often, hardware issues manifest themselves as software issues. And in this matter, support from Tandberg has been excellent at least here in India. Whereas the other tape hardware manufacturers are absent to non-existent in India.

A huge problem in India are high Customs tariffs and slow import/export procedures. So, anything needing shipping hardware back and forth is a huge no-no. So, if anyone said there were making an LTO drive and were US based, and their support needed sending a drive back and they would send a replacement same day, for us in India it would mean two week turnaround with a possible customs duty added.

So all of this support thing is pretty regional. There really is a huge world out there, and a lot of it is also outside the US.

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index


Tom Goldberg
Re: LTO Archive vs Near line storage
on Nov 22, 2018 at 9:30:39 pm
Last Edited By Tom Goldberg on Nov 23, 2018 at 4:06:27 am

OK, I just have to jump in here and argue with the statements made about Cache-A.

Promax may no longer support Cache-A products, but they can still read and write Cache-A tapes and catalogs with their current products.

And statements about not being able to read Cache-A tapes are just plain false. They are all written either in LTFS or open source tar (at the user's discretion), documentation is available as posted elsewhere on this forum. Nothing was ever written in a proprietary format and no special drivers are needed.

As to the thread topic, while the economics and speed arguments over the value of using LTO may be valid, the fearmongering that you won't be able to buy drives is also patently false. You can still buy every generation LTO drive ever built on ebay - hell, you can even buy floppy disk drives - check yourself!

Tom Goldberg
TGCS
30201 Rainbow Hill Rd.
Evergreen, CO 80439
mailto:tomgoldberg@gmail.com
http://tomgoldberg.net




Return to posts index

Neil Sadwelkar
Re: LTO Archive vs Near line storage
on Nov 23, 2018 at 2:31:21 pm

In my short experience of writing LTO tapes from LTO-4 to the recent LTO-8, one of the enduring formats I've found is Bru-PE.

A client recently had to sell a TV show from the 2009-11 period which was then mastered as ProResHQ HD mov files to hard disk, and DigiBeta tapes as SD for telecast in 2009-11. When this show had to be sold again to an OTT platform as HD, the only files they could use were on LTO-4 tapes written with Bru-PE during that period.

We could open these on a recent LTO-6 drive and even with a newer version of Bru-PE and extract all of about 40 tapes. Hard drives of that period (G-drives and a G-StudioXL) refused to mount consistently enough to extract data off them. We didn't have the catalog info for all the tapes, but reading the content info off the tape wasn't difficult.

Of the same period, I had another client approach me with tapes written using Windows NTBackup. This format has been discontinued by Microsoft and newer versions of Windows from Win 7 onwards have no support for tape based NTBackup backups. I need to figure out setting up a working WinXP machine to help this client get his data back. Even after I do, I'm not sure, if, in the absence of any catalog info which gets saved on the Win system which wrote these NTBackup tapes, air all the data will ever be recovered.

Another client had tapes from earlier, which were written with Symantec Backup (according to them), but the newer version of this software they found, cannot retrieve that data.

So, while LTFS is probably more open and common now, and gain forward may be a better format to backup LTOs to, LTO tapes from about pre-2008 while still in good shape, but the means to extract data from them could be challenging.

-----------------------------------
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India


Return to posts index

Tim Jones
Re: LTO Archive vs Near line storage
on Nov 30, 2018 at 5:16:51 pm

As Tom mentions - just because the vendor is gone, doesn't mean that existing devices cease to function. And, in the case of Cache-A, even if the Cache-A devices fail, the tar and LTFS tapes can be accessed by other systems (unlike Quantum's DLT-A and LTO-4A devices).

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index


Tim Jones
Re: LTO Archive vs Near line storage
on Nov 30, 2018 at 5:13:55 pm

Bob,

That post is a bad example and Barry has been hit by some serious issues with our prior outdated system (from 2004). I don't respond to Support Requests here generally to maintain my position as vendor neutral. TOLIS Group is still alive and well and the support system has been replaced with a modern 64bit server with a modern LAMP stack and the latest version of osTicket. You can open a ticket and get a response at http://support.tolisgroup.com.

Also, regarding technology obsolescence, we have LTO-1 drives from 2001, AIT-2 drives from 2002, DLT-80 drives from 1997, 2GB DAT drives from 1991, and 250GB QIC cartridge drives from 1988 that still function properly and allow our current version of BRU to restore the information that was recorded onto their respective tape formats. Not all technology goes away simply because vendors release new formats. Think about that - a 2GB DAT tape that was recorded in 1992 on an SCO XENIX system is still recoverable on a 2018 Linux system.

The big issue with cloud that is not on-premises will always be bandwidth. When a relatively modest wedding shoot brings in 4-5TB of multiple camera data, there's no way to get that data onto your cloud storage before your next job adds that much more data. Imaging trying to manage that when shooting a larger production.

While nothing that has been mentioned about disk is incorrect, many myths about tape storage are.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

Tim Gerhard
Re: LTO Archive vs Near line storage
on Nov 30, 2018 at 9:09:41 pm

You're lucky if you can read a DAT / DDS tape after so long. The media is expected to last 10 years. Our data recovery department encountered a DAT tape from the 90's and the data was just long gone.

The main problem that I see with LTO TAR systems is that the companies that make the software get swallowed so often. I'm wrong to say proprietary, but when you need an IT pro that knows how to use TAR or data recovery service to help you get your data off your tape, I can't say that many people are very happy about it. Most people don't even know what software was used to write the tape when they come looking for help.

TAR is still much easier to deal with at least, just make sure you have a plan to recover your data assuming a bad scenario of that software becoming unavailable. Old tape hardware is still pretty easy to get, we sell LTO2 drives quite frequently still for support contracts.

Tim Gerhard
MagStor Inc.
614-505-6333
tgerhard@magstor.com


Return to posts index

Tim Jones
Re: LTO Archive vs Near line storage
on Dec 4, 2018 at 6:23:58 am

I would have loved to have had out team be able to examine that tape. Unless the tape itself was aggressively degaussed, the drive should have been able to find the servo tracking information and at least been able to give you the hard format header info. Granted, I have some magic in my drives because I was part of the team responsible for the original design in conjunction with SGI and HP, and I can coax a bit of recovery efforts from them, but I've never seen a DAT tape just "lose" it's tracks and data unless it was horribly mishandled. I still own the first customer unit DAT from our Maynard division (complete with the ADAT digital audio I/O port for SGI in our lab.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!


Return to posts index


Eric Strand
Re: LTO Archive vs Near line storage
on Dec 12, 2018 at 8:50:04 pm

Thanks to the Tims' and Neil for jumping in!

@ericstrand11


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2018 CreativeCOW.net All Rights Reserved
[TOP]