FORUMS: list search recent posts

LTO Library Automation vs Standalone Drive?

COW Forums : Archiving and Back-Up

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Dan Olsen
LTO Library Automation vs Standalone Drive?
on Apr 14, 2017 at 4:58:20 pm

Hoping for a quick poll and some feedback--how many of you are using libraries with automation vs standalone LTO drives?

I used standalone LTO-5 drives on BRU PE for years at our facility and issues were extremely rare. Job sizes started to grow as we started adopting 6K etc and I found myself stopping in regularly on weekends to swap tapes just to keep our backups on course, so LTO-6 library was the obvious next step.

Over the last 2+ years, the library has been extremely unreliable in its operations, most notably with tapes being marked full early (job will request or move on to a new tape well below the tape's capacity) and archive jobs or verifies grinding to a halt. It seems like the hardware is way more fickle in the library--the rest of our environment has stayed exactly the same as when we were operating with only standalone LTO-5 drives.

Is anyone else seeing more issues with the library units than standalone drives?


Return to posts index

David Roth Weiss
Re: LTO Library Automation vs Standalone Drive?
on Apr 14, 2017 at 5:53:45 pm

Dan,

While most of abhor paying yearly support fees to resellers, your post is a perfect example of what often happens when those of us who think we're technically savvy purchase, deploy, and try to maintain these types of systems on our own. When I worked for a well known reseller, I used to give my customers a piece of advice when they would balk at purchasing a support contract, "I'd rather have you yell at my support guys when something doesn't work properly, than risk your boss yelling at or firing you."

David Roth Weiss
Director/Editor/Colorist & Workflow Consultant
David Weiss Productions
Los Angeles


David is a Creative COW contributing editor and a forum host of the Apple Final Cut Pro forum.


Return to posts index

Dan Olsen
Re: LTO Library Automation vs Standalone Drive?
on Apr 14, 2017 at 6:06:14 pm

Thanks David, I appreciate the response. Unfortunately, all of these issues continue to persist despite working closely with TOLIS support from day one. Sometimes I'll go months with the library just churning through archives and being well worth the price tag, but I've also hit strides of weeks or months where one issue or another is totally impeding my ability to archive media.

With the standalone LTO-5 drives, contacting support was extremely rare and usually for a minor issue. With this library, troubleshooting has pretty much become a staple of my workday.


Return to posts index


James Vorley
Re: LTO Library Automation vs Standalone Drive?
on Apr 15, 2017 at 12:18:14 am

Using TOLIS LTO-6 standalone just now.

Initial results were positive but now seem to be experiencing randon slow read/verify times. TOLIS have acknowledged the issue but as drive is HPE we're reliant on them diagnosing. Which hasn't happened yet, even though it's been 4 weeks. It seems to be a very slow process. L&TT tool showsservo framing errors and some other issues.

Looking to get an LTO-6 autoloader from TOLIS imminently for backup (rather than archive) workflow. But slightly reluctant if our standalone isn't working properly.


Return to posts index

Dan Olsen
Re: LTO Library Automation vs Standalone Drive?
on Apr 15, 2017 at 6:10:30 pm

Thanks James, and good luck with HPE. I spent weeks trying to work with their tech support and in the end they wouldn't even acknowledge that their hardware was compatible with BRU PE (I believe the HPE docs list BRU Server, but not BRU PE)--which obviously isn't true because TOLIS (& Archiware, YoYotta, etc I'm sure) works with HPE reps--but their lower to mid-level support personnel are severely misinformed on some of the applications of their hardware. Rather than working on the problem, I was spending time trying to have them understand what I was trying to do and that it was a known & accepted practice, which ultimately they did not buy into at all. We never reached a solution.

It was then that we ditched the HPE branded MSL2024 and got a TOLIS library with their firmware, so that support for software & hardware could come from the same place. The nightmare of working with HPE's general tech support is gone, but unfortunately we're still having a lot of issues with the library.

When the library is working properly, which it has for months at a time, it is fantastic. But I definitely would not say that it is anywhere near as stable as the rock solid performance I had with 2x LTO-5 standalone drives for many years. I'm actually in the process of ordering a cube from TOLIS with 2x standalone LTO-6 drives supplant the library when it isn't functioning properly. Not ideal to spend that money, but I still think TOLIS is the best option, and we quite literally can't afford to have our backup/archive system down as much as it has been with the library. Hoping the standalone/cube is of the same stability that we're used to.

I truly hope you have more success with HPE support than I did!


Return to posts index

Dan Olsen
Re: LTO Library Automation vs Standalone Drive?
on Apr 15, 2017 at 6:44:41 pm

Hey James--when your drive is experiencing slow reads, have you noticed if its making more noise, or different sounds, than it does during normal operations?


Return to posts index


James Vorley
Re: LTO Library Automation vs Standalone Drive?
on Apr 15, 2017 at 8:02:27 pm

To clarify - our drive is from TOLIS but the drive inside is HPE (I believe it's only HPE and IBM that make drives LTO-6 drives).

So our support contact is TOLIS but they then deal with HPE as this is potentially a hardware fault with an HPE part.

When the drive is in "slow mode" it sounds like it is stopping and starting. The drive is quite whiny when in operation but I believe this is to be expected.

I'm curious as to exactly what issues you've had with the LTO-6 autoloader as this is what we're looking to purchase shortly?


Return to posts index

Dan Olsen
Re: LTO Library Automation vs Standalone Drive?
on Apr 16, 2017 at 6:40:31 pm

Ah, okay. It sounds like we may be experiencing a somewhat similar issue, or at least similar symptoms. When the drive is operating normally, it "whines", whizzes, etc, constantly for long periods of time--sounds like it is constantly doing work, moving through tape. Great!

When its having issues, it will whine, slow down, stop, make a knocking/clacking/repositioning noise, then whine up again, and repeat every few minutes or more, each time interrupting the job's progress. Recently, its doing less of the knocking/clacking and just whining and stopping over a period of a few seconds--i.e. if running a verify pass, it will whine, verify maybe 1 or 2GB, slow down, whine through another GB or 2, and repeat. Both behaviors have happened with both drives (auto loader has two LTO-6 drives.

As far as other issues with the autoloader, the single biggest problem and persisting issue is that the autoloader reports that a tape is full when it is not. I always load/allocate the right number of tapes for a job, and over the course of the 2+ years we've had the autoloader, I've regularly arrived in the morning to find a prompt for more tapes. The autoloader will mark a tape full when its written only 200GB, 500GB, 1TB, etc.--varying amounts. This is definitely an error because if I load one of the "full" tapes back in, BRU will continue to write to it--but then the volumes for the job get all screwy. Completely unpredictable behavior. I cancel and restart the job each time this happens, and most times the 2nd attempt completes successfully--the exact same media from the same storage, everything the same. So definitely an issue with the hardware or BRU PE.

Beyond that, just incredibly inconsistent speeds, jobs just stalling out during the write or verify--all things that point to issues with the drives, and it has been very persistent since the machine arrived. I'm not sure if the robotics in the library are just significantly more fragile or what, but its caused me to at times want to abandon the entire ~$10K investment and go back to standalone drives since they were more reliable--or maybe they aren't!

It's been the most frustrating issue we've ever had to deal with by far.


Return to posts index

James Vorley
Re: LTO Library Automation vs Standalone Drive?
on Apr 16, 2017 at 7:03:11 pm

Ok, it sounds like our stopping/starting behaviours are very similar.

Right now we're waiting for HPE to provide a solution but this is a very slow process. It was last November I first discussed the issue with TOLIS and about 4 weeks ago that HPE got involved.

What gets me is the randomness of the issue, it doesn't always appear. And I will see wildly varying speeds when reverifying the same tape repeatedly: sometimes 180 MB/s for the full data set, sometimes as low as 30-40 MB/s.

Have also tried four different Macs (2x nMP, 2 x old Mac Pros) and two different HBAs (ATTO H680 and the Highpoint Rocketsor 6328 and all see the same issue.

So if you're unlucky enough to end up with a drive like mine the standalone may not fix your issue.


Return to posts index


David Roth Weiss
Re: LTO Library Automation vs Standalone Drive?
on Apr 16, 2017 at 7:08:30 pm

I hate hearing about these kinds of issues, and I'm really feeling your pain Dan.

Given that the entire LTO food chain, including hardware, software, and media is, in my opinion, both overpriced and not nearly as foolproof and reliable as it should be, I think I'd be inclined to build a redundant nearline diskbased archival system if I were running a facility. While nothing is perfect, at least it would be faster, cheaper, and more reliable than the current state of LTO.

David Roth Weiss
Director/Editor/Colorist & Workflow Consultant
David Weiss Productions
Los Angeles


David is a Creative COW contributing editor and a forum host of the Apple Final Cut Pro forum.


Return to posts index

Dan Olsen
Re: LTO Library Automation vs Standalone Drive?
on Apr 17, 2017 at 4:13:38 pm

Thank you David, I completely agree on the lack of reliability, which was a total shock after so many years of great performance from LTO-5. It has definitely been very frustrating and has eaten away at quite a bit of sleep over the last few years.

I've considered NL disk-based, but it would grow incredibly fast at our facility. If I had a functioning LTO system I'd write about 1PB of new data to tape each year. And the nature of our work requires us to constantly use footage going back many years (as well as for stock footage sales) so it is crucial that everything is archived safely.

If you are actually able to get media on to tape, its tough to argue that there is a safer method--I can't recall more than a handful of issues ever restoring data, and they were all fixable (well, I'm starting to restore from our old SDLT tapes to bring them onto new tape, so maybe that isn't completely accurate..but an issue for another day!)

James - I'm with you. Completely random success or failure, and have tried every configuration under the sun. H680, HighPoint 2722, mac mini, old mac pro, nMP, multiple versions of OS X, different SAS cables, etc. I'll run a job once and it'll have any of the issues mentioned in my previous post, abort and start over, run exactly the same job, everything constant, and have success. Verify stalls or is crawling, cancel, run it terminal, and its fine and runs at speed. Doesn't make any sense at all!


Return to posts index

James Vorley
Re: LTO Library Automation vs Standalone Drive?
on Apr 17, 2017 at 8:06:06 pm



Here's my SAN i/o during a write I'm running at the moment...

I'm curious as to why are there periods where the speeds aren't reaching full speed?

I understand that the drive will slow down when writing lots of small files (vs large files), but this pattern is clearly cyclical and repetitive in nature so it doesn't look like it's related to the file sizes.

There is no other activity on my SAN at all today apart from this tape write.


Return to posts index


Dan Olsen
Re: LTO Library Automation vs Standalone Drive?
on Apr 18, 2017 at 5:44:23 pm

That's a good question James. I ditched archiving from anything except directly attached thunderbolt RAIDs to remove storage from the list of potential culprits.

I'm starting to agree more and more with David--the whole chain for tape storage is just unreliable, and for me it started with LTO-6 (and automation). I've done a lot of troubleshooting over the years but LTO takes the cake by a long shot. Currently listening to one of the drives revving up and down, write speed dropping well below that of the other drive, when all day yesterday the exact same configuration (same exact type of data on the same source drive) cruised right along and finished at top expected speed. I don't know what else that kind of behavior could point to beyond an unreliable system at the core.

I'd love to be proven wrong but its been an almost daily battle for two years with nothing even resembling an explanation or solution.


Return to posts index

James Vorley
Re: LTO Library Automation vs Standalone Drive?
on Apr 18, 2017 at 7:55:36 pm

Dan, here's what my drive sounds like when it's in "slow mode".

This is a write operation. 100 GB of RED rushes being pulled from 2 x USB 3.0 G-Drives in RAID 0 (BM disk speed test says 322 MB/s write and 326 MB/s read).

I aborted the write then tried again, worked better with less clicking but still an oscillation between about half and full speed as per my SAN graph.

Doing some more testing but if it's anything like the issues I've been seeing if I reverify this exact tape multiple times I will get average verify times for the complete data set ranging from 20 MB/s to 180 MB/s.


Return to posts index

James Vorley
Re: LTO Library Automation vs Standalone Drive?
on Apr 18, 2017 at 9:21:52 pm

Hopefully recording attached this time...

11274_slowlto.m4a.zip


Return to posts index


Dan Olsen
Re: LTO Library Automation vs Standalone Drive?
on Apr 19, 2017 at 5:53:00 pm

Not exactly the same, but similar sounds from a base-level perspective. When the drives are acting properly in the library, they make the same constant sound for sustained periods of time--many minutes straight if not longer. When they aren't working properly, its very clear because the spinning/whining noise of productivity slows to a halt, sometimes makes the knocking/clacking sound which seems like something is physically repositioning, and then starts whining again before repeating the aforementioned events. Or, it'll just keep whining up and down over periods of a few seconds without the clacking. That's when its really reading/writing slowly.

It's been working properly for the last two days but I'll nab a sound recording next time it inevitably starts screwing up again. But yeah, similar but different noises.


Return to posts index

Dan Olsen
Re: LTO Library Automation vs Standalone Drive?
on Apr 21, 2017 at 4:13:49 pm

And the lack of reliability continues. Had an 8TB archive job complete the write and get through 4.3TB of the verify, and all of a sudden one of the drives decides it doesn't want to do its job anymore. Back to the revving up and verifying a tiny chunk of data, revving down to a halt. Up, down, up, down. This machine really just picks and chooses when it wants to work. Here's a sound bite, you can hear the revving behind the fan noise and the noise of the other drive actually doing what its supposed to.

11285_tgrlibrarynoise.m4a.zip


Return to posts index

Tim Gerhard
Re: LTO Library Automation vs Standalone Drive?
on May 2, 2017 at 4:23:08 pm

Dan,

At Magnext we specialize in tape library repair and refurbishment. I saw you mentioning that your tapes were being marked full early on, this is usually a software related issue rather than hardware.

What software and library are you using when you're experiencing these issues? Also, when you start getting slow downs and such, are you writing the files from a network source or a local disk/raid etc. ?

Tim Gerhard
Magnext
614-433-0011 x114
tgerhard@magnext.com


Return to posts index

Dan Olsen
Re: LTO Library Automation vs Standalone Drive?
on May 17, 2017 at 10:31:41 pm

Thanks Tim, I'm getting my tapes marked full early in BRU PE but I've heard of others in the business getting the same behavior with Archiware P5. Its very inconsistent and seems to come in waves. I only write archives from locally attached thunderbolt raids that can easily handle the speeds of 2x LTO-6 drives.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]