Verify pass reduces tape life?
I've been doing a separate explicit verify pass after write when doing backups to LTO-6 and 7 tapes.
While reading up a wikipedia article on LTO I came across this...
"The tape drives use a strong error correction algorithm that makes data recovery possible when lost data is within one track. Also, when data is written to the tape it is verified by reading it back using the read heads that are positioned just 'behind' the write heads. This allows the drive to write a second copy of any data that fails the verify without the help of the host system."
So, verify is happening after writing and any errors are getting corrected on the fly.
"LTO tape is designed for 15 to 30 years of archival storage. Depending on the generation of LTO technology, a single LTO tape should be able to sustain approximately 200-364 full file passes." and "One full file pass is equal to writing enough data to fill an entire tape and takes between 44 and 136 end-to-end passes."
So, these 44 to 136 end-to-end passes are only one of the 200 to 364 full passes? Meaning the tape life is actually 44x200=8,800 end-to-end passes.
"LTO uses an automatic verify-after-write technology to immediately check the data as it is being written, but some backup systems explicitly perform a completely separate tape reading operation to verify the tape was written correctly. This separate verify operation doubles the number of end-to-end passes for each scheduled backup, and reduces the tape life by half."
So, the question is, should one do a full verify pass after write, or should one not.
This is significant with LTO-7 tapes where my records show that 5.1 TB of data takes about 14 hours to write and about 8 hours to verify. If I eliminate verify I can save 9 hours per tape. To backup about 60 TB rushes of a full feature, that would add up to a significant time saving. Almost a week less. Besides increasing the tape life span.
I've just gone through the past record of a feature backup of 137 TB. And the Bru reports did not record a single error after verify.
So, is it overkill to do a separate verify pass?
FCP Editor, Edit systems consultant
"Reduces the tape life by half" must be assuming a completely different workflow, where you rewrite and verify tapes many times with new data. Performing a verify means you can only do this 4000 times rather than 8000 times. Which still sounds like a lot.
However your tapes are probably only written once and verified occasionally. So using those numbers you can still read back the entire dataset 8000 times ! So the verify has not reduced the life at all.
It does take time, but this is your archive. Also the verify performed by the backup application will check that each file is on tape, whereas the drive just checks the bit level data sent to it. So the drive cannot detect if a file is completely missing.
[Neil Sadwelkar] "So, these 44 to 136 end-to-end passes are only one of the 200 to 364 full passes? Meaning the tape life is actually 44x200=8,800 end-to-end passes."
Correct, the number of end to end passes depends upon how many heads there are and for a lower number of heads more end to end passes are required leading to a tape life of say 200 uses and if they are high requiring less passes this can be say 250 uses and here is an old chart that gives you some idea of this https://www.high-rely.com/blog/advantages-of-removable-drive-vs-tape/attach...
Having said that, out of a theoretical 200/250 uses its generally recommended that the tapes be used only a quarter of this amount say 50/62 times after which I presume they start wearing out and the rewrites start which appear as "soft write errors" and in addition to your "second copy" up to a total of 16 rewrites can be done before this now becomes a "hard write error", the LTO standard sets aside 5% of every tapes capacity just for these expected re-writes and I presume if you exceed this 5% allocation (which most of the time you probably don't) your tape run will start asking for ever increasing numbers of tapes to complete the same sized job.
LTO6 should take 5:30 to do a full tape and LTO7 should take 5:55 so if you need 14 hours then perhaps your not feeding it fast enough? I think BRU uses a 15% overhead so 5.1 TB would be a gross 5.9 TB and dividing by 14 hours gives a measly 117 MB/s (5900000/14/3600) which is the typical max speed of a 1Gbe network. LTO6 should be able to do 160 MB's and LTO7 should do 300 MB's, compression (probably unlikely for your source data) would give even higher figures.
I'd be inclined to buy a second set of tapes and just write out a second backup set, sure your capital costs would be higher for a second set of tapes but if you skip the verify step on both they will each last twice as long so your consumption costs (due to wear out and discarding tapes) would still be the same but at least you would have two copies of the data.
The time taken to write to LTO depends on the speed of the drive that the client provides. Sometimes they give us RAID drives with 300+ MB/sec read speeds. In which case a full LTO-7 takes under 8 hours to write and 5 hrs to verify.
With an 8-bay Areca or Promise, over Thunderbolt 2 could take even less, but that's rarely how media is provided for LTO backup.
Sometimes media is Red raw where there are few but large files. This takes less time to write to LTO. Mostly its Arriraw so, millions of 7-10 MB files per drive. That takes a longer since the drive read speed is slower with file sequences. From VFX too, there are millions of OpenEXR files.
Often we get single LaCie or G-Tech drives, even sometimes LaCie rugged drives. In those cases, the write time goes up to 14 hours per LTO-7 since the drive speed is that much slower.
FCP Editor, Edit systems consultant