FORUMS: list search recent posts

SAN Video Corruption Little Pink and Green rectangles

COW Forums : Apple Final Cut Pro Legacy

<< PREVIOUS   •   FAQ   •   VIEW ALL   •   PRINT   •   NEXT >>
Truckee Lynch
SAN Video Corruption Little Pink and Green rectangles
on Feb 19, 2011 at 2:03:07 am

Hello,

We have a catalog of five minute .mov files on our SAN that we continually reuse. They are encoded Apple ProRes 422, about 5Gb each, 1920X1080 etc. Every so often we'll notice corruption in one of these assets and have to go back to the source are re-render it. A huge pain.

We've been tearing our hair out for months trying to locate the source of this bit rot. We think it may have to do with metadata being assigned by Adobe Bridge but this theory has yet to be substantiated. We use a combination of Final Cut Pro, Bridge, and perhaps Compressor to generate these files and they are stored on our SAN using XSAN 1.4 in conjunction with an XRAID. Our server is a G5 xserve.

I have attached an image of a corrupted frame. Generally the corruption lasts for one frame and so far has only affected a file in one place.



Any ideas on what is creating this? It is making us crazy.


Return to posts index

Jerry Hofmann
Re: SAN Video Corruption Little Pink and Green rectangles
on Feb 19, 2011 at 3:11:03 pm

Last time I saw something like this, I reseated my graphics card, reset the display prefs to refresh them, and it went bye bye. Not sure if this is the problem, but it's sure worth a try.

Jerry

Apple Certified Trainer, Producer, Writer, Director Editor, Gun for Hire and other things. I ski. My Blog: http://blogs.creativecow.net/Jerry-Hofmann

Current DVD:
http://store.creativecow.net/p/81/jerry_hofmanns_final_cut_system_setup

8-Core 3.0 Intel Mac Pro, Dual 2 gig G5, AJA Kona SD, AJA Kona 2, Huge Systems Array UL3D, AJA Io HD, 17" MBP, Matrox MXO2 with MAX - Cinema Displays I have a 22" that I paid 4k for still working. G4 with Kona SD card, and SCSI card.


Return to posts index

Truckee Lynch
Re: SAN Video Corruption Little Pink and Green rectangles
on Feb 22, 2011 at 5:57:46 pm

We're pretty sure this isn't the issue because the files are fine when they are rendered, and display the same way (with the gnarly little rectangles) across multiple computers. Leading us to believe that no one graphics card could be the issue. At some point when the file is just sitting on the SAN something(or someone I suppose) corrupts it. Cosmic rays? Goblins? Adobe Bridge? All three?

-Truckee


Return to posts index


Jack Lymon
Re: SAN Video Corruption Little Pink and Green rectangles
on Jul 29, 2012 at 11:21:36 pm

Truckee Lynch - did you ever fix the issue?


Return to posts index

Truckee Lynch
Re: SAN Video Corruption Little Pink and Green rectangles
on Jul 30, 2012 at 3:16:20 am

Not really. We stopped using Adobe Bridge and the problem went nearly completely away. However at roughly the same time we also changed our workflow in a few other ways.

A couple of weeks ago we found that it began happening again, although not on the same scale. Perhaps 3 files not 50. This time, something else, CATDV or Compressor, had recently touched the files.

Our best guess is that the header data is written somewhere near the image data and that some poorly written operations occasionally flip bits in the image data. Either that or some .mov files are written poorly making them susceptible to this kind of corruption.

Do you have a similar problem?


Return to posts index

Jack Lymon
Re: SAN Video Corruption Little Pink and Green rectangles
on Jul 30, 2012 at 4:40:41 am

So, you didn't change anything with XSAN?

We're having the same issue at work and it's the exact same situtation.

Files just sit there and get corrupted.
It's only Prores, we use FCP and Compressor and quicktime 7 pro.
It's like after a week the files go bad, we're trying to solve it but haven't come to any solutions. It also only happens in the first minute or two in the file, nowhere else does that bit-rot appear.


Return to posts index


Truckee Lynch
Re: SAN Video Corruption Little Pink and Green rectangles
on Jul 30, 2012 at 8:58:18 pm

Your situation appears to match ours exactly.

We didn't change anything with XSAN.

What do you use for metadata tagging or logging? Does anything scan you San and assign metadata?

Apparently this last time we think the culprit was actually Cinema Tools. All the files that got corrupted had their metadata read and rewritten by Cinema Tools.

We also found that the corruption consistently happens between the first and second minute of footage, we're assuming it's the same block of bits getting corrupted and how compressed the footage leading up to the bit rot determines how far into the file you see it.

If we learn anything more, I will reply to this thread. Please do the same! We would love to know we solved this problem once and for all.

If you want to reach me directly my email address it truckee (at) vafitness (dot) com


Return to posts index

Garret Maki
Re: SAN Video Corruption Little Pink and Green rectangles
on Nov 10, 2012 at 3:12:47 pm

Hello. Glad I found this thread as we have had almost exactly the same problem using ProRes HQ 4:2:2 in an Isilon NAS environment.

We have been struggling with this for months or more than a year and finally we assigned an operator to just sit there and do iterative tests until we could repeat the problem.

Well it didn't take very long and here is what we have discovered.

If the file is on the NAS, and it is opened in Quicktime on a MAC and then saved, whether anything is changed or not, it will be corrupted within the first two minutes exactly as described in the initial post.

If the file is on the local drive, no corruption.
If you use Quicktime on a PC, no corruption.

I suspect this began happening with a specific version of QT, as we had been working for years with this same process without a single corruption and then suddenly they just started creeping in and becoming more and more common.

Anyway I found this thread because I was searching for more info and to see if anyone had discovered the same thing and perhaps a solution. I hope this helps.

We will be doing more testing, different OS's, different QT versions etc... to try and find a workaround. Of course simply using a PC for QT is an immediate, although not optimum solution, We have so many users and so much going on it is inevitable that someone might out of habit click save on one of them while viewing on a mac so I am looking for something better than just telling everyone to not do that.


Return to posts index

Jack Lymon
Re: SAN Video Corruption Little Pink and Green rectangles
on Feb 17, 2013 at 9:36:24 am

Did this issue ever get resolved, and I'm also curious what version of OSX you are on.

We are still having this issue and we are using LION, everything you said makes sense, but
there's got to be a way to get around this, cause it never happened till we upgraded to fibre/xsan/lion.


Return to posts index


Truckee Lynch
Re: SAN Video Corruption Little Pink and Green rectangles
on Feb 19, 2013 at 10:18:46 pm

We have not run into the problem for a while so we haven't invested any more resources to tracking it down. At one point we were able to recreate the issue by saving a pro-res file after it was opened in quicktime, even though nothing had been changed in the file. Aside from removing quicktime from your workflow(if it is indeed their.) I haven't got many other suggestions than that unfortunately.


Return to posts index

Benoît Godard
Re: SAN Video Corruption Little Pink and Green rectangles
on Mar 25, 2016 at 11:22:56 am

Hi Garret,

I have the exactly same issue here...

Have you got more information about this ?

Thanks by advance.

Benoît G.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]