4th Red Project. Bit hard by corrupt and missing frames. :(
Preamble: This is more of a general discussion on how to avoid this in the future and also to see if anyone has run across these issues. My faith in Red has taken a leery turn for the worse. It's too bad, I was really really beginning to enjoy and trust this workflow. I am approaching this from the post side as I didn't handle production, but I am curious about opinions from the production side.
Red support has been great on the "how-to" work around for these issues. It took a solid 5 days (and nights) to track down all the problems and get everything sorted and fixed for offline editing. Obviously, still no Rocket card for me or that time might have been shorter.
The latest shoot came back to me on two identical hard drives full of the requisite red files. It was a 2 day shoot, for two 30 second spots, so not an exorbitant amount of footage.
Day 01 had three clips that would crash every single program I have at my disposal. RCX, FCP L&T, Clip Finder, Storm Beta, CS5.
I moved on to Day 02 and that's when disaster struck and I realized how much of a problem I had. Day 02 is full of files that hard crashed every program and if the files didn't crash, all viewers turned green.
I then contacted Red and they told me what to do. Not only did I have corrupt/damaged frames, I also had missing frames, frames that are completely gone. These files were proving to be more problematic than the corrupt frames. It turns out, corrupt files seemed to turn green (or sometimes grey and cause audio to turn to static) and missing frames crashed all programs.
After talking with the hired/freelance support crew involved with the shoot, it seems that everything was done correctly. All of the transfers/copies were verified, media played back and spot checked (but not totally checked end-to-end), no red flags. The DP said he saw a dropped frames warning once, but the CF card was switched immediately, checked by DIT, they shot more takes and moved on.
The DIT was using R3D DataManager to verify all file transfers. Everything checked OK. I know if the frames were corrupt to begin with, the corruption is just getting copied and verified, and I love every minute of that.
Searching the internet, (where one goes to find nothing but the best tech support), speaking with other Red faithful and of course to the helpful Red support team all info seemed to point to a mishandling of files.
We were miraculously able to salvage three CF cards before they were sent out on the next job from the rental house in California (I am in Chicago). Once I received the cards, I checked the corrupt frames on the CF cards against the corrupt frames on the hard drives, and all the points matched. This leads me to believe that the corrupt frames were recorded this way on to the CF card, and NOT point to file mishandling, leading to me to either a faulty camera or faulty CF cards. In some wicked twist of fate, the CF cards and camera were rented from different companies. Weird.
Also, I noticed that the audio guide track that was sent to camera was extremely over driven. The DP (who is local to me and I work with a lot and trust) remembered the audio levels looking very low on the Red meters. This seems to indicate something, but what I am not sure.
On to some questions:
-Has anyone seen this level of corruption before?
-Did you ever track down likely causes after ruling out file mismanagement?
-What is it about R3d Files that causes everything to crash? Is there a way to make them more 'robust' or shall I say error tolerant rather than error prone? This is a double edged sword, I'm sure, but it's Red and you live by the sword.
-Is it possible that a mishandled audio signal could corrupt a shoot like this?
-Do RedRocket's help with corrupt frames? How about dropped frames?
-Besides having someone on set to watch every single frame of shot video before erasing a card on set, what have you done to ensure Red reliability after an incident like this?
Thanks for any insight.
Never seen anything that bad, but reduser.net would be a better place to ask around. That gets far more traffic than this forum. There's 2 or 3 forums there that this would be suitable for
[Stuart Smith] " but reduser.net would be a better place to ask around. "
I've tried. They only blame the data handling. It's never really anything but that, but I have found that yo note be true.
Maybe you were part of that thread that turned pretty nasty a week or so ago about this issue. I do some DIT work, and they're wrong. It's next to impossible to check every single frame during a shoot. Of course Im usually there as a post consultant also, so not strictly a legitimate DIT.
I didnt contribute anything to that thread, and Im not suggesting you did either, just mentioning it as it was about this topic and was a somewhat interesting read
I didn't see that one. It's a minefield over there. Do you have a link or can point me in the right direction?
Yeah, unfortunately it's still the best place to go though. However, there's WAY too much noise on those forums, which is a shame, because the speed at which Red releases updates, beta versions etc, they need to have a place where you can get good quick responses without all the fanboy "me too's" taking up 15 pages of a thread.
Yeah, I do love Red, but is a business after all and we aer the one's that have to work with it
rant over, at least they started the crucial ordinance forum
Not sure which forum that thread was in, or what it was called, but it got big, I do know it was in either - Recon, Red Discussion, Red One General, Red CineX or Apple Workflow
All the other forums I follow wouldn't allow a thread like that, or it wouldn't be on topic.
Sorry, not being lazy, but with that info you're now in the same boat as me for trying to find it. 2 weeks ago was the last time I really had any time to go through those forums, so I know it'll have been updated that recently. But nothing got resolved. 2 trains of thought, a DIT should physically check every single frame - or - there's no way a DIT can physically check every single frame during a full shooting day.
Your case sounded like a nightmare though, I've had dropped frames, but good grief, something was seriously wrong with the shoot that those files came from, thats really strange.
[Stuart Smith] "Not sure which forum that thread was in, or what it was called, but it got big, I do know it was in either - Recon, Red Discussion, Red One General, Red CineX or Apple Workflow"
Thanks, I'll try and take a stab at it by throwing a dart at those forums.
[Stuart Smith] "a DIT should physically check every single frame"
To me, that's pretty crazy. This means that reliability goes right out the window. I can't imagine Red, the company, would want their users to feel that every single frame needs to be checked in a production environment in order to garner complete confidence that the money and effort being spent was spent wisely. That seems like a bug that would be in thier best interest to squash in order to compete, but what do I know? Professional production usually goes hand in hand with reliable production.
[Stuart Smith] "there's no way a DIT can physically check every single frame during a full shooting day."
It would depend on the shoot, I guess but that's a lot to ask to take on that kind of responsibility. What if you did find a bad frame 5 hours later? 10 hours later? Does that mean you have to rerig that shot? Who's paying the triple golden overtime on that one?
[Stuart Smith] "Your case sounded like a nightmare though, I've had dropped frames, but good grief, something was seriously wrong with the shoot that those files came from, thats really strange."
That's exactly what I took away from this, simply, "something went really wrong". The trouble was, there was no indication at the shoot (or at least I was told there was no indication at the shoot). It seems to me if there where dropped AND corrupt frames in the same R3D, the camera should tell you something is amiss, dropped frames at the very least, corrupt frames are probably harder to catch/verify while recording. Alexa is on the tip of everyone's tongue around here, not that anything is perfect.
I appreciate your responses, I'll look around on the other forum.
I gave up searching after an hour. Do you remember other details from it? Subject line perhaps?
I've never heard of that massive of a corruption. I would strongly suggest trying to talk RED into looking at one of your files that crash everything. Maybe they can find some clues.
In the couple of years that we've worked with REDs, when some sort of corruption has happened, there has usually been red flags or some sort of warning sign on set that there might be a problem (like the camera not "posting" after recording).
But again, trying to have RED look at one of your R3D files might help.
Colorist/Digital Cinema Specialist
Salt Lake City, UT
[Russell Lasson] "I would strongly suggest trying to talk RED into looking at one of your files that crash everything. Maybe they can find some clues."
Thanks, Russell. Actually, they took two of them and checked them out for me. One of them, they told me exactly where the bad frames were (which I had figured out from their previous instructions). On the other file, they said that the corruption probably happened at or around the 'spanned' area causing a problem with connecting the spanned R3D files together :/ They looked, and they helped me get back on my feet, but there was no explanation as to how and why, and more importantly what can prevent this in the future. I'm just glad we were able to salvage day02.
The audio thing is what is really disconcerting. I am wondering if the wrong power load (phantom power, etc) can cause something like this?
I feel silly even asking, but did anyone listen to the audio signal going into the camera? Were the inputs set to line or mic? This is all normal camera setup stuff so I'm quite sure you would have had all those settings right.
What type of batteries were you using? Anything unusual or not made by RED?
What version of R3D Data Manager were they using? And what copy settings did they use? Were they doing checksums or skipping that?
I also wonder if you would have been able to put the CF cards back in the camera if they would have played back okay.
I'm not sure what you did to upset the RED God, but he sure got you on this one. Maybe he could read your thoughts on how cool you think Alexa is:)
Overall though, we have had maybe around 100 commercials shoots and worked with several feature length films since we got in the RED game and things have become reliable for us. In general we shoot to RED DRIVES and we don't transfer in the field. At the end of they day the files are delivered to post for processing. On our own commercial work we usually just copy the files over for archiving/processing just using Finder. For features we usually use R3D Data Manager to copy and create checksums.
It has really been left up to the camera department to watch for warning signs of possible issues. When they see an issue (like dropped frames or the camera stalling while posting a file) they get another take, play it back on camera to check it, then move on. There have been a couple of times when post gets a phone call to talk about a particular concern and we figure out the safest thing to do. Because we own the cameras and media, this works for us because we have the time to do it after the shoot. There have been a couple of issues along the way, but really nothing that has made us ever feel like we needed to not shoot RED in the future, especially with how promising the EPIC is looking.
After what you've been through though, I can't blame you if you never shoot RED again.
Colorist/Digital Cinema Specialist
Salt Lake City, UT
[Russell Lasson] "I feel silly even asking, but did anyone listen to the audio signal going into the camera? Were the inputs set to line or mic? This is all normal camera setup stuff so I'm quite sure you would have had all those settings right. "
You would think. Unfortunately, I wasn't at the shoot and all fo my quires have turned up nothing. It was obvious the guide track to the camera was setup incorrectly as a lot of the audio is way over driven and unusable (although the DP said that the actually levels received on the camera were super low, perhaps indicating a mismatch of inputs). Whoever the audio guy was did not monitor the return as they would have caught this mistake immediately. The double system audio that was recorded, was perfect.
[Russell Lasson] "What type of batteries were you using? Anything unusual or not made by RED?"
I'll have to check.
[Russell Lasson] "What version of R3D Data Manager were they using? And what copy settings did they use? Were they doing checksums or skipping that?"
I think they used v6.1. I have 'ProjectInformation.txt" files in every reel, but they are weird. There are MD5 numbers. What I don't see is the MD5 number at source and the MD5 number at the destination. This makes it hard to compare. Shouldn't there me a compare at some point and doesn't that info get written to the log? I have never used R3D Manager.
[Russell Lasson] "Maybe he could read your thoughts on how cool you think Alexa is:)"
Heh heh. Good one!
[Russell Lasson] "Because we own the cameras and media, this works for us because we have the time to do it after the shoot. "
All the equipment we had was rented and located half way across the country, which makes tracking down the problems even that much harder. When I get a minute, I do have a friend that will help me to try and get the media to fail with mismatched audio signals. Right now, it's the only tangible thing I have to go on. My previous three Red jobs have been fantastic, the only thing different about this one, was a terribly handled guide track. I need to at least check that out.
[Russell Lasson] "After what you've been through though, I can't blame you if you never shoot RED again. "
It's hard to not be a little Red shy at this point.
Thanks a lot for your responses.
Sorry for chiming in that late, I'm on location in Brazil.
Yeah, it's hard at times to find the useful information at reduser…
I've seen such trouble before and I'm afraid it is a problem with the CF card or it's connections. It's pretty easy to bend some pins or get dirt/dust into the connector. Bad thing is: It doesn't always show immediately and can easily lead to an intermittent problem.
Yes, the DIT needs to check every frame. This doesn't mean that he/she needs to go step-frame, but just browsing the clips doesn't cut it either. You can sit down and have it run normal speed and keep your eyes on the screen.
I'd rather suggest doing a quick transcode with low quality de-bayer to small HD – even a laptop can do that at reasonable speed. The problem will definitely show, if it doesn't, a later high-quality transcode will be fine too.
This doesn't help with your current footage, though. The best way of getting back all frames which are not defective is a RED Rocket, which will normally not crash at thoes frames, while all software does.
BTW, I've already suggested to The Foundry to add a frame-checking feature to Storm…
Director of the Institute of Media Research (IMF) at Braunschweig University of Arts
[Uli Plank] "Sorry for chiming in that late, I'm on location in Brazil."
I feel sorry for you. ;)
[Uli Plank] "I've seen such trouble before and I'm afraid it is a problem with the CF card or it's connections. It's pretty easy to bend some pins or get dirt/dust into the connector. Bad thing is: It doesn't always show immediately and can easily lead to an intermittent problem. "
I read about that. Unfortunately, since the equipment is rented it will be hard to track that down. Is this something that Red will fix? Can the owner send the camera in to get checked?
[Uli Plank] "Yes, the DIT needs to check every frame. This doesn't mean that he/she needs to go step-frame, but just browsing the clips doesn't cut it either. You can sit down and have it run normal speed and keep your eyes on the screen."
Seems pretty crazy. It's the only camera in existence that breeds this level of paranoia. That's a huge problem, don't you think?
[Uli Plank] "The best way of getting back all frames which are not defective is a RED Rocket, which will normally not crash at thoes frames, while all software does."
Yes, good to know. We are still on the fence about pulling the trigger on the Rocket. If we would have bought it for the first job, it would be paid for by now, but Red is still new to us and we are testing the waters with it. Frame only mode out of RCX has done well in identifying the corrupt frame and skipping the dropped frames without crashing.
Thanks for your time and get back to work (or the beach). :)
Well, RED can fix it if it's the connectors on the cam…
But I think there's good reason for them to move towards SSD. They tried to give users a very low-priced option for the start, but CF is not the most reliable system for moving images. I've seen similar problems with DSLRs. It may hurt if you loose a single one from thousands of photographs, but it surely hurts if you loose a single frame in moving images.
I didn't try the reliability of frame-mode in RCX for such cases yet, but it might be OK. So, if you consider a RED Rocket, check it's price/performance against a good 12-core and look at the workflow you want to establish. The RR may not be the most flexible decision.
P.S. It's not the beach, but a natural reserve in the rainforest and there's good reason to call it that way.
Mosquitoes all over us…
Director of the Institute of Media Research (IMF) at Braunschweig University of Arts