More Frustrations with XDCam EX Format
We shoot for pay per view television using the PMW-350 as our main camera and an EX1R for Bcam. I have had considerable frustration in my postproduction workflow and I'm about to sell all the gear and give up using this format.
I won't go into all the history but most of the problems seem to have been with the XDCam Transfer and XDCam Browser software. My workflow after each shoot has been to take the approximately 20 SD cards that we use on location and copy them to harddrives by dragging the intact BPAV folders into directories named REEL 1, REEL 2, REEL 3, etc. I don't touch any of the native folders or file structure. It's just a straight transfer.
I then check each BPAV folder with the orginal SD card to make sure every byte of data got transfered over. When I am satisfied I have everything from a shoot, I mirror the hard drive to another hard drive and place the SD cards back into service for the next location shooting.
To deploy the footage into a program, I navigate to the desired "REEL" folder with XDCAm transfer and import. This works about 75% of the time. Frequently, not all the clips come up in the XDCam browser and I get errors. Sometimes, entire folders can't be read in XDCam Transfer. This is entirely random.
When this happens, I transfer the BPAV folder to a local drive on the MacPro machine and try to read the same folder with XDCam Browser. Thankfully, this seems to save me most of the time. A folder or clip that chokes XD Tranfer will nearly always be viewable in XD Browser. (I then copy the files into a new folder and start over with XDCam Transfer)
Today, my luck ran out. I have 10 BPAV folders of clips from the same shooting that can't be read in either Transfer or Browser. I have tried every work around, including copying the BPAV folders back to sDcards and trying to read them from the camera. Even that that failed.
I know ALL the data is there. I can see it. My records also refelct that I got every byte of data from the original SD cards before they were formated for another shoot. I know the camera is working properly because I just pulled the SD cards from our latest shoot and so far, all of them can be read in both Transfer and Browser. Please help me read and save these 10 BPAV folders. We will be out approximately $10,000 if we lost this material.
I just tried the latest version of Browser [2.6] and after loading once, it now just crashes and cannot be relaunched. I've rebooted and that does not help.
Many thanks for any help or input.
Thank you very much.
[W. Lawrence Stevens] "My workflow after each shoot has been to take the approximately 20 SD cards that we use on location and copy them to harddrives by dragging the intact BPAV folders"
Don't blame the format when it's your workflow. You shouldn't drag folder every. ClipBrowser with CRC On so it can verify the copy. Anything else and it's an accident waiting to (or already in your case) happening.
[W. Lawrence Stevens] "Please help me read and save these 10 BPAV folders. We will be out approximately $10,000 if we lost this material."
Try reading the individual MP4 files using Clip Browser or XDCAM Browser ... if the MP4 can be read then you can use the same app to rebuild a new BPAV directory for those clips, and so import them into your NLE as per normal.
Another option might be to buy and use Calibrated Software's MP4-EX Import component which would allow you to read and edit directly with the native MP4 files (so you don't need the BPAV directory) ... as would Sony's Cinemon import component (which they call their "workflow accelerator").
Using Browser to "import" the individual MP4 files is tedious but it appears to work. It just goes to show that all the footage is there intact.
I still don't understand why the copied BPAV folders can't be read in Browser or Transfer when byte for byte, they are exact duplicates of what originally existed on the SD Cards. I'm not feeling this is a reliable system if you have to jump through hoops just to get simple concepts to work.
[W. Lawrence Stevens] " when byte for byte, they are exact duplicates of what originally existed on the SD Cards"
You can't be sure of that. That's what CRC On determines. The function truly exists for a utilitarian purpose. It's not window dressing.
If you want something a bit more powerful when moving files there's ShotPut Pro
Sorry but your workflow is not reliable. If you drive without a seat belt you risk a fatal accident.
[W. Lawrence Stevens] "I'm not feeling this is a reliable system if you have to jump through hoops just to get simple concepts to work."
Craig is absolutely right Lawrence, it's your existing workflow that is flawed, not the reliability of the system. Follow the right practice and you'll be fine, use a shortcut and you'll probably be fine too ... right up to the day when you're not.
[W. Lawrence Stevens] Using Browser to "import" the individual MP4 files is tedious but it appears to work.
Is it 89 USD worth of tediousness? That's all it costs for the EX-MP4 import component. Then again, using Clip Browser is free. Just think how much more tedious it would have been if you'd lost 10 grands worth of footage ...
Glad I could help
I am glad that you were able to get your files to work.
At the risk of piling on, I have to agree with Craig and Andy. I download 6 full SDHC cards worth of clips somedays. I use Clip Browser with the CRC on to transfer the files to my back-up hard drive. Clip Browser puts them in a randomly named folder which you can rename anything you want. This, in combination with proper camera metadata, makes it easy to search hundreds of files. It takes a bit longer but not nearly as long (or traumatic) as trying to find missing clips. For my purposes I then use XDCam Transfer to bring those files into Final Cut Server for my client access.
It is much easier, faster, cheaper(?) than tape. It is NOT the format.
Thanks gentlemen. I appreciate the advice and collective wisdom. The problem I have (which I didn't want to bore anyone with) is that I have an assistant upload all the SD Cards in the field immediately after they are shot on a small windows computer.
Our location shoots last over 15 hours so uploading the 20 or so full SD cards is easy during that time. It saves me many hours of babysitting the uploads afterwards if I had to do it myself. Instead of spending all that time uploading data, my job I just to compare each BPAV folder with each corresponding SD Card before it is reformated to make sure the upload was done correctly. That task takes me about an hour.
I need something idiot proof that can be done by an assistant in the field with a Windows computer without much training or fanfare. Apparently, a basic folder copy operation in Windows is not reliable (if someone can explain why, I'm really interested!)
Cost is really no object. I need something easy and fool proof. Clip Browser crashed on my Mac on the second go and won't come back. Is Shotput automated? Any further advice is appreciated.
Many thanks again!
I think you are putting too much emphasis on how long a card takes to copy correctly using clip browser with CRC. It takes longer but it is still faster than realtime. If you are recording 113 minutes onto a 32GB card it will definitely be done copying before you are finished recording the next card. If you are using multiple cameras and recording more cards you could get a little backed up. I would buy more cards. They are VERY cheap.
I have Shotput but it is very confusing for my little brain. I really need to be able to copy a card and change it's name one at a time. Especially if I am working on two or three different job numbers at the same time.
I bought a four-headed card reader (Medusa....just kidding) but that doesn't work either.
MXM (SD adaptor) Card Readers worth looking at
MXR (SD adaptor) MultiCard Reader
I have had the odd problem with clip browser but the most effective way if you want to dump everything into your NLE is to use
Sony's Cinemon import component software. You set finder on a single directory and search Mpeg4s, open your bins and drag and drop the clips into your system. The re-wrapping occurs very rapidly.
Sometimes when I've been unable to access a file, this procedure always recovers them.
All the best,
I have to agree that the work flow is a problem. If we could just have XDCAM Transfer for Windows it would solve a big problem for me. Browser does not let you set in and out points. All you can do is copy the entire contents of the card over. Thats crazy. I have read the entire manual and it never says how to review certain clips, set in and out points and "add to que".
Also, media that works is not cheap. I have had several errors on SD cards and so has my partner with his EX-1. SXS seems to be the only reliable card and they ARE expensive.
It would seem that Sony could come out with a one step process to get .mov files into both Apple and Windows computers.
Ian are you listening?
Apple Certified Master Pro FCS 2
New convert to Adobe CS5.5 Production Premium
[Ron Pestes] "Also, media that works is not cheap. I have had several errors on SD cards and so has my partner with his EX-1. SXS seems to be the only reliable card and they ARE expensive. "
Roger that. I've never had a problem with SxS cards, while I have had one problem with an SD card. Many people do fine with SD cards, but it only takes one blown shoot to more than pay for a set of SxS cards.
I don't use clip browser. Since Final Cut now can do a log and transfer, I don't have a need for it.
I make a folder with a title, the drag the BPAV folder into it. I don't have a SxS card reader, so when I need to grab files off of my SxS card, I need to plug in the camera and copy it over. I use an SD card reader for my SD card adapter.
Once I have all my BPAV files transferred, I then back them up on my external drive, do a quick review to make sure the clips are there, and then delete my old clips from my cards to prep for the next shoot.
It's nice to have an assistant, but they have to be trained and dependable.
Owner / Director / Editor
Video Marketing Toolbox.net
Canon 5D Mark II
Mac Pro Tower, Quad Core,
with Final Cut Studio
HP i7 Quad laptop
Adobe CS-5 Production Suite
[Brent Dunn] "I make a folder with a title, the drag the BPAV folder into it."
If you've been following this and other threads on this forum, you'd know this is fundamentally the wrong thing to do. A single bad copy can cost a professional a fortune. ClipBrowser with CRC or ShotPut and the equivalent is a must for any professional. Sorry but what you suggest is flatly the wrong thing to suggest for any professional.
The Plot Thickens:
I checked back to compare the bytes from the SD Card to one of my in-field transfers done by an assistant. It turns out there was a bad file copy and I did not catch it when I was doing my double check. The number of bytes were the same but oddly, files and folders were different. I was able to save this MP4 by purchasing some software (mentioned above) which allows the Mac to play MP4 files natively. Fine. Now here's the problem.
I'm looking at the latest location shooting where I have 12 A Cam SD cards shot with our PMW-350 and 12 B Cam SD cards shot with the Ex-1R. All the transfers were perfect. Byte for byte, file for file, folder for folder. They show the exact same file structure, size, etc. Guess what?
All the A Cam transfers are readible in Transfer and Browser and Clip Browser. But NONE of the B Cam tranfers are working. This is frustrating becase both A and B Cam files were uploaded in the exact same fashion and under the same conditions. I have done this hudreds of times previously without a problem. Nothing has changed in my workflow.
Any thoughts? Another mystery to solve.
(I'm sorry, I can't use Shotput in the field... I need to use Windows 7 folder copy and make it work reliably somehow.)
[W. Lawrence Stevens] "But NONE of the B Cam tranfers are working. This is frustrating becase both A and B Cam files were uploaded in the exact same fashion and under the same conditions. "
If you didn't use ClipBrowser with CRC it makes no difference. You're driving without a seatbelt and car accidents can be fatal.
An odd experience with ShotPut Pro, and I wonder whether anyone else has had a similar one. Yesterday, I tried several times to transfer an SxS card, using ShotPut Pro. It kept getting stuck at the point where it said 5 minutes remained. I used Sony's Clip Browser ver. 2.6 to do the transfer, one hard drive at a time. Then I tried ShotPut Pro again - and it froze at the 5 minute mark - but I went out to dinner, and when I came back, it was finished. So all is well, sort of.
Should I be concerned? The one thing I didn't think to try was a restart of the MacBook Pro.
I received this notice from ShoPut Pro.
CRITICAL ShotPut ProTM Update -- v4.1.2 for Macintosh
Yesterday we discovered and fixed a serious problem specific to ShotPut Pro v4.1.1 for Macintosh. Users that have recently purchased or updated to v4.1.1 are advised to immediately update to v4.1.2 to avoid potential data loss.
We found that in certain instances, without warning, the software was not copying all selected files. Unfortunately, this problem was not detected during our normal tests prior to release.
Please update your copy of the software immediately to v4.1.2 to correct this potential problem.
To update, click "Check for Updates" within the application menu. Or log into your account at http://www.imagineproducts.com. Click My Serial Numbers at the top of the page, and click the download link next to your v4 serial number (ending with "D1112").
We apologize for any inconveniences this may have caused, and are very grateful to those who alerted us in a timely manner to it. Please know we've also taken steps to detect such problems in the future releases.
Again, this problem is specific to v4.1.1 only. Prior versions of ShotPut Pro for Macintosh are not affected. ShotPut Pro for Windows is not affected.
[W. Lawrence Stevens] "(I'm sorry, I can't use Shotput in the field... I need to use Windows 7 folder copy and make it work reliably somehow.)"
Sorry, just not seeing why you can't use XDCAM Browser, it's just putting a GUI on the copy function, works just fine on Windows, and offers a CRC check option after copy ... why so dead set on using a dodgy folder copy instead?
There's no way of copying reliably with either Windows folder copy or Mac Finder copy.
If you have a problem with copy speeds, switch to UHS-1 SD-HC cards (Panasonic, Sandisk and Delkin make them). If you have a decent USB 2 bus you'll download at half the time you used to, if you have USB 3 and appropriate reader you'll get around 70MBytes/s real life download speed, maybe more depending on system, etc. You might have to get some new SD to SxS adapters to use the new UHS-1 speed class SD cards. Or, if price is really no object, just get some 64GB SxS cards... ;-) They ARE more reliable for copy. And much faster!
In my experience 90% of copy errors, when using SD or CF cards, originate in the card reader (mostly overheating). So, if you are using Shot Put Pro don't use 'File Size Comparison', use 'MD5 checksum' or 'Byte Size Comparison'. There are card readers that are solid enough to be able to use 'File Size Comparison' verification method and be reasonably safe. F.ex. SxS or P2 in Panasonic USB Reader. SD or CF readers that you buy for $50 are not that safe. In my experience, that is...
Hope that helps.
Aasulv Wolf Austad, fnf
Director of Photography
Los Angeles based
UPDATE: The frustration continues.
All recently shot cards uploaded from the A Camera work fine with my workflow. All the cards from the B cam, however, exhibit the same problem which started this thread. Every byte of data is transferred and accounted for from the original SD discs but the drives which hold the data won't read in Browser or Transfer. The data is managed exactly the same for each camera. Only the B camera (EX1R) files won't read in Transfer or Browser. I have experimented with every possible combination to figure out why there is a difference when the workflow for A cam files and B cam files is identical.
The only difference is the type of SD cards. The A Cam is using Sandisk "Extreme" and the B Cam is using the Extreme "Pro" discs. How could this be an issue if all the data is there?
I'm fairly fed up.
[W. Lawrence Stevens] "I'm fairly fed up."
Please don't take this the wrong way, but whatever the issue is, blame it on the fact that you use SDHC cards and not the recommended SxS cards. In other words, don't blame the camera. Sony won't condone the use of 3rd party cards for a very good reason: some work wonderfully and some will not work properly at all, even those of the same make, model and manufacturing lot number. Same goes for the SDHC 3rd party adapters. Go figure. If you find an SDHC card and adapter that works reliably then you've found some inexpensive media and more power to you. But you can also be badly burned and lose clients in the process.
Don't get me wrong. I've talked to camera people that shoot with EX cameras and travel all over the world and use SDHC cards exclusively. They can't understand why I don't use them too because they haven't had a problem. That's fine for them. But I also hear stories like yours. I don't hear these stories from using SxS.
Personally, I'm willing to pay (a lot) more for SxS Pro cards and at the end of the day know I'm going to be able to transfer and edit all my clips without a hitch. I've owned an EX1 since December of 2007 and now have an EX1R and an EX3. In that time I've never had a media transfer or recording issue. Never.
I don't like losing clients because of something I did, like having equipment failure. Your choice.
A Vancouver Video Production Company
Hi Don: Your points are all well taken and you have helped me before.
I just checked some older B cam files and it appears everything was working fine with the B camera until July of this year when half way through the day, it appears something broke down causing the problem I'm having now with resulting transfer of files. I have 14 8Gb SD cards (for each camera) and two adaptors and they all functioned without a hitch. I'm pretty certain this is not a card or adaptor failure or inconsistence or I would have experienced it before.
I'm now focusing on the a potential problem with the EX1-R. Ever since the shooting in July, my B cam disks have not been able to transfer to hardrive back up. I am going to start another thread to see if anyone else has experienced this.
Best of luck with solving this issue. I wish I could be more helpful but I have no experience at all with SDHC.
If the issue is the same with a borrowed SxS card then send the damn thing into Sony and tell them to fix it. The expense will be well worth it if the problem goes away.
Cheers my friend.
A Vancouver Video Production Company
Probably too obvious, but have you tried the cards and adapter from the A cam in the B cam? You might be able to isolate the problem or fault as being with the specific media and/or SDHD adapter. Also, have you updated your camera firmware anywhere along the way?