CatDV Re-archive footage with Cache-A
Mac system, CatDV Client version 9.06.
We're doing some testing with archiving and recalling footage via the Cache-A CatDV plugin. We had an error on our first transfer. We had been told that Cache-A (via the CatDV plug-in) would ask for a new tape when the first one was full. However, in earlier testing someone had deselected "multiple volumes" and we figured that is what caused the problem.
So, we erased the tape and now that we have "multiple volumes" turned on we want to re-run the batch we originally backed up since they did not all get written to the tape. However, CatDV in the "Archive Summary" field shows ALL the files as backed up... (not true) and every time I select all files and go to Tools > Archive Media Files > Archive it comes back in an instant saying "Copied 0 files... 746 file(s) already copied). Blarg.
I can't change the "Archive Summary" field (or any other system generated archive related field) so not sure how to get CatDV / Cache-A to re-archive these files. Any help would be appreciated.
Whoops... figured it out. The material was still on the Cache-A "VTape" and thus the reason CatDV / Cache-A was refusing to rewrite it. We deleted the material on the VTape through the Cache-A interface and then were able to start the backup again.
I was evaluating the Cache-A integration last year but I don't believe I had the latest Cache-A software. We haven't purchased CATDV yet but we are using the Cache-A and tape capacity notification for me is an issue. Can you clarify for me that in order for CATDV to ask for a new tape when it reaches capacity that "multiple volumes" must be enabled on the Cache-A? I can't test this and the last time I chatted with Cache-A tech support they suggested that I not enable that option.
When we enabled multi-volume and Cache-A did ask for another tape (through the web interface) when the first was full. However, we have not successfully recovered files from a spanned set. In fact, the Cache-A plug-in for CatDV doesn't seem to be performing reliably at all and we aren't using it or the Cache-A at this point. I can't detail everything we've tried but basically, it doesn't work. We get conflicting stories from our integrator, CatDV and Cache-A about whether or not we can use spanning. It's been very frustrating because we can't get a straight answer out of anyone.
Right now I have a technician dedicated to getting it figured out. I personally can't spend more time on it as we have actual projects to get done. I would be happy to update you on our progress and what we discover. If you don't hear from me in a while give me a nudge and I will update you here.
Wow, you have no idea how strangely comforting it is to have evidence of another Cache-A/CatDV user experiencing difficulty with products and workflow. Our issues are different however.
I have yet to even play with the multiple volumes options,as at this point with the media types we utilize (3 varieties of MXF)I have yet to have a successful archive out of CatDV to Cache-A. That is an archive that transfers error free as logged by the Cache-A UI
May I inquire as to what media types you are importing into CatDV for cataloging?
Based on our staff, equipment, software etc., it has been convienent and effective to edit directly from the native MXF pulled from tapeless camera media, including P2 cards, SD cards and Ikegami GFPaks. So of course these formats would provide some great challenges in terms of archive! We do have the MXF option purchased for CatDV as well as all of the Calibrated MXF plug-ins too, but it seems that we will be forced to convert from MXF to something else to archive using CatDV. At this point we have no money to invest in a worker node, and I guess I can really only blame myself for not recognizing that issue when the CatDV purchase was made specifically for the archiving function.
We are a PC based system, just FYI.
At this point...I almost miss tape...
I feel for both of you guys. We've had two techs, myself and another, working for the last couple weeks trying to get our archiving system on track. Archiving from CatDV to Cache-A has been a roller-coaster of an experience.
At first things were going very well. The footage would be backed up to LTO and a field in CatDV would auto-fill with the path to where the footage was archived. Our first problem was trying to easily verify that the footage had been successfully backed up. There is no way to compare file size, as Cache-A compresses the files. We did learn that by hovering over a file, it does show you the original size, but it took several days to discover that. You also cannot view the footage from Cache-A, to double check that it's actually there.
We proceeded with archiving this way for a few hours, then started running into issues. Occasionally, a file would not back up and we would receive an error message that CatDV could not find the path to Cache-A. I would try to reconnect to the Cache-A and it would work for a little while before failing again. Sometimes files will not back up and we receive an error message “Unable to make directory.” We have contacted Cache-A and CatDV and have yet to get a useful response.
I attempted to bring footage back in from after purging some test footage. I got an error message saying Cache-A cannot locate the file. Using the Cache-A web gui, we can locate the files on the LTO, but for whatever reason it is not found when trying to bring back the footage.
Every time we think we're making progress, something quirky happens. The Cache-A wont mount, error messages pop up. Our most recent issue, we have a 1.5TB LTO. Put 150GB onto it through CatDV and it said we had 1200GB of "space lost" so the tape was full.
So frustrating. I keep waiting for things to just fall into place and work but as of now, it hasn't happened. I hope things change, because we have a lot of time and money thrown into this and we really want it to work as advertised.
Yeah, I hate to be a pessimist but I have yet to bump into anyone who is using this successfully. I have had integrators demonstrate it to me by writing a couple files and bringing them back but that is a far cry from backing up and recalling thousands of files without issues.
We aren't having any issues on the transfer or errors during Cache-A writing. It's recall that's the issue for us right now. We did discover today that you have to OPEN the catalog and then do the backup in order to retain the information in the "Archived Status" field. If you are in read-only server mode it will show the tape name and such that the clip was archived too but the info. will be lost if you restart CatDV and no other client will see those changes (because they are not actually saved). So, you have to open the catalog, send to to Cache through the plug-in and then save the catalog.
Strangely, we have discovered that when we don't do that CatDV STILL knows where the clip was saved (but doesn't display it in the "Archived Status field)... If you select a clip that has nothing in the "Archive Status" and then go Tools > Archive Status it reports the tape name where it's located! Figure that one out.
I was also going to mention to you and Nathan to watch out for illegal characters in filenames... we are in a Mac environment and anything that had non-alphanumeric characters gave us errors. And unfortunately, we had a ton of those files. Those files would trip the creation of "Apple Double" files and they took up a LOT of space. We eventually got this mostly resolved with CatDV. I am wondering if you and Nathan are getting the same type of problem on the PC side... since you both get seemingly random errors. Could it be filename issues?
To answer your question about file formats we have quite a variety... We have some stuff prior to when I was here shot on Panasonic P2 format but that has mostly been dropped to ProRes. We now shoot AVCHD footage generated from our Sony NEX VG-10's (LOVE this camera).
We did not pay for the MXF plug-in for CatDV because ultimately we decided we didn't need it. We don't really have a need but I could create a Worker Node routine to trip ClipWrap and generate ProRes preview files so the clips could be reviewed in CatDV just as we do with our AVCHD files. But we just haven't had call for it yet since most our stuff was "log and transferred" by FCP 7 into ProRes and the MXF files are long gone. We have a ton of other stuff that is mostly .MP4 or some flavor of Quicktime. But, we haven't had any particular issue with Cache-A or CatDV not wanting to back up a certain type of video... which I think was the point of your question.
We are doing more testing today. I will share anything I think that is useful to you, Nathan, or anyone else interested. Let's keep talking.
Is it sad that at this point I am happy to accept a pessimistic outlook on this workflow? It's just nice to not feel quite so crazy or that it is entirely based on operator error.
Yet its very frustrating indeed. Over the past few months, as I have been solely responsible for establishing this media archive system, I have felt like a hot potato at times, bouncing between CatDV and Cache-A support with little headway made.
I also have little trust when it comes to having absolute knowledge that files have been properly archived. My latest attempt to 'Restore Media Files' in a CatDV to CacheA test resulted in this error:
Perhaps I am missing a step in terms of restoring through CatDV? But it seems to me if I have to manually locate the file, or media directory containing a specific file, through the Cache UI and move it to the VTAPE within the CacheA UI, this defeats the purpose of restoring through CatDV.
Jeff, correct me if I'm wrong, but what you are doing is archiving by moving a CatDV catalog file to the VTAPE. Correct? I haven't tried that method yet. I have only used the function built into CatDV itself, and am careful to select 20 to 30 GB chunks of clips inside a specific catalog to send to archive at one time. The chunk size limit was recommended by CatDV or CacheA (I can't remember at this point). Also just FYI we do not operate with a CatDV server here. We are a two-person crew with only one edit system utilizing a raid array for main media storage and attempting to establish this CatDV/CacheA based archive system.
I just did a quick repeat test of a P2 media directory archive, performed three different ways on a freshly re-initialized tape.
Attempt 1 (first event in attached transfer log) was out of CatDV utilizing the 'Archive Media Files' tool.
Attempt 2 was a basic, staged transfer where the media directory was copied to a 'staging folder' on the CacheA share drive and then copy/pasted from the stage folder to the VTAPE folder. This is when the directory file size seems to change.
Attempt 3 was a basic, staged transfer where the media directory was copied to a 'staging folder' on the CacheA share drive and then dragged and dropped from the staging folder to the VTAPE. This is the only way so far that I have performed successful transfers. I have tested this recall successfully (going to test again now...just to verify).
Nate, you had mentioned looking at file size by hovering over a file in the CacheA UI correct? I had not noticed that feature yet, but when I tried it, I get nothing. One thing I have noticed in regards to file size is that on occasion a media directory will be a larger size according to the Cache UI log at the end of the transfer.
Nate, what type of media are you archiving through CatDV? Does your production house convert all media upon return from a shoot out of it's native format?
Jeff thanks for sharing media type...yes I was curious as to whether it was a media type specific issue. Forgive my ignorance, but is worker node a feature only accessible if you have a server-based CatDV system? Not sure I entirely understand the role of the server though I realize it is a key tool when working in a larger production house.
I had some concerns about portions of the MXF file structure not being archived with the associated media files through CatDV, such as MXF .txt files and empty folders built into the structure for proxies and additional metadata. I feared that when restored our NLE wouldn't be able to read the MXF structure if it was missing components, empty or not. I have been able to ease my fears of this matter with a bit of testing at least with my P2 media.
In terms of basic archives to the CacheA, skipping CatDV, has anyone noticed any difference when performing a staged transfer as to whether the desired media in the staging folder on the CacheA Share is copy/pasted to the VTAPE folder or dragged/dropped to the VTAPE folder? Numerous tests on my end with these slightly different methods with an almost empty LTO tape and CacheA share drive, have proved that if media is copy/pasted to the VTAPE rather dragged/dropped, the transfer will produce various, seemingly random errors and not be completely successfully.
As far as illegal characters and file naming issues, I have tested that as well and haven't come up with any road blocks on our end in regards to that. I performed various tests with the same directory with a name from the most basic to more complex using underscores and numbers, all with success if, and only if I transfer via the basic, staged transfer using the drag and drop method. I even tested file path length as the MXF structure can be deep and file path did not prove to be a limiting factor.
Having spent a significant amount of time importing, adding metadata, and sorting our raw media directories in category/subject based catalogs for efficient, long term accessibility post-archive and with our main raid storage almost at capacity, I am extremely eager to verify a successful CatDV to CacheA Archive and Restore with a successful test. I've taught myself not to get my hopes up too much. Every time I think I've sorted it out, something different seems to pop up.
Anyhow, now that I have mastered the staged, basic transfer, for now I am sadly skipping CatDV until I can figure out how to successfully restore from an archived catalog etc. Any suggestions on that issue?
Lets' continue this conversation. Thanks and good luck to all on this archiving journey!
We are using the CatDV plug-in exclusively, what you are calling the "Archive Media Files" tool. This only appears in CatDV if you have the plug-in. It didn't for us until our licenses came through.
We have to do it this way because we need CatDV to remember which LTO tape a given video clip was cataloged too.
I had a thought on your issue with it working when you drag and drop from Staging to VTape but not when you COPY from Staging to the VTape.
I know that as soon as you drop anything in the Vtape directory Cache-A takes off writing it immediately to tape. Just like a CD-writing session if the writing device (the LTO writer in this case) gets "starved" for data it will create an error (a buffer under-run). I wonder if that is what's happening... Cache-A starts writing but "catches up" to the copying and gets starved for data.
Early on, when we were using Cache-A outside of CatDV to create backups the people at Cache-A were telling our tech. to copy everything to a "Staging" folder first (seems like they are telling you that too) and then once everything was there, drag and drop it into Vtape. Their reasoning was that if you copy across a network directly into the Vtape folder the Cache-A might get starved for data at some point (like when copying a large file) and cause an error.
However, this doesn't work for our situation. First problem: we have to work with the plug-in as I mentioned. Second: we can't tell the plug-in to write to the staging folder because that is non-unique name... if all our footage was written to "staging" (as far as CatDV knows) then it is going to try to restore from "staging" instead of the LTO tape's serial number. So, we MUST write across the network to the Vtape and the plug-in intends for it to work this way. When you start the archiving process you can "see" Cache-A writing the clips from the catalog into the Vtape folder but the serial number of the tape gets recorded in CatDV.
At first we had our Cache-A passing through a 10/100 network and we were getting buffer under-run issues (I think... we were getting errors anyway). However, now we have a gigabit network and we can write with the Cache-A plug-in to the Vtape (through the plug-in) directly without any trouble at all- we never have errors. We're just working on the recall issues.
So what is your network speed?
We are going to get this to work! :)
Good to know 'Archive Media Tools' = plug-in for future correspondences.
As for the drag/drop vs. copy/paste your explanation definitely makes sense. I'll continue using the drag method, and I was mostly just curious on that issue as I found it strange. Another thought I had on that was permissions differences between simply moving a folder (drag/drop) and re-writing a folder (copy/paste).
Yes when working with the plug-in which is ideally what I want to do, the staging folder does not play and a role and it shouldn't have to. Not sure why our plug-in seems to be failing...which again seems to bring me back to media type, though Rolf and others have said it shouldn't be an issue.
I just checked our network connection and we are running at 1 GB/s just as you are. I just performed another quick archive test and as soon as the data starts appearing in the Cache UI transfer log so do the red 'x's and associated errors which are frequently 'Cannot open: Resource temporarily unavailable' as noted in the transfer log I provided earlier.
I have another thought about the archive fields in CatDV, but it will have to wait until tomorrow. Thanks again for your feedback as I am strangely more optimistic now than I have been for some time regarding this situation. Have a great evening!
As the Cache-A guy on the Cow, I would like to jump in here and try to address a number of the issues being mentioned in this thread.
First, the CatDV plugin does not deal with spanned tapes aka Multiple Volumes. As you add tapes to a spanned set on a Cache-A, we automatically rename each subsequent tape with the volume name of the first tape in the set (i.e. "myfiles) followed by an underscore tape number (i.e. myfiles_2, myfiles_3). The CatDV plugin does not recognize this policy at this time.
The CatDV plugin does display the % space left on tape upon invoking an archive. This allows you to manage your archive transfers to fit on tapes. In general, spanned sets are to be avoided if possible - they're ok for archiving large data sets that need to remain intact, but not needed for CatDV managed files.
In order to restore files with CatDV's plugin, they must already have been restored to the VTAPE by the user first. CatDV is actively working on being able to restore directly from tapes, but they will have to comment on the availability of that feature for what version of their plugin.
Lindsay, the kind of media you are archiving is irrelevant, we are a data device and when it comes to archiving, bits are bits. Because you are getting file transfer errors, I'd suspect your network, your physical connections, your archive source hard drives, your PC configuration... something in your system. Because we have a lot of units deployed, and problems like yours are most likely to be due to such end-user system issues.
Nathan, have you contacted our tech support? We can help you through some of these issues. Are you trying to use LTFS (not tested with the CatDV plugin)? What version is your Cache-A (you should be at v2.1.15)? Is it under warranty (Lindsay's is not)?
Jeff, you seem to be on the right path. I hope some of these explanations help explain issues you've had.
I hope all of you can work through your issues and will be here to address this kind of topic... usually over on the Archiving and Backup forum, but happy to contribute here as well.
602 Park Point Drive
Golden, CO 80401
Wow... thank you Tom for the information, especially about the end user needing to restore the files to the Vtape first... our integrator never told us this and all our frustrations now make sense. This is the reason why we can't get anything to restore properly. In all the demo's we saw of the plug-in the tape was never ejected so the files never left the Vtape... they were just getting recalled from the Vtape when the plug-in called for a restore. That must be why it worked in those demo. settings but not in our "real-world" tests. No wonder the plug-in is saying things like "Can't restore from VTape" while I bang my head on the wall and say "It's on the LTO!"
The information about not using spanned volumes in CatDV is also helpful. Again, I wish we had understood that at the outset so we wouldn't have wasted hours trying to figure out why it wasn't working.
I guess this is why they say information is power... sigh.
Thank again Tom,
Tom, thanks for your input. You brought up several key issues with how we've been archiving.
[Tom Goldberg] "First, the CatDV plugin does not deal with spanned tapes aka Multiple Volumes."
We have been trying to add media from multiple volumes.
[Tom Goldberg] "In order to restore files with CatDV's plugin, they must already have been restored to the VTAPE by the user first."
I simply followed the instructions from Matt Stamos on this YouTube video where there is no manual intervention:
We have been using LTFS but currently are working with a tape that somehow became a .tar format. We have over a TB of "space lost". What do you make of that?
[Tom Goldberg] "have you contacted our tech support? We can help you through some of these issues. What version is your Cache-A (you should be at v2.1.15)?"
We have contacted tech support. Rolf actually gave us a helpful response today which we haven't had a chance to test since we're also having an unrelated archive issue with another company and we've been working with that all day. I will give you an update as to if this fixed some of our problems.
It also looks like we have an old version as we're operating on v2.0.22. I'll update that and see if that helps. Hopefully we will have time to run some test and devote all of our attention to this so I can give a detailed update by the end of week.
I'm sorry that your expectations were not properly set for the interactions between our products. Not much I can do about that except to disclose all the facts to the best of my understanding here. As I've mentioned, I know Rolf is actively working on the restore issue.
As noted in our 2.0.22 release notes, once you format a tape as LTFS, you can't change it back - this explains your "Space Lost" issue. As noted in our 2.1 release notes, this has been fixed, so once you get the update, you will be able to fix that tape - format it back to LTFS and then back to tar again to get all the space back.
We'll need your serial number to schedule that software update for your machine - send me an email with that info or simply fill out the Software Update request form on our web site (http://cache-a.com/uprequest.php)
602 Park Point Drive
Golden, CO 80401
Ok... we are SO confused.
We just did a test and CatDV Cache-A WAS able to recall files from the LTO tape... we deleted the VTape, called for a restore and watched the Cache-A seek up to the files, copy them to the Vtape and then CatDV Cache-A plug-in copied them to our SAN. So, Tom, it seems that this is working after all?
However, some of the footage we wrote to the tapes is missing... LARGE portions and we don't understand why. We wrote in two sessions and only some of the footage from the FIRST session is on the tape.
Tom, can we not write multiple sessions to the same LTO? That poses a big problem if it's true.
I just wanted to post an update after we did more tests today.
Here's what we did:
1) With CatDV Cache-A plugin wrote a catalog of 91 files (30 GB) to an LTO tape. No logged errors or other problems.
2) We ejected this tape and cleared the vtape.
3) Re-inserted the tape and then restored the files with the CatDV Cache-A plugin. PROBLEM. Six files were 1/2 or less of their original size. CatDV could not play them. Couldn't play them in the OS. They are incomplete and / or corrupt. BLARG.
4) We compared the problem files (now on our SAN) against the files on the vtape. THE FILES ON THE VTAPE WERE FINE... wow. So the problem came when CatDV Cache-A plugin copied them from the vtape back to the SAN... WHY?? That seems like a relatively simple operation.
5) We deleted the problematic files from the SAN and the vtape and then called for a restore of just these files through the CatDV Cache-A plugin. They came back from the LTO to the vtape to the SAN without any issue. We are perplexed.
6) We started over... we deleted all the files from this catalog on the SAN and the Vtape. With the CatDV Cache-A plugin we called for a restore of all the files. They were all pulled from the LTO tape to the vtape to the SAN without any issue... HEAD SCRATCHING ENSUES. Why did this not work the first time and now it was flawless?
We move on to seeing if we can write multiple sessions to the SAME tape (NOT spanning tapes). It works only if you NEVER EJECT THE TAPE. If you do eject the tape and then try to write more files to tape in a later session it will overwrite whatever is on the tape... so that means you have to fill the tape (if you want your money's worth) without ever ejecting it.
My question is, is that normal operation or is this some sort of problem with CatDV Cache-A plugin? Any thoughts Tom?
Thanks so much for the extensive testing!
I sure hope that either Rolf or Tom can answer your question, "can you perform multiple sessions to the same tape, eject, erase, reinsert tape and perform restore via CATDV?" This in my mind would be a deal breaker as we would be archiving projects on a daily basis with the chance of needing to restore from a different tape at any given moment.
Let's see how the day pans out.
Okay, so I realize I've been told on a few occasions now by different sources that file/media type shouldn't matter, but I decided to do a bit of testing anyways.
I took a single clip from each of our media types and converted each of them into 8 different formats with a QT wrapper. The formats ranged from 'Animation' codec to basic DV NTSC. I also threw an AVI and H264 example in the mix as well. I then imported these files,which are sitting in an isolated location on our raid drive, into a blank CatDV catalog. Next I fired up the CacheA, re-initialized a tape I have been testing on, configured the CatDV plug-in to point to the correct VTAPE, highlighted a portion of the clips no more than 20 to 30 GB at a time, and started the archive using the CatDV plug-in. Not a single one of these files transferred without error 'Resource temporarily unavailable'.
I moved a few test files over to our C-drive to try an archive from there and eliminate the Raid as a factor, but had the same error filled transfer result.
At this point I feel there has to be something funky going on with my CatDV software/plug-in. Rolf, CatDV, can you guys chime in? I'll try you through your formal support contact as well.
Thank you for clarifying the restore procedure for CatDV. I will be sure to follow that process, once I have an error-free, successfully archived a catalog.
As for the issues that seem to specific to our system, at this point I am at a bit of a loss. I have also been working with our IT specialists on everything from examining permissions on all related hard drives to our network connections, including testing different Ethernet cables.
I will continue archiving directly to the CacheA using the staged transfer method for now, and fingers crossed perhaps will have a new revelation or idea to test out. I am open to any suggestions you might have.
Sorry to hear about the problems people here are experiencing with the archive plug-in.
It's true that spanned volumes are not supported.
What Tom says about restoring manually is not quite true however, as CatDV 9 will ask the Cache-A to restore files from tape to vtape. For this to work you need a reasonably up to date version of both CatDV and Cache-A firmware (within the past 6 months say), and need to enter the hostname of the Cache-A device, not just the vtape share location. We have recently learned that this may not always work realiably however: it seems that the request to restore files might sometimes not return as quickly as we thought, so we may need to increase an internal timeout value to prevent errors when issuing the restore command.
We are also aware of an issue specific to the Mac that can cause problems, both when archiving and restoring. If the Cache-A volume goes offline while using CatDV that CatDV might in some situations then create a folder on your local drive called /Volumes/Cache-A , which means that when the Cache-A is next mounted it will get the name /Volumes/Cache-A-1. To see if this has happened to you, use Finder > Go > Go To Folder and type in /Volumes. If you unmount the Cache-A and still have a folder by that name, move that folder to the trash (first checking to see if it contains any "archived" files!), then remount the Cache-A and reselect the vtape folder in the CatDV archive config settings. CatDV is supposed to protect against that happening so we're trying to find out under precisely which circumstances one of these phantom folders can appear.
We are investigating both these issues and will issue an updated version of CatDV shortly with any changes as necessary.
Regarding not being able to archive to a tape, eject it, load it again, and add more files to it I'm not aware of anything in CatDV that would stop that working, so that may be a Cache-A issue. Can you add more files to an existing tape if you use drag and drop from the Finder (or Explorer)?
[Rolf Howarth] "it seems that the request to restore files might sometimes not return as quickly as we thought, so we may need to increase an internal timeout value to prevent errors when issuing the restore command."
YES. This happened to us the yesterday. We called for a restore and Cache-A restored everything to the vtape perfectly but after we got a timeout error message for CatDV and nothing was restored to our SAN. We repeated the request with the plugin and then CatDV pulled everything from the vtape without issue.
[Rolf Howarth] ". If the Cache-A volume goes offline while using CatDV that CatDV might in some situations then create a folder on your local drive called /Volumes/Cache-A , which means that when the Cache-A is next mounted it will get the name /Volumes/Cache-A-1."
Yes again. We just found out this happened to us. We've had one station that was using the CatDV plugin fine and the all of a sudden would not archive files any longer. Every time we tried it would say "Unable to create directory [target directory]". We just went and checked and sure enough the phantom folder was there. I think Nathan was getting this same error.
[Rolf Howarth] "Regarding not being able to archive to a tape, eject it, load it again, and add more files to it I'm not aware of anything in CatDV that would stop that working, so that may be a Cache-A issue. Can you add more files to an existing tape if you use drag and drop from the Finder (or Explorer)?"
We haven't up to this point. When dropping stuff directly into the Cache-A (without the CatDV plugin) we have only done it in single sessions. But I will try a multi-session (ejecting the tape in between write sessions) and see what I get.
Rolf, is there a difference between a software update from Cache-A and the firmware update you are talking about? Tom, how do we know if we have the latest Cache-A firmware (we have software version 2.1.15)? How do we request a firmware update if we need it? We bought our Cache-A in July 2010.
Rolf and Tom thank you so much for dialoging with us about these issues.
[Jeff Schaap] "Rolf, is there a difference between a software update from Cache-A and the firmware update you are talking about? Tom, how do we know if we have the latest Cache-A firmware (we have software version 2.1.15)? How do we request a firmware update if we need it? We bought our Cache-A in July 2010."
Sorry yes, I meant a Cache-A software update. I believe 2.1.15 is the current version.
Right 2.1.15 is the latest and greatest. You can always check for the status of all our releases on our support page and just click on the SW update request button to get an update scheduled for your serial number.
Warranty or not, we've always provided software updates for free.
That said... a heads up, we are going to v3.0 soon and that will be a paid upgrade, pricing TBD. For 3.0 we'll be moving to CentOS6.2 64 bit and it will be distributed on a flash drive. Keep an eye on our support page for this announcement and details.
602 Park Point Drive
Golden, CO 80401
Just wanted to touch base here as I haven't received a response from the last email I sent you including the system specs you requested. Perhaps my email was caught in your spam filter? I don't mean to seem pushy, I am just eager for some feedback and to continue moving in a positive direction to get our plug-in working.
I am happy to provide any and all details you may feel to be of importance. I can even provide step by step screen shots etc. Just let me know how I can help facilitate this. At this point I'm stumped and out of ideas (at least for today) of what other factors to test in terms of isolating an issue specific to our system. Could an uninstall/re-install of CatDV have any affect?
Thanks again for your patience and feedback,
Sorry, I thought I did reply (see below). We plan to increase the timeout in the next release.
[Rolf Howarth] "Using the Cache-A API, CatDV queues a list of files to be restored one by one. Normally adding individual files returns very quickly, so there's a timeout of 5sec, but it seems that occasionally the Cache-A doesn't respond in time so it times out. We'll increase the timeout in the next release of CatDV but for now you should be able to wait a little while then repeat the restore command until it successfully restores all the files. Apologies for the inconvenience."
Sorry for the confusion, but when I asked about your reply, I was referring to the email correspondence we had begun specifically regarding my systems inability to archive anything from the CatDV plug-in. I responded to your email with all of the information you requested, which I have copied below, and have yet to receive a response in that correspondence.
"Cache-A Version: 2.1.14
CatDV Pro 9.0.6
Windows 7 Ultimate
The Cache-A is only mounted on one machine and it is a NAS mounted drive with 1 Gb/s speed. The source media that is being pushed to the vtape for archive is sitting on a RAID-0 array with an SAS connection.
I am not playing media from the vtape, but yes I am simply using the vtape as the appropriate conduit to archive files to tape.
Yes, it consistently archives okay using the Finder if I first copy files to a CacheA share, internal HD, and then move the files into the vtape. This is the only way I have performed a successful, error free archive. I have documented that if files copied to the Cache share, internal HD, are copied and pasted to the vtape rather than dragged and dropped or moved, I received the same 'resource temporarily unavailable' error.
Yes, it fails every attempt I have made in CatDV, which I agree is very strange. "
We did more testing this past week. Rolf, we do seem plagued by the time out problem you mentioned. Here's what we did:
1) We wrote 650 files, about a terabyte's worth, to LTO through CatDV without any issue.
2) We ejected the tape and deleted the vtape then reinserted it. We called for a restore of all 650 files. We got a time out error (after about 2 minutes). We tried to restore one file and that came back fine. Left for the weekend.
3) Monday, we call for a restore of the 650 files. After about a ten minute pause it took off with the restore without issue. However, it only restored 251 of the 650 files and CatDV said it was "unable to restore" the rest. They were also not on the vtape.
4) We call for another restore. It skips the ones it already has done and recalls the rest from tape without issue. So it works, sorta.
There seems to be some sort of issue with CatDV timing out when there is a large amount of files to be restored. Rolf, what exactly is happening at this stage that is causing the system to pause so long? Is it building and transmitting a list of files to restore to the Cache-A?
We are performing the multiple write to the same tape tests right now (without using CatDV) and we'll let you know what happens there.
[Jeff Schaap] "There seems to be some sort of issue with CatDV timing out when there is a large amount of files to be restored. Rolf, what exactly is happening at this stage that is causing the system to pause so long? Is it building and transmitting a list of files to restore to the Cache-A?"
Using the Cache-A API, CatDV queues a list of files to be restored one by one. Normally adding individual files returns very quickly, so there's a timeout of 5sec, but it seems that occasionally the Cache-A doesn't respond in time so it times out. We'll increase the timeout in the next release of CatDV but for now you should be able to wait a little while then repeat the restore command until it successfully restores all the files. Apologies for the inconvenience.
[Rolf Howarth] "Normally adding individual files returns very quickly, so there's a timeout of 5sec"
So, if there is small delay on the transmission of each file then it wouldn't be abnormal for it to take five or more minutes to transmit 650 "file restore requests" right? We thought maybe something was wrong when we were getting such long pauses. It would be great to have a progress bar or something to show that it's working.
Can you say when the next version will be available?
Thanks for your help,
I have the results from our attempts to do multiple write sessions to the same tape using just the Cache-A interface. Here's what happened:
1) We copied a few files to the staging folder and then dropped them into the vtape. Ejected the tape, clearing the vtape and then re-insterted the tape. Wrote two more sessions (3 total) repeating this procedure of write, eject, erase vtape, reinsert tape.
2) We called for a restore all file from the tape to the vtape. Cache-A went into a pending status and never did anything. Eventually we had to restart the Cache-A.
3) After a restart the Cache-A said the vtape and and tape were not a match. It only offered three options and selected "eject the tape." Then we re-inserted the tape and called for a restore of all files and they all came back without error.
So, Rolf, with some caveats, it does appear that the system can recall files from a multiple session write without issue. So I am left to wonder why this isn't working when we do it with CatDV. Have any thoughts on it?
Just a note, not sure why the Cache-A would have hung at pending, but if that kind of thing happens, restarting is undesirable and slow - you are better off going to System Tools page > Utilities tab > Restart Tape Manager button - much quicker and safer.
602 Park Point Drive
Golden, CO 80401
[Lindsay Simpson] "
Nate, you had mentioned looking at file size by hovering over a file in the CacheA UI correct?"
You must hover over the file size to get the uncompressed file size (figure 01) I haven't noticed the media directory being larger at the end of transfer. I'll watch for that.
[Lindsay Simpson] "Nate, what type of media are you archiving through CatDV? Does your production house convert all media upon return from a shoot out of it's native format?"
We do sometimes get footage from outside sources that has been transcoded but mostly it's our native format and we shoot ProRes. We also have some DVCPROHD footage. Coming up we have an XDCam shoot and am contemplating purchasing the MXF option so we can ingest that directly.
[Lindsay Simpson] "Forgive my ignorance, but is worker node a feature only accessible if you have a server-based CatDV system?"
The server plays a vital role in our workflow because we have CatDV on ten computers as well as two Web Client licenses. We save all of our catalogs to the server so any of the computers can access them. Worker node, I believe, is only available for server-based systems, as we have it on a Windows server (although all our licenses are on Macs). You can send commands to make proxies through Worker Node from any of our systems.
Ah yes, I understand now. Thanks for clarifying Nate.