CatDV to Cache-A archive inconsistencies
I have been establishing our CatDV/Cache-A archive system over the past few months, and I think I have good handle on the best workflow etc. I have recently upgraded to CatDV Pro 9 with Archive and MXF options. The upgrade has solved one particular issue I was having with my JVC MXF (equivalent to XDCAM format) media, but I am still experiencing inconsistencies in the actual archive to data tape. More on the specific issue in a bit, but first a little information on the system I have established to track my archives as I go.
I have established a spreadsheet data base system to track which original media (2 Different types)has been logged and sorted into category specific catalogs. I also have another spreadsheet tracking the category catalogs themselves, when a catalog is first archived to CacheA, which LTO tape it was archived to, and eventually any dates on which the archive is updated.
After lots of trial and error, back and forth between CacheA and CatDV support (both seem to playing a bit of a blame game) I still don't have a successful transfer from a CatDV catalog to CacheA tape as every transfer I've attempted results in errors logged in the CacheA transfer log, which CacheA indicates are never good.
I have carefully archived media using the command provided in CatDV with one category catalog at a time or if the catalog is large, in chunks of 20 to 30 GB within a catalog. During these archives, I do nothing else with the computer and wait until the CacheA browser based user interface indicates a successful transfer in the transfer log before attempting anything else.
The two media types in question are P2 MXF and JVC/XDCAM MXF which are both imported into the catalogs as 'Metaclips' not 'movies' to ensure all of the MXF data is packaged into a single clip and can be successfully restored after dumping to the CacheA.
After upgrading to CatDV 9, I erased all of my LTO tapes and started fresh as much of what I had already archived all contained transfer log errors. (It's a long story, but to a degree I had been ignoring the errors during the first archive transfers because CacheA had indicated another issue with the CacheA UI I was experiencing was not something to worry about) CacheA indicated that the CatDV upgrade should clear that up, but upon my first post-upgrade test the same errors were replicated, specifically an issue with P2 media.
The transfer logs seem to indicate that while some P2 media in a particular directory transfer successfully other metaclips result in a transfer error that indicates the media is not accessible on the disk. I've attached the transfer logs of my most recent catalog archives. I have touched base with CacheA on this matter again, but am still waiting on a response after they deemed it necessary to perform a support connect.
I realize this is a CatDV forum, but perhaps folks out there are also using CatDV to catalog and archive their media via CacheA and hopefully have some insight.
Any thoughts or suggestions are sincerely appreciated. And if you require more information, please let me know I am happy to provide. This issue is one of many I have experienced over a period of months as I work to establish a manageable, efficient and organized catalog/archive system for the long term.
Here are links to the logs from the recent transfer:
I'm not sure what a Cache-A "Resource temporarily unavailable" message means but try breaking things down to see at what stage it goes wrong.
When you archive files to the Cache-A from CatDV what it does it copy the files to the Cache-a VTAPE volume, then wait for the Cache-A device to automatically copy them to tape. First, check that the files are indeed copied to the vtape. Then, try doing an archive operation just using the Finder without using CatDV. Copy an entire P2 or JVC folder structure to the vtape and see if that is correctly archived or also gives you errors.
Thanks for your reply. As for the transfer of media CatDV > CacheA the media begins to transfer from VTAPE to tape almost instantly once the archive process has been initiated in CatDV. Generally the VTAPE to tape transfer does not complete until a few minutes after the CatDV transfer to the VTAPE is complete.
The error message 'Resource temporarily unavailable' at some level implied to me that perhaps there was a permissions issue occurring with the Raid where the raw media sits. I have checked and double checked this and find no issue in regards to permissions.
As for dumping P2/JVC folders directly to the VTAPE for archive, I have not tried it yet with those two particular media types, but I will report back when I do. I have successfully performed a direct VTAPE dump of an Ikegami MXF media directory, as for now I am not ingesting that media into CatDV as it can not be read as a Metaclip, at least on a PC based system, which we have discussed previously.Again, I will report back with findings from direct VTAPE transfers of P2 and JVC today or tomorrow, including transfer logs.
Thank you and have a great day,
I have sent this issue directly to CatDV Support, but will relay here on CreativeCow as well.
I have finally successfully performed a direct archive of all 3 of the MXF media types I am working with (P2, JVC & Ikegami). This was done through a staged transfer process by copying an individual directory folder directly to the CacheA internal HDD, not directly into the VTAPE, but into a staging folder. From the staging folder, if a media directory was copy/pasted into the VTAPE folder, the transfer would result in random, non-repeating errors (noted after multiple tests of the same folder). But if the media directory to be archived was dragged from the staging folder and dropped into the VTAPE folder the transfer completes successfully to tape without error. I have also successfully restored media archived using this direct, staged transfer.
After completing the direct transfer with confidence, I created a new CatDV catalog and 'Import Directory' to bring in one of the media directories that had been used during direct testing, specifically a P2 directory titled 'Alternative_Enterprises.' The directory imported into the catalog successfully as metaclips. I then used the 'Archive Media' tool and let the process run to completion as noted by the end of a session in the transfer log in the Cache-A UI. The transfer still produces a transfer log with errors, and when I attempted to restore the media I discovered that parts of the MXF directory are not being archived. Please review screen shots for details/visuals. The errors still seem to indicate a permissions issue, but Raid serving as main storage for the media directories being archived is open and accessible with no permissions restrictions. Perhaps I am missing a setting inside CatDV or Cache-A? Or perhaps it is something else?
Thanks for any and all input,
It's true that CatDV won't archive the LASTCLIP.TXT file from a P2 card. By default, when you import a folder into CatDV it only creates clips for media files it recognises. In general an arbitrary folder could contains all sorts of irrelevant files - backups, temporary files, hidden files, non-media files etc. - that simply aren't relevant to a catalog of media assets. If you select a number of clips and archive them it will only copy the files directly used by those clips.
Thank you for clarifying Rolf, but how then do I successfully archive the MXF media through the CatDV interface, as it seems the text files such as 'LASTCLIP.txt' and even empty folders within an MXF directory are integral to restoring the media structure properly?
Today I was introduced to the concept of a CatDV worker node (I believe that's what it was called, could be remembering this wrong) to transcode media out of MXF. I have skimmed through my CatDV 8 manual (I don't have the v.9 manual printed off yet)and haven't come across that term specifically. Can you point me to some informational resources to get my head wrapped around this process within CatDV?
Well did a bit of digging on my own, and the worker node is out of reach for us right now. We have a budget of zero to make any purchases. I suppose I held a misconception that the MXF option of CatDV would be enough to facilitate the Cache-A archive function properly with that file type. Back to the workflow drawing board I suppose. Finding a balance between the great ability to work natively with all of our MXF media in the Adobe production suite and get it to archive using the CatDV catalog system is proving challenging. Right now I am struggling to imagine how to efficiently convert the massive library of existing MXF media we have to a more, friendly universal format that is expected to have some longevity, without making another purchase of something like Worker Node which can directed to perform such processes in the background or after hours.
Is anyone aware of a method to set up scheduled conversion of media using Adobe Media Encoder?
That's a good point about the empty folders. We don't actively track empty folders in our catalog but could probably restore them in the specific case of restoring a P2 volume.