FORUMS: list search recent posts

Automatic Scanning for New or Moved Files

COW Forums : Square Box CatDV

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Ernesto Sanchez
Automatic Scanning for New or Moved Files
on Mar 18, 2014 at 5:13:09 pm

We're evaluating CatDV and have a question on how it handles new files and files that are moved after the initial scan. Does CatDV keep track of files as they move or is the only way it knows about a move is to run a new scan. We are testing CatDV on a RAID with 46TB of data.

Also looking into Axle Video's product. They say it does it "will index all the files and update its database in real time as new files are added or removed. Can CatDV to this?

Any info/advise is appreciated.


Cheers,
Ernesto Sanchez
Director of Technology
Toy Box Entertainment


Return to posts index

bryson jones
Re: Automatic Scanning for New or Moved Files
on Mar 19, 2014 at 3:01:26 am

Define "real time" for 46TB, right?

CatDV does not do this, neither does any other sub-$10,000 product, at scale.

Good to see you on here, man!

bryson

bryson "at" northshoreautomation.com

northshoreautomation.com


Return to posts index

Ernesto Sanchez
Re: Automatic Scanning for New or Moved Files
on Mar 19, 2014 at 3:16:33 pm

Hi Bryson,

Say we just cataloged one 2TB project (folder) on this RAID. Within this project files are organized in various subfolders in addition to the new metadata we added via CatDV. Does CatDV loose track If someone accidentally moves files around?


Cheers,
Ernesto


Return to posts index


bryson jones
Re: Automatic Scanning for New or Moved Files
on Mar 19, 2014 at 4:19:58 pm

Yes, just like any path based non-linear editor, CatDV would see those assets as offline.

I'm not sure that any DAM would really be able to track those reliably. I'm excited to see what others are doing about file moves, but I'm cautious about that in the real world. Here are my questions about this:

How many files can a system track in real time?
How do updates and duplications get resolved. (Does the system decide what is a dupe and delete one? This seems strange/scary to me.)
How are temporary offline situations resolved? (We've seen another brand DAM system be set in a way that a short period of the storage being offline caused a re-scan of 2,000,000 files with new thumbnails being pulled etc. Yes, this took days.)

These are all real concerns and I've simply not heard of enough people doing this type of workflow to answer. Do you have a demo running yet?

CatDV can watch an area, like you describe, and ingest assets. In most of the North Shore deployments, we like to have CatDV Worker ingest the assets and then move them to a fixed structure that is under admin control. Then people can access them read-only or call a move from a Worker action so that the system tracks files and delivers it to the new location, if copying isn't appropriate. You can leave them in place but it can get a little confusing as to what to ingest and what not to. The node-based Worker gains a ton of power from being a separate app (You can have 100 of them ingesting if you wanted to in theory) but it not being tied directly to the server at all times does have limitations.

In my training and consulting I use the term, "the myth of editorial asset management." That just means that humans will always outsmart a computer when it comes to tracking assets (and moving/losing them.) I find that handling the assets pre and post edit, is a far more successful approach and then letting the editors edit from the shared storage as needed, which is what they tend to do no matter what you tell them. ;) If they need to change the assets, then of course a copy is made and re-entered as a new asset, (which it would of course be, once changed.)

There are so many issues with letting a user move assets and then expecting a DAM/MAM to "find" it. The biggest is latency. While the DAM engine is scanning or doing some other operation on another part of the system, what if a user moves a file? How long until the system shows that change? As we all know, "instant" isn't really likely in a decent sized file system. (Heck, some SAN's don't even refresh the filesystem that fast unless you do it manually.)

Until we can tie DAMs directly to the filesystem I'm not sure how well that would work. Clients/users often hate to hear this.

Let us know the results of your testing, there are many ways to tackle this one, but I would think the answer is a workflow modification if you want to use CatDV in this environment.

bryson

bryson "at" northshoreautomation.com

northshoreautomation.com


Return to posts index

Scott Goddard
Re: Automatic Scanning for New or Moved Files
on Mar 21, 2014 at 12:23:17 pm

CatDV can be used in many ways throughout the production workflow. When it comes to archiving though, you really need to have a strong file naming and folder structure for your back end. This will keep you in control of the assets and will build a foundation to allow you to access this material 5-10-50 years down the line, which in my opinion is really what it's all about.

It's worth taking the extra time here.

Some of the workflows that I use involve having seperate stages of production, such as a working catalog and an archiving catalog. Depending on the workflow you employ, this may help you decide when and how to move those files in to differnt filenames and folder structures. CatDV has some very useful Move and Rename tools.

Scott Goddard

Neo Verite Limited
http://neoverite.com


Return to posts index

John Heagy
Re: Automatic Scanning for New or Moved Files
on Mar 31, 2014 at 10:10:01 pm

This is similar to my question about determining whether a file is online.

I would like to see a command that can check if a file is online on a file by file or catalog by catalog basis. I would never what a storage wide scan. Most processes can be narrowed down to a file or a catalog so I thing checking for online status at that level would work for most.

John


Return to posts index


bryson jones
Re: Automatic Scanning for New or Moved Files
on Mar 31, 2014 at 11:24:50 pm

This is not really an issue of "can it be done" in regards to checking a file's status at a given moment in time to a given path. That's easily scriptable, as a "check online status" action.

However, what that status means is the main issue. CatDV's path mapping and flexibility makes it difficult to say what "online" means to all users at all paths at any given time.

That said, if you want an action to check for a file in a given path, then update a field stating that status as of a date and time, that can certainly be done now.

Check file in path

Write "online status"

Write date and time of check

When and how often this happens is where your choice comes in, as well as the speed of checking a large set of files.

Let us know if there's widespread interest in this sort of status check. This could be offered as a middleware product I'm sure with the above stated workflow restrictions.

bryson

bryson "at" northshoreautomation.com

northshoreautomation.com


Return to posts index

John Heagy
Re: Automatic Scanning for New or Moved Files
on Apr 2, 2014 at 1:46:38 pm

Well there's a "widespread" need here ;)

We are planning a restore workflow where the restored files do not go back to there original location but to a "restore" location that is very transient. We plan on using CatDV to catalog these files and communicate what is there. Since these files are regularly deleted and never restored it would be useful for CatDV too actually delete the files from the catalog when they are deleted from the restore location, Otherwise we will need some sort or online status for catalog/s that are full of deleted files.

John


Return to posts index

bryson jones
Re: Automatic Scanning for New or Moved Files
on Apr 2, 2014 at 5:35:09 pm

So the workflow is:

Restored "temp" files delivered to folder and temp records created in CatDV, new catalog.

Editors use files and when done delete or flag for deletion (perhaps an auto delete based on time?)

Then the idea is that you would check the status of a "temp" file before re-restoring it?

If that's the case, then I'd assume the archive device would check that. If not, then draw up the various scenarios and this could likely be implemented via the web API.

On that note, if you all saw the "Automation API" mention in the Squarebox NAB release, that's something we've been discussing for a while. I'm going to post a bit about what that is next week, but basically we're seeing the 1 to 2 second time for Worker Node actions reduced to just a few milliseconds for updates done through the Web API. Everything CatDV automates can be faster now.

bryson

bryson "at" northshoreautomation.com

northshoreautomation.com


Return to posts index


John Heagy
Re: Automatic Scanning for New or Moved Files
on Apr 3, 2014 at 8:46:43 pm

[bryson jones] "Then the idea is that you would check the status of a "temp" file before re-restoring it?
"


We would never restore the "temp" file but simply delete it. Once deleted it would ideally be deleted from the catalog. The alternative would be to create a new catalog very 1/2hr and delete all previous. We would use CatDV to tell us "what's there" and report the metadata to our in house linking app.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]