FORUMS: list search recent posts

WorkerNode command line script problem

COW Forums : Square Box CatDV

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Jeff Schaap
WorkerNode command line script problem
on Oct 7, 2013 at 4:00:50 pm

In processing proxies for AVCHD .mts files I have the following command in a worker node server query under the "Execute command 1":

mv /Volumes/maxxraid1/Media_Encoder_Watch/Output/$g.mov /Volumes/maxxraid1/CatDV_Previews$p/

This works perfectly to move proxies I've generated to their proper location. However, it all falls apart if there are spaces in the name or directories. Ugh. I've tried many things including:

mv "/Volumes/maxxraid1/Media_Encoder_Watch/Output/$g.mov" "/Volumes/maxxraid1/CatDV_Previews$p/"

But I get an exit status of 1 from the mv command in worknode (which means general error). I am guessing my variables are being taken as characters because I am enclosing them within the quotes but I am not sure how to get around that because I need them for the spaces in the file and directory names. Anyone know how this should be written?


Return to posts index

Tim Jones
Re: WorkerNode command line script problem
on Oct 7, 2013 at 4:56:30 pm

Just a shot in the dark here, but what if you escape the $ in the variable:

mv "/Volumes/maxxraid1/Media_Encoder_Watch/Output/$g.mov" "/Volumes/maxxraid1/CatDV_Previews$p/"


Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.productionbackup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

Jeff Schaap
Re: WorkerNode command line script problem
on Oct 7, 2013 at 6:08:23 pm

That looks exactly the same as the command line I had posted that I already tried. Or am I am missing something?


Return to posts index


Tim Jones
Re: WorkerNode command line script problem
on Oct 7, 2013 at 6:30:50 pm

Ugh - HTML "fixing". There should be a backslash before each of the $ characters.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.productionbackup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

Jeff Schaap
Re: WorkerNode command line script problem
on Oct 7, 2013 at 8:14:24 pm

If you pretend my #'s are backslashes then this is what I did:

mv "/Volumes/maxxraid1/Media_Encoder_Watch/Output/#$g.mov" "/Volumes/maxxraid1/CatDV_Previews#$p/"

and unfortunately, it returned exit status 1, so, thanks for the idea but that didn't solve it.


Return to posts index

bryson jones
Re: WorkerNode command line script problem
on Oct 8, 2013 at 12:07:37 am

Jeff, the Worker is full of dark mysteries.

We use single quotes, or "ticks" to escape spaces etc.

Don't change the variables, there's no need to escape the "$g" or whatever, that's not your issue.

We find that more advanced functions are best served by taking things out to a shell or Python script where you can get some control over things.

Hope this helps you a bit. Make sure you reach out to your dealer or Squarebox support, they can offer advice with a clear understanding of your environment and systems. I'm just some guy with a beard and an internet connection. ;)

bryson

bryson "at" northshoreautomation.com

northshoreautomation.com


Return to posts index


Jeff Schaap
Re: WorkerNode command line script problem
on Oct 8, 2013 at 1:50:42 pm

Hey Bryson,

Your beard is awesome.

Thanks for the input, yes there is some deep mystery in WorkerNode at times...

Replacing the " with ' gave exit status 1. I would imagine because a '$g' is interpreted literally as $g and not a variable.

It seems crazy that I have to write a script to do one mv operation... but at this point I am willing to try almost anything. From a terminal window, when I set g equal to the correct file (g = "My File" and p the the destination path (p = "/Volumes/maxxraid1/Footage/Domestic/Personnel/Creson Bob" and do this:

mv /Volumes/maxxraid1/Media_Encoder_Watch/Output/"$g".mov /Volumes/CatDV_Previews/"$p"/

It works and moves the file. Even with the spaces on the name and path. I was thrilled and figured I would just write a script that looked like this:

#!/bin/sh
mv /Volumes/maxxraid1/Media_Encoder_Watch/Output/"$g".mov /Volumes/CatDV_Previews/"$p"/

and I called it from Worknode by doing this under Command 1:

/Volumes/maxxraid1/Scripts/move_proxies.sh

But, that didn't work because I later found out that my variable weren't being passed to the shell. So I did this under command 1:

/Volumes/maxxraid1/Scripts/move_proxies.sh $g $p

and changed my script to:

mv /Volumes/maxxraid1/Media_Encoder_Watch/Output/"$1".mov /Volumes/CatDV_Previews/"$2"/

And exit staus 1... ARGH. Not sure what to try next. I can contact support as you suggested.


Return to posts index

bryson jones
Re: WorkerNode command line script problem
on Oct 9, 2013 at 3:08:52 pm

I think I see what's up here.

This is not well documented in the Worker Read Me. We've spent forever dialing this in and basically there are syntax rules, such as calling bash or sh when you enter commands. I imagine that's what you're missing.

Hit up support and I'm sure they can sort you out.

They don't doc this part of things deeply and I'm pretty sure it's because they traditionally did the heavier deployments themselves or with a known team of dealers.

bryson

bryson "at" northshoreautomation.com

northshoreautomation.com


Return to posts index

Jeff Schaap
Re: WorkerNode command line script problem
on Oct 10, 2013 at 12:48:29 pm

Thanks Tim and Bryson for your responses. I did figure out what was going on and unfortunately, it had nothing to do with spaces in the filenames or paths and my original mv command was working properly. Sorry for the wild goose chase.

The issue is a Mac/PC environment. We began as an all Mac environment but now are a mix of PC and Mac. Files that were ingested on a Mac are getting paths like /Volumes/maxxraid1/Footage/... and the same file, when ingested from a PC gets Y:maxxraid1Footage... and herein lies the problem.

For operations within a client I can set up original media search paths to account for the fact that files maybe located under /Volumes/... or as Y:.... That works great. But in WorkNode, it's a problem. Apparently it doesn't (can't?) use that same information to search for original media when running jobs.

So, when I have files that were ingested over a PC and the paths start with Y: you can understand why, when executing a mv command in a Mac environment, it becomes a problem. The mv command can't find the file because it's being referenced from a PC mapping, the path is slightly different and the slashes within the path are reversed.

I also noticed that variables change...(figured this out from status emails I had the Workernode send me). $g is supposed to be the base part of the file name (i.e. my_clip rather than my_clip.mov). But when dealing with a file ingested from a PC this variable becomes the WHOLE path and the file name (without extension):

$g from Mac ingest file = IMCFTG3471_Creson Bob and Dallas_0002

$g from PC ingested file = Y:FootageDomesticWycliffe PersonnelCreson Bob and DallasIMCFTG3471_Creson Bob and Dallas_002

I don't get that at all... is this a bug??

It probably is time to talk to support but I wondered if I might be missing some simple thing that would make this work. I can't be the only person with this problem.


Return to posts index


Tim Jones
Re: WorkerNode command line script problem
on Oct 10, 2013 at 3:31:28 pm

Hi Jeff,

A very thorough bit of research on your part. I did a bit of experimentation on this and have submitted a feature request to the Square Box team to handle these situations a bit more logically.

My recommendation is to internally store everything with forward slashes and prepend a forward slash on drive letters from Windows systems. For example:

F:\A Folder\Some Path\A Movie File.mov

becomes

/F:/A Folder/Some Path/A Movie File.mov

The leading slash could then be dropped and the forward slashes would be converted to back slashes for human-readable display on Windows.

This would allow Server and Worker Node scripts to work on any platform regardless of the host platform of the media file(s).

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.productionbackup.com
BRU ... because it's the RESTORE that matters!


Return to posts index

bryson jones
Re: WorkerNode command line script problem
on Oct 10, 2013 at 4:22:01 pm

Back to my "dark art" comment, there's a lot to consider when you start to automate. Great work for digging in like that.

Bigger issue, it's a best practice to use UNC paths in CatDV and Worker. Mapped drives can create issues with Quicktime (google that for some fun) and also are more confusing for writing scripts.

In defense of the system, not many systems (any?) will dynamically allow you to send a command based on a Mac or PC path being auto sensed and it's fairly easy to have 2 scripts that run depending on the root of the path if you get outside of the Worker itself.

Great work and let us know how you do with the mts files. (an even bigger mess) We can't seem to get them to span correctly or carry TC when we just raw process them via a watch. Has Adobe sorted this yet?

If so, we'll get to work rolling out some CLI scripts for Media Encoder so people don't have to drop and move files.

bryson

bryson "at" northshoreautomation.com

northshoreautomation.com


Return to posts index

Jeff Schaap
Re: WorkerNode command line script problem
on Oct 11, 2013 at 2:51:14 pm

Hey Bryson, thanks for the response. Our solution, for the time being, is to just ingest via a Mac workstation. This way, all the paths are usable by WorkerNode and we move on with life.

[bryson jones] "
Bigger issue, it's a best practice to use UNC paths in CatDV and Worker. Mapped drives can create issues with Quicktime (google that for some fun) and also are more confusing for writing scripts."


I am not sure how that is possible in a Mac environment (our CatDV server is a Mac) since UNC paths would be misinterpreted by WorkNode as well. Maybe I am misunderstanding something.

[bryson jones] "In defense of the system, not many systems (any?) will dynamically allow you to send a command based on a Mac or PC path being auto sensed and it's fairly easy to have 2 scripts that run depending on the root of the path if you get outside of the Worker itself."

I am not intending to bad mouth CatDV. It has some great features. It has some shortcomings. What would be great, for our purposes, is if the "original media path" search behavior in the client extended to WorkNode. Maybe there is some great reason it doesn't but, in a mixed environment like ours it's an issue.

I considered the "two scripts" idea but.. yikes. It would be easy enough to detect if the path was from a Mac or PC (you could do that within WorkNode's interface I believe) but quite a scripting feat to convert a PC path to the correct corresponding Mac path. At least, for me. While I've written some python in the past I don't have the time or inclination to spend the hours it would take me to figure it out. Deadlines are always looming. Much easier to just ingest via Mac for now.

[bryson jones] "Great work and let us know how you do with the mts files. (an even bigger mess) We can't seem to get them to span correctly or carry TC when we just raw process them via a watch. Has Adobe sorted this yet?

If so, we'll get to work rolling out some CLI scripts for Media Encoder so people don't have to drop and move files"


Yes.. the AVCHD conundrum is at the heart of this whole problem. If CatDV could play AVCHD natively I think I would cry. Haha. When pulling .MTS stuff into Premiere I don't see timecode and it isn't adjoining clips that are technically all one recording. But, it's of little issue for us. I guess some people might need that functionality.

When we were all Mac I used to command line ClipWrap from WorkerNode and create proxies for all our .MTS clips. A second pass from WorkNode would compress them down. This worked great most of the time but every once in while WorkerNode couldn't process a clip generated by ClipWrap. So I looked for another route.

Tried FFMPEG and got random freezes. Tried Adobe Media Encoder through a watch a folder arrangement and that works most of the time but on big jobs it can hang. Frustrating that nothing is rock solid.

There's Xuggle of course but the play back is too rough to be useful. If only there were an AVCHD codec for Quicktime. VLC can play it! Sigh.


Return to posts index


bryson jones
Re: WorkerNode command line script problem
on Oct 11, 2013 at 10:42:36 pm

Hey, nice to see someone write novels like I do. ;)

UNC, that's correct, only on PC's. But just know that you shouldn't use D: or whatever when you do set up PC paths.

Totally, know that you're not knocking the system. (I don't work for SB, hey, knock away!.) But if you tie the Worker Node to the pathing for a server, what happens when you have a different Production Group on different storage on the same server? If you follow, the reason the path mapping needs to be separate is that you may have a Worker Node working for multiple "Groups" which if you know your CatDV, means different settings, including path mapping are possible. Maybe it could be read, but then you'd have to tell it which pathing to use from which Prod Group in a large environment. (In our shops people often mount the same storage different ways depending on access level.)

"While I've written some python in the past I don't have the time or inclination to spend the hours it would take me to figure it out. Deadlines are always looming. Much easier to just ingest via Mac for now." - That's why I founded NSA ;) (BLATANT PLUG)

AVC is like the poor, it will be with us always (now there's a reference for you.) Actually I hope it is EOL someday. But if you import into Premiere as oppose to just dragging it in, you actually get good spanning and usually TC as well. That gives me hope.

Your drop folder is probably best for now. But yes, reliability is always an issue on any automated process.

You are doing good work over there man. Keep it up and thx for contributing.

bryson

bryson "at" northshoreautomation.com

northshoreautomation.com


Return to posts index

Jeff Schaap
Re: WorkerNode command line script problem
on Oct 15, 2013 at 5:11:50 pm

Hey Bryson,

[bryson jones] "AVC is like the poor, it will be with us always (now there's a reference for you.)"

Hahaha, yep, Matthew 26:11 :)

Thanks for the encouraging word. I've benefited from your wisdom more than once and greatly appreciate your willingness to answer questions.


Return to posts index

Jeff Schaap
Re: WorkerNode command line script problem
on Oct 15, 2013 at 5:07:07 pm

Hey Bryson,

I wanted to ask you about:

[bryson jones] "Bigger issue, it's a best practice to use UNC paths in CatDV and Worker. Mapped drives can create issues with Quicktime (google that for some fun) and also are more confusing for writing scripts"

I did google it but I can't find one definitive issue... So I was wondering exactly what the problem may be. Could I trouble you for a link to a specific site detailing the issue?

I ask because we have in fact had what I would call "access delay" in CatDV when previewing files on drive letter mapped locations. So, I am wondering if this is the issue.

Thanks,
Jeff


Return to posts index


bryson jones
Re: WorkerNode command line script problem
on Oct 18, 2013 at 11:02:39 pm

http://lmgtfy.com/?q=quicktime+windows+drive+mapping+problem

Jeff, I wouldn't share that if you didn't have a good sense of humor.

Seriously though, this is all over the place. We've watched it since CatDV v7 (Ah, the grey beard...)

I'm hoping for a fix, or someone to decide that QT is worth saving but not holding my breath.

bryson

bryson "at" northshoreautomation.com

northshoreautomation.com


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]