FORUMS: list search recent posts

Compressor 4 "Render Farm" :: What is Required?

COW Forums : Compression Techniques

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Doug Daulton
Compressor 4 "Render Farm" :: What is Required?
on May 26, 2012 at 10:46:18 pm

One of the reported advantages of Compressor 4 is the effective integration of Qmaster and the ability to more easily distribute renders to other Macs with Compressor 4 installed. I am trying to get this working and am having some trouble doing so.

I've tried setting up QuickClusters & Managed clusters, but the jobs are not distributing across the machines as I thought they would. I've looked at the C4 tutorials on Lynda.com, searched the forums here and scrubbed the Interwebs for example setups. I've read the docs as well and I am clearly not groking them or missing something crucial in my read.

The only thing I think could be a problem is I am not setting all of the nodes up with a central data store on the network. Each is working off of their local hard drive.

So, before I invest in a SAN, a couple of questions:

1) Have I misunderstood C4's ability to create a video render farm?

2) If not, does distributed render in C4 require a central data store (CDS)? If not, what else might be the issue?

3) If the absence of a CDS is the issue, does it require a SAN? Or, can I used a mapped drive to a NAS? If a SAN is required, does it have to be Apple XSAN? Or, could I use something like Bob Zelin's DIY SAN (link below).

http://magazine.creativecow.net/article/build-your-own-affordable-san-that-...

A little about me, I am pretty technical. I build and modify machines all the time. I also have a pretty firm understanding of networking principles, but would not be considered a network engineer. I have experience with Episode Engine on both OSX & PC, so I know this can be done. I am just looking to do it using C5, or if needed C3.5.

In any case, feel free to be pretty technical in your responses. I should be able to follow you and ask good questions when I don't.

Thanks,

Doug

PS: If there is another forum which is more appropriate for this conversation, please point me there. I am new to the COW, and just learning my way around the community.


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 26, 2012 at 11:04:29 pm

My experience has been that you need a central data store. It can be an afp drive.



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 27, 2012 at 12:42:29 am

Thanks Ian. Will give it a go with that configuration.


Return to posts index


Bob Zelin
Re: Compressor 4 "Render Farm" :: What is Required?
on May 27, 2012 at 1:16:01 pm

hi Doug -
I cannot answer your question. I am responding here to your email to me on 5/27/12
A SAN is not a render farm. I do not know how to setup a C4D render farm. But I can tell you one thing. In the article that I wrote that you referenced, if you create a SAN like I described and turn QMaster on, and try to do this type of C4D render, it will lock up the entire system, and no one will be able to work. I had this exact example happen at a client with 5 workstattions (FCP), and one C4D workstation. They turned on QMaster on all workstations, and started a large render and within 30 minutes, everyone locked up.

So it doesn't work.

Bob Zelin



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 27, 2012 at 6:10:53 pm

Bob,

RE: SAN is not a render farm. Understood. I was thinking of the SAN as attached storage for the render farm, as it seems that Compressor 4 requires a central data store to allow distributed rendering.

I plan to use Ethernet, probably link aggregated Gigabit as I am working with ProRes and not Uncompressed. So, if my research is accurate, it should give me the through out I need.

Having said that, I now suspect a SAN is overkill for my needs. Rather, I think I could do a DIY NAS an mount it to all the render nodes using AFP. In this case, I still plan to explore using NICs and a managed switch with link aggregation to get the maximum throughput across the "farm". Any gotchas with that idea?

Thanks,

Doug

PS: Did I send you an email? I don't think I have you email address. :) If I did, I don't seem to have it. In any case, thanks for taking the time to answer. We are exploring adding C4D to the mix, so your use case is very timely and valuable.


Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 27, 2012 at 6:14:23 pm

Wow. Just reread that. Lots of typos I missed. Sorry for my fat fingers.

In any case, the only one that may require clarification is "through out". I meant "throughput". I think auto-correct got me. :D


Return to posts index


Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 27, 2012 at 6:21:02 pm

For simple rendering and not necessarily streaming video, a managed switch is not necessary. While the cost difference may not be that high I'm just trying to save you a buck. You can just throw a NAS on your current network or use one of your systems to serve out a file share.



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 5:26:11 pm

Have tried the AFP mount method and am having trouble. Here are the methods I have tried. In each case, I have all machines mapped to the same folder mounted locally over AFP.

1) Primary machine as "as Services and cluster controller" with other machines "as Services only".

2) Primary machine as "QuickCluster with services" with other machines "as Services only".

3) All machines as "QuickCluster with services".

In each case, I submit a job using "This Computer Plus" and the job goes into the queue and just hangs. There is no movement or status updating. I even let it run overnight thinking it might need to copy some files. In the AM, it was still hung.

Here is a video walking through the setup and render.

http://screencast.com/t/Nv5kLnP0GTPx

What am I missing?

Thanks,


Doug


Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 6:14:33 pm

By the way, I have read am continuing to try to peruse the Compressor 4 online docs to find my misstep:

http://bit.ly/MYloRh


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 6:54:54 pm

I would stick with actually creating clusters - I have never been a fan of that quick thing. YOu did not show in your video where the destination is - is that destination also on the AFP share?



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 7:35:56 pm

Ian,

I figured going with a cluster controller would be the best option as QuickCluster is intended for making a cluster out of a multi-core machine. Here is a more detailed video, this time with audio, of the setup and the issues.

http://screencast.com/t/7DPoBFBm8x5

Thanks,

Doug


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 7:43:43 pm

Star wars fan??? I do not hear any audio.



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 8:13:13 pm

Star Wars Fan? ... Yep. Aren't we all?

Hmmm. Will check audio and repost.


Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 8:36:37 pm

OK, here is the video, now, with audio!

http://screencast.com/t/b1e0qNtVFc

In addition, here is a screencap of my share monitor, with "hung jobs".

http://screencast.com/t/yfTvTffqUD9


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 8:50:25 pm

Uncheck allow use of unmanaged services and check that each client needs to be managed. This should allow you to add them to the cluster. Also, I would, to start off with, put a file in that share location and render from and to that location. If that works then you can try a file from a local volume and cross your fingers that it gets copied correctly.



Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 8:52:30 pm

Also tell me if there is a Qmaster in system preferences.



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 9:12:40 pm

On Gold Leader, there is a QMaster on in System Preferences. It looks to be the 3.5 QMaster as it has none of the settings I assigned.

I moved the cluster controller to Red Leader as it is a more robust box and it is the home of the central data store. Seemed to make sense that would be home base.

DD


Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 9:10:33 pm

Ian,

I followed your instructions and this did, indeed, allow me to add the nodes to the cluster. Everything looked good until I hit "apply". At that point, "QAdministrator" threw this error:

"Error occurred while creating new cluster: SwampDaemon::NFSPublisher: operation not authorized"

Am now researching.

Doug


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 9:15:49 pm

Try to add one computer at a time - see if one of those spits out errors. Also, I might try a restart on all of the clients and a qmaster reset. let me know how you make out.



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 9:43:58 pm

Aside from this thread, the following is the only result returned by Google. This does not bode well.

https://discussions.apple.com/thread/3809792?start=0&tstart=0


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 9:51:19 pm

I know - I posted in it - did you try my suggestion?



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 10:04:34 pm

Ian - It did not register that was your reply. :D

Trying your solution now.

DD


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 10:36:36 pm

Also - look in console when you get the error (the same one should appear in the console app). Let me know if you see any other errors around it (maybe ones that are more recognizable).



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 10:00:38 pm

Opened a discussion in Apple's Compressor forums (link below) and opened a ticket with the Compressor team.

https://discussions.apple.com/thread/3991369


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 10:00:34 pm

Did you try adding one computer at a time?



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 10:54:27 pm

Ian - Have tried adding one node at a time. The results vary. Here are the steps I've taken thus far:

I have power cycled all of my machines, the cluster controller and the render nodes. Now, I get a slightly different error message, but the same result. Here is the message:

"Error occurred while creating new cluster: SwampDaemon::NFSPublisher: authorization has expired"

Also, occasionally, when attempting to build the cluster, I get this message:

"The controller cannot be reached. Please try later."

This is odd because the controller IS the same machine. So, there should be no network issue.

Also, per the screen capture linked below, when selected, the cluster shows:

Storage: "Info Not Available"

http://screencast.com/t/vG4iLBplaB

I am not sure if that applies, but it seems odd.

Here is a video capture of the process at work.

http://screencast.com/t/6H2qj3dHvbqk


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 10:55:42 pm

Try the other thing I posted about console
Also - are the clocks in sync?



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 11:34:28 pm

Ian,

I've include relevant bits of my console dump below. The following screencast explains what I THINK they are, given that they are out of context.

http://screencast.com/t/ak4cvrS8SK

Thanks,

Doug

5/30/12 12:34:04.012 AM [0x0-0x291291].com.apple.Compressor 2012-05-30 00:34:04.006 compressord[26945:6a03] In '__CFPasteboardCopyData', file /SourceCache/CF/CF-635.21/AppServices.subproj/CFPasteboard.c, line 2372, during unlock, spin lock 0x6821a38 has value 0x0, which is not locked. The memory has been smashed or the lock is being unlocked when not locked.

*****************************

5/30/12 2:02:18.553 PM Apple Qadministrator exception (NSException raised by 'NSPortTimeoutException', reason = 'Distributed objects message send timed out (timeout: 360104568.550303 at time: 360104538.550667) 1') occured in connectAndCaptureService

*****************************

5/30/12 2:51:19.442 PM Apple Qadministrator ERROR: connect failed while updating service attribute info..

*****************************

5/30/12 3:32:33.484 PM qmasterd CDNSharedStorageServer::publishNotification: CException: SwampDaemon::NFSPublisher: authorization has expired
5/30/12 3:32:33.485 PM [0x0-0x14014].com.apple.Compressor 2012-05-30 15:32:33.483 qmasterd[293:1403] CDNSharedStorageServer::publishNotification: CException: SwampDaemon::NFSPublisher: authorization has expired

*****************************

5/30/12 3:40:32.000 PM kernel IOSurface: buffer allocation size is zero

*****************************

5/30/12 3:41:29.034 PM Apple Qadministrator How did this happen!? JC is not captured!! Something is reeeeeaaaallly wrong! :-((
5/30/12 3:41:29.035 PM Apple Qadministrator JCName = RED LEADER (CC)


*****************************

5/30/12 3:42:30.591 PM DiskImages UI Agent Could not find image named 'background'.
5/30/12 3:42:30.591 PM [0x0-0x3d03d].com.apple.DiskImageMounter 2012-05-30 15:42:30.589 DiskImages UI Agent[641:707] Could not find image named 'background'.
5/30/12 3:42:35.398 PM DiskImages UI Agent *** -[NSMachPort handlePortMessage:]: dropping incoming DO message because the connection is invalid
5/30/12 3:42:35.399 PM [0x0-0x3d03d].com.apple.DiskImageMounter 2012-05-30 15:42:35.397 DiskImages UI Agent[641:707] *** -[NSMachPort handlePortMessage:]: dropping incoming DO message because the connection is invalid


Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 30, 2012 at 11:48:20 pm

Ian - All of the machine clocks are synced in Date/Time to the Apple time server. Is this what you meant?

DD


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 31, 2012 at 2:29:23 am

I swear I'm not just trying everything in the book but:

Are you connecting to the AFP share with an account that has write access?

I am assuming the firewall is off.

Have you tried resetting compressor/qmaster on the systems?



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 31, 2012 at 3:45:49 am

Ian - I really appreciate the help. So, don't sweat it. :D

1) When setting up the AFP share (in System Preferences >> Sharing), I gave Read/Write permissions to everyone. So, permissions should not be an issue, unless there is another way of setting them (CHMOD?) that I should be using.

2) No firewall of which I am aware. Is there something in OSX I should check?

3) As for resetting Compressor/QMaster, I have stopped and started Compressor, cleaned Compressor/QMaster 3.5 off of the systems and power cycled all of the boxes in the cluster. Is there is a "reset" switch in Compressor I am missing?

Thanks again.

Doug


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 31, 2012 at 4:48:54 pm

TO check firewall

Apple->System Preferences->Security & Privacy->Firewall

My primary question with AFP is when you are connecting -what account are you using to login to the remote server - the local admin?

In the old compressor it was quite evident how to reset the services. In the new one I have no clue. You can try the command line: qmasterprefs -reset

I have no way of testing this as I do not have a system with only Compressor 4 installed.



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 31, 2012 at 5:59:56 pm

Checked all machines ... Firewall not on.


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 31, 2012 at 4:55:30 pm

To expand on my last post - in compressor:

Compressor->Reset background processes

Also, you may want to look at this:

http://www.digitalrebellion.com/blog/posts/compressor_error_unable_to_conne...


Repair disk permissions, etc as well.



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 31, 2012 at 6:01:28 pm

Resetting background processes let me build the cluster without error. Test render in process, but still appears to be hanging.

This process would be much easier if Apple incorporated more meaningful/useful error messages. :D


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 31, 2012 at 6:11:04 pm

We're getting somewhere. Look at the console before, during, and after you submit the batch. Grab the error logs from there.



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on May 31, 2012 at 6:41:43 pm

We have success! I can submit to the cluster and it is tearing up the test file. Thanks for all of your help Ian.

Oddly, trying to submit to the local machine seems to hang. The local machine is part of the cluster and required to be part of managed services. So, maybe that is the issue? You would think Apple would put in an error message like "This machine is part of XWZ cluster and cannot process jobs independently".

Incidentally, I went back and my storage mounts after seeing 0 machines mounted in sharing. I had just connected to machines over Bonjour, which should work. In any case, I specifically mounted the drive on all nodes via AFP://. I also created a specific cluster user and connected to the drive using that user, and saved the login creds to the keychain on each machine.

I am not sure that is a factor though, as I forgot to do it on one of the machines and the cluster is still using it. Weird huh?

So, one final quirk ... one of my nodes (Green Leader) is not being seen by the cluster controller (Red Leader). GL sees RL and the rest of the nodes, but RL does not see it. So, it is not in the cluster. Again, weird.

Any ideas?


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on May 31, 2012 at 7:51:41 pm

You could try adding it manually via the IP address.



Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on Jun 1, 2012 at 12:10:54 am

I added the box by IP both in Compressor a Qadmin Preferences. Both se te box as online, but it does not appear in the Qadminstrator queue to be added to the cluster.

Did I do it right?


Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on Jun 1, 2012 at 9:49:37 pm

Ian,

I now have the entire cluster online. Woot!

Essentially, I went node by node by node, beginning with the Cluster Controller and did the following:

1) Compressor: Reset Background Processing

2) Ran "Compressor Fix" from Digital Rebellion, to blow away caches etc. Note to the concerned, this did not blow away my compressor configurations.

3) Shut down C4 & Qa4. NOTE: I did not power cycle the boxes. I just closed the app.

4) I brought up C4/Qa4 on the nodes FIRST and the Cluster Controller last.

5) I rebuilt the cluster with all 4 nodes and the CC in the mix.

I offer a little more detail in the screencast linked below.

Thanks to Ian and Bob for their help.

Doug

http://screencast.com/t/217vGrGC


Return to posts index

Doug Daulton
Re: Compressor 4 "Render Farm" :: What is Required?
on Jun 2, 2012 at 1:33:30 am

A little follow-up. The managed cluster seems ... fragile. It seems to break ... a lot.


Return to posts index

Ian Liuzzi-Fedun
Re: Compressor 4 "Render Farm" :: What is Required?
on Jun 2, 2012 at 2:23:12 am

I could have told you that and I meant to mention it. Like so many other things apple comes out with-it aint designed for prime time



Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]