FORUMS: list search recent posts

Using Multiple Local Computers for Encoding

COW Forums : Adobe Media Encoder

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Cody Walters
Using Multiple Local Computers for Encoding
on Dec 14, 2012 at 3:03:48 am

I was curious if it is possible to queue out finished projects from Premiere to a different computer to encode using Adobe Media Encoder. Can I used that other computer to strictly encode while I continue working on my editing computer? Would an Ethernet connection between the two be fast enough, or does this even work?

Cody Walters
Houston Video Production
Houston Wedding Videographer

Final Cut Studio 3
Adobe CS6 Master Suite
Panasonic HVX-200
Canon 7D, 60D



Return to posts index

Mike Beattie
Re: Using Multiple Local Computers for Encoding
on May 19, 2015 at 2:31:17 pm

This is an old post but yes, it can be done. Let me know if you are still interested in achieving it

Mike.


Return to posts index

Clif Davis
Re: Using Multiple Local Computers for Encoding
on Sep 23, 2015 at 4:41:02 pm

Hi Mike,
This is exactly what I am trying to achieve, I would love to hear the details of how this is done.

Thanks,
Clif


Return to posts index


Mike Beattie
Re: Using Multiple Local Computers for Encoding
on Sep 23, 2015 at 5:10:39 pm

Hi Cliff

In this installation there are 20 Macs, each running Premiere. There are also 15 servers each running Proxmox + 4 virtual W7 instances, and each of those with Adobe Premiere running too. So capacity for 80 renders.

When a job is edited on one of the Macs, once finished the editor hits a big green button that saves the .prproj to a special folder that reads the file and sends the job to a render farm server. This server determines which one of the machines to send it to (one of the 60 VMs or one of the Macs if they are not being used for editing, and if ProRes is required in the final format). The HBR media is made available to the rendering machine, and the rendering kicked off via some special software. Once finished, the rendered file is added to the MAM system, along with its metadata, or sent off to some other workflow like publishing /distribution/ CDN /OTT according to the job requirements.

It's a scalable model... so it's working here with 80 simultaneous renders, but is equally happy with 800 or 8.

Hope that helps...let me know if you need more details.

Mike.


Return to posts index

Ht Davis
Re: Using Multiple Local Computers for Encoding
on Feb 14, 2017 at 10:39:36 am

That's a bit overblown, there mike.

Simpler way with a few machines:
Using After effects to render your video is the key. You'll need a drive big enough to hold every single frame in JPEG format, then you'll need to have after effects installed on every machine (doesn't require activation, just needs to be installed). This will also install the After effects rendering engine. Now install the same plugins on every rendering machine (If you use any specialty plugins, you can render up to a frame before and then after... ...and so on... ...then let your main machine handle them with a second render operation to save on plugin costs). You'll have to set up the after effects rendering engine on each machine so it can access the same watch folder. In current CC (2015 or higher by my knowledge), you can even do this across platforms... ...PC\mac makes no difference, so long as they can all see the files and the job in the watch folder. You must create the folder in a shared location (this means all computers can see it; can be network drive, NAS, or a drive on a computer, but I recommend using a tower with a multi-connector NIC, and linking them all to it, then using usb3 or thunderbolt RAID drive enclosure). Set up another location for the output (this will be a folder of images). Place a rip of your audio track into the output folder. Now go to the other computers and start the render engine (not after effects itself) on each one, making sure that you can read and write to the watch folder, and the output folder. In the render engine, set the watch folder, the output folder will also be selected automatically. Once you set this up, set them to start scanning or watching the watch folder. There's nothing there yet, but there will be soon. Do the same thing for either win or mac. You'll need the same plugins if you added any, or you can render portions without the plugins first, then render pluginned portions in a second, single machine pass (I've tried it and it works well enough without a lot of extra overhead when the plugin area is short. It requires setting up multiple render jobs, but do it enough and you get old hat. Now go to the first machine and tell after effects to COLLECT files (usually found in the file menu), and tell it to place them in the watch folder. Depending on the size of your project, this could take some time... ...Unless the Watch folder is the same location where you placed your files initially. It's a catch all step to make sure all the computers can access the files. Now start the job. You can add and subtract machines as necessary. If you put the audio into the output folder and run it through AME, you can compress the video back down with the audio attached in a lot less time.

This works all the way down to CS4 but then you needed to have licenses for every machine. At cs5, you paid half license, and entered a special key. In CS6, you just add a txt file to a specific location. Look up renderfarm for CS6... ...Cheap way to get it done. However, it hasn't been cross platform until adobe bought a farm engine protocol that remapped the project directory so the commands from the machine went to the right location. If you do have 2 or 3 machines and CS6, you can run up to 4 render engines, before they want you to add a license, but that still equates to 4 times the power. With plugins, you might not be so lucky. While it's perfectly legal to install some older plugins on multiple machines and have them render just fine, some will reject the rendering engine if the key provided doesn't support it, or if it detects the same key somewhere else on the same network. Check your plugins and check the license agreement.

ETHERNET is best costwise, especially if you get GIGABIT, and yes it can work between several computers as an intranet (not internet like you see in your browser, but contained to those machines only), and it will be faster; you'll need a server style NIC in at least one machine that will serve as the central hub.
You're better off speedwise with a fiber connected network (you can find these nics online now but expensive), or if you can find them online, thunderbolt\usb3 crossover networking wires. These wires allow you to connect computers together directly at speeds reaching 10gb\s. 10mb ethernet can work, but it's pretty slow by comparison.

Finally...
You won't want to be working on the master machine while it's handling the render job, or any others. However, if you must do some work, you can get away with it. Because you're outputting only the images, uncompressing the video file, you're not using as much of your computing power. Compression takes up a lot more calculation power as it has to compare each frame with those around it, and then drop unnecessary data, and place the new image into the video file's data stream. If you have audio to go with it, it's even tougher. If your images are compressed video already... ...um, glacial bowel syndrome comes to mind as a description... Split up those processes and do the decompress on many machines, then recompress on only one, and you could cut the processing time by up to half in theory (though you'd have to have one computer for every frame, a fast network, and be able to set up the render engine on all of them at once... ....yeah... ...maybe not). Two or three computers working on this cuts down the time greatly.

And an FYI... ...GPU cards only help you to process playback data for most operations unless you have a special engine for them installed, or you know how to hack your settings to utilize them. Most often, the primary rendering of the photos is done by your main processor. Your copies of AE need to match, plugins need to match, all need access to the data, and your processor\os type has to match (x64 to x64). Other than that, you can go cross-platform with this, get some cheap pc's and build a farm that'll keep the hot delivery under 30mins, all run from a mac... ...then laugh your face off, grab some coffee, sit back and pray your chair don't break.

I've done this with CS6 with old and new machines working together. I borrowed the new ones from pals, and set up CS6 for rendering. They have CC or don't even use adobe, but don't mind the space being used. I link them with an ASUS gigabit router or two, and two 2tb drives. I work from external drives on my home machine, but I use disk images to move the data around. It's so fast to move one big file to an area where it can be loaded to multiple machines as if local, then I can run my CS6 render engine. It finds the location locally, which routes to the networked drive, and outputs to another networked drive. Then I just encode from there. Easy peasy. A 4 hour hd video with transitions etc and high end audio takes roughly 3-4hours to render for output media. Since I go to DVD\Bluray anyway for most things, I can even multi-target and run them all in parallel on my fastest machines to get 2, 3, 4... ..10 different outputs with different settings so I can check quality... ...Now that's service. All with CS6, one really old core2duo, 2 to 4 i7's, all with 4-8gb ram and only my 4gb core2duo macbook pro with a video card... ...it's pretty fast if I don't do a lot of weird effects. And for the work I do with non-profit groups, it's fast enough that it doesn't make me wanna puke. I'm updating to cc2015 now, and hope to include some Cheap PC's really soon so I can work even faster. I cannot handle 2016 or 2017 on my laptop, but 2015 I can do, and I can install it on two more besides... ...no prob bob. I'm there. 2 i# or i5 pcs with decent networking, ssd's for the main drive, CC2015 and solitaire... For pro-res rendering, you just use the AME on the mac for best results. For other, go either way with AME cc2015. It handles so many formats and the advanced tools are really powerful by comparison. Just outstanding.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]