I wanted to add this post in hopes I can help some of you with an issue you may well get while using Encore CS3 (or other encore versions possibly).
I was authoring my DVD without any issue until I realized i had an audio track I was unhappy with in one of my chapters. I changed out the audio.
I also added a new first play with an audio track under it, an Mp3.
I started getting PGC errors. I had no idea what to do.
So after hours of google'ing the error, I had no luck, in fact I even killed Encore with a runtime error.
I added a new user and reopened the project. I got past the PGC error (by putting all files for the project in one foler and making sure there were no long filename and no underscores) but then started getting the dreaded "Error Starting Transcoder" error. I made sure all my files were transcoded prior to attempting to build. I cleared the cache, restarted, reopened and pulled almost all my hair out for hours.
In the end I started digging through the project on my hard drive. I found a few curious things.
1. For some reason, even when clearing the cache, a lot of the files I had used along the authoring still were in the cache, I deleted them. You may want to clear these manually yourself, the project now opens a ton quicker.
2. I found that some of my audio encoding in the "ProjectName"\Sources\Transcodes folder had underscores in the titles, when they did not have them normally. So I deleted them after reverting all said files to original.
3. This is the kicker and this was the problem that was giving me the Error Setting up Transcoder, so pay attention here folks. There was a file in the "ProjectName"\Sources\Transcodes called _PGC_Fpgc_track_Otrk_1_audio.ac3_temp along with corresponding .xmp files with the same name. I remembered the PGC problem immediately because it was in the filename. I deleted this file (and any others with the same name) and went back to encore. I proceeded to click build. No problems, no issues, clean build. I think the PGC file was corrupt. Whenever Encore went to build my DVD, it was either trying to access or replace this file and it could not, so it gave me the error setting up transcoder.
Hopefully, this may help some of you because I tried about ever work around I found but none mentioned deleting this file.
Hope it works for you and saves you some very irritating hours.