I have been frustratingly trying to work with Encore CS4 for the past few weeks. I come from a Final Cut Studio background so I assumed, like DVD Studio Pro, Encore would like compressed media to work with. This is far from right.
My original process was similar to that of DVD Studio where you import your menu and assets, create timelines and a flowchart for the disk. I had about an hour of content split into about 8 parts. I compressed it in Apple's Compressor with a modified bluray preset. I found it odd how it was taking such a long time to import and dump into a new timeline but I thought nothing of it. After 6 or 7 parts were in the timeline it would crash every time. No matter how many projects I made or how many times I cleared the cache, like clockwork, it crashed.
Next I tried bringing the footage in as straight quicktimes from my original final cut sequence to avoid quality loss. It took the files instantly. When I dropped them into a new timeline, they had no issue. This was my solution. After I prepped the project with a flowchart I set my transcode settings to H.264 Maximum quality for all of my media. Expect a long wait time for the transcode. On a Quad 2.26 Intel MacPro, we benchmarked a 12-15:1 ratio. A one hour bluray took just under 12 hours. The quality was great and clients were happy.
You and I are in the same boat with the same kind of a machine. I highly recommend that you invest in a Matrox Compress HD card for the Mac. It works directly with Compressor 3.0 or above. It encodes in real time with pretty good results. Your 1 hour sequence would have taken 1 hour to make the H.264 file.
Of course, authoring is a different story. I had similar problems and I am still having similar problems, but I eventually got my bluRay authored after a few hours....Quite frankly if you ask me Encore CS4 for a mac is a piece of junk, but unfortunately that is the only option I have to author good bluRays...Toast 10 just doesn't cut it...I do use it to burn the BluRay though
*Note that the Mac version of the Compress HD card doesn't work with Adobe Media Encoder*
Have you had any problems with Audio not being in sync with the Video?