Final Cut Pro unusable for big projects?
by Christoph Strothjohann
on
Jun 21, 2008 at 3:02:21 pm
I´m an Assistant Editor on a european movie.
It was shot on 35mm film runniing 24fps.
We´re editing on final cut pro 6.0.3 (on OsX 10.5.3) using the aja io hd (latest software and firmware) to capture video from hd tapes, that were telecined at the lab running at 24psf (not 23.98).
log files were provide by the lab, so each take is captured as an individual clip.
the aja io hd also puts out the picture to a 52" full HD monitor, converting to 25fps.
we are using the the apple pro res codec (not HQ) at 1920*1080.
Sound was recorded seperatly (24bit and 48hz) and imported into the project.
sound and picture were synced to slates within final cut pro using the "merge clip" function an then sub clipped to get rid of black head and tail before editing with it.
our project has about 8000 files (sound and picture), which take up 3,5 terabyte spread on 3 internal sata drives and 2 external esata drives.
the main project file is about 125mb big.
on the hardware side we are using an early 2008 MacPro 8core 2.8ghz with 16gb of ram. with the geForce 8800GT graphics.
only leopard, quicktime, itunes, final cut studio2, and the aja software installed.
Now to our problem:
This setup runs extremly instable. with one or two crashes on a good day or up to 8 crashes (including complete system crashes) on a bad day. most of the crashes happen during playback or when we try to move files to a second opened project. all crashes seem to be random.
that is already very annoying and takes a lot of our time in the editing room.
but the worst is that we have a problem with longer playbacks.
each time we screen rough cuts ( at the moment ca. 100 mins long) out of one sequence after a 15-20 minutes fcp gets a short "hiccup" with visibly stuttering frames for a moment then continuing to run on, but completly out of sync. when we then hit pause and play quickly it continues running in sync again. Until after another 15 to 20Minutes we have the same "hiccup" occurs again...
this is completly unacceptable when u screen a cut and has never happenend to us in 10 years editing on the avid media composer.
after we tried all tips that were given (like turning of "high quality scrubbing", deleting preferences, using without the aja io hd, etc.) we moved our project two days ago to a brand new mac pro with the same specs and fresh install of leopard and final cut.
The same problems occur on this machine.
does anyone have similar problems or experience with final cut pro and bigger projects and can maybe shed some light on what we could do???
Re: Final Cut Pro unusable for big projects? by walter biscardi on Jun 21, 2008 at 3:47:28 pm
[Christoph Strothjohann]"our project has about 8000 files (sound and picture), which take up 3,5 terabyte spread on 3 internal sata drives and 2 external esata drives."
There's a HUGE problem right there. You have a massive project spread out over multiple drives. FCP has to search through all of those to find the files and that takes time. This REALLY slows down your performance.
With a project this big I don't see why you or whomever owns the system did not invest in a large, high speed SATA array. We own an 8TB MaxxDigital EVO HD unit which is 8 drives striped together for around 400 - 500MB/s. And more importantly, it's all one media array.
[Christoph Strothjohann]"This setup runs extremly instable. with one or two crashes on a good day or up to 8 crashes (including complete system crashes) on a bad day. most of the crashes happen during playback or when we try to move files to a second opened project. all crashes seem to be random."
Again, a symptom of your media being spread around instead of in one location. Very bad project management for a project this large.
[Christoph Strothjohann]"each time we screen rough cuts ( at the moment ca. 100 mins long) out of one sequence after a 15-20 minutes fcp gets a short "hiccup" with visibly stuttering frames for a moment then continuing to run on, but completly out of sync. when we then hit pause and play quickly it continues running in sync again. Until after another 15 to 20Minutes we have the same "hiccup" occurs again..."
My guess is the hiccups happen when FCP is switching from one media drive to another and there's a lag in the information getting to FCP, so you get a hiccup.
[Christoph Strothjohann]"after we tried all tips that were given (like turning of "high quality scrubbing", deleting preferences, using without the aja io hd, etc.) we moved our project two days ago to a brand new mac pro with the same specs and fresh install of leopard and final cut.
The same problems occur on this machine."
If you're using the same media arrays spread out, this will continue to be a problem. You need a high speed, single media array for a project this large, preferably running RAID 5 in the case of a drive failure. MaxxDigital, Dulce, Sonnet, Ciprico, Facilis all make excellent large media arrays for this type of work
Walter Biscardi, Jr.
Biscardi Creative Media HD and SD Production for Broadcast and Independent Productions.
Re: Final Cut Pro unusable for big projects? by Mike Parfit on Jun 21, 2008 at 5:51:57 pm
Hi,
Yes, we have a big project, and we have almost exactly the same problems. Our crashes happen mostly during rendering, but also at apparent random. We also have had the playback problems you describe, and sometimes they happen even when the program is set to stop on dropped frames. In other words sometimes the program does not think it has dropped frames but apparently it has; it then goes out of sync and we have to stop and start to get it back.
We reduce the crashes by reducing the size of the project file, mostly by restricting the numbers of sequences in the project. We tend to duplicate sequences often for backup and to retain earlier versions, so we often save a new project and delete the old sequences. We find that it helps to have the project size well under 100 MB. When we need to do a lot of rendering, we also remove our Kona 3 card from the system, which helps a great deal.
In the case of the IoHD, we used one briefly and found something different. With our Kona 3 card all we had to do was remove the card itself, not the software, to dramatically improve our stability. With the IoHD, however, unplugging the unit and rebooting did not help at all, but when we uninstalled the software, that did help.
These removals do not solve the problem entirely, but greatly reduce the crashing issues. (The playback can't be helped this way, of course, because we have to playback through the Kona.)
We think there is a memory leak in Final Cut that steadily locks up memory and eventually leads to a crash. We have come to that conclusion because several times we set the program to render a long clip on one monitor and used the other monitor to show the OSX Activity Monitor. Since it was a very long render, we set up a video camera to record the Activity Monitor. As the render continued, the memory use gradually increased, until it passed 2GB (we have 4GB in the machine), at which point it worked for about another five or ten minutes, then crashed. We repeated this test several times with different size projects, and it was always the same. With the Kona 3 card out of the system the machine took longer to get to the 2GB spot and longer to crash, but it usually did.
This guess at the source of the problem is also supported by another symptom. We sometimes crash just when scrolling through a non-rendered clip on the timeline, even when we have not been doing much rendering. These kinds of crashes never occur during the first few hours of working with Final Cut. They always happen after we've been at the desk for some time. This, too, would seem to indicate that there's a memory leak somewhere, that builds up as the program is used over time.
We significantly reduce our problems by rendering small clips from the timeline of the bigger projects, then putting those smaller clips on a separate timeline. When the timeline is restricted to one video track and two or four audio tracks, we are able to render 93-minute clips with no problem. With a single big clip like that on a single drive, the playback is no problem.
Like you, we have tried MANY, MANY fixes suggested by people on this forum. Not one of those fixes have changed the outcome in any way at all. The only fix we haven't tried is what Walter suggests, the addition of a very fast array that looks like a single drive to the system. I think that might help, but we simply don't have the thousands of dollars necessary. And if there's a memory leak that makes large project files crash more frequently than smaller ones, it won't help much.
My additional suggestion is that you do what we've been planning but haven't gotten around to -- chop your film into reels and have a separate (much smaller) project file for each reel. It's not difficult to work with a couple of files in FCP when you need to shift sequences from one to another, either nested or just cut and pasted, so it shouldn't handicap you much. Also, it seems to help somewhat to have as few video tracks as possible, so if you have a few places with complex composites, I'd put them on a separate timeline, render them out and put the result on the main timeline on track 1.
I hope you find some solutions. Personally, I think this is a problem that has been with FCP from the start and will only be fixed by the eventual complete code rewrite. But I may be wrong and there may be some simple fix. So, good luck.
Re: Final Cut Pro unusable for big projects? by walter biscardi on Jun 21, 2008 at 6:59:25 pm
[Mike Parfit]"The only fix we haven't tried is what Walter suggests, the addition of a very fast array that looks like a single drive to the system. I think that might help, but we simply don't have the thousands of dollars necessary. And if there's a memory leak that makes large project files crash more frequently than smaller ones, it won't help much."
All I can tell you is that if you're doing very large projects, there's no reason to bite the bullet and get a very fast array. The prices are so much cheaper these days.
The 8TB unit we're running is $5995 US and that includes the ATTO R380 Host Adapter. The 6TB is just $4995 US. Considering I spent about $8995 4 years ago for a 2TB Fibre Channel array, that's an amazing price. I purchased two of the 8TB units last year and they have been cutting projects that have as many as 200 raw tapes per project with zero stability issues. The only issues we ran in to were some Quicktime / Render issues with one version of Quicktime.
Fast and reliable storage is simply a must for any large project or really any shop that is doing a lot of work. You just can't be stringing your media along multiple drives or even regular DV projects can have slow-down issues, instability and the like. One, large fast media array is always better than multiple media drives.
Yes, FCP does have some sort of a memory issue that has been around since probably day one. The longer you run your FCP system, the slower it will get, especially if you open any other apps while you're working. We always quit FCP during lunch and re-launch the app after lunch. Sometimes we'll do a complete reboot after lunch. This is just common practice to ensure a smooth edit session.
Walter Biscardi, Jr.
Biscardi Creative Media HD and SD Production for Broadcast and Independent Productions.
Re: Final Cut Pro unusable for big projects? by Mike Parfit on Jun 21, 2008 at 7:37:07 pm
Hi, Walter,
Thanks. I don't think we disagree here, except in terms of budget. $5,000 is just out of our reach. I mean, it is more than we paid for the machine itself.
It has so much to do with what a filmmaker is doing. You are putting out a series and a lot of other stuff regularly and your cash flow is quite different from ours. We've been working on a single feature doc for four years. We've won awards out on the festival circuit and have significant distributor interest, but we're taking the highest-risk route here and trying to have an effective theatrical release, so we are flat broke, and have been for the whole four years, as we've put everything into shooting and editing time, and then publicity. Five grand is just out of reach. It doesn't mean that we're less professional, but it does mean that we're on a different budget track.
Sure, it probably makes sense for us to have borrowed enough at the start to buy one of those excellent external raid systems that you and many others have been reviewing for us all these years, but this comes down to my only real crankiness about all this. If working on large projects requires such a system in order for FCP to be stable, Apple should tell us in its system requirements, and then we could take that cost into consideration at purchase time. Instead, the stated requirements and ALL the advertising indicate otherwise. The only stuff published about high-performance raid systems in both the FCP and AJA documents is related to whether you have a system fast enough to avoid dropping frames on capture or playback. There is nothing about system stability, but that is the big problem.
That's my gripe -- that Apple does not accept this problem. And the other thing that is still preventing us from finding that five grand somewhere is the strong likelihood that it just won't improve things much. For a long time we used DVCPRO HD and HDV all on a 3-disk internal array, with great read and write numbers, far beyond what those high-compression codecs require, and that didn't make any difference in the marginal stability of the system.
Oddly, one of the other very effective things we've done is to take OUT 2 GB of ram, which takes our system down to just 2 GB total. The system doesn't crash then, but it does give a lot of "Out of Memory" errors. (By the way, just in case you're thinking, 'Ah ha, he has bad memory,' I have alternately swapped out different sticks of memory just to be sure it's not something in one of the sticks.) It's as if FCP has proper error-catching code at 2GB of memory, but at 4GB or larger it just crashes.
Re: Final Cut Pro unusable for big projects? by Michael Sacci on Jun 21, 2008 at 9:25:46 pm
My 2 cents, I think it is crazy to think you should be able to do large editing projects with JBOD, internal and external. A good array is part of the tools of the trade. Now you can get the speed you need for less but you give up redundancy, I put together 7.5TB Raid 0 systems in the 2K range. With my system I have edited 8 streams of ProRes HQ (HD) multicam so it can keep up, All things being equal I would like to get a higher end array but I also want to eat, but there is no way I would except to do my job with single Sata drives.
If you have to use multiple single drives you ready need to be editing DV25 footage and then do a final recapture of the HD footage.
Re: Final Cut Pro unusable for big projects? by Steven Gonzales on Jun 21, 2008 at 9:59:39 pm
I've edited a lot of features which had very low budgets, through all permutations of Final Cut.
You might have to scale down your codec to match your hard drives. Also, keeping the project files size down makes everything work better.
FCP can definitely do large features at a low budget. However, using a very high throughput codec, without the proper drive subsystem, will be very painful and unworkable.
You might have to sacrifice editing image when using low cost hard drives and a smaller "offline" type of codec. If you're finishing on film, the important thing is your film lists are correct, and you can deliver sound properly, not the quality of the codec you're editing with.
Re: Final Cut Pro unusable for big projects? by walter biscardi on Jun 21, 2008 at 10:47:20 pm
[Mike Parfit]"Thanks. I don't think we disagree here, except in terms of budget. $5,000 is just out of our reach. I mean, it is more than we paid for the machine itself."
Absolutely, high speed, high volume storage does cost more. When you get into shared FibreChannel it goes through the roof, maybe $20,000 for 6 to 8TB shared for two workstations that cost maybe $8,000 combined.
[Mike Parfit]"Sure, it probably makes sense for us to have borrowed enough at the start to buy one of those excellent external raid systems that you and many others have been reviewing for us all these years, but this comes down to my only real crankiness about all this. If working on large projects requires such a system in order for FCP to be stable, Apple should tell us in its system requirements, and then we could take that cost into consideration at purchase time."
Apple has been recommending their own X-RAID for years for working with video production. That's a high speed, very expensive array. So that should be a warning right there.
Also, Final Cut Pro is designed to work in many MANY different configurations depending on your needs. Small projects require small arrays. Larger projects require careful consideration, especially on media management. It really makes common sense that if you are going to need a lot of storage, it's better to purchase a large, fast array in the beginning.
[Mike Parfit]"It has so much to do with what a filmmaker is doing. You are putting out a series and a lot of other stuff regularly and your cash flow is quite different from ours."
Ok, let's start from the beginning. In 2001 when I opened Biscardi Creative Media I was coming off a failed partnership that put me $50,000 in the hole. As in I still owed my ex-partner $50,000 to pay off the debts of the failed company. I had zero equipment.
So I borrowed $27,000 to purchase a brand new Final Cut Pro System with a Pinnacle Cin�Wave card along with a BetaSP Recorder and what was at the time a very large 240GB Med�a SCSI high speed array.
So opening my doors I'm $77,000 in the hole before I start my first project. I had exactly 1 full time client. A corporate client for whom I did a lot of short children's stories. Sure I had a great reputation and resume with Emmy awards and the like, but just 1 full time client and whole heaping load of debt opening my doors.
You know how I got Good Eats? A very lucky circumstance with the D.P. asking for help on the Cow and I happened to answer his call. Then I went out on a limb and invested approx. $32,000 that I didn't have into a Panasonic 1200A DVCPro HD VTR. When Good Eats switched to HD, they asked me to help them develop the HD Post workflow and we've now delivered over 60 episodes of the series in HD.
So I get very annoyed when I see posts that say "Well, you're in a different league, we can't spend that kind of money." I'm just a small business guy who believes I the only way to make a successful business for yourself and to make you stand out from the others AND to have a STABLE system, you need to spend money in certain areas.
High speed, stable, reliable storage is one of those places.
[Mike Parfit]"Oddly, one of the other very effective things we've done is to take OUT 2 GB of ram, which takes our system down to just 2 GB total. The system doesn't crash then, but it does give a lot of "Out of Memory" errors. (By the way, just in case you're thinking, 'Ah ha, he has bad memory,' I have alternately swapped out different sticks of memory just to be sure it's not something in one of the sticks.) It's as if FCP has proper error-catching code at 2GB of memory, but at 4GB or larger it just crashes."
4GB of RAM is a major problem as I have noted in my blog and in other posts here on the Cow. Our Mac Pro Quad had all sorts of issues with 4GB RAM that went away when we dropped down to 2GB. Now it's running 8GB RAM with zero issues. The new Octo Core is running 10GB with no issues.
Walter Biscardi, Jr.
Biscardi Creative Media HD and SD Production for Broadcast and Independent Productions.
Re: Final Cut Pro unusable for big projects? by Steve Eisen on Jun 22, 2008 at 12:42:34 am
Thank you Walter. You said it all. I wish more shooters/editors can understand the requirements needed in this business. If you are going to shoot and edit in HD, you need to do it the right way. If not, expect undesirable results.
I too get annoyed when I see posts complaining that something doesn't work or blaming Apple. The majority of the time it is user error. Apple is giving you the basic tools needed to get you started in this business. You still have to add more RAM, hard drive storage, monitors, capture card, etc.
I am a small business owner that started with a $15k loan. I don't need to go into detail on the equipment I own or how much money I make. All I can say is I am very educated when it comes to the necessary equipment needed to complete a project. My system has never hiccuped or experienced any crashing like Mike and Christoph has described above.
Can an HD project be done on a budget? Yes it can. It still costs money.
Steve Eisen
Eisen Video Productions
Board of Directors
Chicago Final Cut Pro Users Group
Re: Final Cut Pro unusable for big projects? by Rafael Amador on Jun 22, 2008 at 1:48:12 am
Instead of talking about how to get an stable system, we end up talking about to spend more money.
My bigger project (no that long) have been 47 minutes. I worked with some 120 tapes. A 4 years old G5 and few external FW400-800 HDs. Not a single crack.
I live 600 miles away from the closet Apple service. I can not allow my self any kind of crashes and I get it. I don't understand why people with the same equipments and configurations, some have problems, some have no problems. For me the word is : Maintenance.
Re: Final Cut Pro unusable for big projects? by Mike Parfit on Jun 22, 2008 at 1:55:52 am
Hi, all,
Sorry. I am not trying to annoy people. I am trying to get answers. My understanding is that this is what forums like this are for. Perhaps they are exactly for helping educate people who do not have access to, let's say, the urban areas where one can find users groups and folks who can come over and help with your difficulties. Perhaps they are exactly for people who try many things to solve their problems -- such as searching these forums -- but may have overlooked one parameter in the software that twitches everything. Perhaps they are exactly for people who trying to reach beyond their grasp and learn while doing.
I'm sorry I tried to describe why I can't come up with $5,000 for an array right now, because it is not fair to compare wealth and poverty and that tends to rub people the wrong way, so my genuine apologies for that.
But I have to say that expressing annoyance that I may not know as much as you do seems exactly contrary to the purpose of all these forums, in which everyone is still learning one facet or another of an extremely complex and swiftly-evolving business. I am here exactly because I am hoping that you know more than I do. I respect that knowledge, and when I post here I try to do so in great detail, including the things I have tried and the way I operate, precisely because I am hoping that my problems are indeed user error, and someone else who has made it through the same mistake will have the generosity of his or her time to help me get past it.
And as far as the money goes -- does no one in this business besides me have a person or a department in charge of budgeting in which someone regularly rolls his or her eyes when you come into the room, knowing that you're going to ask for yet another piece of expensive hardware? Does no one recall or regularly experience that same individual or department simply saying, 'You said the same thing last time, and this time you're just going to have to make do?' Does no one ever have the experience that when you ask for money the person with the money, or rather, what's left of it, reads you the equipment requirements for certain software, and points out that what you already have far exceeds those requirements, but still you're asking for more?
We are editing with video -- HDV and DVCProHD -- that does not require particularly fast disks, and we have an internal raid array that far exceeds the required speed. We do not suffer from significant numbers of dropped frames and if we do because of a cluttered timeline we have excellent workarounds to solve that by making single clips of what we need to play back. We routinely output to HDCAM which gets shown on the big screen with no problems. What we suffer from is regular, repeated crashing during editing and rendering. Now, can one of you go in to that department that has the money and tell those folks that it makes sense to spend $5,000 on faster disks primarily to prevent renders from crashing? Can any of you say with authority that rendering requires very fast disk read and write numbers? If you can, you know more than I do and you're smarter than I am, which is probably the bitter truth anyway. But I haven't been able to make that case.
Best wishes to all. I have learned a lot in these forums. Just not enough yet.
Re: Final Cut Pro unusable for big projects? by Alan Okey on Jun 22, 2008 at 6:33:39 am
Two observations about your hardware system that have not been addressed:
You may want to switch to the ATI Radeon X1900XT for better Core Image performance. The 8800GT is only better for gaming, not for Pro Apps.
If you plan on finishing on the FInal Cut system (especially if you're going to grade in Color), you should strongly consider changing the graphics card to the recommended model (the ATI X1900XT) for better performance.
Regarding RAM: IS your system configured with Apple OEM RAM, or third party RAM? If third party, who? While there are several reputable third party RAM suppliers out there, many are pretty sketchy. Flaky RAM is known to cause problems in Mac Pro systems. Also, there is an optimal configuration of memory modules detailed in Apple's Final Cut Pro support pages:
What is your final delivery format? Are you doing a film out, or finishing to HD video? I'm assuming film out, as you're working at 24fps, not 23.98. If you're doing a film out, why edit in HD at all? Why not adopt a traditional offline SD editing workflow to generate a negative cut list? Even if you are finishing in HD, you might have better luck offlining in SD and then conforming to HD at the end of the editing process, given your storage limitations.
Re: Final Cut Pro unusable for big projects? by walter biscardi on Jun 22, 2008 at 12:57:24 pm
[Alan Okey]"You may want to switch to the ATI Radeon X1900XT for better Core Image performance. The 8800GT is only better for gaming, not for Pro Apps."
This is true. I actually purchased a new Octo Core from the Apple Store that already had the 8800 in it. I pulled it out and put the 1900 in there. I'm waiting to see how that new ATI card is going to perform and might switch to that at some point.
[Alan Okey]"Regarding RAM: IS your system configured with Apple OEM RAM, or third party RAM? If third party, who?"
Good point, all the RAM should match too if possible. We purchase from Crucial.com for all our RAM. They have a fantastic system checker to ensure that you're getting exactly the correct RAM for your system.
Walter Biscardi, Jr.
Biscardi Creative Media HD and SD Production for Broadcast and Independent Productions.
Re: Final Cut Pro unusable for big projects? by gary adcock on Jun 22, 2008 at 12:32:41 pm
[Mike Parfit]"And as far as the money goes -- does no one in this business besides me have a person or a department in charge of budgeting in which someone regularly rolls his or her eyes when you come into the room, knowing that you're going to ask for yet another piece of expensive hardware?"
Yes we have all had that same issue. try it with $150,000.00 requests instead of $5,000.00.
Solving issues is about presentation of the problems at hand and how the purchase will save money in the long run. Post is about Time, you time, the editors time and guess what - crew costs at every stage of a production are greater than the hardware costs. You convince a money person that spending $5K for hardware will take 2 weeks off the Edit or reduce the OT because of these issues you will be able to reinforce your position that you are looking out for them.
IMHO your posts here are example of poor communication - You cannot come in "whining" that you need more, it doesn't work, life sucks and get respect.
Look at the title of this thread, it does not ask for help, it declares that it does not work. Asking if others have had the same issues, without the declarations.
Maybe try a different tack and you might have better results.
gary adcock
Studio37
HD & Film Consultation
Post and Production Workflows
Inside look at the IoHD
Re: Final Cut Pro unusable for big projects? by walter biscardi on Jun 22, 2008 at 12:54:57 pm
[Mike Parfit]"Sorry. I am not trying to annoy people. I am trying to get answers."
One answer I have give you and Christoph is the need for a large central media array. That IS the answer for a project with a LOT of media. You simply cannot string together multiple drives, throw 1000s of clip on to all of those drives and expect FCP to be stable. That is not me telling you to spend money. This is me telling you to have a stable system with 1000s of clips, the first step is to consolidate all of that media on to one array.
[Mike Parfit]"And as far as the money goes -- does no one in this business besides me have a person or a department in charge of budgeting in which someone regularly rolls his or her eyes when you come into the room, knowing that you're going to ask for yet another piece of expensive hardware?"
Yep, I worked for two corporate companies for about 3 years prior to starting my own business. The rule there was be absolutely sure you have EVERYTHING you will need for the next 12 months because you simply can't come back and ask for more unless there are extenuating circumstances. That's when I learned to ask a LOT of "what if" questions to the vendors. This ensured what I asked for would service me today and for the next 12 months. Generally we purchased much more than what was absolutely needed at that moment and storage was always something we overpurchased on.
Today I have a VERY good VAR here in Atlanta and I ask him a LOT of questions. I review the Cow postings, I look at the Cow Reviews, I especially read the reviews and posts from Bob Zelin.
Maybe because I've been cutting on NLE's since 1993 I learned early on that Storage is the key to stability and performance on just about any NLE system. This is the one thing that I see most people on these forums just don't get. They get the largest processor, the fastest graphics card, but then they skimp on what is really the most important part of your entire setup. Storage.
I always purchase as least double of what I think we'll ever need. In the case of the 8TB units, they replaced 2TB units. I could have simply added another 2 or 4TB SATA array to the same card and just ran media off multiple arrays, but this is not as stable as the single large array. And we never have more than 3 or 4TB on either of the arrays, but by having all that overhead and empty space on the array, they run very efficiently and fast.
So yes, Final Cut Pro is absolutely usable for large projects, but step one is maximizing your media storage efficiency. Without that, you will have problems cutting very large projects.
[Mike Parfit]"Now, can one of you go in to that department that has the money and tell those folks that it makes sense to spend $5,000 on faster disks primarily to prevent renders from crashing? Can any of you say with authority that rendering requires very fast disk read and write numbers? If you can, you know more than I do and you're smarter than I am, which is probably the bitter truth anyway. But I haven't been able to make that case."
Here's the neat thing when you start asking questions about making very large purchases. You can ask for a demo. MaxxDigital will allow you to demo one of their products for 30 days. Don't like it, return it. Whenever you, or anyone else reading this, is about ready to make a very large purchase, don't just go to the cheapest online source. Go directly to the manufacturer or to a reputable VAR, like WH Platts here in Atlanta. Either the VAR will have demo units available for testing or some manufacturers, like Maxx will allow you to demo a product. And always mention you saw their products on the Cow either in a banner or in a post.
If you want to try it, you can get one of those high speed arrays in to your system for a week, a couple of weeks, whatever. If the stability issues go away, you've solved the problem. If they don't, well then we are looking at some conflict in the system or some corrupted media along the way.
Walter Biscardi, Jr.
Biscardi Creative Media HD and SD Production for Broadcast and Independent Productions.
Re: Final Cut Pro unusable for big projects? by Mitch Ives on Jun 23, 2008 at 4:37:22 pm
[walter biscardi]"4GB of RAM is a major problem as I have noted in my blog and in other posts here on the Cow. Our Mac Pro Quad had all sorts of issues with 4GB RAM that went away when we dropped down to 2GB. Now it's running 8GB RAM with zero issues. The new Octo Core is running 10GB with no issues. "
I'm glad Walter addressed this. 8Gb should be the minimum.
FWIW, our new 8-core crashed regularly with 10GB (two fours, two ones) of ram. At 8GB (two fours) it doesn't crash at all. If you read Apple's tech note, they pretty much tell you that with FCP you need even matching modules on both risers, which means from 8GB you will probably want to go to 16GB next.
Re: Final Cut Pro unusable for big projects? by gary adcock on Jun 22, 2008 at 12:15:10 pm
[Mike Parfit]"working on large projects requires such a system in order for FCP to be stable, Apple should tell us in its system requirements, and then we could take that cost into consideration at purchase time. Instead, the stated requirements and ALL the advertising indicate otherwise"
Sorry, I am tossing up a bull*@^ flag.
Apple in every high level presentation and article about editing in film-style projects always mention and shows high-end storage solutions. I have not seen an Apple demo in the last 3 years that did not mention Xraids or Fibre connectivity.
And Apple and AJA only ever publish BASE MINIMUM SPECS to let you know what the basic needs will be. NO mfg's I know up has ever posted the info you are requesting- it is not possible since every production has different needs. If you were working with a professional video integrator they would (or Should) have told you what was needed.
If you DIY it -its your fault for not doing the proper research.
[Mike Parfit]"Oddly, one of the other very effective things we've done is to take OUT 2 GB of ram, which takes our system down to just 2 GB total."
Mike- are you sure on your software / hardware specs?
this is not an issue on the OctoCores running leopard- it was the previous machines / OS combo that had the 2 gig Ram issues- I have not seen this since 10.5 and QT 7.4 were released even on the older machines.
gary adcock
Studio37
HD & Film Consultation
Post and Production Workflows
Inside look at the IoHD
Re: Final Cut Pro unusable for big projects? by walter biscardi on Jun 22, 2008 at 12:36:06 pm
[gary adcock]"I have not seen this since 10.5 and QT 7.4 were released even on the older machines."
We have on our Mac Pro Quad 3.0. That's the machine that really blew up and we had to remove 2GB RAM to make the machine work. Now we're at 8GB and it's happy.
Walter Biscardi, Jr.
Biscardi Creative Media HD and SD Production for Broadcast and Independent Productions.
Re: Final Cut Pro unusable for big projects? by gary adcock on Jun 22, 2008 at 2:55:00 pm
[walter biscardi]"We have on our Mac Pro Quad 3.0. That's the machine that really blew up and we had to remove 2GB RAM to make the machine work. Now we're at 8GB and it's happy."
That sounds more like the RAM issue rather than the memory buffer- if it was a buffer issue, adding extra ram would not resolve the problem.
Funny how many of these errors are fixed by just reseating the RAM in the MacPro's
gary adcock
Studio37
HD & Film Consultation
Post and Production Workflows
Inside look at the IoHD
Re: Final Cut Pro unusable for big projects? by Mike Parfit on Jun 22, 2008 at 3:38:02 pm
Two simple questions:
How will faster storage fix a problem of crashing while rendering? No journaling. No sleep, ever. No attached firewire drives. Lots of room on the SATA drives. Memory swapped and moved and checked and resettled. OEM memory, proper configuration.
How do you explain that the crashes only seem to occur when the real memory use in the Activity Monitor builds up over time during a render to a point beyond 2GB? Is that somehow related to storage? It seems counterintuitive to me to blame this on storage, but I have certainly been around computers long enough to understand that counterintuitive is sometimes right.
Re: Final Cut Pro unusable for big projects? by walter biscardi on Jun 23, 2008 at 1:46:31 am
[Mike Parfit]"How will faster storage fix a problem of crashing while rendering? No journaling. No sleep, ever. No attached firewire drives. Lots of room on the SATA drives. Memory swapped and moved and checked and resettled. OEM memory, proper configuration."
If you have media on multiple drives, the risk of corruption is greater. Hit one corrupted piece of media and crash.
If you have media on multiple drives there can be momentary lags when FCP switches from drive to drive to access a file. Can cause a crash.
There are a myriad of issues that can occur when you have media spread out across multiple drives.
There can be a myriad of issues if you update FCP during a project.
There can be a myriad of issues if you update Quicktime during a project.
[Mike Parfit]"How do you explain that the crashes only seem to occur when the real memory use in the Activity Monitor builds up over time during a render to a point beyond 2GB? I"
The only render issues I've seen are if you have exactly 2GB RAM in a machine and do a long series of renders, like a 20 - 30 minute section of timeline at once. I documented this in my blog about 2 or 3 months ago. This occurred on both our G5 Quad and Mac Pro Quad.
Walter Biscardi, Jr.
Biscardi Creative Media HD and SD Production for Broadcast and Independent Productions.
Re: Final Cut Pro unusable for big projects? by Andy Mees on Jun 23, 2008 at 5:48:10 am
[walter biscardi]"f you have media on multiple drives there can be momentary lags when FCP switches from drive to drive to access a file. Can cause a crash."
if true, a sad illustration of the fallibility of technology ie when computers just crash for no damn good reason
Re: Final Cut Pro unusable for big projects? by gary adcock on Jun 22, 2008 at 11:35:43 am
Hello Christoph,
I am with Walter on the media, also i am guessing that the audio in on the externals? That would create a large issue with the timing due to the seek speeds needed- the files may be smaller but the storage needs to be pretty fast to support seeking for the small files.
Your crashes may be the result of the sata drives spinning down when not in constant use, when FCP reaches for them it cannot find the media in a place it expects- hence the crash.
More than a few major motion pictures have been cut on FCP, not to mention a much larger number of Episodic TV shows that are done with volumes of files larger than this. I have heard of a couple 1 hr TV shows that deal with this much data on a weekly basis.
[Christoph Strothjohann]"only leopard, quicktime, itunes, final cut studio2, and the aja software installed. "
Doubt that those are the only apps
What about Journalling for the media?
No Spaces or desktop switching
- and I have always recommended NOT having itunes open and running while editing.
gary adcock
Studio37
HD & Film Consultation
Post and Production Workflows
Inside look at the IoHD
Re: Final Cut Pro unusable for big projects? by James Sullivan on Jun 22, 2008 at 9:26:50 pm
I am going to ask a dumb question. Did you start the project at the same version of Final Cut? Upgrading anything of that size, even a dot upgrade can wreak havoc and sink ships off the coast of small countries I have been told. Nasa images of the day are photoshopped right?
I understand the risk to dig deep into gear at what could be a long drawn out indy situation. I currently have a wonderfully stable SD system. I have been waiting for "HD" to figure itself out. Blue-ray being the final piece that still sucks. I do not have clients that need it and therefor I am holding out on Encore/Fastmac internal drive/Toast. I can wait while others cannot bless them.
On the other hand, at lease you have a new system. I have nightmares about working on a project captured on firewire with second system sound that started on an ibook and is now doing a film out. Forget Jaws, or Saw Part 14 "Home Despot", Final Cut lets you mess up and it has come to bite me more then once. It sounds like you were able to put together a decent edit suite and can finish this puppy to much applause and knowledge gained. There is too much to know about HD, post, and the future. I am finding out that patience and planing are more fundamental to money and time. This is where the Cow (Thank your creator), friends, and people who have done it before come in handy.
There was another post recently that reminded me that we are in this for the artistic creation end of things and not so much the tools. Dropped frames with producers in the room still blows.
Good luck and get that puppy out the door to be downloaded and crunched to all hell when it hits cable and ipods everywhere!
James
P.S. I am in TV please do not take any offense if in fact you are creating art. The real cost of reality television is the depletion of one's soul one piece at a time.
Re: Final Cut Pro unusable for big projects? by Rafael Amador on Jun 23, 2008 at 5:45:42 am
Jounaling helps the system to keep track of the files in the HDs, but also can slow down the process.
In Hds with many (many) files is convenient to have the Jounaling ON. HDs used for media storing, where you can have few hundreds or few thousand of files, do not really needs Journaling.
I have jounaled the System HDs (that is a must) and an external USB HD where I save docs, pictures, music etc. The media HDs, all Un-journaled.
Cheers,
rafael
Re: Final Cut Pro unusable for big projects? by Kevin Monahan on Jun 23, 2008 at 6:36:59 pm
The more complex your show is, the more hardcore you have to be about the following:
System Config
Drive Speed
Quality and proper config of RAM
System Maintenance
Project management
I agree with all these guys. Since I opened my shop with FCP 1, I've always been a purveyor of OVERKILL. Especially when it comes to drive speed. Even back then, I was able to play out a broadcast quality SD output to Beta SP with the ancient Targa 2K pro while others were doing mere DV. The secret was a high speed SCSI array at RAID 0.
It is a huge mistake not to build a system around a high speed data array if you are doing large or complex projects. This has always been the case. Doing good research indicates this and always has since FCP 1. Rule of thumb? If you are adding a capture card to your system, you must add a high speed data array of some kind.
Walter, Gary, Raf and others. Thank you for what is working out to be a great post and even better advice for Christoph.
the truth: FCP handles big projects badly. When we edited our feature we came across some of the same problems, however not so severe. When you get a lot of small clips in your project, fcp gets more and more unpredictable. I have written about this a couple of times. It is really annoying that Avid handles this so much better. FCP is fantastic, but big projects are one of it's biggest weaknesses. When we had screenings, we had to make self-contained movies to be sure nothing screwed up.
Let's hope the next version of FCP fixes this BIG problem.
Re: Final Cut Pro unusable for big projects? by Christoph Strothjohann on Jun 25, 2008 at 10:02:39 am
thanx walter, mike, gary and all you others for the active discussion on this post and all your advice.
one thing upfront to walter. by labelling the post "final cut pro unusable for big projects?"
i didn´t want to state but rather ask (hence the "?") if it the size of our projects is where our problems stem from. but english is not my mothertongue, so I´m sorry, if somebody finds it to provoking and was upset about it.
we talked to the production company and took your advive, walter, to order a raid.
i hope it will arrive on friday and we´ll copy our media to the drive over the weekend.
there was no money for top class fibre channel etc.
so we ordered a sonnet d800 with 8tb storage for ca 4500 dollars without VAT.
as soon as it´s up and running i will give feedback on this post on how it affects our working.
one question about "good" media management with raid: shall we put all media files on there (video, sound, render files) or would it be wiser to keep the sound files on a seperate internal sata drive?
journaling is of course turned off for all our media drives. so that is not where our problems come from.
no updates were made either since the main project was set up.
although the problems we encounter are similar to mike´s, we haven´t run into render issues.
just yesterday i rendered the widescreen effect, video and audio transitions on a 2hour sequence in one go without any problems.
so i will be back in a few days with a description on how our switch to the raid went.
