FORUMS: list search recent posts

FCPX developers, please optimize disk access

COW Forums : Apple Final Cut Pro X Debates

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Ladislav Zamba
FCPX developers, please optimize disk access
on Jun 30, 2012 at 7:23:42 am

Hello,

I was editing large project with about 500 clips on timeline.
All media files (ProRes FullHD) I copied from internal to external drive and relinked all clips from timeline to external drive. Event and Project folders remained on my internal drive.

External drive was connected via USB and then I realized how often is FCPX accessing original media :-(

I'm using latest 10.0.5 version and with my large project (500 clips, 3 hours video) FCPX was accessing original media after EVERY EDIT action I did.

It was not problem with smaller projects, but with this large project it was really pain to see disk working 15 seconds after every edit action.
It was also working when I selected clip on timeline and hit Cmd-C :-(

My question: why is FCPX always accessing original media? Is it necessary always checking clips used on timeline?

Thank you


Return to posts index

Ben Holmes
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 11:56:37 am

USB2 or USB3?

Edit Out Ltd
----------------------------
FCP Editor/Trainer/System Consultant
EVS/VT Supervisor for live broadcast
RED camera transfer/post
Independent Director/Producer

http://www.blackmagic-design.com/community/communitydetails/?UserStoryId=87...


Return to posts index

Ladislav Zamba
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 12:47:52 pm

USB2


Return to posts index


Davee Schulte
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 1:45:07 pm

When you choose to optimize media, it creates the ProRes files. Once the media is optimized, it automatically switches over and refers to the new optimized (pro res) media files. The media should definitely be on an external drive but you should stick with firewire (or thunderbolt).


Return to posts index

Ladislav Zamba
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 1:57:49 pm

Why optimize when my timeline is ProRes422 and media files are too?

Why is FCPX so heavily accessing all media files with every edit?

It's not problem to buy thunderbolt external RAID. But what if I will have 30 hours timeline with 5000 clips?

All I ask is to optimize this step. I think that FCPX is doing something unnecessarily.


Return to posts index

Oliver Peters
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 2:16:54 pm

I don't understand the issue. If you are editing with the media linked to the original files (which is what is sounds like you are doing), then yes, original media will be accessed with every edit. That's the way every NLE works. USB is too slow to efficiently work with HD media.

- Oliver

Oliver Peters Post Production Services, LLC
Orlando, FL
http://www.oliverpeters.com


Return to posts index


Ladislav Zamba
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 2:22:56 pm

Yes, it's my case.

But why it is so? Why it is accessing all clips? What is it checking? If they still exist? Alter every edit?
If I append new clip to the end of timeline how does it affect existing clips on timeline?

I hope taht FCPX developers read this thread and make some little change :-)
I want to help make FCPX even better.


Return to posts index

olof ekbergh
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 3:11:26 pm

In order to play a clip a NLE needs to access the media that is linked to the timeline. How else would it play?

Also when you edit it needs to know about the clip. If you have thumbnails and audio waveforms visible in the timeline, those need to be updated as well, this is done by reading the media and analyzing it.

If your HD's and system is slow try editing proxy media. USB2 with ProRes would be pretty slow, possible even unwatchable.

I hope this helps a little.

Olof Ekbergh


Return to posts index

Ladislav Zamba
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 4:51:20 pm

I understand you, but my example:

Timeline has displayed all waveforms, disk is quiet. Then I trim last clip by one frame from right. Disk is working 15 seconds and then it is quiet again. Then I trim same clip by one frame again and disk is working another 15 seconds. I don't think that it is optimal behaviour and I don't understand reason for this disk activity.

Remember - disk contains media clips only. No project or event files.


Return to posts index


Dave Jenkins
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 5:05:50 pm

It sounds like your disc is going to sleep and when you access the clips it waking back up.
Are you working on a laptop?
Is the box in the energy saver preference set to "Put hard disks to sleep when possible" checked?

Dajen Productions, Santa Barbara, CA
MacPro Two 2.8GHz Quad Core - AJA Kona LHe
FCS 3 OS X 10.6 QT 10


Return to posts index

Ladislav Zamba
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 5:07:02 pm

No, it's not going to sleep.


Return to posts index

Bill Davis
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 5:50:56 pm

Lasislav,

Perhaps this will help you look at things in a different light...

One of the absolutely best things about X is that it's taking constant "snapshots" of all the decisions you make - every edit, every color correction, every sound change - it tracks the state of your project.

All FCP-X does is track text changes. That's the metadata that everyone is always talking about.

So you can think of it as whenever you point to ANYTHING in the X interface, the program must refer back to the original text that describes the state of the thing you've pointed to.

So access to "historical" data is critical - since every new change has to be linked to that. Always.

That's the problem it sounds like you're facing. Your system is extremely slow to access the historical editing data X needs to attach your new instructions. So it's driving you nuts.

You might try using the MOVE command to re-locate the PROJECT and EVENT files you're working with to the same drive as your footage. It's possible that if the machine can locate everything it needs in the same place, it might fractionally speed things up since that drive is where the controllers will look for everything and therefore it will never go to "sleep" and slow things down more.

But at the heart of things, you're simply working with a system that's doesn't shuffle enough data around fast enough to keep up with the editing your doing.

As others have noted, I think that USB-2 just isn't fast enough to work with anything larger than Proxy files.

Good luck.

"Before speaking out ask yourself whether your words are true, whether they are respectful and whether they are needed in our civil discussions."-Justice O'Connor


Return to posts index


Ladislav Zamba
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 6:21:25 pm

Bill and all guys here,

I appreciate your answers. I work with high speed RAIDS. This USB2 experience was extreme a occasioanal case.

As old SQL database developer I'm aware of not optimal programming. For example not optimal SELECTs work fine with small databases on speed hardware, but with thousands of gigabytes of data such SELECTs are weak. And if thousands of users run such SELECTs at same time, servers are in trouble then.

My USB2 experience tells me that something can be made better.

Do you remember problem with doubling project file size when compound clip was made? Apple fixed it.
So, I believe that FCPX developers can avoid some unnecessary disk activity and make editing on slow drives and machines faster.


Return to posts index

Andrew Richards
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 8:31:43 pm

[Ladislav Zamba] "As old SQL database developer I'm aware of not optimal programming. For example not optimal SELECTs work fine with small databases on speed hardware, but with thousands of gigabytes of data such SELECTs are weak. And if thousands of users run such SELECTs at same time, servers are in trouble then."

As a SQL developer you must also know that the most effective way to accelerate a database is to give it lower latency storage to operate on. Depending on the size of the database, that is often a matter of loading the entire thing into RAM. Moore's Law and Brute force have been the saving grace of accelerating large databases for a long time, and the same is true for a 64-bit, hardware-optimized NLE.

[Ladislav Zamba] "My USB2 experience tells me that something can be made better."

My USB2 experience tells me that it is some of the slowest, highest-latency storage you can get short of trying to edit over 100BASE-T.

[Ladislav Zamba] "Do you remember problem with doubling project file size when compound clip was made? Apple fixed it. So, I believe that FCPX developers can avoid some unnecessary disk activity and make editing on slow drives and machines faster."

Maybe they could if you had a huge glut of RAM they could load more of the utilized source media into, since they have to store the timeline media somewhere if it isn't going to be read from disk when called upon. However, for a large project, this too is untenable. Think about how much RAM you have, and then how much media you are using in your large project's timeline. I can't imagine what they could do to achieve what you are describing. Sometimes you just need better hardware.

That said, I do agree they need to optimize how FCPX assets are stored- specifically, allowing for separation of Render media from the Project database storage. The Events and Projects databases will be much happier on SSD, while the media can tolerate higher density, lower cost HDD arrays that are built for streaming media and not so good with the IOPS.

Best,
Andy


Return to posts index

Ladislav Zamba
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 9:09:10 pm

Andy - I have Project and Event folder on internal drive. Only original media files was on USB2 disk.

And with SQL optimisation I was pointing to sequential vs. index searching in tables. Even if you have modern servers with lot of RAM, it's not always possible to put 500GB table into RAM :-)

So, modern HW is good to have, but programmers must optimize their code always.
What if Apple is planing to implement multiuser access to Events. Imagine 100 editors accessing same media at same time.

I still hope that there is no reason for FCPX to access all original media used on timeline.


Return to posts index


Andrew Richards
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 10:02:07 pm

[Ladislav Zamba] "Andy - I have Project and Event folder on internal drive. Only original media files was on USB2 disk."

I understand, but I maintain that a single USB2 HDD is usually inadequate media storage for Pro Res. What brand/model USB2 drive are you using? They vary a lot.

[Ladislav Zamba] "And with SQL optimisation I was pointing to sequential vs. index searching in tables. Even if you have modern servers with lot of RAM, it's not always possible to put 500GB table into RAM :-)"

Yes, developers should always optimize their code to be efficient, I don't dispute that. All I'm arguing is that hardware is often a much more significant variable when it comes to NLE responsiveness. Software tuning will only get you so far.

500 GB tables? No problem. Now if you're getting into the 10s of TBs, then you'll really have to spend some money!

[Ladislav Zamba] "What if Apple is planing to implement multiuser access to Events. Imagine 100 editors accessing same media at same time."

Oh, that's easy. Xsan (StorNext) can scale any which way you need. It ain't cheap, but if you really have 100 editors, it can be built to handle it.

I stand by my original point- hardware performance is a lot more of a factor than software optimization when it comes to NLEs. Software optimization is very important, and should certainly be stressed, but at some point you bump up against physics and you just need a bigger boat.



Best,
Andy


Return to posts index

Ladislav Zamba
Re: FCPX developers, please optimize disk access
on Jun 30, 2012 at 10:18:48 pm

It was desktop 3.5'' 7200 rpm drive with 32MB cache in USB2/eSata enclosure

BTW playback and skimming was fine and acceptable.


Return to posts index

Rafael Amador
Re: FCPX developers, please optimize disk access
on Jul 1, 2012 at 3:35:44 am

USB 2 is not even up to the task of capturing DV.
The minimum storage requirement for FC historically has been FW400.
USB2 offers a similar transfer rate but with many peaks.
rafael

http://www.nagavideo.com


Return to posts index

Davee Schulte
Re: FCPX developers, please optimize disk access
on Jul 1, 2012 at 4:33:22 am

I have the program on internal drive, project and render files on on a second drive and then all media on a larger third drive. Regardless, I think you're fighting the obvious. USB for video isn't fast enough, never has been. Hence the importance of firewire. I doubt that Apple will take the time to optimize FCPX for USB2 when its looking to thunderbolt for the future.


Return to posts index

Ladislav Zamba
Re: FCPX developers, please optimize disk access
on Jul 1, 2012 at 5:43:02 am

I only wanted to say that working with large timeline on very slow media disk can reveal some bottlenecks in SW too.


Return to posts index

Scott Sheriff
Re: FCPX developers, please optimize disk access
on Jul 1, 2012 at 10:29:58 pm

[Ladislav Zamba] "So, I believe that FCPX developers can avoid some unnecessary disk activity and make editing on slow drives and machines faster."

Sure, they could. But why?
Isn't the whole concept of X to be a "new paradigm in editing"? Isn't it about moving into the future? Once you sign on to a deal like X, it is not a smorgasbord. You have to take it all, including frequent hardware upgrades to the latest and greatest in order to keep up.
When you (FCP X users) start saying developers need to write code in order to accommodate your (FCP X users) hardware deficiencies, or even that the app isn't getting everything out of the hardware that it can, well this sounds like the previous FCS/FCP7 users complaints.
need to upgrade and move into the future.
Oh, BTW apple will be happy to supply you with a faster machine and drives.

Scott Sheriff
Director
http://www.sstdigitalmedia.com


"If you think it's expensive to hire a professional to do the job, wait until you hire an amateur." ---Red Adair

Where were you on 6/21?


Return to posts index

Ladislav Zamba
Re: FCPX developers, please optimize disk access
on Jul 1, 2012 at 5:53:43 am

Larry Jordan has new article

http://www.larryjordan.biz/app_bin/wordpress/archives/1951#utm_source=feed&...

And there is workaround:

"However, for larger projects, compound clips that aren’t nested inside each other can actually make the system run better. If you are loading lots of clips into the Timeline, Final Cut needs to track each of these clips individually. Instead, if there is a section of the Timeline that you are done editing, select all the clips in that section and convert it to a compound clip using File > New Compound Clip.

This tells Final Cut to treat those clips as a group. This improves memory management and overall performance, especially as projects and clip counts get larger."


Return to posts index

Rafael Amador
Re: FCPX developers, please optimize disk access
on Jul 1, 2012 at 9:06:48 am

Whatever the way a NLE may works, USB is not up to the task of moving three Prores streams in real time.

[Ladislav Zamba] ""However, for larger projects, compound clips that aren’t nested inside each other can actually make the system run better. If you are loading lots of clips into the Timeline, Final Cut needs to track each of these clips individually. Instead, if there is a section of the Timeline that you are done editing, select all the clips in that section and convert it to a compound clip using File > New Compound Clip."
This has been discussed in this forum months ago, but makes no sense to blame the NLE or the complexity of the project when you system do not reach the minimum performance.
Get rid of the USB HD.
rafael

http://www.nagavideo.com


Return to posts index

Jiri Fiala
Re: FCPX developers, please optimize disk access
on Sep 5, 2012 at 5:06:49 pm

Yeah, but - every other NLE has no issue whatsoever to play a single stream of, say, 720/25p ProRes from a measly USB2 drive. And simple cuts editing is fast.

FCPX has BIG issues working with larger projects, even from eSATA G-Tech RAID 0 with speeds over 130 MBps on a 8-core Mac Pro. Opening a keyword (bin) almost always slowly re-reads thumbnails, with RAID thrashing. When I switch to another bin and back, it SLOWLY rereads thumbs again. And most of my 22 GB RAM sits there unused. FCPX should be WAY more aggressive with RAM.


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]