FORUMS: list search recent posts

Slowwwww sparse bundles?

COW Forums : Apple Final Cut Pro X Debates

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Ian Johnson
Slowwwww sparse bundles?
on Jul 9, 2013 at 11:39:17 pm

We've been using Sparse Bundles to cut with FCPX on our server. It works fairly well overall, but they're so slow!! We usually see speeds of around 115MB/s both ways (whether transferring files or cutting in FCP7), but when working off a sparse bundle the speed averages drop to around 40MB/s, but sometimes can drop to as low as 15MB/s. Fifteen! This usually doesn't affect performance too much, but once you start layering streams or simply copying files in and out of them, you definitely feel it.

Has anyone noticed this? I can't find a peep about it anywhere on the forums. But with all the people now using FCPX and sparse bundles there's no way we're the only company to run into this...


Return to posts index

Nicholas Zimmerman
Re: Slowwwww sparse bundles?
on Jul 10, 2013 at 12:00:14 am

I use Sparse Bundles on a local RAID and have never experienced this problem, even with some bundles reaching over 2TB.

--------------------------
Avid MC, PPro CS6, FCP7 - wasting away on my SSD.
I just can't quit X.
--------------------------


Return to posts index

John Davidson
Re: Slowwwww sparse bundles?
on Jul 10, 2013 at 12:51:42 am

Are you storing your bulky media in them? I always suggest keeping media in an outside, referenced location.

Right now I'm getting 140 MB/s write and 73.7 MB/s reads on a server mounted Sparse Bundle.

John Davidson | President / Creative Director | Magic Feather Inc.


Return to posts index


Ian Johnson
Re: Slowwwww sparse bundles?
on Jul 10, 2013 at 5:29:37 pm

The only media we store in the bundles are proxies and the odd still/SFX/music track. All original raw media sits in a separate, non-bundled folder on the server that operates at full speed.

As a result, the bundles are usually under 200GB.


Return to posts index

Bill Davis
Re: Slowwwww sparse bundles?
on Jul 10, 2013 at 5:50:43 pm

It seems to me that there's some confusion about how SparseBundles work here.

Just so people who are considering the workflow understand, the SparseBundle package by itself is a storage container. When you LAUNCH the bundle by double clicking on it - it opens on the desktop in the finder as if the original disk itself has been mounted. In the launched state, it operates as fast as the bus allows - exactly as if you've inserted the original card.

So the HD or LAN connection between the device the Sparse Bundle resides on - and the throughput of the connection being used to mount that drive will be the limiting factors in the speed.

Hope that helps.

Know someone who teaches video editing in elementary school, high school or college? Tell them to check out http://www.StartEditingNow.com - video editing curriculum complete with licensed practice content.


Return to posts index

Ian Johnson
Re: Slowwwww sparse bundles?
on Jul 10, 2013 at 8:33:16 pm

Hey Bill,

Thanks for the input. I don't think there's any confusion (at least on our end) on the function of the bundles. The issue that we're having is that the read/write speeds to and from the bundles are are far lower than what the bus allows and operates at in all other cases. We're trying to figure out where the bottleneck is...


Return to posts index


Jason Jenkins
Re: Slowwwww sparse bundles?
on Jul 10, 2013 at 8:49:56 pm

There is definitely a speed difference. Here is one test I ran:

AJA System Test

Volume: Promise Pegasus R6
Video Frame Size: 1920x1080
File Size: 1.0 GB
Write: 317.7 MB/s
Read: 446.8 MB/s

Volume: 20 GB Sparse Bundle on Promise Pegasus R6
Video Frame Size: 1920x1080
File Size: 1.0 GB
Write: 266.0 MB/s
Read: 236.7 MB/s

Jason Jenkins
Flowmotion Media
Video production... with style!

Check out my Mormon.org profile.


Return to posts index

John Heagy
Re: Slowwwww sparse bundles?
on Jul 10, 2013 at 9:38:20 pm

[Ian Johnson] "We've been using Sparse Bundles to cut with FCPX on our server."

If your server can export an NFS share it will act exactly like a SAN to FCPX with all the SAN Location functionality intact.

John


Return to posts index

Eric Newbauer
Re: Slowwwww sparse bundles?
on Jul 10, 2013 at 10:12:30 pm

Ian, do you see such a dramatic slowdown with a new and unused sparse bundle?

Eric Newbauer
Studio Network Solutions (SNS)
http://www.studionetworksolutions.com


Return to posts index


Ian Johnson
Re: Slowwwww sparse bundles?
on Jul 11, 2013 at 3:56:37 pm

Eric,

They seem to behave similarly no matter how fresh and unused. Though I haven't tested this thoroughly enough to be 100% sure it's not a factor. Is this something that might affect performance?


Return to posts index

Craig Alan
Re: Slowwwww sparse bundles?
on Jul 11, 2013 at 6:10:21 pm

Later this summer i will be setting up a 20 computer FCP X lab and plan on using sparse images as a way to control ownership of projects/media and make it easier to move projects to computers in the studio and elsewhere.

Would love to hear/understand best practices to do so. I plan to put all media associated with an active project in the same sparse image rather than having the original media in its own location. Is it best that the events folder itself only point to the media in a different folder(s)?

Based on above and doing more research on google I will switch to sparse image 'bundles' since they copy faster, shrink easier, and can get corrupted in a smaller chunks rather than the whole image going down. Disk warrior is probably a great app for them either way.

Question: I read somewhere that it is best when you begin to use a new sparse image to let FCP X create the folders (Final Cut Events/Final Cut Projects) instead of creating them in the finder. Is this true? I know just putting new media in them in the finder doesn't work but I haven't noticed any problems creating the folders - though maybe something is duplicated?

Mac Pro, macbook pro, Imacs (i7); Camcorders: Panasonic AG-HPX170/AG-HPX250P, Canon HV30/40, Sony Z7U, VX2000, PD170; FCP 6 certified; write professionally for a variety of media; teach video production in L.A.


Return to posts index

Michael Hadley
Re: Slowwwww sparse bundles?
on Jul 12, 2013 at 12:14:06 am

Here's a thought. Ever run Disk Warrior or similar utilities on a sparse bundle? Don't know if it's possible but maybe that might help improve speeds?


Return to posts index


Eric Newbauer
Re: Slowwwww sparse bundles?
on Jul 16, 2013 at 3:57:21 pm

Hi Ian,

Yes, usage can affect the performance of your sparse volume. Counterintuitively, the performance problem can be caused by fragmentation. (This is something we've seen in our lab and can reproduce easily by just using generic stuff.)

FWIW, I'd like to point out that our EVO storage server also provides NFS. Please have a look if this is of interest to you:

http://www.studionetworksolutions.com/evo/features/

Eric Newbauer
Studio Network Solutions (SNS)
http://www.studionetworksolutions.com


Return to posts index

Craig Alan
Re: Slowwwww sparse bundles?
on Jul 16, 2013 at 4:53:04 pm

I ran some tests using both AJA and Black Magic software. All my media is on a 4TB internal raid on a Mac Pro. I have projects stored In Sparse Images. The results were that the sparse image read/write speeds were very close to the parent raid speeds. Sometimes faster. What would be considered good read/write speeds on a software raided media drive? Or more importantly good speeds for editing in FCP X?

And related but a different problem -

I need a way for students in a FCP X lab to NOT have access to each other's media and projects. The only way I have come up with so far is storing the projects in sparse images and setting up a student user account with simple finder in OS X. In simple finder, sparse images are grayed out when you do an import search in FC. And completely invisible in the finder. I'm hoping there are no gotchas. This would be whole lot easier than setting up tons of user accounts and setting up permissions for every external folder.

If I mount a sparse image using the admin account then the sparse image shows up in FC. They could store all their files in that sparse image and then log out at the end of their session. Then I unmount the image and load the next one.

This will also make help transfer user projects to a different computer or to the trash or to a back-up. The media will be stored on an external raid. I have not tested this on external storage. The media drive I'm using is a software raided drives in a Mac Pro.

My first plan was to use password protected sparse images but copying media to an encrypted image takes way too long.

Do you see any problems with this plan and would it work equally well with a server?

Mac Pro, macbook pro, Imacs (i7); Camcorders: Panasonic AG-HPX170/AG-HPX250P, Canon HV30/40, Sony Z7U, VX2000, PD170; FCP 6 certified; write professionally for a variety of media; teach video production in L.A.


Return to posts index

Caspian Brand
Re: Slowwwww sparse bundles?
on Jul 16, 2013 at 8:35:56 pm

[Craig Alan] "I need a way for students in a FCP X lab to NOT have access to each other's media and projects. The only way I have come up with so far is storing the projects in sparse images and setting up a student user account with simple finder in OS X. In simple finder, sparse images are grayed out when you do an import search in FC. And completely invisible in the finder. I'm hoping there are no gotchas. This would be whole lot easier than setting up tons of user accounts and setting up permissions for every external folder.

If I mount a sparse image using the admin account then the sparse image shows up in FC. They could store all their files in that sparse image and then log out at the end of their session. Then I unmount the image and load the next one.

This will also make help transfer user projects to a different computer or to the trash or to a back-up. The media will be stored on an external raid. I have not tested this on external storage. The media drive I'm using is a software raided drives in a Mac Pro.

My first plan was to use password protected sparse images but copying media to an encrypted image takes way too long. "


Hi Craig,

Keep in mind FCP X only allows for one user per Event/Project period. No Read users, regardless of network file system. You can, however, import media w/o copying from another active Event/Project. Not that it sounds like you're looking for simultaneous users in the same Project like an Avid workflow, but just stating for the record on how FCP X behaves in general.

The workflow you describe actually lends itself very well to a volume managed solution. We have a software product called SANmp which enables workspaces to be assigned per student just as you describe over iSCSI. Permissions can be easily set so that student volumes would only be visible to the assigned student and their instructor, not by other classmates, regardless of which computer the student logs in from. SANmp is very popular in educational media production environments like this for it's ease of setting up such permission schemes as you described.

Common class media can be centrally stored on it's own volume(s) for Read Only access by all students and Write access for the instructor. A CLI interface is even included for scripting so you can auto-mount volumes with a specific user's account. It is available as stand-alone software to manage access to iSCSI targets with your existing storage, as well as with our turnkey Shared Storage platform called EVO.

With our more advanced EVO solution, SANmp is included free at no extra per-seat cost. EVO provides more advanced file level managed workflows with NFS, as well as AFP and SMB in addition to high performance media workspaces using SANmp. With EVO you can:
-configure an all Sparse Disk Image workflow over NAS protocols
-configure an all high performance media sharing workflow with SANmp volumes per Class/Student
-configure a combination of the two, with central/common elements for everyone on a high performance media volume and student specific edits/renders/projects on Sparse Disk Images
-configure XSAN over iSCSI and use the FCP X 'Add SAN Location' in conjunction with XSAN volumes/permissions

As has been mentioned by others on this thread, performance of Sparse Disk Images is not as consistent when introduced in a network compared to using them on local disks, so care should be taken on a properly configured RAID as well as keeping your production network 'clean' and keeping your storage system over as direct a path to the edit systems as you can without traversing a bunch of routers or switches.

Regards,
-Caspian

Product Specialist
Studio Network Solutions


Return to posts index


Craig Alan
Re: Slowwwww sparse bundles?
on Jul 16, 2013 at 10:46:38 pm

[Caspian Brand] "Keep in mind FCP X only allows for one user per Event/Project period. No Read users, regardless of network file system"

On the mac pro I have been experimenting with, I set up a student account with a simple finder. If a sparse image is mounted on the teacher account (the account that it was created in), the student account can open it's events and projects.

Or do you mean allow one user at a time?

Will I be able to use Apple Remote desktop to see what they are doing in an open project and help them if needed?

[Caspian Brand] "performance of Sparse Disk Images is not as consistent when introduced in a network compared to using them on local disks, so care should be taken on a properly configured RAID as well as keeping your production network 'clean' and keeping your storage system over as direct a path to the edit systems as you can without traversing a bunch of routers or switches."

I have 20 computers and external raids for each. Since I had heard that FCP X was not working well in networked facilities, I decided to keep each computer paired with its own media drive. Also I did not get the funding all at once so it was put together piecemeal.

Is there a problem with sparse image performance even on a thunderbolt connected raid to a single computer?

Your solution is definitely‎ networked : so are you using volumes within the server storage instead of sparse images with software to assign ownership to a volume? Are sparse image speeds compromised?

Could we use the thunderbolt raids in such a system? Maybe daisy chained together as a large network storage. They are pegasus raids with four hard drives each.(8TB total, but can be changed out).

What would be a ballpark figure for this type of package for a 20 seat computer lab? Please keep in mind that in our school district we do not have a budget. What I do is apply for grants and I need to know ahead of time how much things cost so I can put in a request long before I get to order anything. It's a very time consuming, overly complicated way to fund programs. But it is what it is.

Mac Pro, macbook pro, Imacs (i7); Camcorders: Panasonic AG-HPX170/AG-HPX250P, Canon HV30/40, Sony Z7U, VX2000, PD170; FCP 6 certified; write professionally for a variety of media; teach video production in L.A.


Return to posts index

Eric Newbauer
Re: Slowwwww sparse bundles?
on Jul 18, 2013 at 3:04:15 pm

[Craig Alan] "Your solution is definitely‎ networked : so are you using volumes within the server storage instead of sparse images with software to assign ownership to a volume? Are sparse image speeds compromised?"

Hi Craig, we (SNS) have a few different ways of tackling FCPX workflows with our storage products, but the one Caspian mentioned is based on using our SAN volume sharing software (SANmp), and because of that we don't have to deal with sparse bundles. It's very simple this way, and as Caspian pointed out it's a very popular solution for schools.


[Craig Alan] "Could we use the thunderbolt raids in such a system? Maybe daisy chained together as a large network storage. They are pegasus raids with four hard drives each.(8TB total, but can be changed out)."

Yes. We also have a product (Xtarget) that will enable you to take Thunderbolt storage and present it over Ethernet as SAN storage. You can then use your Thunderbolt storage with our SAN volume sharing software that I mentioned earlier.


[Craig Alan] "What would be a ballpark figure for this type of package for a 20 seat computer lab?"

There are several different ways to solve the problem you're describing. We could do it all in software and you can use your own storage, or we could put together an entire solution: hardware and all. I'll add some links below so you can try out our stuff for free, and if you like it please give us a call and chat with one of our reps about costs.

Here's a link to our volume sharing software (SANmp):
http://www.studionetworksolutions.com/sanmp/

Here's a link to the software that enables you to present direct attach devices as SAN storage (Xtarget):
http://www.studionetworksolutions.com/globalsan-xtarget/

Eric Newbauer
Studio Network Solutions (SNS)
http://www.studionetworksolutions.com


Return to posts index

Ivan Radovanovic
Re: Slowwwww sparse bundles?
on Jul 11, 2013 at 6:30:10 pm

I'm not FCP X editor but I wonder why is everybody ignoring NFS as an option when a lot of servers can export NFS share?
I have tested with our NAS couple months ago, 1G connection, 110 MB/s Read and Write. Sparse bundles are always slower.


Return to posts index


Ian Johnson
Re: Slowwwww sparse bundles?
on Jul 11, 2013 at 7:56:58 pm

I had never considered (or configured) NFS.

Our current setup is a Mac Pro with a set of SAS raids sharing over a LAN. Can NFS be setup on a system like this?


Return to posts index

John Heagy
Re: Slowwwww sparse bundles?
on Jul 11, 2013 at 9:48:33 pm

The short answer is yes absolutely.

It helps if it's OS X Server and 10.6 where it's a simple file sharing setting. In 10.7 and 10.8 Apple has eliminated this setting so a bit of Terminal work is needed or use NFSManager.

http://www.bresink.com/osx/NFSManager.html

John


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]