FORUMS: list search recent posts

Sparse Bundle Users: How do you backup?

COW Forums : Apple Final Cut Pro X

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Jeff Kirkland
Sparse Bundle Users: How do you backup?
on Jul 23, 2013 at 12:03:50 am

This has come up in another thread and I thought I'd pull it out into into one of it's own.

How do those of you using sparse bundles for organising your projects manage backups? I loved using sparse bundles but recently stopped because I couldn't get them to back up reliably if they were open and in use - and it's not always practical to quit FCPX, then eject every sparse bundle that's open before a backup runs.

I use Chronosync to backup my 4tb RAID to another 6th RAID (larger so I can keep deletions for a few days) and it's set to run and sync changes every couple of hours.

As a test I ran ten consecutive backups of a typical project and two of the ten sparse bundle backups were corrupt. I don't consider a 20% chance of failure good enough so I had to stop using sparse bundles.

I can't always guarantee I'll be able to close a bundle before a backup runs, or I'll have to leave a bundle mounted so FCPX can do a long render - so how do you guys deal with your backups?

Is there a different approach or better software I should be using?

Jeff Kirkland | Video Producer | Southern Creative Media | Melbourne Australia
http://www.southerncreative.com.au | G+: http://gplus.to/jeffkirkland | Twitter: @jeffkirkland


Return to posts index

Nicholas Zimmerman
Re: Sparse Bundle Users: How do you backup?
on Jul 23, 2013 at 12:41:51 am

I tend to do it at the end of the day. When I'm taking a 6 hour or longer break I just move them onto my backup RAID. Because they are bundled I just have 1-2 to move and overwrite the previous days version. Of course, I also don't use Time Machine or any other automated backup. Always just preferred to do it myself I guess.

--------------------------
Avid MC, PPro CS6, FCP7 - wasting away on my SSD.
I just can't quit X.
--------------------------


Return to posts index

Jeff Kirkland
Re: Sparse Bundle Users: How do you backup?
on Jul 23, 2013 at 12:59:30 am

Sometimes keeping it simple is the best :-)

Experience tells me that it might not be wise to trust myself with a manual solution though :-/

Jeff Kirkland | Video Producer | Southern Creative Media | Melbourne Australia
http://www.southerncreative.com.au | G+: http://gplus.to/jeffkirkland | Twitter: @jeffkirkland


Return to posts index


Andreas Kiel
Re: Sparse Bundle Users: How do you backup?
on Jul 23, 2013 at 10:11:25 am

Jeff,

It's not a good idea to backup while things are in progress.

Quit FCPX and any other app before backups starts.
For your info: Apple's Time Machine is also based on Sparsebundles.

-Andreas

Spherico
http://www.spherico.com/filmtools

"He who fights with monsters should be careful lest he thereby
become a monster. And if thou gaze long into an abyss, the abyss will
also gaze into thee." - Friedrich Nietzsche, Beyond Good and Evil


Return to posts index

Brett Sherman
Re: Sparse Bundle Users: How do you backup?
on Jul 23, 2013 at 6:52:42 pm

Corruption of .fcpevent and .fcpproject files does seem to be a problem for me with incremental backups. I always make sure I run a backup of those files to another location. But this problem is why I want to ditch sparse bundles. If Apple just had a better way to manage and share projects and events. At this point sparse bundles are a necessary evil.



Return to posts index

Craig Alan
Re: Sparse Bundle Users: How do you backup?
on Jul 23, 2013 at 8:08:13 pm

I would back up your small project files using FCP itself.

http://help.apple.com/finalcutpro/mac/10.0.6/#verb6acb2fb

"Control-click your project in the Project Library, and choose Duplicate Project from the shortcut menu.

In the window that appears, type a name for the new (duplicate) project and choose your external storage device from the Location pop-up menu.

Select an option to specify what is duplicated:

Duplicate Project Only: Duplicates files specific to the project you selected. You may want to select this option if you use the same clips in multiple projects and plan to back up your events separately (perhaps in your backup copy of your entire Project Library and events). This option does not duplicate event media."

And to back up the entire sparse image, I would use CCC (or the like) to make a clone of it.

If it had been previously backed up and you haven't added any new media and you have at least two copies of the project file, I think you are pretty well covered. If the footage is valuable I would have a second back up of the raw footage on yet another drive.

This is the one thing I miss about tape. It was a pain to digitize but the tape itself was way more reliable and easier to store. A labeled tape cannot be mistaken for anything but what it is. I think as SSDs get bigger & better this might all become easier and more reliable. They have no moving parts. Run cooler. Etc.

Mac Pro, macbook pro, Imacs (i7); Camcorders: Panasonic AG-HPX170/AG-HPX250P, Canon HV30/40, Sony Z7U, VX2000, PD170; FCP 6 certified; write professionally for a variety of media; teach video production in L.A.


Return to posts index


Jeff Kirkland
Re: Sparse Bundle Users: How do you backup?
on Jul 24, 2013 at 6:10:02 am

Thanks everyone - but surely there must be a way of automating the backup?

This year I have lost disks in my RAID0 primary work drive twice - both times with absolutely no warning. With that in mind I want at least a daily complete clone of my work drive, preferably with deletions archived for a few days just in case.

The notion that I have to completely stop work and wait for a backup to finish is exactly why I've stopped using sparse images. Chronosync has the odd issue with the actual FCPX project and event files but warns me if there was a problem so I can quickly do a manual backup of just those two files. Only happens very occasionally though.

Looks like sparse images are still a no-go for me at least... i was hoping maybe there was something that could recognise a sparse image was open and skip backing it up until next time. A 48 hour old copy would be far preferable to a completely corrupt copy.

Jeff Kirkland | Video Producer | Southern Creative Media | Melbourne Australia
http://www.southerncreative.com.au | G+: http://gplus.to/jeffkirkland | Twitter: @jeffkirkland


Return to posts index

Jeff Kirkland
Re: Sparse Bundle Users: How do you backup?
on Jul 24, 2013 at 6:17:36 am

Thanks Andreas... I've not really looked at time machine because it doesn't have the control that Chronosync does.

Jeff Kirkland | Video Producer | Southern Creative Media | Melbourne Australia
http://www.southerncreative.com.au | G+: http://gplus.to/jeffkirkland | Twitter: @jeffkirkland


Return to posts index

Craig Alan
Re: Sparse Bundle Users: How do you backup?
on Jul 23, 2013 at 7:52:52 pm

As long as we are revisiting from other threads:

some end users are reporting that their sparse images are slower than the media drives they are in and a call to Apple Care pro app department got me a strong recommendation not to use sparse images - that they would slow things down. My tests on a raid with plenty of overhead and relatively small images housed in it did not show this however. I used Black Magic and AJA apps to test the speed of the raid and the sparse image. If anything the images were faster.

Do any of you experience speed/performance issues using sparse images in FC?

Mac Pro, macbook pro, Imacs (i7); Camcorders: Panasonic AG-HPX170/AG-HPX250P, Canon HV30/40, Sony Z7U, VX2000, PD170; FCP 6 certified; write professionally for a variety of media; teach video production in L.A.


Return to posts index


Jeff Kirkland
Re: Sparse Bundle Users: How do you backup?
on Jul 24, 2013 at 6:17:06 am

In the time I was using sparse bundles I didn't really notice any great speed difference. Just ran a test with Blackmagic's test app and the speed difference if any was negligible. This was on a 50% full Thunderbolt RAID drive. Write was 3mb/s slower and read was actually 10mb/s faster than the non-sparsebundle test.

i get similar feedback from other FCPX editors I know so while I have no doubt sparsebundles may be slightly slower sometimes, I think in general there's no great difference.

Jeff Kirkland | Video Producer | Southern Creative Media | Melbourne Australia
http://www.southerncreative.com.au | G+: http://gplus.to/jeffkirkland | Twitter: @jeffkirkland


Return to posts index

Jason Jenkins
Re: Sparse Bundle Users: How do you backup?
on Jul 24, 2013 at 6:30:49 am

Here is one test I ran:

AJA System Test

Volume: Promise Pegasus R6
Video Frame Size: 1920x1080
File Size: 1.0 GB
Write: 317.7 MB/s
Read: 446.8 MB/s

Volume: 20 GB Sparse Bundle on Promise Pegasus R6
Video Frame Size: 1920x1080
File Size: 1.0 GB
Write: 266.0 MB/s
Read: 236.7 MB/s

Jason Jenkins
Flowmotion Media
Video production... with style!

Check out my Mormon.org profile.


Return to posts index

Jeff Kirkland
Re: Sparse Bundle Users: How do you backup?
on Jul 24, 2013 at 6:48:33 am

Just to keep a sort-of apples and apples comparison, I downloaded AJA System Test and ran it again...

Volume: Thunderbolt (LaCie 2big 4tb RAID 0 about 50% full)
Video Frame Size: 1920x1080 (8bit)
File Size: 1.0 GB
Write: 258.4 MB/s
Read: 260.9 MB/s

Volume: Sparse Bundle (1tb / 24gb in use)
Video Frame Size: 1920x1080
File Size: 1.0 GB
Write: 252.1 MB/s
Read: 282.7 MB/s

Jeff Kirkland | Video Producer | Southern Creative Media | Melbourne Australia
http://www.southerncreative.com.au | G+: http://gplus.to/jeffkirkland | Twitter: @jeffkirkland


Return to posts index


Andreas Kiel
Re: Sparse Bundle Users: How do you backup?
on Jul 24, 2013 at 7:54:59 am

@ Jason
That's a pretty poor result.

I never got results that had been less than 15% off the volume speed.
Since sparsebundles are only directories (with a little overhead) there is no reason why the should be way slower than the volume itself.

@ Jeff
I personally never got a corrupted sparsebundle and I'm using them for many things since many years (even before X).
I don't know about Chronosync as I a use CCC since the day it was released. But there can't a big difference between those apps cause both (and all backup apps) just use the system Unix calls.
Normally during the backup the checksum of files are compared so nothing can happen (in most cases) if the source file is okay.

A good idea though is to unmount sparsebundles during backup times, they might be busy for whatever reason and this might influence the way they are "backuped".

-Andreas

Spherico
http://www.spherico.com/filmtools

"He who fights with monsters should be careful lest he thereby
become a monster. And if thou gaze long into an abyss, the abyss will
also gaze into thee." - Friedrich Nietzsche, Beyond Good and Evil


Return to posts index

Brett Sherman
Re: Sparse Bundle Users: How do you backup?
on Jul 24, 2013 at 10:04:24 am

For me the sparse bundle is okay, media is okay, but the event and project files can get corrupted. This has happened on more than one occasion. This is with the bundles unmounted. I think there is an issue with incremental backups of sparse bundles and the way FCP X modifies the database files. I'm not 100% sure, but I think it may be more likely to happen when the backup sparse bundle is launched and opened in FCP X in between backups.



Return to posts index

Dave Gage
Re: Sparse Bundle Users: How do you backup?
on Jul 25, 2013 at 1:38:09 am

Brett,

This may have nothing to do with your issue, but have you tried running DiskWarrior on the Sparse Bundle? I don't use them, but I seem to remember hearing that DiskWarrior should work find on Sparse Bundles as well as standard drives.

Early on with FCPX, I major issues with external drives used solely for use as FCPX drives getting corrupted to the point where I had to wipe them clean and start over. I then began to use DW regularly and have not had a problem since. As I said about a year and a half ago, it "appeared" that FCPX was hosing my external RAIDs, which might have translated into the drive directory structures becoming a mess. All is fine now.

Dave


Return to posts index


Craig Alan
Re: Sparse Bundle Users: How do you backup?
on Jul 25, 2013 at 7:12:22 am

[Dave Gage] "This may have nothing to do with your issue, but have you tried running DiskWarrior on the Sparse Bundle? I don't use them, but I seem to remember hearing that DiskWarrior should work find on Sparse Bundles as well as standard drives."

I've heard that too but is there any issues with DiskWarrior and FCP X file structure? Now that I think about it, I used it on my system drive when running FCP 4-7 where the project file was on the system drive and never had an issue. But FCP X has a completely different type of file. Hmmm. Gonna do a search.

Mac Pro, macbook pro, Imacs (i7); Camcorders: Panasonic AG-HPX170/AG-HPX250P, Canon HV30/40, Sony Z7U, VX2000, PD170; FCP 6 certified; write professionally for a variety of media; teach video production in L.A.


Return to posts index

Dave Gage
Re: Sparse Bundle Users: How do you backup?
on Jul 25, 2013 at 5:33:25 pm

[Craig Alan] "Gonna do a search."

Cool, let us know what you find out. By the way for the record, I'm still on 10.6.8 and FCPX 10.0.4. I really doubt the file structure has changed since it's debut, but I have no idea. You might consider contacting the DW Tech Support at Alsoft. The senior engineers there are awfully helpful and knowledgeable.

Thanks,
Dave


Return to posts index

Craig Alan
Diskwarrior tech support
on Jul 25, 2013 at 9:46:05 pm

Google was no help but this might be:

From Diskwarrior tech support:

"As for DiskWarrior, yes it can be used with drives that contain Final Cut Pro X
media. However, a project file being damaged while in a .sparsebundle may or
may not indicate a directory problem."

"To Rebuild a .sparseimage or .sparsebundle, the image can be mounted or
unmounted. Simply, select "Rebuild Disk Image" from the File menu within
DiskWarrior."

"Is it possible for you to send both a working as well as damaged project file? "

I did not have a corrupted project to send them. But here's the address if anyone cares to:

Alsoft Technical Support

Mac Pro, macbook pro, Imacs (i7); Camcorders: Panasonic AG-HPX170/AG-HPX250P, Canon HV30/40, Sony Z7U, VX2000, PD170; FCP 6 certified; write professionally for a variety of media; teach video production in L.A.


Return to posts index


Andreas Kiel
Re: Diskwarrior tech support
on Jul 26, 2013 at 10:45:55 am

The most important thing is that you use sparsebundles -- not sparseimages.
Sparsebundles in most cases even can be repaired with Disk Utility.
But I do really recommend to have Disk Warrior in your toolset -- it's a must have!

A corrupted "CurrentVersion.xxxx" in most cases has nothing to do with the location where it is saved. But a disk might be bad and on a bad disk a sparsebundle (remember it's a folder) will work as bad as any other folder.

So finally you should have a backup strategy like having your projects or events saved quite often to a different file -- a different file on a different disk is even better. You can do that within FCPX by duplicating the project and collecting used things into a new event or you can do that by using ProVersioner or my X-Files Pro which both will only save events/project without anything around like render files etc.
Next thing is to keep media at a safe place and organize them upfront before using them. If media need to be transcoded put the transcoded media in a new folder at a safe place and start from there.
Having a strict organization will allow a good backup strategy using backup apps which allow additional scripts.
Another (not cheap) recommendation is to have a disk/raid for each of the disks/volumes. If you let your machine 24/7 running the OS will try to repair and defragment things (means also you should have at least 20% free space on the disk/volumes). Fragmentation will happen anyway as really big files won't be touched. So a safe copy to a wiped disk is the only safe way and will make your system more responsive -- take the weekend off, have some fun with your family or friends.

Just some notes.

-Andreas

Spherico
http://www.spherico.com/filmtools

"He who fights with monsters should be careful lest he thereby
become a monster. And if thou gaze long into an abyss, the abyss will
also gaze into thee." - Friedrich Nietzsche, Beyond Good and Evil


Return to posts index

Brett Sherman
Re: Diskwarrior tech support
on Jul 26, 2013 at 3:03:18 pm

[Andreas Kiel] "A corrupted "CurrentVersion.xxxx" in most cases has nothing to do with the location where it is saved. But a disk might be bad and on a bad disk a sparsebundle (remember it's a folder) will work as bad as any other folder."

I don't think that's the case here. I'm not sure what the problem is exactly. But somehow my backup with sparse bundles is causing corruption in the event and project files. Not all the time, but enough to be a concern and not rely on.

For me, ProVersioner is a buggy mess right now. To the point that it is impossible to work with. I'm using Backups for Final Cut, but don't like the laborious manual setup process. Can't wait to try out X-Files Pro when it's out of beta.



Return to posts index

Matt Larson
Re: Sparse Bundle Users: How do you backup?
on Jul 30, 2013 at 4:12:53 pm

Looks like a lot of good info on this thread already, but I'll add my solution (it's worked well so far...)

I've been keeping my events on a separate image than my projects. This works well for me for two reasons: 1) I tend to have video that I use across multiple projects, so I can start a new project with an Event I've already keyworded, etc and not have the old project cluttering up FCPX; 2) The Images with the Events are large and don't need to be backed up as often as the project files do. So I have just the project file on a small disk image that backs up quickly. I add a "_X" to the end of the file name to distinguish it from the event image, but this could be anything. (As long as it's consistent it lets you do this next part.)

I've also set up an Automator action to make daily backups of any file that ends in "_X.sparsebundle" to another drive.

There are also some Terminal commands that let you either resize the Image to make it larger if you need to, or compact the image and reclaim free space when you have deleted Render Files or Transcoded media before you archive the project. Reminds me of my days dealing with the Avid Unity system!

2 x3Ghz Quad MacPro
9 GB RAM
Mac OS X 10.5.5
QT 7.5.5
FCP 6.0.5
AJA Kona 3 (6.0.1 drivers)
G-Speed XL 12 RAID


Return to posts index

Jeff Kirkland
Re: Sparse Bundle Users: How do you backup?
on Jul 31, 2013 at 12:36:52 am

I'd thought about keeping events in separate bundles but I was constantly having panic attacks because the project would open with offline media because I'd forgotten to mount the right event bundle.

I'll revisit it once Andreas' x-files app comes out of beta as that supports defining and mounting the bundles required for a project.

Jeff Kirkland | Video Producer | Southern Creative Media | Melbourne Australia
http://www.southerncreative.com.au | G+: http://gplus.to/jeffkirkland | Twitter: @jeffkirkland


Return to posts index

Matt Larson
Re: Sparse Bundle Users: How do you backup?
on Aug 1, 2013 at 4:34:56 pm

Hadn't seen that X-files program before, thanks for mentioning it. Looks like it could be pretty useful. Even just the part that creates and resizes sparse bundles looks much better than using Terminal all the time

2 x3Ghz Quad MacPro
9 GB RAM
Mac OS X 10.5.5
QT 7.5.5
FCP 6.0.5
AJA Kona 3 (6.0.1 drivers)
G-Speed XL 12 RAID


Return to posts index

Andreas Kiel
Re: Sparse Bundle Users: How do you backup?
on Aug 2, 2013 at 11:14:12 am

I've updated the X-Files Pro Beta.

I now works till October 12th.

There are several little things updated under the hood.
One of the more obvious is that the user now can convert a sparse image to a sparse bundle, leaving the sparse image in tact to double check before removing it.
As mentioned all the sparse bundle stuff is free and doesn't have a time limitation.

Andreas

Spherico
http://www.spherico.com/filmtools

"He who fights with monsters should be careful lest he thereby
become a monster. And if thou gaze long into an abyss, the abyss will
also gaze into thee." - Friedrich Nietzsche, Beyond Good and Evil


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]