FORUMS: list search recent posts

Deleting never fully complete

COW Forums : Autodesk Smoke

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Josh Spivack
Deleting never fully complete
on Feb 19, 2013 at 1:24:41 am

We have a 2TB storage limit on our smoke imported material and renders. Because of this I occasionally go through old projects and archive or delete them altogether, however I am concerned at how I still have 1.3TB in use. I went though the storage and discovered dpx frames from projects totally deleted that should not be there at all.
When I do delete, I go to all my libraries, select the clips, go to "unstore" and "unrender" and then delete them. I think deleting would be enough but just to be safe I follow those steps. Then I go to the edit desk and delete the clips also. Then I delete the project, if I am removing the entire project.
Yes, I've hit "force purge" and all that but I still don't see enough space and as noted, clips from deleted projects in folders such as this: 2e0e or 3ef7. I see dpx frames of these deleted projects and it worries me about how Smoke's really managing my media.
My volume statistics indicates I am using a total (added them up) of 655,691 Nb FullRes Frames. No proxy frames, audio negligible.
What can I do here?

Thanks,
Josh


Return to posts index

Brian Mulligan
Re: Deleting never fully complete
on Feb 19, 2013 at 2:11:21 am

Normally if you reboot you system then VIC will run and when you restart Smoke those frames should be deleted.

You can run VIC manually.
/usr/discreet/io/bin/vic -v stonefs7

See if that helps.

Brian Mulligan
Senior Editor - Autodesk Smoke
WTHR-TV Indianapolis,IN, USA
Twitter: @bkmeditor


Return to posts index

Josh Spivack
Re: Deleting never fully complete
on Feb 19, 2013 at 6:32:58 pm

Brian,

Thank you for your response. I have certainly restarted since the referred to projects were deleted. Where would I enter that code? My best guest is using Manual Edit in my Media Storage area for my Setups. Perhaps under:
[Partition7]
Name=AutodeskMediaStorage
Path=/Volumes/SCRATCH/Autodesk Media Storage

But I'm not sure and know better than to haphazardly enter codes. Smoke's storage does seem to respond to deleting from the library or projects but simply not enough, and since I can see deleted projects' files obviously there's an issue.

Thank you again,
Josh


Return to posts index


Brian Mulligan
Re: Deleting never fully complete
on Feb 19, 2013 at 6:39:26 pm

Open up the Terminal program in utlilties and enter it there.

Brian Mulligan
Senior Editor - Autodesk Smoke
WTHR-TV Indianapolis,IN, USA
Twitter: @bkmeditor


Return to posts index

Josh Spivack
Re: Deleting never fully complete
on Feb 19, 2013 at 6:56:31 pm

Well terminal seemed to do a lot, extracting frames here and there etc. but even after a restart and doing it twice (which oddly yielded precisely the same results...I have a saved terminal output) there is no change in the storage available and the folders that have media from deleted projects remain. Thank you nevertheless.
I do have what I consider an anomaly that might be related. If I go to my localhost/wiretapcentral I discover two gateways and two IFFFS. Now the gateways I understand since they're pretty much just my network. One gateway is for a computer called "fudge" another is the main CPU where the storage is called "biscuit." I know, great names. Anyways, I am able to access the less important Fudge's IFFFS and underneath that I see "stonefs7." I regret to say I'm not 100% certain what this is but since it refers to stonefs I believe it's talking about storage.
When I go to the real IFFFS that I should be concerned with however, I receive an error message: "Unable to access the volume stonefs: The given volume does not have a corresponding hardware partition." You may recall my storage is partition 7, it probably was accidentally made 7 rather than 1 because awhile back we added external drives to process to, and maybe along the way we've killed a connection.

This might be totally unrelated but I thought it was worth mentioning. If you have any other advice, including waiting on what we did for terminal to perhaps updates, I'm all ears. Thank you again!

Josh


Return to posts index

Robert Adam
Re: Deleting never fully complete
on Feb 21, 2013 at 6:04:33 pm

Hi Josh.

The default storage partition that the smoke installer creates is stonefs7. When you add a second partition, it is named stonefs, then stonefs1 and so on.
Given the error message you're getting, it seems that at some time you've added a second partition (stonefs) which was later disconnected from your Mac but was not removed from Smoke's config.

Open the Smoke Setup app and go to the Media Storage tab. Are there two entries in that list? If one of them corresponds to a drive that's no longer connected to your Mac, please click "Manual Edit", then scroll down until you find the "[Partition]", "Name=" and "Path=" lines that match that partition and add a hash before each of those three lines. Then click OK, Yes and Apply to save the changes. There should only be one partition listed in the setup app now.
Then open the Service Monitor app and restart the Stone+Wire and Wiretap Server services.

Next, open the Terminal app and type this command. This is a variant of the command Brian already gave you, but this version checks all partitions, not just stonefs7:

vic -a -f

After vic finishes, restart your Mac and check your disk space again.


Regards,
Robert.

Autodesk M&E Support
http://www.facebook.com/AutodeskMECF


Return to posts index


Josh Spivack
Re: Deleting never fully complete
on Feb 21, 2013 at 7:03:59 pm

Robert,
Thank you greatly for the response. There was only 1 media storage partition saved to Smoke's setup. This is because I had already canceled those drives out quite some time back. I do agree though that this is likely related to my problem. I can see that the one media storage added is the correct one that goes to where all my files should be stored or rendered. It'd "Id" is 7 which may be related to the fact that it's partition 7. I feel as though it should be partition 1 since it was the first drive we ever used but perhaps it got bumped up as others were added and then stayed at 7 when those other drives got canceled out (I can send you this).

Nevertheless, to make a long response short, I had already canceled those out but when I enter your code into terminal I get the result "-bash: vic: command not found". I am literally entering "vic -a -f".

Any ideas?
Thanks again,
Josh


Return to posts index

Robert Adam
Re: Deleting never fully complete
on Feb 21, 2013 at 8:31:51 pm

Hi Josh.

if volume zero is not in the Smoke Setup app but you're getting those errors, that means that the metadata folder for volume 0 is still on your system drive.

Open Terminal and type:
ls /usr/discreet/clip

Normally the output should only list "stonefs7" but I bet that your mac will also list "stonefs" which corresponds to the partition you've disconnected.
If so, type this command to remove the "stonefs" entry:
sudo mv /usr/discreet/clip/stonefs /var/tmp/

Then run vic like with the full path, like this:
/usr/discreet/io/bin/vic -a -f

Then restart your Mac.

Regards,
Robert.

Autodesk M&E Support
http://www.facebook.com/AutodeskMECF


Return to posts index

Josh Spivack
Re: Deleting never fully complete
on Feb 26, 2013 at 6:49:08 pm

Thank you for this reply and for all the help. Sorry I am getting back to this a few days later, I had Friday plus Monday off. Anyways, when I enter that first code I get from terminal this response:
biscuits-Mac-Pro:~ biscuit$ ls /usr/discreet/clip
stonefs1 stonefs2 stonefs3 stonefs7
Now in my setup the [Partion0] is canceled out, if that is what you mean. I think this has to be where in lies the problem, somewhere in the setup. I'm embarrassed to say it's a bit messy. Two partion 1's, a partion 3, all canceled out, and my media storage drive being a partion 7 that isn't cancelled out.

I enter the second code and receive this:

biscuits-Mac-Pro:~ biscuit$ sudo mv /usr/discreet/clip/stonefs /var/tmp/
mv: /usr/discreet/clip/stonefs: No such file or directory.
The third code seems to extract a lot of files but no change occurs in the media storage, and I can still see deleted projects' renders or media. Obviously I've restarted since.
Thank you for your help,
Josh


Return to posts index


Robert Adam
Re: Deleting never fully complete
on Feb 26, 2013 at 6:53:34 pm

Hi Josh.

So you are only using partition 7, right? That means the stonefs1, stonefs2 and stonefs3 folders under /usr/discreet/clip are no longer needed in there.

Enter these commands to move those folders away:
sudo mv /usr/discreet/clip/stonefs1 /var/tmp/
sudo mv /usr/discreet/clip/stonefs2 /var/tmp/
sudo mv /usr/discreet/clip/stonefs3 /var/tmp/

Restart the Stone+Wire service, run vic again, restart your Mac and tell me if it made any difference.

thanks!
robert.

Autodesk M&E Support
http://www.facebook.com/AutodeskMECF


Return to posts index

Josh Spivack
Re: Deleting never fully complete
on Feb 26, 2013 at 7:14:34 pm

Thanks Robert,
Terminal did delete those directories. This is evidenced also by the fact that when reentered no directory existed but originally it does delete it. However while running the vic code from the previous email does seem to apply a different result of extraction, no change in my media storage has yet to be seen. I followed your directions exactly.
The media storage I'm trying to have old files be deleted from is a scratch disk installed directly into the MAC, literally an extra drive connected to the system only for Smoke. Along the way, we had to add external drives to render to due to projects that required literally 1.5 or so TB just to really work, full debayed r3d stuff. I think this might be why deleting is not as easy since we did at one point have multiple scratch disks. That being said, those drives have all been cancelled out but for whatever reason files that were set to render to my "local" scratch disk storage but then deleted or at least had their projects become deleted still reside in this disk. Short of going through the 254 folders and examining if the dpxs are from a current or deleted project, what are my options? I don't assume that is even much of an option considering I don't exactly understand how the Smoke storage is setup and could delete something more crucial.
Thanks,
Josh


Return to posts index

Robert Adam
Re: Deleting never fully complete
on Feb 26, 2013 at 7:39:02 pm

Hi Josh.

Please don't delete anything by hand. You might delete frames used in current projects, or even worse, .mio files that contain metadata on soft-imported clips.

Here's how you could poll the frames to see if they belong to a current project. Do you remember how long ago you last worked on those older projects?

If you do, open Terminal and type this command to go to the Media Storage folder (replace vol_name with the actual name of your internal disk):
cd /Volumes/vol_name/Autodesk\ Media\ Storage

Now type the following command and replace DAYS with the approximate number of days that have passed since you last worked on those old projects:
find ./ -mtime +DAYS

This will return a list of files that are older than DAYS, including the folder they're in.
For example:
.//9b25/0x052f58c9700000a1.dpx
.//9b25/0x052f58c9700000a2.dpx

The part that I underlined in the above example is the frame ID.

Now pick a few random frame IDs from the output and run this command on each of them to see what project and clip they're being used in (replace FRAME_ID with the actual ID):
/usr/discreet/io/bin/fcof stonefs7 FRAME_ID

Did you get any "FrameID not found" message or do you get reports of any frames belonging to projects that you have deleted?

Thanks!
Robert.

Autodesk M&E Support
http://www.facebook.com/AutodeskMECF


Return to posts index


Josh Spivack
Re: Deleting never fully complete
on Feb 26, 2013 at 8:18:36 pm

Ran everything like you said, cool way to look at the files. I did some random testing at first and everything belong to a project, "test" "tutorials" whatever.
So I went to my media storage and found the folders that I had noted as having dpx's from old, deleted projects. Using "find" I confirmed these dpx clip ID's were found during my terminal search (after the "days" code) and they were. I then entered these clips ID's and the were routed to a live project, a "test" project. Perhaps at one point I brought them in here to test a few things, not sure.
So that is very interesting. What I can't figure then is why those files are there. Should I delete "test project" since it seems to have quite a bit of old information stored in it? I can always start a new test or move over elements that I'd like to maintain (I might need to figure out how to do that haha but still).
These clips don't exist on the edit desk or any library in the project they are assigned to, so how can I delete them without axing the project itself?
Real useful information thank you Robert.
Josh


Return to posts index

Robert Adam
Re: Deleting never fully complete
on Feb 26, 2013 at 8:25:35 pm

That's interesting. Yes, if that test project doesn't contain anything you need, just delete the entire test project (and any other project you don't need anymore) and run vic again to clean up the frames:
/usr/discreet/io/bin/vic -a -f

Restart stone+wire after running vic, give it a few minutes and then check the disk space again.
If free disk space has still not increased, then the path of least resistance might be to just go into Smoke, archive all the projects you want to keep, then delete all the projects, delete all the contents of the "Autodesk Media Folder" and then restore the project archives. It's a rather radical approach, but it will probably take less time than to manually hunt for "orphan" frames in the scratch folder and delete them by hand.

Regards,
Robert.

Autodesk M&E Support
http://www.facebook.com/AutodeskMECF


Return to posts index

Josh Spivack
Re: Deleting never fully complete
on Feb 26, 2013 at 9:26:16 pm

Okay I appreciate all this information thank you Robert. To delete "test" would require some time on my end to figure out to preserve open work. Test is a project that implies the obvious. I use Smoke for a lot of wonderful things, love it actually, and I use the test project to increase my understanding of Action and other aspects as well as occasionally work on something that could be useful. One of those sort of learn and use applications is alive, a project involving a lot of layers that I'll bring into action. I'd want this to be preserved. I'm not sure yet how to move libraries and save them to another project (or folders on an edit desk), so before deleting this I'd need to do that and that's not such a bad thing anyways since it's time I learn this step (hadn't been necessary before). I tell you this because I won't have a result back to you for a little while, since I still have about 700GB of storage I know I can rely on until I reach critical.
Nevertheless a solution, it doesn't answer the question of why these dpx frames even exist. If deleted from edit desk and library why are they in the media storage? While you don't need to have a direct answer since maybe it could be a number of causes, obviously having the luxury of a project I can delete is not always a guarantee.
Thanks immensely Robert I'm all ears for whatever you have to say and if I do end up deleting the project I will let you know how these results fare.
Josh


Return to posts index

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2017 CreativeCOW.net All Rights Reserved
[TOP]