Creative COW SIGN IN :: SPONSORS :: ADVERTISING :: ABOUT US :: CONTACT US
Creative COW's LinkedIn GroupCreative COW's Facebook PageCreative COW on TwitterCreative COW's Google+ PageCreative COW on YouTube
FORUMS:listlist (w/ descriptions)archivetagssearchhall of famerecent posts

Issues with metaSAN

COW Forums : metaSAN

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Share on Facebook
Jeremy GarchowIssues with metaSAN
by on Aug 18, 2011 at 4:08:44 pm

Hi-

We are beta testing the new Sonnet Technologies VFibre shared storage system:

http://www.sonnettech.com/news/pr2010/pr091010_fusionrx1600vfibre.html

It is a fiber and ethernet based SAN with a windows server built in to a 48TB fibre channel raid (using SAS internally) using metaSAN as the metadata server and windows computer as the master. We've had it for about 2.5 weeks now. I hope I explained all that correctly, I am new to the SAN admin world.

Right now we have two metaSAN licenses (desktops connected via fibre), and run between 1-3 metaLAN licenses (ethernet). These are all running on Macs, the windows server is NTFS, of course.

Everything was going pretty well until this past weekend when without warning, the San disappeared from everyone's desktop, knocking all files offline in FCP. The volume is still listed as mounted on the windows server, but the SAN is "Not Available" according to metaSAN. I then have to go in to metaSAN and add the volumes back to both metaSAN and then redefine the metaLAN volume as well. Everything then mounts back up, and away we go.

No data loss or corruption of any sort, but it is a bit disconcerting that the volumes keep getting "lost" on the master. This has now happened two more times since Sunday for a total of three times in a week. Here's a log from a laptop based ethernet connection:

08/17/11 16:26 Volume "HighInFibre" was removed from volumes list.
08/17/11 16:26 Failed to process master changed notification for volume "HighInFibre".
08/17/11 16:26 Volume "HighInFibre" was unmounted from client.
08/17/11 16:26 Volume "HighInFibre" was removed from volumes list.
08/17/11 16:26 Master changed notification received for volume "HighInFibre".
08/17/11 09:04 Volume "HighInFibre" was mounted as client.
08/17/11 09:04 Volume "HighInFibre" was added to volumes list.
08/17/11 09:04 Volume "HighInFibre" was added to volumes list.
08/17/11 09:04 Currently logged system user set successfully.
08/17/11 09:04 Trying to set currently logged system user...

I also have logs from the master if you need them. Teamviewer is also installed on the server in case you want to take a look.

Any ideas on what is going on here? I have also been working with Sonnet tech support, but my contact had to go away on emergency so I figured I would try here until he is able to respond.

Thanks so much,

Jeremy


Return to posts index
Reply   Like  

Brandon KraemerRe: Issues with metaSAN
by on Aug 18, 2011 at 9:40:52 pm

Jeremy,

I would certainly suggest you contact Tiger with support logs generated from the metaSAN interface. We have dealt with very similar behavior in the past, but it was on a Mac server and was usually caused by a kernel panic crash that resulted in drives that failed to remount cleanly on the master. We have sonnet RX1600s.

Two things seemed to solve this. 1. we reformatted our sonnet volumes. 2. in SAN Managment... Advanced... Master Priorities, we set the master IP to priority High and all other client nodes to priority None. This way when the master crashed or had a kernel panic and rebooted, the volumes on the clients would disappear and then reappear. there was only momentarily a loss of volumes and they never went unavailable from the master.

bk


Return to posts index
Reply   Like  

Jeremy GarchowRe: Issues with metaSAN
by on Aug 18, 2011 at 9:46:29 pm

[Brandon Kraemer] "I would certainly suggest you contact Tiger with support logs generated from the metaSAN interface. "

Will do.

It just happened again. You know what is weird, it seems to happen at around 4:30PM all the time. I have no idea if that's just coincidence, or what.

[Brandon Kraemer] "1. we reformatted our sonnet volumes."

Yikes. That would be no good. They are brand new.

[Brandon Kraemer] "2. in SAN Managment... Advanced... Master Priorities, we set the master IP to priority High and all other client nodes to priority None."

Yes, that is already setup that way. Thanks.

What about the "Data Masters" button? It appears all the metaSAN clients are setup for masters. That's normal, right?

Jeremy


Return to posts index
Reply   Like  


Brandon KraemerRe: Issues with metaSAN
by on Aug 18, 2011 at 10:03:40 pm

yes, all machines should be data masters.

we noticed this crash behavior at all times of the day, but one thing that seemed to trigger it was when the server would check for updates.

our XServe raid apparently has a known issue with the en0/1 port that causes kernel panics and logs have shown the en0 port specifically causes our crashes. that port is only used for internet traffic for the master, not metadata (en1) and no AFP/SMB sharing (we use a bonded line for that). all of that likely doesn't apply to your situation being windows.

also, i would generate support logs from both the master and a client, and Tiger want's to know the exact time the crash happens in the email you send. If your logs are too big to attach, they can set you up with an FTP site to upload.

worth mentioning, we now defrag our volumes every 2 months and we reformat them twice a year. we use one volume as a way point for data while we reformat the other, vice versa. we had a major crash/freeze once that was caused by fragmentation. when you move your data around avoid multiple copy processes, try to cue up all your data to copy at once.

best of luck,

bk


Return to posts index
Reply   Like  

Jeremy GarchowRe: Issues with metaSAN
by on Aug 18, 2011 at 10:05:21 pm

Thanks so much, Brandon.

Jeremy


Return to posts index
Reply   Like  

Bernard LamborelleRe: Issues with metaSAN
by on Aug 19, 2011 at 12:56:24 am

Hi Jeremy,

I did notice you are now in contact with our support team, which is good. I find it quite odd that the volume would get knocked out of the SAN Definition like this. You mentioned this happening twice in a row around 4h30 and only recently. Do you have any special processes or activities taking place at that moment (i.e. anti-virus, defrag or backup application that could interfere)?

I have seen occasional corrupted SAN Definition leading to odd behaviour before, but it is usually triggered by excessive manipulation of the SAN Definition (such as quickly adding/removing members without giving enough time to metaSAN to propagate the new Definition around). But if you have not been making lots of changes to your SAN Definition, it could also be something else.

Whatever the issue is, the fact everything else is working smoothly and this issue only started manifesting itself recently leads me to believe that our guys will be able to quickly figure it out.

Let us know how it goes!

Cheers,

Bernard


Return to posts index
Reply   Like  


Jeremy GarchowRe: Issues with metaSAN
by on Aug 19, 2011 at 1:40:28 pm

[Bernard Lamborelle] " You mentioned this happening twice in a row around 4h30 and only recently. Do you have any special processes or activities taking place at that moment (i.e. anti-virus, defrag or backup application that could interfere)?"

Not that I know of. The windows part of the machine and the drives come preformatted and ready to go. The only thing to do is get the DNS setup, and define everything in metasan. I will have to do some more digging on that. Unfortunately, windows is very foreign to me.

[Bernard Lamborelle] "But if you have not been making lots of changes to your SAN Definition, it could also be something else."

No. It's stayed the same. What is weird is that the drive stays mounted to the VFibre during this, it's just the SAN that gets booted.

It has now happened 4 times. From memory, three of those times were right around 4:30PM (16:30).

It was Sunday, Monday, Wednesday, Thursday.

Another thing I should mention, the first time it happened, one of the drives in the raid6 config reported as faulted, it's odd as we only had the machine for barely two weeks at that point. I rebuilt the raid, and hasn't faulted since then.

Thanks for your response.

Jeremy

Jeremy


Return to posts index
Reply   Like  

Bernard LamborelleRe: Issues with metaSAN
by on Aug 19, 2011 at 7:29:11 pm

Hi Jeremy,

Our guys have received the logs you sent them and are reviewing them. They will be in touch shortly.

Bernard


Return to posts index
Reply   Like  

Jeremy GarchowRe: Issues with metaSAN
by on Aug 19, 2011 at 7:30:21 pm

Awesome.

Thanks so much!

Jeremy


Return to posts index
Reply   Like  


Jeremy GarchowRe: Issues with metaSAN
by on Aug 23, 2011 at 10:29:55 pm

Just to keep you informed, it happened again today (now the fifth time overall) and it coincided with a client shut down.

I sent a Debug log. Hopefully that explains something.

Jeremy


Return to posts index
Reply   Like  

Bernard LamborelleRe: Issues with metaSAN
by on Aug 30, 2011 at 1:48:29 pm

Hi Jeremy,

I was hoping you could give us an update on how things have been going since the config of your system was redone. Is it too early to tell yet?

Cheers,

Bernard

Bernard Lamborelle
bernard at tiger-technology dot com
http://www.tiger-technology.com
514-667-2015


Return to posts index
Reply   Like  

Jeremy GarchowRe: Issues with metaSAN
by on Aug 30, 2011 at 2:31:02 pm

Hi Bernard.

First of all, the tiger support team was great. We had everything up and running in a little over an hour with zero down time as this was done before the work day got started for us.

I think it might be too early to tell as it was hard to get this to fail consistently, but so far, so good. I have tried to "break" it and everything seems to be holding up. There was the crucial step of adding in the master IP to the
Metasan config file that was the difference. I don't think that was done on our initial setup.

Thanks so much to you and your team.

I do have a few other weird things happening, but I think those might already be addressed through Sonnet support. If I find that it isn't, I will be in contact.

Jeremy


Return to posts index
Reply   Like  


Bernard LamborelleRe: Issues with metaSAN
by on Oct 2, 2011 at 2:34:58 pm

Hi Jeremy,

It's been a while since we heard from you...

Has your system been rock solid? Did you manage to fix the "few other weird things"?

Cheers,

Bernard

Bernard Lamborelle
bernard at tiger-technology dot com
http://www.tiger-technology.com
514-667-2015


Return to posts index
Reply   Like  

Jeremy GarchowRe: Issues with metaSAN
by on Oct 3, 2011 at 1:13:48 pm

Hi, Bernard. Thanks for checking.

There's still a few little things, but we are getting there.

There's a new Target Driver that I am installing today that should help with some SAN to LAN failovers we have been getting and should hopefully fix a "flaky" connection on one of our fibre clients.

P2 Flow still doesn't work, but the appropriate people are in contact on that.

I have some files that are permanently stuck in the trash that came from my LTO system. I am trying to work with the Cache-A team to see if they have idea how this happened, if they don't I was going to contact you about it. Basically, there are some files with odd permissions that are stuck in .Trashes > 501 and Terminal or the Windows equivalent won't delete them.

But all in all, things have been going well.

Jeremy


Return to posts index
Reply   Like  

Morad ShalbakRe: Issues with metaSAN
by on Sep 19, 2011 at 3:12:04 pm

HII ,,,

Do you need to metaSAN two Ethernet networks such as the Xsan , To dedication one of the private (metadata) network and one for the public ??


Return to posts index
Reply   Like  


Brandon KraemerRe: Issues with metaSAN
by on Sep 19, 2011 at 3:24:03 pm

yes, it needs a meta data network on a separate subnet from your standard network, so two network interfaces are required.


Return to posts index
Reply   Like  

Morad ShalbakRe: Issues with metaSAN
by on Sep 19, 2011 at 3:59:16 pm

but Xsan Create a private network with open directory users ( ACL Permissions ) Is there a special configuration of the metaSAN?


Return to posts index
Reply   Like  

Morad ShalbakRe: Issues with metaSAN
by on Sep 20, 2011 at 5:06:13 pm

but Xsan Create a private network with open directory users Is there a special configuration of the metaSAN


Return to posts index
Reply   Like  

<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
Share on Facebook


FORUMSTUTORIALSMAGAZINESTOCKYARDVIDEOSPODCASTSEVENTSSERVICESNEWSLETTERNEWSBLOGS

Creative COW LinkedIn Group Creative COW Facebook Page Creative COW on Twitter
© 2013 CreativeCOW.net All rights are reserved. - Privacy Policy

[Top]