Change font size
It is currently Mon Mar 30, 2020 6:10 pm

Forum rules


Post a new topicPost a reply Page 1 of 1   [ 1 post ]
Author Message
 Post subject: Latest News Re: Grid Downtime
PostPosted: Wed Aug 27, 2014 10:06 am 
Site Admin
User avatar

Joined: Sun Feb 22, 2009 1:39 pm
Posts: 395
Location: Wisconsin, USA
Hi everyone, Hiro Protagonist has made an announcement on the IRC #osgrid channel giving the current status of the grid's downtime. The gist of Hiro's message is that a professional data recovery services firm is looking at the damaged drives from the RAID array, but in the interest of economy over speed, it could be a few days before we know the extent of the damage/ feasibility and cost of a recovery.

Thanks everyone for your patience, and thanks Hiro for the update! Here is the chat copied & pasted verbatim:

07:42 < leTwitch> hey y'all this is aka Hiro Protagonist (I lost the password for my Hiro IRC login a few months back)
07:42 < leTwitch> here's the deal
07:43 < leTwitch> we suffered an outage as you are aware; upon investigation it was discovered there was problems with the array
07:43 < leTwitch> the controller card was subbed with a few different new ones to no avail
07:43 < leTwitch> by way of substitution (and supported by log messages) it was discovered one of the drives was severely troubled
07:44 < leTwitch> unfortunately, it is the drive containing the raid metadata
07:44 < leTwitch> as best anyone can tell, the data is still intact in the secondary raid mirror
07:44 < leTwitch> however, it's a bit beyond any of our skill levels to attempt recovering from that condition
07:44 < leTwitch> so
07:44 < leTwitch> we have a professional data recovery services firm looking at it
07:45 < leTwitch> they are not cheap
07:45 < leTwitch> for instance, we have to pay a (small) fee for an estimate of recovery time and potential recovery success
07:45 < leTwitch> one thing that impacts cost, assuming they are able to recover the array, is how fast do we want it done
07:46 < leTwitch> so while there is a fairly good chance they can effect a recovery, I would say that it is safe to say that
everyone should adopt an attitude of patience
07:46 < AllenKerensky> Ed Gruberman voice: so, what... like an hour?
07:47 < MarcusLlew> Heh
07:47 < leTwitch> keep doing what you're doing - check your backups, fall back to standalones and small grids, and rest assured that
we are fully invested in recovery and taking every prudent and effective course of action we have at our disposal
to get the grid back online, intact.
07:47 < leTwitch> :p
07:47 < AllenKerensky> and for those who don't know "Ed Gruberman" look up the old "Boot to the Head" song
07:47 < MarcusLlew> We know you are, Hiro. Thanks for filling in the blanks. :)
07:48 < leTwitch> I just couldn't sit and watch y'all twist in the wind about this
07:48 < AllenKerensky> exactly - we know you folks are once again going above and beyond the call for "a test grid" and we do
appreciate it
07:49 < leTwitch> I have seen remarkable patience exhibited by all the users about this, and I dont want that to change because of
too much silence on our end
07:49 < leTwitch> but this is why we've been telling everyone 'no news is good news'
07:49 < Shez> Thanks for the update Hiro
07:50 < leTwitch> the status right now is that the drives have been shipped to the data recovery s3ervices company; we used slow-go
so the shipping cost could be kept down
07:50 < leTwitch> they should be getting the drives in-hand sometime today
07:50 < leTwitch> You're welcome Shez :)
07:50 < MarcusLlew> So we're now waiting on their initial assessment.
07:51 < leTwitch> yes
07:52 < MarcusLlew> Cool beans.
07:52 < leTwitch> I suspect that might actually take a few days
07:52 < leTwitch> they're the sort of folks with grade ten cleanrooms where they go in wearing something close to a spacesuit and
disassemble the drives and mount the platters on a test bench etc in an hermetically sealed lab
07:53 < leTwitch> srs brsns
07:53 < leTwitch> if anybody can get our data back, these folks can
07:53 < MarcusLlew> You got our support, whatever the results. I ain't going nowhere.
07:54 < leTwitch> Damned glad to hear that Marcus :D
07:55 < MarcusLlew> Heck, the plazas are safe, and Wright plaza has half the grid's content anyway. :P
07:56 < Shez> Dang.. I just can't click that X!
07:56 < leTwitch> LMAO Marcus too right bub
07:56 < MarcusLlew> lol
07:57 < leTwitch> the other half is probably not worth looking at and stored in my IAR :p
07:57 < MarcusLlew> Everyone's OARs and IARs alone would go a long, long way toward content restoration.
07:58 * Shez bites bullet and clicks X 6 times
07:58 < leTwitch> I think so too, but I hate to let even one user lose anything
07:58 < leTwitch> hey y'all share this convo with friends and associates, copypasta preferred so we dont end up playing 'telephone'

Display posts from previous:  Sort by  
Post a new topicPost a reply Page 1 of 1   [ 1 post ]

Who is online

Users browsing this forum: No registered users and 1 guest

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
610nm Style by Daniel St. Jules of