nope - no grok or grope
it was there, but not being used at all
darn it - some mailing list I was on was looking for alternatives to easydns beacause they thought that group had pulled plug on wikileaks
now I know it's not true, so I want to fix them
one of my million mailing lists :0
found it
man I love these proliant dl360's.   4 of them sitting in a rack next to my desk.  when they are idle (which they are when I'm configuring them) they are absolutely silent.
wow jpalmer
usually when configuring
they are on full blast
i freaking hate that
the dells are like that.   when you first turn the R410's on,  for like 2 minutes, you won't hear anything else.
yeah
its annoying
wow
those dl360's can have up to 192GB of RAM huh
that's kinda insane
the only part I don't like about them,  is the 2.5" disk.
but you can fit more gigabytes with a smaller disk :P
is there anything wrong with mercury
i cant ssh or vnc into it
what's mercury?
a server at arp
looks like at about 20 after my vps stopped responding
it is also on mercury
yep thats the time i see when my bot pinged out
did you email support@ at that time?
no
I just sent an email incase I'm the first
my friend just told me his bot pinged out so i went to investigate
hrm
i can't ping mercury either
jpalmer: yeah, I love HP gear.
yup, kernel panic
Why do this shirt have a pleasant smell? It should smell like shit
Oh well
well, this is great, grub just hangs at "Loading GRUB..."
Can you get into the GRUB console?
worst case, presumably it's just grub that got trashed somehow...
though I doubt you want to *go* anywhere ;)
mike-burns: nope
sometimes "Error 15" pops up
looks like it can't find a file
will keep working...
Did you change partitions, kernel, mountpoints, etc?
at that stage of the game, the mbr is accessable but the rest of the disk not so much, check that the partition table didn't get munged somehow (I've had this happen to me in the past)
aka I've `recovered' by `from memory' recalling the sizes of the partitions. may not be your problem. yymv. just a thought. ;-)
jdoe: agreed.   everyone complains about the price of HP gear, until they actually use it.
it's hard to beat the proliant line.
"well,  this similar spec dell is 1/3rd the price!"
I've never owned Dell anything ^_^
we use dells at work though
meh
anyone complaining about the price of HP gear has never bought from a non-whitebox vendor
heh
I dunno,  a Dell R410,  is about half the price of a DL360 G6.  configured similarly.  and the main difference is only that the HP would use a 2.5" disk.  the dell would use a 3.5"
well, t hat and the R410 is a bit limited when it comes to expansion.
jpalmer: my experience w/ HP gear has been rather painful
jpalmer: give me IBM gear anyday
Looks like there is some corruption on mercury's root fs :(  The installer USB key I have in there does not provide enough rescue tools :(  Will need to make a data center trip
shitty :/
lol
Host mercury has been restored and boots properly. We're doing last minute checks + kernel upgrade before starting VMs.
what was the problem?
fucking hell
corruption again
hardware must be bad :(
memtest?
memtest will take forever
you can let it run for x mins (1 usually spots really bad memory quickly)
but I'm sure you'll handle it in any event. bugging out for the evening. enjoy!
could be a controller too
could be almost anything :/
I'm guessing the new host that I am on is called Mercury.
DDevine: let me guess,  outages?
Actually, no!
Sometimes my SSH takes a long time to prompt for password
like 30 seconds +
then, I doubt it.  mercury is having some hardware issues it seems.
oh,   slow ssh logins is usually do to bad rDNS
Probably. I'll get around to fixing that soon.
I have already requested the transfer.
DDevine: you're not on mercury
up_the_irons: Yeah, just the delay on SSH made it look so. I'll fix up my rdns today probably.
any updates on mercury?
wwrl: still looks bad.  I think the raid controller is bad (although it still appears to work), or something severely corrupted the disks (even the LVM volumes on which VPS' reside :(
some of my own VMs are on there, and if i try to mount the LVM volumes, i get bad blocks
up_the_irons: backups?
fink: i have backups of the host root fs, but individual VMs are not backed up (responsibility of the customer to backup their stuff)
heh
am i on this machine?
fink: don't know, PM your uuid
if my VPS was on mercury, there's no chance that I could just have it moved to a different host machine, correct?
wwrl: well, i'm trying to do that now as it is, but it is proving difficult
wwrl: what is your vps uuid? (pm it to me)
up_the_irons: My VPS probably isn't on Mercury. There's no chance that I could have it moved there now that it's broken and stuff?
dxtr: i hope you're joking ;)
huh, looks like FreeBSD 8.2-BETA1 was released recently fwiw to anyone that follows the -STABLE branch.
up_the_irons: can you move me to mercury too plox?
what's special about mercury anyways other than its broken right now.
nothing, it's broken
and why is it called mercury and not like...kvm14?
or is there a special name for all kvms and I just don't know the name of mine.
whitefang: it was created before i had a naming convention
you could call it kvm0 :P
or you could let me call kvm14 "venus" because it rhymes with a naughty word.
personally I call all the machines on my home network + VPS' after NZ Native Birds
wow, that's a lot of machines named kiwi
the machinethat I called Kiwi is now 'retired'
Silly flightless birds...
hmmm, I see the amount of bandwidth for my package under My Services, but I don't see my current usage.
am I blind?
<sigh...>
all data on mercury has been catastrophically lost
emails sent to customers offering a fresh VPS (with two months of free service)
fuck I'm going to have a long night...
and day tomorrow
i've never lost a box before, sigh...  I guess this is the cost of doing business
does ANYONE know an effective way of backing up LVM volumes that doesn't tax the fuck out of the machine?
was mercury one of the first machines?
whitefang: it was the very first
that's unfortunate :/
it is the only machine that does not have the same hardware config as the others
up_the_irons, testdisk was no good?
http://www.cgsecurity.org/wiki/TestDisk
did you use this?
I thought a RAID array was supposed to stop things like complete data loss from happening.
whitefang: if a single disk fails, yes, it is great for that.  but what if the controller goes bad?  that's what I think happened here
vapor: I'm passed that at this point.  I *can* get a lot of data from the lost volumes, but it is very hit or miss.  running an fsck produces so many errors it is unbelievable.  one may be able to find individual files they need, but the filesystems will never boot
is there any way to give the customers on that machine access to their specific filesystems do recover what they can?
whitefang: it's possible, but not without some work; right now, i wouldn't have time for that, just need to focus on getting the base VMs up
up_the_irons: yeah, of course...but it might soften the wrath if you can offer them said access to their possibly corrupted filesystems "soon".
and by "soon" I mean Blizzard(tm)'s sense of the word "soon".
whitefang, RAID is "high availability", not a "backup". :-)
whitefang: yeah, you're right
up_the_irons: just tell them jpalmer is personally responsible for getting them their old filesystems, he was just telling me how bored he was the other night. :P
LOL