RandalSchwartz: it was there, but not being used at all
darn it - some mailing list I was on was looking for alternatives to easydns beacause they thought that group had pulled plug on wikileaks
now I know it's not true, so I want to fix them
one of my million mailing lists :0
found it ***: LT has joined #arpnetworks
nakano is now known as nakano_
schmir has joined #arpnetworks
_Ehtyar has joined #arpnetworks
Ehtyar has quit IRC (Remote host closed the connection)
ziyourenxiang has joined #arpnetworks
schmir has quit IRC (Remote host closed the connection)
fink has joined #arpnetworks
fink has quit IRC (Client Quit)
fink has joined #arpnetworks
DDevine has quit IRC (Remote host closed the connection) jpalmer: man I love these proliant dl360's. 4 of them sitting in a rack next to my desk. when they are idle (which they are when I'm configuring them) they are absolutely silent. ***: fink has quit IRC (Quit: fink) vcs: wow jpalmer
usually when configuring
they are on full blast
i freaking hate that jpalmer: the dells are like that. when you first turn the R410's on, for like 2 minutes, you won't hear anything else. ***: fink has joined #arpnetworks vcs: yeah
its annoying whitefang: wow
those dl360's can have up to 192GB of RAM huh
that's kinda insane jpalmer: the only part I don't like about them, is the 2.5" disk. whitefang: but you can fit more gigabytes with a smaller disk :P ***: cubelogic has joined #arpnetworks
Sheath is now known as MuskyHusky
richardquisumbin has quit IRC (Ping timeout: 265 seconds) MuskyHusky: is there anything wrong with mercury
i cant ssh or vnc into it ***: LT has quit IRC (Quit: Leaving)
d^_^b has quit IRC (Ping timeout: 265 seconds)
ziyourenxiang has quit IRC (Quit: ziyourenxiang)
cubelogic has quit IRC (Remote host closed the connection) whitefang: what's mercury? MuskyHusky: a server at arp toddf: looks like at about 20 after my vps stopped responding
it is also on mercury MuskyHusky: yep thats the time i see when my bot pinged out toddf: did you email support@ at that time? MuskyHusky: no toddf: I just sent an email incase I'm the first MuskyHusky: my friend just told me his bot pinged out so i went to investigate up_the_irons: hrm
i can't ping mercury either -: jdoe blames up_the_irons jdoe: jpalmer: yeah, I love HP gear. up_the_irons: yup, kernel panic dxtr: Why do this shirt have a pleasant smell? It should smell like shit
Oh well ***: cubelogic has joined #arpnetworks
nerdd_ has joined #arpnetworks
nerdd_ is now known as nerdd up_the_irons: well, this is great, grub just hangs at "Loading GRUB..." mike-burns: Can you get into the GRUB console? jdoe: worst case, presumably it's just grub that got trashed somehow...
though I doubt you want to *go* anywhere ;) up_the_irons: mike-burns: nope
sometimes "Error 15" pops up
looks like it can't find a file
will keep working... mike-burns: Did you change partitions, kernel, mountpoints, etc? toddf: at that stage of the game, the mbr is accessable but the rest of the disk not so much, check that the partition table didn't get munged somehow (I've had this happen to me in the past)
aka I've `recovered' by `from memory' recalling the sizes of the partitions. may not be your problem. yymv. just a thought. ;-) ***: nerdd_ has joined #arpnetworks
nerdd has quit IRC (Ping timeout: 255 seconds)
nerdd_ is now known as nerdd jpalmer: jdoe: agreed. everyone complains about the price of HP gear, until they actually use it.
it's hard to beat the proliant line.
"well, this similar spec dell is 1/3rd the price!" whitefang: I've never owned Dell anything ^_^
we use dells at work though jdoe: meh
anyone complaining about the price of HP gear has never bought from a non-whitebox vendor
heh jpalmer: I dunno, a Dell R410, is about half the price of a DL360 G6. configured similarly. and the main difference is only that the HP would use a 2.5" disk. the dell would use a 3.5"
well, t hat and the R410 is a bit limited when it comes to expansion. ***: cubelogic has quit IRC (Remote host closed the connection) G: jpalmer: my experience w/ HP gear has been rather painful
jpalmer: give me IBM gear anyday ***: vcs has quit IRC (Read error: Operation timed out)
whitefang has quit IRC (Read error: Operation timed out) up_the_irons: Looks like there is some corruption on mercury's root fs :( The installer USB key I have in there does not provide enough rescue tools :( Will need to make a data center trip ***: vcs has joined #arpnetworks
schmir has joined #arpnetworks
schmir has quit IRC (Ping timeout: 240 seconds) jdoe: shitty :/ ***: cubelogic has joined #arpnetworks
richardquisumbin has joined #arpnetworks
cubelogic has quit IRC (Remote host closed the connection) vcs: lol ***: whitefang has joined #arpnetworks
amdprophet has quit IRC (Quit: amdprophet)
amdprophet has joined #arpnetworks up_the_irons: Host mercury has been restored and boots properly. We're doing last minute checks + kernel upgrade before starting VMs. fink: what was the problem? up_the_irons: fucking hell
corruption again
hardware must be bad :( toddf: memtest? up_the_irons: memtest will take forever toddf: you can let it run for x mins (1 usually spots really bad memory quickly)
but I'm sure you'll handle it in any event. bugging out for the evening. enjoy! jdoe: could be a controller too
could be almost anything :/ ***: robotarmy has quit IRC (Remote host closed the connection)
Tranceo has joined #arpnetworks
Tranceo has left
nuke- has quit IRC (Ping timeout: 265 seconds)
DDevine has joined #arpnetworks DDevine: I'm guessing the new host that I am on is called Mercury. jpalmer: DDevine: let me guess, outages? DDevine: Actually, no!
Sometimes my SSH takes a long time to prompt for password
like 30 seconds + jpalmer: then, I doubt it. mercury is having some hardware issues it seems.
oh, slow ssh logins is usually do to bad rDNS DDevine: Probably. I'll get around to fixing that soon.
I have already requested the transfer. ***: robotarmy has joined #arpnetworks up_the_irons: DDevine: you're not on mercury DDevine: up_the_irons: Yeah, just the delay on SSH made it look so. I'll fix up my rdns today probably. ***: wwrl has joined #arpnetworks wwrl: any updates on mercury? up_the_irons: wwrl: still looks bad. I think the raid controller is bad (although it still appears to work), or something severely corrupted the disks (even the LVM volumes on which VPS' reside :(
some of my own VMs are on there, and if i try to mount the LVM volumes, i get bad blocks fink: up_the_irons: backups? up_the_irons: fink: i have backups of the host root fs, but individual VMs are not backed up (responsibility of the customer to backup their stuff) fink: heh
am i on this machine? up_the_irons: fink: don't know, PM your uuid wwrl: if my VPS was on mercury, there's no chance that I could just have it moved to a different host machine, correct? up_the_irons: wwrl: well, i'm trying to do that now as it is, but it is proving difficult
wwrl: what is your vps uuid? (pm it to me) ***: fink has quit IRC (Quit: fink) dxtr: up_the_irons: My VPS probably isn't on Mercury. There's no chance that I could have it moved there now that it's broken and stuff? ***: vapor has quit IRC (Ping timeout: 272 seconds) up_the_irons: dxtr: i hope you're joking ;) whitefang: huh, looks like FreeBSD 8.2-BETA1 was released recently fwiw to anyone that follows the -STABLE branch.
up_the_irons: can you move me to mercury too plox?
what's special about mercury anyways other than its broken right now. up_the_irons: nothing, it's broken whitefang: and why is it called mercury and not like...kvm14?
or is there a special name for all kvms and I just don't know the name of mine. up_the_irons: whitefang: it was created before i had a naming convention whitefang: you could call it kvm0 :P
or you could let me call kvm14 "venus" because it rhymes with a naughty word. G: personally I call all the machines on my home network + VPS' after NZ Native Birds whitefang: wow, that's a lot of machines named kiwi -: whitefang ducks. G: the machinethat I called Kiwi is now 'retired' DDevine: Silly flightless birds... whitefang: hmmm, I see the amount of bandwidth for my package under My Services, but I don't see my current usage.
am I blind? ***: _Ehtyar has quit IRC (Remote host closed the connection)
vcs has quit IRC (*.net *.split)
toddf has quit IRC (*.net *.split)
RandalSchwartz has quit IRC (*.net *.split)
islandfox has quit IRC (*.net *.split)
nuke^ has quit IRC (*.net *.split)
richardquisumbin has quit IRC (*.net *.split)
amdprophet has quit IRC (*.net *.split)
koan has quit IRC (*.net *.split)
azmarco has quit IRC (*.net *.split)
awyeah has quit IRC (*.net *.split)
cmeiklejohn has quit IRC (*.net *.split)
Yamazaki-kun has quit IRC (*.net *.split)
mike-burns has quit IRC (*.net *.split)
phlux has quit IRC (*.net *.split)
up_the_irons has quit IRC (*.net *.split)
tooth has quit IRC (*.net *.split)
whitefang has quit IRC (*.net *.split)
mattx86 has quit IRC (*.net *.split)
tuv has quit IRC (*.net *.split)
infrared has quit IRC (*.net *.split)
cedwards has quit IRC (*.net *.split)
baklava has quit IRC (*.net *.split)
blovett has quit IRC (*.net *.split)
Dougy has quit IRC (*.net *.split)
mtve has quit IRC (*.net *.split)
Lefty has quit IRC (*.net *.split)
N1JER has quit IRC (*.net *.split)
MuskyHusky has quit IRC (*.net *.split)
bitslip has quit IRC (*.net *.split)
pete__c_ has quit IRC (*.net *.split)
nipuL has quit IRC (*.net *.split)
robotarmy has quit IRC (*.net *.split)
hiro_dSn_ has quit IRC (*.net *.split)
jpalmer has quit IRC (*.net *.split)
mjp has quit IRC (*.net *.split)
ww has quit IRC (*.net *.split)
bsdguy has quit IRC (*.net *.split)
dxtr has quit IRC (*.net *.split)
bob^^ has quit IRC (*.net *.split)
G has quit IRC (*.net *.split)
jdoe has quit IRC (*.net *.split)
DDevine has quit IRC (*.net *.split)
mhoran has quit IRC (*.net *.split)
BarberRonny has quit IRC (*.net *.split)
heidar has quit IRC (*.net *.split)
nerdd has quit IRC (*.net *.split)
boogeyman has quit IRC (*.net *.split)
wwrl has quit IRC (*.net *.split)
tuvwx has quit IRC (*.net *.split)
wwrl has joined #arpnetworks
robotarmy has joined #arpnetworks
DDevine has joined #arpnetworks
amdprophet has joined #arpnetworks
whitefang has joined #arpnetworks
richardquisumbin has joined #arpnetworks
vcs has joined #arpnetworks
nerdd has joined #arpnetworks
dxtr has joined #arpnetworks
mattx86 has joined #arpnetworks
cedwards has joined #arpnetworks
bob^^ has joined #arpnetworks
toddf has joined #arpnetworks
blovett has joined #arpnetworks
mhoran has joined #arpnetworks
awyeah has joined #arpnetworks
RandalSchwartz has joined #arpnetworks
Dougy has joined #arpnetworks
hiro_dSn_ has joined #arpnetworks
ww has joined #arpnetworks
baklava has joined #arpnetworks
G has joined #arpnetworks
BarberRonny has joined #arpnetworks
mtve has joined #arpnetworks
cmeiklejohn has joined #arpnetworks
boogeyman has joined #arpnetworks
jpalmer has joined #arpnetworks
Yamazaki-kun has joined #arpnetworks
mjp has joined #arpnetworks
islandfox has joined #arpnetworks
koan has joined #arpnetworks
MuskyHusky has joined #arpnetworks
jdoe has joined #arpnetworks
Lefty has joined #arpnetworks
bitslip has joined #arpnetworks
pete__c_ has joined #arpnetworks
heidar has joined #arpnetworks
bsdguy has joined #arpnetworks
mike-burns has joined #arpnetworks
azmarco has joined #arpnetworks
nuke^ has joined #arpnetworks
phlux has joined #arpnetworks
up_the_irons has joined #arpnetworks
holmes.freenode.net sets mode: +oooo toddf mhoran mike-burns up_the_irons
tuv has joined #arpnetworks
N1JER has joined #arpnetworks
infrared has joined #arpnetworks
tooth has joined #arpnetworks
tuvwx has joined #arpnetworks
nipuL has joined #arpnetworks
whitefang has quit IRC (Quit: http://lice.muppetz.com)
whitefang has joined #arpnetworks
vapor has joined #arpnetworks up_the_irons: <sigh...>
all data on mercury has been catastrophically lost
emails sent to customers offering a fresh VPS (with two months of free service)
fuck I'm going to have a long night...
and day tomorrow
i've never lost a box before, sigh... I guess this is the cost of doing business
does ANYONE know an effective way of backing up LVM volumes that doesn't tax the fuck out of the machine? whitefang: was mercury one of the first machines? up_the_irons: whitefang: it was the very first whitefang: that's unfortunate :/ up_the_irons: it is the only machine that does not have the same hardware config as the others vapor: up_the_irons, testdisk was no good?
http://www.cgsecurity.org/wiki/TestDisk
did you use this? whitefang: I thought a RAID array was supposed to stop things like complete data loss from happening. up_the_irons: whitefang: if a single disk fails, yes, it is great for that. but what if the controller goes bad? that's what I think happened here
vapor: I'm passed that at this point. I *can* get a lot of data from the lost volumes, but it is very hit or miss. running an fsck produces so many errors it is unbelievable. one may be able to find individual files they need, but the filesystems will never boot whitefang: is there any way to give the customers on that machine access to their specific filesystems do recover what they can? up_the_irons: whitefang: it's possible, but not without some work; right now, i wouldn't have time for that, just need to focus on getting the base VMs up whitefang: up_the_irons: yeah, of course...but it might soften the wrath if you can offer them said access to their possibly corrupted filesystems "soon".
and by "soon" I mean Blizzard(tm)'s sense of the word "soon". tooth: whitefang, RAID is "high availability", not a "backup". :-) up_the_irons: whitefang: yeah, you're right whitefang: up_the_irons: just tell them jpalmer is personally responsible for getting them their old filesystems, he was just telling me how bored he was the other night. :P up_the_irons: LOL