#arpnetworks 2010-12-09,Thu

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
RandalSchwartznope - no grok or grope
it was there, but not being used at all
darn it - some mailing list I was on was looking for alternatives to easydns beacause they thought that group had pulled plug on wikileaks
now I know it's not true, so I want to fix them
one of my million mailing lists :0
[00:42]
found it [00:49]
.... (idle for 17mn)
***LT has joined #arpnetworks [01:06]
nakano is now known as nakano_ [01:15]
............... (idle for 1h14mn)
schmir has joined #arpnetworks [02:29]
......... (idle for 44mn)
_Ehtyar has joined #arpnetworks [03:13]
.... (idle for 18mn)
Ehtyar has quit IRC (Remote host closed the connection) [03:31]
................... (idle for 1h30mn)
ziyourenxiang has joined #arpnetworks [05:01]
................ (idle for 1h16mn)
schmir has quit IRC (Remote host closed the connection) [06:17]
...... (idle for 28mn)
fink has joined #arpnetworks
fink has quit IRC (Client Quit)
fink has joined #arpnetworks
[06:45]
...... (idle for 29mn)
DDevine has quit IRC (Remote host closed the connection) [07:17]
........ (idle for 39mn)
jpalmerman I love these proliant dl360's. 4 of them sitting in a rack next to my desk. when they are idle (which they are when I'm configuring them) they are absolutely silent. [07:56]
***fink has quit IRC (Quit: fink) [07:58]
vcswow jpalmer
usually when configuring
they are on full blast
i freaking hate that
[08:02]
jpalmerthe dells are like that. when you first turn the R410's on, for like 2 minutes, you won't hear anything else. [08:03]
.... (idle for 15mn)
***fink has joined #arpnetworks [08:18]
vcsyeah
its annoying
[08:20]
whitefangwow
those dl360's can have up to 192GB of RAM huh
that's kinda insane
[08:29]
.... (idle for 17mn)
jpalmerthe only part I don't like about them, is the 2.5" disk. [08:46]
whitefangbut you can fit more gigabytes with a smaller disk :P [08:49]
***cubelogic has joined #arpnetworks
Sheath is now known as MuskyHusky
[08:52]
.... (idle for 17mn)
richardquisumbin has quit IRC (Ping timeout: 265 seconds) [09:13]
MuskyHuskyis there anything wrong with mercury
i cant ssh or vnc into it
[09:17]
***LT has quit IRC (Quit: Leaving)
d^_^b has quit IRC (Ping timeout: 265 seconds)
ziyourenxiang has quit IRC (Quit: ziyourenxiang)
cubelogic has quit IRC (Remote host closed the connection)
[09:19]
whitefangwhat's mercury? [09:32]
MuskyHuskya server at arp [09:35]
toddflooks like at about 20 after my vps stopped responding
it is also on mercury
[09:39]
MuskyHuskyyep thats the time i see when my bot pinged out [09:39]
toddfdid you email support@ at that time? [09:40]
MuskyHuskyno [09:40]
toddfI just sent an email incase I'm the first [09:40]
MuskyHuskymy friend just told me his bot pinged out so i went to investigate [09:41]
......... (idle for 43mn)
up_the_ironshrm
i can't ping mercury either
[10:24]
jdoejdoe blames up_the_irons
jpalmer: yeah, I love HP gear.
[10:25]
up_the_ironsyup, kernel panic [10:26]
dxtrWhy do this shirt have a pleasant smell? It should smell like shit
Oh well
[10:27]
***cubelogic has joined #arpnetworks [10:37]
nerdd_ has joined #arpnetworks
nerdd_ is now known as nerdd
[10:45]
up_the_ironswell, this is great, grub just hangs at "Loading GRUB..." [10:52]
mike-burnsCan you get into the GRUB console? [10:55]
jdoeworst case, presumably it's just grub that got trashed somehow...
though I doubt you want to *go* anywhere ;)
[10:55]
up_the_ironsmike-burns: nope
sometimes "Error 15" pops up
looks like it can't find a file
will keep working...
[10:58]
mike-burnsDid you change partitions, kernel, mountpoints, etc? [11:00]
toddfat that stage of the game, the mbr is accessable but the rest of the disk not so much, check that the partition table didn't get munged somehow (I've had this happen to me in the past)
aka I've `recovered' by `from memory' recalling the sizes of the partitions. may not be your problem. yymv. just a thought. ;-)
[11:07]
.... (idle for 19mn)
***nerdd_ has joined #arpnetworks
nerdd has quit IRC (Ping timeout: 255 seconds)
[11:27]
nerdd_ is now known as nerdd [11:38]
jpalmerjdoe: agreed. everyone complains about the price of HP gear, until they actually use it.
it's hard to beat the proliant line.
"well, this similar spec dell is 1/3rd the price!"
[11:43]
whitefangI've never owned Dell anything ^_^
we use dells at work though
[11:46]
jdoemeh
anyone complaining about the price of HP gear has never bought from a non-whitebox vendor
heh
[11:47]
jpalmerI dunno, a Dell R410, is about half the price of a DL360 G6. configured similarly. and the main difference is only that the HP would use a 2.5" disk. the dell would use a 3.5"
well, t hat and the R410 is a bit limited when it comes to expansion.
[11:50]
........ (idle for 36mn)
***cubelogic has quit IRC (Remote host closed the connection) [12:28]
Gjpalmer: my experience w/ HP gear has been rather painful
jpalmer: give me IBM gear anyday
[12:38]
***vcs has quit IRC (Read error: Operation timed out)
whitefang has quit IRC (Read error: Operation timed out)
[12:49]
up_the_ironsLooks like there is some corruption on mercury's root fs :( The installer USB key I have in there does not provide enough rescue tools :( Will need to make a data center trip [12:50]
***vcs has joined #arpnetworks
schmir has joined #arpnetworks
[12:52]
.... (idle for 19mn)
schmir has quit IRC (Ping timeout: 240 seconds) [13:12]
.......... (idle for 45mn)
jdoeshitty :/ [13:57]
***cubelogic has joined #arpnetworks
richardquisumbin has joined #arpnetworks
[13:58]
cubelogic has quit IRC (Remote host closed the connection) [14:07]
vcslol [14:09]
...... (idle for 25mn)
***whitefang has joined #arpnetworks
amdprophet has quit IRC (Quit: amdprophet)
amdprophet has joined #arpnetworks
[14:34]
............. (idle for 1h0mn)
up_the_ironsHost mercury has been restored and boots properly. We're doing last minute checks + kernel upgrade before starting VMs. [15:34]
finkwhat was the problem? [15:35]
up_the_ironsfucking hell
corruption again
hardware must be bad :(
[15:45]
toddfmemtest? [15:57]
up_the_ironsmemtest will take forever [16:03]
toddfyou can let it run for x mins (1 usually spots really bad memory quickly)
but I'm sure you'll handle it in any event. bugging out for the evening. enjoy!
[16:04]
...... (idle for 26mn)
jdoecould be a controller too
could be almost anything :/
[16:30]
***robotarmy has quit IRC (Remote host closed the connection)
Tranceo has joined #arpnetworks
Tranceo has left
[16:42]
.... (idle for 17mn)
nuke- has quit IRC (Ping timeout: 265 seconds) [16:59]
.... (idle for 17mn)
DDevine has joined #arpnetworks [17:16]
DDevineI'm guessing the new host that I am on is called Mercury. [17:17]
jpalmerDDevine: let me guess, outages? [17:18]
DDevineActually, no!
Sometimes my SSH takes a long time to prompt for password
like 30 seconds +
[17:18]
jpalmerthen, I doubt it. mercury is having some hardware issues it seems.
oh, slow ssh logins is usually do to bad rDNS
[17:18]
DDevineProbably. I'll get around to fixing that soon.
I have already requested the transfer.
[17:19]
..... (idle for 22mn)
***robotarmy has joined #arpnetworks [17:41]
up_the_ironsDDevine: you're not on mercury [17:45]
DDevineup_the_irons: Yeah, just the delay on SSH made it look so. I'll fix up my rdns today probably. [17:52]
***wwrl has joined #arpnetworks [17:56]
.... (idle for 16mn)
wwrlany updates on mercury? [18:12]
up_the_ironswwrl: still looks bad. I think the raid controller is bad (although it still appears to work), or something severely corrupted the disks (even the LVM volumes on which VPS' reside :(
some of my own VMs are on there, and if i try to mount the LVM volumes, i get bad blocks
[18:22]
finkup_the_irons: backups? [18:23]
up_the_ironsfink: i have backups of the host root fs, but individual VMs are not backed up (responsibility of the customer to backup their stuff) [18:29]
finkheh
am i on this machine?
[18:30]
up_the_ironsfink: don't know, PM your uuid [18:32]
wwrlif my VPS was on mercury, there's no chance that I could just have it moved to a different host machine, correct? [18:35]
up_the_ironswwrl: well, i'm trying to do that now as it is, but it is proving difficult [18:39]
..... (idle for 21mn)
wwrl: what is your vps uuid? (pm it to me) [19:00]
***fink has quit IRC (Quit: fink) [19:04]
dxtrup_the_irons: My VPS probably isn't on Mercury. There's no chance that I could have it moved there now that it's broken and stuff? [19:07]
***vapor has quit IRC (Ping timeout: 272 seconds) [19:18]
up_the_ironsdxtr: i hope you're joking ;) [19:21]
whitefanghuh, looks like FreeBSD 8.2-BETA1 was released recently fwiw to anyone that follows the -STABLE branch. [19:30]
up_the_irons: can you move me to mercury too plox?
what's special about mercury anyways other than its broken right now.
[19:36]
up_the_ironsnothing, it's broken [19:36]
whitefangand why is it called mercury and not like...kvm14?
or is there a special name for all kvms and I just don't know the name of mine.
[19:37]
up_the_ironswhitefang: it was created before i had a naming convention [19:39]
whitefangyou could call it kvm0 :P
or you could let me call kvm14 "venus" because it rhymes with a naughty word.
[19:40]
Gpersonally I call all the machines on my home network + VPS' after NZ Native Birds [19:45]
whitefangwow, that's a lot of machines named kiwi
whitefang ducks.
[19:47]
Gthe machinethat I called Kiwi is now 'retired' [19:48]
.......... (idle for 48mn)
DDevineSilly flightless birds... [20:36]
......... (idle for 40mn)
whitefanghmmm, I see the amount of bandwidth for my package under My Services, but I don't see my current usage.
am I blind?
[21:16]
........ (idle for 35mn)
***_Ehtyar has quit IRC (Remote host closed the connection)
vcs has quit IRC (*.net *.split)
toddf has quit IRC (*.net *.split)
RandalSchwartz has quit IRC (*.net *.split)
islandfox has quit IRC (*.net *.split)
nuke^ has quit IRC (*.net *.split)
richardquisumbin has quit IRC (*.net *.split)
amdprophet has quit IRC (*.net *.split)
koan has quit IRC (*.net *.split)
azmarco has quit IRC (*.net *.split)
awyeah has quit IRC (*.net *.split)
cmeiklejohn has quit IRC (*.net *.split)
Yamazaki-kun has quit IRC (*.net *.split)
mike-burns has quit IRC (*.net *.split)
phlux has quit IRC (*.net *.split)
up_the_irons has quit IRC (*.net *.split)
tooth has quit IRC (*.net *.split)
whitefang has quit IRC (*.net *.split)
mattx86 has quit IRC (*.net *.split)
tuv has quit IRC (*.net *.split)
infrared has quit IRC (*.net *.split)
cedwards has quit IRC (*.net *.split)
baklava has quit IRC (*.net *.split)
blovett has quit IRC (*.net *.split)
Dougy has quit IRC (*.net *.split)
mtve has quit IRC (*.net *.split)
Lefty has quit IRC (*.net *.split)
N1JER has quit IRC (*.net *.split)
MuskyHusky has quit IRC (*.net *.split)
bitslip has quit IRC (*.net *.split)
pete__c_ has quit IRC (*.net *.split)
nipuL has quit IRC (*.net *.split)
robotarmy has quit IRC (*.net *.split)
hiro_dSn_ has quit IRC (*.net *.split)
jpalmer has quit IRC (*.net *.split)
mjp has quit IRC (*.net *.split)
ww has quit IRC (*.net *.split)
bsdguy has quit IRC (*.net *.split)
dxtr has quit IRC (*.net *.split)
bob^^ has quit IRC (*.net *.split)
G has quit IRC (*.net *.split)
jdoe has quit IRC (*.net *.split)
DDevine has quit IRC (*.net *.split)
mhoran has quit IRC (*.net *.split)
BarberRonny has quit IRC (*.net *.split)
heidar has quit IRC (*.net *.split)
nerdd has quit IRC (*.net *.split)
boogeyman has quit IRC (*.net *.split)
wwrl has quit IRC (*.net *.split)
tuvwx has quit IRC (*.net *.split)
wwrl has joined #arpnetworks
robotarmy has joined #arpnetworks
DDevine has joined #arpnetworks
amdprophet has joined #arpnetworks
whitefang has joined #arpnetworks
richardquisumbin has joined #arpnetworks
vcs has joined #arpnetworks
nerdd has joined #arpnetworks
dxtr has joined #arpnetworks
mattx86 has joined #arpnetworks
cedwards has joined #arpnetworks
bob^^ has joined #arpnetworks
toddf has joined #arpnetworks
blovett has joined #arpnetworks
mhoran has joined #arpnetworks
awyeah has joined #arpnetworks
RandalSchwartz has joined #arpnetworks
Dougy has joined #arpnetworks
hiro_dSn_ has joined #arpnetworks
ww has joined #arpnetworks
baklava has joined #arpnetworks
G has joined #arpnetworks
BarberRonny has joined #arpnetworks
mtve has joined #arpnetworks
cmeiklejohn has joined #arpnetworks
boogeyman has joined #arpnetworks
jpalmer has joined #arpnetworks
Yamazaki-kun has joined #arpnetworks
mjp has joined #arpnetworks
islandfox has joined #arpnetworks
koan has joined #arpnetworks
MuskyHusky has joined #arpnetworks
jdoe has joined #arpnetworks
Lefty has joined #arpnetworks
bitslip has joined #arpnetworks
pete__c_ has joined #arpnetworks
heidar has joined #arpnetworks
bsdguy has joined #arpnetworks
mike-burns has joined #arpnetworks
azmarco has joined #arpnetworks
nuke^ has joined #arpnetworks
phlux has joined #arpnetworks
up_the_irons has joined #arpnetworks
holmes.freenode.net sets mode: +oooo toddf mhoran mike-burns up_the_irons
tuv has joined #arpnetworks
N1JER has joined #arpnetworks
infrared has joined #arpnetworks
tooth has joined #arpnetworks
tuvwx has joined #arpnetworks
nipuL has joined #arpnetworks
[21:51]
......... (idle for 42mn)
whitefang has quit IRC (Quit: http://lice.muppetz.com) [22:40]
whitefang has joined #arpnetworks
vapor has joined #arpnetworks
[22:50]
........... (idle for 51mn)
up_the_irons<sigh...>
all data on mercury has been catastrophically lost
emails sent to customers offering a fresh VPS (with two months of free service)
fuck I'm going to have a long night...
and day tomorrow
i've never lost a box before, sigh... I guess this is the cost of doing business
does ANYONE know an effective way of backing up LVM volumes that doesn't tax the fuck out of the machine?
[23:43]
whitefangwas mercury one of the first machines? [23:45]
up_the_ironswhitefang: it was the very first [23:45]
whitefangthat's unfortunate :/ [23:45]
up_the_ironsit is the only machine that does not have the same hardware config as the others [23:45]
vaporup_the_irons, testdisk was no good?
http://www.cgsecurity.org/wiki/TestDisk
did you use this?
[23:45]
whitefangI thought a RAID array was supposed to stop things like complete data loss from happening. [23:48]
up_the_ironswhitefang: if a single disk fails, yes, it is great for that. but what if the controller goes bad? that's what I think happened here
vapor: I'm passed that at this point. I *can* get a lot of data from the lost volumes, but it is very hit or miss. running an fsck produces so many errors it is unbelievable. one may be able to find individual files they need, but the filesystems will never boot
[23:49]
whitefangis there any way to give the customers on that machine access to their specific filesystems do recover what they can? [23:51]
up_the_ironswhitefang: it's possible, but not without some work; right now, i wouldn't have time for that, just need to focus on getting the base VMs up [23:52]
whitefangup_the_irons: yeah, of course...but it might soften the wrath if you can offer them said access to their possibly corrupted filesystems "soon".
and by "soon" I mean Blizzard(tm)'s sense of the word "soon".
[23:53]
toothwhitefang, RAID is "high availability", not a "backup". :-) [23:56]
up_the_ironswhitefang: yeah, you're right [23:56]
whitefangup_the_irons: just tell them jpalmer is personally responsible for getting them their old filesystems, he was just telling me how bored he was the other night. :P [23:57]
up_the_ironsLOL [23:59]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)