[00:42] <RandalSchwartz> nope - no grok or grope
[00:42] <RandalSchwartz> it was there, but not being used at all
[00:43] <RandalSchwartz> darn it - some mailing list I was on was looking for alternatives to easydns beacause they thought that group had pulled plug on wikileaks
[00:43] <RandalSchwartz> now I know it's not true, so I want to fix them
[00:44] <RandalSchwartz> one of my million mailing lists :0
[00:49] <RandalSchwartz> found it
[01:06] *** LT has joined #arpnetworks
[01:15] *** nakano is now known as nakano_
[02:29] *** schmir has joined #arpnetworks
[03:13] *** _Ehtyar has joined #arpnetworks
[03:31] *** Ehtyar has quit IRC (Remote host closed the connection)
[05:01] *** ziyourenxiang has joined #arpnetworks
[06:17] *** schmir has quit IRC (Remote host closed the connection)
[06:45] *** fink has joined #arpnetworks
[06:48] *** fink has quit IRC (Client Quit)
[06:48] *** fink has joined #arpnetworks
[07:17] *** DDevine has quit IRC (Remote host closed the connection)
[07:56] <jpalmer> man I love these proliant dl360's.   4 of them sitting in a rack next to my desk.  when they are idle (which they are when I'm configuring them) they are absolutely silent.
[07:58] *** fink has quit IRC (Quit: fink)
[08:02] <vcs> wow jpalmer
[08:02] <vcs> usually when configuring
[08:02] <vcs> they are on full blast
[08:02] <vcs> i freaking hate that
[08:03] <jpalmer> the dells are like that.   when you first turn the R410's on,  for like 2 minutes, you won't hear anything else.
[08:18] *** fink has joined #arpnetworks
[08:20] <vcs> yeah
[08:20] <vcs> its annoying
[08:29] <whitefang> wow
[08:29] <whitefang> those dl360's can have up to 192GB of RAM huh
[08:29] <whitefang> that's kinda insane
[08:46] <jpalmer> the only part I don't like about them,  is the 2.5" disk.
[08:49] <whitefang> but you can fit more gigabytes with a smaller disk :P
[08:52] *** cubelogic has joined #arpnetworks
[08:56] *** Sheath is now known as MuskyHusky
[09:13] *** richardquisumbin has quit IRC (Ping timeout: 265 seconds)
[09:17] <MuskyHusky> is there anything wrong with mercury
[09:19] <MuskyHusky> i cant ssh or vnc into it
[09:19] *** LT has quit IRC (Quit: Leaving)
[09:20] *** d^_^b has quit IRC (Ping timeout: 265 seconds)
[09:21] *** ziyourenxiang has quit IRC (Quit: ziyourenxiang)
[09:21] *** cubelogic has quit IRC (Remote host closed the connection)
[09:32] <whitefang> what's mercury?
[09:35] <MuskyHusky> a server at arp
[09:39] <toddf> looks like at about 20 after my vps stopped responding
[09:39] <toddf> it is also on mercury
[09:39] <MuskyHusky> yep thats the time i see when my bot pinged out
[09:40] <toddf> did you email support@ at that time?
[09:40] <MuskyHusky> no
[09:40] <toddf> I just sent an email incase I'm the first
[09:41] <MuskyHusky> my friend just told me his bot pinged out so i went to investigate
[10:24] <up_the_irons> hrm
[10:24] <up_the_irons> i can't ping mercury either
[10:25] * jdoe blames up_the_irons
[10:26] <jdoe> jpalmer: yeah, I love HP gear.
[10:26] <up_the_irons> yup, kernel panic
[10:27] <dxtr> Why do this shirt have a pleasant smell? It should smell like shit
[10:27] <dxtr> Oh well
[10:37] *** cubelogic has joined #arpnetworks
[10:45] *** nerdd_ has joined #arpnetworks
[10:48] *** nerdd_ is now known as nerdd
[10:52] <up_the_irons> well, this is great, grub just hangs at "Loading GRUB..."
[10:55] <mike-burns> Can you get into the GRUB console?
[10:55] <jdoe> worst case, presumably it's just grub that got trashed somehow...
[10:55] <jdoe> though I doubt you want to *go* anywhere ;)
[10:58] <up_the_irons> mike-burns: nope
[10:59] <up_the_irons> sometimes "Error 15" pops up
[10:59] <up_the_irons> looks like it can't find a file
[10:59] <up_the_irons> will keep working...
[11:00] <mike-burns> Did you change partitions, kernel, mountpoints, etc?
[11:07] <toddf> at that stage of the game, the mbr is accessable but the rest of the disk not so much, check that the partition table didn't get munged somehow (I've had this happen to me in the past)
[11:08] <toddf> aka I've `recovered' by `from memory' recalling the sizes of the partitions. may not be your problem. yymv. just a thought. ;-)
[11:27] *** nerdd_ has joined #arpnetworks
[11:28] *** nerdd has quit IRC (Ping timeout: 255 seconds)
[11:38] *** nerdd_ is now known as nerdd
[11:43] <jpalmer> jdoe: agreed.   everyone complains about the price of HP gear, until they actually use it.
[11:43] <jpalmer> it's hard to beat the proliant line.
[11:44] <jpalmer> "well,  this similar spec dell is 1/3rd the price!"
[11:46] <whitefang> I've never owned Dell anything ^_^
[11:46] <whitefang> we use dells at work though
[11:47] <jdoe> meh
[11:47] <jdoe> anyone complaining about the price of HP gear has never bought from a non-whitebox vendor
[11:47] <jdoe> heh
[11:50] <jpalmer> I dunno,  a Dell R410,  is about half the price of a DL360 G6.  configured similarly.  and the main difference is only that the HP would use a 2.5" disk.  the dell would use a 3.5"
[11:52] <jpalmer> well, t hat and the R410 is a bit limited when it comes to expansion.
[12:28] *** cubelogic has quit IRC (Remote host closed the connection)
[12:38] <G> jpalmer: my experience w/ HP gear has been rather painful
[12:38] <G> jpalmer: give me IBM gear anyday
[12:49] *** vcs has quit IRC (Read error: Operation timed out)
[12:49] *** whitefang has quit IRC (Read error: Operation timed out)
[12:50] <up_the_irons> Looks like there is some corruption on mercury's root fs :(  The installer USB key I have in there does not provide enough rescue tools :(  Will need to make a data center trip
[12:52] *** vcs has joined #arpnetworks
[12:53] *** schmir has joined #arpnetworks
[13:12] *** schmir has quit IRC (Ping timeout: 240 seconds)
[13:57] <jdoe> shitty :/
[13:58] *** cubelogic has joined #arpnetworks
[14:02] *** richardquisumbin has joined #arpnetworks
[14:07] *** cubelogic has quit IRC (Remote host closed the connection)
[14:09] <vcs> lol
[14:34] *** whitefang has joined #arpnetworks
[14:34] *** amdprophet has quit IRC (Quit: amdprophet)
[14:34] *** amdprophet has joined #arpnetworks
[15:34] <up_the_irons> Host mercury has been restored and boots properly. We're doing last minute checks + kernel upgrade before starting VMs.
[15:35] <fink> what was the problem?
[15:45] <up_the_irons> fucking hell
[15:45] <up_the_irons> corruption again
[15:45] <up_the_irons> hardware must be bad :(
[15:57] <toddf> memtest?
[16:03] <up_the_irons> memtest will take forever
[16:04] <toddf> you can let it run for x mins (1 usually spots really bad memory quickly)
[16:04] <toddf> but I'm sure you'll handle it in any event. bugging out for the evening. enjoy!
[16:30] <jdoe> could be a controller too
[16:30] <jdoe> could be almost anything :/
[16:42] *** robotarmy has quit IRC (Remote host closed the connection)
[16:42] *** Tranceo has joined #arpnetworks
[16:42] *** Tranceo has left 
[16:59] *** nuke- has quit IRC (Ping timeout: 265 seconds)
[17:16] *** DDevine has joined #arpnetworks
[17:17] <DDevine> I'm guessing the new host that I am on is called Mercury.
[17:18] <jpalmer> DDevine: let me guess,  outages?
[17:18] <DDevine> Actually, no!
[17:18] <DDevine> Sometimes my SSH takes a long time to prompt for password
[17:18] <DDevine> like 30 seconds +
[17:18] <jpalmer> then, I doubt it.  mercury is having some hardware issues it seems.
[17:19] <jpalmer> oh,   slow ssh logins is usually do to bad rDNS
[17:19] <DDevine> Probably. I'll get around to fixing that soon.
[17:19] <DDevine> I have already requested the transfer.
[17:41] *** robotarmy has joined #arpnetworks
[17:45] <up_the_irons> DDevine: you're not on mercury
[17:52] <DDevine> up_the_irons: Yeah, just the delay on SSH made it look so. I'll fix up my rdns today probably.
[17:56] *** wwrl has joined #arpnetworks
[18:12] <wwrl> any updates on mercury?
[18:22] <up_the_irons> wwrl: still looks bad.  I think the raid controller is bad (although it still appears to work), or something severely corrupted the disks (even the LVM volumes on which VPS' reside :(
[18:22] <up_the_irons> some of my own VMs are on there, and if i try to mount the LVM volumes, i get bad blocks
[18:23] <fink> up_the_irons: backups?
[18:29] <up_the_irons> fink: i have backups of the host root fs, but individual VMs are not backed up (responsibility of the customer to backup their stuff)
[18:30] <fink> heh
[18:30] <fink> am i on this machine?
[18:32] <up_the_irons> fink: don't know, PM your uuid
[18:35] <wwrl> if my VPS was on mercury, there's no chance that I could just have it moved to a different host machine, correct?
[18:39] <up_the_irons> wwrl: well, i'm trying to do that now as it is, but it is proving difficult
[19:00] <up_the_irons> wwrl: what is your vps uuid? (pm it to me)
[19:04] *** fink has quit IRC (Quit: fink)
[19:07] <dxtr> up_the_irons: My VPS probably isn't on Mercury. There's no chance that I could have it moved there now that it's broken and stuff?
[19:18] *** vapor has quit IRC (Ping timeout: 272 seconds)
[19:21] <up_the_irons> dxtr: i hope you're joking ;)
[19:30] <whitefang> huh, looks like FreeBSD 8.2-BETA1 was released recently fwiw to anyone that follows the -STABLE branch.
[19:36] <whitefang> up_the_irons: can you move me to mercury too plox?
[19:36] <whitefang> what's special about mercury anyways other than its broken right now.
[19:36] <up_the_irons> nothing, it's broken
[19:37] <whitefang> and why is it called mercury and not like...kvm14?
[19:37] <whitefang> or is there a special name for all kvms and I just don't know the name of mine.
[19:39] <up_the_irons> whitefang: it was created before i had a naming convention
[19:40] <whitefang> you could call it kvm0 :P
[19:40] <whitefang> or you could let me call kvm14 "venus" because it rhymes with a naughty word.
[19:45] <G> personally I call all the machines on my home network + VPS' after NZ Native Birds
[19:47] <whitefang> wow, that's a lot of machines named kiwi
[19:47] * whitefang ducks.
[19:48] <G> the machinethat I called Kiwi is now 'retired'
[20:36] <DDevine> Silly flightless birds...
[21:16] <whitefang> hmmm, I see the amount of bandwidth for my package under My Services, but I don't see my current usage.
[21:16] <whitefang> am I blind?
[21:51] *** _Ehtyar has quit IRC (Remote host closed the connection)
[21:54] *** vcs has quit IRC (*.net *.split)
[21:54] *** toddf has quit IRC (*.net *.split)
[21:54] *** RandalSchwartz has quit IRC (*.net *.split)
[21:54] *** islandfox has quit IRC (*.net *.split)
[21:54] *** nuke^ has quit IRC (*.net *.split)
[21:54] *** richardquisumbin has quit IRC (*.net *.split)
[21:54] *** amdprophet has quit IRC (*.net *.split)
[21:54] *** koan has quit IRC (*.net *.split)
[21:54] *** azmarco has quit IRC (*.net *.split)
[21:54] *** awyeah has quit IRC (*.net *.split)
[21:54] *** cmeiklejohn has quit IRC (*.net *.split)
[21:54] *** Yamazaki-kun has quit IRC (*.net *.split)
[21:54] *** mike-burns has quit IRC (*.net *.split)
[21:54] *** phlux has quit IRC (*.net *.split)
[21:54] *** up_the_irons has quit IRC (*.net *.split)
[21:54] *** tooth has quit IRC (*.net *.split)
[21:54] *** whitefang has quit IRC (*.net *.split)
[21:54] *** mattx86 has quit IRC (*.net *.split)
[21:54] *** tuv has quit IRC (*.net *.split)
[21:54] *** infrared has quit IRC (*.net *.split)
[21:54] *** cedwards has quit IRC (*.net *.split)
[21:54] *** baklava has quit IRC (*.net *.split)
[21:54] *** blovett has quit IRC (*.net *.split)
[21:54] *** Dougy has quit IRC (*.net *.split)
[21:54] *** mtve has quit IRC (*.net *.split)
[21:54] *** Lefty has quit IRC (*.net *.split)
[21:54] *** N1JER has quit IRC (*.net *.split)
[21:54] *** MuskyHusky has quit IRC (*.net *.split)
[21:54] *** bitslip has quit IRC (*.net *.split)
[21:54] *** pete__c_ has quit IRC (*.net *.split)
[21:54] *** nipuL has quit IRC (*.net *.split)
[21:54] *** robotarmy has quit IRC (*.net *.split)
[21:54] *** hiro_dSn_ has quit IRC (*.net *.split)
[21:54] *** jpalmer has quit IRC (*.net *.split)
[21:54] *** mjp has quit IRC (*.net *.split)
[21:54] *** ww has quit IRC (*.net *.split)
[21:54] *** bsdguy has quit IRC (*.net *.split)
[21:54] *** dxtr has quit IRC (*.net *.split)
[21:54] *** bob^^ has quit IRC (*.net *.split)
[21:54] *** G has quit IRC (*.net *.split)
[21:54] *** jdoe has quit IRC (*.net *.split)
[21:54] *** DDevine has quit IRC (*.net *.split)
[21:54] *** mhoran has quit IRC (*.net *.split)
[21:54] *** BarberRonny has quit IRC (*.net *.split)
[21:54] *** heidar has quit IRC (*.net *.split)
[21:54] *** nerdd has quit IRC (*.net *.split)
[21:54] *** boogeyman has quit IRC (*.net *.split)
[21:54] *** wwrl has quit IRC (*.net *.split)
[21:54] *** tuvwx has quit IRC (*.net *.split)
[21:58] *** wwrl has joined #arpnetworks
[21:58] *** robotarmy has joined #arpnetworks
[21:58] *** DDevine has joined #arpnetworks
[21:58] *** amdprophet has joined #arpnetworks
[21:58] *** whitefang has joined #arpnetworks
[21:58] *** richardquisumbin has joined #arpnetworks
[21:58] *** vcs has joined #arpnetworks
[21:58] *** nerdd has joined #arpnetworks
[21:58] *** dxtr has joined #arpnetworks
[21:58] *** mattx86 has joined #arpnetworks
[21:58] *** cedwards has joined #arpnetworks
[21:58] *** bob^^ has joined #arpnetworks
[21:58] *** toddf has joined #arpnetworks
[21:58] *** blovett has joined #arpnetworks
[21:58] *** mhoran has joined #arpnetworks
[21:58] *** awyeah has joined #arpnetworks
[21:58] *** RandalSchwartz has joined #arpnetworks
[21:58] *** Dougy has joined #arpnetworks
[21:58] *** hiro_dSn_ has joined #arpnetworks
[21:58] *** ww has joined #arpnetworks
[21:58] *** baklava has joined #arpnetworks
[21:58] *** G has joined #arpnetworks
[21:58] *** BarberRonny has joined #arpnetworks
[21:58] *** mtve has joined #arpnetworks
[21:58] *** cmeiklejohn has joined #arpnetworks
[21:58] *** boogeyman has joined #arpnetworks
[21:58] *** jpalmer has joined #arpnetworks
[21:58] *** Yamazaki-kun has joined #arpnetworks
[21:58] *** mjp has joined #arpnetworks
[21:58] *** islandfox has joined #arpnetworks
[21:58] *** koan has joined #arpnetworks
[21:58] *** MuskyHusky has joined #arpnetworks
[21:58] *** jdoe has joined #arpnetworks
[21:58] *** Lefty has joined #arpnetworks
[21:58] *** bitslip has joined #arpnetworks
[21:58] *** pete__c_ has joined #arpnetworks
[21:58] *** heidar has joined #arpnetworks
[21:58] *** bsdguy has joined #arpnetworks
[21:58] *** mike-burns has joined #arpnetworks
[21:58] *** azmarco has joined #arpnetworks
[21:58] *** nuke^ has joined #arpnetworks
[21:58] *** phlux has joined #arpnetworks
[21:58] *** up_the_irons has joined #arpnetworks
[21:58] *** holmes.freenode.net sets mode: +oooo toddf mhoran mike-burns up_the_irons
[21:58] *** tuv has joined #arpnetworks
[21:58] *** N1JER has joined #arpnetworks
[21:58] *** infrared has joined #arpnetworks
[21:58] *** tooth has joined #arpnetworks
[21:58] *** tuvwx has joined #arpnetworks
[21:58] *** nipuL has joined #arpnetworks
[22:40] *** whitefang has quit IRC (Quit: http://lice.muppetz.com)
[22:50] *** whitefang has joined #arpnetworks
[22:52] *** vapor has joined #arpnetworks
[23:43] <up_the_irons> <sigh...>
[23:43] <up_the_irons> all data on mercury has been catastrophically lost
[23:44] <up_the_irons> emails sent to customers offering a fresh VPS (with two months of free service)
[23:44] <up_the_irons> fuck I'm going to have a long night...
[23:44] <up_the_irons> and day tomorrow
[23:44] <up_the_irons> i've never lost a box before, sigh...  I guess this is the cost of doing business
[23:45] <up_the_irons> does ANYONE know an effective way of backing up LVM volumes that doesn't tax the fuck out of the machine?
[23:45] <whitefang> was mercury one of the first machines?
[23:45] <up_the_irons> whitefang: it was the very first
[23:45] <whitefang> that's unfortunate :/
[23:45] <up_the_irons> it is the only machine that does not have the same hardware config as the others
[23:45] <vapor> up_the_irons, testdisk was no good?
[23:47] <vapor> http://www.cgsecurity.org/wiki/TestDisk
[23:47] <vapor> did you use this?
[23:48] <whitefang> I thought a RAID array was supposed to stop things like complete data loss from happening.
[23:49] <up_the_irons> whitefang: if a single disk fails, yes, it is great for that.  but what if the controller goes bad?  that's what I think happened here
[23:50] <up_the_irons> vapor: I'm passed that at this point.  I *can* get a lot of data from the lost volumes, but it is very hit or miss.  running an fsck produces so many errors it is unbelievable.  one may be able to find individual files they need, but the filesystems will never boot
[23:51] <whitefang> is there any way to give the customers on that machine access to their specific filesystems do recover what they can?
[23:52] <up_the_irons> whitefang: it's possible, but not without some work; right now, i wouldn't have time for that, just need to focus on getting the base VMs up
[23:53] <whitefang> up_the_irons: yeah, of course...but it might soften the wrath if you can offer them said access to their possibly corrupted filesystems "soon".
[23:55] <whitefang> and by "soon" I mean Blizzard(tm)'s sense of the word "soon".
[23:56] <tooth> whitefang, RAID is "high availability", not a "backup". :-)
[23:56] <up_the_irons> whitefang: yeah, you're right
[23:57] <whitefang> up_the_irons: just tell them jpalmer is personally responsible for getting them their old filesystems, he was just telling me how bored he was the other night. :P
[23:59] <up_the_irons> LOL