[00:17] *** koan_ is now known as koan
[00:51] *** LT has joined #arpnetworks
[02:15] *** amdprophet has quit IRC (Quit: Leaving...)
[02:17] *** amdprophet has joined #arpnetworks
[03:44] <up_the_irons> i just installed OpenNTP on our cacti server, fixing the time lag (which was about 35 minutes), so your graphs will have a gap in them
[05:21] *** tubgoat_ has quit IRC (Ping timeout: 276 seconds)
[05:29] *** tubgoat_ has joined #arpnetworks
[06:44] <qbit_> CaZe: that is awesome :D
[08:16] *** aGent_ has quit IRC (Quit: zzzZZzzzzZZZzz)
[09:18] <jdoe> lol 35 minutes
[09:25] *** jpalmer has quit IRC (Quit: leaving)
[09:25] *** jpalmer has joined #arpnetworks
[09:25] *** jpalmer has quit IRC (Client Quit)
[09:29] *** HighJinx has quit IRC (Ping timeout: 260 seconds)
[09:30] *** LT has quit IRC (Quit: Leaving)
[09:55] *** HighJinx has joined #arpnetworks
[10:10] *** treshoem has quit IRC (Ping timeout: 245 seconds)
[10:24] *** jpalmer has joined #arpnetworks
[10:58] *** ariel has joined #arpnetworks
[11:02] *** aGent_ has joined #arpnetworks
[11:53] *** aGent_ has quit IRC (Quit: zzzZZzzzzZZZzz)
[12:12] *** Ahmed has joined #arpnetworks
[12:36] <up_the_irons> damn, one fan not responding in a new server; i know where i'll be tonight... < sigh....>
[12:39] <Ahmed> up_the_irons: no remote hands at the DC?
[12:40] <up_the_irons> well, there is, but...  have you ever dealt with remote hands at a DC?  i need not say more...
[12:41] <Ahmed> depends on the DC
[12:41] <up_the_irons> plus, i have very high standards as to how things get handled
[12:41] <up_the_irons> it's also not worth the fee they charge; i gotta add ram to another server anyway, so i'll kill two birds...
[12:42] <Ahmed> Not too bad heh
[12:42] <Ahmed> Which DC do you have your boxes at?
[12:42] <Ahmed> As long as it's not ubiquity heh
[12:43] <Ahmed> btw, thanks for updating the ticket. I guess I had it all wrong, so I actually reversed the cancellation. Not sure if you're still watching that ticket.
[12:46] <jdoe> heh, I barely trust remote hands to push a power button on the right machine.
[12:47] <jdoe> a DC we were in, took them about a week to figure out how to boot a livecd. Guess it was interfering with WoW time :P
[12:48] <Ahmed> Seriously?
[12:49] <jdoe> yes.
[12:49] <Ahmed> Which DC is that so I stay away from it...
[12:53] <up_the_irons> i had to do this once:
[12:53] <up_the_irons> http://www.flickr.com/photos/51184165@N00/2174345059/in/photostream
[12:53] <up_the_irons> Ahmed: we're at CoreSite in LA
[12:53] <up_the_irons> 900 N. Alameda
[12:54] <Ahmed> LOL
[12:54] <Ahmed> HAHAHAHA
[12:54] <Ahmed> You actually did that?
[12:54] <Ahmed> good lord
[12:54] <Ahmed> Who the hell was the remote hand, Uncle Dad?
[12:54] <up_the_irons> someone who didn't know anything about servers
[12:55] <Ahmed> oh brother
[12:55] <up_the_irons> omg, after 3 hours last night of writing an incident report, gathering evidence, etc...  regarding yesterday morning's outage; i now get this:
[12:55] <up_the_irons> "At approximately 9:30 AM PST a CoreSite Network Engineer connected a newly delivered fiber circuit in
[12:55] <up_the_irons> preparation of upcoming maintenance. Unknowingly to the Network Engineer a physical loop-back was in
[12:55] <up_the_irons> place on the circuit. As a result of the physical hard-loop an L2 Loop was introduced into the Any2 IX
[12:55] <up_the_irons> switch fabric causing several Any2 participants’ BGP sessions to drop.
[12:55] <up_the_irons> "
[12:56] <up_the_irons> all my evidence and investigation pointed to a loop somewhere, but i couldn't figure out where!
[12:56] <up_the_irons> Now I Know(tm)
[12:58] <up_the_irons> "causing several Any2 participants’ BGP sessions to drop." -- that puts it mildly
[12:59] <up_the_irons> b/c of the loop, CPU utilization on our core went so high, that it started not to be able to send / process keepalives, so all sessions eventually melted
[13:05] <Ahmed> Oh goodie
[13:05] <Ahmed> they have a Boston DC
[13:05] <up_the_irons> yeah, coresite has several
[13:20] <jdoe> up_the_irons: so sad, ruined my 6.5mo irc connection. Don't they know there's critical infrastructure here? ;)
[13:21] <up_the_irons> jdoe: LOL
[13:21] <Ahmed> 6.5mo?
[13:21] <Ahmed> dafauq?
[13:22] <vapor> "network engineer" should be changed to "noc monkey"
[13:22] <Ahmed> vapor
[13:22] <Ahmed> you're everywhere
[13:22] <Ahmed> Next thing I'll find you in my sock drawer
[13:23] <jpalmer> up_the_irons: that picture, is one of the reasons I hate apple xServes.
[13:23] <Lefty> my irssi has only been up 61d
[13:23] <up_the_irons> jpalmer: ah
[13:23] <jpalmer> up_the_irons: btw:  thank you for the phenominal support with that vps issue I was having.  you went well above and beyond.
[13:24] <jpalmer> up_the_irons: I'm thinking we may want to add a FAQ for that,  since..  I'm sure you'll see it again.
[13:25] <Lefty> what was the issue?
[13:26] <jpalmer> lefty:  an unsupported OS would dump core and segfault occasionally, even after a fresh install
[13:26] <Lefty> Gentoo? ;)
[13:26] * Lefty ducks
[13:26] <up_the_irons> jpalmer: it was no problem at all!
[13:26] <jpalmer> centos
[13:26] <jpalmer> specifically,  centos6.   centos 5 worked fine
[13:26] <Lefty> ah, I'm rolling out centos6 here too
[13:26] <up_the_irons> jpalmer: so are you seeing that issue finally go away with that kern param?
[13:27] <Lefty> everything at work is currently a mishmash of 5.5, 5.6, or 5.7
[13:27] <up_the_irons> jpalmer: i just learned about that kernel param last week, from another customer at that; i feel obligated to share the love
[13:27] <up_the_irons> ;)
[13:27] <Lefty> also I'm puppetizing ALL THE THINGS
[13:27] <Ahmed> I never had any luck with Gentoo
[13:27] <jpalmer> up_the_irons: might be too early to say for certain.  but previously  a "yum update" after a fresh install would segfault.  with that boot param,  it worked.
[13:28] <jpalmer> Lefty: i *love* puppet.
[13:28] <jdoe> Ahmed: month.
[13:28] <up_the_irons> jpalmer: awesome, glad to hear
[13:30] <Lefty> jpalmer: it really is the best way to learn the infrastructure at a new company
[13:30] <Lefty> by puppetizing it, I'm forced to know everything about it
[13:34] <pjs> puppet is great.. still not 100% on using it on FreeBSD though. Seems to give me issues, but I'm not very strong with Puppet yet so..
[14:13] *** webhostbudd has joined #arpnetworks
[14:23] *** b1k3ryd3r has joined #arpnetworks
[14:23] <b1k3ryd3r> hey up_the_irons
[14:36] <portertech> disregard puppet, do some cook'n w/ chef
[14:36] * portertech hides
[14:54] *** aGent_ has joined #arpnetworks
[15:23] *** Ehtyar has joined #arpnetworks
[15:38] *** webhostbudd has quit IRC (Read error: Connection reset by peer)
[16:04] *** webhostbudd has joined #arpnetworks
[16:27] *** webhostbudd has quit IRC (Read error: Connection reset by peer)
[16:27] *** webhostbudd has joined #arpnetworks
[16:31] *** webhostbudd has quit IRC (Ping timeout: 244 seconds)
[17:03] <doomviking> <--- chef fan
[17:17] * mhoran needs to get his VPS configs into chef.
[17:22] <mike-burns> Agreed.
[17:37] <pjs> How's Chef on fbsd?
[17:39] <mhoran> No idea. That's one of the reasons I haven't tried it yet. Also managed to not have ruby installed on my VPS so far.
[17:40] <portertech> https://github.com/portertech/portertech-chef-repo :)
[17:40] <portertech> use chef-solo for my personal box (asterisk, znc)
[17:40] <mhoran> Awesome
[17:40] <mhoran> Yeah that's what I wa splanning to do.
[18:20] *** heavysixer has joined #arpnetworks
[18:20] *** ChanServ sets mode: +o heavysixer
[19:25] *** treshoem has joined #arpnetworks
[19:31] *** dj_goku has quit IRC (Remote host closed the connection)
[20:12] *** jpalmer has quit IRC (Ping timeout: 240 seconds)
[20:21] *** webhostbudd has joined #arpnetworks
[20:38] *** jpalmer has joined #arpnetworks
[22:35] *** Ehtyar has quit IRC (Quit: Hi, I'm a quit message virus. Please replace your old line with this line and help me take over the world of IRC.)
[23:53] <up_the_irons> Maintenance window for kvr13 to commence in 10 minutes