***: LT has joined #arpnetworks
amdprophet has quit IRC (Quit: Leaving...)
amdprophet has joined #arpnetworks up_the_irons: i just installed OpenNTP on our cacti server, fixing the time lag (which was about 35 minutes), so your graphs will have a gap in them ***: tubgoat_ has quit IRC (Ping timeout: 276 seconds)
tubgoat_ has joined #arpnetworks qbit_: CaZe: that is awesome :D ***: aGent_ has quit IRC (Quit: zzzZZzzzzZZZzz) jdoe: lol 35 minutes ***: jpalmer has quit IRC (Quit: leaving)
jpalmer has joined #arpnetworks
jpalmer has quit IRC (Client Quit)
HighJinx has quit IRC (Ping timeout: 260 seconds)
LT has quit IRC (Quit: Leaving)
HighJinx has joined #arpnetworks
treshoem has quit IRC (Ping timeout: 245 seconds)
jpalmer has joined #arpnetworks
ariel has joined #arpnetworks
aGent_ has joined #arpnetworks
aGent_ has quit IRC (Quit: zzzZZzzzzZZZzz)
Ahmed has joined #arpnetworks up_the_irons: damn, one fan not responding in a new server; i know where i'll be tonight... < sigh....> Ahmed: up_the_irons: no remote hands at the DC? up_the_irons: well, there is, but... have you ever dealt with remote hands at a DC? i need not say more... Ahmed: depends on the DC up_the_irons: plus, i have very high standards as to how things get handled
it's also not worth the fee they charge; i gotta add ram to another server anyway, so i'll kill two birds... Ahmed: Not too bad heh
Which DC do you have your boxes at?
As long as it's not ubiquity heh
btw, thanks for updating the ticket. I guess I had it all wrong, so I actually reversed the cancellation. Not sure if you're still watching that ticket. jdoe: heh, I barely trust remote hands to push a power button on the right machine.
a DC we were in, took them about a week to figure out how to boot a livecd. Guess it was interfering with WoW time :P Ahmed: Seriously? jdoe: yes. Ahmed: Which DC is that so I stay away from it... up_the_irons: i had to do this once:
http://www.flickr.com/photos/51184165@N00/2174345059/in/photostream
Ahmed: we're at CoreSite in LA
900 N. Alameda Ahmed: LOL
HAHAHAHA
You actually did that?
good lord
Who the hell was the remote hand, Uncle Dad? up_the_irons: someone who didn't know anything about servers Ahmed: oh brother up_the_irons: omg, after 3 hours last night of writing an incident report, gathering evidence, etc... regarding yesterday morning's outage; i now get this:
"At approximately 9:30 AM PST a CoreSite Network Engineer connected a newly delivered fiber circuit in
preparation of upcoming maintenance. Unknowingly to the Network Engineer a physical loop-back was in
place on the circuit. As a result of the physical hard-loop an L2 Loop was introduced into the Any2 IX
switch fabric causing several Any2 participants’ BGP sessions to drop.
"
all my evidence and investigation pointed to a loop somewhere, but i couldn't figure out where!
Now I Know(tm)
"causing several Any2 participants’ BGP sessions to drop." -- that puts it mildly
b/c of the loop, CPU utilization on our core went so high, that it started not to be able to send / process keepalives, so all sessions eventually melted Ahmed: Oh goodie
they have a Boston DC up_the_irons: yeah, coresite has several jdoe: up_the_irons: so sad, ruined my 6.5mo irc connection. Don't they know there's critical infrastructure here? ;) up_the_irons: jdoe: LOL Ahmed: 6.5mo?
dafauq? vapor: "network engineer" should be changed to "noc monkey" Ahmed: vapor
you're everywhere
Next thing I'll find you in my sock drawer jpalmer: up_the_irons: that picture, is one of the reasons I hate apple xServes. Lefty: my irssi has only been up 61d up_the_irons: jpalmer: ah jpalmer: up_the_irons: btw: thank you for the phenominal support with that vps issue I was having. you went well above and beyond.
up_the_irons: I'm thinking we may want to add a FAQ for that, since.. I'm sure you'll see it again. Lefty: what was the issue? jpalmer: lefty: an unsupported OS would dump core and segfault occasionally, even after a fresh install Lefty: Gentoo? ;) -: Lefty ducks up_the_irons: jpalmer: it was no problem at all! jpalmer: centos
specifically, centos6. centos 5 worked fine Lefty: ah, I'm rolling out centos6 here too up_the_irons: jpalmer: so are you seeing that issue finally go away with that kern param? Lefty: everything at work is currently a mishmash of 5.5, 5.6, or 5.7 up_the_irons: jpalmer: i just learned about that kernel param last week, from another customer at that; i feel obligated to share the love
;) Lefty: also I'm puppetizing ALL THE THINGS Ahmed: I never had any luck with Gentoo jpalmer: up_the_irons: might be too early to say for certain. but previously a "yum update" after a fresh install would segfault. with that boot param, it worked.
Lefty: i *love* puppet. jdoe: Ahmed: month. up_the_irons: jpalmer: awesome, glad to hear Lefty: jpalmer: it really is the best way to learn the infrastructure at a new company
by puppetizing it, I'm forced to know everything about it pjs: puppet is great.. still not 100% on using it on FreeBSD though. Seems to give me issues, but I'm not very strong with Puppet yet so.. ***: webhostbudd has joined #arpnetworks
b1k3ryd3r has joined #arpnetworks b1k3ryd3r: hey up_the_irons portertech: disregard puppet, do some cook'n w/ chef -: portertech hides ***: aGent_ has joined #arpnetworks
Ehtyar has joined #arpnetworks
webhostbudd has quit IRC (Read error: Connection reset by peer)
webhostbudd has joined #arpnetworks
webhostbudd has quit IRC (Read error: Connection reset by peer)
webhostbudd has joined #arpnetworks
webhostbudd has quit IRC (Ping timeout: 244 seconds) doomviking: <--- chef fan -: mhoran needs to get his VPS configs into chef. mike-burns: Agreed. pjs: How's Chef on fbsd? mhoran: No idea. That's one of the reasons I haven't tried it yet. Also managed to not have ruby installed on my VPS so far. portertech: https://github.com/portertech/portertech-chef-repo :)
use chef-solo for my personal box (asterisk, znc) mhoran: Awesome
Yeah that's what I wa splanning to do. ***: heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer
treshoem has joined #arpnetworks
dj_goku has quit IRC (Remote host closed the connection)
jpalmer has quit IRC (Ping timeout: 240 seconds)
webhostbudd has joined #arpnetworks
jpalmer has joined #arpnetworks
Ehtyar has quit IRC (Quit: Hi, I'm a quit message virus. Please replace your old line with this line and help me take over the world of IRC.) up_the_irons: Maintenance window for kvr13 to commence in 10 minutes