#arpnetworks 2012-02-01,Wed

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)
WhoWhatWhen
***koan_ is now known as koan [00:17]
....... (idle for 34mn)
LT has joined #arpnetworks [00:51]
................. (idle for 1h24mn)
amdprophet has quit IRC (Quit: Leaving...)
amdprophet has joined #arpnetworks 		[02:15]
.................. (idle for 1h27mn)
up_the_ironsi just installed OpenNTP on our cacti server, fixing the time lag (which was about 35 minutes), so your graphs will have a gap in them [03:44]
.................... (idle for 1h37mn)
***tubgoat_ has quit IRC (Ping timeout: 276 seconds) [05:21]
tubgoat_ has joined #arpnetworks [05:29]
................ (idle for 1h15mn)
qbit_CaZe: that is awesome :D [06:44]
................... (idle for 1h32mn)
***aGent_ has quit IRC (Quit: zzzZZzzzzZZZzz) [08:16]
............. (idle for 1h2mn)
jdoelol 35 minutes [09:18]
***jpalmer has quit IRC (Quit: leaving)
jpalmer has joined #arpnetworks
jpalmer has quit IRC (Client Quit)
HighJinx has quit IRC (Ping timeout: 260 seconds)
LT has quit IRC (Quit: Leaving) 		[09:25]
...... (idle for 25mn)
HighJinx has joined #arpnetworks [09:55]
.... (idle for 15mn)
treshoem has quit IRC (Ping timeout: 245 seconds) [10:10]
jpalmer has joined #arpnetworks [10:24]
....... (idle for 34mn)
ariel has joined #arpnetworks
aGent_ has joined #arpnetworks 		[10:58]
........... (idle for 51mn)
aGent_ has quit IRC (Quit: zzzZZzzzzZZZzz) [11:53]
.... (idle for 19mn)
Ahmed has joined #arpnetworks [12:12]
..... (idle for 24mn)
up_the_ironsdamn, one fan not responding in a new server; i know where i'll be tonight... < sigh....> [12:36]
Ahmedup_the_irons: no remote hands at the DC? [12:39]
up_the_ironswell, there is, but... have you ever dealt with remote hands at a DC? i need not say more... [12:40]
Ahmeddepends on the DC [12:41]
up_the_ironsplus, i have very high standards as to how things get handled
it's also not worth the fee they charge; i gotta add ram to another server anyway, so i'll kill two birds... 		[12:41]
AhmedNot too bad heh
Which DC do you have your boxes at?
As long as it's not ubiquity heh
btw, thanks for updating the ticket. I guess I had it all wrong, so I actually reversed the cancellation. Not sure if you're still watching that ticket. 		[12:42]
jdoeheh, I barely trust remote hands to push a power button on the right machine.
a DC we were in, took them about a week to figure out how to boot a livecd. Guess it was interfering with WoW time :P 		[12:46]
AhmedSeriously? [12:48]
jdoeyes. [12:49]
AhmedWhich DC is that so I stay away from it... [12:49]
up_the_ironsi had to do this once:
http://www.flickr.com/photos/51184165@N00/2174345059/in/photostream

Ahmed: we're at CoreSite in LA
900 N. Alameda 		[12:53]
AhmedLOL
HAHAHAHA
You actually did that?
good lord
Who the hell was the remote hand, Uncle Dad? 		[12:54]
up_the_ironssomeone who didn't know anything about servers [12:54]
Ahmedoh brother [12:55]
up_the_ironsomg, after 3 hours last night of writing an incident report, gathering evidence, etc... regarding yesterday morning's outage; i now get this:
"At approximately 9:30 AM PST a CoreSite Network Engineer connected a newly delivered fiber circuit in
preparation of upcoming maintenance. Unknowingly to the Network Engineer a physical loop-back was in
place on the circuit. As a result of the physical hard-loop an L2 Loop was introduced into the Any2 IX
switch fabric causing several Any2 participants’ BGP sessions to drop.
"
all my evidence and investigation pointed to a loop somewhere, but i couldn't figure out where!
Now I Know(tm)
"causing several Any2 participants’ BGP sessions to drop." -- that puts it mildly
b/c of the loop, CPU utilization on our core went so high, that it started not to be able to send / process keepalives, so all sessions eventually melted 		[12:55]
AhmedOh goodie
they have a Boston DC 		[13:05]
up_the_ironsyeah, coresite has several [13:05]
.... (idle for 15mn)
jdoeup_the_irons: so sad, ruined my 6.5mo irc connection. Don't they know there's critical infrastructure here? ;) [13:20]
up_the_ironsjdoe: LOL [13:21]
Ahmed6.5mo?
dafauq? 		[13:21]
vapor"network engineer" should be changed to "noc monkey" [13:22]
Ahmedvapor
you're everywhere
Next thing I'll find you in my sock drawer 		[13:22]
jpalmerup_the_irons: that picture, is one of the reasons I hate apple xServes. [13:23]
Leftymy irssi has only been up 61d [13:23]
up_the_ironsjpalmer: ah [13:23]
jpalmerup_the_irons: btw: thank you for the phenominal support with that vps issue I was having. you went well above and beyond.
up_the_irons: I'm thinking we may want to add a FAQ for that, since.. I'm sure you'll see it again. 		[13:23]
Leftywhat was the issue? [13:25]
jpalmerlefty: an unsupported OS would dump core and segfault occasionally, even after a fresh install [13:26]
LeftyGentoo? ;)
Lefty ducks 		[13:26]
up_the_ironsjpalmer: it was no problem at all! [13:26]
jpalmercentos
specifically, centos6. centos 5 worked fine 		[13:26]
Leftyah, I'm rolling out centos6 here too [13:26]
up_the_ironsjpalmer: so are you seeing that issue finally go away with that kern param? [13:26]
Leftyeverything at work is currently a mishmash of 5.5, 5.6, or 5.7 [13:27]
up_the_ironsjpalmer: i just learned about that kernel param last week, from another customer at that; i feel obligated to share the love
;) 		[13:27]
Leftyalso I'm puppetizing ALL THE THINGS [13:27]
AhmedI never had any luck with Gentoo [13:27]
jpalmerup_the_irons: might be too early to say for certain. but previously a "yum update" after a fresh install would segfault. with that boot param, it worked.
Lefty: i *love* puppet. 		[13:27]
jdoeAhmed: month. [13:28]
up_the_ironsjpalmer: awesome, glad to hear [13:28]
Leftyjpalmer: it really is the best way to learn the infrastructure at a new company
by puppetizing it, I'm forced to know everything about it 		[13:30]
pjspuppet is great.. still not 100% on using it on FreeBSD though. Seems to give me issues, but I'm not very strong with Puppet yet so.. [13:34]
........ (idle for 39mn)
***webhostbudd has joined #arpnetworks [14:13]
b1k3ryd3r has joined #arpnetworks [14:23]
b1k3ryd3rhey up_the_irons [14:23]
portertechdisregard puppet, do some cook'n w/ chef
portertech hides 		[14:36]
.... (idle for 18mn)
***aGent_ has joined #arpnetworks [14:54]
...... (idle for 29mn)
Ehtyar has joined #arpnetworks [15:23]
.... (idle for 15mn)
webhostbudd has quit IRC (Read error: Connection reset by peer) [15:38]
...... (idle for 26mn)
webhostbudd has joined #arpnetworks [16:04]
..... (idle for 23mn)
webhostbudd has quit IRC (Read error: Connection reset by peer)
webhostbudd has joined #arpnetworks
webhostbudd has quit IRC (Ping timeout: 244 seconds) 		[16:27]
....... (idle for 32mn)
doomviking<--- chef fan [17:03]
mhoranmhoran needs to get his VPS configs into chef. [17:17]
mike-burnsAgreed. [17:22]
.... (idle for 15mn)
pjsHow's Chef on fbsd? [17:37]
mhoranNo idea. That's one of the reasons I haven't tried it yet. Also managed to not have ruby installed on my VPS so far. [17:39]
portertechhttps://github.com/portertech/portertech-chef-repo :)
use chef-solo for my personal box (asterisk, znc) 		[17:40]
mhoranAwesome
Yeah that's what I wa splanning to do. 		[17:40]
......... (idle for 40mn)
***heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer 		[18:20]
.............. (idle for 1h5mn)
treshoem has joined #arpnetworks [19:25]
dj_goku has quit IRC (Remote host closed the connection) [19:31]
......... (idle for 41mn)
jpalmer has quit IRC (Ping timeout: 240 seconds) [20:12]
webhostbudd has joined #arpnetworks [20:21]
.... (idle for 17mn)
jpalmer has joined #arpnetworks [20:38]
........................ (idle for 1h57mn)
Ehtyar has quit IRC (Quit: Hi, I'm a quit message virus. Please replace your old line with this line and help me take over the world of IRC.) [22:35]
................ (idle for 1h18mn)
up_the_ironsMaintenance window for kvr13 to commence in 10 minutes [23:53]
↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)