i just installed OpenNTP on our cacti server, fixing the time lag (which was about 35 minutes), so your graphs will have a gap in them
CaZe: that is awesome :D
lol 35 minutes
damn, one fan not responding in a new server; i know where i'll be tonight... < sigh....>
up_the_irons: no remote hands at the DC?
well, there is, but...  have you ever dealt with remote hands at a DC?  i need not say more...
depends on the DC
plus, i have very high standards as to how things get handled
it's also not worth the fee they charge; i gotta add ram to another server anyway, so i'll kill two birds...
Not too bad heh
Which DC do you have your boxes at?
As long as it's not ubiquity heh
btw, thanks for updating the ticket. I guess I had it all wrong, so I actually reversed the cancellation. Not sure if you're still watching that ticket.
heh, I barely trust remote hands to push a power button on the right machine.
a DC we were in, took them about a week to figure out how to boot a livecd. Guess it was interfering with WoW time :P
Seriously?
yes.
Which DC is that so I stay away from it...
i had to do this once:
http://www.flickr.com/photos/51184165@N00/2174345059/in/photostream
Ahmed: we're at CoreSite in LA
900 N. Alameda
LOL
HAHAHAHA
You actually did that?
good lord
Who the hell was the remote hand, Uncle Dad?
someone who didn't know anything about servers
oh brother
omg, after 3 hours last night of writing an incident report, gathering evidence, etc...  regarding yesterday morning's outage; i now get this:
"At approximately 9:30 AM PST a CoreSite Network Engineer connected a newly delivered fiber circuit in
preparation of upcoming maintenance. Unknowingly to the Network Engineer a physical loop-back was in
place on the circuit. As a result of the physical hard-loop an L2 Loop was introduced into the Any2 IX
switch fabric causing several Any2 participants’ BGP sessions to drop.
"
all my evidence and investigation pointed to a loop somewhere, but i couldn't figure out where!
Now I Know(tm)
"causing several Any2 participants’ BGP sessions to drop." -- that puts it mildly
b/c of the loop, CPU utilization on our core went so high, that it started not to be able to send / process keepalives, so all sessions eventually melted
Oh goodie
they have a Boston DC
yeah, coresite has several
up_the_irons: so sad, ruined my 6.5mo irc connection. Don't they know there's critical infrastructure here? ;)
jdoe: LOL
6.5mo?
dafauq?
"network engineer" should be changed to "noc monkey"
vapor
you're everywhere
Next thing I'll find you in my sock drawer
up_the_irons: that picture, is one of the reasons I hate apple xServes.
my irssi has only been up 61d
jpalmer: ah
up_the_irons: btw:  thank you for the phenominal support with that vps issue I was having.  you went well above and beyond.
up_the_irons: I'm thinking we may want to add a FAQ for that,  since..  I'm sure you'll see it again.
what was the issue?
lefty:  an unsupported OS would dump core and segfault occasionally, even after a fresh install
Gentoo? ;)
jpalmer: it was no problem at all!
centos
specifically,  centos6.   centos 5 worked fine
ah, I'm rolling out centos6 here too
jpalmer: so are you seeing that issue finally go away with that kern param?
everything at work is currently a mishmash of 5.5, 5.6, or 5.7
jpalmer: i just learned about that kernel param last week, from another customer at that; i feel obligated to share the love
;)
also I'm puppetizing ALL THE THINGS
I never had any luck with Gentoo
up_the_irons: might be too early to say for certain.  but previously  a "yum update" after a fresh install would segfault.  with that boot param,  it worked.
Lefty: i *love* puppet.
Ahmed: month.
jpalmer: awesome, glad to hear
jpalmer: it really is the best way to learn the infrastructure at a new company
by puppetizing it, I'm forced to know everything about it
puppet is great.. still not 100% on using it on FreeBSD though. Seems to give me issues, but I'm not very strong with Puppet yet so..
hey up_the_irons
disregard puppet, do some cook'n w/ chef
<--- chef fan
Agreed.
How's Chef on fbsd?
No idea. That's one of the reasons I haven't tried it yet. Also managed to not have ruby installed on my VPS so far.
https://github.com/portertech/portertech-chef-repo :)
use chef-solo for my personal box (asterisk, znc)
Awesome
Yeah that's what I wa splanning to do.
Maintenance window for kvr13 to commence in 10 minutes