***: ziyourenxiang has joined #arpnetworks
ziyourenxiang has quit IRC (Quit: ziyourenxiang)
RandalSchwartz has quit IRC (Ping timeout: 272 seconds)
heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer
RandalSchwartz has joined #arpnetworks
RandalSchwartz has quit IRC (Changing host)
RandalSchwartz has joined #arpnetworks RandalSchwartz: up_the_irons - having problems with my box # 3 for neil
it comes up, then goes off the net
kvr13
and I can't get to the console from there either
anyone else on kvr13 with problems? toddf: I presume the 'vnc server' is also the host system name the vps is run off of? RandalSchwartz: ok, I'm sitting on the console now, after rebooting the vm
I'm gonna watch to see what goes wrong
vnc logged in as root on console toddf: zfs: malloc failed! RandalSchwartz: looping with date; sleep 1
whoa. it just went bonkers
it's still running internally, but all network activity stop toddf: ping6 -n -w ff02::1%em0 ?
40 bytes from fe80::5054:ff:fe27:2122%em0: 0.v.freedaemon.com.
44 bytes from fe80::5054:ff:fe27:9007%em0: s3.lax.arpnetworks.com.
thats what I get on mine, just to see if `anything else' is on the same net, though I'm sure pinging the default gateway (v6/v4) would tell you that as well RandalSchwartz: even the vnc goes away at the same time
this isn't good
this is very bad
this is the kind of thing for which I need urgent support. :)
ironic, since we talked about that yesterday. toddf: vnc going away means the kvm process itself dies RandalSchwartz: NO. already confirmed
the box is still *running*
it's just network stack
my cron jobs continue to fire off toddf: I suspect serial console might shed some light if you can force it to spew console messages and the like to serial console
since serial console can't `go away' like vnc RandalSchwartz: doesn't the serial console still use the network stack to talk to me? toddf: if vnc goes away and your vps looses net .. almost sounds like the host nic has connectivity issues .. RandalSchwartz: Yeah... that's why I think this is an up_the_irons issue, not mine mattx86: I could see if I have problems.. I'm on kvr13 as well toddf: well, true, serial console uses a tcp connection from a local vps to your kvm instance's serial port on a tcp port
however, if it did disconnect, at least you'd see the last message as opposed to vnc going 'blip' and disappearing
again, kvr13 .. is that your vnc server hostname? if so, I'm on `mercury' thus can't help ;-( RandalSchwartz: yes
it's not like there's a lot of web activity or anything either up_the_irons: ugh
See my latest tweet on @bsdvps ***: Hien has joined #arpnetworks toddf: bsdvps: Host "kvr13" experienced kernel memory corruption earlier this morning. We performed emergency maintenance: kernel upgrade and reboot RandalSchwartz: aha.
that's me
and that's why I'm having troubles
is it all good now?
neil's website was down for two hours
he hates that up_the_irons: There were a bunch of conntrack over limit errors on the serial port, yet I saw almost no traffic go to the box. Could not get anything on the monitor. Box still appeared alive b/c it was logging to serial. After working with it for about 10 minutes, I deemed the only thing left to do was a reboot RandalSchwartz: already ragging on me to "move chris somewhere else" toddf: bah, I even have a window open from 'microblog-purple' in pidgin that said so at 13 after, 10 mins later, I still hadn't noticed. bah! RandalSchwartz: is there anything I can tell neil about how rare this is?
I really don't want to spend a lot more time planning for something that has enough 9's normally mattx86: yeah, i cant connect on kvr13 either up_the_irons: RandalSchwartz: this is the first time i've seen something like this in 18 months (basically, since i started with this product line) RandalSchwartz: ok - I passed that along to him
I don't know if he knows how much more expensive 5 9's is over 4 9's :) toddf: randal: heh, like load balanced vps's on separate kvm hosts? ;-) RandalSchwartz: what we'll probably do is set up some sort of alternate hosting for the brochure web site and loadbal it, yeah up_the_irons: mattx86: your VPS should be coming up now mattx86: up_the_irons: yep, there it is RandalSchwartz: if kvr13 keeps having problems, can you migrate us fairly rapidly?
or do you have to copy disk around to do that?
not sure how your disk-to-cpu mapping works mattx86: would it be possible that the bug I discovered in mikrotik's routeros product would have affected the host in any way? up_the_irons: RandalSchwartz: i have to copy the disk to another host; doesn't take all that long toddf: disks are local to each kvm host up_the_irons: mattx86: i would think not, but ya never know RandalSchwartz: yeah, it's only about 30G for that box
my settings wouldn't change though, right?
except for my virtual console access mattx86: true RandalSchwartz: Neil's saying "oof" about the two hour downtime up_the_irons: RandalSchwartz: vnc host would change, that is about it RandalSchwartz: well - please help me out by keeping on top of this, and feel free to migrate me if needed, even if it means a few minutes downtime
gotta keep neil happy
freebsd responds nicely to graceful down. takes about 30 seconds up_the_irons: RandalSchwartz: definitely going to stay on top of this RandalSchwartz: I need to figure out why nagios didn't tell me either
I must have something configured wrong
it notified me about ssh down and up, but not ping or website
and ssh is apparently "under" ping, so if ping is down, that is supposed to control all other notifications
but I still should have got the ping-down notice up_the_irons: if you get a HOST DOWN message, no services on that host are checked further -: up_the_irons wanders off ***: LT has quit IRC (Quit: Leaving)
cedwards has quit IRC (Changing host)
cedwards has joined #arpnetworks Hien: it was raining outside...
a little cold.... ***: Hien has quit IRC (Quit: Page closed) RandalSchwartz: oh bah. someone thought it clever that freebsd box admins should only be notified during normal working hours in the default nagios config
everything else was 24x7. Dumb.
I erased that puppy right away, reloaded nagios IPv6Freely: morning doods
<3 nagios RandalSchwartz: but that explains why I didn't get my down notice for the 6am fault IPv6Freely: yeah, i wish it would queue notifications or something, so if youre set to 9-5 and a fault is at 6am, it will queue it and notify you at 9am about it
maybe there is a way to do that, but i havent found it RandalSchwartz: if it's still a problem at 9am, don't you get the notice anyway? jpalmer: you'd get teh next 'reminder" notification, yes.
but, it may not be immediately at 9.
for instance, you have 9-5 notifications. you have a failure at 8:55, and a 2 hour reminder.. you may not get the reminder until 10:55ish IPv6Freely: yea
itd be nice if the 6am event would go out immediately when the notification timeperiod starts
for that reason i just had everything 24x7
but page/sms only went out 9-5
but email worked 24x7
so id at least see it immediately when i woke up jpalmer: IPv6Freely: thats generally what I do, as well. IPv6Freely: this job is a fucking waste of time
i show up to install a new juniper network... and they havent even decided if they are ACTUALLY going juniper, going cisco, or going some other noname garbage ive never heard of... not to mention the lack of an MPLS circuit being ordered yet
so im sitting here... mattx86: wow
that sucks ***: hsien has joined #arpnetworks hsien: yo jpalmer: yo back hsien: cool toddf: IPv6Freely: so install OpenBSD on a pc of theirs and setup MPLS for them while you wait.. IPv6Freely: toddf: hahahahahhahaa
oh man thatd be funny -: toddf grins IPv6Freely: LOL the cisco quote came back
303k for one router. mattx86: that's rich :> jpalmer: yikes, I'm hoping they are doing some SERIOUS traffic. IPv6Freely: very little actually... we're just tryng to quote hardware comparable to the juniper equivalent
303k for one cisco router. They need four of them.
103k for four juniper SRX650s, and four EX4200s.
...yet theyre leaning cisco -: IPv6Freely facepalm toddf: wow, and I thought comparing openbsd to an $18k ASA for IPSec was a rather large price difference, maybe I should learn more about MPLS jpalmer: very little traffic? I have a 2611, and a 2621xm I'll happily ship you (overnight, morning delivery) for $250k. look, a discount! I'll even buy you a smartnet contract. IPv6Freely: heh
hell, one single 10x1gigE module for the Cisco ASR1000 is $17,700... and this router needs 5 of those jpalmer: interesting. my VPS rebooted a couple hours ago. RandalSchwartz: jpalmer - you on kvr13? jpalmer: RandalSchwartz: yes RandalSchwartz: ... http://twitter.com/#!/bsdvps/status/25988394637
memory corruption jpalmer: I was wondering why my mysql DB had corrupted itself. now I know ;) RandalSchwartz: the WTF there is "mysql - why?" IPv6Freely: heh RandalSchwartz: something legacy that demands mysql instead of a sane database? jpalmer: yes IPv6Freely: i should probably follow that account RandalSchwartz: sad to hear that jpalmer: I'd much rather be using postrge. but, eh. it is what it is. RandalSchwartz: postrge? :) ***: schmir has joined #arpnetworks RandalSchwartz: maybe you mean postgres jpalmer: ugg, I hate when i see others type that. why the hell did I just do it :P RandalSchwartz: postgres aka postgresql and never postgre
and sometimes pg :) toddf: and definately not postrge either RandalSchwartz: or mysql :) jpalmer: pg is acceptable. postgresql is acceptable. the rest, not acceptable. RandalSchwartz: postgres is also typical
just not postgre jpalmer: so is "u" but thats not acceptible either. ;) RandalSchwartz: and it's "post gres cue ell"
not "post grey see quel" - which grates my ears
if Monty gets it right with "my ess cue ell", the pg people can learn to say it right too jpalmer: I remember when they first put up the mp3/wave of the pronunciation. heh it obviously bothered a few people :P RandalSchwartz: people who say "my see quel" should be shot
it's... like typing Perl in "CAPS" :) jpalmer: PERL! -: jpalmer stops RandalSchwartz: a clue of the clueless :)
or as that comedian would say, "here's your sign!" jpalmer: I was joking, but have no problems admitting.. I'm clueless with perl. it makes my eyes bleed. RandalSchwartz: ... http://en.wikipedia.org/wiki/Bill_Engvall toddf: its close to C but without the buffer overflows (just leaks memory if you don't watch it..) IPv6Freely: i love making words out of acronyms. MPLS is MIPPLES
heh
:P jpalmer: if I said tha tin the office, I'd get some looks. IPv6Freely: haha
The best one is "e-grep" for EIGRP
where the hell do you get "grep" from that?
I also love ackels for ACLs
by love, i mean it makes me want to punch babies -: RandalSchwartz wanders to a different location mattx86: :) dxtr: I wonder where up_the_irons have been. started a support ticket a couple of days ago and he answered once :P toddf: interesting, https://bugzilla.redhat.com/show_bug.cgi?id=508801 suggests mpbios issues in OpenBSD vs kvm can be resolved by the bios being updated inside qemu-kvm .. heh! ***: schmir has quit IRC (Remote host closed the connection) dxtr: up_the_irons! mattx86: fwiw, they're sold out again with new hardware on the way
that'll keep up_the_irons busy, or to a certain extent anyways ***: ficovh has joined #arpnetworks
ficovh has quit IRC (Client Quit) up_the_irons: dxtr: i've been resting a lot this week b/c I got sick, so support is a little slow
dxtr: oh and i see you wanted the upgrade; yeah, those take a little time. i've mainly been sticking with the quick requests like rdns, until i feel better jpalmer: up_the_irons: I know someone asked the other day. in our VPS's, we have 2 cdroms. can we request 2 seperate iso's be permanently mounted, or is thee second one unused? mattx86: up_the_irons: ah, that makes me more understanding. I'm sorry about giving you a hard time earlier.
hope you get to feeling better up_the_irons: mattx86: :)
jpalmer: the 2nd cd-rom is a kvm bug i think. there's only 1 in the VM config. Technically, I *could* configure two if you wanted -: up_the_irons wanders of up_the_irons: *off jpalmer: up_the_irons: gotcha. I don't personally need it, but I've seen it asked before. good to have an answer
up_the_irons: thanks for the answer. get some chicken soup and rest.