up_the_irons - having problems with my box # 3 for neil it comes up, then goes off the net kvr13 and I can't get to the console from there either anyone else on kvr13 with problems? I presume the 'vnc server' is also the host system name the vps is run off of? ok, I'm sitting on the console now, after rebooting the vm I'm gonna watch to see what goes wrong vnc logged in as root on console zfs: malloc failed! looping with date; sleep 1 whoa. it just went bonkers it's still running internally, but all network activity stop ping6 -n -w ff02::1%em0 ? 40 bytes from fe80::5054:ff:fe27:2122%em0: 0.v.freedaemon.com. 44 bytes from fe80::5054:ff:fe27:9007%em0: s3.lax.arpnetworks.com. thats what I get on mine, just to see if `anything else' is on the same net, though I'm sure pinging the default gateway (v6/v4) would tell you that as well even the vnc goes away at the same time this isn't good this is very bad this is the kind of thing for which I need urgent support. :) ironic, since we talked about that yesterday. vnc going away means the kvm process itself dies NO. already confirmed the box is still *running* it's just network stack my cron jobs continue to fire off I suspect serial console might shed some light if you can force it to spew console messages and the like to serial console since serial console can't `go away' like vnc doesn't the serial console still use the network stack to talk to me? if vnc goes away and your vps looses net .. almost sounds like the host nic has connectivity issues .. Yeah... that's why I think this is an up_the_irons issue, not mine I could see if I have problems.. I'm on kvr13 as well well, true, serial console uses a tcp connection from a local vps to your kvm instance's serial port on a tcp port however, if it did disconnect, at least you'd see the last message as opposed to vnc going 'blip' and disappearing again, kvr13 .. is that your vnc server hostname? if so, I'm on `mercury' thus can't help ;-( yes it's not like there's a lot of web activity or anything either ugh See my latest tweet on @bsdvps bsdvps: Host "kvr13" experienced kernel memory corruption earlier this morning. We performed emergency maintenance: kernel upgrade and reboot aha. that's me and that's why I'm having troubles is it all good now? neil's website was down for two hours he hates that There were a bunch of conntrack over limit errors on the serial port, yet I saw almost no traffic go to the box. Could not get anything on the monitor. Box still appeared alive b/c it was logging to serial. After working with it for about 10 minutes, I deemed the only thing left to do was a reboot already ragging on me to "move chris somewhere else" bah, I even have a window open from 'microblog-purple' in pidgin that said so at 13 after, 10 mins later, I still hadn't noticed. bah! is there anything I can tell neil about how rare this is? I really don't want to spend a lot more time planning for something that has enough 9's normally yeah, i cant connect on kvr13 either RandalSchwartz: this is the first time i've seen something like this in 18 months (basically, since i started with this product line) ok - I passed that along to him I don't know if he knows how much more expensive 5 9's is over 4 9's :) randal: heh, like load balanced vps's on separate kvm hosts? ;-) what we'll probably do is set up some sort of alternate hosting for the brochure web site and loadbal it, yeah mattx86: your VPS should be coming up now up_the_irons: yep, there it is if kvr13 keeps having problems, can you migrate us fairly rapidly? or do you have to copy disk around to do that? not sure how your disk-to-cpu mapping works would it be possible that the bug I discovered in mikrotik's routeros product would have affected the host in any way? RandalSchwartz: i have to copy the disk to another host; doesn't take all that long disks are local to each kvm host mattx86: i would think not, but ya never know yeah, it's only about 30G for that box my settings wouldn't change though, right? except for my virtual console access true Neil's saying "oof" about the two hour downtime RandalSchwartz: vnc host would change, that is about it well - please help me out by keeping on top of this, and feel free to migrate me if needed, even if it means a few minutes downtime gotta keep neil happy freebsd responds nicely to graceful down. takes about 30 seconds RandalSchwartz: definitely going to stay on top of this I need to figure out why nagios didn't tell me either I must have something configured wrong it notified me about ssh down and up, but not ping or website and ssh is apparently "under" ping, so if ping is down, that is supposed to control all other notifications but I still should have got the ping-down notice if you get a HOST DOWN message, no services on that host are checked further it was raining outside... a little cold.... oh bah. someone thought it clever that freebsd box admins should only be notified during normal working hours in the default nagios config everything else was 24x7. Dumb. I erased that puppy right away, reloaded nagios morning doods <3 nagios but that explains why I didn't get my down notice for the 6am fault yeah, i wish it would queue notifications or something, so if youre set to 9-5 and a fault is at 6am, it will queue it and notify you at 9am about it maybe there is a way to do that, but i havent found it if it's still a problem at 9am, don't you get the notice anyway? you'd get teh next 'reminder" notification, yes. but, it may not be immediately at 9. for instance, you have 9-5 notifications. you have a failure at 8:55, and a 2 hour reminder.. you may not get the reminder until 10:55ish yea itd be nice if the 6am event would go out immediately when the notification timeperiod starts for that reason i just had everything 24x7 but page/sms only went out 9-5 but email worked 24x7 so id at least see it immediately when i woke up IPv6Freely: thats generally what I do, as well. this job is a fucking waste of time i show up to install a new juniper network... and they havent even decided if they are ACTUALLY going juniper, going cisco, or going some other noname garbage ive never heard of... not to mention the lack of an MPLS circuit being ordered yet so im sitting here... wow that sucks yo yo back cool IPv6Freely: so install OpenBSD on a pc of theirs and setup MPLS for them while you wait.. toddf: hahahahahhahaa oh man thatd be funny LOL the cisco quote came back 303k for one router. that's rich :> yikes, I'm hoping they are doing some SERIOUS traffic. very little actually... we're just tryng to quote hardware comparable to the juniper equivalent 303k for one cisco router. They need four of them. 103k for four juniper SRX650s, and four EX4200s. ...yet theyre leaning cisco wow, and I thought comparing openbsd to an $18k ASA for IPSec was a rather large price difference, maybe I should learn more about MPLS very little traffic? I have a 2611, and a 2621xm I'll happily ship you (overnight, morning delivery) for $250k. look, a discount! I'll even buy you a smartnet contract. heh hell, one single 10x1gigE module for the Cisco ASR1000 is $17,700... and this router needs 5 of those interesting. my VPS rebooted a couple hours ago. jpalmer - you on kvr13? RandalSchwartz: yes ... http://twitter.com/#!/bsdvps/status/25988394637 memory corruption I was wondering why my mysql DB had corrupted itself. now I know ;) the WTF there is "mysql - why?" heh something legacy that demands mysql instead of a sane database? yes i should probably follow that account sad to hear that I'd much rather be using postrge. but, eh. it is what it is. postrge? :) maybe you mean postgres ugg, I hate when i see others type that. why the hell did I just do it :P postgres aka postgresql and never postgre and sometimes pg :) and definately not postrge either or mysql :) pg is acceptable. postgresql is acceptable. the rest, not acceptable. postgres is also typical just not postgre so is "u" but thats not acceptible either. ;) and it's "post gres cue ell" not "post grey see quel" - which grates my ears if Monty gets it right with "my ess cue ell", the pg people can learn to say it right too I remember when they first put up the mp3/wave of the pronunciation. heh it obviously bothered a few people :P people who say "my see quel" should be shot it's... like typing Perl in "CAPS" :) PERL! a clue of the clueless :) or as that comedian would say, "here's your sign!" I was joking, but have no problems admitting.. I'm clueless with perl. it makes my eyes bleed. ... http://en.wikipedia.org/wiki/Bill_Engvall its close to C but without the buffer overflows (just leaks memory if you don't watch it..) i love making words out of acronyms. MPLS is MIPPLES heh :P if I said tha tin the office, I'd get some looks. haha The best one is "e-grep" for EIGRP where the hell do you get "grep" from that? I also love ackels for ACLs by love, i mean it makes me want to punch babies :) I wonder where up_the_irons have been. started a support ticket a couple of days ago and he answered once :P interesting, https://bugzilla.redhat.com/show_bug.cgi?id=508801 suggests mpbios issues in OpenBSD vs kvm can be resolved by the bios being updated inside qemu-kvm .. heh! up_the_irons! fwiw, they're sold out again with new hardware on the way that'll keep up_the_irons busy, or to a certain extent anyways dxtr: i've been resting a lot this week b/c I got sick, so support is a little slow dxtr: oh and i see you wanted the upgrade; yeah, those take a little time. i've mainly been sticking with the quick requests like rdns, until i feel better up_the_irons: I know someone asked the other day. in our VPS's, we have 2 cdroms. can we request 2 seperate iso's be permanently mounted, or is thee second one unused? up_the_irons: ah, that makes me more understanding. I'm sorry about giving you a hard time earlier. hope you get to feeling better mattx86: :) jpalmer: the 2nd cd-rom is a kvm bug i think. there's only 1 in the VM config. Technically, I *could* configure two if you wanted *off up_the_irons: gotcha. I don't personally need it, but I've seen it asked before. good to have an answer up_the_irons: thanks for the answer. get some chicken soup and rest.