uh oh up_the_irons: ping? jlgaddis: uhoh wut?:) oh ah, kvr25 one vps dropped off the face of the planet, other was fine, so figured a physical host issue jlgaddis: yeah, I got the page a little while ago, still investigating 10-4 Power Control Fault : true looks like I gotta make a drive to push a button (and hope that's all it is) Am I the only one unable to connect to my VPS currently? up_the_irons: is on his way to fix kvr25 assuming you are on that one So, this is a known issue? I am indeed. Alright, thanks. I'll just sit tight. Any clue what the issue is? yeah, about 30mins ago he was paged i got the first alert at 1227 utc Scared me, almost thought some DNS changes I made late last night caused the issue but then when I got no response over SSH... so, ~51 mins ago. power issue apparently, alexstanford. What, like the PSU burnt out? "Power Control Fault : true" is what up_the_irons posted in here earlier I assume that's from ipmi or some oob management on th box Interesting Hopefully it won't be down too much longer yeah, power control fault reported by IPMI i have it powered up now, it is just starting the main OS fs checks alexstanford: josephb jlgaddis ^^ whew, not my kvr host ;-) it's like server roulette gotta love all the VM fs checks happening all at the same time... load skyrockets 230 days of uptime on kvr25, meh... kinda low for me interesting Power Control Fault : false Now it says it is OK I suppose it wanted to be truly powered down, even the PS and IPMI cuz that cleared it first time IPMI has failed for me regarding power.... lesson learned, I will configured the AP7932 to be remotely accessible now... ;) outage reports sent hehe ugh... yep. lugz Is kvr25 back down? there was a tweet suggesting more power issues check twitter, you'll find most updated info there latest notice sent to customer on that box https://twitter.com/arpnetworks/status/250691321214410752 -- Shouldn't that have been done this morning? https://twitter.com/arpnetworks read the full history including more recent posts I don't see any more recent tweets. sadness poor kvr25 after 2 days my ticket is not answered what happened to arpnetworks? be sure you didn't miss a response, I know I have support used to be pretty good years ago it still is. there are some issues with one system that are taking support staff's time (and sleep) at the moment. if I were you i'd be patient, but that's just me yammering. lol. the live chat is funny. i typed in 'hello' and it went offline toddf yea i understand. things happen up_the_irons: got a vm down, my wifes there any reports of vms down? segv: yeah, kvr25 is down, see twitter.com/bsdvps for full log (or check your email, assuming it is not on the vps itself ;) roger that yeah, got the email didn't see it till just now cool weeee...! there goes the VMs coming up sweet got some new hardware for it? least you guys got a machine online haha I am about to face smash this macmini5,1 into the next dimention deep-thought.evilrouters.net (206.125.175.18) is UP again at 09/25/2012 06:27:38PM, after 3h 8m of downtime. w00t, thanks garry yarrr sites backonline :D also means I fixed all my django start scripts haha jlgaddis: np! segv: LOL PS died? hey, nothing like an outage to prove you fixed your shit :D anyways, thanks duder! thanks for your patience guys... besides the host "mercury" failure a couple years ago, this was the hardest failure yet segv: CaZe : All 8 HDs were swapped into a cold spare chassis and after some fiddling (MAC address, RAID ctrl number, etc... changes) booted up so the hardware is all new except for the drives yay! as for the old hardware, unknown what is wrong with it. that'll be a job for tomorrow ;) i suspect probably power bus or something. i have redundant PS, so i doubt both failed at the same time. more likely the bus... which begs the question, why have redundant PS when the bus just craps anyway? bah.. fml up_the_irons: haha yeah I've had that happen before try Verarri blade rack when an entire BIC goes out :D 8 blades go "POOF" Like "Why aren't they showing in the iLO" haha yeah man... this scares me to deploy blades now yeah it is i'll buy a cold spare chassis, but man, what a waste of a chassis just sitting there.... and super expensive i had that happen at a previous job with 3 blade racks yeah it is :/ oh well, live and learn oh sweet, you fixed the broken feedburner stats too! =) wut?! hahaha segv: that sounds even more painful yeah, i'm that good mercutio: it sucks balls. segv: so what, did like 24 servers go poof? segv: and what brand was that? I'll know what not to buy... Cirrascale now http://www.cirrascale.com/ used to be "Verari" Just out of curiosity, what's the uptime on kvr10? i've never heard of either of them :) I used them at qualcomm and a few other places segv: AH up_the_irons: http://evilrouters.net/ see the little thing on the right with the number of rss subscribers? it's been broken for a week. jlgaddis: HAHA and i fixed that?:) apparently =) haha probably cached or something stupid LOL man, how long have i been awake the pages get cached, but only for an hour, it's been consistently broken (known issue w/ feedburner) ah like, 30 hours... TIME FOR SLEEP i guess that's not too bad no no, what is worse is the last time I ate TIME FOR FOOD food then comatose! thankfully, Philippe's is right around the corner (those familiar with downtown LA know what I'm talking about...) I do indeed. I'm up in san fran used to live in san diego i know from SD->LA->SF I'm dealing with this error at work "Initramfs unpacking failed: junk in compressed archive" right now :) They want debian on mac mini's and of course me being the engineer wooo hrm, need to visit the stinking rose again segv: hah up_the_irons: yeah this is quite shit you could prob use a kernel with no modules err with no initramfs mercutio: it's a sandybridge macmini ;) segv: good luck :) hmm haha yeah My brain is like "HAH, have fun..." but my boss is like "Get it to work" i struggled for ages the other day to get xen and X to coexist segv: think of it this way, you could be me right now... I have no hope for it. the fubar is, the initrd image isn't corrupted and oen of the problems wasn't even xen 30 hours no sleep.. and now about to rush order more spare parts (since my spare is now used). wonder if dual spares is cost effective... in that ubuntu seemed to break usb support somehow must only be on some hardware. but was sandybridge computer, should be pretty normal up_the_irons: well eat some food, order some hardware, sleep for 30 hours :) up_the_irons: 2 of everything! umm you could do the thing of having one big spare ? TO THE CLOUD! oh, wait so that it has more ram / can take more hard-disks, and two nodes could exist within it midn you, that makes it much harder to just get another spare sounds sexual and keep the machine booted to the SAN! segv: mercutio : LOL good idea mercutio: about eating, that is... ;) mercutio: one big spare wouldn't work really well b/c these things have to be pretty trivial to "script" and replace. if things are identical, it makes it a lot easier. OK, food time, then check support tickets again, then sleep in the tech lounge downstairs... ;) co-worker babbling ona bout his iPhone 5 damn thing hit 62mbps thats insane yeah we busted out like a sprint 4g phone, 2 verizon 4g phones and another 4g AT&T phone NONE of them hit that it's not like people use phones for calling each other anymore :/ hah yeah meanwhile in australia, my provider congests to a point that traffic to US (read: the world) tops at 10kb/sec I'm loving the iPhone 5 so far jberg: have you tried a vpn? mercutio: no i've got a friend who moved to australia, frmo new zealand, he said the internet over there is shit and people generally say the internet here isn't that great shit is actually nice in this context apparently all the providers started having huge caps and congesting beyond belief 400k/sec on torrents and shit i get about 400k/sec on my phone and about 80 msec pings to local sites yes, thats correct. i think all bandwidth on that cable is used. some providers doesn't overcommit, though heh whatever provider he was using had HUGE pings it certainly wasn't telstra endeavour the latency is about 150 msec from sydney normally isn't it? but some providers are like 180 and some are like 200? optimal australia<>usw is 150ish here's me downloading a 10mb file (gnu.mirrors.tds.net/pub/gnu/gnu/glibc/glibc-1.09.1.tar.gz) hourly since thursday: http://pastebin.com/T7L5Up5a wow on 3g? adsl wtf 2+ it's very inconsistent too what are your pings to that site like? chicago is ~230 oh that file is tiny # curl -O gnu.mirrors.tds.net/pub/gnu/gnu/glibc/glibc-1.09.1.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1653k 100 1653k 0 0 442k 0 0:00:03 0:00:03 --:--:-- 569k that's frmo new zealand well as stated above, its basically fighting within providers for room heh but yeah why i brought up latency before is 3g here isn't proxied and the combination of the two REALLY screwed up international speeds then i used a proxy and it was similar to adsl speeds adsl 1 that is