[00:26] *** Ehtyar has quit IRC (Quit: Never look down on someone unless you're helping them up.) [01:13] *** LT has joined #arpnetworks [02:01] *** mtve has quit IRC (*.net *.split) [02:01] *** henderb has quit IRC (*.net *.split) [02:01] *** Lefty has quit IRC (*.net *.split) [02:01] *** henderb has joined #arpnetworks [02:13] *** Webhostbudd_ has quit IRC (Quit: Leaving) [02:18] *** Lefty has joined #arpnetworks [02:38] *** HighJinx has quit IRC (Remote host closed the connection) [02:38] *** HighJinx has joined #arpnetworks [04:20] *** k3asd` has quit IRC (Ping timeout: 246 seconds) [04:20] *** k3asd` has joined #arpnetworks [04:23] *** k3asd` has quit IRC (Read error: Connection reset by peer) [04:23] *** k3asd` has joined #arpnetworks [04:47] *** k3asd` has quit IRC (Ping timeout: 248 seconds) [04:47] *** k3asd` has joined #arpnetworks [04:48] *** ameise has joined #arpnetworks [04:55] *** ant has quit IRC (Remote host closed the connection) [04:55] *** ameise is now known as ant [05:16] *** plett_ is now known as plett [05:28] *** segv has quit IRC (Ping timeout: 240 seconds) [05:31] uh oh [05:41] up_the_irons: ping? [05:41] jlgaddis: uhoh wut?:) [05:42] oh ah, kvr25 [05:42] one vps dropped off the face of the planet, other was fine, so figured a physical host issue [05:44] jlgaddis: yeah, I got the page a little while ago, still investigating [05:45] 10-4 [06:00] [06:00] Power Control Fault : true [06:00] looks like I gotta make a drive to push a button (and hope that's all it is) [06:11] *** ameise has joined #arpnetworks [06:13] *** alexstanford has joined #arpnetworks [06:15] Am I the only one unable to connect to my VPS currently? [06:15] up_the_irons: is on his way to fix kvr25 [06:16] assuming you are on that one [06:16] So, this is a known issue? [06:16] I am indeed. [06:16] Alright, thanks. I'll just sit tight. Any clue what the issue is? [06:17] yeah, about 30mins ago he was paged [06:18] i got the first alert at 1227 utc [06:18] Scared me, almost thought some DNS changes I made late last night caused the issue but then when I got no response over SSH... [06:18] so, ~51 mins ago. power issue apparently, alexstanford. [06:19] What, like the PSU burnt out? [06:20] "Power Control Fault : true" is what up_the_irons posted in here earlier [06:20] I assume that's from ipmi or some oob management on th box [06:20] Interesting [06:34] Hopefully it won't be down too much longer [06:35] *** ant has quit IRC (Quit: Leaving) [06:35] *** ameise is now known as ant [07:10] *** ix33 has quit IRC (Quit: WeeChat 0.3.9-rc2) [07:11] *** ix33 has joined #arpnetworks [07:46] yeah, power control fault reported by IPMI [07:46] i have it powered up now, it is just starting the main OS fs checks [07:46] alexstanford: josephb jlgaddis ^^ [07:52] whew, not my kvr host ;-) [07:52] it's like server roulette [07:53] gotta love all the VM fs checks happening all at the same time... load skyrockets [07:55] 230 days of uptime on kvr25, meh... kinda low for me [07:55] interesting [07:55] Power Control Fault : false [07:55] Now it says it is OK [07:56] I suppose it wanted to be truly powered down, even the PS and IPMI [07:56] cuz that cleared it [07:57] first time IPMI has failed for me regarding power.... lesson learned, I will configured the AP7932 to be remotely accessible now... ;) [08:10] outage reports sent [08:29] *** LT has quit IRC (Quit: Leaving) [08:31] hehe [09:14] *** dzup has quit IRC (Read error: Connection reset by peer) [10:28] *** HighJinx has quit IRC (Quit: Computer has gone to sleep.) [10:43] *** k3asd` has quit IRC (Ping timeout: 246 seconds) [10:44] *** k3asd` has joined #arpnetworks [10:45] *** segv has joined #arpnetworks [10:46] *** HighJinx has joined #arpnetworks [10:52] *** dzup has joined #arpnetworks [12:23] *** segv has quit IRC (Ping timeout: 244 seconds) [12:24] ugh... [12:35] yep. [12:44] lugz [12:55] *** ryk is now known as ServNick [12:56] *** ServNick is now known as ryk [13:19] Is kvr25 back down? [13:19] there was a tweet suggesting more power issues [13:20] check twitter, you'll find most updated info there [13:21] latest notice sent to customer on that box [13:21] https://twitter.com/arpnetworks/status/250691321214410752 -- Shouldn't that have been done this morning? [13:22] https://twitter.com/arpnetworks [13:22] read the full history including more recent posts [13:24] I don't see any more recent tweets. [13:25] sadness [13:25] poor kvr25 [13:26] *** LEMONed has joined #arpnetworks [13:28] after 2 days my ticket is not answered [13:28] what happened to arpnetworks? [13:29] be sure you didn't miss a response, I know I have [13:30] support used to be pretty good years ago [13:31] it still is. there are some issues with one system that are taking support staff's time (and sleep) at the moment. if I were you i'd be patient, but that's just me yammering. [13:31] lol. the live chat is funny. i typed in 'hello' and it went offline [13:31] toddf yea i understand. things happen [13:44] *** LEMONed has quit IRC (Quit: !stfu) [15:15] *** segv has joined #arpnetworks [15:15] up_the_irons: got a vm down, my wifes there any reports of vms down? [15:16] segv: yeah, kvr25 is down, see twitter.com/bsdvps for full log (or check your email, assuming it is not on the vps itself ;) [15:16] roger that [15:23] yeah, got the email [15:23] didn't see it till just now [15:24] cool [15:25] weeee...! there goes the VMs coming up [15:25] sweet [15:25] got some new hardware for it? [15:26] *** flotwig has joined #arpnetworks [15:26] least you guys got a machine online haha [15:27] I am about to face smash this macmini5,1 into the next dimention [15:28] deep-thought.evilrouters.net (206.125.175.18) is UP again at 09/25/2012 06:27:38PM, after 3h 8m of downtime. [15:28] w00t, thanks garry [15:28] yarrr [15:28] sites backonline [15:28] :D [15:28] also means I fixed all my django start scripts haha [15:29] jlgaddis: np! [15:29] segv: LOL [15:29] PS died? [15:29] hey, nothing like an outage to prove you fixed your shit [15:29] :D [15:29] anyways, thanks duder! [15:29] thanks for your patience guys... besides the host "mercury" failure a couple years ago, this was the hardest failure yet [15:30] segv: CaZe : All 8 HDs were swapped into a cold spare chassis and after some fiddling (MAC address, RAID ctrl number, etc... changes) booted up [15:30] so the hardware is all new except for the drives [15:30] yay! [15:31] as for the old hardware, unknown what is wrong with it. that'll be a job for tomorrow ;) [15:31] i suspect probably power bus or something. i have redundant PS, so i doubt both failed at the same time. more likely the bus... which begs the question, why have redundant PS when the bus just craps anyway? bah.. fml [15:31] up_the_irons: haha yeah [15:31] I've had that happen before [15:31] try Verarri blade rack [15:32] when an entire BIC goes out :D [15:32] 8 blades go "POOF" [15:32] Like "Why aren't they showing in the iLO" [15:32] haha [15:32] yeah man... this scares me to deploy blades now [15:32] yeah it is [15:32] i'll buy a cold spare chassis, but man, what a waste of a chassis just sitting there.... and super expensive [15:33] i had that happen at a previous job with 3 blade racks [15:33] yeah it is :/ [15:33] oh well, live and learn [15:34] oh sweet, you fixed the broken feedburner stats too! =) [15:36] wut?! [15:36] hahaha [15:36] * up_the_irons blows the smoke from his pistol [15:36] segv: that sounds even more painful [15:36] yeah, i'm that good [15:36] mercutio: it sucks balls. [15:36] segv: so what, did like 24 servers go poof? [15:36] segv: and what brand was that? I'll know what not to buy... [15:37] Cirrascale now [15:37] http://www.cirrascale.com/ [15:37] used to be "Verari" [15:37] Just out of curiosity, what's the uptime on kvr10? [15:37] i've never heard of either of them :) [15:37] I used them at qualcomm and a few other places [15:37] segv: AH [15:37] up_the_irons: http://evilrouters.net/ see the little thing on the right with the number of rss subscribers? it's been broken for a week. [15:38] jlgaddis: HAHA and i fixed that?:) [15:38] apparently =) [15:38] haha [15:38] probably cached or something stupid [15:38] LOL [15:39] man, how long have i been awake [15:39] the pages get cached, but only for an hour, it's been consistently broken (known issue w/ feedburner) [15:39] ah [15:39] like, 30 hours... [15:39] TIME FOR SLEEP [15:39] i guess that's not too bad [15:39] no no, what is worse is the last time I ate [15:39] TIME FOR FOOD [15:39] food then comatose! [15:40] thankfully, Philippe's is right around the corner (those familiar with downtown LA know what I'm talking about...) [15:42] I do indeed. [15:42] I'm up in san fran [15:42] used to live in san diego [15:42] i know from SD->LA->SF [15:43] I'm dealing with this error at work "Initramfs unpacking failed: junk in compressed archive" right now [15:43] :) [15:43] They want debian on mac mini's and of course me being the engineer [15:43] wooo [15:44] hrm, need to visit the stinking rose again [15:52] segv: hah [15:52] up_the_irons: yeah this is quite shit [15:52] you could prob use a kernel with no modules [15:53] err with no initramfs [15:53] mercutio: it's a sandybridge macmini ;) [15:53] segv: good luck :) [15:53] hmm [15:53] haha yeah [15:53] My brain is like "HAH, have fun..." but my boss is like "Get it to work" [15:53] i struggled for ages the other day to get xen and X to coexist [15:53] segv: think of it this way, you could be me right now... [15:53] I have no hope for it. [15:54] the fubar is, the initrd image isn't corrupted [15:54] and oen of the problems wasn't even xen [15:54] 30 hours no sleep.. and now about to rush order more spare parts (since my spare is now used). wonder if dual spares is cost effective... [15:54] in that ubuntu seemed to break usb support somehow [15:54] must only be on some hardware. but was sandybridge computer, should be pretty normal [15:55] up_the_irons: well eat some food, order some hardware, sleep for 30 hours :) [15:55] up_the_irons: 2 of everything! [15:55] umm you could do the thing of having one big spare ? [15:55] TO THE CLOUD! [15:55] oh, wait [15:56] so that it has more ram / can take more hard-disks, and two nodes could exist within it [15:56] midn you, that makes it much harder to just get another spare [15:56] sounds sexual [15:56] and keep the machine booted [15:56] to the SAN! [16:00] *** amdprophet has quit IRC (Ping timeout: 260 seconds) [16:00] segv: mercutio : LOL good idea [16:01] mercutio: about eating, that is... ;) [16:01] mercutio: one big spare wouldn't work really well b/c these things have to be pretty trivial to "script" and replace. if things are identical, it makes it a lot easier. [16:02] OK, food time, then check support tickets again, then sleep in the tech lounge downstairs... ;) [16:05] *** amdprophet has joined #arpnetworks [16:08] *** amdprophet has quit IRC (Remote host closed the connection) [16:09] co-worker babbling ona bout his iPhone 5 [16:09] damn thing hit 62mbps [16:09] thats insane [16:09] yeah [16:09] we busted out like [16:09] a sprint 4g phone, 2 verizon 4g phones [16:09] and another 4g AT&T phone [16:09] NONE of them hit that [16:10] it's not like people use phones for calling each other anymore :/ [16:10] hah [16:10] yeah [16:10] meanwhile in australia, my provider congests to a point that traffic to US (read: the world) tops at 10kb/sec [16:11] I'm loving the iPhone 5 so far [16:11] jberg: have you tried a vpn? [16:11] mercutio: no [16:11] i've got a friend who moved to australia, frmo new zealand, he said the internet over there is shit [16:11] and people generally say the internet here isn't that great [16:11] shit is actually nice in this context [16:12] apparently all the providers started having huge caps and congesting beyond belief [16:12] 400k/sec on torrents and shit [16:12] i get about 400k/sec on my phone [16:13] and about 80 msec pings to local sites [16:13] yes, thats correct. i think all bandwidth on that cable is used. some providers doesn't overcommit, though [16:13] heh whatever provider he was using had HUGE pings [16:13] it certainly wasn't telstra endeavour [16:14] the latency is about 150 msec from sydney normally isn't it? [16:14] but some providers are like 180 and some are like 200? [16:15] optimal australia<>usw is 150ish [16:15] here's me downloading a 10mb file (gnu.mirrors.tds.net/pub/gnu/gnu/glibc/glibc-1.09.1.tar.gz) hourly since thursday: http://pastebin.com/T7L5Up5a [16:17] wow [16:17] on 3g? [16:17] adsl [16:17] wtf [16:17] 2+ [16:17] it's very inconsistent too [16:17] what are your pings to that site like? [16:18] chicago is ~230 [16:18] oh that file is tiny [16:18] # curl -O gnu.mirrors.tds.net/pub/gnu/gnu/glibc/glibc-1.09.1.tar.gz [16:18] % Total % Received % Xferd Average Speed Time Time Time Current [16:18] Dload Upload Total Spent Left Speed [16:18] 100 1653k 100 1653k 0 0 442k 0 0:00:03 0:00:03 --:--:-- 569k [16:18] that's frmo new zealand [16:19] well as stated above, its basically fighting within providers for room [16:19] heh [16:19] but yeah why i brought up latency before [16:19] is 3g here isn't proxied [16:20] and the combination of the two REALLY screwed up international speeds [16:20] then i used a proxy [16:20] and it was similar to adsl speeds [16:20] adsl 1 that is [16:22] *** segv has quit IRC (Quit: Lost terminal) [16:23] *** amdprophet has joined #arpnetworks [17:00] *** Ehtyar has joined #arpnetworks [18:38] *** HighJinx has quit IRC (Quit: Computer has gone to sleep.) [22:14] *** nixbag has joined #arpnetworks [22:57] *** Webhostbudd has joined #arpnetworks [23:01] *** HighJinx has joined #arpnetworks [23:15] *** CaZe` has joined #arpnetworks [23:15] *** CaZe has quit IRC (Read error: Connection reset by peer) [23:15] *** CaZe` is now known as CaZe