| ↑back Search ←Prev date Next date→ Show only urls | (Click on time to select a line by its url) |
| Who | What | When |
|---|---|---|
| *** | Ehtyar has quit IRC (Quit: Never look down on someone unless you're helping them up.) | [00:26] |
| .......... (idle for 47mn) | ||
| LT has joined #arpnetworks | [01:13] | |
| .......... (idle for 48mn) | ||
| mtve has quit IRC (*.net *.split)
henderb has quit IRC (*.net *.split) Lefty has quit IRC (*.net *.split) henderb has joined #arpnetworks | [02:01] | |
| Webhostbudd_ has quit IRC (Quit: Leaving) | [02:13] | |
| Lefty has joined #arpnetworks | [02:18] | |
| ..... (idle for 20mn) | ||
| HighJinx has quit IRC (Remote host closed the connection)
HighJinx has joined #arpnetworks | [02:38] | |
| ..................... (idle for 1h42mn) | ||
| k3asd` has quit IRC (Ping timeout: 246 seconds)
k3asd` has joined #arpnetworks k3asd` has quit IRC (Read error: Connection reset by peer) k3asd` has joined #arpnetworks | [04:20] | |
| ..... (idle for 24mn) | ||
| k3asd` has quit IRC (Ping timeout: 248 seconds)
k3asd` has joined #arpnetworks ameise has joined #arpnetworks | [04:47] | |
| ant has quit IRC (Remote host closed the connection)
ameise is now known as ant | [04:55] | |
| ..... (idle for 21mn) | ||
| plett_ is now known as plett | [05:16] | |
| segv has quit IRC (Ping timeout: 240 seconds) | [05:28] | |
| jlgaddis | uh oh | [05:31] |
| up_the_irons: ping? | [05:41] | |
| up_the_irons | jlgaddis: uhoh wut?:)
oh ah, kvr25 | [05:41] |
| jlgaddis | one vps dropped off the face of the planet, other was fine, so figured a physical host issue | [05:42] |
| up_the_irons | jlgaddis: yeah, I got the page a little while ago, still investigating | [05:44] |
| jlgaddis | 10-4 | [05:45] |
| .... (idle for 15mn) | ||
| up_the_irons | <sigh>
Power Control Fault : true looks like I gotta make a drive to push a button (and hope that's all it is) | [06:00] |
| *** | ameise has joined #arpnetworks
alexstanford has joined #arpnetworks | [06:11] |
| alexstanford | Am I the only one unable to connect to my VPS currently? | [06:15] |
| josephb | up_the_irons: is on his way to fix kvr25
assuming you are on that one | [06:15] |
| alexstanford | So, this is a known issue?
I am indeed. Alright, thanks. I'll just sit tight. Any clue what the issue is? | [06:16] |
| josephb | yeah, about 30mins ago he was paged | [06:17] |
| jlgaddis | i got the first alert at 1227 utc | [06:18] |
| alexstanford | Scared me, almost thought some DNS changes I made late last night caused the issue but then when I got no response over SSH... | [06:18] |
| jlgaddis | so, ~51 mins ago. power issue apparently, alexstanford. | [06:18] |
| alexstanford | What, like the PSU burnt out? | [06:19] |
| josephb | "Power Control Fault : true" is what up_the_irons posted in here earlier
I assume that's from ipmi or some oob management on th box | [06:20] |
| alexstanford | Interesting | [06:20] |
| Hopefully it won't be down too much longer | [06:34] | |
| *** | ant has quit IRC (Quit: Leaving)
ameise is now known as ant | [06:35] |
| ........ (idle for 35mn) | ||
| ix33 has quit IRC (Quit: WeeChat 0.3.9-rc2)
ix33 has joined #arpnetworks | [07:10] | |
| ........ (idle for 35mn) | ||
| up_the_irons | yeah, power control fault reported by IPMI
i have it powered up now, it is just starting the main OS fs checks alexstanford: josephb jlgaddis ^^ | [07:46] |
| toddf | whew, not my kvr host ;-) | [07:52] |
| up_the_irons | it's like server roulette
gotta love all the VM fs checks happening all at the same time... load skyrockets 230 days of uptime on kvr25, meh... kinda low for me interesting Power Control Fault : false Now it says it is OK I suppose it wanted to be truly powered down, even the PS and IPMI cuz that cleared it first time IPMI has failed for me regarding power.... lesson learned, I will configured the AP7932 to be remotely accessible now... ;) | [07:52] |
| outage reports sent | [08:10] | |
| .... (idle for 19mn) | ||
| *** | LT has quit IRC (Quit: Leaving) | [08:29] |
| milki | hehe | [08:31] |
| ......... (idle for 43mn) | ||
| *** | dzup has quit IRC (Read error: Connection reset by peer) | [09:14] |
| ............... (idle for 1h14mn) | ||
| HighJinx has quit IRC (Quit: Computer has gone to sleep.) | [10:28] | |
| .... (idle for 15mn) | ||
| k3asd` has quit IRC (Ping timeout: 246 seconds)
k3asd` has joined #arpnetworks segv has joined #arpnetworks HighJinx has joined #arpnetworks | [10:43] | |
| dzup has joined #arpnetworks | [10:52] | |
| ................... (idle for 1h31mn) | ||
| segv has quit IRC (Ping timeout: 244 seconds) | [12:23] | |
| up_the_irons | ugh... | [12:24] |
| jdoe | yep. | [12:35] |
| qbit | lugz | [12:44] |
| *** | ryk is now known as ServNick
ServNick is now known as ryk | [12:55] |
| ..... (idle for 23mn) | ||
| alexstanford | Is kvr25 back down? | [13:19] |
| toddf | there was a tweet suggesting more power issues
check twitter, you'll find most updated info there | [13:19] |
| up_the_irons | latest notice sent to customer on that box | [13:21] |
| alexstanford | https://twitter.com/arpnetworks/status/250691321214410752 -- Shouldn't that have been done this morning? | [13:21] |
| toddf | https://twitter.com/arpnetworks
read the full history including more recent posts | [13:22] |
| alexstanford | I don't see any more recent tweets. | [13:24] |
| milki | sadness
poor kvr25 | [13:25] |
| *** | LEMONed has joined #arpnetworks | [13:26] |
| LEMONed | after 2 days my ticket is not answered
what happened to arpnetworks? | [13:28] |
| toddf | be sure you didn't miss a response, I know I have | [13:29] |
| LEMONed | support used to be pretty good years ago | [13:30] |
| toddf | it still is. there are some issues with one system that are taking support staff's time (and sleep) at the moment. if I were you i'd be patient, but that's just me yammering. | [13:31] |
| LEMONed | lol. the live chat is funny. i typed in 'hello' and it went offline
toddf yea i understand. things happen | [13:31] |
| *** | LEMONed has quit IRC (Quit: !stfu) | [13:44] |
| ................... (idle for 1h31mn) | ||
| segv has joined #arpnetworks | [15:15] | |
| segv | up_the_irons: got a vm down, my wifes there any reports of vms down? | [15:15] |
| up_the_irons | segv: yeah, kvr25 is down, see twitter.com/bsdvps for full log (or check your email, assuming it is not on the vps itself ;) | [15:16] |
| segv | roger that | [15:16] |
| yeah, got the email
didn't see it till just now | [15:23] | |
| up_the_irons | cool
weeee...! there goes the VMs coming up | [15:24] |
| segv | sweet
got some new hardware for it? | [15:25] |
| *** | flotwig has joined #arpnetworks | [15:26] |
| segv | least you guys got a machine online haha
I am about to face smash this macmini5,1 into the next dimention | [15:26] |
| jlgaddis | deep-thought.evilrouters.net (206.125.175.18) is UP again at 09/25/2012 06:27:38PM, after 3h 8m of downtime.
w00t, thanks garry | [15:28] |
| segv | yarrr
sites backonline :D also means I fixed all my django start scripts haha | [15:28] |
| up_the_irons | jlgaddis: np!
segv: LOL | [15:29] |
| CaZe | PS died? | [15:29] |
| segv | hey, nothing like an outage to prove you fixed your shit
:D anyways, thanks duder! | [15:29] |
| up_the_irons | thanks for your patience guys... besides the host "mercury" failure a couple years ago, this was the hardest failure yet
segv: CaZe : All 8 HDs were swapped into a cold spare chassis and after some fiddling (MAC address, RAID ctrl number, etc... changes) booted up so the hardware is all new except for the drives | [15:29] |
| milki | yay! | [15:30] |
| up_the_irons | as for the old hardware, unknown what is wrong with it. that'll be a job for tomorrow ;)
i suspect probably power bus or something. i have redundant PS, so i doubt both failed at the same time. more likely the bus... which begs the question, why have redundant PS when the bus just craps anyway? bah.. fml | [15:31] |
| segv | up_the_irons: haha yeah
I've had that happen before try Verarri blade rack when an entire BIC goes out :D 8 blades go "POOF" Like "Why aren't they showing in the iLO" haha | [15:31] |
| up_the_irons | yeah man... this scares me to deploy blades now | [15:32] |
| segv | yeah it is | [15:32] |
| up_the_irons | i'll buy a cold spare chassis, but man, what a waste of a chassis just sitting there.... and super expensive | [15:32] |
| segv | i had that happen at a previous job with 3 blade racks
yeah it is :/ | [15:33] |
| up_the_irons | oh well, live and learn | [15:33] |
| jlgaddis | oh sweet, you fixed the broken feedburner stats too! =) | [15:34] |
| up_the_irons | wut?! | [15:36] |
| segv | hahaha | [15:36] |
| up_the_irons | up_the_irons blows the smoke from his pistol | [15:36] |
| mercutio | segv: that sounds even more painful | [15:36] |
| up_the_irons | yeah, i'm that good | [15:36] |
| segv | mercutio: it sucks balls. | [15:36] |
| up_the_irons | segv: so what, did like 24 servers go poof?
segv: and what brand was that? I'll know what not to buy... | [15:36] |
| segv | Cirrascale now
http://www.cirrascale.com/ used to be "Verari" | [15:37] |
| CaZe | Just out of curiosity, what's the uptime on kvr10? | [15:37] |
| mercutio | i've never heard of either of them :) | [15:37] |
| segv | I used them at qualcomm and a few other places | [15:37] |
| up_the_irons | segv: AH | [15:37] |
| jlgaddis | up_the_irons: http://evilrouters.net/ see the little thing on the right with the number of rss subscribers? it's been broken for a week. | [15:37] |
| up_the_irons | jlgaddis: HAHA and i fixed that?:) | [15:38] |
| jlgaddis | apparently =) | [15:38] |
| segv | haha
probably cached or something stupid | [15:38] |
| up_the_irons | LOL
man, how long have i been awake | [15:38] |
| jlgaddis | the pages get cached, but only for an hour, it's been consistently broken (known issue w/ feedburner) | [15:39] |
| segv | ah | [15:39] |
| up_the_irons | like, 30 hours... | [15:39] |
| segv | TIME FOR SLEEP | [15:39] |
| up_the_irons | i guess that's not too bad
no no, what is worse is the last time I ate TIME FOR FOOD | [15:39] |
| segv | food then comatose! | [15:39] |
| up_the_irons | thankfully, Philippe's is right around the corner (those familiar with downtown LA know what I'm talking about...) | [15:40] |
| segv | I do indeed.
I'm up in san fran used to live in san diego i know from SD->LA->SF I'm dealing with this error at work "Initramfs unpacking failed: junk in compressed archive" right now :) They want debian on mac mini's and of course me being the engineer wooo | [15:42] |
| jlgaddis | hrm, need to visit the stinking rose again | [15:44] |
| up_the_irons | segv: hah | [15:52] |
| segv | up_the_irons: yeah this is quite shit | [15:52] |
| mercutio | you could prob use a kernel with no modules
err with no initramfs | [15:52] |
| segv | mercutio: it's a sandybridge macmini ;) | [15:53] |
| up_the_irons | segv: good luck :) | [15:53] |
| mercutio | hmm | [15:53] |
| segv | haha yeah
My brain is like "HAH, have fun..." but my boss is like "Get it to work" | [15:53] |
| mercutio | i struggled for ages the other day to get xen and X to coexist | [15:53] |
| up_the_irons | segv: think of it this way, you could be me right now... | [15:53] |
| segv | I have no hope for it.
the fubar is, the initrd image isn't corrupted | [15:53] |
| mercutio | and oen of the problems wasn't even xen | [15:54] |
| up_the_irons | 30 hours no sleep.. and now about to rush order more spare parts (since my spare is now used). wonder if dual spares is cost effective... | [15:54] |
| mercutio | in that ubuntu seemed to break usb support somehow
must only be on some hardware. but was sandybridge computer, should be pretty normal | [15:54] |
| segv | up_the_irons: well eat some food, order some hardware, sleep for 30 hours :) | [15:55] |
| mercutio | up_the_irons: 2 of everything!
umm you could do the thing of having one big spare ? | [15:55] |
| jlgaddis | TO THE CLOUD!
oh, wait | [15:55] |
| mercutio | so that it has more ram / can take more hard-disks, and two nodes could exist within it
midn you, that makes it much harder to just get another spare | [15:56] |
| jlgaddis | sounds sexual | [15:56] |
| mercutio | and keep the machine booted
to the SAN! | [15:56] |
| *** | amdprophet has quit IRC (Ping timeout: 260 seconds) | [16:00] |
| up_the_irons | segv: mercutio : LOL good idea
mercutio: about eating, that is... ;) mercutio: one big spare wouldn't work really well b/c these things have to be pretty trivial to "script" and replace. if things are identical, it makes it a lot easier. OK, food time, then check support tickets again, then sleep in the tech lounge downstairs... ;) | [16:00] |
| *** | amdprophet has joined #arpnetworks
amdprophet has quit IRC (Remote host closed the connection) | [16:05] |
| segv | co-worker babbling ona bout his iPhone 5
damn thing hit 62mbps | [16:09] |
| jbergstroem | thats insane | [16:09] |
| segv | yeah
we busted out like a sprint 4g phone, 2 verizon 4g phones and another 4g AT&T phone NONE of them hit that | [16:09] |
| mercutio | it's not like people use phones for calling each other anymore :/ | [16:10] |
| pjs | hah | [16:10] |
| segv | yeah | [16:10] |
| jbergstroem | meanwhile in australia, my provider congests to a point that traffic to US (read: the world) tops at 10kb/sec | [16:10] |
| pjs | I'm loving the iPhone 5 so far | [16:11] |
| mercutio | jberg: have you tried a vpn? | [16:11] |
| jbergstroem | mercutio: no | [16:11] |
| mercutio | i've got a friend who moved to australia, frmo new zealand, he said the internet over there is shit
and people generally say the internet here isn't that great | [16:11] |
| jbergstroem | shit is actually nice in this context | [16:11] |
| mercutio | apparently all the providers started having huge caps and congesting beyond belief
400k/sec on torrents and shit i get about 400k/sec on my phone and about 80 msec pings to local sites | [16:12] |
| jbergstroem | yes, thats correct. i think all bandwidth on that cable is used. some providers doesn't overcommit, though | [16:13] |
| mercutio | heh whatever provider he was using had HUGE pings
it certainly wasn't telstra endeavour the latency is about 150 msec from sydney normally isn't it? but some providers are like 180 and some are like 200? | [16:13] |
| jbergstroem | optimal australia<>usw is 150ish
here's me downloading a 10mb file (gnu.mirrors.tds.net/pub/gnu/gnu/glibc/glibc-1.09.1.tar.gz) hourly since thursday: http://pastebin.com/T7L5Up5a | [16:15] |
| mercutio | wow
on 3g? | [16:17] |
| jbergstroem | adsl | [16:17] |
| mercutio | wtf | [16:17] |
| jbergstroem | 2+ | [16:17] |
| mercutio | it's very inconsistent too
what are your pings to that site like? | [16:17] |
| jbergstroem | chicago is ~230 | [16:18] |
| mercutio | oh that file is tiny
# curl -O gnu.mirrors.tds.net/pub/gnu/gnu/glibc/glibc-1.09.1.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1653k 100 1653k 0 0 442k 0 0:00:03 0:00:03 --:--:-- 569k that's frmo new zealand | [16:18] |
| jbergstroem | well as stated above, its basically fighting within providers for room | [16:19] |
| mercutio | heh
but yeah why i brought up latency before is 3g here isn't proxied and the combination of the two REALLY screwed up international speeds then i used a proxy and it was similar to adsl speeds adsl 1 that is | [16:19] |
| *** | segv has quit IRC (Quit: Lost terminal)
amdprophet has joined #arpnetworks | [16:22] |
| ........ (idle for 37mn) | ||
| Ehtyar has joined #arpnetworks | [17:00] | |
| .................... (idle for 1h38mn) | ||
| HighJinx has quit IRC (Quit: Computer has gone to sleep.) | [18:38] | |
| ............................................ (idle for 3h36mn) | ||
| nixbag has joined #arpnetworks | [22:14] | |
| ......... (idle for 43mn) | ||
| Webhostbudd has joined #arpnetworks
HighJinx has joined #arpnetworks | [22:57] | |
| CaZe` has joined #arpnetworks
CaZe has quit IRC (Read error: Connection reset by peer) CaZe` is now known as CaZe | [23:15] | |
| ↑back Search ←Prev date Next date→ Show only urls | (Click on time to select a line by its url) |