***: LT has joined #arpnetworks
mtve has quit IRC (*.net *.split)
henderb has quit IRC (*.net *.split)
Lefty has quit IRC (*.net *.split)
henderb has joined #arpnetworks
Webhostbudd_ has quit IRC (Quit: Leaving)
Lefty has joined #arpnetworks
HighJinx has quit IRC (Remote host closed the connection)
HighJinx has joined #arpnetworks
k3asd` has quit IRC (Ping timeout: 246 seconds)
k3asd` has joined #arpnetworks
k3asd` has quit IRC (Read error: Connection reset by peer)
k3asd` has joined #arpnetworks
k3asd` has quit IRC (Ping timeout: 248 seconds)
k3asd` has joined #arpnetworks
ameise has joined #arpnetworks
ant has quit IRC (Remote host closed the connection)
ameise is now known as ant
plett_ is now known as plett
segv has quit IRC (Ping timeout: 240 seconds) jlgaddis: uh oh
up_the_irons: ping? up_the_irons: jlgaddis: uhoh wut?:)
oh ah, kvr25 jlgaddis: one vps dropped off the face of the planet, other was fine, so figured a physical host issue up_the_irons: jlgaddis: yeah, I got the page a little while ago, still investigating jlgaddis: 10-4 up_the_irons: <sigh>
Power Control Fault : true
looks like I gotta make a drive to push a button (and hope that's all it is) ***: ameise has joined #arpnetworks
alexstanford has joined #arpnetworks alexstanford: Am I the only one unable to connect to my VPS currently? josephb: up_the_irons: is on his way to fix kvr25
assuming you are on that one alexstanford: So, this is a known issue?
I am indeed.
Alright, thanks. I'll just sit tight. Any clue what the issue is? josephb: yeah, about 30mins ago he was paged jlgaddis: i got the first alert at 1227 utc alexstanford: Scared me, almost thought some DNS changes I made late last night caused the issue but then when I got no response over SSH... jlgaddis: so, ~51 mins ago. power issue apparently, alexstanford. alexstanford: What, like the PSU burnt out? josephb: "Power Control Fault : true" is what up_the_irons posted in here earlier
I assume that's from ipmi or some oob management on th box alexstanford: Interesting
Hopefully it won't be down too much longer ***: ant has quit IRC (Quit: Leaving)
ameise is now known as ant
ix33 has quit IRC (Quit: WeeChat 0.3.9-rc2)
ix33 has joined #arpnetworks up_the_irons: yeah, power control fault reported by IPMI
i have it powered up now, it is just starting the main OS fs checks
alexstanford: josephb jlgaddis ^^ toddf: whew, not my kvr host ;-) up_the_irons: it's like server roulette
gotta love all the VM fs checks happening all at the same time... load skyrockets
230 days of uptime on kvr25, meh... kinda low for me
interesting
Power Control Fault : false
Now it says it is OK
I suppose it wanted to be truly powered down, even the PS and IPMI
cuz that cleared it
first time IPMI has failed for me regarding power.... lesson learned, I will configured the AP7932 to be remotely accessible now... ;)
outage reports sent ***: LT has quit IRC (Quit: Leaving) milki: hehe ***: dzup has quit IRC (Read error: Connection reset by peer)
HighJinx has quit IRC (Quit: Computer has gone to sleep.)
k3asd` has quit IRC (Ping timeout: 246 seconds)
k3asd` has joined #arpnetworks
segv has joined #arpnetworks
HighJinx has joined #arpnetworks
dzup has joined #arpnetworks
segv has quit IRC (Ping timeout: 244 seconds) up_the_irons: ugh... jdoe: yep. qbit: lugz ***: ryk is now known as ServNick
ServNick is now known as ryk alexstanford: Is kvr25 back down? toddf: there was a tweet suggesting more power issues
check twitter, you'll find most updated info there up_the_irons: latest notice sent to customer on that box alexstanford: https://twitter.com/arpnetworks/status/250691321214410752 -- Shouldn't that have been done this morning? toddf: https://twitter.com/arpnetworks
read the full history including more recent posts alexstanford: I don't see any more recent tweets. milki: sadness
poor kvr25 ***: LEMONed has joined #arpnetworks LEMONed: after 2 days my ticket is not answered
what happened to arpnetworks? toddf: be sure you didn't miss a response, I know I have LEMONed: support used to be pretty good years ago toddf: it still is. there are some issues with one system that are taking support staff's time (and sleep) at the moment. if I were you i'd be patient, but that's just me yammering. LEMONed: lol. the live chat is funny. i typed in 'hello' and it went offline
toddf yea i understand. things happen ***: LEMONed has quit IRC (Quit: !stfu)
segv has joined #arpnetworks segv: up_the_irons: got a vm down, my wifes there any reports of vms down? up_the_irons: segv: yeah, kvr25 is down, see twitter.com/bsdvps for full log (or check your email, assuming it is not on the vps itself ;) segv: roger that
yeah, got the email
didn't see it till just now up_the_irons: cool
weeee...! there goes the VMs coming up segv: sweet
got some new hardware for it? ***: flotwig has joined #arpnetworks segv: least you guys got a machine online haha
I am about to face smash this macmini5,1 into the next dimention jlgaddis: deep-thought.evilrouters.net (206.125.175.18) is UP again at 09/25/2012 06:27:38PM, after 3h 8m of downtime.
w00t, thanks garry segv: yarrr
sites backonline
:D
also means I fixed all my django start scripts haha up_the_irons: jlgaddis: np!
segv: LOL CaZe: PS died? segv: hey, nothing like an outage to prove you fixed your shit
:D
anyways, thanks duder! up_the_irons: thanks for your patience guys... besides the host "mercury" failure a couple years ago, this was the hardest failure yet
segv: CaZe : All 8 HDs were swapped into a cold spare chassis and after some fiddling (MAC address, RAID ctrl number, etc... changes) booted up
so the hardware is all new except for the drives milki: yay! up_the_irons: as for the old hardware, unknown what is wrong with it. that'll be a job for tomorrow ;)
i suspect probably power bus or something. i have redundant PS, so i doubt both failed at the same time. more likely the bus... which begs the question, why have redundant PS when the bus just craps anyway? bah.. fml segv: up_the_irons: haha yeah
I've had that happen before
try Verarri blade rack
when an entire BIC goes out :D
8 blades go "POOF"
Like "Why aren't they showing in the iLO"
haha up_the_irons: yeah man... this scares me to deploy blades now segv: yeah it is up_the_irons: i'll buy a cold spare chassis, but man, what a waste of a chassis just sitting there.... and super expensive segv: i had that happen at a previous job with 3 blade racks
yeah it is :/ up_the_irons: oh well, live and learn jlgaddis: oh sweet, you fixed the broken feedburner stats too! =) up_the_irons: wut?! segv: hahaha -: up_the_irons blows the smoke from his pistol mercutio: segv: that sounds even more painful up_the_irons: yeah, i'm that good segv: mercutio: it sucks balls. up_the_irons: segv: so what, did like 24 servers go poof?
segv: and what brand was that? I'll know what not to buy... segv: Cirrascale now
http://www.cirrascale.com/
used to be "Verari" CaZe: Just out of curiosity, what's the uptime on kvr10? mercutio: i've never heard of either of them :) segv: I used them at qualcomm and a few other places up_the_irons: segv: AH jlgaddis: up_the_irons: http://evilrouters.net/ see the little thing on the right with the number of rss subscribers? it's been broken for a week. up_the_irons: jlgaddis: HAHA and i fixed that?:) jlgaddis: apparently =) segv: haha
probably cached or something stupid up_the_irons: LOL
man, how long have i been awake jlgaddis: the pages get cached, but only for an hour, it's been consistently broken (known issue w/ feedburner) segv: ah up_the_irons: like, 30 hours... segv: TIME FOR SLEEP up_the_irons: i guess that's not too bad
no no, what is worse is the last time I ate
TIME FOR FOOD segv: food then comatose! up_the_irons: thankfully, Philippe's is right around the corner (those familiar with downtown LA know what I'm talking about...) segv: I do indeed.
I'm up in san fran
used to live in san diego
i know from SD->LA->SF
I'm dealing with this error at work "Initramfs unpacking failed: junk in compressed archive" right now
:)
They want debian on mac mini's and of course me being the engineer
wooo jlgaddis: hrm, need to visit the stinking rose again up_the_irons: segv: hah segv: up_the_irons: yeah this is quite shit mercutio: you could prob use a kernel with no modules
err with no initramfs segv: mercutio: it's a sandybridge macmini ;) up_the_irons: segv: good luck :) mercutio: hmm segv: haha yeah
My brain is like "HAH, have fun..." but my boss is like "Get it to work" mercutio: i struggled for ages the other day to get xen and X to coexist up_the_irons: segv: think of it this way, you could be me right now... segv: I have no hope for it.
the fubar is, the initrd image isn't corrupted mercutio: and oen of the problems wasn't even xen up_the_irons: 30 hours no sleep.. and now about to rush order more spare parts (since my spare is now used). wonder if dual spares is cost effective... mercutio: in that ubuntu seemed to break usb support somehow
must only be on some hardware. but was sandybridge computer, should be pretty normal segv: up_the_irons: well eat some food, order some hardware, sleep for 30 hours :) mercutio: up_the_irons: 2 of everything!
umm you could do the thing of having one big spare ? jlgaddis: TO THE CLOUD!
oh, wait mercutio: so that it has more ram / can take more hard-disks, and two nodes could exist within it
midn you, that makes it much harder to just get another spare jlgaddis: sounds sexual mercutio: and keep the machine booted
to the SAN! ***: amdprophet has quit IRC (Ping timeout: 260 seconds) up_the_irons: segv: mercutio : LOL good idea
mercutio: about eating, that is... ;)
mercutio: one big spare wouldn't work really well b/c these things have to be pretty trivial to "script" and replace. if things are identical, it makes it a lot easier.
OK, food time, then check support tickets again, then sleep in the tech lounge downstairs... ;) ***: amdprophet has joined #arpnetworks
amdprophet has quit IRC (Remote host closed the connection) segv: co-worker babbling ona bout his iPhone 5
damn thing hit 62mbps jbergstroem: thats insane segv: yeah
we busted out like
a sprint 4g phone, 2 verizon 4g phones
and another 4g AT&T phone
NONE of them hit that mercutio: it's not like people use phones for calling each other anymore :/ pjs: hah segv: yeah jbergstroem: meanwhile in australia, my provider congests to a point that traffic to US (read: the world) tops at 10kb/sec pjs: I'm loving the iPhone 5 so far mercutio: jberg: have you tried a vpn? jbergstroem: mercutio: no mercutio: i've got a friend who moved to australia, frmo new zealand, he said the internet over there is shit
and people generally say the internet here isn't that great jbergstroem: shit is actually nice in this context mercutio: apparently all the providers started having huge caps and congesting beyond belief
400k/sec on torrents and shit
i get about 400k/sec on my phone
and about 80 msec pings to local sites jbergstroem: yes, thats correct. i think all bandwidth on that cable is used. some providers doesn't overcommit, though mercutio: heh whatever provider he was using had HUGE pings
it certainly wasn't telstra endeavour
the latency is about 150 msec from sydney normally isn't it?
but some providers are like 180 and some are like 200? jbergstroem: optimal australia<>usw is 150ish
here's me downloading a 10mb file (gnu.mirrors.tds.net/pub/gnu/gnu/glibc/glibc-1.09.1.tar.gz) hourly since thursday: http://pastebin.com/T7L5Up5a mercutio: wow
on 3g? jbergstroem: adsl mercutio: wtf jbergstroem: 2+ mercutio: it's very inconsistent too
what are your pings to that site like? jbergstroem: chicago is ~230 mercutio: oh that file is tiny
# curl -O gnu.mirrors.tds.net/pub/gnu/gnu/glibc/glibc-1.09.1.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1653k 100 1653k 0 0 442k 0 0:00:03 0:00:03 --:--:-- 569k
that's frmo new zealand jbergstroem: well as stated above, its basically fighting within providers for room mercutio: heh
but yeah why i brought up latency before
is 3g here isn't proxied
and the combination of the two REALLY screwed up international speeds
then i used a proxy
and it was similar to adsl speeds
adsl 1 that is ***: segv has quit IRC (Quit: Lost terminal)
amdprophet has joined #arpnetworks
Ehtyar has joined #arpnetworks
HighJinx has quit IRC (Quit: Computer has gone to sleep.)
nixbag has joined #arpnetworks
Webhostbudd has joined #arpnetworks
HighJinx has joined #arpnetworks
CaZe` has joined #arpnetworks
CaZe has quit IRC (Read error: Connection reset by peer)
CaZe` is now known as CaZe