#arpnetworks 2012-08-27,Mon

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***gcw|mbpro has quit IRC (Remote host closed the connection)
gcw|mbpro has joined #arpnetworks
[00:08]
Webhostbudd has quit IRC (Quit: Leaving) [00:19]
.... (idle for 15mn)
Ehtyar has quit IRC (Quit: Going!) [00:34]
.................. (idle for 1h28mn)
`ariel has quit IRC (Ping timeout: 245 seconds)
ariel has joined #arpnetworks
[02:02]
........................................................................... (idle for 6h12mn)
RandalSchwartz has joined #arpnetworks [08:15]
RandalSchwartzMy disk (on kvr05) is acting horribly slow.
is there a system status update that might be relevant?
up_the_irons?
[08:15]
***himuraken has joined #arpnetworks [08:26]
RandalSchwartz has quit IRC (Ping timeout: 246 seconds) [08:36]
......... (idle for 42mn)
ryk has joined #arpnetworks
ryk has quit IRC (Changing host)
ryk has joined #arpnetworks
[09:18]
Webhostbudd has joined #arpnetworks
RandalSchwartz has joined #arpnetworks
RandalSchwartz has quit IRC (Changing host)
RandalSchwartz has joined #arpnetworks
[09:27]
k3asd` has quit IRC (Ping timeout: 246 seconds) [09:43]
k3asd` has joined #arpnetworks [09:56]
sako has joined #arpnetworks [10:06]
.... (idle for 16mn)
hive-mind has quit IRC (Remote host closed the connection)
hive-mind has joined #arpnetworks
[10:22]
........ (idle for 37mn)
k3asd` has quit IRC (Ping timeout: 246 seconds) [11:01]
k3asd` has joined #arpnetworks [11:14]
...... (idle for 25mn)
k3asd has joined #arpnetworks
k3asd` has quit IRC (Ping timeout: 248 seconds)
[11:39]
....... (idle for 30mn)
nestea has quit IRC (Ping timeout: 260 seconds)
nestea has joined #arpnetworks
[12:11]
RandalSchwartz has quit IRC (Ping timeout: 246 seconds)
mtve has quit IRC (Ping timeout: 244 seconds)
mtve has joined #arpnetworks
RandalSchwartz has joined #arpnetworks
RandalSchwartz has quit IRC (Changing host)
RandalSchwartz has joined #arpnetworks
[12:19]
RandalSchwartzup_the_irons - any idea about my slow disk?
been crappy all morning
do I have a noisy neighbor again?
[12:24]
DAMN... this needs fixing.
1048576 bytes transferred in 22.494234 secs (46615 bytes/sec)
it took 22 seconds to write A MEGABYTE
up_the_irons ding ding ding
[12:38]
up_the_ironsRandalSchwartz: yeah, i've been having intermittent problems with the host kvr05 [12:40]
RandalSchwartzcan you migrate me to a different box then?
it's interfering with business and tasks.
[12:40]
jdoemine's fairly slow too.
52428800 bytes (52 MB) copied, 71.7589 s, 731 kB/s
:/
[12:41]
up_the_ironsRandalSchwartz: the first priority is to fix the issue with kvr05; i think it is a noisy neighbor, as you said [12:45]
***RandalSchwartz has quit IRC (Ping timeout: 246 seconds) [12:45]
up_the_ironsjdoe: are you able to reach your vps at all? (i assume perhaps so, b/c you could get those test results) [12:49]
jdoeup_the_irons: yeah. [12:54]
up_the_ironsjdoe: and it is on kvr05? [12:54]
jdoenot sure offhand.
how would I tell?
[12:55]
up_the_ironsjdoe: it's the same as your vnc host (listed in portal under vm details) [12:55]
jdoeyou can tell how often I log in ;)
sec.
[12:55]
up_the_irons:) [12:56]
arielmm i'm on KVR05 and is sloww, my monitoring software is detecting the VM as down/up/down/up [12:57]
up_the_ironsi can't ssh into kvr05. serial console is responsive, however, it also won't let me in past me typing in my login name. i think the disk is completely locked up [12:57]
CaZekvr10 is doing fine. :D [12:57]
up_the_ironsariel: although it is interesting that you can still get in, even though i can't get into the host. [12:57]
jdoeup_the_irons: lol... I have at least two portal accounts, neither showing my vps.
nope, I'm on kvr06
[12:58]
up_the_ironsjdoe: roger [13:00]
jdoeseems better now, 6-ish M/s. [13:03]
up_the_ironsjdoe: i'm not touching kvr06 ;)
but 6M/s is still pretty slow
[13:05]
jdoeI know, I'm just saying whatever it was, it's improved slightly. And no argument ;) [13:06]
up_the_ironskvr05, i'm in!
jdoe: roger :)
now to find out who is hogging the disk
[13:06]
jdoeprolly that schwartz guy.
jdoe NODS SAGELY.
[13:07]
up_the_ironsrandal actually _does_ use quite a bit of I/O, but not enough to take down the box.. ;) [13:08]
jdoeI'm surprised any guest can make it *that* unresponsive. [13:08]
toddfmaybe someone is swapping? [13:09]
jpalmerup_the_irons: sorry for the delay on centos. I had to board up house and horses for the storm. I'll be unboarding and such tonight and tomorrow.. then can get back to work on it. [13:11]
up_the_ironsjdoe: well, kvm/qemu doesn't really have too much in the way of I/O isolation. Like, each VM is just a Linux process and if it wants to really load up the disk, the scheduler isn't gonna stop it
jpalmer: oh problem at all, take your time. I'm actually going to make the 5.8 version :)
[13:15]
twobithackerI saw a presentation from Verio a few years back where they had talked about all the custom work they had done to give a level of control over disk I/O, memory and CPU usage to their VPS product
they claimed they were going to release that back to the FreeBSD project, but never did :/
[13:16]
jdoeup_the_irons: linux does kinda suck for that. [13:17]
up_the_ironsI have come up with a new way of dealing with disk provisioning for Linux guests that will afford me a *much* easier time supporting more distros; therefore, I'm pumped to make CentOS 5.8 templates, then Ubuntu 12.04 after that, then who knows... Fedora, Gentoo, Arch, ...? you name it [13:17]
jdoeLFS PLZ. [13:18]
up_the_ironstwobithacker: jdoe yeah, i've seen several proprietary solutions talked about for that stuff, but nothing out in the open (but i admit, i haven't looked into it in a while) [13:18]
toddfas the list grows longer so does the amount of effort to maintain such a list. I know it won't effect our bottom line but still... ;-)
<sarcasm> doesn't ionice work? </sarcasm>
[13:18]
jdoeup_the_irons: yeah I dunno, it doesn't seem like a problem anyone is anxious to solve, despite the work that's gone into cfq and deadline. I dunno why. I can load the shit out of the disk on fbsd, and the machine is still responsive.
try that on linux and it craters.
[13:19]
up_the_ironstoddf: well, that's the thing, up until now the effort to maintain such a list has been prohibitive; but, with my new disk provisioning strategy, no longer :) [13:19]
toddfI'm not in on all the details, but with a new disk provisioning strategy, doesn't that still mean someone has to manually test install a distribution for it to work? [13:20]
up_the_ironsjdoe: i think it is a problem that people have accepted; kinda like DoS attacks, they suck but what can ya do? ;)
toddf: yes, but only once, then the template is copied for every VM down the line. new distro versions don't come out _that_ often; openbsd is actually the most frequent publisher, where I find myself having to update my templates every 6 months ;)
[13:20]
toddfup_the_irons: request multiple vps's across different kvr* hosts like randalschwartz, and make it a cluster that will lessen the effect of one kvr* with temporary io starvation ;_) [13:21]
jdoeeh, ubuntu is on a 6 month cycle too isn't it? [13:22]
up_the_ironstoddf: wait wut?;)
jdoe: is it? i haven't even noticed... for ubuntu i tend to do only the LTS versions
if you want non-LTS, a ubuntu fan can just get an LTS and dist-upgrade, very simple process with like 2 commands
[13:22]
toddfup_the_irons: my 'request multiple vps' thing was in rsponse to your 'what can ya do?' and my response involves more business for you *grin* [13:23]
up_the_ironstoddf: :) [13:23]
jdoeup_the_irons: if you're tracking every version, then yeah it's every 6 months, most of the time.
XX.04 and XX.10
[13:23]
up_the_ironsroger [13:26]
........ (idle for 39mn)
time for kvr05 to admit failure and reboot
up 968 days, 13:33, 5 users, load average: 11.82, 12.28, 20.87
of *course* i can't reach 1000, ever...
the load was getting better, but now it is back to worse
[14:05]
***beandog has joined #arpnetworks [14:08]
beandog has quit IRC (Quit: Leaving) [14:17]
plettup_the_irons: re your comments earlier about protecting one guest from another on KVM, cgroups might be what you're after [14:17]
jdoedidn't know those worked for io... wonder how well they work. [14:24]
andoljdoe: Well, if we all start doing really heavy i/o usage, perhaps up_the_irons will find out for us? :-) [14:27]
***Webhostbudd_ has joined #arpnetworks
Webhostbudd has quit IRC (Read error: Operation timed out)
heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer
jbum has joined #arpnetworks
[14:33]
up_the_ironsplett: i'll check it out, tnx [14:50]
....... (idle for 32mn)
***k3asd has quit IRC (Ping timeout: 265 seconds) [15:22]
.... (idle for 19mn)
RandalSchwartz has joined #arpnetworks
RandalSchwartz has quit IRC (Changing host)
RandalSchwartz has joined #arpnetworks
[15:41]
RandalSchwartzup_the_irons - there's no PST until november. :(
"at approximately 08/27/2012 14:30 PST" - not possible
[15:41]
***arenlor has joined #arpnetworks [15:41]
RandalSchwartzbut at least my system is working again. :) [15:42]
arenlorDo we know the cause?
Also, my VPS seems to be working fine.
[15:42]
RandalSchwartzit was just kvr05 acting up
900+ uptime days
[15:54]
arenlorRandalSchwartz: I'm on kvr05 [15:55]
..... (idle for 21mn)
***Ehtyar has joined #arpnetworks [16:16]
jdoearenlor: note past tense [16:22]
arenlorjdoe: So the answer is no that we don't know what the cause was. [16:22]
***henderb_ has quit IRC (Quit: Changing server)
henderb has joined #arpnetworks
[16:24]
up_the_ironsRandalSchwartz: bah, i always end up screwing the date in some way
RandalSchwartz: so, no further issues now? (i don't see any)
arenlor: cause is unknown, but i simply suspect after 960+ days of uptime, something just got kinked. I've analyzed enough logs and i'm writing the resolution as "Power cycling the server fixed the high I/O wait issue" and leaving it at that. I'll take 960+ days of uptime and move on... :)
[16:29]
***sako has quit IRC (Ping timeout: 268 seconds) [16:35]
jdoeup_the_irons: quitters never win and winners never quit... [16:36]
milkii usually win more sleep if i quit early
:o
[16:47]
RandalSchwartzyeah - seems fine now [16:58]
........ (idle for 39mn)
***arenlor has quit IRC (Read error: Connection reset by peer)
arenlor has joined #arpnetworks
[17:37]
up_the_ironsRandalSchwartz: cool [17:40]
..... (idle for 20mn)
***sako has joined #arpnetworks [18:00]
up_the_ironssako: so looks like you didn't make it to la dev ops? ;)
up_the_irons wanders off
[18:03]
sakostop spying on me
lol
[18:04]
***jbum has quit IRC (Quit: jbum) [18:08]
pjsup_the_irons, turns out we have a few friends in common, sako being one
and Lars
[18:21]
.... (idle for 18mn)
***sako has quit IRC (Ping timeout: 244 seconds) [18:39]
sako has joined #arpnetworks [18:45]
................. (idle for 1h21mn)
himuraken has quit IRC (Ping timeout: 246 seconds) [20:06]
himuraken has joined #arpnetworks [20:18]
sako has quit IRC (Ping timeout: 260 seconds) [20:31]
............. (idle for 1h2mn)
up_the_ironspjs: you know sako and lars? hah, nice
pjs: do u go to LADevOps at all?
[21:33]
.... (idle for 16mn)
perhaps pjs is there now... [21:49]
........ (idle for 38mn)
up_the_irons looks around
we had 75 nicks in here earlier, a record!
[22:27]
***sako has joined #arpnetworks [22:27]
up_the_ironsup_the_irons spies on sako [22:27]
***Webhostbudd has joined #arpnetworks
Webhostbudd_ has quit IRC (Ping timeout: 250 seconds)
[22:29]
Webhostbuddup_the_irons: still do =) [22:37]
up_the_ironsWebhostbudd: still do wut? :)
oh, 75 nicks!
[22:37]
CaZeHmm.
How many VPSes?
[22:38]
arenlorCaZe: I have two, what about you? [22:41]
CaZeInfinity. [22:43]
arenlorNice trick, gave an interger overflow on the billing platform I'm sure. [22:43]
***Webhostbudd has quit IRC (Ping timeout: 250 seconds)
Webhostbudd has joined #arpnetworks
Webhostbudd_ has joined #arpnetworks
Webhostbudd has quit IRC (Ping timeout: 250 seconds)
[22:51]
Aerosonic has joined #arpnetworks [23:10]
up_the_ironsholy crap, 76 nicks! [23:12]
Aerosonicup_the_irons: Yeah, so I decided to check out ARP Networks. [23:14]
up_the_ironsAerosonic: ah, your nick looked new
:)
although it sounds oddly familiar...
[23:22]
AerosonicI used to be on Linode?
Maybe you know me from their old support chan
[23:24]
***sako has quit IRC (Ping timeout: 256 seconds) [23:29]
up_the_ironsAerosonic: hmm.. i was never on their old support chan; i used to be in the slicehost one... oh well, doesn't matter )
:)
[23:32]
.... (idle for 15mn)
***sorressean has quit IRC (Read error: Operation timed out) [23:47]
sorressean has joined #arpnetworks [23:59]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)