↑back Search ←Prev date Next date→ Show only urls | (Click on time to select a line by its url) |
Who | What | When |
---|---|---|
*** | gcw|mbpro has quit IRC (Remote host closed the connection)
gcw|mbpro has joined #arpnetworks | [00:08] |
Webhostbudd has quit IRC (Quit: Leaving) | [00:19] | |
.... (idle for 15mn) | ||
Ehtyar has quit IRC (Quit: Going!) | [00:34] | |
.................. (idle for 1h28mn) | ||
`ariel has quit IRC (Ping timeout: 245 seconds)
ariel has joined #arpnetworks | [02:02] | |
........................................................................... (idle for 6h12mn) | ||
RandalSchwartz has joined #arpnetworks | [08:15] | |
RandalSchwartz | My disk (on kvr05) is acting horribly slow.
is there a system status update that might be relevant? up_the_irons? | [08:15] |
*** | himuraken has joined #arpnetworks | [08:26] |
RandalSchwartz has quit IRC (Ping timeout: 246 seconds) | [08:36] | |
......... (idle for 42mn) | ||
ryk has joined #arpnetworks
ryk has quit IRC (Changing host) ryk has joined #arpnetworks | [09:18] | |
Webhostbudd has joined #arpnetworks
RandalSchwartz has joined #arpnetworks RandalSchwartz has quit IRC (Changing host) RandalSchwartz has joined #arpnetworks | [09:27] | |
k3asd` has quit IRC (Ping timeout: 246 seconds) | [09:43] | |
k3asd` has joined #arpnetworks | [09:56] | |
sako has joined #arpnetworks | [10:06] | |
.... (idle for 16mn) | ||
hive-mind has quit IRC (Remote host closed the connection)
hive-mind has joined #arpnetworks | [10:22] | |
........ (idle for 37mn) | ||
k3asd` has quit IRC (Ping timeout: 246 seconds) | [11:01] | |
k3asd` has joined #arpnetworks | [11:14] | |
...... (idle for 25mn) | ||
k3asd has joined #arpnetworks
k3asd` has quit IRC (Ping timeout: 248 seconds) | [11:39] | |
....... (idle for 30mn) | ||
nestea has quit IRC (Ping timeout: 260 seconds)
nestea has joined #arpnetworks | [12:11] | |
RandalSchwartz has quit IRC (Ping timeout: 246 seconds)
mtve has quit IRC (Ping timeout: 244 seconds) mtve has joined #arpnetworks RandalSchwartz has joined #arpnetworks RandalSchwartz has quit IRC (Changing host) RandalSchwartz has joined #arpnetworks | [12:19] | |
RandalSchwartz | up_the_irons - any idea about my slow disk?
been crappy all morning do I have a noisy neighbor again? | [12:24] |
DAMN... this needs fixing.
1048576 bytes transferred in 22.494234 secs (46615 bytes/sec) it took 22 seconds to write A MEGABYTE up_the_irons ding ding ding | [12:38] | |
up_the_irons | RandalSchwartz: yeah, i've been having intermittent problems with the host kvr05 | [12:40] |
RandalSchwartz | can you migrate me to a different box then?
it's interfering with business and tasks. | [12:40] |
jdoe | mine's fairly slow too.
52428800 bytes (52 MB) copied, 71.7589 s, 731 kB/s :/ | [12:41] |
up_the_irons | RandalSchwartz: the first priority is to fix the issue with kvr05; i think it is a noisy neighbor, as you said | [12:45] |
*** | RandalSchwartz has quit IRC (Ping timeout: 246 seconds) | [12:45] |
up_the_irons | jdoe: are you able to reach your vps at all? (i assume perhaps so, b/c you could get those test results) | [12:49] |
jdoe | up_the_irons: yeah. | [12:54] |
up_the_irons | jdoe: and it is on kvr05? | [12:54] |
jdoe | not sure offhand.
how would I tell? | [12:55] |
up_the_irons | jdoe: it's the same as your vnc host (listed in portal under vm details) | [12:55] |
jdoe | you can tell how often I log in ;)
sec. | [12:55] |
up_the_irons | :) | [12:56] |
ariel | mm i'm on KVR05 and is sloww, my monitoring software is detecting the VM as down/up/down/up | [12:57] |
up_the_irons | i can't ssh into kvr05. serial console is responsive, however, it also won't let me in past me typing in my login name. i think the disk is completely locked up | [12:57] |
CaZe | kvr10 is doing fine. :D | [12:57] |
up_the_irons | ariel: although it is interesting that you can still get in, even though i can't get into the host. | [12:57] |
jdoe | up_the_irons: lol... I have at least two portal accounts, neither showing my vps.
nope, I'm on kvr06 | [12:58] |
up_the_irons | jdoe: roger | [13:00] |
jdoe | seems better now, 6-ish M/s. | [13:03] |
up_the_irons | jdoe: i'm not touching kvr06 ;)
but 6M/s is still pretty slow | [13:05] |
jdoe | I know, I'm just saying whatever it was, it's improved slightly. And no argument ;) | [13:06] |
up_the_irons | kvr05, i'm in!
jdoe: roger :) now to find out who is hogging the disk | [13:06] |
jdoe | prolly that schwartz guy.
jdoe NODS SAGELY. | [13:07] |
up_the_irons | randal actually _does_ use quite a bit of I/O, but not enough to take down the box.. ;) | [13:08] |
jdoe | I'm surprised any guest can make it *that* unresponsive. | [13:08] |
toddf | maybe someone is swapping? | [13:09] |
jpalmer | up_the_irons: sorry for the delay on centos. I had to board up house and horses for the storm. I'll be unboarding and such tonight and tomorrow.. then can get back to work on it. | [13:11] |
up_the_irons | jdoe: well, kvm/qemu doesn't really have too much in the way of I/O isolation. Like, each VM is just a Linux process and if it wants to really load up the disk, the scheduler isn't gonna stop it
jpalmer: oh problem at all, take your time. I'm actually going to make the 5.8 version :) | [13:15] |
twobithacker | I saw a presentation from Verio a few years back where they had talked about all the custom work they had done to give a level of control over disk I/O, memory and CPU usage to their VPS product
they claimed they were going to release that back to the FreeBSD project, but never did :/ | [13:16] |
jdoe | up_the_irons: linux does kinda suck for that. | [13:17] |
up_the_irons | I have come up with a new way of dealing with disk provisioning for Linux guests that will afford me a *much* easier time supporting more distros; therefore, I'm pumped to make CentOS 5.8 templates, then Ubuntu 12.04 after that, then who knows... Fedora, Gentoo, Arch, ...? you name it | [13:17] |
jdoe | LFS PLZ. | [13:18] |
up_the_irons | twobithacker: jdoe yeah, i've seen several proprietary solutions talked about for that stuff, but nothing out in the open (but i admit, i haven't looked into it in a while) | [13:18] |
toddf | as the list grows longer so does the amount of effort to maintain such a list. I know it won't effect our bottom line but still... ;-)
<sarcasm> doesn't ionice work? </sarcasm> | [13:18] |
jdoe | up_the_irons: yeah I dunno, it doesn't seem like a problem anyone is anxious to solve, despite the work that's gone into cfq and deadline. I dunno why. I can load the shit out of the disk on fbsd, and the machine is still responsive.
try that on linux and it craters. | [13:19] |
up_the_irons | toddf: well, that's the thing, up until now the effort to maintain such a list has been prohibitive; but, with my new disk provisioning strategy, no longer :) | [13:19] |
toddf | I'm not in on all the details, but with a new disk provisioning strategy, doesn't that still mean someone has to manually test install a distribution for it to work? | [13:20] |
up_the_irons | jdoe: i think it is a problem that people have accepted; kinda like DoS attacks, they suck but what can ya do? ;)
toddf: yes, but only once, then the template is copied for every VM down the line. new distro versions don't come out _that_ often; openbsd is actually the most frequent publisher, where I find myself having to update my templates every 6 months ;) | [13:20] |
toddf | up_the_irons: request multiple vps's across different kvr* hosts like randalschwartz, and make it a cluster that will lessen the effect of one kvr* with temporary io starvation ;_) | [13:21] |
jdoe | eh, ubuntu is on a 6 month cycle too isn't it? | [13:22] |
up_the_irons | toddf: wait wut?;)
jdoe: is it? i haven't even noticed... for ubuntu i tend to do only the LTS versions if you want non-LTS, a ubuntu fan can just get an LTS and dist-upgrade, very simple process with like 2 commands | [13:22] |
toddf | up_the_irons: my 'request multiple vps' thing was in rsponse to your 'what can ya do?' and my response involves more business for you *grin* | [13:23] |
up_the_irons | toddf: :) | [13:23] |
jdoe | up_the_irons: if you're tracking every version, then yeah it's every 6 months, most of the time.
XX.04 and XX.10 | [13:23] |
up_the_irons | roger | [13:26] |
........ (idle for 39mn) | ||
time for kvr05 to admit failure and reboot
up 968 days, 13:33, 5 users, load average: 11.82, 12.28, 20.87 of *course* i can't reach 1000, ever... the load was getting better, but now it is back to worse | [14:05] | |
*** | beandog has joined #arpnetworks | [14:08] |
beandog has quit IRC (Quit: Leaving) | [14:17] | |
plett | up_the_irons: re your comments earlier about protecting one guest from another on KVM, cgroups might be what you're after | [14:17] |
jdoe | didn't know those worked for io... wonder how well they work. | [14:24] |
andol | jdoe: Well, if we all start doing really heavy i/o usage, perhaps up_the_irons will find out for us? :-) | [14:27] |
*** | Webhostbudd_ has joined #arpnetworks
Webhostbudd has quit IRC (Read error: Operation timed out) heavysixer has joined #arpnetworks ChanServ sets mode: +o heavysixer jbum has joined #arpnetworks | [14:33] |
up_the_irons | plett: i'll check it out, tnx | [14:50] |
....... (idle for 32mn) | ||
*** | k3asd has quit IRC (Ping timeout: 265 seconds) | [15:22] |
.... (idle for 19mn) | ||
RandalSchwartz has joined #arpnetworks
RandalSchwartz has quit IRC (Changing host) RandalSchwartz has joined #arpnetworks | [15:41] | |
RandalSchwartz | up_the_irons - there's no PST until november. :(
"at approximately 08/27/2012 14:30 PST" - not possible | [15:41] |
*** | arenlor has joined #arpnetworks | [15:41] |
RandalSchwartz | but at least my system is working again. :) | [15:42] |
arenlor | Do we know the cause?
Also, my VPS seems to be working fine. | [15:42] |
RandalSchwartz | it was just kvr05 acting up
900+ uptime days | [15:54] |
arenlor | RandalSchwartz: I'm on kvr05 | [15:55] |
..... (idle for 21mn) | ||
*** | Ehtyar has joined #arpnetworks | [16:16] |
jdoe | arenlor: note past tense | [16:22] |
arenlor | jdoe: So the answer is no that we don't know what the cause was. | [16:22] |
*** | henderb_ has quit IRC (Quit: Changing server)
henderb has joined #arpnetworks | [16:24] |
up_the_irons | RandalSchwartz: bah, i always end up screwing the date in some way
RandalSchwartz: so, no further issues now? (i don't see any) arenlor: cause is unknown, but i simply suspect after 960+ days of uptime, something just got kinked. I've analyzed enough logs and i'm writing the resolution as "Power cycling the server fixed the high I/O wait issue" and leaving it at that. I'll take 960+ days of uptime and move on... :) | [16:29] |
*** | sako has quit IRC (Ping timeout: 268 seconds) | [16:35] |
jdoe | up_the_irons: quitters never win and winners never quit... | [16:36] |
milki | i usually win more sleep if i quit early
:o | [16:47] |
RandalSchwartz | yeah - seems fine now | [16:58] |
........ (idle for 39mn) | ||
*** | arenlor has quit IRC (Read error: Connection reset by peer)
arenlor has joined #arpnetworks | [17:37] |
up_the_irons | RandalSchwartz: cool | [17:40] |
..... (idle for 20mn) | ||
*** | sako has joined #arpnetworks | [18:00] |
up_the_irons | sako: so looks like you didn't make it to la dev ops? ;)
up_the_irons wanders off | [18:03] |
sako | stop spying on me
lol | [18:04] |
*** | jbum has quit IRC (Quit: jbum) | [18:08] |
pjs | up_the_irons, turns out we have a few friends in common, sako being one
and Lars | [18:21] |
.... (idle for 18mn) | ||
*** | sako has quit IRC (Ping timeout: 244 seconds) | [18:39] |
sako has joined #arpnetworks | [18:45] | |
................. (idle for 1h21mn) | ||
himuraken has quit IRC (Ping timeout: 246 seconds) | [20:06] | |
himuraken has joined #arpnetworks | [20:18] | |
sako has quit IRC (Ping timeout: 260 seconds) | [20:31] | |
............. (idle for 1h2mn) | ||
up_the_irons | pjs: you know sako and lars? hah, nice
pjs: do u go to LADevOps at all? | [21:33] |
.... (idle for 16mn) | ||
perhaps pjs is there now... | [21:49] | |
........ (idle for 38mn) | ||
up_the_irons looks around
we had 75 nicks in here earlier, a record! | [22:27] | |
*** | sako has joined #arpnetworks | [22:27] |
up_the_irons | up_the_irons spies on sako | [22:27] |
*** | Webhostbudd has joined #arpnetworks
Webhostbudd_ has quit IRC (Ping timeout: 250 seconds) | [22:29] |
Webhostbudd | up_the_irons: still do =) | [22:37] |
up_the_irons | Webhostbudd: still do wut? :)
oh, 75 nicks! | [22:37] |
CaZe | Hmm.
How many VPSes? | [22:38] |
arenlor | CaZe: I have two, what about you? | [22:41] |
CaZe | Infinity. | [22:43] |
arenlor | Nice trick, gave an interger overflow on the billing platform I'm sure. | [22:43] |
*** | Webhostbudd has quit IRC (Ping timeout: 250 seconds)
Webhostbudd has joined #arpnetworks Webhostbudd_ has joined #arpnetworks Webhostbudd has quit IRC (Ping timeout: 250 seconds) | [22:51] |
Aerosonic has joined #arpnetworks | [23:10] | |
up_the_irons | holy crap, 76 nicks! | [23:12] |
Aerosonic | up_the_irons: Yeah, so I decided to check out ARP Networks. | [23:14] |
up_the_irons | Aerosonic: ah, your nick looked new
:) although it sounds oddly familiar... | [23:22] |
Aerosonic | I used to be on Linode?
Maybe you know me from their old support chan | [23:24] |
*** | sako has quit IRC (Ping timeout: 256 seconds) | [23:29] |
up_the_irons | Aerosonic: hmm.. i was never on their old support chan; i used to be in the slicehost one... oh well, doesn't matter )
:) | [23:32] |
.... (idle for 15mn) | ||
*** | sorressean has quit IRC (Read error: Operation timed out) | [23:47] |
sorressean has joined #arpnetworks | [23:59] |
↑back Search ←Prev date Next date→ Show only urls | (Click on time to select a line by its url) |