[00:08] *** gcw|mbpro has quit IRC (Remote host closed the connection) [00:12] *** gcw|mbpro has joined #arpnetworks [00:19] *** Webhostbudd has quit IRC (Quit: Leaving) [00:34] *** Ehtyar has quit IRC (Quit: Going!) [02:02] *** `ariel has quit IRC (Ping timeout: 245 seconds) [02:03] *** ariel has joined #arpnetworks [08:15] *** RandalSchwartz has joined #arpnetworks [08:15] My disk (on kvr05) is acting horribly slow. [08:16] is there a system status update that might be relevant? [08:16] up_the_irons? [08:26] *** himuraken has joined #arpnetworks [08:36] *** RandalSchwartz has quit IRC (Ping timeout: 246 seconds) [09:18] *** ryk has joined #arpnetworks [09:19] *** ryk has quit IRC (Changing host) [09:19] *** ryk has joined #arpnetworks [09:27] *** Webhostbudd has joined #arpnetworks [09:29] *** RandalSchwartz has joined #arpnetworks [09:29] *** RandalSchwartz has quit IRC (Changing host) [09:29] *** RandalSchwartz has joined #arpnetworks [09:43] *** k3asd` has quit IRC (Ping timeout: 246 seconds) [09:56] *** k3asd` has joined #arpnetworks [10:06] *** sako has joined #arpnetworks [10:22] *** hive-mind has quit IRC (Remote host closed the connection) [10:24] *** hive-mind has joined #arpnetworks [11:01] *** k3asd` has quit IRC (Ping timeout: 246 seconds) [11:14] *** k3asd` has joined #arpnetworks [11:39] *** k3asd has joined #arpnetworks [11:41] *** k3asd` has quit IRC (Ping timeout: 248 seconds) [12:11] *** nestea has quit IRC (Ping timeout: 260 seconds) [12:12] *** nestea has joined #arpnetworks [12:19] *** RandalSchwartz has quit IRC (Ping timeout: 246 seconds) [12:19] *** mtve has quit IRC (Ping timeout: 244 seconds) [12:20] *** mtve has joined #arpnetworks [12:24] *** RandalSchwartz has joined #arpnetworks [12:24] *** RandalSchwartz has quit IRC (Changing host) [12:24] *** RandalSchwartz has joined #arpnetworks [12:24] up_the_irons - any idea about my slow disk? [12:24] been crappy all morning [12:25] do I have a noisy neighbor again? [12:38] DAMN... this needs fixing. [12:38] 1048576 bytes transferred in 22.494234 secs (46615 bytes/sec) [12:38] it took 22 seconds to write A MEGABYTE [12:38] up_the_irons ding ding ding [12:40] RandalSchwartz: yeah, i've been having intermittent problems with the host kvr05 [12:40] can you migrate me to a different box then? [12:41] it's interfering with business and tasks. [12:41] mine's fairly slow too. [12:41] 52428800 bytes (52 MB) copied, 71.7589 s, 731 kB/s [12:41] :/ [12:45] RandalSchwartz: the first priority is to fix the issue with kvr05; i think it is a noisy neighbor, as you said [12:45] *** RandalSchwartz has quit IRC (Ping timeout: 246 seconds) [12:49] jdoe: are you able to reach your vps at all? (i assume perhaps so, b/c you could get those test results) [12:54] up_the_irons: yeah. [12:54] jdoe: and it is on kvr05? [12:55] not sure offhand. [12:55] how would I tell? [12:55] jdoe: it's the same as your vnc host (listed in portal under vm details) [12:55] you can tell how often I log in ;) [12:55] sec. [12:56] :) [12:57] mm i'm on KVR05 and is sloww, my monitoring software is detecting the VM as down/up/down/up [12:57] i can't ssh into kvr05. serial console is responsive, however, it also won't let me in past me typing in my login name. i think the disk is completely locked up [12:57] kvr10 is doing fine. :D [12:57] ariel: although it is interesting that you can still get in, even though i can't get into the host. [12:58] up_the_irons: lol... I have at least two portal accounts, neither showing my vps. [12:59] nope, I'm on kvr06 [13:00] jdoe: roger [13:03] seems better now, 6-ish M/s. [13:05] jdoe: i'm not touching kvr06 ;) [13:06] but 6M/s is still pretty slow [13:06] I know, I'm just saying whatever it was, it's improved slightly. And no argument ;) [13:06] kvr05, i'm in! [13:06] jdoe: roger :) [13:06] now to find out who is hogging the disk [13:07] prolly that schwartz guy. [13:07] * jdoe NODS SAGELY. [13:08] randal actually _does_ use quite a bit of I/O, but not enough to take down the box.. ;) [13:08] I'm surprised any guest can make it *that* unresponsive. [13:09] maybe someone is swapping? [13:11] up_the_irons: sorry for the delay on centos. I had to board up house and horses for the storm. I'll be unboarding and such tonight and tomorrow.. then can get back to work on it. [13:15] jdoe: well, kvm/qemu doesn't really have too much in the way of I/O isolation. Like, each VM is just a Linux process and if it wants to really load up the disk, the scheduler isn't gonna stop it [13:15] jpalmer: oh problem at all, take your time. I'm actually going to make the 5.8 version :) [13:16] I saw a presentation from Verio a few years back where they had talked about all the custom work they had done to give a level of control over disk I/O, memory and CPU usage to their VPS product [13:16] they claimed they were going to release that back to the FreeBSD project, but never did :/ [13:17] up_the_irons: linux does kinda suck for that. [13:17] I have come up with a new way of dealing with disk provisioning for Linux guests that will afford me a *much* easier time supporting more distros; therefore, I'm pumped to make CentOS 5.8 templates, then Ubuntu 12.04 after that, then who knows... Fedora, Gentoo, Arch, ...? you name it [13:18] LFS PLZ. [13:18] twobithacker: jdoe yeah, i've seen several proprietary solutions talked about for that stuff, but nothing out in the open (but i admit, i haven't looked into it in a while) [13:18] as the list grows longer so does the amount of effort to maintain such a list. I know it won't effect our bottom line but still... ;-) [13:18] doesn't ionice work? [13:19] up_the_irons: yeah I dunno, it doesn't seem like a problem anyone is anxious to solve, despite the work that's gone into cfq and deadline. I dunno why. I can load the shit out of the disk on fbsd, and the machine is still responsive. [13:19] try that on linux and it craters. [13:19] toddf: well, that's the thing, up until now the effort to maintain such a list has been prohibitive; but, with my new disk provisioning strategy, no longer :) [13:20] I'm not in on all the details, but with a new disk provisioning strategy, doesn't that still mean someone has to manually test install a distribution for it to work? [13:20] jdoe: i think it is a problem that people have accepted; kinda like DoS attacks, they suck but what can ya do? ;) [13:21] toddf: yes, but only once, then the template is copied for every VM down the line. new distro versions don't come out _that_ often; openbsd is actually the most frequent publisher, where I find myself having to update my templates every 6 months ;) [13:21] up_the_irons: request multiple vps's across different kvr* hosts like randalschwartz, and make it a cluster that will lessen the effect of one kvr* with temporary io starvation ;_) [13:22] eh, ubuntu is on a 6 month cycle too isn't it? [13:22] toddf: wait wut?;) [13:22] jdoe: is it? i haven't even noticed... for ubuntu i tend to do only the LTS versions [13:23] if you want non-LTS, a ubuntu fan can just get an LTS and dist-upgrade, very simple process with like 2 commands [13:23] up_the_irons: my 'request multiple vps' thing was in rsponse to your 'what can ya do?' and my response involves more business for you *grin* [13:23] toddf: :) [13:23] up_the_irons: if you're tracking every version, then yeah it's every 6 months, most of the time. [13:24] XX.04 and XX.10 [13:26] roger [14:05] time for kvr05 to admit failure and reboot [14:05] up 968 days, 13:33, 5 users, load average: 11.82, 12.28, 20.87 [14:05] of *course* i can't reach 1000, ever... [14:05] the load was getting better, but now it is back to worse [14:08] *** beandog has joined #arpnetworks [14:17] *** beandog has quit IRC (Quit: Leaving) [14:17] up_the_irons: re your comments earlier about protecting one guest from another on KVM, cgroups might be what you're after [14:24] didn't know those worked for io... wonder how well they work. [14:27] jdoe: Well, if we all start doing really heavy i/o usage, perhaps up_the_irons will find out for us? :-) [14:33] *** Webhostbudd_ has joined #arpnetworks [14:36] *** Webhostbudd has quit IRC (Read error: Operation timed out) [14:36] *** heavysixer has joined #arpnetworks [14:36] *** ChanServ sets mode: +o heavysixer [14:37] *** jbum has joined #arpnetworks [14:50] plett: i'll check it out, tnx [15:22] *** k3asd has quit IRC (Ping timeout: 265 seconds) [15:41] *** RandalSchwartz has joined #arpnetworks [15:41] *** RandalSchwartz has quit IRC (Changing host) [15:41] *** RandalSchwartz has joined #arpnetworks [15:41] up_the_irons - there's no PST until november. :( [15:41] "at approximately 08/27/2012 14:30 PST" - not possible [15:41] *** arenlor has joined #arpnetworks [15:42] but at least my system is working again. :) [15:42] Do we know the cause? [15:45] Also, my VPS seems to be working fine. [15:54] it was just kvr05 acting up [15:54] 900+ uptime days [15:55] RandalSchwartz: I'm on kvr05 [16:16] *** Ehtyar has joined #arpnetworks [16:22] arenlor: note past tense [16:22] jdoe: So the answer is no that we don't know what the cause was. [16:24] *** henderb_ has quit IRC (Quit: Changing server) [16:25] *** henderb has joined #arpnetworks [16:29] RandalSchwartz: bah, i always end up screwing the date in some way [16:30] RandalSchwartz: so, no further issues now? (i don't see any) [16:31] arenlor: cause is unknown, but i simply suspect after 960+ days of uptime, something just got kinked. I've analyzed enough logs and i'm writing the resolution as "Power cycling the server fixed the high I/O wait issue" and leaving it at that. I'll take 960+ days of uptime and move on... :) [16:35] *** sako has quit IRC (Ping timeout: 268 seconds) [16:36] up_the_irons: quitters never win and winners never quit... [16:47] i usually win more sleep if i quit early [16:47] :o [16:58] yeah - seems fine now [17:37] *** arenlor has quit IRC (Read error: Connection reset by peer) [17:37] *** arenlor has joined #arpnetworks [17:40] RandalSchwartz: cool [18:00] *** sako has joined #arpnetworks [18:03] sako: so looks like you didn't make it to la dev ops? ;) [18:03] * up_the_irons wanders off [18:04] stop spying on me [18:04] lol [18:08] *** jbum has quit IRC (Quit: jbum) [18:21] up_the_irons, turns out we have a few friends in common, sako being one [18:21] and Lars [18:39] *** sako has quit IRC (Ping timeout: 244 seconds) [18:45] *** sako has joined #arpnetworks [20:06] *** himuraken has quit IRC (Ping timeout: 246 seconds) [20:18] *** himuraken has joined #arpnetworks [20:31] *** sako has quit IRC (Ping timeout: 260 seconds) [21:33] pjs: you know sako and lars? hah, nice [21:33] pjs: do u go to LADevOps at all? [21:49] perhaps pjs is there now... [22:27] * up_the_irons looks around [22:27] we had 75 nicks in here earlier, a record! [22:27] *** sako has joined #arpnetworks [22:27] * up_the_irons spies on sako [22:29] *** Webhostbudd has joined #arpnetworks [22:31] *** Webhostbudd_ has quit IRC (Ping timeout: 250 seconds) [22:37] up_the_irons: still do =) [22:37] Webhostbudd: still do wut? :) [22:38] oh, 75 nicks! [22:38] Hmm. [22:38] How many VPSes? [22:41] CaZe: I have two, what about you? [22:43] Infinity. [22:43] Nice trick, gave an interger overflow on the billing platform I'm sure. [22:51] *** Webhostbudd has quit IRC (Ping timeout: 250 seconds) [22:54] *** Webhostbudd has joined #arpnetworks [22:55] *** Webhostbudd_ has joined #arpnetworks [22:58] *** Webhostbudd has quit IRC (Ping timeout: 250 seconds) [23:10] *** Aerosonic has joined #arpnetworks [23:12] holy crap, 76 nicks! [23:14] up_the_irons: Yeah, so I decided to check out ARP Networks. [23:22] Aerosonic: ah, your nick looked new [23:22] :) [23:22] although it sounds oddly familiar... [23:24] I used to be on Linode? [23:24] Maybe you know me from their old support chan [23:29] *** sako has quit IRC (Ping timeout: 256 seconds) [23:32] Aerosonic: hmm.. i was never on their old support chan; i used to be in the slicehost one... oh well, doesn't matter ) [23:32] :) [23:47] *** sorressean has quit IRC (Read error: Operation timed out) [23:59] *** sorressean has joined #arpnetworks