[00:08] *** gcw|mbpro has quit IRC (Remote host closed the connection)
[00:12] *** gcw|mbpro has joined #arpnetworks
[00:19] *** Webhostbudd has quit IRC (Quit: Leaving)
[00:34] *** Ehtyar has quit IRC (Quit: Going!)
[02:02] *** `ariel has quit IRC (Ping timeout: 245 seconds)
[02:03] *** ariel has joined #arpnetworks
[08:15] *** RandalSchwartz has joined #arpnetworks
[08:15] <RandalSchwartz> My disk (on kvr05) is acting horribly slow.
[08:16] <RandalSchwartz> is there a system status update that might be relevant?
[08:16] <RandalSchwartz> up_the_irons?
[08:26] *** himuraken has joined #arpnetworks
[08:36] *** RandalSchwartz has quit IRC (Ping timeout: 246 seconds)
[09:18] *** ryk has joined #arpnetworks
[09:19] *** ryk has quit IRC (Changing host)
[09:19] *** ryk has joined #arpnetworks
[09:27] *** Webhostbudd has joined #arpnetworks
[09:29] *** RandalSchwartz has joined #arpnetworks
[09:29] *** RandalSchwartz has quit IRC (Changing host)
[09:29] *** RandalSchwartz has joined #arpnetworks
[09:43] *** k3asd` has quit IRC (Ping timeout: 246 seconds)
[09:56] *** k3asd` has joined #arpnetworks
[10:06] *** sako has joined #arpnetworks
[10:22] *** hive-mind has quit IRC (Remote host closed the connection)
[10:24] *** hive-mind has joined #arpnetworks
[11:01] *** k3asd` has quit IRC (Ping timeout: 246 seconds)
[11:14] *** k3asd` has joined #arpnetworks
[11:39] *** k3asd has joined #arpnetworks
[11:41] *** k3asd` has quit IRC (Ping timeout: 248 seconds)
[12:11] *** nestea has quit IRC (Ping timeout: 260 seconds)
[12:12] *** nestea has joined #arpnetworks
[12:19] *** RandalSchwartz has quit IRC (Ping timeout: 246 seconds)
[12:19] *** mtve has quit IRC (Ping timeout: 244 seconds)
[12:20] *** mtve has joined #arpnetworks
[12:24] *** RandalSchwartz has joined #arpnetworks
[12:24] *** RandalSchwartz has quit IRC (Changing host)
[12:24] *** RandalSchwartz has joined #arpnetworks
[12:24] <RandalSchwartz> up_the_irons - any idea about my slow disk?
[12:24] <RandalSchwartz> been crappy all morning
[12:25] <RandalSchwartz> do I have a noisy neighbor again?
[12:38] <RandalSchwartz> DAMN... this needs fixing.
[12:38] <RandalSchwartz> 1048576 bytes transferred in 22.494234 secs (46615 bytes/sec)
[12:38] <RandalSchwartz> it took 22 seconds to write A MEGABYTE
[12:38] <RandalSchwartz> up_the_irons ding ding ding
[12:40] <up_the_irons> RandalSchwartz: yeah, i've been having intermittent problems with the host kvr05
[12:40] <RandalSchwartz> can you migrate me to a different box then?
[12:41] <RandalSchwartz> it's interfering with business and tasks.
[12:41] <jdoe> mine's fairly slow too.
[12:41] <jdoe> 52428800 bytes (52 MB) copied, 71.7589 s, 731 kB/s
[12:41] <jdoe> :/
[12:45] <up_the_irons> RandalSchwartz: the first priority is to fix the issue with kvr05; i think it is a noisy neighbor, as you said
[12:45] *** RandalSchwartz has quit IRC (Ping timeout: 246 seconds)
[12:49] <up_the_irons> jdoe: are you able to reach your vps at all?  (i assume perhaps so, b/c you could get those test results)
[12:54] <jdoe> up_the_irons: yeah.
[12:54] <up_the_irons> jdoe: and it is on kvr05?
[12:55] <jdoe> not sure offhand.
[12:55] <jdoe> how would I tell?
[12:55] <up_the_irons> jdoe: it's the same as your vnc host (listed in portal under vm details)
[12:55] <jdoe> you can tell how often I log in ;)
[12:55] <jdoe> sec.
[12:56] <up_the_irons> :)
[12:57] <ariel> mm i'm on KVR05 and is sloww, my monitoring software is detecting the VM as down/up/down/up
[12:57] <up_the_irons> i can't ssh into kvr05.  serial console is responsive, however, it also won't let me in past me typing in my login name.  i think the disk is completely locked up
[12:57] <CaZe> kvr10 is doing fine. :D
[12:57] <up_the_irons> ariel: although it is interesting that you can still get in, even though i can't get into the host.
[12:58] <jdoe> up_the_irons: lol... I have at least two portal accounts, neither showing my vps.
[12:59] <jdoe> nope, I'm on kvr06
[13:00] <up_the_irons> jdoe: roger
[13:03] <jdoe> seems better now, 6-ish M/s.
[13:05] <up_the_irons> jdoe: i'm not touching kvr06 ;)
[13:06] <up_the_irons> but 6M/s is still pretty slow
[13:06] <jdoe> I know, I'm just saying whatever it was, it's improved slightly. And no argument ;)
[13:06] <up_the_irons> kvr05, i'm in!
[13:06] <up_the_irons> jdoe: roger :)
[13:06] <up_the_irons> now to find out who is hogging the disk
[13:07] <jdoe> prolly that schwartz guy.
[13:07] * jdoe NODS SAGELY.
[13:08] <up_the_irons> randal actually _does_ use quite a bit of I/O, but not enough to take down the box.. ;)
[13:08] <jdoe> I'm surprised any guest can make it *that* unresponsive.
[13:09] <toddf> maybe someone is swapping?
[13:11] <jpalmer> up_the_irons: sorry for the delay on centos.  I had to board up house and horses for the storm.  I'll be unboarding and such tonight and tomorrow.. then can get back to work on it.
[13:15] <up_the_irons> jdoe: well, kvm/qemu doesn't really have too much in the way of I/O isolation. Like, each VM is just a Linux process and if it wants to really load up the disk, the scheduler isn't gonna stop it
[13:15] <up_the_irons> jpalmer: oh problem at all, take your time.  I'm actually going to make the 5.8 version :)
[13:16] <twobithacker> I saw a presentation from Verio a few years back where they had talked about all the custom work they had done to give a level of control over disk I/O, memory and CPU usage to their VPS product
[13:16] <twobithacker> they claimed they were going to release that back to the FreeBSD project, but never did :/
[13:17] <jdoe> up_the_irons: linux does kinda suck for that.
[13:17] <up_the_irons> I have come up with a new way of dealing with disk provisioning for Linux guests that will afford me a *much* easier time supporting more distros; therefore, I'm pumped to make CentOS 5.8 templates, then Ubuntu 12.04 after that, then who knows... Fedora, Gentoo, Arch, ...? you name it
[13:18] <jdoe> LFS PLZ.
[13:18] <up_the_irons> twobithacker: jdoe yeah, i've seen several proprietary solutions talked about for that stuff, but nothing out in the open (but i admit, i  haven't looked into it in a while)
[13:18] <toddf> as the list grows longer so does the amount of effort to maintain such a list. I know it won't effect our bottom line but still... ;-)
[13:18] <toddf> <sarcasm> doesn't ionice work? </sarcasm>
[13:19] <jdoe> up_the_irons: yeah I dunno, it doesn't seem like a problem anyone is anxious to solve, despite the work that's gone into cfq and deadline. I dunno why. I can load the shit out of the disk on fbsd, and the machine is still responsive.
[13:19] <jdoe> try that on linux and it craters.
[13:19] <up_the_irons> toddf: well, that's the thing, up until now the effort to maintain such a list has been prohibitive; but, with my new disk provisioning strategy, no longer :)
[13:20] <toddf> I'm not in on all the details, but with a new disk provisioning strategy, doesn't that still mean someone has to manually test install a distribution for it to work?
[13:20] <up_the_irons> jdoe: i think it is a problem that people have accepted; kinda like DoS attacks, they suck but what can ya do? ;)
[13:21] <up_the_irons> toddf: yes, but only once, then the template is copied for every VM down the line.  new distro versions don't come out _that_ often; openbsd is actually the most frequent publisher, where I find myself having to update my templates every 6 months ;)
[13:21] <toddf> up_the_irons: request multiple vps's across different kvr* hosts like randalschwartz, and make it a cluster that will lessen the effect of one kvr* with temporary io starvation ;_)
[13:22] <jdoe> eh, ubuntu is on a 6 month cycle too isn't it?
[13:22] <up_the_irons> toddf: wait wut?;)
[13:22] <up_the_irons> jdoe: is it?  i haven't even noticed...   for ubuntu i tend to do only the LTS versions
[13:23] <up_the_irons> if you want non-LTS, a ubuntu fan can just get an LTS and dist-upgrade, very simple process with like 2 commands
[13:23] <toddf> up_the_irons: my 'request multiple vps' thing was in rsponse to your 'what can ya do?' and my response involves more business for you *grin*
[13:23] <up_the_irons> toddf: :)
[13:23] <jdoe> up_the_irons: if you're tracking every version, then yeah it's every 6 months, most of the time.
[13:24] <jdoe> XX.04 and XX.10
[13:26] <up_the_irons> roger
[14:05] <up_the_irons> time for kvr05 to admit failure and reboot
[14:05] <up_the_irons> up 968 days, 13:33,  5 users,  load average: 11.82, 12.28, 20.87
[14:05] <up_the_irons> of *course* i can't reach 1000, ever...
[14:05] <up_the_irons> the load was getting better, but now it is back to worse
[14:08] *** beandog has joined #arpnetworks
[14:17] *** beandog has quit IRC (Quit: Leaving)
[14:17] <plett> up_the_irons: re your comments earlier about protecting one guest from another on KVM, cgroups might be what you're after
[14:24] <jdoe> didn't know those worked for io... wonder how well they work.
[14:27] <andol> jdoe: Well, if we all start doing really heavy i/o usage, perhaps up_the_irons will find out for us? :-)
[14:33] *** Webhostbudd_ has joined #arpnetworks
[14:36] *** Webhostbudd has quit IRC (Read error: Operation timed out)
[14:36] *** heavysixer has joined #arpnetworks
[14:36] *** ChanServ sets mode: +o heavysixer
[14:37] *** jbum has joined #arpnetworks
[14:50] <up_the_irons> plett: i'll check it out, tnx
[15:22] *** k3asd has quit IRC (Ping timeout: 265 seconds)
[15:41] *** RandalSchwartz has joined #arpnetworks
[15:41] *** RandalSchwartz has quit IRC (Changing host)
[15:41] *** RandalSchwartz has joined #arpnetworks
[15:41] <RandalSchwartz> up_the_irons - there's no PST until november. :(
[15:41] <RandalSchwartz> "at approximately 08/27/2012 14:30 PST" - not possible
[15:41] *** arenlor has joined #arpnetworks
[15:42] <RandalSchwartz> but at least my system is working again. :)
[15:42] <arenlor> Do we know the cause?
[15:45] <arenlor> Also, my VPS seems to be working fine.
[15:54] <RandalSchwartz> it was just kvr05 acting up
[15:54] <RandalSchwartz> 900+ uptime days
[15:55] <arenlor> RandalSchwartz: I'm on kvr05
[16:16] *** Ehtyar has joined #arpnetworks
[16:22] <jdoe> arenlor: note past tense
[16:22] <arenlor> jdoe: So the answer is no that we don't know what the cause was.
[16:24] *** henderb_ has quit IRC (Quit: Changing server)
[16:25] *** henderb has joined #arpnetworks
[16:29] <up_the_irons> RandalSchwartz: bah, i always end up screwing the date in some way
[16:30] <up_the_irons> RandalSchwartz: so, no further issues now?  (i don't see any)
[16:31] <up_the_irons> arenlor: cause is unknown, but i simply suspect after 960+ days of uptime, something just got kinked.  I've analyzed enough logs and i'm writing the resolution as "Power cycling the server fixed the high I/O wait issue" and leaving it at that.  I'll take 960+ days of uptime and move on... :)
[16:35] *** sako has quit IRC (Ping timeout: 268 seconds)
[16:36] <jdoe> up_the_irons: quitters never win and winners never quit...
[16:47] <milki> i usually win more sleep if i quit early
[16:47] <milki> :o
[16:58] <RandalSchwartz> yeah - seems fine now
[17:37] *** arenlor has quit IRC (Read error: Connection reset by peer)
[17:37] *** arenlor has joined #arpnetworks
[17:40] <up_the_irons> RandalSchwartz: cool
[18:00] *** sako has joined #arpnetworks
[18:03] <up_the_irons> sako: so looks like you didn't make it to la dev ops? ;)
[18:03] * up_the_irons wanders off
[18:04] <sako> stop spying on me
[18:04] <sako> lol
[18:08] *** jbum has quit IRC (Quit: jbum)
[18:21] <pjs> up_the_irons, turns out we have a few friends in common, sako being one
[18:21] <pjs> and Lars
[18:39] *** sako has quit IRC (Ping timeout: 244 seconds)
[18:45] *** sako has joined #arpnetworks
[20:06] *** himuraken has quit IRC (Ping timeout: 246 seconds)
[20:18] *** himuraken has joined #arpnetworks
[20:31] *** sako has quit IRC (Ping timeout: 260 seconds)
[21:33] <up_the_irons> pjs: you know sako and lars?  hah, nice
[21:33] <up_the_irons> pjs: do u go to LADevOps at all?
[21:49] <up_the_irons> perhaps pjs is there now...
[22:27] * up_the_irons looks around
[22:27] <up_the_irons> we had 75 nicks in here earlier, a record!
[22:27] *** sako has joined #arpnetworks
[22:27] * up_the_irons spies on sako
[22:29] *** Webhostbudd has joined #arpnetworks
[22:31] *** Webhostbudd_ has quit IRC (Ping timeout: 250 seconds)
[22:37] <Webhostbudd> up_the_irons: still do =)
[22:37] <up_the_irons> Webhostbudd: still do wut? :)
[22:38] <up_the_irons> oh, 75 nicks!
[22:38] <CaZe> Hmm.
[22:38] <CaZe> How many VPSes?
[22:41] <arenlor> CaZe: I have two, what about you?
[22:43] <CaZe> Infinity.
[22:43] <arenlor> Nice trick, gave an interger overflow on the billing platform I'm sure.
[22:51] *** Webhostbudd has quit IRC (Ping timeout: 250 seconds)
[22:54] *** Webhostbudd has joined #arpnetworks
[22:55] *** Webhostbudd_ has joined #arpnetworks
[22:58] *** Webhostbudd has quit IRC (Ping timeout: 250 seconds)
[23:10] *** Aerosonic has joined #arpnetworks
[23:12] <up_the_irons> holy crap, 76 nicks!
[23:14] <Aerosonic> up_the_irons: Yeah, so I decided to check out ARP Networks.
[23:22] <up_the_irons> Aerosonic: ah, your nick looked new
[23:22] <up_the_irons> :)
[23:22] <up_the_irons> although it sounds oddly familiar...
[23:24] <Aerosonic> I used to be on Linode?
[23:24] <Aerosonic> Maybe you know me from their old support chan
[23:29] *** sako has quit IRC (Ping timeout: 256 seconds)
[23:32] <up_the_irons> Aerosonic: hmm.. i was never on their old support chan; i used to be in the slicehost one...  oh well, doesn't matter )
[23:32] <up_the_irons> :)
[23:47] *** sorressean has quit IRC (Read error: Operation timed out)
[23:59] *** sorressean has joined #arpnetworks