[06:31] *** nathani has quit IRC (Ping timeout: 276 seconds)
[06:40] *** nathani has joined #arpnetworks
[08:01] <mhoran> I'm having some funky IO issues on my VPS, anyone else?
[08:03] <mike-burns> Update: I wasn't before, but now I am.
[08:03] <mike-burns> Took me a minute to SSH in.
[08:03] <mhoran> Maybe something going on with the ceph cluster?
[08:11] <brycec> My loadavg is unusually high, seems to stem from time spent waiting on disk IO.
[08:11] <brycec> aka "me too"
[08:18] <brycec> Oct 04 08:13:47 vps3 kernel: sd 0:0:0:1: [sdb] tag#9 abort
[08:18] <brycec> :X
[08:18] <brycec> That's my disk from the spinning-rust Ceph pool...
[08:19] * brycec fires the up_the_irons and mercutio flare
[08:39] *** up_the_irons2 has joined #arpnetworks
[08:39] *** up_the_irons2 is now known as up_the_irons
[08:39] <up_the_irons> why do I keep getting booted from the chan
[08:39] *** up_the_irons is now known as Guest54746
[08:40] <brycec> Because your client isn't identifying to Nickserv before joining the channel, Guest54746 ?
[08:41] <Guest54746> but I do have it set to identify
[08:41] <Guest54746> It keeps renaming me
[08:42] <brycec> heh, likely because nickserv thinks your nick's in use somewhere on the network, you may have to ghost it to get it back
[08:42] *** Guest54746 has quit IRC (Quit: WeeChat 1.2)
[08:42] <brycec> Anyway, IRC aside, let's get ceph back :)
[08:45] *** up_the_irons2 has joined #arpnetworks
[08:46] <brycec> Not seeing any more kernel disk errors at least, and services are slowly beginning to respond. Hooray
[08:47] <up_the_irons2> I'm seeing the same on my end
[08:47] <up_the_irons2> A failing disk was finally kicked out of the cluster
[08:47] <up_the_irons2> and the cluster is rebalancing...
[08:48] <up_the_irons2> the failed disk is in the SATA pool, so the SSD pool _shouldn't_ have been affected
[08:49] <brycec> (I only noticed it in the SATA pool, fwiw)
[08:49] <up_the_irons2> Roger that
[09:04] <up_the_irons2> How's things looking for others?
[09:04] <up_the_irons2> Cluster is almost completely rebalanced
[09:04] <brycec> (Rebalancing is magic)
[09:05] <brycec> (Also, don't let your datacenter hands/monkeys *not* label drives... I pulled 4 drives before I found the right ones. Thank goodness for rebalancing.)
[09:05] <mike-burns> Things seem to be back to normal-ish.
[09:43] <up_the_irons2> Of course, everything is labeled
[10:09] *** ziyourenxiang_ has quit IRC (Ping timeout: 268 seconds)
[17:31] *** ziyourenxiang_ has joined #arpnetworks