#arpnetworks 2019-10-04,Fri

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***nathani has quit IRC (Ping timeout: 276 seconds) [06:31]
nathani has joined #arpnetworks [06:40]
................. (idle for 1h21mn)
mhoranI'm having some funky IO issues on my VPS, anyone else? [08:01]
mike-burnsUpdate: I wasn't before, but now I am.
Took me a minute to SSH in.
[08:03]
mhoranMaybe something going on with the ceph cluster? [08:03]
brycecMy loadavg is unusually high, seems to stem from time spent waiting on disk IO.
aka "me too"
[08:11]
Oct 04 08:13:47 vps3 kernel: sd 0:0:0:1: [sdb] tag#9 abort
:X
That's my disk from the spinning-rust Ceph pool...
brycec fires the up_the_irons and mercutio flare
[08:18]
..... (idle for 20mn)
***up_the_irons2 has joined #arpnetworks
up_the_irons2 is now known as up_the_irons
[08:39]
up_the_ironswhy do I keep getting booted from the chan [08:39]
***up_the_irons is now known as Guest54746 [08:39]
brycecBecause your client isn't identifying to Nickserv before joining the channel, Guest54746 ? [08:40]
Guest54746but I do have it set to identify
It keeps renaming me
[08:41]
brycecheh, likely because nickserv thinks your nick's in use somewhere on the network, you may have to ghost it to get it back [08:42]
***Guest54746 has quit IRC (Quit: WeeChat 1.2) [08:42]
brycecAnyway, IRC aside, let's get ceph back :) [08:42]
***up_the_irons2 has joined #arpnetworks [08:45]
brycecNot seeing any more kernel disk errors at least, and services are slowly beginning to respond. Hooray [08:46]
up_the_irons2I'm seeing the same on my end
A failing disk was finally kicked out of the cluster
and the cluster is rebalancing...
the failed disk is in the SATA pool, so the SSD pool _shouldn't_ have been affected
[08:47]
brycec(I only noticed it in the SATA pool, fwiw) [08:49]
up_the_irons2Roger that [08:49]
.... (idle for 15mn)
How's things looking for others?
Cluster is almost completely rebalanced
[09:04]
brycec(Rebalancing is magic)
(Also, don't let your datacenter hands/monkeys *not* label drives... I pulled 4 drives before I found the right ones. Thank goodness for rebalancing.)
[09:04]
mike-burnsThings seem to be back to normal-ish. [09:05]
........ (idle for 38mn)
up_the_irons2Of course, everything is labeled [09:43]
...... (idle for 26mn)
***ziyourenxiang_ has quit IRC (Ping timeout: 268 seconds) [10:09]
......................................................................................... (idle for 7h22mn)
ziyourenxiang_ has joined #arpnetworks [17:31]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)