| ↑back Search ←Prev date Next date→ Show only urls | (Click on time to select a line by its url) |
| Who | What | When |
|---|---|---|
| *** | nathani has quit IRC (Ping timeout: 276 seconds) | [06:31] |
| nathani has joined #arpnetworks | [06:40] | |
| ................. (idle for 1h21mn) | ||
| mhoran | I'm having some funky IO issues on my VPS, anyone else? | [08:01] |
| mike-burns | Update: I wasn't before, but now I am.
Took me a minute to SSH in. | [08:03] |
| mhoran | Maybe something going on with the ceph cluster? | [08:03] |
| brycec | My loadavg is unusually high, seems to stem from time spent waiting on disk IO.
aka "me too" | [08:11] |
| Oct 04 08:13:47 vps3 kernel: sd 0:0:0:1: [sdb] tag#9 abort
:X That's my disk from the spinning-rust Ceph pool... brycec fires the up_the_irons and mercutio flare | [08:18] | |
| ..... (idle for 20mn) | ||
| *** | up_the_irons2 has joined #arpnetworks
up_the_irons2 is now known as up_the_irons | [08:39] |
| up_the_irons | why do I keep getting booted from the chan | [08:39] |
| *** | up_the_irons is now known as Guest54746 | [08:39] |
| brycec | Because your client isn't identifying to Nickserv before joining the channel, Guest54746 ? | [08:40] |
| Guest54746 | but I do have it set to identify
It keeps renaming me | [08:41] |
| brycec | heh, likely because nickserv thinks your nick's in use somewhere on the network, you may have to ghost it to get it back | [08:42] |
| *** | Guest54746 has quit IRC (Quit: WeeChat 1.2) | [08:42] |
| brycec | Anyway, IRC aside, let's get ceph back :) | [08:42] |
| *** | up_the_irons2 has joined #arpnetworks | [08:45] |
| brycec | Not seeing any more kernel disk errors at least, and services are slowly beginning to respond. Hooray | [08:46] |
| up_the_irons2 | I'm seeing the same on my end
A failing disk was finally kicked out of the cluster and the cluster is rebalancing... the failed disk is in the SATA pool, so the SSD pool _shouldn't_ have been affected | [08:47] |
| brycec | (I only noticed it in the SATA pool, fwiw) | [08:49] |
| up_the_irons2 | Roger that | [08:49] |
| .... (idle for 15mn) | ||
| How's things looking for others?
Cluster is almost completely rebalanced | [09:04] | |
| brycec | (Rebalancing is magic)
(Also, don't let your datacenter hands/monkeys *not* label drives... I pulled 4 drives before I found the right ones. Thank goodness for rebalancing.) | [09:04] |
| mike-burns | Things seem to be back to normal-ish. | [09:05] |
| ........ (idle for 38mn) | ||
| up_the_irons2 | Of course, everything is labeled | [09:43] |
| ...... (idle for 26mn) | ||
| *** | ziyourenxiang_ has quit IRC (Ping timeout: 268 seconds) | [10:09] |
| ......................................................................................... (idle for 7h22mn) | ||
| ziyourenxiang_ has joined #arpnetworks | [17:31] | |
| ↑back Search ←Prev date Next date→ Show only urls | (Click on time to select a line by its url) |