[00:07] *** hazardous has joined #arpnetworks
[03:16] *** MeltedLux has quit IRC (Quit: R.I.P)
[03:21] *** MeltedLux has joined #arpnetworks
[05:30] *** ziyourenxiang has joined #arpnetworks
[07:00] *** mkb has joined #arpnetworks
[07:34] *** qbit has quit IRC (Quit: WeeChat 1.9.1)
[08:02] *** qbit has joined #arpnetworks
[10:05] *** ziyourenxiang has quit IRC (Ping timeout: 240 seconds)
[10:24] *** hive-mind has quit IRC (Ping timeout: 240 seconds)
[10:27] *** hive-mind has joined #arpnetworks
[10:43] *** hive-mind has quit IRC (Ping timeout: 260 seconds)
[10:45] *** hive-mind has joined #arpnetworks
[18:48] *** mkb has quit IRC (Quit: leaving)
[18:52] *** mkb has joined #arpnetworks
[21:08] *** dj_goku has joined #arpnetworks
[21:08] *** dj_goku has quit IRC (Changing host)
[21:08] *** dj_goku has joined #arpnetworks
[21:34] *** hive-mind has quit IRC (Ping timeout: 248 seconds)
[21:53] *** hive-mind has joined #arpnetworks
[22:08] *** incin has quit IRC (Read error: Connection reset by peer)
[22:09] *** incin has joined #arpnetworks
[22:19] *** milki has quit IRC (Ping timeout: 240 seconds)
[22:27] *** milki has joined #arpnetworks
[22:33] *** milki has quit IRC (Ping timeout: 255 seconds)
[22:48] *** IM_AWESOME1 has quit IRC (Remote host closed the connection)
[22:48] *** CoBryceMatrixBo4 has quit IRC (Remote host closed the connection)
[22:50] *** milki has joined #arpnetworks
[22:59] <KILLALLHUMANS01> sct04 / ceph cluster have a little bump? Oct 11 22:08:24 vps3 kernel: [12522051.816013] sd 0:0:0:0: [sda] abort
[22:59] <KILLALLHUMANS01> (And my Thunder instance has been tumbling downhill since then, having remounted to read-only, Postgres is unhappy, etc etc etc
[23:13] *** brycec has quit IRC (Ping timeout: 246 seconds)
[23:13] *** BryceBot has quit IRC (Ping timeout: 246 seconds)
[23:15] *** BryceBot has joined #arpnetworks
[23:15] <up_the_irons> KILLALLHUMANS01: had a couple reports of disk timeouts, so i'm looking into it
[23:15] <KILLALLHUMANS01> Thanks up_the_irons
[23:16] <KILLALLHUMANS01> I ended up hard-rebooting things
[23:16] <KILLALLHUMANS01> and am gradually getting back to normal, you should see brycec join shortly
[23:16] <KILLALLHUMANS01> btw this was the SSD-backed pool specifically
[23:17] <up_the_irons> KILLALLHUMANS01: but are you still seeing disk issues?
[23:18] <KILLALLHUMANS01> Not currently
[23:18] * KILLALLHUMANS01 is dealing with some rather unhappy database tables
[23:20] <up_the_irons> OK
[23:20] <KILLALLHUMANS01> According to logs, the issue started at 22:10 and continued for a few minutes at least (when the system dropped to read-only mode and logging was useless) -- And anything logged after read-only is muddy at best since there was a lot of stuff just flapping
[23:21] <up_the_irons> PST ?
[23:22] <KILLALLHUMANS01> "ARP Standard Time" :P yeah
[23:22] <up_the_irons> LOL
[23:22] <up_the_irons> OK
[23:26] *** brycec has joined #arpnetworks
[23:26] <brycec> works
[23:28] <up_the_irons> what works
[23:29] <up_the_irons> brycec: ^
[23:30] <KILLALLHUMANS01> my connection (except actually I was in the middle of typing "/join #arpnetworks" when Weechat rejoined the room on its own, so you just got the last 5 letters :P
[23:30] <up_the_irons> I kicked out a particularly noisy and dying drive from our Ceph cluster a few moments ago.  Not sure why it wasn't kicked out on its own, but I suspect those who had some blocks on that particular one may have got some timeouts too.
[23:30] <brycec> interesting
[23:33] <up_the_irons> I think so too