***: MeltedLux has quit IRC (Quit: R.I.P)
MeltedLux has joined #arpnetworks
ziyourenxiang has joined #arpnetworks
mkb has joined #arpnetworks
qbit has quit IRC (Quit: WeeChat 1.9.1)
qbit has joined #arpnetworks
ziyourenxiang has quit IRC (Ping timeout: 240 seconds)
hive-mind has quit IRC (Ping timeout: 240 seconds)
hive-mind has joined #arpnetworks
hive-mind has quit IRC (Ping timeout: 260 seconds)
hive-mind has joined #arpnetworks
mkb has quit IRC (Quit: leaving)
mkb has joined #arpnetworks
dj_goku has joined #arpnetworks
dj_goku has quit IRC (Changing host)
dj_goku has joined #arpnetworks
hive-mind has quit IRC (Ping timeout: 248 seconds)
hive-mind has joined #arpnetworks
incin has quit IRC (Read error: Connection reset by peer)
incin has joined #arpnetworks
milki has quit IRC (Ping timeout: 240 seconds)
milki has joined #arpnetworks
milki has quit IRC (Ping timeout: 255 seconds)
IM_AWESOME1 has quit IRC (Remote host closed the connection)
CoBryceMatrixBo4 has quit IRC (Remote host closed the connection)
milki has joined #arpnetworks KILLALLHUMANS01: sct04 / ceph cluster have a little bump? Oct 11 22:08:24 vps3 kernel: [12522051.816013] sd 0:0:0:0: [sda] abort
(And my Thunder instance has been tumbling downhill since then, having remounted to read-only, Postgres is unhappy, etc etc etc ***: brycec has quit IRC (Ping timeout: 246 seconds)
BryceBot has quit IRC (Ping timeout: 246 seconds)
BryceBot has joined #arpnetworks up_the_irons: KILLALLHUMANS01: had a couple reports of disk timeouts, so i'm looking into it KILLALLHUMANS01: Thanks up_the_irons
I ended up hard-rebooting things
and am gradually getting back to normal, you should see brycec join shortly
btw this was the SSD-backed pool specifically up_the_irons: KILLALLHUMANS01: but are you still seeing disk issues? KILLALLHUMANS01: Not currently -: KILLALLHUMANS01 is dealing with some rather unhappy database tables up_the_irons: OK KILLALLHUMANS01: According to logs, the issue started at 22:10 and continued for a few minutes at least (when the system dropped to read-only mode and logging was useless) -- And anything logged after read-only is muddy at best since there was a lot of stuff just flapping up_the_irons: PST ? KILLALLHUMANS01: "ARP Standard Time" :P yeah up_the_irons: LOL
OK ***: brycec has joined #arpnetworks brycec: works up_the_irons: what works
brycec: ^ KILLALLHUMANS01: my connection (except actually I was in the middle of typing "/join #arpnetworks" when Weechat rejoined the room on its own, so you just got the last 5 letters :P up_the_irons: I kicked out a particularly noisy and dying drive from our Ceph cluster a few moments ago. Not sure why it wasn't kicked out on its own, but I suspect those who had some blocks on that particular one may have got some timeouts too. brycec: interesting up_the_irons: I think so too