| ↑back Search ←Prev date Next date→ Show only urls | (Click on time to select a line by its url) |
| Who | What | When |
|---|---|---|
| *** | Koshi has joined #arpnetworks | [05:20] |
| .................... (idle for 1h37mn) | ||
| Koshja has joined #arpnetworks
Koshi has quit IRC (Ping timeout: 250 seconds) | [06:57] | |
| ....... (idle for 34mn) | ||
| Koshja has quit IRC (Quit: Leaving) | [07:31] | |
| ..... (idle for 24mn) | ||
| ziyourenxiang has quit IRC (Quit: Leaving) | [07:55] | |
| .................. (idle for 1h25mn) | ||
| _iwc has quit IRC (K-Lined) | [09:20] | |
| ............................ (idle for 2h16mn) | ||
| dne has quit IRC (Remote host closed the connection)
dne has joined #arpnetworks | [11:36] | |
| ......... (idle for 44mn) | ||
| BryceBot has quit IRC (Ping timeout: 256 seconds)
brycec has quit IRC (Ping timeout: 260 seconds) | [12:23] | |
| .............. (idle for 1h9mn) | ||
| Lucifer333 has joined #arpnetworks
BryceBot has joined #arpnetworks brycec has joined #arpnetworks | [13:33] | |
| BryceBot has quit IRC (Ping timeout: 260 seconds)
brycec has quit IRC (Ping timeout: 268 seconds) | [13:50] | |
| ........ (idle for 38mn) | ||
| KILLALLHUMANS01 | ^ grrrr | [14:28] |
| sjackso | bad times in bryce land? | [14:34] |
| KILLALLHUMANS01 | Yes.
VPS seems to be having some issues with its underlying storage . Or that's my hypothesis. Load average suddenly spiked (over 100) and eventually it went "dead" Now it's sortof up (haven't rebooted yet, would rather not), kindof responding on the VNC terminal Actually I think it's very slowly rebooting (from when I accidentally clicked the "Send CtrlAltDel" button thinking it was the "Send key sequence..." menu I'm familiar with in other VM stuff) Yep, exactly what I did. Well damn. Was hoping I could "recover" it. | [14:34] |
| *** | BryceBot has joined #arpnetworks | [14:40] |
| mercutio | usually if load average spikes heaps it's due to lots of swapping | [14:41] |
| *** | brycec has joined #arpnetworks | [14:45] |
| .... (idle for 17mn) | ||
| KILLALLHUMANS01 | mercutio: Indeed there appears to have been a bunch of swapping that began to happen around the time, from the spotty information I have.
And if swapping isn't keeping up or worse is blocking then it all goes to shit. | [15:02] |
| mercutio | on desktop's i would normally be lead to believe it was chrome's fault
well swap by definition blocks :) | [15:02] |
| brycec | (Right but I meant blocks and never returns...) | [15:04] |
| mercutio | oh right
did you get a process list? you might be able to figure out what was causing it from that | [15:04] |
| brycec | Remarkably yes, as things sorted themselves out on the [accidental] shutdown, my top refreshed | [15:04] |
| mercutio | haha | [15:04] |
| brycec | Unfortunately, it says *everybody* was swapping.
Because they were. Because it was 2+ hours since it started. | [15:05] |
| mercutio | was anything using lots of ram? | [15:05] |
| brycec | Just the daily backup job which has $never caused this before
but other variables may be at play. | [15:05] |
| mercutio | it may be lots of apache processes or such
none using that much on their own but just with lots of them adding up... | [15:06] |
| brycec | LOL Apache... | [15:06] |
| mercutio | although i suspect you may not use apache :)
well apache seems to be one of the common examples for that behaviour. | [15:06] |
| brycec | Actually I do, but it's a single small instance for handling DAV traffic to a single private vhost.
Anyhow, from iotop: Actual DISK READ: 3.85 M/s | Actual DISK WRITE: 80.04 M/s | [15:06] |
| mercutio | that's decent speed for swap. | [15:07] |
| brycec | It's one of the newer nodes no less :)
kct03 | [15:07] |
| mercutio | i gathered that
the old nodes wouldn't swap that quick.. | [15:07] |
| brycec | haha, exactly | [15:08] |
| mercutio | but yeah it doesn't sound like it's disk performance issue so much as some kind of extra ram utilisation. | [15:08] |
| brycec | Indeed.
It swapped itself to death Some sort of race condition or the like that caused it to swap harder and harder and harder, the OOM-killer never did a thing it seems :( (oh this explains part of the load - it was time for the full, not incremental, backup) | [15:09] |
| mercutio | i hvae never had much luck with the oom killer
it often kills the wrong thing and like apache is spwaning way too many processes that are bloating up so it kills mysql | [15:11] |
| brycec | lol, whatever it takes to keep people from browsing your LAMP website right? :P
I wouldn't say I rely on OOM-killer, I was just surprised it didn't kick-in. | [15:13] |
| mercutio | tbh, i haven't had any exposure to the OOM killer in a long time. | [15:13] |
| brycec | brycec restarts the backup... let's see if it blows up again :) | [15:15] |
| .................... (idle for 1h37mn) | ||
| *** | BryceBot has quit IRC (Ping timeout: 240 seconds)
brycec has quit IRC (Ping timeout: 248 seconds) | [16:52] |
| KILLALLHUMANS01 | There we go. Broke it again.
[ 8160.104100] INFO: task jbd2/dm-4-8:404 blocked for more than 120 seconds. That's why I wonder if there are issues with the underlying storage... Or Linux. Or the host. Or some driver. (Yes I slammed it with some relatively high load activity - copying from one large database to another, so big chunk of RAM getting used I imagine) At present, it appears the kernel's waiting for data to get written to disk / filesystem writes to commit, and it's just blocking. http://sprunge.us/PBgJ | [17:03] |
| JC_Denton | anyone here use mutt? | [17:09] |
| KILLALLHUMANS01 | A bit in the past. I liked it, but I was never "great" at it. | [17:10] |
| JC_Denton | trying to run down an annoying bug. was hoping someone was using the latest with the integrated sidebar and running a mac client. | [17:11] |
| KILLALLHUMANS01 | integrated sidebar? (the fact I have no idea what that is should give you an idea how minimal my experience has been) | [17:12] |
| JC_Denton | yeah, in 1.7 they merged in the popular sidebar patch
makes for a nice addition to the interface | [17:13] |
| KILLALLHUMANS01 | Damn... Ended up having to hard power-off/on my VPS :( | [17:14] |
| *** | BryceBot has joined #arpnetworks | [17:16] |
| brycec has joined #arpnetworks | [17:22] | |
| ................ (idle for 1h17mn) | ||
| ziyourenxiang has joined #arpnetworks | [18:39] | |
| nathani | brycec: do you think something is up with the disks presented to kct0* hosts via ceph? | [18:42] |
| ................. (idle for 1h21mn) | ||
| *** | dne has quit IRC (Ping timeout: 260 seconds)
staticsafe has quit IRC (Ping timeout: 250 seconds) staticsafe has joined #arpnetworks | [20:03] |
| ............................................ (idle for 3h35mn) | ||
| dne has joined #arpnetworks | [23:40] | |
| ↑back Search ←Prev date Next date→ Show only urls | (Click on time to select a line by its url) |