***: carvite has quit IRC (*.net *.split)
sjackso has quit IRC (*.net *.split)
jpalmer has quit IRC (*.net *.split)
KDE_Perry has quit IRC (*.net *.split)
KILLALLHUMANS01 has quit IRC (*.net *.split)
DaCa has quit IRC (*.net *.split)
milki has quit IRC (*.net *.split)
toddf has quit IRC (*.net *.split)
up_the_irons has quit IRC (*.net *.split)
d^_^b has quit IRC (*.net *.split)
mike-burns has quit IRC (*.net *.split)
mhoran has quit IRC (*.net *.split)
carvite has joined #arpnetworks
sjackso has joined #arpnetworks
jpalmer has joined #arpnetworks
KDE_Perry has joined #arpnetworks
KILLALLHUMANS01 has joined #arpnetworks
DaCa has joined #arpnetworks
milki has joined #arpnetworks
toddf has joined #arpnetworks
up_the_irons has joined #arpnetworks
d^_^b has joined #arpnetworks
mike-burns has joined #arpnetworks
mhoran has joined #arpnetworks
asimov.freenode.net sets mode: +oo toddf up_the_irons
nesta has quit IRC (Ping timeout: 245 seconds)
nesta has joined #arpnetworks
Lucifer333 has quit IRC (Quit: Leaving)
Lucifer333 has joined #arpnetworks brycec: I couldn't say. I have neither the expertise nor the data to fully form a hypothesis. ***: BryceBot has quit IRC (Ping timeout: 258 seconds) KILLALLHUMANS01: Well cool/fuck.
Reproduced my issue even though I slowed down the whole process. ***: brycec has quit IRC (Ping timeout: 248 seconds) KILLALLHUMANS01: So I'm hitting this when I copy from one database to another (moving BryceBot and other services off MySQL, finally.) I wrote a Ruby script based on Sequel's (bin/sequel) copy-database mode. It does a big SELECT on the source database (some half-million rows as I recall) and then for each result, INSERTs into the new database.
I thought the act of slamming so many rows into the database at such a high rate was responsible for bringing my system down, so I put a sleep in - 1 row every 5 seconds. BAM still died.
What's kindof funny is that neither MySQL nor Postgres (nor my script) are showing in htop (well MySQL is, just not at high load)
Also weird is that kswapd is responsible for a relatively large chunk of CPU, along with kworker/0:3 (I'll need to check which device this refers to), but there is ZERO swap usage as of yet, and still a decent chunk of free memory for that matter.
https://www.dropbox.com/s/0o1e1e39we6drcp/screenshot_2016-12-29_09-12-03.png?dl=0
I'm guessing that something (MySQL?) tried to allocate a big piece of memory, forcing the system to start trying to figure out what all it can swap out, but it's having trouble... Either deciding, or actually writing to swap.
Regrettably I don't have an iotop this time.
Also of note, pretty low load average... Then again, htop hasn't refreshed any time recently. It probably froze when the kernel started looking for volunteers to swap out.
So... Yeah. There's a bit more data, I guess. At least as much as I can get. Seems like I can reproduce it relatively easily, at least with this current state of things.
(Or maybe it's the Ruby process trying to allocate?)
Unrelated question - On consule.cust, the "broadcast message" function ^Ecb, does that go to all users on consule.cust, or just those connected to my VPS? (If the former, then I apologize for my language just now.) ***: BryceBot has joined #arpnetworks
brycec has joined #arpnetworks nathani: maybe you need moar RAM brycec mercutio: or maybe it's some kernel bug
hmm i'm not sure if it broadcasts to all
we should try it :) brycec: nathani: You don't even work for ARP, why are you trying to sell me on more RAM? :P (Once I'm off MySQL, I'll actually have an even smaller memory footprint :) ) nathani: :-) mercutio: linux's memory management used to be really really bad
and having more ram could make it worse sometimes
linux aggressively forces memory into being used as disk cache
so you can get memory pressure even when you have plenty of spare memory if you read a lot of data in
in 2.2.x kernels it was all over the place
erk, i just remembered that was like 15 years back
haha brycec: Yeah I'm pretty sure it was better by 3.16.0-4-amd64 mercutio: i don't notice much changes in new kernels these days
i do wonder when BBR will be merged though
oh it's in 4.9 brycec: Anyhow, now paginating my input so hopefully no random memory pressure. So far, so good...
3+ million rows, 1000 at a time.
With periodic breaks every 5 seconds to allow disk writes to flush if they want, etc -: brycec is watching the source MySQL database gradually use more RAM while Postgres just sits back and takes it. mike-burns: My current client uses MySQL; it's been fun watching each new dev they hire raise the same question within their first week: "So, when are we switching to Postgres?" brycec: lol up_the_irons: nathani: i'm using weechat and screen. non-ascii is no problem.:)
JC_Denton: i use mutt ***: nesta has quit IRC (Quit: WeeChat 1.6)
nesta has joined #arpnetworks JC_Denton: up_the_irons: 1.7.x? up_the_irons: no 1.5.x
it's old JC_Denton: *shrug*
1.7 isn't that new ;)
but it did mainline the sidebar patch, which is why i built from source on my debian login host
was having a god awful annoying screen corruption bug
i finally figured it out, though. i needed to compile against wide-ncurses vs. plain mercutio: i thought it was slang, hmm
mine is curses too up_the_irons: i c ***: ziyourenxiang has joined #arpnetworks mercutio: i'm using 1.4?
it seems i am way behind JC_Denton: you can compile it with slang