[01:38] *** Guest51708 has quit IRC (Ping timeout: 268 seconds) [01:40] *** jcv has joined #arpnetworks [06:31] *** vom has joined #arpnetworks [06:32] hello all - anyone seeing disk latency on vps hosts ? [06:33] i had a kernel panic in the wee hours from the disk "falling out" [06:33] and now im back up - but IO is abyssmal [06:33] this is using sysbench [06:33] Read 32.812Mb Written 21.875Mb Total transferred 54.688Mb (186.26Kb/sec) [08:25] *** incin has joined #arpnetworks [09:08] vom: out of curiosity, is that under openbsd? [09:28] johnny-o: its not - its ubuntu server 16.04 - kernel 4.4.0 [09:29] i had the kernel panic a few weeks ago - but not the slow down [09:29] this morning i got both [09:29] the server is very lightly loaded - ram is never an issue [09:38] is that rand or seq r/w? just curious, going to try the same thing on my openbsd host [09:39] here's my cmdline ive been playing with this morning [09:39] sysbench --test=fileio --file-total-size=2G prepare [09:39] sysbench --test=fileio --file-total-size=2G --file-test-mode=rndrw --max-time=600 --max-requests=0 run [09:39] sysbench --test=fileio --file-total-size=2G cleanup [09:39] so to answer your question - looks like random read/write [09:40] i began having disk i/o performance issues after migrating to a new host, so i thought maybe it was unique to openbsd [09:40] ive never ran sysbench before - so im running it on some local bare metal boxes too just to get some more data points [09:41] my vm was migrated a few months ago i think - and (huge grain of salt here) - i never had disk issues like this previous [09:41] i didn't report anything official, but mentioned something briefly here and mercutio mentioned a workaround [09:42] hmm, interesting [09:42] i guess i don't know enough about the underlying host configuration changes, but i also know i had not a single issue on the old host [09:44] *** ziyourenxiang has quit IRC (Ping timeout: 248 seconds) [09:44] FWIW I frequently see high load averages overnight in the wee hours. My VPS itself is doing anything more or less at the time, just seems that disk IO slows down. (my backups run ~12 hours later) wee hours = 01:40 - 01:50 ARP standard time (PDT, GMT-7) [09:45] (It rebounds though. Don't know what your ongoing issue may be, sorry) [10:08] yeah, mine is much more intermittent. i'm compiling as much information as i can to speak to it in a more useful manner, but i can't seem to force reproducibility and repeatability [10:11] For comparison, results from my busy VPS' SSD-backed Ceph pool: Read 827.95Mb Written 551.97Mb Total transferred 1.3476Gb (2.2975Mb/sec) 147.04 Requests/sec executed [10:11] For the spinning-rust volume (less busy, at least by my VPS): Read 164.06Mb Written 109.38Mb Total transferred 273.44Mb (466.01Kb/sec) 29.13 Requests/sec executed [10:11] I wouldn't say I notice any significant latency though. Not currently, as of those sysbench runs, anyways. [10:49] consider a lot of vm's with daily cronjobs roughly the same time of day ... [11:48] *** dferris__ has joined #arpnetworks [11:53] *** dferris has quit IRC (*.net *.split) [14:31] my performance seems to have slowly gotten better as the day has gone on... [14:31] Read 289.69Mb Written 193.12Mb Total transferred 482.81Mb (1.5972Mb/sec) [14:32] thats about 10x what it was running first thing this morning [14:36] i don't mean this as an accusation at all - but could there be resource (storage) contention on the host ? [14:37] and if a good chunk of VMs are running various crons at roughly the same time, its running too hot and timeouts and hangs are manifesting at the vm level ? [15:20] that's closer to what i've seen during a couple of runs at random times throughout the day using your sysbench params [15:22] i have no idea if i can systematically detect what i'm seeing, as it may be abstracted away [15:23] seems to manifest itself in annoying interactive ways, like deleting messages in mutt and having to wait 5-7 seconds [17:05] *** Tsesarevich_ has joined #arpnetworks [17:05] *** Tsesarevich has quit IRC (Ping timeout: 265 seconds) [17:37] *** qbit[m] has quit IRC (Ping timeout: 245 seconds) [17:59] *** qbit[m] has joined #arpnetworks [19:23] *** mercutio has joined #arpnetworks [19:23] *** ChanServ sets mode: +o mercutio