[07:24] *** hive-mind has quit IRC (Ping timeout: 250 seconds) [07:43] no she said they were running up until a month ago [07:43] not that I expect her to really know anything [07:58] *** awyeah has quit IRC (Ping timeout: 252 seconds) [07:58] packet loss again [07:59] *** milki has quit IRC (Ping timeout: 252 seconds) [08:00] *** jpalmer has quit IRC (Ping timeout: 256 seconds) [08:00] *** KDE_Perry has quit IRC (Ping timeout: 252 seconds) [08:00] yeah, not just me i guess [08:00] haha [08:01] *** RandalSchwartz has quit IRC (Ping timeout: 265 seconds) [08:01] *** KDE_Perry has joined #arpnetworks [08:01] *** mkb has quit IRC (Ping timeout: 265 seconds) [08:01] *** awyeah has joined #arpnetworks [08:06] *** milki has joined #arpnetworks [08:12] *** mkb has joined #arpnetworks [08:19] *** hive-mind has joined #arpnetworks [08:35] *** mnathani_ has quit IRC (Ping timeout: 264 seconds) [08:36] *** mnathani_ has joined #arpnetworks [09:56] *** jpalmer has joined #arpnetworks [13:16] *** RandalSchwartz has joined #arpnetworks [13:20] *** RandalSchwartz has quit IRC (Changing host) [13:20] *** RandalSchwartz has joined #arpnetworks [13:35] that was lot pf people that left at once 5 hours ago [13:35] or so [13:36] i wonder if they were all using ipv6 [13:44] i was getting awful packet loss on both v6 and v4 [13:45] i wonder if it's that same annoying thing that was happening last week [13:47] well my graphs are fine :/ [13:48] but my route goes over any2ix [13:48] mine is via level3 [13:48] i've been dreaming about making a ping thingy [13:48] the plot is starting to come together [13:49] finding good hosts to monitor is the complex part [13:50] nothing's sticking out as terrible on hte monitoring that's on arp's network [13:52] i think my test hosts don't really test level3 well though [13:52] this packet loss stuff seems to start at approximately the same time each day it occurs [13:53] at like 6 hours ago? [13:53] yea, around there [13:53] i updated my smokeping with my newer targets list at least. [13:53] happens between 0830-0900 MDT [13:53] i'm never around then :) [13:54] yeah [13:54] do you have any mtr's or anything? [13:54] i can't MTR the arp box because of how severe the packet loss is [13:54] does it get unusable? [13:54] yep [13:54] i can't stay connected via ssh at all [13:54] can you setup a cron on the arp box to mtr out? [13:54] does packet loss start before it hits arps network? [13:55] no [13:55] aside from the naturally occurring deprioritization stuff [13:55] and it's lossy to arp's web site or other arp hosts at the same time as your vm? [13:55] haven't tested it [13:55] yeah i hate lboody deprioritisation :) [13:55] i guess i will next time this happens [13:56] what kvr are you on? [13:56] 03 [13:56] oh [13:56] yeah :) [13:56] well i'm monitoring that host [13:57] and i've only seen loss a couple of hours ago [13:57] my latency to the vm is jittery as all hell [13:57] +/- 12ms [13:57] even now? [13:57] yep [13:58] does it get unusable?i'm seeing 0.1 msec jitter or something [13:58] oops [13:58] i must have pressed up arrow a bit [13:58] yeh no jitter here [13:58] well a mtr now could be useful. [13:58] i usually use mtr -i=0.1 when i'm testing things :/ [13:59] which shows up deprioritisation even more [13:59] err mtr -i0.1 [13:59] but shows jitter easier [14:00] r1 drops packets with that frequency [14:02] ihttp://pastebin.com/F7SJ0GjA [14:03] wow [14:04] oh you traced to kvr03 :) [14:04] yeah [14:04] tracing to te vps now [14:04] it's largely the same [14:04] that was with -i1 ? [14:05] it could be a comcast issue [14:05] negative [14:05] it's not [14:05] oh [14:05] no packet loss to any other destination i've tested [14:06] ther's jitter to 96.120.1.73 [14:06] buddy in FL was using the voice server and started noticing packet loss and let me know [14:06] but i mean between arp/comcast. [14:06] yes, because this is a residential cable connection [14:06] That's what she said!! [14:06] BryceBot: no [14:06] Oh, okay... I'm sorry. 'yes, because this is a residential cable connection' [14:06] but there' snot jitter to the hop prior of 689.86.182.122 [14:06] but if 96.120.1.73 is a bras, that could be deprioritisation [14:07] it's the headend [14:07] so yes [14:07] yeah [14:07] mind sharing your ip? :) [14:07] 73.26.90.138 [14:08] it's worse to your ip [14:08] i'm still seeing no loss though [14:09] and it's just as bad from another provider [14:09] my latency to a box w/another provider is 30.1ms +/- .2ms [14:09] oh but it's fine from new zealand [14:09] well 3 msec jitter [14:09] or just under [14:09] That's what she said!! [14:09] BryceBot: no [14:09] Oh, okay... I'm sorry. 'or just under' [14:09] can you try tracing to 202.49.71.24 ? [14:10] that's my nz test server [14:10] i have less jitter to that destination than to arp [14:10] it's gtt->comcast for forward route [14:10] +/- 1.8ms [14:10] but i suspect it's forward path from youu that's the issue [14:10] yeah i know right [14:11] i imagine it's not going comcast->level3 [14:11] but that doesn't explain why a bunch of other peoples' connections took a dump around 30 seconds after i started seeing packet loss [14:11] they may all be on comcast? [14:11] congestion could explain the jitter, but i couldn't care less about jitter [14:11] no, their sessions to freenode dumped [14:12] but yeah level3 may be having issues [14:12] oh hmm [14:12] on arp V6 IPs (i saw at least one with no rdns that was arp's prefix) [14:12] it's not even peak time though [14:12] true [14:12] the mind boggles :) [14:12] that's why i'm wondering if someone's machine is doing something at that time [14:12] because it happens right around that time each time [14:12] That's what she said!! [14:13] BryceBot: no [14:13] Oh, okay... I'm sorry. 'because it happens right around that time each time' [14:13] yeah [14:13] like maybe saturating the link to upload or download something [14:13] i don't think kvr03 is having issues [14:13] and i don't think it's likely you're all on kvr03 [14:13] and i don't think r1 is having issues [14:13] but s7 coudl be having issues [14:14] it doesn't really explain ipv6 screwing up thouguh [14:14] nope [14:15] actually it does [14:15] ntt is on s7 [14:15] if it's a general issue [14:15] so, having looked at the users w/out cloaks who dropped when i mentioned packet loss, they were all connected via some arp svc [14:15] yeah [14:16] incoming skips s7 [14:16] for ipv4 from r1 [14:16] and my r1 based routes have been fine [14:17] but outgoing doesn't skip it [14:17] hey, milki jpalmer RandalSchwartz awyeah - are you guys connecting to IRC via an ARP VPS or dedi? if you're on a VPS, mind sharing which host? [14:17] but you wouldn't think someone would ddos at the asme time every day [14:17] right, that's why i'm thinking it's something saturating a link [14:18] or similar [14:18] and downloading at high speed over transit enough to saturate links isn't normal [14:18] offsite backup, something like that [14:18] yeah it could be that they're pushing backups onto arp [14:18] if they all happen to be on kvr03, that'd be great [14:18] haha [14:18] on a dedicated server [14:19] i really don't think kvr03 is at fault [14:19] but i'm happy to be wrong [14:19] oh hmm [14:20] kvr03 still goes throuugh s1 [14:20] i was wondering if hitting a host directly would bypass s1 [14:20] outbound route from the vm on kvr03 is via s7 [14:21] and that may happen on peering but not transit [14:21] *** hoggworm has joined #arpnetworks [14:21] inbound route is via s7 too i think you'll find [14:21] it just responds with the link address to level3 probably [14:21] but it'll be the same host [14:21] it's only r1 that's assymetric [14:21] because s1 doesn't have full route table [14:21] oh, were you meaning tracing the host kvr03.arpnetworks.com indicates s1? [14:22] well i was meaning kvr03 may have more direct path than your vm [14:22] but nah it's the same path ish [14:22] yeah, looks the same [14:22] it's pretty symmetrical. much better than before level3 was turned on [14:23] yeah level3 isn't bad [14:23] That's what she said!! [14:23] used to be inbound from nlayer/mzima/whatever and outbound via ntt [14:23] nlayer would have been fine probably [14:23] and it was unusable during "peak" time for me [14:23] yeah, it was NTT that sucked [14:23] way way way oversubbed at comcast [14:23] nlayer's been having weird shit happen recently [14:24] nlayer/mzima/tinet/gtt [14:24] i can't tell which is which haha [14:24] yeah, hahah [14:24] as4436 is having issues [14:24] which is that [14:24] mzimtinettttttt [14:24] it says nlayer :/ [14:24] yeah, some stuff still does [14:24] http://www.peeringdb.com/view.php?asn=4436 [14:25] yeah so that's the legacy nlayer [14:25] the net formerly known as nlayer [14:25] what happened to mzima haha [14:26] i wonder how many routes comcast has [14:26] 245 [14:27] including ipv6 [14:28] there's some non aggregated stuff in there too [14:28] i imagine they have more asn's, that's just AS7922 [14:31] it kind of sucks that comcast are huge and don't peer :/ [14:34] they started splitting up their network a bit more [14:34] based on market and stuff, tons of ASNs [14:34] ahh [14:34] probably a good thing [14:35] yeah [14:35] mkb isn't on comcast is he [14:35] That's what she said!! [14:35] as you said it doesn't seem like it's just a comcast issue [14:36] but prioritising making sure users connections well seems sensible [14:36] freenode on the other hand.. [14:36] dropping on freenode could be anything [14:36] that said, i've been stable for a whil again [14:44] does this packet loss happen at toher times than 7:30 to 8:00 MDT? [14:44] it has before, yes [14:44] it happened a ton last week (i think it was last week) [14:44] but atm i mean [14:44] yeh i remember that [14:44] well, it hasn't happened since last week [14:45] ahh [14:45] today's the first day. the time when it occurred in the past started at 0830-0900 MDT and then would occur again throughout the day [14:45] that gave me loss though [14:45] this one didn't give me loss [14:46] also kvr03's reoccuring issues in the past got dealt with [14:46] the specific to kvr03 issues [14:47] which was fixed about 10 days ago [14:48] but there was the issue efecting everyone around then too [14:48] and the effecting everyone was never determined the cause of i think [14:49] borrowing smokeping data from acf_ - http://kremvax.acfsys.net/smokeping.cgi?epoch_start=1426863600;hierarchy=;epoch_end=1426888048;target=Remote.arpnet-lsanca;displaymode=n;start=2015-03-20%2008%3A00;end=now;Generate%21=Generate%21 [14:49] it's on there [14:50] ahh i see [14:50] 0750 PDT 0850 MDT [14:50] it's level3 outbound [14:50] to ec2 [14:51] i wonder if it's level3 having issues [14:51] i think this is local though [14:51] microsecond ping time [14:51] on that graph, avg is .688ms [14:51] oh [14:52] nah his smokeping is weird [14:52] the top is to arpnetworks.com [14:52] but the last 30 hours is to ec2.acfsys.net [14:52] yeah, from the local box [14:52] second is from ec2 [14:52] it's the second that's having issues thoguh [14:52] the graph i linked, top part, showed 30+% packet loss to arpnetworks.com from within the network [14:53] for a period of 10 mins [14:53] oh the link didn't work [14:53] oh [14:53] i dununo why [14:53] weird [14:53] it wa so long, and it didn't make sense why [14:53] ok, well set the time today (03/20/2015) and 0700 as the starting time [14:53] to now [14:54] http://i.imgur.com/Z5WZ0y4.png as an alternative [14:54] oh that's better [14:54] mine was set to 08:00 [14:54] yeah [14:54] and it was ll grreen :/ [14:54] that weird little blip is there, even on arp>arp stuff [14:54] you're so right [14:55] http://la.meh.net.nz:24/cgi-bin/smokeping.cgi?target=meh.arpmeh [14:55] that's to arp vm from vm on arp dedicated [14:56] hmm [14:56] maybe it is kvr03 issue? [14:57] i can't rmeember what host acf is on [14:57] http://i.imgur.com/jFD4Yic.png [14:57] wishing i still had my server at home so i could have additional data [14:57] but i shut down everything here but an rpi [14:57] haha [14:57] a rpi* [14:59] acf ios on kvr27 [14:59] looking at logs [15:00] he may have more than one host thouugh [15:00] i saw something about kvr10 - i was on 27 before (had a project that ended) [15:01] oh [15:01] i saw discussion about 10 too [15:01] but he said he had high latencies to 27 [15:01] but that may have been you :) [15:01] yeah, we were trying to figure out the ntt weirdness [15:01] around then [15:01] i should just monitor them all :/ [15:23] Thanks for the reminder m0unds, I keep meaning to setup smokeping on my rpi [15:23] (but it still won't be today) [15:23] That's what she said!! [15:29] i need to set it up on my san jose vultr host [15:29] which seeems to be ntt [15:29] if i weren't running an awful java app on mine i would [15:29] so useful for testing transit from arp i suppose [15:29] i need to get another one [15:29] the network is pretty bad on vultr [15:29] is it? [15:30] well it depnds [15:30] http://sj.meh.net.nz:24/10m [15:30] vs http://la.meh.net.nz:24/10m [15:30] they're both the same for me, 5MB/sec [15:30] maybe youu're closer [15:30] i likely am [15:30] vultr has random issues more often too [15:31] oh wow [15:31] from uk sj is way faster than la [15:31] ntt vs level3 [15:31] i've only used a couple hosts long term over the last several years [15:31] arp and linode for the last 2 or so [15:32] i only use vultr cos they're cheap [15:32] just for smokeping and bw tests haha [15:32] and somewhre convenient to ssh to [15:33] ok this is strange [15:33] vultr is going ntt->level3 [15:33] and getting better performance than arp going level3->level3 [15:40] i wonder if something is up with level3 [15:41] arp to vultr was fast. [15:41] but sj vultr shouldn't be twice the speed of arp [15:41] esp when they're both l3 to uk [15:41] (that was doing a uk test) [15:42] and vultr has the extra ntt to level3 hop [15:49] http://pastebin.com/HNz6uQtK [15:49] anyone want to try my curl download test? [15:49] it downloads 7 10mb files [15:50] *** YogeeBear has joined #arpnetworks [15:51] i probably should just stick it on site [15:51] http://weallsee.net/testspeed [16:02] mercutio: http://sprunge.us/GdAh but 10MB is a pretty small test file for a good connection with 100mbps Internet. [16:02] brycec: it's good to be small [16:02] (that's from my home cable connection) [16:03] large fikles you usually have in background [16:03] smaller ones you have in foreground [16:03] and it gives you a good idea if connection is going wekll [16:03] But for a speed test, a 10MB file on a connection that can carry 10MB/s is a poor sample size. [16:03] i usually time the whole script [16:03] but adding up the numbers it looks pretty good [16:03] Ah [16:03] but la is slower than sj [16:04] why is la so slow [16:04] (I'm also closer to SJ) [16:04] oh [16:04] but look how fast you get to new zealand [16:04] the top one is new zealand [16:04] ah [16:04] faster than .au, impressive and surprising [16:04] la.meh.net.nz is a nz ip advertised on an arp vm [16:04] i hope it's not going via nz [16:04] nz is actually closer to the US than asutralia [16:05] and syd is vultr :/ [16:05] yeah but aren't most trans-pacific cables going to Sydney directly? [16:05] i thought it was more imrpessive that it's faste than los angeles [16:05] nope [16:05] well they are [16:05] but it's a longer cable [16:06] i probably should use an arp ip [16:06] but as a quick test, la.meh.net.nz ping looks fien? [16:06] (currently running mtr's to see) [16:07] (Looks like I get to NZ via Cogent's SJ -> Plain.net.nz direct) [16:08] (And Sydney via NTT LA / Equinix Sydney) [16:11] http://sprunge.us/GNCS if you're curious [16:11] la pings were slightly slower than san jose, and both went over NTT [16:13] well it matters more what my route to you is [16:14] it's 3.18 kernel on la vs 3.19 on the others [16:14] it's gtt->comcast from nz [16:14] and it's going direct via san jose [16:15] and la is sending out via level3 to comcast. [16:16] with way worse jitter [16:16] i mean icmp deprioritisation and all that, but it's 0.3 msec jitter vs 2.1 msec jitter. [16:17] the jitter on seattle level3 is showing 0.3 msec jitter though [16:17] the jitter is on te-0-1-1-ten01.spokane.wa.seattle.comcast.net [16:17] and te-0-1-0 [16:17] they both hit that hop [16:17] which alternates. [16:18] (I should point out that Spokane, roughly wear I live, is a city 5 hours East of Seattle) [16:18] actually they both hit be-37-sur03.spokane.wa.seattle.comcast.net [16:18] and the jitter isn't there [16:18] That's what she said!! [16:18] (So, presumably Seattle serves as our upstream, but it is aways away) [16:18] so probably it's forward path [16:19] ahh your mtrs only have one ping [16:20] Yeah, for expediency [16:20] it's hard to know thouugh [16:20] yeah [16:23] can you run that test again just to make sure it's not random that la is going slow? [16:23] sure [16:23] i mean 30 megabit isn't /that/ slow [16:24] do you want just la's mtr? [16:24] just la's mtr [16:24] and la's download speed [16:24] actually just doing la a few times over for the 10mb test would be fine [16:24] it gets over 40mb/sec from vultr [16:25] so i doubt it's a general throughput issue [16:25] which is making me wonder if level3 is being weird [16:25] DCC SEND STARTKEYLOGGER 0 0 0 سمَـَّوُوُحخ ̷̴̐خ ̷̴̐خ ̷̴̐خ امارتيخ ̷̴̐خ [16:25] *** YogeeBear has left "DCC SEND STARTKEYLOGGER 0 0 0 سمَـَّوُوُحخ ̷̴̐خ ̷̴̐خ ̷̴̐خ امارتيخ ̷̴̐خ" [16:26] if it's slightly problematic generally, it increases the chances that it's being more problematic other times [16:26] mercutio: http://sprunge.us/LRJG [16:27] (and traffic on my own net is relatively quiet) [16:28] wow [16:28] it's all ove rthe place. [16:28] i reckon there's packet loss [16:29] can you try the same thing with garlic.plain.net.nz:24/10m ? [16:30] i expect it'll be the same [16:33] sure, one sec [16:33] anyone use policy based routing to redirect certain traffic out a different gateway? [16:34] i've done it before mnathani_ [16:34] mercutio: http://sprunge.us/dMJd [16:34] using ip rule and multiple tables [16:34] I redirect specific subnets and addresses over my work vpn, yeah [16:35] yeah same diff with garlic [16:35] it's xen, garlic is dom0 [16:36] why does your router have such high pings? [16:36] just seems a bit strange :) [16:37] because I don't have direct ethernet to it [16:37] oh it's bridged? [16:37] (ethernet-coax-ethernet) [16:37] ahh [16:37] decanet specifically [16:38] pings to other hosts on this side of the bridge are .3ms [16:38] maybe we could try iperf udp test? [16:38] sup with yogeebear trying to dccc the file "startkeylogger" [16:38]