[00:48] *** Guest88801 is now known as easymac [00:49] *** easymac is now known as Guest55375 [01:49] *** Guest55375 is now known as easymac [01:49] *** easymac is now known as Guest45837 [02:50] *** Guest45837 is now known as easymac [02:50] *** easymac is now known as Guest61474 [03:50] *** Guest61474 is now known as easymac [03:51] *** easymac is now known as Guest79080 [04:51] *** Guest79080 is now known as easymac [04:52] *** easymac is now known as Guest90147 [05:52] *** Guest90147 is now known as easymac [05:52] *** easymac is now known as Guest69670 [06:08] mercutio: you around? [06:08] yip [06:08] what's up [06:09] mercutio: my v6 prefix's announce disappeared sometime last night, as far as I can see the prefix is exported on my end. Can you take a look? [06:09] 2620:98:4001::/48 [06:09] ok [06:13] yeah that is curious [06:13] i can see 4000 but not 4001 [06:14] but 4000 isn't going via arp [06:14] i can reach it from my nagios instance indicating some peers are seeing it [06:14] (softlayer) [06:14] so he.net isn't seeing it [06:14] yeah [06:15] is the /48 just advertised via ape [06:15] arp even [06:15] yeah [06:15] with having no path via a supporting /32 or such [06:18] i think there's some route object issues with your subnet getting out in general [06:18] and he.net's been having issues [06:21] hmm [06:21] so 2620:98:4000::/48 isn't advertised by arp [06:21] no, its advertised elsewhere [06:22] yeah it seems to be advertised anyway, due to the way the filters are setup [06:22] this needs fixing :) [06:22] do you want the ability to advertise 4000 still? [06:53] *** Guest69670 is now known as easymac [06:53] *** easymac is now known as Guest71394 [07:54] *** Guest71394 is now known as easymac [07:54] *** easymac is now known as Guest55703 [08:55] *** Guest55703 is now known as easymac [08:55] *** easymac is now known as Guest30476 [09:56] *** Guest30476 is now known as easymac [09:56] *** easymac is now known as Guest13865 [10:56] *** Guest13865 is now known as easymac [10:57] *** easymac is now known as Guest97469 [11:57] *** Guest97469 is now known as easymac [11:58] *** easymac is now known as Guest78531 [12:59] *** Guest78531 is now known as easymac [12:59] *** easymac is now known as Guest1206 [13:59] *** Guest1206 is now known as easymac [14:00] *** easymac is now known as Guest36637 [15:00] *** Guest36637 is now known as easymac [15:00] *** easymac is now known as Guest85919 [16:01] *** Guest85919 is now known as easymac [16:01] *** easymac is now known as Guest89094 [17:02] *** Guest89094 is now known as easymac [17:02] *** easymac is now known as Guest46254 [17:30] *** sjackso has quit IRC (Quit: leaving) [17:56] *** m0unds has quit IRC (Quit: WeeChat 1.1.1) [17:57] *** m0unds has joined #arpnetworks [18:02] *** Guest46254 is now known as easymac [18:02] *** m0unds has quit IRC (Quit: stupid tmux) [18:03] tmux is great, dunno what he's talking about :) [18:03] *** easymac is now known as Guest91502 [18:04] hrm, my prefix is gone again [18:04] oh? [18:04] yeah [18:04] damn hangon [18:05] should be back [18:05] trying to fix these filters up better [18:05] but maybe i shoudl wait until a bit later [18:07] :o [18:15] The only time tmux has ever fouled me up beyond recovery was when I upgraded tmux and couldn't reattach to an existing session running the previous binary. (And even then, the interim solution was to downgrade) [18:16] i'm having refresh issues w/tmux [18:16] on one particular client, not on another [18:16] so it's annoying as hell [18:16] so..in this instance, annoying and not so great [18:16] What exactly is a refresh issue in tmux? [18:16] when you switch between windows and it redraws the content in it [18:16] like..if text is scrolling or something [18:16] Mmhm [18:17] never had an issue that redraw/switching windows didn't fix [18:17] switch to the other window, slow redraw [18:17] switch back, slow redraw [18:17] it's irritating [18:17] I blame IPv6 >.> [18:17] it's like i'm running it over a serial connection [18:17] <.< [18:17] haha, seems legit [18:17] that sounds like a terminal issue [18:17] are you using putty or something [18:17] doesn't matter [18:17] oh hangon i've seen something like that before m0unds. [18:17] i've connected to it different ways, behaves the same regardless [18:17] in light of recent network issues it's legit. (though not to worry, not seeing anything at this moment :) ) [18:18] it's not on arp :) [18:18] some terminal was insanely slow with radeon view drivers. [18:18] s/view/video/ [18:18] some terminal was insanely slow with radeon video drivers. [18:18] and i seem to remember tmux did something that made it slower [18:18] this client (a raspi) is fine [18:18] probably something weird with the way it does background colours or some other less common thing [18:21] protip: Don't PM brycec, he won't see it for at least 20minutes :p [18:21] heh [18:21] i used to have huge issues with notion and radeon video drivers. [18:22] whenever i resized windows using line drawing it was painfully slow [18:22] the fallback path for unaccelerated stuff is shocking. [18:22] fwiw mercutio, trace from my home to ARP looks to be taking the same route as it has before. But at the moment, mostly clean. Dropped 3 out of 300 packets. [18:22] it's the path from arp to you that may be different atm [18:23] mmkay I'll look in a sec :) [18:24] I can see the same nightly IPv6 abuse on ARP starting again in Smokeping https://smokeping.cobryce.com/?target=ARP.Upstream6 (that's from my ARP VPS to ARP's v6 router [i believe]) [18:24] heh [18:24] there is another change to come [18:25] dun dun DUN [18:25] yeah the nightly abuse is pretty bad [18:25] it seemed to start a few minutes late tonight [18:26] I do see packet loss the first hop after ARP (ge-101-0-0-13.r04.lsanca03.us.bb.gin.ntt.net) (my mtr to the feed:dead address) [18:26] i saw 98% loss to that hop [18:26] i think it's gone some chronic icmp deprioritisation [18:26] s/gone/got/ [18:26] i think it's got some chronic icmp deprioritisation [18:26] 80%+ over the last 2 minutes [18:27] the hop one after was fine though right [18:27] Yeah could be (I'm well aware of that pifall) [18:27] Right [18:27] yeah [18:27] And there's another ntt.net hop that's lossy, but through to the destination is clear. [18:28] mercutio: I don't suppose ARP can warn the customer that's beating up the v6 router or anything, can they? [18:29] for doing traffic? [18:29] For disrupting service? [18:30] yeah i'm not sure [18:30] it's not like they're ddosing [18:30] (Or maybe ARP just needs to get the Cisco v6 router going...) [18:30] just heavy usage. [18:30] improving the router situation is important medium to long term [18:30] not having dropouts is important short term [18:31] The Acceptable Use Policy section 2.2 and 2.3 could be broadly interpreted as being violated :P [18:32] But like I've said before, I don't care too much, I just connect over ipv4 and carry on. [18:33] heh [18:33] (even if I would be happier with v6... and not running up my bandwidth quota :p) [18:33] i'm sure you're nowhere near your quota [18:34] whoa I've used 25% of my monthly quota actually [18:34] 208.68/800GB [18:34] but that's also normal [18:34] 25% [18:34] nothing to worry about [18:34] for me. Not exceptionally high due to all my terminal traffic now going over v4 :P [18:34] heh [18:34] If I was worried, I'd tunnel through my employer's dedicated with 12TB or whatever it is [18:35] (Yep, 12TB of bandwidth) [18:35] it's 10TB on arp metal [18:35] We have 12TB [18:35] And 504GB used this month [18:35] heh [18:36] All-time we've used only 3.4TB [18:36] wow [18:36] i think i've used more than that personally [18:37] I thought so too personally, but my own yearly usage was <2TB [18:37] And in our defence, we weren't using that host until July [18:38] 56.11GB/year [18:38] mercutio: can you confirm which is inbound/outbound on the graphs - is inbound internet->VPS/dedi? (or is it relative to the switch port, inbound is host -> switch) [18:39] * brycec is wondering why he has a 200Mbps spike"in" [18:39] *spike "inbound" [18:39] i think it's reversed [18:39] okay that would make sense to me then (uploading a chunk of data) [18:39] because it's monitoring the router rather than the host [18:39] but don't hold me to that :) [18:40] If you really didn't want to be held accountable for that remark, should've started your message with [OFF] :P [18:40] Now we have logs.... [18:40] or just say i think [18:40] rather than i know :) [18:41] status update on my home HE -> ARP VPS mtr: 17 dropped out of 1500. [18:41] So, pretty quiet [18:41] i'll be back in a bit [18:41] sweet that sounds way better [18:41] i'll check it agian in a bit :) [18:42] * brycec reconnects over ipv6 [18:42] we'll see how this goes... assuming I'm at my tmux to notice [19:03] *** Guest91502 is now known as easymac [19:04] *** easymac is now known as Guest36744 [19:59] *** m0unds has joined #arpnetworks [20:05] *** Guest36744 is now known as easymac [20:05] *** easymac is now known as Guest44560 [20:50] blargh dropped a few packets just then [20:50] but still better than the other day [20:51] yea, was pretty bad [20:52] hmm i'm seeing a bit of bursts of jitter up to like 25 msec [20:52] i think it's happened a few times in the last mo or so [20:52] then the next second it's fine [20:53] when you say drop a few packets you mean like jittery ssh not responding quickly then fine? [20:53] * brycec only noticed it because he was trying to change tmux windows at the time and it just wouldn't respond [20:53] or like it feels like it's dropping out. [20:53] mercutio: I mean mtr says packets were dropped [20:53] And also ssh was non-responsive [20:53] a few in a row? [20:53] Hard to say [20:54] Yes [20:54] well m0unds noticed it too [20:54] (thank you mtr "Display Mode") [20:54] heh [20:54] yeah when i use mtr i can glance at it cos i see the colour change [20:54] although it wasn't on screen atm [20:54] I'm seeing 4 packets here and there over the last 90 seconds (12 total) [20:55] * brycec goes back to abusing Syncthing [20:55] haha [20:55] that seems like heaps [20:55] ahh [20:55] i'm seeing loss to www.he.net [20:55] but no loss to the gateway [20:56] actually i'm not sure if it's loss to or from [20:56] I should add, mercutio, the the loss I'm seeing is on my home HE tunnel -> ARP VPS, starting at the ntt hop (one and only, second to last hop) and continuing to the last hop of my VPS [20:56] Another burst of loss there and there [20:56] yeah i'm still seeing loss to www.he.net [20:56] i'm trying to find a host site [20:57] for some reason uk vultr seems down :( [20:57] i thought that maybe on he [20:58] 10 packets in a row just then :( [20:58] Cloudflare has native IPv6 IIRC [20:58] could test to them [20:58] cloudflare doesn't cache my file properly [20:58] i was hoping to use iperf to test direction [20:58] I figured you were just pinging/mtr'ing [20:58] i was testing with vultr before [20:59] but that was hitting gtt [20:59] ah [21:00] anyone here using freenas? [21:00] my route over ipv4 ius hitting he.net to vultr uk [21:00] so hopefully it'll help :) [21:00] weird it's hitting trit for this route [21:01] JC_Denton: Yes [21:01] when it was worst for me was when my v6 route from home traversed he [21:01] when it goes over just ntt, it's fine [21:02] m0unds: i think there's at least double issues atm [21:02] and one potential issue :) [21:02] daily, haha [21:02] my conclusion the other day, m0unds, was that it is a particular HE-NTT handoff that is over-burdened. [21:02] i mean of separate issues [21:02] yeah [21:02] brycec: know off the top of your head if the ZFS setup on it supports POSIX ACLs? [21:02] brycec: yeah, that's how it looked to me [21:02] m0unds: but he.net direct was screwed too right? [21:02] i mean brycec [21:02] no [21:02] JC_Denton: hm no I don't [21:03] but my route changed off he->ntt before i could really dig into it at all [21:03] mercutio: can you clarify what you mean by HE direct? [21:03] he.net to he.net [21:03] when not hitting ntt was also breaking [21:03] JC_Denton: IIRC xacl support is in the underlying ZFS (because FreeBSD) but I never tested it. [21:03] mer [21:04] DAMMIT IPV6 [21:04] brb [21:04] hahahahah [21:04] I'm back to v4. Trying to chat with y'all is impossible :p [21:04] this trit route is like 10 megabit tcp to uk [21:04] need moar missed keystrokes pls [21:05] mercutio: How would I test HE-HE? ARP traffic always crossed NTT. [21:05] oh really? [21:05] m0unds: bless SSH/TCP it always kept up. [21:05] brycec: are you hitting lax2.he.net? [21:05] when you're enroute to arp via your tunnel [21:05] *** Guest44560 is now known as easymac [21:05] m0unds: yes [21:05] check this out [21:05] lax2.he -> lsanca.ntt [21:06] http://pastebin.com/eLLZEMkP [21:06] (and it's that lsanca hop that drops) [21:06] this is ntt->cogent and level3->trit and seems ok in both directions [21:06] *** easymac is now known as Guest39524 [21:06] that's LAX2-> comcast in NM, and LAX2 -> arp [21:06] (hence my conclusion that it's the lax2he->lsanca.ntt hop that is saturated) [21:06] yup [21:07] 2001:418:0:5000::52c is the ntt hop i was seeing drop when it was fine before [21:07] I see 52d in my current mtr [21:07] probably the other end of such :) [21:08] but yeah that alone seems to mean nothing [21:08] or could be another parallel route [21:08] cos 2001:418:0:2000::1ee is fine [21:08] (that hop doesn't even appear in my mtr) [21:09] hah why is he.net looking glass showinghigh pings for every hop [21:09] this is outbound from arp over ntt [21:09] lax1->lax2 is fine [21:09] lax2->lax1 is fine (he to he) [21:09] it's that ntt hop borking stuff [21:10] because he->comcast on ipv6 is shitty (goes via ntt) and he->arp is shitty (also goes via ntt iirc) [21:11] 2001:470:0:6c::1 [21:11] what's your trace to that ip like? [21:13] for me, sea1.ipv6.he.net -> sea1.he.nt -> sjc2.he.net -> lax2.he.net no loss (from my home tunnel) [21:13] (obviously) [21:13] yeah and that's screwed from arp [21:14] And from ARP https://dl.dropboxusercontent.com/u/3167967/screenshot_2015-09-05_21-11-23.png [21:19] 216.218.226.238 tracerouting to this via v4 looks awful, goes through any2ix [21:19] (tunnel server in seattle) [21:21] is there something special that makes www.he.net special? [21:21] what do you mean? [21:21] with loss [21:21] is it not bad? [21:21] it doesn't go through any2ix [21:22] it is bad [21:22] nm [21:22] it does [21:22] but it seems to be bad suddenly [21:22] just no rdns on the hop [21:22] any2ix to HE sucks from arp [21:22] every single one i've tried [21:22] so i'm wondering if it's rate limit [21:23] so it's always been bad? [21:23] and ntt->he is bad [21:24] no, don't think it always has [21:25] but for quite a while? [21:25] maybe, i don't usually see he on my route [21:25] but when i saw tons of pkt loss w/unreliable ssh the other night, it involved he [21:25] and that ntt hop brycec mentioned [21:26] and any2ix on either the return or other path [21:26] i'm looking for more looking glasse [21:29] i foundm something that's slow, but it's even slower over ipv4 [21:30] and fine from linux, go figure [21:37] huh [21:37] he.net's giving issues again [21:38] i'ev found dns server as something beter to ping [21:56] and it's he.net incoming rather than outgoing that's the issue [22:06] *** Guest39524 is now known as easymac [22:07] *** easymac is now known as Guest13147 [23:01] s/beter/better [23:01] i'ev found dns server as something better to ping [23:01] heh [23:01] hey where did the 2 come from? [23:01] i got my own tunnel [23:01] can test easy now :) [23:01] *** mnathani2_ is now known as mnathani_ [23:01] probably from another netsplit [23:02] when it says so much loss with mtr, it's actually single direction [23:02] there, thats better [23:02] it's traffic into arp via he.net that's the issue [23:02] via any2ix? [23:02] and that ntt hop is return path, where arp sends traffic out, so it's not the issue [23:03] yeah [23:03] congestion? [23:03] i don't think it's actually congestion. [23:03] i'm not sure what it is though [23:03] it's like suddenly packets stop coming through [23:04] and then it's fine again [23:08] *** Guest13147 is now known as easymac [23:08] *** easymac is now known as Guest52803