***: easymac is now known as Guest55375
Guest55375 is now known as easymac
easymac is now known as Guest45837
Guest45837 is now known as easymac
easymac is now known as Guest61474
Guest61474 is now known as easymac
easymac is now known as Guest79080
Guest79080 is now known as easymac
easymac is now known as Guest90147
Guest90147 is now known as easymac
easymac is now known as Guest69670 staticsafe: mercutio: you around? mercutio: yip
what's up staticsafe: mercutio: my v6 prefix's announce disappeared sometime last night, as far as I can see the prefix is exported on my end. Can you take a look?
2620:98:4001::/48 mercutio: ok
yeah that is curious
i can see 4000 but not 4001
but 4000 isn't going via arp staticsafe: i can reach it from my nagios instance indicating some peers are seeing it
(softlayer) mercutio: so he.net isn't seeing it staticsafe: yeah mercutio: is the /48 just advertised via ape
arp even staticsafe: yeah mercutio: with having no path via a supporting /32 or such
i think there's some route object issues with your subnet getting out in general
and he.net's been having issues
hmm
so 2620:98:4000::/48 isn't advertised by arp staticsafe: no, its advertised elsewhere mercutio: yeah it seems to be advertised anyway, due to the way the filters are setup
this needs fixing :)
do you want the ability to advertise 4000 still? ***: Guest69670 is now known as easymac
easymac is now known as Guest71394
Guest71394 is now known as easymac
easymac is now known as Guest55703
Guest55703 is now known as easymac
easymac is now known as Guest30476
Guest30476 is now known as easymac
easymac is now known as Guest13865
Guest13865 is now known as easymac
easymac is now known as Guest97469
Guest97469 is now known as easymac
easymac is now known as Guest78531
Guest78531 is now known as easymac
easymac is now known as Guest1206
Guest1206 is now known as easymac
easymac is now known as Guest36637
Guest36637 is now known as easymac
easymac is now known as Guest85919
Guest85919 is now known as easymac
easymac is now known as Guest89094
Guest89094 is now known as easymac
easymac is now known as Guest46254
sjackso has quit IRC (Quit: leaving)
m0unds has quit IRC (Quit: WeeChat 1.1.1)
m0unds has joined #arpnetworks
Guest46254 is now known as easymac
m0unds has quit IRC (Quit: stupid tmux) mercutio: tmux is great, dunno what he's talking about :) ***: easymac is now known as Guest91502 staticsafe: hrm, my prefix is gone again mercutio: oh? staticsafe: yeah mercutio: damn hangon
should be back
trying to fix these filters up better
but maybe i shoudl wait until a bit later staticsafe: :o brycec: The only time tmux has ever fouled me up beyond recovery was when I upgraded tmux and couldn't reattach to an existing session running the previous binary. (And even then, the interim solution was to downgrade) m0unds_: i'm having refresh issues w/tmux
on one particular client, not on another
so it's annoying as hell
so..in this instance, annoying and not so great brycec: What exactly is a refresh issue in tmux? m0unds_: when you switch between windows and it redraws the content in it
like..if text is scrolling or something brycec: Mmhm
never had an issue that redraw/switching windows didn't fix m0unds_: switch to the other window, slow redraw
switch back, slow redraw
it's irritating brycec: I blame IPv6 >.> m0unds_: it's like i'm running it over a serial connection brycec: <.< m0unds_: haha, seems legit mercutio: that sounds like a terminal issue
are you using putty or something m0unds_: doesn't matter mercutio: oh hangon i've seen something like that before m0unds. m0unds_: i've connected to it different ways, behaves the same regardless brycec: in light of recent network issues it's legit. (though not to worry, not seeing anything at this moment :) ) m0unds_: it's not on arp :) mercutio: some terminal was insanely slow with radeon view drivers.
s/view/video/ BryceBot: <mercutio> some terminal was insanely slow with radeon video drivers. mercutio: and i seem to remember tmux did something that made it slower m0unds_: this client (a raspi) is fine mercutio: probably something weird with the way it does background colours or some other less common thing brycec: protip: Don't PM brycec, he won't see it for at least 20minutes :p mercutio: heh
i used to have huge issues with notion and radeon video drivers.
whenever i resized windows using line drawing it was painfully slow
the fallback path for unaccelerated stuff is shocking. brycec: fwiw mercutio, trace from my home to ARP looks to be taking the same route as it has before. But at the moment, mostly clean. Dropped 3 out of 300 packets. mercutio: it's the path from arp to you that may be different atm brycec: mmkay I'll look in a sec :)
I can see the same nightly IPv6 abuse on ARP starting again in Smokeping https://smokeping.cobryce.com/?target=ARP.Upstream6 (that's from my ARP VPS to ARP's v6 router [i believe]) mercutio: heh
there is another change to come brycec: dun dun DUN mercutio: yeah the nightly abuse is pretty bad
it seemed to start a few minutes late tonight brycec: I do see packet loss the first hop after ARP (ge-101-0-0-13.r04.lsanca03.us.bb.gin.ntt.net) (my mtr to the feed:dead address) mercutio: i saw 98% loss to that hop
i think it's gone some chronic icmp deprioritisation
s/gone/got/ BryceBot: <mercutio> i think it's got some chronic icmp deprioritisation brycec: 80%+ over the last 2 minutes mercutio: the hop one after was fine though right brycec: Yeah could be (I'm well aware of that pifall)
Right mercutio: yeah brycec: And there's another ntt.net hop that's lossy, but through to the destination is clear.
mercutio: I don't suppose ARP can warn the customer that's beating up the v6 router or anything, can they? mercutio: for doing traffic? brycec: For disrupting service? mercutio: yeah i'm not sure
it's not like they're ddosing brycec: (Or maybe ARP just needs to get the Cisco v6 router going...) mercutio: just heavy usage.
improving the router situation is important medium to long term
not having dropouts is important short term brycec: The Acceptable Use Policy section 2.2 and 2.3 could be broadly interpreted as being violated :P
But like I've said before, I don't care too much, I just connect over ipv4 and carry on. mercutio: heh brycec: (even if I would be happier with v6... and not running up my bandwidth quota :p) mercutio: i'm sure you're nowhere near your quota brycec: whoa I've used 25% of my monthly quota actually
208.68/800GB
but that's also normal mercutio: 25%
nothing to worry about brycec: for me. Not exceptionally high due to all my terminal traffic now going over v4 :P mercutio: heh brycec: If I was worried, I'd tunnel through my employer's dedicated with 12TB or whatever it is
(Yep, 12TB of bandwidth) mercutio: it's 10TB on arp metal brycec: We have 12TB
And 504GB used this month mercutio: heh brycec: All-time we've used only 3.4TB mercutio: wow
i think i've used more than that personally brycec: I thought so too personally, but my own yearly usage was <2TB
And in our defence, we weren't using that host until July mercutio: 56.11GB/year brycec: mercutio: can you confirm which is inbound/outbound on the graphs - is inbound internet->VPS/dedi? (or is it relative to the switch port, inbound is host -> switch) -: brycec is wondering why he has a 200Mbps spike"in" brycec: *spike "inbound" mercutio: i think it's reversed brycec: okay that would make sense to me then (uploading a chunk of data) mercutio: because it's monitoring the router rather than the host
but don't hold me to that :) brycec: If you really didn't want to be held accountable for that remark, should've started your message with [OFF] :P
Now we have logs.... mercutio: or just say i think
rather than i know :) brycec: status update on my home HE -> ARP VPS mtr: 17 dropped out of 1500.
So, pretty quiet mercutio: i'll be back in a bit
sweet that sounds way better
i'll check it agian in a bit :) -: brycec reconnects over ipv6 brycec: we'll see how this goes... assuming I'm at my tmux to notice ***: Guest91502 is now known as easymac
easymac is now known as Guest36744
m0unds has joined #arpnetworks
Guest36744 is now known as easymac
easymac is now known as Guest44560 brycec: blargh dropped a few packets just then
but still better than the other day m0unds: yea, was pretty bad mercutio: hmm i'm seeing a bit of bursts of jitter up to like 25 msec m0unds: i think it's happened a few times in the last mo or so mercutio: then the next second it's fine
when you say drop a few packets you mean like jittery ssh not responding quickly then fine? -: brycec only noticed it because he was trying to change tmux windows at the time and it just wouldn't respond mercutio: or like it feels like it's dropping out. brycec: mercutio: I mean mtr says packets were dropped
And also ssh was non-responsive mercutio: a few in a row? brycec: Hard to say
Yes mercutio: well m0unds noticed it too brycec: (thank you mtr "Display Mode") mercutio: heh
yeah when i use mtr i can glance at it cos i see the colour change
although it wasn't on screen atm brycec: I'm seeing 4 packets here and there over the last 90 seconds (12 total) -: brycec goes back to abusing Syncthing m0unds: haha mercutio: that seems like heaps
ahh
i'm seeing loss to www.he.net
but no loss to the gateway
actually i'm not sure if it's loss to or from brycec: I should add, mercutio, the the loss I'm seeing is on my home HE tunnel -> ARP VPS, starting at the ntt hop (one and only, second to last hop) and continuing to the last hop of my VPS
Another burst of loss there and there mercutio: yeah i'm still seeing loss to www.he.net
i'm trying to find a host site
for some reason uk vultr seems down :(
i thought that maybe on he brycec: 10 packets in a row just then :(
Cloudflare has native IPv6 IIRC
could test to them mercutio: cloudflare doesn't cache my file properly
i was hoping to use iperf to test direction brycec: I figured you were just pinging/mtr'ing mercutio: i was testing with vultr before
but that was hitting gtt brycec: ah JC_Denton: anyone here using freenas? mercutio: my route over ipv4 ius hitting he.net to vultr uk
so hopefully it'll help :)
weird it's hitting trit for this route brycec: JC_Denton: Yes m0unds: when it was worst for me was when my v6 route from home traversed he
when it goes over just ntt, it's fine mercutio: m0unds: i think there's at least double issues atm
and one potential issue :) m0unds: daily, haha brycec: my conclusion the other day, m0unds, was that it is a particular HE-NTT handoff that is over-burdened. mercutio: i mean of separate issues m0unds: yeah JC_Denton: brycec: know off the top of your head if the ZFS setup on it supports POSIX ACLs? m0unds: brycec: yeah, that's how it looked to me mercutio: m0unds: but he.net direct was screwed too right?
i mean brycec m0unds: no brycec: JC_Denton: hm no I don't m0unds: but my route changed off he->ntt before i could really dig into it at all brycec: mercutio: can you clarify what you mean by HE direct? mercutio: he.net to he.net
when not hitting ntt was also breaking brycec: JC_Denton: IIRC xacl support is in the underlying ZFS (because FreeBSD) but I never tested it.
mer
DAMMIT IPV6
brb m0unds: hahahahah brycec: I'm back to v4. Trying to chat with y'all is impossible :p mercutio: this trit route is like 10 megabit tcp to uk m0unds: need moar missed keystrokes pls brycec: mercutio: How would I test HE-HE? ARP traffic always crossed NTT. mercutio: oh really? brycec: m0unds: bless SSH/TCP it always kept up. m0unds: brycec: are you hitting lax2.he.net?
when you're enroute to arp via your tunnel ***: Guest44560 is now known as easymac brycec: m0unds: yes m0unds: check this out brycec: lax2.he -> lsanca.ntt m0unds: http://pastebin.com/eLLZEMkP brycec: (and it's that lsanca hop that drops) mercutio: this is ntt->cogent and level3->trit and seems ok in both directions ***: easymac is now known as Guest39524 m0unds: that's LAX2-> comcast in NM, and LAX2 -> arp brycec: (hence my conclusion that it's the lax2he->lsanca.ntt hop that is saturated) m0unds: yup mercutio: 2001:418:0:5000::52c is the ntt hop i was seeing drop when it was fine before brycec: I see 52d in my current mtr mercutio: probably the other end of such :)
but yeah that alone seems to mean nothing brycec: or could be another parallel route mercutio: cos 2001:418:0:2000::1ee is fine brycec: (that hop doesn't even appear in my mtr) mercutio: hah why is he.net looking glass showinghigh pings for every hop
this is outbound from arp over ntt m0unds: lax1->lax2 is fine
lax2->lax1 is fine (he to he)
it's that ntt hop borking stuff
because he->comcast on ipv6 is shitty (goes via ntt) and he->arp is shitty (also goes via ntt iirc) mercutio: 2001:470:0:6c::1
what's your trace to that ip like? brycec: for me, sea1.ipv6.he.net -> sea1.he.nt -> sjc2.he.net -> lax2.he.net no loss (from my home tunnel)
(obviously) mercutio: yeah and that's screwed from arp brycec: And from ARP https://dl.dropboxusercontent.com/u/3167967/screenshot_2015-09-05_21-11-23.png m0unds: 216.218.226.238 tracerouting to this via v4 looks awful, goes through any2ix
(tunnel server in seattle) mercutio: is there something special that makes www.he.net special? m0unds: what do you mean? mercutio: with loss m0unds: is it not bad?
it doesn't go through any2ix mercutio: it is bad m0unds: nm
it does mercutio: but it seems to be bad suddenly m0unds: just no rdns on the hop
any2ix to HE sucks from arp
every single one i've tried mercutio: so i'm wondering if it's rate limit
so it's always been bad?
and ntt->he is bad m0unds: no, don't think it always has mercutio: but for quite a while? m0unds: maybe, i don't usually see he on my route
but when i saw tons of pkt loss w/unreliable ssh the other night, it involved he
and that ntt hop brycec mentioned
and any2ix on either the return or other path mercutio: i'm looking for more looking glasse
i foundm something that's slow, but it's even slower over ipv4
and fine from linux, go figure m0unds: huh mercutio: he.net's giving issues again
i'ev found dns server as something beter to ping
and it's he.net incoming rather than outgoing that's the issue ***: Guest39524 is now known as easymac
easymac is now known as Guest13147 mnathani2_: s/beter/better BryceBot: <mercutio> i'ev found dns server as something better to ping mercutio: heh mnathani2_: hey where did the 2 come from? mercutio: i got my own tunnel
can test easy now :) ***: mnathani2_ is now known as mnathani_ mercutio: probably from another netsplit
when it says so much loss with mtr, it's actually single direction mnathani_: there, thats better mercutio: it's traffic into arp via he.net that's the issue mnathani_: via any2ix? mercutio: and that ntt hop is return path, where arp sends traffic out, so it's not the issue
yeah mnathani_: congestion? mercutio: i don't think it's actually congestion.
i'm not sure what it is though
it's like suddenly packets stop coming through
and then it's fine again ***: Guest13147 is now known as easymac
easymac is now known as Guest52803