mercutio: you around?
yip
what's up
mercutio: my v6 prefix's announce disappeared sometime last night, as far as I can see the prefix is exported on my end. Can you take a look?
2620:98:4001::/48
ok
yeah that is curious
i can see 4000 but not 4001
but 4000 isn't going via arp
i can reach it from my nagios instance indicating some peers are seeing it
(softlayer)
so he.net isn't seeing it
yeah
is the /48 just advertised via ape
arp even
yeah
with having no path via a supporting /32 or such
i think there's some route object issues with your subnet getting out in general
and he.net's been having issues
hmm
so 2620:98:4000::/48 isn't advertised by arp
no, its advertised elsewhere
yeah it seems to be advertised anyway, due to the way the filters are setup
this needs fixing :)
do you want the ability to advertise 4000 still?
tmux is great, dunno what he's talking about :)
hrm, my prefix is gone again
oh?
yeah
damn hangon
should be back
trying to fix these filters up better
but maybe i shoudl wait until a bit later
:o
The only time tmux has ever fouled me up beyond recovery was when I upgraded tmux and couldn't reattach to an existing session running the previous binary. (And even then, the interim solution was to downgrade)
i'm having refresh issues w/tmux
on one particular client, not on another
so it's annoying as hell
so..in this instance, annoying and not so great
What exactly is a refresh issue in tmux?
when you switch between windows and it redraws the content in it
like..if text is scrolling or something
Mmhm
never had an issue that redraw/switching windows didn't fix
switch to the other window, slow redraw
switch back, slow redraw
it's irritating
I blame IPv6 >.>
it's like i'm running it over a serial connection
<.<
haha, seems legit
that sounds like a terminal issue
are you using putty or something
doesn't matter
oh hangon i've seen something like that before m0unds.
i've connected to it different ways, behaves the same regardless
in light of recent network issues it's legit. (though not to worry, not seeing anything at this moment :) )
it's not on arp :)
some terminal was insanely slow with radeon view drivers.
s/view/video/
<mercutio> some terminal was insanely slow with radeon video drivers.
and i seem to remember tmux did something that made it slower
this client (a raspi) is fine
probably something weird with the way it does background colours or some other less common thing
protip: Don't PM brycec, he won't see it for at least 20minutes :p
heh
i used to have huge issues with notion and radeon video drivers.
whenever i resized windows using line drawing it was painfully slow
the fallback path for unaccelerated stuff is shocking.
fwiw mercutio, trace from my home to ARP looks to be taking the same route as it has before. But at the moment, mostly clean. Dropped 3 out of 300 packets.
it's the path from arp to you that may be different atm
mmkay I'll look in a sec :)
I can see the same nightly IPv6 abuse on ARP starting again in Smokeping https://smokeping.cobryce.com/?target=ARP.Upstream6 (that's from my ARP VPS to ARP's v6 router [i believe])
heh
there is another change to come
dun dun DUN
yeah the nightly abuse is pretty bad
it seemed to start a few minutes late tonight
I do see packet loss the first hop after ARP (ge-101-0-0-13.r04.lsanca03.us.bb.gin.ntt.net) (my mtr to the feed:dead address)
i saw 98% loss to that hop
i think it's gone some chronic icmp deprioritisation
s/gone/got/
<mercutio> i think it's got some chronic icmp deprioritisation
80%+ over the last 2 minutes
the hop one after was fine though right
Yeah could be (I'm well aware of that pifall)
Right
yeah
And there's another ntt.net hop that's lossy, but through to the destination is clear.
mercutio: I don't suppose ARP can warn the customer that's beating up the v6 router or anything, can they?
for doing traffic?
For disrupting service?
yeah i'm not sure
it's not like they're ddosing
(Or maybe ARP just needs to get the Cisco v6 router going...)
just heavy usage.
improving the router situation is important medium to long term
not having dropouts is important short term
The Acceptable Use Policy section 2.2 and 2.3 could be broadly interpreted as being violated :P
But like I've said before, I don't care too much, I just connect over ipv4 and carry on.
heh
(even if I would be happier with v6... and not running up my bandwidth quota :p)
i'm sure you're nowhere near your quota
whoa I've used 25% of my monthly quota actually
208.68/800GB
but that's also normal
25%
nothing to worry about
for me. Not exceptionally high due to all my terminal traffic now going over v4 :P
heh
If I was worried, I'd tunnel through my employer's dedicated with 12TB or whatever it is
(Yep, 12TB of bandwidth)
it's 10TB on arp metal
We have 12TB
And 504GB used this month
heh
All-time we've used only 3.4TB
wow
i think i've used more than that personally
I thought so too personally, but my own yearly usage was <2TB
And in our defence, we weren't using that host until July
56.11GB/year
mercutio: can you confirm which is inbound/outbound on the graphs - is inbound internet->VPS/dedi? (or is it relative to the switch port, inbound is host -> switch)
*spike "inbound"
i think  it's reversed
okay that would make sense to me then (uploading a chunk of data)
because it's monitoring the router rather than the host
but don't hold me to that :)
If you really didn't want to be held accountable for that remark, should've started your message with [OFF] :P
Now we have logs....
or just say i think
rather than i know :)
status update on my home HE -> ARP VPS mtr: 17 dropped out of 1500.
So, pretty quiet
i'll be back in a bit
sweet that sounds way better
i'll check it agian in a bit :)
we'll see how this goes... assuming I'm at my tmux to notice
blargh dropped a few packets just then
but still better than the other day
yea, was pretty bad
hmm i'm seeing a bit of bursts of jitter up to like 25 msec
i think it's happened a few times in the last mo or so
then the next second it's fine
when you say drop a few packets you mean like jittery ssh not responding quickly then fine?
or like it feels like it's dropping out.
mercutio: I mean mtr says packets were dropped
And also ssh was non-responsive
a few in a row?
Hard to say
Yes
well m0unds noticed it too
(thank you mtr "Display Mode")
heh
yeah when i use mtr i can glance at it cos i see the colour change
although it wasn't on screen atm
I'm seeing 4 packets here and there over the last 90 seconds (12 total)
haha
that seems like heaps
ahh
i'm seeing loss to www.he.net
but no loss to the gateway
actually i'm not sure if it's loss to or from
I should add, mercutio, the the loss I'm seeing is on my home HE tunnel -> ARP VPS, starting at the ntt hop (one and only, second to last hop) and continuing to the last hop of my VPS
Another burst of loss there and there
yeah i'm still seeing loss to www.he.net
i'm trying to find a host site
for some reason uk vultr seems down :(
i thought that maybe on he
10 packets in a row just then :(
Cloudflare has native IPv6 IIRC
could test to them
cloudflare doesn't cache my file properly
i was hoping to use iperf to test direction
I figured you were just pinging/mtr'ing
i was testing with vultr before
but that was hitting gtt
ah
anyone here using freenas?
my route over ipv4 ius hitting he.net to vultr uk
so hopefully it'll help :)
weird it's hitting trit for this route
JC_Denton: Yes
when it was worst for me was when my v6 route from home traversed he
when it goes over just ntt, it's fine
m0unds: i think there's at least double issues atm
and one potential issue :)
daily, haha
my conclusion the other day, m0unds, was that it is a particular HE-NTT handoff that is over-burdened.
i mean of separate issues
yeah
brycec: know off the top of your head if the ZFS setup on it supports POSIX ACLs?
brycec: yeah, that's how it looked to me
m0unds: but he.net direct was screwed too right?
i mean brycec
no
JC_Denton: hm no I don't
but my route changed off he->ntt before i could really dig into it at all
mercutio: can you clarify what you mean by HE direct?
he.net to he.net
when not hitting ntt was also breaking
JC_Denton: IIRC xacl support is in the underlying ZFS (because FreeBSD) but I never tested it.
mer
DAMMIT IPV6
brb
hahahahah
I'm back to v4. Trying to chat with y'all is impossible :p
this trit route is like 10 megabit tcp to uk
need moar missed keystrokes pls
mercutio: How would I test HE-HE? ARP traffic always crossed NTT.
oh really?
m0unds: bless SSH/TCP it always kept up.
brycec: are you hitting lax2.he.net?
when you're enroute to arp via your tunnel
m0unds: yes
check this out
lax2.he -> lsanca.ntt
http://pastebin.com/eLLZEMkP
(and it's that lsanca hop that drops)
this is ntt->cogent and level3->trit and seems ok in both directions
that's LAX2-> comcast in NM, and LAX2 -> arp
(hence my conclusion that it's the lax2he->lsanca.ntt hop that is saturated)
yup
2001:418:0:5000::52c is the ntt hop i was seeing drop when it was fine before
I see 52d in my current mtr
probably the other end of such :)
but yeah that alone seems to mean nothing
or could be another parallel route
cos 2001:418:0:2000::1ee is fine
(that hop doesn't even appear in my mtr)
hah why is he.net looking glass showinghigh pings for every hop
this is outbound from arp over ntt
lax1->lax2 is fine
lax2->lax1 is fine (he to he)
it's that ntt hop borking stuff
because he->comcast on ipv6 is shitty (goes via ntt) and he->arp is shitty (also goes via ntt iirc)
2001:470:0:6c::1
what's your trace to that ip like?
for me, sea1.ipv6.he.net -> sea1.he.nt -> sjc2.he.net -> lax2.he.net no loss (from my home tunnel)
(obviously)
yeah and that's screwed from arp
And from ARP https://dl.dropboxusercontent.com/u/3167967/screenshot_2015-09-05_21-11-23.png
216.218.226.238 tracerouting to this via v4 looks awful, goes through any2ix
(tunnel server in seattle)
is there something special that makes www.he.net special?
what do you mean?
with loss
is it not bad?
it doesn't go through any2ix
it is bad
nm
it does
but it seems to be bad suddenly
just no rdns on the hop
any2ix to HE sucks from arp
every single one i've tried
so i'm wondering if it's rate limit
so it's always been bad?
and ntt->he is bad
no, don't think it always has
but for quite a while?
maybe, i don't usually see he on my route
but when i saw tons of pkt loss w/unreliable ssh the other night, it involved he
and that ntt hop brycec mentioned
and any2ix on either the return or other path
i'm looking for more looking glasse
i foundm something that's slow, but it's even slower over ipv4
and fine from linux, go figure
huh
he.net's giving issues again
i'ev found dns server as something beter to ping
and it's he.net incoming rather than outgoing that's the issue
s/beter/better
<mercutio> i'ev found dns server as something better to ping
heh
hey where did the 2 come from?
i got my own tunnel
can test easy now :)
probably from another netsplit
when it says so much loss with mtr, it's actually single direction
there, thats better
it's traffic into arp via he.net that's the issue
via any2ix?
and that ntt hop is return path, where arp sends traffic out, so it's not the issue
yeah
congestion?
i don't think it's actually congestion.
i'm not sure what it is though
it's like suddenly packets stop coming through
and then it's fine again