liking the BIRD filter / function language, even though it seems a bit weird at first
  import filter {
    bgp_community.add((our_asn,20000));
    accept;
  };
easy enough
a little bit weird is, show all HE routes:
sh ro filter { if 6939 = bgp_path.first then accept; }
seem verbose, but meh, it works
i like openbgpd syntax etc
but bird is faster at converging from what i understand
and is supported on linux
i'd like to see openbgp for linux though
i left that comcast trace going, and there's still no packet loss to comcast
i think i need to do it during earlier hours
mercutio: bird seems to be insanely fast; peers go into "Established" state almost instantly after I reload config with a new peer
up_the_irons: pulp = repo management,  candlepin = subscription management (system A has repos A,B, and F, but not D or E)  etc
jpalmer: oh cool
foreman is a frontend dashboard, and ENC to puppet.
cool
up_the_irons: does that mean any2xi is up now?
http://arstechnica.com/information-technology/2014/02/netflix-performance-on-verizon-and-comcast-has-been-dropping-for-months/
Ars Technica: "Netflix performance on Verizon and Comcast has been dropping for months"
it's interesting that verizon and comcast were the two destinations ntt were haveing issues too
s/too/to/
<mercutio> it's interesting that verizon and comcast were the two destinations ntt were haveing issues to
Hey. I could use some help on an ipv6 /48 ubuntu configuration. No matter the search query in google I can't seem to find anyone that describes it the way arp networks does. Someone know how to set up the /48 on a single Ubuntu VPS?
at one point it was routed to your vps. at one point the lowest /64 was an ethernet segment and the rest was avilable on a support ticket request basis for routing. I'm not sure what the defaults are at this point. if you're a recent customer, just try setting the lowest /64 subnet on your ethernet segment and see how that goes .. try fe80::1 and <yourv6network>::1 for a default router, one of those should work. perhaps there's a ...
... wiki page I'm unaware of. hope that helps.
I'm told it has been routed to link-local and that I should set my side to fe80::2/64
I'm not exactly sure what they mean by "my side".. default gateway, local address or?
http://wiki.arpnetworks.com/wiki/48%20IPv6%20on%20OpenBSD is good reference
So "set your side to..." means set the IP on the interface to fe80::2
the default gateway is fe80::1
(because ARP's side is fe80::1 and routing the /48 to fe80::2)
(Also: Requisite "if you don't know how to do this stuff, then you probably shouldn't be messing with it.")
you shouldn't need a /48
Oh I know it's expert only ;) But I have other servers where /48 is routed differently (I think)
but I do
basically
but yeah I know that I shouldn't
ARP's method of routing is actually pretty common too, fwiw.
Though the majority of tutorials and howtos are written for people with HE tunnels and the like, so I can see how that drowns out the useful information.
I dont doubt it. It's just really hard to find it described that way anywhere else
yep, it's mostly two lines about native ipv6 and then 4 pages about tunnels
native ipv6 is easy though
Yep.
And the /48 too once you realize it's two lines or so
I got some ipv6 connectivity now. Thanks a lot for your help
arp's default config is great for a single vps. if you have multiple, you have to route v6 to the others from your first vps, or ask arpnetworks for changes.  I opted for plan b *grin*, one /64 on the ethernet segment.
yea, i wrote a post about configuring SRX devices with a roll-your-own ipv6 tunnel in flow mode because so many of the HE tunnel broker tutorials are silly and tell you to switch off flow mode on your appliance and stuff
hopefully it'll help someone sooner or later - same with working srcnat for xbox live, since it seems people way overthink that stuff
heh
m0unds: link?
mercutio: not yet
toddf: for the record, our default is no routing at all, just /64 on your VLAN, so no single vps is a point of failure
up_the_irons: ah. I've been around too long to know what the actual current default is, hope I made that clear above ;-)
lol
sine up_the_irons missed it: 15:46:39 <@toddf> up_the_irons: ah. I've been around too long to know what the actual current default is, hope I made that clear above ;-)
*since
brycec: tnx!
If you had multiple VPS'en and a /48, I suppose you could always CARP them all, but routing would be annoying/tricky.
yeah
I still haven't worked out a good way to give my CARP backup IPv6 access to an HE tunnel :/ Not without watching for the state change and scripting route changes anyways.
(It's also not high on the priority list)
brycec: convince he.net it needs to do ospf6 with you and have two tunnels one to each router?
not always doable because some people only have a single ip, carp can be done in this case, but v4 connectivity is always fun in the backup router instance
Two tunnels but same subnet?
Not to worry, both routers have public v4 IPs
plus one shared
you'd need two tunnels and ospf6 should handle routing of the same subnet yes
Well all that's left is to convince HE of anything, lol
(note I've never heard of anyone doing it, but if you want to avoid scripting and wish to do it up proper...)
Yeah that would be proper. But given how much I'm paying them... I don't expect them to do anything "for me"
you could of course get two vps'en from arpnetworks and do ospf6 across two gif tunnels to your home for full redundancy on your side ..
they do permit bgp6 over a tunnel for a fee, if I read their website properly
That sounds like fun :) And I'm still meaning to move my IPv6 tunnels to ARP.  However lately, HE's reliability has been > ARP :(
toddf: Actually I can request a BGP tunnel for free
But first I'd need an ASN...
details
And only 7 POPs support it
(As in: not my closest POP)
brycec: http://chris.vanvoro.us/2013/12/26/fun-with-ipv6/
Thanks m0unds
sure, it's not the best, but it's better than most of what i'd read, haha
that your site?
yep, terribleness that it is
octopress + nginx
code repo on bitbucket
i was too lazy to octopress so i just went back to wordpress
i started using nitrous.io as a quick IDE for posting
i'm hosting a friend's wordpress site - it's the only reason i still have mysql and php running on my vps
Man I have no idea why I thought this would be more difficult... Using my ARP VPS as a v6 tunnel endpoint accomplished! (Still need to setup routing and firewalling, but that's all)
thanks for the kick in the butt m0unds
you betcha!
i sat on mine for 6 mos before i did it
I'm over 1yr now
i think
then got bored at work and went 'meh' and just did it
apparently cogent -> verizon is even more broken than normal
tl;dr just need matching gif/v4tunnel/etc sections on both ends, that's it
the srx part was what i hung up on initially though because i was on junos 11.4, which doesn't support ipv6 in flow mode
cogent issues have been going on for something like two years now?
so when i updated to the final build for my srx (discontinued model) it fixed it
cool, congrats
(my oldest invoice seems to be Nov 2012. Over a year now, woo)
brycec: yeh it pretty simple to tunnel ipv6
you may have to mss clamp if you're forwarding traffic though
Hardest part now is deciding on address allocations
fac3 ?
I'll keep that in mind, thanks mercutio
lol
i dunno :)
err face would work too
face:b00c is pretty well-known ;)
1337 ?
yeh i know
Yeah there are a bunch of "clever" ones out there. I'm far more practical.... But I can't just start at "1"
you only have 16 bits to play with
(0 is already in use)
hahaha
i started at 2
16 bits?
48 to 64
my clients at home are 4, iirc
Oh sure, duh
bcec ?
removing r and y from your nick, that don't map to hex :)
ha
Probably gonna start at f00a
it does kind of sound like "be sick" though
or f00f like the pentium bug?
like "sick" as in, WAY SICK DOODZ
So help whichever net ends up on f00f ;)
1c12
( i see one too)
actually I should just migrate my current HE prefixes
bryce: you only have 16 bits, it's not that many
you need the /64 for autoconfig
16 bits is till pretty big
(And I know I can't really sub-divide the /64)
it never felt very big on pc's :)
dammn those 64k memory limits
it was a real pita
but yeah it's a lot better than like 1 or 8 or such
for those running their own ntp server
"1. If you run ntpd, upgrading to the latest version, which removes the "monlist" command that is used for these attacks; alternately, disabling the monitoring function by adding "disable
+monitor" to your /etc/ntp.conf file."
we're getting LOTS of notices for NTP-based UDP amplification attacks
up_the_irons: Any way to forward those notices to the responsible party?
*parties
brycec: i am in the process of doing so, yes
a very big time suck
39 notices based on IP.  gotta lookup the IP, get email of customer, then foward.
*forward
Bummer
maybe i could write some filter..
(Oh good, I was already secure)
procmail or something
actually, would anyone *else* like to write something?  I'll pay (obviously).  Basic flow would be: 1) I get an abuse complaint, 2) i forward to some special address, 3) something / script on that address looks up IP with regex, 4) IP returns an email address (with our REST API), 5) forward that email
or, pointers to how this would be done would help
i can try to code something up
Seems straight-forward enough
up_the_irons: it very well could happen for dns too
mercutio: dns?
so maybe having an easy way to email ip's would be good
up_the_irons: the any thing, and open recursive are being hit on authorative and recursive a lot recently too
mercutio: if you mean the amplification attacks, yes, very much so
up_the_irons: what about having a sepcial domain you email with users ip@blah.arpnetworks.com
or such
and then it emails the right person, and a sepcial mailbox to keep note
which would just mean cutting and pasting the ip
which isn't automated, but is simpler to test, ..
mercutio: ah true
mercutio: i like it
I LIKE IT
can you map from ip to user with a mysql query or such?
more like a bit of ruby
4) IP returns an email address (with our REST API)"
(obviously not a public REST API ;) )
obviously :)
i wonder if for things like recursive dns there should be tests every now and then
but with a little script magic if such a system was setup it'd be easy to email effected users
err affected?
^
hmm as an addition could have some extra things to bounce to which would send automated message content that say how to fix open dns etc
or maybe just keep a list of the various things, and people can parse themselves.
for ntp i'm in favour of openntpd which doesn't listen by default
affected
are there any web based test tools for the NTP or DNS amplification attacks yet?
(I don't know how to exploit it offhand,  but would like to verify my DNS and NTP servers are ok.
host -t any <your domain name>
the any thing is complicated
basically more providers need to do bcp38 to improve the situation
as the predominant issue is that it's valid to do an any request for a domain name.
yeah, that returns several records for all of my domains.
see that's normal
now the problem is someone can spoof an address
so that your response goes to another address
esp if one has lots of entries
like say host -t any microsoft.com has quite a bit of data
it's only like 4x amplification normally with that htough
but still if it's 10 megabit of requests that makes 40 megabit of response
yep yep
arp defaults to 5 megabit rate limit for udp, so you'll only be able to return 5 megabit
but that could impact other services..
generally speaking most people seem to be ignoring the amplification attack and suggesting that it's the people sending spoofed requests that are the problem
heh.   it's not the misconfigured SMTP servers,  it's the spammers!
well udp will limit what response size normally
and tcp won't work
bcp38 means people can't spoof addresses as easily so it cna't work as easily
from memory comcast is the biggest provider with no protection
I'll have to read up on bcp38.  not familiar with it.
https://www.nanog.org/sites/default/files/mon_general_weber_defeat_23.pdf
it only really matters for providers
basically it means that you can't send packets with my source ip address
which arp do btw
but basically if the any requests aren't terribly long it's probably mostly ok
Are there network anomalies at present? I am getting about 2% packetloss from Toronto
via ntt?
ntt -> verizon still seems lossy
acf had a smokeping
uhh
acf's smokeping was really good in the middle of the night
and his comcast gets better earlier
acf: did you check out your smpkeping?
via nlayer / mzima
oh prob diff issue then
2% isn't so bad
thats the forward path
unless that's averaged over time
acf's issue was forward path from arp
err and not just arp
going via ntt in san jose was also broken
reverse path is: trit > he
i'd suspect that trit->he path
nlayer do heaps of icmp deprioritisation too
i'd do iperf in udp mode at low bandwidth
to check which direction
not that you can necessarily change anything
hence is the nature of the Internet / Inter webs
yeh
i feel better having more idea of where things are going wrong even if i can't change them :)
It surprising it actually works at all
try 20% packet loss
that is hell to use
one time i was playing dota and there was a ddos attack and had 50% packet loss
and the game was going terribly
so i cehcked with mtr etc
and then i thought it was doing well considering there was ilke 50% packet loss
with ssh if there's a bit of packet loss often typing another key can help things along
ilek if something's not appearing you can press backspace or something
but if it's completely broken often it's better not to touch anything at all
and have the connection time out / disconnect
I always use tmux anyway so the session stays alive
ahh yip
oh was that you that posted to outages@ acf? :)
who/what is acf?
the guy who brought up comcast/verizon packet loss before
or did i get it wrong?
 http://kremvax.acfsys.net/smokeping.cgi?target=Remote.verizon-snloca
he linked that
well someone posted to outages@
who uses arp
https://puck.nether.net/pipermail/outages/2014-February/006596.html
oh ok..
I keep trying tmux,  then switching back to screen.  hehe
I like its default config, works out of the box
screen, I keep needing to paste in configs before I can use it
also the splitting of windows / panes is nice
I keep getting fumbled up by the default keybindings in tmux,  so used to screen's
I need a basic "idiots guide to tmux"  and just start using it with irssi.  when I get more comfortable with it,  then install it on all my machines with puppet.
http://www.amazon.com/tmux-Productive-Development-Brian-Hogan/dp/1934356964/ref=sr_1_1?ie=UTF8&qid=1392089683&sr=8-1&keywords=tmux
Amazon: "tmux: Productive Mouse-Free Development"
keybinding should be pretty basic to reconfigure
although you might not want to for sake of running screen within a tmux session
there are books on tmux?
wow
open resolver project is handy for identifying open resolvers on a network
http://openresolverproject.org
re: the ntpd thing, the default config for freebsd was changed when that vuln was identified, and freebsd10 ships with the modified config by default
http://www.itnews.com.au/News/372033,worlds-largest-ddos-strikes-us-europe.aspx
there was a ddos today
apparentyl 400 gigabit
oh and it was using ntp
did up_the_irons reports all come today?
whoa
jeez, that's enormous
so that could haev effected canda traffic
canada
cool, equiv of openresolverproject for ntpd
also, cloudflare's not on aws, but whatever, hahaha
i dunno
cloudflare is terrible
they may haev some stuff on aws
maybe staging or something, but they pride themselves on owning their hardware
i haven't found anything about this ddos on nanog yet
i was avoiding reading nanog to not get swamped :)
didn't see anything in nanog digests today
i don't know if 400 gigabit is actually the biggest ddos too
i been reading this carrier comparison
for some reason i can't find any other articles or mentioning of ddos
mercutio: all today, yeah
in fact, i dunno why i didn't look before, but like 30 minutes ago i noticed all our egress links are at like 300 Mbps!
ouch
lots of VPS' participating in the attacks (i'm sure innocent victims)
so i'm going to be blocking all NTP inbound
hmm
on all hosts
probably prudent
as a stop gap until people start fixing their setup
there's some debate whether it's a good idea to block all ntp
as some ntp like to use the same source/dest port
but yeah as stop-gap it makes a hell of a lot of sense
there's no debate in my mind when my network is hitting some target with > 1 Gbps of UDP
heh
well the debate was whether it shoudl be rate limited
or blocked ocmpletely
i reckon blocked completely
i'm kind of against rate limiting
rate limiting won't do shit
i mean, it will, but if 99% of the incoming is illegit traffic
your rate limit will effectively block all legit traffic too
so wtf
rather
it won't matter
hmm
won't people be hitting that 5 megabit udp rate limit anyway?
i shouldn't distract you
that's only in one direction
the wrong direction ;)
and  yes, i'll take questions later :)
oof
for freebsd guests: http://www.freebsd.org/security/advisories/FreeBSD-SA-14:02.ntpd.asc
i still can't see anything on nanog
i wonder if tehre's another mailing list i should follow too
someone posted about it on nznog
So I saw on some intertwitters about ntp blockage?
I don't have ntp running, but now that you mention it, is there an internal ntp server that can be peered with at the moment?
pcn: its inbound ntp requests, outbound as to get time from say pool.ntp.org should work just fine
OK
it's still valid question
i don't know of any
i think i just use pool.ntp.org
whats the nmap check or ntp check to ensure a host isnt configured incorrectly so as to be used in a UDP / ntp based DDOS attack?
uhh is saw something somewhere
<http://nmap.org/nsedoc/scripts/ntp-monlist.html>
mnathani: either upgrade ntp or just disable monlist command
up_the_irons: that nmap thing checks for monlist
so you could port scan your ranges if you wanted to find out who is vulnerable to it
mercutio: oh sweet
which frmo your own ip could prob bypass any blocks
this is old, but still looks like it'd work:
http://railspikes.com/2007/6/1/rails-email-processing
no need to set up procmail or Postfix filter to fork into ruby process.  just have a daemon check a special email box!
cool.
not that it really matters which way it is done
up_the_irons: roger
mercutio: yeah, I see that in the smokeping
after prodding NTT a bit more
http://paste.unixcube.org/k/246aaa
acf_: was it you that posted to outages@?
hmm? I emailed noc@us.ntt.net again
not technical in nature
oh i just saw that same address as you were saying on outages mailing list
how do i search scrollback? :)
peering disagreement or something likely
so it wasn't you that posted to oustages mailing list?
nope
https://puck.nether.net/pipermail/outages/2014-February/006596.html
maybe it not someone in irc even
any connection to the recent ddos news things you think?
i was wondering that
but i don't think it is
esp with your email response
it's ntt getting into messy situation like cogent
with not wanting to pay to send data
i imagine
wow. that guy sounds exactly like me
see how i wondered?
he even on arp :)
idk if "not technical in nature" means ntt/verizon is purpousely degrading connectivity
could be
or just that verizon/ntt have to negotiate bigger pipes to take the data
did you see the uhh
god damnit
i weant to find a way to find urls i pasted to irc :)
 http://arstechnica.com/information-technology/2014/02/netflix-performance-on-verizon-and-comcast-has-been-dropping-for-months/
Ars Technica: "Netflix performance on Verizon and Comcast has been dropping for months"
is that something bryce can do?
it's the same two providers
even if diff origin
i think ntt is generally considered tier 1 and cogent not though?
but they're both huge
yeah
cogent is usually considered tier crap afaik
heh i was reading on nanog about cogent
again :/
but yeah i not a fan
godaddy is crap tier too
i still reckon up_the_irons should just route verizon/comcast a different way
maybe with max prefix limit
a bit of testing seems to reveal that rerouting through nlayer would still go through ntt
these things never seem to get fixed very quickly
and it usually gets worse before it gets better
we'll probably have to wait for level3 if up_the_irons wants to reroute
oh
idk much though
he has tata too
but i dunno
level3 is sure to fix it
how did you test via nlayer?
yeah, it doesn't look like we're near the end of this
oh from a lg?
i exepct level3 shouldn't take long to get connected up
i imagine it's just however long it takes to get a cross connect
which should be quicker for a big data centre you'd think
i mean they could probably turn it up tommorow if they felt like it
but if you tried to ask for tommorow they'd probably want to charge heaps for urgency
yeah. I certainly hope it's soon
well, nlayer-> verizon is direct
but he'd probably have to go there to plug in cross connect
but verizon-> nlayer is via ntt
oh
but it's -> verizon that is bad
judging by my routing via verizon being fine
so maybe that would fix it
i doubt reverse path is via verizon
but i dunno trace me ?
202.49.67.22
(from verizon) verizon->ntt->new zealand stuff
i actually think the routing side of things is somewhere the internet could really improve
so it's ntt return with no packet loss
which city is the verizon-> ntt in ?
oh of course california
is it san jose or los angeles?
or something else?
lax for me
and yeh we tested sending via ntt in la and sj
and both were bad
and sending via verizon in la was fine
yeah, no packet loss to you
you have nlayer path to verizon?
hmm
i misplaced your ip
i dunno what route it taking atm
108.40.173.223
yeh that forward path via verizon
uhh it's showing packet loss now
i wonder if it doesn't like two traceroutes at once
probably verizon just really sucks
i'll pingplotter from the other ip
yeah it looks fine
but i think it is rate limiting icmp too
oh no now some loss
this was all fine last night!
the internet is breaking down!
it does that
it's not as bad as before
it's 1.14%
and it was easily 7% usually going via ntt
http://kremvax.acfsys.net/smokeping.cgi?target=Remote.verizon-lsanca
yeh i saw that earlier today
the overnight thing is like wow
i can smokeping you from nz maybe?
that overnight thing was awesome
if it's clean it suggests that it's single direction
the latency dropped from 30-40ms. I wonder if the ARP route changed
i doubit it
it lpooks like congestion
but it's hard to know
it dropped off a bit sharply. does it do that?
you mind if i add you to my smokeping?
go for it
sometimes
i have curl testing too
but i assume you're not hosting any files on your dsl :)
nope
what was that comcast ip?
oh it was comcast.net
also 72.55.8.69
btw my friend in sj on comcast cable wasn't packet loss
what's 72.55.8.69?
that's via level 3 forwaered route
the non-rate-limiting router in front of work's internet
work blocks icmp :0
ahh
is it on level3?
that's comcast
normal comcast is via ntt
whereas this is via level3
no ip's even say comcast.net
strange. I think they have some legacy IP address block, but I didn't think it would affect routing
it says comcast business
yeah, that's it
as13385
wheras comcast.net is AS7922
ok first syas comcast telecommunications, second says comcast cable
both go ntt-> comcast via tata over arp
not from here though
i'll do b oth
arp doesn't have level3 yet
it will be interesting to see the difference
maybe i shoudl do a subgroup
nah screw it
i wnat to be able to subgroup and not subgroup at the same time
it's handy scrolling through liwst