[00:19] *** dj_goku has joined #arpnetworks
[01:00] <up_the_irons> liking the BIRD filter / function language, even though it seems a bit weird at first
[01:00] <up_the_irons>   import filter {
[01:00] <up_the_irons>     bgp_community.add((our_asn,20000));
[01:00] <up_the_irons>     accept;
[01:00] <up_the_irons>   };
[01:00] <up_the_irons> easy enough
[01:01] <up_the_irons> a little bit weird is, show all HE routes:
[01:01] <up_the_irons> sh ro filter { if 6939 = bgp_path.first then accept; }
[01:01] <up_the_irons> seem verbose, but meh, it works
[01:09] *** tehfink has joined #arpnetworks
[02:14] *** tehfink has quit IRC (Quit: tehfink)
[02:40] <mercutio> i like openbgpd syntax etc
[02:40] <mercutio> but bird is faster at converging from what i understand
[02:40] <mercutio> and is supported on linux
[02:41] <mercutio> i'd like to see openbgp for linux though
[02:42] *** tehfink has joined #arpnetworks
[03:04] <mercutio> i left that comcast trace going, and there's still no packet loss to comcast
[03:05] <mercutio> i think i need to do it during earlier hours
[03:05] *** tehfink has quit IRC (Quit: tehfink)
[03:49] <up_the_irons> mercutio: bird seems to be insanely fast; peers go into "Established" state almost instantly after I reload config with a new peer
[04:14] <jpalmer> up_the_irons: pulp = repo management,  candlepin = subscription management (system A has repos A,B, and F, but not D or E)  etc
[04:14] <up_the_irons> jpalmer: oh cool
[04:15] <jpalmer> foreman is a frontend dashboard, and ENC to puppet.
[04:17] <up_the_irons> cool
[04:42] *** tehfink has joined #arpnetworks
[04:44] *** kevr has quit IRC (Read error: Operation timed out)
[04:50] *** kevr has joined #arpnetworks
[05:00] *** tehfink has quit IRC (Quit: tehfink)
[05:03] *** tehfink has joined #arpnetworks
[05:11] *** kevr has quit IRC (Ping timeout: 252 seconds)
[05:13] *** tehfink has quit IRC (Quit: tehfink)
[05:17] *** kevr has joined #arpnetworks
[05:55] *** tehfink has joined #arpnetworks
[06:44] *** toddf has quit IRC (Quit: leaving)
[06:47] *** toddf has joined #arpnetworks
[06:47] *** ChanServ sets mode: +o toddf
[09:09] *** tehfink has quit IRC (Quit: tehfink)
[09:41] *** NiTeMaRe has quit IRC (Ping timeout: 265 seconds)
[09:43] *** NiTeMaRe has joined #arpnetworks
[10:14] *** SpeedBus has quit IRC (Ping timeout: 245 seconds)
[10:35] *** pjs_ has quit IRC (Quit: EPIC5-1.1.2[1638] - amnesiac : Help! The paranoids are out to get me!)
[10:35] *** pjs has joined #arpnetworks
[10:45] *** SpeedBus has joined #arpnetworks
[12:02] *** hazardous has quit IRC (*.net *.split)
[12:02] *** gizmoguy has quit IRC (*.net *.split)
[12:02] *** plett has quit IRC (*.net *.split)
[12:03] *** plett has joined #arpnetworks
[12:03] *** gizmoguy has joined #arpnetworks
[12:04] *** laotzi has joined #arpnetworks
[12:05] *** jpalmer has quit IRC (Excess Flood)
[12:05] *** jpalmer has joined #arpnetworks
[13:11] <mercutio> up_the_irons: does that mean any2xi is up now?
[13:33] *** hp_ has joined #arpnetworks
[13:34] *** hp_ is now known as Guest58998
[14:01] <mercutio> http://arstechnica.com/information-technology/2014/02/netflix-performance-on-verizon-and-comcast-has-been-dropping-for-months/
[14:01] <BryceBot> Ars Technica: "Netflix performance on Verizon and Comcast has been dropping for months"
[14:01] <mercutio> it's interesting that verizon and comcast were the two destinations ntt were haveing issues too
[14:01] <mercutio> s/too/to/
[14:01] <BryceBot> <mercutio> it's interesting that verizon and comcast were the two destinations ntt were haveing issues to
[14:06] <Guest58998> Hey. I could use some help on an ipv6 /48 ubuntu configuration. No matter the search query in google I can't seem to find anyone that describes it the way arp networks does. Someone know how to set up the /48 on a single Ubuntu VPS?
[14:09] <toddf> at one point it was routed to your vps. at one point the lowest /64 was an ethernet segment and the rest was avilable on a support ticket request basis for routing. I'm not sure what the defaults are at this point. if you're a recent customer, just try setting the lowest /64 subnet on your ethernet segment and see how that goes .. try fe80::1 and <yourv6network>::1 for a default router, one of those should work. perhaps there's a ...
[14:09] <toddf> ... wiki page I'm unaware of. hope that helps.
[14:10] <Guest58998> I'm told it has been routed to link-local and that I should set my side to fe80::2/64
[14:10] <Guest58998> I'm not exactly sure what they mean by "my side".. default gateway, local address or?
[14:11] <brycec> http://wiki.arpnetworks.com/wiki/48%20IPv6%20on%20OpenBSD is good reference
[14:11] <brycec> So "set your side to..." means set the IP on the interface to fe80::2
[14:11] <brycec> the default gateway is fe80::1
[14:11] <brycec> (because ARP's side is fe80::1 and routing the /48 to fe80::2)
[14:12] <brycec> (Also: Requisite "if you don't know how to do this stuff, then you probably shouldn't be messing with it.")
[14:13] <mercutio> you shouldn't need a /48
[14:13] <Guest58998> Oh I know it's expert only ;) But I have other servers where /48 is routed differently (I think)
[14:13] <Guest58998> but I do
[14:13] <mercutio> basically
[14:14] <Guest58998> but yeah I know that I shouldn't
[14:16] <brycec> ARP's method of routing is actually pretty common too, fwiw.
[14:17] <brycec> Though the majority of tutorials and howtos are written for people with HE tunnels and the like, so I can see how that drowns out the useful information.
[14:17] <Guest58998> I dont doubt it. It's just really hard to find it described that way anywhere else
[14:18] <Guest58998> yep, it's mostly two lines about native ipv6 and then 4 pages about tunnels
[14:18] <mercutio> native ipv6 is easy though
[14:19] <brycec> Yep.
[14:19] <brycec> And the /48 too once you realize it's two lines or so
[14:20] <Guest58998> I got some ipv6 connectivity now. Thanks a lot for your help
[14:23] *** Guest58998 has quit IRC (Quit: Leaving)
[14:26] <toddf> arp's default config is great for a single vps. if you have multiple, you have to route v6 to the others from your first vps, or ask arpnetworks for changes.  I opted for plan b *grin*, one /64 on the ethernet segment.
[14:45] <m0unds> yea, i wrote a post about configuring SRX devices with a roll-your-own ipv6 tunnel in flow mode because so many of the HE tunnel broker tutorials are silly and tell you to switch off flow mode on your appliance and stuff
[14:45] <m0unds> hopefully it'll help someone sooner or later - same with working srcnat for xbox live, since it seems people way overthink that stuff
[15:15] <brycec> heh
[15:15] <brycec> m0unds: link?
[15:45] <up_the_irons> mercutio: not yet
[15:47] <up_the_irons> toddf: for the record, our default is no routing at all, just /64 on your VLAN, so no single vps is a point of failure
[15:47] *** up_the_irons has quit IRC (Read error: Operation timed out)
[15:47] <toddf> up_the_irons: ah. I've been around too long to know what the actual current default is, hope I made that clear above ;-)
[15:48] *** up_the_irons has joined #arpnetworks
[15:48] *** ChanServ sets mode: +o up_the_irons
[15:49] *** mhoran has quit IRC (Ping timeout: 246 seconds)
[15:49] <brycec> lol
[15:49] <brycec> sine up_the_irons missed it: 15:46:39 <@toddf> up_the_irons: ah. I've been around too long to know what the actual current default is, hope I made that clear above ;-)
[15:50] <brycec> *since
[15:50] <up_the_irons> brycec: tnx!
[15:51] <brycec> If you had multiple VPS'en and a /48, I suppose you could always CARP them all, but routing would be annoying/tricky.
[15:51] <up_the_irons> yeah
[15:52] <brycec> I still haven't worked out a good way to give my CARP backup IPv6 access to an HE tunnel :/ Not without watching for the state change and scripting route changes anyways.
[15:52] <brycec> (It's also not high on the priority list)
[16:03] <toddf> brycec: convince he.net it needs to do ospf6 with you and have two tunnels one to each router?
[16:04] <toddf> not always doable because some people only have a single ip, carp can be done in this case, but v4 connectivity is always fun in the backup router instance
[16:04] <brycec> Two tunnels but same subnet?
[16:04] <brycec> Not to worry, both routers have public v4 IPs
[16:04] <brycec> plus one shared
[16:04] <toddf> you'd need two tunnels and ospf6 should handle routing of the same subnet yes
[16:05] <brycec> Well all that's left is to convince HE of anything, lol
[16:05] <toddf> (note I've never heard of anyone doing it, but if you want to avoid scripting and wish to do it up proper...)
[16:06] <brycec> Yeah that would be proper. But given how much I'm paying them... I don't expect them to do anything "for me"
[16:06] <toddf> you could of course get two vps'en from arpnetworks and do ospf6 across two gif tunnels to your home for full redundancy on your side ..
[16:06] <toddf> they do permit bgp6 over a tunnel for a fee, if I read their website properly
[16:06] <brycec> That sounds like fun :) And I'm still meaning to move my IPv6 tunnels to ARP.  However lately, HE's reliability has been > ARP :(
[16:07] <brycec> toddf: Actually I can request a BGP tunnel for free
[16:08] <brycec> But first I'd need an ASN...
[16:08] <toddf> details
[16:08] <brycec> And only 7 POPs support it
[16:09] <brycec> (As in: not my closest POP)
[16:44] *** KDE_Perry has quit IRC (Ping timeout: 246 seconds)
[16:45] *** KDE_Perry has joined #arpnetworks
[16:46] <m0unds> brycec: http://chris.vanvoro.us/2013/12/26/fun-with-ipv6/
[16:47] <brycec> Thanks m0unds
[16:47] <m0unds> sure, it's not the best, but it's better than most of what i'd read, haha
[16:47] <staticsafe> that your site?
[16:49] <m0unds> yep, terribleness that it is
[16:49] <m0unds> octopress + nginx
[16:49] <m0unds> code repo on bitbucket
[16:50] <staticsafe> i was too lazy to octopress so i just went back to wordpress
[16:50] <m0unds> i started using nitrous.io as a quick IDE for posting
[16:51] <m0unds> i'm hosting a friend's wordpress site - it's the only reason i still have mysql and php running on my vps
[16:59] *** laotzi has quit IRC (Remote host closed the connection)
[17:09] <brycec> Man I have no idea why I thought this would be more difficult... Using my ARP VPS as a v6 tunnel endpoint accomplished! (Still need to setup routing and firewalling, but that's all)
[17:09] <brycec> thanks for the kick in the butt m0unds
[17:09] <m0unds> you betcha!
[17:09] <m0unds> i sat on mine for 6 mos before i did it
[17:10] <brycec> I'm over 1yr now
[17:10] <brycec> i think
[17:10] <m0unds> then got bored at work and went 'meh' and just did it
[17:10] <mercutio> apparently cogent -> verizon is even more broken than normal
[17:10] <brycec> tl;dr just need matching gif/v4tunnel/etc sections on both ends, that's it
[17:10] <m0unds> the srx part was what i hung up on initially though because i was on junos 11.4, which doesn't support ipv6 in flow mode
[17:10] <mercutio> cogent issues have been going on for something like two years now?
[17:11] <m0unds> so when i updated to the final build for my srx (discontinued model) it fixed it
[17:11] <brycec> cool, congrats
[17:11] <brycec> (my oldest invoice seems to be Nov 2012. Over a year now, woo)
[17:11] <mercutio> brycec: yeh it pretty simple to tunnel ipv6
[17:11] <mercutio> you may have to mss clamp if you're forwarding traffic though
[17:12] <brycec> Hardest part now is deciding on address allocations
[17:12] <mercutio> fac3 ?
[17:12] <brycec> I'll keep that in mind, thanks mercutio
[17:12] <brycec> lol
[17:12] <mercutio> i dunno :)
[17:12] <mercutio> err face would work too
[17:12] <brycec> face:b00c is pretty well-known ;)
[17:12] <mercutio> 1337 ?
[17:12] <mercutio> yeh i know
[17:13] <brycec> Yeah there are a bunch of "clever" ones out there. I'm far more practical.... But I can't just start at "1"
[17:13] <mercutio> you only have 16 bits to play with
[17:13] <brycec> (0 is already in use)
[17:13] <m0unds> hahaha
[17:13] <m0unds> i started at 2
[17:13] <brycec> 16 bits?
[17:13] *** laotzi has joined #arpnetworks
[17:13] <mercutio> 48 to 64
[17:13] <m0unds> my clients at home are 4, iirc
[17:13] <brycec> Oh sure, duh
[17:15] <mercutio> bcec ?
[17:15] <mercutio> removing r and y from your nick, that don't map to hex :)
[17:15] <brycec> ha
[17:16] <brycec> Probably gonna start at f00a
[17:16] <mercutio> it does kind of sound like "be sick" though
[17:16] <mercutio> or f00f like the pentium bug?
[17:16] <m0unds> like "sick" as in, WAY SICK DOODZ
[17:16] <brycec> So help whichever net ends up on f00f ;)
[17:17] * brycec spirals into the IPv6 "OMG SO MANY ADDRESSES" oblivion
[17:17] <mercutio> 1c12
[17:17] <mercutio> ( i see one too)
[17:17] <brycec> actually I should just migrate my current HE prefixes
[17:18] <mercutio> bryce: you only have 16 bits, it's not that many
[17:18] <mercutio> you need the /64 for autoconfig
[17:19] <brycec> 16 bits is till pretty big
[17:19] <brycec> (And I know I can't really sub-divide the /64)
[17:19] <mercutio> it never felt very big on pc's :)
[17:19] <mercutio> dammn those 64k memory limits
[17:20] <mercutio> it was a real pita
[17:20] <mercutio> but yeah it's a lot better than like 1 or 8 or such
[17:21] *** hazardous has joined #arpnetworks
[17:28] *** laotzi has quit IRC (Quit: SIGQUIT)
[17:28] <up_the_irons> for those running their own ntp server
[17:28] <up_the_irons> "1. If you run ntpd, upgrading to the latest version, which removes the "monlist" command that is used for these attacks; alternately, disabling the monitoring function by adding "disable
[17:28] <up_the_irons> +monitor" to your /etc/ntp.conf file."
[17:28] <up_the_irons> we're getting LOTS of notices for NTP-based UDP amplification attacks
[17:29] <brycec> up_the_irons: Any way to forward those notices to the responsible party?
[17:29] <brycec> *parties
[17:29] <up_the_irons> brycec: i am in the process of doing so, yes
[17:29] <up_the_irons> a very big time suck
[17:29] <up_the_irons> 39 notices based on IP.  gotta lookup the IP, get email of customer, then foward.
[17:30] <up_the_irons> *forward
[17:30] *** dne has quit IRC (Ping timeout: 264 seconds)
[17:30] *** Spitfire has quit IRC (Ping timeout: 264 seconds)
[17:30] <brycec> Bummer
[17:31] *** Yamazaki-kun has quit IRC (Ping timeout: 245 seconds)
[17:32] <up_the_irons> maybe i could write some filter..
[17:32] <brycec> (Oh good, I was already secure)
[17:33] <up_the_irons> procmail or something
[17:35] *** Spitfire has joined #arpnetworks
[17:37] *** dne has joined #arpnetworks
[17:39] *** Yamazaki-kun has joined #arpnetworks
[17:41] <up_the_irons> actually, would anyone *else* like to write something?  I'll pay (obviously).  Basic flow would be: 1) I get an abuse complaint, 2) i forward to some special address, 3) something / script on that address looks up IP with regex, 4) IP returns an email address (with our REST API), 5) forward that email
[17:41] <up_the_irons> or, pointers to how this would be done would help
[17:41] <up_the_irons> i can try to code something up
[17:45] *** laotzi has joined #arpnetworks
[17:46] *** Yamazaki-kun has quit IRC (Ping timeout: 245 seconds)
[17:46] <brycec> Seems straight-forward enough
[17:48] <mercutio> up_the_irons: it very well could happen for dns too
[17:48] <up_the_irons> mercutio: dns?
[17:48] <mercutio> so maybe having an easy way to email ip's would be good
[17:48] <mercutio> up_the_irons: the any thing, and open recursive are being hit on authorative and recursive a lot recently too
[17:48] <up_the_irons> mercutio: if you mean the amplification attacks, yes, very much so
[17:49] <mercutio> up_the_irons: what about having a sepcial domain you email with users ip@blah.arpnetworks.com
[17:49] <mercutio> or such
[17:50] <mercutio> and then it emails the right person, and a sepcial mailbox to keep note
[17:50] <mercutio> which would just mean cutting and pasting the ip
[17:50] <mercutio> which isn't automated, but is simpler to test, ..
[17:50] <up_the_irons> mercutio: ah true
[17:50] <up_the_irons> mercutio: i like it
[17:51] <up_the_irons> I LIKE IT
[17:51] <mercutio> can you map from ip to user with a mysql query or such?
[17:52] <up_the_irons> more like a bit of ruby
[17:52] <brycec> 4) IP returns an email address (with our REST API)"
[17:52] <brycec> (obviously not a public REST API ;) )
[17:52] <up_the_irons> obviously :)
[17:53] <mercutio> i wonder if for things like recursive dns there should be tests every now and then
[17:54] *** Yamazaki-kun has joined #arpnetworks
[17:54] <mercutio> but with a little script magic if such a system was setup it'd be easy to email effected users
[17:54] <mercutio> err affected?
[17:54] <brycec> ^
[17:55] *** mhoran has joined #arpnetworks
[17:55] *** ChanServ sets mode: +o mhoran
[18:01] <mercutio> hmm as an addition could have some extra things to bounce to which would send automated message content that say how to fix open dns etc
[18:01] <mercutio> or maybe just keep a list of the various things, and people can parse themselves.
[18:02] <mercutio> for ntp i'm in favour of openntpd which doesn't listen by default
[18:03] <up_the_irons> affected
[18:21] <jpalmer> are there any web based test tools for the NTP or DNS amplification attacks yet?
[18:22] <jpalmer> (I don't know how to exploit it offhand,  but would like to verify my DNS and NTP servers are ok.
[18:23] <mercutio> host -t any <your domain name>
[18:24] <mercutio> the any thing is complicated
[18:24] <mercutio> basically more providers need to do bcp38 to improve the situation
[18:24] <mercutio> as the predominant issue is that it's valid to do an any request for a domain name.
[18:25] <jpalmer> yeah, that returns several records for all of my domains.
[18:27] <mercutio> see that's normal
[18:27] <mercutio> now the problem is someone can spoof an address
[18:28] <mercutio> so that your response goes to another address
[18:28] <mercutio> esp if one has lots of entries
[18:28] <mercutio> like say host -t any microsoft.com has quite a bit of data
[18:30] <mercutio> it's only like 4x amplification normally with that htough
[18:30] <mercutio> but still if it's 10 megabit of requests that makes 40 megabit of response
[18:30] <jpalmer> yep yep
[18:30] <mercutio> arp defaults to 5 megabit rate limit for udp, so you'll only be able to return 5 megabit
[18:30] <mercutio> but that could impact other services..
[18:31] <mercutio> generally speaking most people seem to be ignoring the amplification attack and suggesting that it's the people sending spoofed requests that are the problem
[18:31] <jpalmer> heh.   it's not the misconfigured SMTP servers,  it's the spammers!
[18:32] <mercutio> well udp will limit what response size normally
[18:32] <mercutio> and tcp won't work
[18:32] <mercutio> bcp38 means people can't spoof addresses as easily so it cna't work as easily
[18:33] <mercutio> from memory comcast is the biggest provider with no protection
[18:33] <jpalmer> I'll have to read up on bcp38.  not familiar with it.
[18:33] <mercutio> https://www.nanog.org/sites/default/files/mon_general_weber_defeat_23.pdf
[18:33] <mercutio> it only really matters for providers
[18:34] <mercutio> basically it means that you can't send packets with my source ip address
[18:34] <mercutio> which arp do btw
[18:35] <mercutio> but basically if the any requests aren't terribly long it's probably mostly ok
[18:45] <mnathani> Are there network anomalies at present? I am getting about 2% packetloss from Toronto
[18:46] <mercutio> via ntt?
[18:48] <mercutio> ntt -> verizon still seems lossy
[18:49] <mercutio> acf had a smokeping
[18:50] <mercutio> uhh
[18:50] <mercutio> acf's smokeping was really good in the middle of the night
[18:50] <mercutio> and his comcast gets better earlier
[18:51] <mercutio> acf: did you check out your smpkeping?
[18:57] <mnathani> via nlayer / mzima
[18:57] <mercutio> oh prob diff issue then
[18:57] <mercutio> 2% isn't so bad
[18:57] <mnathani> thats the forward path
[18:58] <mercutio> unless that's averaged over time
[18:58] <mercutio> acf's issue was forward path from arp
[18:58] <mercutio> err and not just arp
[18:58] <mercutio> going via ntt in san jose was also broken
[18:58] <mnathani> reverse path is: trit > he
[18:58] <mercutio> i'd suspect that trit->he path
[18:58] <mercutio> nlayer do heaps of icmp deprioritisation too
[18:59] <mercutio> i'd do iperf in udp mode at low bandwidth
[18:59] <mercutio> to check which direction
[18:59] <mercutio> not that you can necessarily change anything
[18:59] <mnathani> hence is the nature of the Internet / Inter webs
[18:59] <mercutio> yeh
[18:59] <mercutio> i feel better having more idea of where things are going wrong even if i can't change them :)
[18:59] <mnathani> It surprising it actually works at all
[19:00] <mercutio> try 20% packet loss
[19:00] <mercutio> that is hell to use
[19:00] <mercutio> one time i was playing dota and there was a ddos attack and had 50% packet loss
[19:00] <mercutio> and the game was going terribly
[19:00] <mercutio> so i cehcked with mtr etc
[19:00] <mercutio> and then i thought it was doing well considering there was ilke 50% packet loss
[19:01] <mercutio> with ssh if there's a bit of packet loss often typing another key can help things along
[19:01] <mercutio> ilek if something's not appearing you can press backspace or something
[19:02] <mercutio> but if it's completely broken often it's better not to touch anything at all
[19:02] <mercutio> and have the connection time out / disconnect
[19:03] *** DaCa has quit IRC (Ping timeout: 252 seconds)
[19:15] <mnathani> I always use tmux anyway so the session stays alive
[19:16] <mercutio> ahh yip
[19:19] <mercutio> oh was that you that posted to outages@ acf? :)
[19:21] <mnathani> who/what is acf?
[19:22] <mercutio> the guy who brought up comcast/verizon packet loss before
[19:22] <mercutio> or did i get it wrong?
[19:22] <mercutio>  http://kremvax.acfsys.net/smokeping.cgi?target=Remote.verizon-snloca
[19:22] <mercutio> he linked that
[19:23] <mercutio> well someone posted to outages@
[19:23] <mercutio> who uses arp
[19:23] <mercutio> https://puck.nether.net/pipermail/outages/2014-February/006596.html
[19:29] <mnathani> oh ok..
[19:30] <jpalmer> I keep trying tmux,  then switching back to screen.  hehe
[19:31] <mnathani> I like its default config, works out of the box
[19:31] <mnathani> screen, I keep needing to paste in configs before I can use it
[19:32] <mnathani> also the splitting of windows / panes is nice
[19:33] <jpalmer> I keep getting fumbled up by the default keybindings in tmux,  so used to screen's
[19:34] <jpalmer> I need a basic "idiots guide to tmux"  and just start using it with irssi.  when I get more comfortable with it,  then install it on all my machines with puppet.
[19:35] <mnathani> http://www.amazon.com/tmux-Productive-Development-Brian-Hogan/dp/1934356964/ref=sr_1_1?ie=UTF8&qid=1392089683&sr=8-1&keywords=tmux
[19:35] <BryceBot> Amazon: "tmux: Productive Mouse-Free Development"
[19:36] <mnathani> keybinding should be pretty basic to reconfigure
[19:36] <mnathani> although you might not want to for sake of running screen within a tmux session
[19:43] <mercutio> there are books on tmux?
[19:43] <mercutio> wow
[19:44] <m0unds> open resolver project is handy for identifying open resolvers on a network
[19:44] <m0unds> http://openresolverproject.org
[19:45] <m0unds> re: the ntpd thing, the default config for freebsd was changed when that vuln was identified, and freebsd10 ships with the modified config by default
[19:47] <mercutio> http://www.itnews.com.au/News/372033,worlds-largest-ddos-strikes-us-europe.aspx
[19:47] <mercutio> there was a ddos today
[19:47] <mercutio> apparentyl 400 gigabit
[19:47] <mercutio> oh and it was using ntp
[19:48] <mercutio> did up_the_irons reports all come today?
[19:49] <m0unds> whoa
[19:49] <m0unds> jeez, that's enormous
[19:49] <mercutio> so that could haev effected canda traffic
[19:49] <mercutio> canada
[19:50] <m0unds> cool, equiv of openresolverproject for ntpd
[19:52] <m0unds> also, cloudflare's not on aws, but whatever, hahaha
[19:52] <mercutio> i dunno
[19:52] <mercutio> cloudflare is terrible
[19:52] <mercutio> they may haev some stuff on aws
[19:53] <m0unds> maybe staging or something, but they pride themselves on owning their hardware
[19:53] <mercutio> i haven't found anything about this ddos on nanog yet
[19:53] <mercutio> i was avoiding reading nanog to not get swamped :)
[19:53] <m0unds> didn't see anything in nanog digests today
[19:53] <mercutio> i don't know if 400 gigabit is actually the biggest ddos too
[19:54] <mercutio> i been reading this carrier comparison
[20:11] <mercutio> for some reason i can't find any other articles or mentioning of ddos
[20:15] <up_the_irons> mercutio: all today, yeah
[20:15] <up_the_irons> in fact, i dunno why i didn't look before, but like 30 minutes ago i noticed all our egress links are at like 300 Mbps!
[20:15] <mercutio> ouch
[20:16] <up_the_irons> lots of VPS' participating in the attacks (i'm sure innocent victims)
[20:16] <up_the_irons> so i'm going to be blocking all NTP inbound
[20:16] <mercutio> hmm
[20:16] <up_the_irons> on all hosts
[20:16] <mercutio> probably prudent
[20:17] <up_the_irons> as a stop gap until people start fixing their setup
[20:17] <mercutio> there's some debate whether it's a good idea to block all ntp
[20:17] <mercutio> as some ntp like to use the same source/dest port
[20:17] <mercutio> but yeah as stop-gap it makes a hell of a lot of sense
[20:17] <up_the_irons> there's no debate in my mind when my network is hitting some target with > 1 Gbps of UDP
[20:17] <mercutio> heh
[20:18] <mercutio> well the debate was whether it shoudl be rate limited
[20:18] <mercutio> or blocked ocmpletely
[20:18] <mercutio> i reckon blocked completely
[20:18] <mercutio> i'm kind of against rate limiting
[20:18] <up_the_irons> rate limiting won't do shit
[20:18] <up_the_irons> i mean, it will, but if 99% of the incoming is illegit traffic
[20:18] <up_the_irons> your rate limit will effectively block all legit traffic too
[20:19] <up_the_irons> so wtf
[20:19] <up_the_irons> rather
[20:19] <up_the_irons> it won't matter
[20:19] <mercutio> hmm
[20:19] <mercutio> won't people be hitting that 5 megabit udp rate limit anyway?
[20:19] <mercutio> i shouldn't distract you
[20:19] <up_the_irons> that's only in one direction
[20:19] <up_the_irons> the wrong direction ;)
[20:20] <up_the_irons> and  yes, i'll take questions later :)
[20:23] <m0unds> oof
[20:24] <m0unds> for freebsd guests: http://www.freebsd.org/security/advisories/FreeBSD-SA-14:02.ntpd.asc
[20:32] <mercutio> i still can't see anything on nanog
[20:32] <mercutio> i wonder if tehre's another mailing list i should follow too
[20:33] <mercutio> someone posted about it on nznog
[20:59] *** DaCa has joined #arpnetworks
[21:05] *** pcn has joined #arpnetworks
[21:07] <pcn> So I saw on some intertwitters about ntp blockage?
[21:07] <pcn> I don't have ntp running, but now that you mention it, is there an internal ntp server that can be peered with at the moment?
[21:09] <mnathani> pcn: its inbound ntp requests, outbound as to get time from say pool.ntp.org should work just fine
[21:24] <pcn> OK
[21:26] <mercutio> it's still valid question
[21:26] <mercutio> i don't know of any
[21:26] <mercutio> i think i just use pool.ntp.org
[21:35] <mnathani> whats the nmap check or ntp check to ensure a host isnt configured incorrectly so as to be used in a UDP / ntp based DDOS attack?
[21:35] <mercutio> uhh is saw something somewhere
[21:36] <mercutio> <http://nmap.org/nsedoc/scripts/ntp-monlist.html>
[21:46] *** BryceBot has quit IRC (Excess Flood)
[21:46] *** BryceBot has joined #arpnetworks
[22:19] <up_the_irons> mnathani: either upgrade ntp or just disable monlist command
[22:20] <mercutio> up_the_irons: that nmap thing checks for monlist
[22:21] <mercutio> so you could port scan your ranges if you wanted to find out who is vulnerable to it
[22:21] <up_the_irons> mercutio: oh sweet
[22:21] <mercutio> which frmo your own ip could prob bypass any blocks
[22:44] <up_the_irons> this is old, but still looks like it'd work:
[22:44] <up_the_irons> http://railspikes.com/2007/6/1/rails-email-processing
[22:45] <up_the_irons> no need to set up procmail or Postfix filter to fork into ruby process.  just have a daemon check a special email box!
[22:49] <mercutio> cool.
[22:49] <mercutio> not that it really matters which way it is done
[22:56] <mnathani> up_the_irons: roger
[23:05] <acf_> mercutio: yeah, I see that in the smokeping
[23:06] <acf_> after prodding NTT a bit more
[23:06] <acf_> http://paste.unixcube.org/k/246aaa
[23:07] <mercutio> acf_: was it you that posted to outages@?
[23:07] <acf_> hmm? I emailed noc@us.ntt.net again
[23:07] <mercutio> not technical in nature
[23:07] <mercutio> oh i just saw that same address as you were saying on outages mailing list
[23:08] <mercutio> how do i search scrollback? :)
[23:08] <acf_> peering disagreement or something likely
[23:08] <mercutio> so it wasn't you that posted to oustages mailing list?
[23:08] <acf_> nope
[23:08] <mercutio> https://puck.nether.net/pipermail/outages/2014-February/006596.html
[23:09] <mercutio> maybe it not someone in irc even
[23:09] <acf_> any connection to the recent ddos news things you think?
[23:09] <mercutio> i was wondering that
[23:09] <mercutio> but i don't think it is
[23:09] <mercutio> esp with your email response
[23:10] <mercutio> it's ntt getting into messy situation like cogent
[23:10] <mercutio> with not wanting to pay to send data
[23:10] <mercutio> i imagine
[23:10] <acf_> wow. that guy sounds exactly like me
[23:10] <mercutio> see how i wondered?
[23:10] <mercutio> he even on arp :)
[23:11] <acf_> idk if "not technical in nature" means ntt/verizon is purpousely degrading connectivity
[23:11] <mercutio> could be
[23:11] <acf_> or just that verizon/ntt have to negotiate bigger pipes to take the data
[23:11] <mercutio> did you see the uhh
[23:11] <mercutio> god damnit
[23:11] <mercutio> i weant to find a way to find urls i pasted to irc :)
[23:12] <mercutio>  http://arstechnica.com/information-technology/2014/02/netflix-performance-on-verizon-and-comcast-has-been-dropping-for-months/
[23:12] <BryceBot> Ars Technica: "Netflix performance on Verizon and Comcast has been dropping for months"
[23:12] <mercutio> is that something bryce can do?
[23:12] <mercutio> it's the same two providers
[23:12] <mercutio> even if diff origin
[23:13] <mercutio> i think ntt is generally considered tier 1 and cogent not though?
[23:13] <mercutio> but they're both huge
[23:13] <acf_> yeah
[23:13] <acf_> cogent is usually considered tier crap afaik
[23:13] <mercutio> heh i was reading on nanog about cogent
[23:13] <mercutio> again :/
[23:14] <mercutio> but yeah i not a fan
[23:14] <mnathani> godaddy is crap tier too
[23:14] <mercutio> i still reckon up_the_irons should just route verizon/comcast a different way
[23:14] <mercutio> maybe with max prefix limit
[23:14] <acf_> a bit of testing seems to reveal that rerouting through nlayer would still go through ntt
[23:14] <mercutio> these things never seem to get fixed very quickly
[23:14] <mercutio> and it usually gets worse before it gets better
[23:15] <acf_> we'll probably have to wait for level3 if up_the_irons wants to reroute
[23:15] <mercutio> oh
[23:15] <acf_> idk much though
[23:15] <mercutio> he has tata too
[23:15] <mercutio> but i dunno
[23:15] <mercutio> level3 is sure to fix it
[23:15] <mercutio> how did you test via nlayer?
[23:15] <acf_> yeah, it doesn't look like we're near the end of this
[23:15] <mercutio> oh from a lg?
[23:16] <mercutio> i exepct level3 shouldn't take long to get connected up
[23:16] <mercutio> i imagine it's just however long it takes to get a cross connect
[23:16] <mercutio> which should be quicker for a big data centre you'd think
[23:16] <mercutio> i mean they could probably turn it up tommorow if they felt like it
[23:17] <mercutio> but if you tried to ask for tommorow they'd probably want to charge heaps for urgency
[23:17] <acf_> yeah. I certainly hope it's soon
[23:17] <acf_> well, nlayer-> verizon is direct
[23:17] <mercutio> but he'd probably have to go there to plug in cross connect
[23:17] <acf_> but verizon-> nlayer is via ntt
[23:17] <mercutio> oh
[23:17] <mercutio> but it's -> verizon that is bad
[23:18] <mercutio> judging by my routing via verizon being fine
[23:18] <acf_> so maybe that would fix it
[23:18] <mercutio> i doubt reverse path is via verizon
[23:18] <mercutio> but i dunno trace me ?
[23:18] <mercutio> 202.49.67.22
[23:19] <acf_> (from verizon) verizon->ntt->new zealand stuff
[23:19] <mercutio> i actually think the routing side of things is somewhere the internet could really improve
[23:19] <mercutio> so it's ntt return with no packet loss
[23:19] <mercutio> which city is the verizon-> ntt in ?
[23:19] <mercutio> oh of course california
[23:19] <mercutio> is it san jose or los angeles?
[23:19] <mercutio> or something else?
[23:19] <acf_> lax for me
[23:20] <mercutio> and yeh we tested sending via ntt in la and sj
[23:20] <mercutio> and both were bad
[23:20] <mercutio> and sending via verizon in la was fine
[23:20] <acf_> yeah, no packet loss to you
[23:20] <acf_> you have nlayer path to verizon?
[23:20] <mercutio> hmm
[23:20] <mercutio> i misplaced your ip
[23:20] <mercutio> i dunno what route it taking atm
[23:20] <acf_> 108.40.173.223
[23:21] <mercutio> yeh that forward path via verizon
[23:21] <mercutio> uhh it's showing packet loss now
[23:21] <mercutio> i wonder if it doesn't like two traceroutes at once
[23:21] <acf_> probably verizon just really sucks
[23:21] <mercutio> i'll pingplotter from the other ip
[23:22] <mercutio> yeah it looks fine
[23:22] <mercutio> but i think it is rate limiting icmp too
[23:22] <mercutio> oh no now some loss
[23:23] <mercutio> this was all fine last night!
[23:23] <acf_> the internet is breaking down!
[23:23] <mercutio> it does that
[23:23] <mercutio> it's not as bad as before
[23:23] <mercutio> it's 1.14%
[23:23] <mercutio> and it was easily 7% usually going via ntt
[23:24] <acf_> http://kremvax.acfsys.net/smokeping.cgi?target=Remote.verizon-lsanca
[23:24] <mercutio> yeh i saw that earlier today
[23:24] <mercutio> the overnight thing is like wow
[23:24] <mercutio> i can smokeping you from nz maybe?
[23:24] <acf_> that overnight thing was awesome
[23:25] <mercutio> if it's clean it suggests that it's single direction
[23:25] <acf_> the latency dropped from 30-40ms. I wonder if the ARP route changed
[23:25] <mercutio> i doubit it
[23:25] <mercutio> it lpooks like congestion
[23:25] <mercutio> but it's hard to know
[23:25] <acf_> it dropped off a bit sharply. does it do that?
[23:25] <mercutio> you mind if i add you to my smokeping?
[23:26] <acf_> go for it
[23:26] <mercutio> sometimes
[23:26] <mercutio> i have curl testing too
[23:26] <mercutio> but i assume you're not hosting any files on your dsl :)
[23:26] <acf_> nope
[23:26] <mercutio> what was that comcast ip?
[23:26] <mercutio> oh it was comcast.net
[23:27] <acf_> also 72.55.8.69
[23:27] <mercutio> btw my friend in sj on comcast cable wasn't packet loss
[23:27] <mercutio> what's 72.55.8.69?
[23:27] <mercutio> that's via level 3 forwaered route
[23:27] <acf_> the non-rate-limiting router in front of work's internet
[23:27] <acf_> work blocks icmp :0
[23:27] <mercutio> ahh
[23:28] <mercutio> is it on level3?
[23:28] <acf_> that's comcast
[23:28] <mercutio> normal comcast is via ntt
[23:28] <mercutio> whereas this is via level3
[23:28] <mercutio> no ip's even say comcast.net
[23:28] <acf_> strange. I think they have some legacy IP address block, but I didn't think it would affect routing
[23:29] <mercutio> it says comcast business
[23:29] <acf_> yeah, that's it
[23:29] <mercutio> as13385
[23:29] <mercutio> wheras comcast.net is AS7922
[23:30] <mercutio> ok first syas comcast telecommunications, second says comcast cable
[23:30] <acf_> both go ntt-> comcast via tata over arp
[23:30] <mercutio> not from here though
[23:30] <mercutio> i'll do b oth
[23:31] <mercutio> arp doesn't have level3 yet
[23:31] <acf_> it will be interesting to see the difference
[23:31] <mercutio> maybe i shoudl do a subgroup
[23:32] <mercutio> nah screw it
[23:32] <mercutio> i wnat to be able to subgroup and not subgroup at the same time
[23:32] <mercutio> it's handy scrolling through liwst