[00:19] *** dj_goku has joined #arpnetworks [01:00] <up_the_irons> liking the BIRD filter / function language, even though it seems a bit weird at first [01:00] <up_the_irons> import filter { [01:00] <up_the_irons> bgp_community.add((our_asn,20000)); [01:00] <up_the_irons> accept; [01:00] <up_the_irons> }; [01:00] <up_the_irons> easy enough [01:01] <up_the_irons> a little bit weird is, show all HE routes: [01:01] <up_the_irons> sh ro filter { if 6939 = bgp_path.first then accept; } [01:01] <up_the_irons> seem verbose, but meh, it works [01:09] *** tehfink has joined #arpnetworks [02:14] *** tehfink has quit IRC (Quit: tehfink) [02:40] <mercutio> i like openbgpd syntax etc [02:40] <mercutio> but bird is faster at converging from what i understand [02:40] <mercutio> and is supported on linux [02:41] <mercutio> i'd like to see openbgp for linux though [02:42] *** tehfink has joined #arpnetworks [03:04] <mercutio> i left that comcast trace going, and there's still no packet loss to comcast [03:05] <mercutio> i think i need to do it during earlier hours [03:05] *** tehfink has quit IRC (Quit: tehfink) [03:49] <up_the_irons> mercutio: bird seems to be insanely fast; peers go into "Established" state almost instantly after I reload config with a new peer [04:14] <jpalmer> up_the_irons: pulp = repo management, candlepin = subscription management (system A has repos A,B, and F, but not D or E) etc [04:14] <up_the_irons> jpalmer: oh cool [04:15] <jpalmer> foreman is a frontend dashboard, and ENC to puppet. [04:17] <up_the_irons> cool [04:42] *** tehfink has joined #arpnetworks [04:44] *** kevr has quit IRC (Read error: Operation timed out) [04:50] *** kevr has joined #arpnetworks [05:00] *** tehfink has quit IRC (Quit: tehfink) [05:03] *** tehfink has joined #arpnetworks [05:11] *** kevr has quit IRC (Ping timeout: 252 seconds) [05:13] *** tehfink has quit IRC (Quit: tehfink) [05:17] *** kevr has joined #arpnetworks [05:55] *** tehfink has joined #arpnetworks [06:44] *** toddf has quit IRC (Quit: leaving) [06:47] *** toddf has joined #arpnetworks [06:47] *** ChanServ sets mode: +o toddf [09:09] *** tehfink has quit IRC (Quit: tehfink) [09:41] *** NiTeMaRe has quit IRC (Ping timeout: 265 seconds) [09:43] *** NiTeMaRe has joined #arpnetworks [10:14] *** SpeedBus has quit IRC (Ping timeout: 245 seconds) [10:35] *** pjs_ has quit IRC (Quit: EPIC5-1.1.2[1638] - amnesiac : Help! The paranoids are out to get me!) [10:35] *** pjs has joined #arpnetworks [10:45] *** SpeedBus has joined #arpnetworks [12:02] *** hazardous has quit IRC (*.net *.split) [12:02] *** gizmoguy has quit IRC (*.net *.split) [12:02] *** plett has quit IRC (*.net *.split) [12:03] *** plett has joined #arpnetworks [12:03] *** gizmoguy has joined #arpnetworks [12:04] *** laotzi has joined #arpnetworks [12:05] *** jpalmer has quit IRC (Excess Flood) [12:05] *** jpalmer has joined #arpnetworks [13:11] <mercutio> up_the_irons: does that mean any2xi is up now? [13:33] *** hp_ has joined #arpnetworks [13:34] *** hp_ is now known as Guest58998 [14:01] <mercutio> http://arstechnica.com/information-technology/2014/02/netflix-performance-on-verizon-and-comcast-has-been-dropping-for-months/ [14:01] <BryceBot> Ars Technica: "Netflix performance on Verizon and Comcast has been dropping for months" [14:01] <mercutio> it's interesting that verizon and comcast were the two destinations ntt were haveing issues too [14:01] <mercutio> s/too/to/ [14:01] <BryceBot> <mercutio> it's interesting that verizon and comcast were the two destinations ntt were haveing issues to [14:06] <Guest58998> Hey. I could use some help on an ipv6 /48 ubuntu configuration. No matter the search query in google I can't seem to find anyone that describes it the way arp networks does. Someone know how to set up the /48 on a single Ubuntu VPS? [14:09] <toddf> at one point it was routed to your vps. at one point the lowest /64 was an ethernet segment and the rest was avilable on a support ticket request basis for routing. I'm not sure what the defaults are at this point. if you're a recent customer, just try setting the lowest /64 subnet on your ethernet segment and see how that goes .. try fe80::1 and <yourv6network>::1 for a default router, one of those should work. perhaps there's a ... [14:09] <toddf> ... wiki page I'm unaware of. hope that helps. [14:10] <Guest58998> I'm told it has been routed to link-local and that I should set my side to fe80::2/64 [14:10] <Guest58998> I'm not exactly sure what they mean by "my side".. default gateway, local address or? [14:11] <brycec> http://wiki.arpnetworks.com/wiki/48%20IPv6%20on%20OpenBSD is good reference [14:11] <brycec> So "set your side to..." means set the IP on the interface to fe80::2 [14:11] <brycec> the default gateway is fe80::1 [14:11] <brycec> (because ARP's side is fe80::1 and routing the /48 to fe80::2) [14:12] <brycec> (Also: Requisite "if you don't know how to do this stuff, then you probably shouldn't be messing with it.") [14:13] <mercutio> you shouldn't need a /48 [14:13] <Guest58998> Oh I know it's expert only ;) But I have other servers where /48 is routed differently (I think) [14:13] <Guest58998> but I do [14:13] <mercutio> basically [14:14] <Guest58998> but yeah I know that I shouldn't [14:16] <brycec> ARP's method of routing is actually pretty common too, fwiw. [14:17] <brycec> Though the majority of tutorials and howtos are written for people with HE tunnels and the like, so I can see how that drowns out the useful information. [14:17] <Guest58998> I dont doubt it. It's just really hard to find it described that way anywhere else [14:18] <Guest58998> yep, it's mostly two lines about native ipv6 and then 4 pages about tunnels [14:18] <mercutio> native ipv6 is easy though [14:19] <brycec> Yep. [14:19] <brycec> And the /48 too once you realize it's two lines or so [14:20] <Guest58998> I got some ipv6 connectivity now. Thanks a lot for your help [14:23] *** Guest58998 has quit IRC (Quit: Leaving) [14:26] <toddf> arp's default config is great for a single vps. if you have multiple, you have to route v6 to the others from your first vps, or ask arpnetworks for changes. I opted for plan b *grin*, one /64 on the ethernet segment. [14:45] <m0unds> yea, i wrote a post about configuring SRX devices with a roll-your-own ipv6 tunnel in flow mode because so many of the HE tunnel broker tutorials are silly and tell you to switch off flow mode on your appliance and stuff [14:45] <m0unds> hopefully it'll help someone sooner or later - same with working srcnat for xbox live, since it seems people way overthink that stuff [15:15] <brycec> heh [15:15] <brycec> m0unds: link? [15:45] <up_the_irons> mercutio: not yet [15:47] <up_the_irons> toddf: for the record, our default is no routing at all, just /64 on your VLAN, so no single vps is a point of failure [15:47] *** up_the_irons has quit IRC (Read error: Operation timed out) [15:47] <toddf> up_the_irons: ah. I've been around too long to know what the actual current default is, hope I made that clear above ;-) [15:48] *** up_the_irons has joined #arpnetworks [15:48] *** ChanServ sets mode: +o up_the_irons [15:49] *** mhoran has quit IRC (Ping timeout: 246 seconds) [15:49] <brycec> lol [15:49] <brycec> sine up_the_irons missed it: 15:46:39 <@toddf> up_the_irons: ah. I've been around too long to know what the actual current default is, hope I made that clear above ;-) [15:50] <brycec> *since [15:50] <up_the_irons> brycec: tnx! [15:51] <brycec> If you had multiple VPS'en and a /48, I suppose you could always CARP them all, but routing would be annoying/tricky. [15:51] <up_the_irons> yeah [15:52] <brycec> I still haven't worked out a good way to give my CARP backup IPv6 access to an HE tunnel :/ Not without watching for the state change and scripting route changes anyways. [15:52] <brycec> (It's also not high on the priority list) [16:03] <toddf> brycec: convince he.net it needs to do ospf6 with you and have two tunnels one to each router? [16:04] <toddf> not always doable because some people only have a single ip, carp can be done in this case, but v4 connectivity is always fun in the backup router instance [16:04] <brycec> Two tunnels but same subnet? [16:04] <brycec> Not to worry, both routers have public v4 IPs [16:04] <brycec> plus one shared [16:04] <toddf> you'd need two tunnels and ospf6 should handle routing of the same subnet yes [16:05] <brycec> Well all that's left is to convince HE of anything, lol [16:05] <toddf> (note I've never heard of anyone doing it, but if you want to avoid scripting and wish to do it up proper...) [16:06] <brycec> Yeah that would be proper. But given how much I'm paying them... I don't expect them to do anything "for me" [16:06] <toddf> you could of course get two vps'en from arpnetworks and do ospf6 across two gif tunnels to your home for full redundancy on your side .. [16:06] <toddf> they do permit bgp6 over a tunnel for a fee, if I read their website properly [16:06] <brycec> That sounds like fun :) And I'm still meaning to move my IPv6 tunnels to ARP. However lately, HE's reliability has been > ARP :( [16:07] <brycec> toddf: Actually I can request a BGP tunnel for free [16:08] <brycec> But first I'd need an ASN... [16:08] <toddf> details [16:08] <brycec> And only 7 POPs support it [16:09] <brycec> (As in: not my closest POP) [16:44] *** KDE_Perry has quit IRC (Ping timeout: 246 seconds) [16:45] *** KDE_Perry has joined #arpnetworks [16:46] <m0unds> brycec: http://chris.vanvoro.us/2013/12/26/fun-with-ipv6/ [16:47] <brycec> Thanks m0unds [16:47] <m0unds> sure, it's not the best, but it's better than most of what i'd read, haha [16:47] <staticsafe> that your site? [16:49] <m0unds> yep, terribleness that it is [16:49] <m0unds> octopress + nginx [16:49] <m0unds> code repo on bitbucket [16:50] <staticsafe> i was too lazy to octopress so i just went back to wordpress [16:50] <m0unds> i started using nitrous.io as a quick IDE for posting [16:51] <m0unds> i'm hosting a friend's wordpress site - it's the only reason i still have mysql and php running on my vps [16:59] *** laotzi has quit IRC (Remote host closed the connection) [17:09] <brycec> Man I have no idea why I thought this would be more difficult... Using my ARP VPS as a v6 tunnel endpoint accomplished! (Still need to setup routing and firewalling, but that's all) [17:09] <brycec> thanks for the kick in the butt m0unds [17:09] <m0unds> you betcha! [17:09] <m0unds> i sat on mine for 6 mos before i did it [17:10] <brycec> I'm over 1yr now [17:10] <brycec> i think [17:10] <m0unds> then got bored at work and went 'meh' and just did it [17:10] <mercutio> apparently cogent -> verizon is even more broken than normal [17:10] <brycec> tl;dr just need matching gif/v4tunnel/etc sections on both ends, that's it [17:10] <m0unds> the srx part was what i hung up on initially though because i was on junos 11.4, which doesn't support ipv6 in flow mode [17:10] <mercutio> cogent issues have been going on for something like two years now? [17:11] <m0unds> so when i updated to the final build for my srx (discontinued model) it fixed it [17:11] <brycec> cool, congrats [17:11] <brycec> (my oldest invoice seems to be Nov 2012. Over a year now, woo) [17:11] <mercutio> brycec: yeh it pretty simple to tunnel ipv6 [17:11] <mercutio> you may have to mss clamp if you're forwarding traffic though [17:12] <brycec> Hardest part now is deciding on address allocations [17:12] <mercutio> fac3 ? [17:12] <brycec> I'll keep that in mind, thanks mercutio [17:12] <brycec> lol [17:12] <mercutio> i dunno :) [17:12] <mercutio> err face would work too [17:12] <brycec> face:b00c is pretty well-known ;) [17:12] <mercutio> 1337 ? [17:12] <mercutio> yeh i know [17:13] <brycec> Yeah there are a bunch of "clever" ones out there. I'm far more practical.... But I can't just start at "1" [17:13] <mercutio> you only have 16 bits to play with [17:13] <brycec> (0 is already in use) [17:13] <m0unds> hahaha [17:13] <m0unds> i started at 2 [17:13] <brycec> 16 bits? [17:13] *** laotzi has joined #arpnetworks [17:13] <mercutio> 48 to 64 [17:13] <m0unds> my clients at home are 4, iirc [17:13] <brycec> Oh sure, duh [17:15] <mercutio> bcec ? [17:15] <mercutio> removing r and y from your nick, that don't map to hex :) [17:15] <brycec> ha [17:16] <brycec> Probably gonna start at f00a [17:16] <mercutio> it does kind of sound like "be sick" though [17:16] <mercutio> or f00f like the pentium bug? [17:16] <m0unds> like "sick" as in, WAY SICK DOODZ [17:16] <brycec> So help whichever net ends up on f00f ;) [17:17] * brycec spirals into the IPv6 "OMG SO MANY ADDRESSES" oblivion [17:17] <mercutio> 1c12 [17:17] <mercutio> ( i see one too) [17:17] <brycec> actually I should just migrate my current HE prefixes [17:18] <mercutio> bryce: you only have 16 bits, it's not that many [17:18] <mercutio> you need the /64 for autoconfig [17:19] <brycec> 16 bits is till pretty big [17:19] <brycec> (And I know I can't really sub-divide the /64) [17:19] <mercutio> it never felt very big on pc's :) [17:19] <mercutio> dammn those 64k memory limits [17:20] <mercutio> it was a real pita [17:20] <mercutio> but yeah it's a lot better than like 1 or 8 or such [17:21] *** hazardous has joined #arpnetworks [17:28] *** laotzi has quit IRC (Quit: SIGQUIT) [17:28] <up_the_irons> for those running their own ntp server [17:28] <up_the_irons> "1. If you run ntpd, upgrading to the latest version, which removes the "monlist" command that is used for these attacks; alternately, disabling the monitoring function by adding "disable [17:28] <up_the_irons> +monitor" to your /etc/ntp.conf file." [17:28] <up_the_irons> we're getting LOTS of notices for NTP-based UDP amplification attacks [17:29] <brycec> up_the_irons: Any way to forward those notices to the responsible party? [17:29] <brycec> *parties [17:29] <up_the_irons> brycec: i am in the process of doing so, yes [17:29] <up_the_irons> a very big time suck [17:29] <up_the_irons> 39 notices based on IP. gotta lookup the IP, get email of customer, then foward. [17:30] <up_the_irons> *forward [17:30] *** dne has quit IRC (Ping timeout: 264 seconds) [17:30] *** Spitfire has quit IRC (Ping timeout: 264 seconds) [17:30] <brycec> Bummer [17:31] *** Yamazaki-kun has quit IRC (Ping timeout: 245 seconds) [17:32] <up_the_irons> maybe i could write some filter.. [17:32] <brycec> (Oh good, I was already secure) [17:33] <up_the_irons> procmail or something [17:35] *** Spitfire has joined #arpnetworks [17:37] *** dne has joined #arpnetworks [17:39] *** Yamazaki-kun has joined #arpnetworks [17:41] <up_the_irons> actually, would anyone *else* like to write something? I'll pay (obviously). Basic flow would be: 1) I get an abuse complaint, 2) i forward to some special address, 3) something / script on that address looks up IP with regex, 4) IP returns an email address (with our REST API), 5) forward that email [17:41] <up_the_irons> or, pointers to how this would be done would help [17:41] <up_the_irons> i can try to code something up [17:45] *** laotzi has joined #arpnetworks [17:46] *** Yamazaki-kun has quit IRC (Ping timeout: 245 seconds) [17:46] <brycec> Seems straight-forward enough [17:48] <mercutio> up_the_irons: it very well could happen for dns too [17:48] <up_the_irons> mercutio: dns? [17:48] <mercutio> so maybe having an easy way to email ip's would be good [17:48] <mercutio> up_the_irons: the any thing, and open recursive are being hit on authorative and recursive a lot recently too [17:48] <up_the_irons> mercutio: if you mean the amplification attacks, yes, very much so [17:49] <mercutio> up_the_irons: what about having a sepcial domain you email with users ip@blah.arpnetworks.com [17:49] <mercutio> or such [17:50] <mercutio> and then it emails the right person, and a sepcial mailbox to keep note [17:50] <mercutio> which would just mean cutting and pasting the ip [17:50] <mercutio> which isn't automated, but is simpler to test, .. [17:50] <up_the_irons> mercutio: ah true [17:50] <up_the_irons> mercutio: i like it [17:51] <up_the_irons> I LIKE IT [17:51] <mercutio> can you map from ip to user with a mysql query or such? [17:52] <up_the_irons> more like a bit of ruby [17:52] <brycec> 4) IP returns an email address (with our REST API)" [17:52] <brycec> (obviously not a public REST API ;) ) [17:52] <up_the_irons> obviously :) [17:53] <mercutio> i wonder if for things like recursive dns there should be tests every now and then [17:54] *** Yamazaki-kun has joined #arpnetworks [17:54] <mercutio> but with a little script magic if such a system was setup it'd be easy to email effected users [17:54] <mercutio> err affected? [17:54] <brycec> ^ [17:55] *** mhoran has joined #arpnetworks [17:55] *** ChanServ sets mode: +o mhoran [18:01] <mercutio> hmm as an addition could have some extra things to bounce to which would send automated message content that say how to fix open dns etc [18:01] <mercutio> or maybe just keep a list of the various things, and people can parse themselves. [18:02] <mercutio> for ntp i'm in favour of openntpd which doesn't listen by default [18:03] <up_the_irons> affected [18:21] <jpalmer> are there any web based test tools for the NTP or DNS amplification attacks yet? [18:22] <jpalmer> (I don't know how to exploit it offhand, but would like to verify my DNS and NTP servers are ok. [18:23] <mercutio> host -t any <your domain name> [18:24] <mercutio> the any thing is complicated [18:24] <mercutio> basically more providers need to do bcp38 to improve the situation [18:24] <mercutio> as the predominant issue is that it's valid to do an any request for a domain name. [18:25] <jpalmer> yeah, that returns several records for all of my domains. [18:27] <mercutio> see that's normal [18:27] <mercutio> now the problem is someone can spoof an address [18:28] <mercutio> so that your response goes to another address [18:28] <mercutio> esp if one has lots of entries [18:28] <mercutio> like say host -t any microsoft.com has quite a bit of data [18:30] <mercutio> it's only like 4x amplification normally with that htough [18:30] <mercutio> but still if it's 10 megabit of requests that makes 40 megabit of response [18:30] <jpalmer> yep yep [18:30] <mercutio> arp defaults to 5 megabit rate limit for udp, so you'll only be able to return 5 megabit [18:30] <mercutio> but that could impact other services.. [18:31] <mercutio> generally speaking most people seem to be ignoring the amplification attack and suggesting that it's the people sending spoofed requests that are the problem [18:31] <jpalmer> heh. it's not the misconfigured SMTP servers, it's the spammers! [18:32] <mercutio> well udp will limit what response size normally [18:32] <mercutio> and tcp won't work [18:32] <mercutio> bcp38 means people can't spoof addresses as easily so it cna't work as easily [18:33] <mercutio> from memory comcast is the biggest provider with no protection [18:33] <jpalmer> I'll have to read up on bcp38. not familiar with it. [18:33] <mercutio> https://www.nanog.org/sites/default/files/mon_general_weber_defeat_23.pdf [18:33] <mercutio> it only really matters for providers [18:34] <mercutio> basically it means that you can't send packets with my source ip address [18:34] <mercutio> which arp do btw [18:35] <mercutio> but basically if the any requests aren't terribly long it's probably mostly ok [18:45] <mnathani> Are there network anomalies at present? I am getting about 2% packetloss from Toronto [18:46] <mercutio> via ntt? [18:48] <mercutio> ntt -> verizon still seems lossy [18:49] <mercutio> acf had a smokeping [18:50] <mercutio> uhh [18:50] <mercutio> acf's smokeping was really good in the middle of the night [18:50] <mercutio> and his comcast gets better earlier [18:51] <mercutio> acf: did you check out your smpkeping? [18:57] <mnathani> via nlayer / mzima [18:57] <mercutio> oh prob diff issue then [18:57] <mercutio> 2% isn't so bad [18:57] <mnathani> thats the forward path [18:58] <mercutio> unless that's averaged over time [18:58] <mercutio> acf's issue was forward path from arp [18:58] <mercutio> err and not just arp [18:58] <mercutio> going via ntt in san jose was also broken [18:58] <mnathani> reverse path is: trit > he [18:58] <mercutio> i'd suspect that trit->he path [18:58] <mercutio> nlayer do heaps of icmp deprioritisation too [18:59] <mercutio> i'd do iperf in udp mode at low bandwidth [18:59] <mercutio> to check which direction [18:59] <mercutio> not that you can necessarily change anything [18:59] <mnathani> hence is the nature of the Internet / Inter webs [18:59] <mercutio> yeh [18:59] <mercutio> i feel better having more idea of where things are going wrong even if i can't change them :) [18:59] <mnathani> It surprising it actually works at all [19:00] <mercutio> try 20% packet loss [19:00] <mercutio> that is hell to use [19:00] <mercutio> one time i was playing dota and there was a ddos attack and had 50% packet loss [19:00] <mercutio> and the game was going terribly [19:00] <mercutio> so i cehcked with mtr etc [19:00] <mercutio> and then i thought it was doing well considering there was ilke 50% packet loss [19:01] <mercutio> with ssh if there's a bit of packet loss often typing another key can help things along [19:01] <mercutio> ilek if something's not appearing you can press backspace or something [19:02] <mercutio> but if it's completely broken often it's better not to touch anything at all [19:02] <mercutio> and have the connection time out / disconnect [19:03] *** DaCa has quit IRC (Ping timeout: 252 seconds) [19:15] <mnathani> I always use tmux anyway so the session stays alive [19:16] <mercutio> ahh yip [19:19] <mercutio> oh was that you that posted to outages@ acf? :) [19:21] <mnathani> who/what is acf? [19:22] <mercutio> the guy who brought up comcast/verizon packet loss before [19:22] <mercutio> or did i get it wrong? [19:22] <mercutio> http://kremvax.acfsys.net/smokeping.cgi?target=Remote.verizon-snloca [19:22] <mercutio> he linked that [19:23] <mercutio> well someone posted to outages@ [19:23] <mercutio> who uses arp [19:23] <mercutio> https://puck.nether.net/pipermail/outages/2014-February/006596.html [19:29] <mnathani> oh ok.. [19:30] <jpalmer> I keep trying tmux, then switching back to screen. hehe [19:31] <mnathani> I like its default config, works out of the box [19:31] <mnathani> screen, I keep needing to paste in configs before I can use it [19:32] <mnathani> also the splitting of windows / panes is nice [19:33] <jpalmer> I keep getting fumbled up by the default keybindings in tmux, so used to screen's [19:34] <jpalmer> I need a basic "idiots guide to tmux" and just start using it with irssi. when I get more comfortable with it, then install it on all my machines with puppet. [19:35] <mnathani> http://www.amazon.com/tmux-Productive-Development-Brian-Hogan/dp/1934356964/ref=sr_1_1?ie=UTF8&qid=1392089683&sr=8-1&keywords=tmux [19:35] <BryceBot> Amazon: "tmux: Productive Mouse-Free Development" [19:36] <mnathani> keybinding should be pretty basic to reconfigure [19:36] <mnathani> although you might not want to for sake of running screen within a tmux session [19:43] <mercutio> there are books on tmux? [19:43] <mercutio> wow [19:44] <m0unds> open resolver project is handy for identifying open resolvers on a network [19:44] <m0unds> http://openresolverproject.org [19:45] <m0unds> re: the ntpd thing, the default config for freebsd was changed when that vuln was identified, and freebsd10 ships with the modified config by default [19:47] <mercutio> http://www.itnews.com.au/News/372033,worlds-largest-ddos-strikes-us-europe.aspx [19:47] <mercutio> there was a ddos today [19:47] <mercutio> apparentyl 400 gigabit [19:47] <mercutio> oh and it was using ntp [19:48] <mercutio> did up_the_irons reports all come today? [19:49] <m0unds> whoa [19:49] <m0unds> jeez, that's enormous [19:49] <mercutio> so that could haev effected canda traffic [19:49] <mercutio> canada [19:50] <m0unds> cool, equiv of openresolverproject for ntpd [19:52] <m0unds> also, cloudflare's not on aws, but whatever, hahaha [19:52] <mercutio> i dunno [19:52] <mercutio> cloudflare is terrible [19:52] <mercutio> they may haev some stuff on aws [19:53] <m0unds> maybe staging or something, but they pride themselves on owning their hardware [19:53] <mercutio> i haven't found anything about this ddos on nanog yet [19:53] <mercutio> i was avoiding reading nanog to not get swamped :) [19:53] <m0unds> didn't see anything in nanog digests today [19:53] <mercutio> i don't know if 400 gigabit is actually the biggest ddos too [19:54] <mercutio> i been reading this carrier comparison [20:11] <mercutio> for some reason i can't find any other articles or mentioning of ddos [20:15] <up_the_irons> mercutio: all today, yeah [20:15] <up_the_irons> in fact, i dunno why i didn't look before, but like 30 minutes ago i noticed all our egress links are at like 300 Mbps! [20:15] <mercutio> ouch [20:16] <up_the_irons> lots of VPS' participating in the attacks (i'm sure innocent victims) [20:16] <up_the_irons> so i'm going to be blocking all NTP inbound [20:16] <mercutio> hmm [20:16] <up_the_irons> on all hosts [20:16] <mercutio> probably prudent [20:17] <up_the_irons> as a stop gap until people start fixing their setup [20:17] <mercutio> there's some debate whether it's a good idea to block all ntp [20:17] <mercutio> as some ntp like to use the same source/dest port [20:17] <mercutio> but yeah as stop-gap it makes a hell of a lot of sense [20:17] <up_the_irons> there's no debate in my mind when my network is hitting some target with > 1 Gbps of UDP [20:17] <mercutio> heh [20:18] <mercutio> well the debate was whether it shoudl be rate limited [20:18] <mercutio> or blocked ocmpletely [20:18] <mercutio> i reckon blocked completely [20:18] <mercutio> i'm kind of against rate limiting [20:18] <up_the_irons> rate limiting won't do shit [20:18] <up_the_irons> i mean, it will, but if 99% of the incoming is illegit traffic [20:18] <up_the_irons> your rate limit will effectively block all legit traffic too [20:19] <up_the_irons> so wtf [20:19] <up_the_irons> rather [20:19] <up_the_irons> it won't matter [20:19] <mercutio> hmm [20:19] <mercutio> won't people be hitting that 5 megabit udp rate limit anyway? [20:19] <mercutio> i shouldn't distract you [20:19] <up_the_irons> that's only in one direction [20:19] <up_the_irons> the wrong direction ;) [20:20] <up_the_irons> and yes, i'll take questions later :) [20:23] <m0unds> oof [20:24] <m0unds> for freebsd guests: http://www.freebsd.org/security/advisories/FreeBSD-SA-14:02.ntpd.asc [20:32] <mercutio> i still can't see anything on nanog [20:32] <mercutio> i wonder if tehre's another mailing list i should follow too [20:33] <mercutio> someone posted about it on nznog [20:59] *** DaCa has joined #arpnetworks [21:05] *** pcn has joined #arpnetworks [21:07] <pcn> So I saw on some intertwitters about ntp blockage? [21:07] <pcn> I don't have ntp running, but now that you mention it, is there an internal ntp server that can be peered with at the moment? [21:09] <mnathani> pcn: its inbound ntp requests, outbound as to get time from say pool.ntp.org should work just fine [21:24] <pcn> OK [21:26] <mercutio> it's still valid question [21:26] <mercutio> i don't know of any [21:26] <mercutio> i think i just use pool.ntp.org [21:35] <mnathani> whats the nmap check or ntp check to ensure a host isnt configured incorrectly so as to be used in a UDP / ntp based DDOS attack? [21:35] <mercutio> uhh is saw something somewhere [21:36] <mercutio> <http://nmap.org/nsedoc/scripts/ntp-monlist.html> [21:46] *** BryceBot has quit IRC (Excess Flood) [21:46] *** BryceBot has joined #arpnetworks [22:19] <up_the_irons> mnathani: either upgrade ntp or just disable monlist command [22:20] <mercutio> up_the_irons: that nmap thing checks for monlist [22:21] <mercutio> so you could port scan your ranges if you wanted to find out who is vulnerable to it [22:21] <up_the_irons> mercutio: oh sweet [22:21] <mercutio> which frmo your own ip could prob bypass any blocks [22:44] <up_the_irons> this is old, but still looks like it'd work: [22:44] <up_the_irons> http://railspikes.com/2007/6/1/rails-email-processing [22:45] <up_the_irons> no need to set up procmail or Postfix filter to fork into ruby process. just have a daemon check a special email box! [22:49] <mercutio> cool. [22:49] <mercutio> not that it really matters which way it is done [22:56] <mnathani> up_the_irons: roger [23:05] <acf_> mercutio: yeah, I see that in the smokeping [23:06] <acf_> after prodding NTT a bit more [23:06] <acf_> http://paste.unixcube.org/k/246aaa [23:07] <mercutio> acf_: was it you that posted to outages@? [23:07] <acf_> hmm? I emailed noc@us.ntt.net again [23:07] <mercutio> not technical in nature [23:07] <mercutio> oh i just saw that same address as you were saying on outages mailing list [23:08] <mercutio> how do i search scrollback? :) [23:08] <acf_> peering disagreement or something likely [23:08] <mercutio> so it wasn't you that posted to oustages mailing list? [23:08] <acf_> nope [23:08] <mercutio> https://puck.nether.net/pipermail/outages/2014-February/006596.html [23:09] <mercutio> maybe it not someone in irc even [23:09] <acf_> any connection to the recent ddos news things you think? [23:09] <mercutio> i was wondering that [23:09] <mercutio> but i don't think it is [23:09] <mercutio> esp with your email response [23:10] <mercutio> it's ntt getting into messy situation like cogent [23:10] <mercutio> with not wanting to pay to send data [23:10] <mercutio> i imagine [23:10] <acf_> wow. that guy sounds exactly like me [23:10] <mercutio> see how i wondered? [23:10] <mercutio> he even on arp :) [23:11] <acf_> idk if "not technical in nature" means ntt/verizon is purpousely degrading connectivity [23:11] <mercutio> could be [23:11] <acf_> or just that verizon/ntt have to negotiate bigger pipes to take the data [23:11] <mercutio> did you see the uhh [23:11] <mercutio> god damnit [23:11] <mercutio> i weant to find a way to find urls i pasted to irc :) [23:12] <mercutio> http://arstechnica.com/information-technology/2014/02/netflix-performance-on-verizon-and-comcast-has-been-dropping-for-months/ [23:12] <BryceBot> Ars Technica: "Netflix performance on Verizon and Comcast has been dropping for months" [23:12] <mercutio> is that something bryce can do? [23:12] <mercutio> it's the same two providers [23:12] <mercutio> even if diff origin [23:13] <mercutio> i think ntt is generally considered tier 1 and cogent not though? [23:13] <mercutio> but they're both huge [23:13] <acf_> yeah [23:13] <acf_> cogent is usually considered tier crap afaik [23:13] <mercutio> heh i was reading on nanog about cogent [23:13] <mercutio> again :/ [23:14] <mercutio> but yeah i not a fan [23:14] <mnathani> godaddy is crap tier too [23:14] <mercutio> i still reckon up_the_irons should just route verizon/comcast a different way [23:14] <mercutio> maybe with max prefix limit [23:14] <acf_> a bit of testing seems to reveal that rerouting through nlayer would still go through ntt [23:14] <mercutio> these things never seem to get fixed very quickly [23:14] <mercutio> and it usually gets worse before it gets better [23:15] <acf_> we'll probably have to wait for level3 if up_the_irons wants to reroute [23:15] <mercutio> oh [23:15] <acf_> idk much though [23:15] <mercutio> he has tata too [23:15] <mercutio> but i dunno [23:15] <mercutio> level3 is sure to fix it [23:15] <mercutio> how did you test via nlayer? [23:15] <acf_> yeah, it doesn't look like we're near the end of this [23:15] <mercutio> oh from a lg? [23:16] <mercutio> i exepct level3 shouldn't take long to get connected up [23:16] <mercutio> i imagine it's just however long it takes to get a cross connect [23:16] <mercutio> which should be quicker for a big data centre you'd think [23:16] <mercutio> i mean they could probably turn it up tommorow if they felt like it [23:17] <mercutio> but if you tried to ask for tommorow they'd probably want to charge heaps for urgency [23:17] <acf_> yeah. I certainly hope it's soon [23:17] <acf_> well, nlayer-> verizon is direct [23:17] <mercutio> but he'd probably have to go there to plug in cross connect [23:17] <acf_> but verizon-> nlayer is via ntt [23:17] <mercutio> oh [23:17] <mercutio> but it's -> verizon that is bad [23:18] <mercutio> judging by my routing via verizon being fine [23:18] <acf_> so maybe that would fix it [23:18] <mercutio> i doubt reverse path is via verizon [23:18] <mercutio> but i dunno trace me ? [23:18] <mercutio> 202.49.67.22 [23:19] <acf_> (from verizon) verizon->ntt->new zealand stuff [23:19] <mercutio> i actually think the routing side of things is somewhere the internet could really improve [23:19] <mercutio> so it's ntt return with no packet loss [23:19] <mercutio> which city is the verizon-> ntt in ? [23:19] <mercutio> oh of course california [23:19] <mercutio> is it san jose or los angeles? [23:19] <mercutio> or something else? [23:19] <acf_> lax for me [23:20] <mercutio> and yeh we tested sending via ntt in la and sj [23:20] <mercutio> and both were bad [23:20] <mercutio> and sending via verizon in la was fine [23:20] <acf_> yeah, no packet loss to you [23:20] <acf_> you have nlayer path to verizon? [23:20] <mercutio> hmm [23:20] <mercutio> i misplaced your ip [23:20] <mercutio> i dunno what route it taking atm [23:20] <acf_> 108.40.173.223 [23:21] <mercutio> yeh that forward path via verizon [23:21] <mercutio> uhh it's showing packet loss now [23:21] <mercutio> i wonder if it doesn't like two traceroutes at once [23:21] <acf_> probably verizon just really sucks [23:21] <mercutio> i'll pingplotter from the other ip [23:22] <mercutio> yeah it looks fine [23:22] <mercutio> but i think it is rate limiting icmp too [23:22] <mercutio> oh no now some loss [23:23] <mercutio> this was all fine last night! [23:23] <acf_> the internet is breaking down! [23:23] <mercutio> it does that [23:23] <mercutio> it's not as bad as before [23:23] <mercutio> it's 1.14% [23:23] <mercutio> and it was easily 7% usually going via ntt [23:24] <acf_> http://kremvax.acfsys.net/smokeping.cgi?target=Remote.verizon-lsanca [23:24] <mercutio> yeh i saw that earlier today [23:24] <mercutio> the overnight thing is like wow [23:24] <mercutio> i can smokeping you from nz maybe? [23:24] <acf_> that overnight thing was awesome [23:25] <mercutio> if it's clean it suggests that it's single direction [23:25] <acf_> the latency dropped from 30-40ms. I wonder if the ARP route changed [23:25] <mercutio> i doubit it [23:25] <mercutio> it lpooks like congestion [23:25] <mercutio> but it's hard to know [23:25] <acf_> it dropped off a bit sharply. does it do that? [23:25] <mercutio> you mind if i add you to my smokeping? [23:26] <acf_> go for it [23:26] <mercutio> sometimes [23:26] <mercutio> i have curl testing too [23:26] <mercutio> but i assume you're not hosting any files on your dsl :) [23:26] <acf_> nope [23:26] <mercutio> what was that comcast ip? [23:26] <mercutio> oh it was comcast.net [23:27] <acf_> also 72.55.8.69 [23:27] <mercutio> btw my friend in sj on comcast cable wasn't packet loss [23:27] <mercutio> what's 72.55.8.69? [23:27] <mercutio> that's via level 3 forwaered route [23:27] <acf_> the non-rate-limiting router in front of work's internet [23:27] <acf_> work blocks icmp :0 [23:27] <mercutio> ahh [23:28] <mercutio> is it on level3? [23:28] <acf_> that's comcast [23:28] <mercutio> normal comcast is via ntt [23:28] <mercutio> whereas this is via level3 [23:28] <mercutio> no ip's even say comcast.net [23:28] <acf_> strange. I think they have some legacy IP address block, but I didn't think it would affect routing [23:29] <mercutio> it says comcast business [23:29] <acf_> yeah, that's it [23:29] <mercutio> as13385 [23:29] <mercutio> wheras comcast.net is AS7922 [23:30] <mercutio> ok first syas comcast telecommunications, second says comcast cable [23:30] <acf_> both go ntt-> comcast via tata over arp [23:30] <mercutio> not from here though [23:30] <mercutio> i'll do b oth [23:31] <mercutio> arp doesn't have level3 yet [23:31] <acf_> it will be interesting to see the difference [23:31] <mercutio> maybe i shoudl do a subgroup [23:32] <mercutio> nah screw it [23:32] <mercutio> i wnat to be able to subgroup and not subgroup at the same time [23:32] <mercutio> it's handy scrolling through liwst