[00:19] *** dj_goku has joined #arpnetworks [01:00] liking the BIRD filter / function language, even though it seems a bit weird at first [01:00] import filter { [01:00] bgp_community.add((our_asn,20000)); [01:00] accept; [01:00] }; [01:00] easy enough [01:01] a little bit weird is, show all HE routes: [01:01] sh ro filter { if 6939 = bgp_path.first then accept; } [01:01] seem verbose, but meh, it works [01:09] *** tehfink has joined #arpnetworks [02:14] *** tehfink has quit IRC (Quit: tehfink) [02:40] i like openbgpd syntax etc [02:40] but bird is faster at converging from what i understand [02:40] and is supported on linux [02:41] i'd like to see openbgp for linux though [02:42] *** tehfink has joined #arpnetworks [03:04] i left that comcast trace going, and there's still no packet loss to comcast [03:05] i think i need to do it during earlier hours [03:05] *** tehfink has quit IRC (Quit: tehfink) [03:49] mercutio: bird seems to be insanely fast; peers go into "Established" state almost instantly after I reload config with a new peer [04:14] up_the_irons: pulp = repo management, candlepin = subscription management (system A has repos A,B, and F, but not D or E) etc [04:14] jpalmer: oh cool [04:15] foreman is a frontend dashboard, and ENC to puppet. [04:17] cool [04:42] *** tehfink has joined #arpnetworks [04:44] *** kevr has quit IRC (Read error: Operation timed out) [04:50] *** kevr has joined #arpnetworks [05:00] *** tehfink has quit IRC (Quit: tehfink) [05:03] *** tehfink has joined #arpnetworks [05:11] *** kevr has quit IRC (Ping timeout: 252 seconds) [05:13] *** tehfink has quit IRC (Quit: tehfink) [05:17] *** kevr has joined #arpnetworks [05:55] *** tehfink has joined #arpnetworks [06:44] *** toddf has quit IRC (Quit: leaving) [06:47] *** toddf has joined #arpnetworks [06:47] *** ChanServ sets mode: +o toddf [09:09] *** tehfink has quit IRC (Quit: tehfink) [09:41] *** NiTeMaRe has quit IRC (Ping timeout: 265 seconds) [09:43] *** NiTeMaRe has joined #arpnetworks [10:14] *** SpeedBus has quit IRC (Ping timeout: 245 seconds) [10:35] *** pjs_ has quit IRC (Quit: EPIC5-1.1.2[1638] - amnesiac : Help! The paranoids are out to get me!) [10:35] *** pjs has joined #arpnetworks [10:45] *** SpeedBus has joined #arpnetworks [12:02] *** hazardous has quit IRC (*.net *.split) [12:02] *** gizmoguy has quit IRC (*.net *.split) [12:02] *** plett has quit IRC (*.net *.split) [12:03] *** plett has joined #arpnetworks [12:03] *** gizmoguy has joined #arpnetworks [12:04] *** laotzi has joined #arpnetworks [12:05] *** jpalmer has quit IRC (Excess Flood) [12:05] *** jpalmer has joined #arpnetworks [13:11] up_the_irons: does that mean any2xi is up now? [13:33] *** hp_ has joined #arpnetworks [13:34] *** hp_ is now known as Guest58998 [14:01] http://arstechnica.com/information-technology/2014/02/netflix-performance-on-verizon-and-comcast-has-been-dropping-for-months/ [14:01] Ars Technica: "Netflix performance on Verizon and Comcast has been dropping for months" [14:01] it's interesting that verizon and comcast were the two destinations ntt were haveing issues too [14:01] s/too/to/ [14:01] it's interesting that verizon and comcast were the two destinations ntt were haveing issues to [14:06] Hey. I could use some help on an ipv6 /48 ubuntu configuration. No matter the search query in google I can't seem to find anyone that describes it the way arp networks does. Someone know how to set up the /48 on a single Ubuntu VPS? [14:09] at one point it was routed to your vps. at one point the lowest /64 was an ethernet segment and the rest was avilable on a support ticket request basis for routing. I'm not sure what the defaults are at this point. if you're a recent customer, just try setting the lowest /64 subnet on your ethernet segment and see how that goes .. try fe80::1 and ::1 for a default router, one of those should work. perhaps there's a ... [14:09] ... wiki page I'm unaware of. hope that helps. [14:10] I'm told it has been routed to link-local and that I should set my side to fe80::2/64 [14:10] I'm not exactly sure what they mean by "my side".. default gateway, local address or? [14:11] http://wiki.arpnetworks.com/wiki/48%20IPv6%20on%20OpenBSD is good reference [14:11] So "set your side to..." means set the IP on the interface to fe80::2 [14:11] the default gateway is fe80::1 [14:11] (because ARP's side is fe80::1 and routing the /48 to fe80::2) [14:12] (Also: Requisite "if you don't know how to do this stuff, then you probably shouldn't be messing with it.") [14:13] you shouldn't need a /48 [14:13] Oh I know it's expert only ;) But I have other servers where /48 is routed differently (I think) [14:13] but I do [14:13] basically [14:14] but yeah I know that I shouldn't [14:16] ARP's method of routing is actually pretty common too, fwiw. [14:17] Though the majority of tutorials and howtos are written for people with HE tunnels and the like, so I can see how that drowns out the useful information. [14:17] I dont doubt it. It's just really hard to find it described that way anywhere else [14:18] yep, it's mostly two lines about native ipv6 and then 4 pages about tunnels [14:18] native ipv6 is easy though [14:19] Yep. [14:19] And the /48 too once you realize it's two lines or so [14:20] I got some ipv6 connectivity now. Thanks a lot for your help [14:23] *** Guest58998 has quit IRC (Quit: Leaving) [14:26] arp's default config is great for a single vps. if you have multiple, you have to route v6 to the others from your first vps, or ask arpnetworks for changes. I opted for plan b *grin*, one /64 on the ethernet segment. [14:45] yea, i wrote a post about configuring SRX devices with a roll-your-own ipv6 tunnel in flow mode because so many of the HE tunnel broker tutorials are silly and tell you to switch off flow mode on your appliance and stuff [14:45] hopefully it'll help someone sooner or later - same with working srcnat for xbox live, since it seems people way overthink that stuff [15:15] heh [15:15] m0unds: link? [15:45] mercutio: not yet [15:47] toddf: for the record, our default is no routing at all, just /64 on your VLAN, so no single vps is a point of failure [15:47] *** up_the_irons has quit IRC (Read error: Operation timed out) [15:47] up_the_irons: ah. I've been around too long to know what the actual current default is, hope I made that clear above ;-) [15:48] *** up_the_irons has joined #arpnetworks [15:48] *** ChanServ sets mode: +o up_the_irons [15:49] *** mhoran has quit IRC (Ping timeout: 246 seconds) [15:49] lol [15:49] sine up_the_irons missed it: 15:46:39 <@toddf> up_the_irons: ah. I've been around too long to know what the actual current default is, hope I made that clear above ;-) [15:50] *since [15:50] brycec: tnx! [15:51] If you had multiple VPS'en and a /48, I suppose you could always CARP them all, but routing would be annoying/tricky. [15:51] yeah [15:52] I still haven't worked out a good way to give my CARP backup IPv6 access to an HE tunnel :/ Not without watching for the state change and scripting route changes anyways. [15:52] (It's also not high on the priority list) [16:03] brycec: convince he.net it needs to do ospf6 with you and have two tunnels one to each router? [16:04] not always doable because some people only have a single ip, carp can be done in this case, but v4 connectivity is always fun in the backup router instance [16:04] Two tunnels but same subnet? [16:04] Not to worry, both routers have public v4 IPs [16:04] plus one shared [16:04] you'd need two tunnels and ospf6 should handle routing of the same subnet yes [16:05] Well all that's left is to convince HE of anything, lol [16:05] (note I've never heard of anyone doing it, but if you want to avoid scripting and wish to do it up proper...) [16:06] Yeah that would be proper. But given how much I'm paying them... I don't expect them to do anything "for me" [16:06] you could of course get two vps'en from arpnetworks and do ospf6 across two gif tunnels to your home for full redundancy on your side .. [16:06] they do permit bgp6 over a tunnel for a fee, if I read their website properly [16:06] That sounds like fun :) And I'm still meaning to move my IPv6 tunnels to ARP. However lately, HE's reliability has been > ARP :( [16:07] toddf: Actually I can request a BGP tunnel for free [16:08] But first I'd need an ASN... [16:08] details [16:08] And only 7 POPs support it [16:09] (As in: not my closest POP) [16:44] *** KDE_Perry has quit IRC (Ping timeout: 246 seconds) [16:45] *** KDE_Perry has joined #arpnetworks [16:46] brycec: http://chris.vanvoro.us/2013/12/26/fun-with-ipv6/ [16:47] Thanks m0unds [16:47] sure, it's not the best, but it's better than most of what i'd read, haha [16:47] that your site? [16:49] yep, terribleness that it is [16:49] octopress + nginx [16:49] code repo on bitbucket [16:50] i was too lazy to octopress so i just went back to wordpress [16:50] i started using nitrous.io as a quick IDE for posting [16:51] i'm hosting a friend's wordpress site - it's the only reason i still have mysql and php running on my vps [16:59] *** laotzi has quit IRC (Remote host closed the connection) [17:09] Man I have no idea why I thought this would be more difficult... Using my ARP VPS as a v6 tunnel endpoint accomplished! (Still need to setup routing and firewalling, but that's all) [17:09] thanks for the kick in the butt m0unds [17:09] you betcha! [17:09] i sat on mine for 6 mos before i did it [17:10] I'm over 1yr now [17:10] i think [17:10] then got bored at work and went 'meh' and just did it [17:10] apparently cogent -> verizon is even more broken than normal [17:10] tl;dr just need matching gif/v4tunnel/etc sections on both ends, that's it [17:10] the srx part was what i hung up on initially though because i was on junos 11.4, which doesn't support ipv6 in flow mode [17:10] cogent issues have been going on for something like two years now? [17:11] so when i updated to the final build for my srx (discontinued model) it fixed it [17:11] cool, congrats [17:11] (my oldest invoice seems to be Nov 2012. Over a year now, woo) [17:11] brycec: yeh it pretty simple to tunnel ipv6 [17:11] you may have to mss clamp if you're forwarding traffic though [17:12] Hardest part now is deciding on address allocations [17:12] fac3 ? [17:12] I'll keep that in mind, thanks mercutio [17:12] lol [17:12] i dunno :) [17:12] err face would work too [17:12] face:b00c is pretty well-known ;) [17:12] 1337 ? [17:12] yeh i know [17:13] Yeah there are a bunch of "clever" ones out there. I'm far more practical.... But I can't just start at "1" [17:13] you only have 16 bits to play with [17:13] (0 is already in use) [17:13] hahaha [17:13] i started at 2 [17:13] 16 bits? [17:13] *** laotzi has joined #arpnetworks [17:13] 48 to 64 [17:13] my clients at home are 4, iirc [17:13] Oh sure, duh [17:15] bcec ? [17:15] removing r and y from your nick, that don't map to hex :) [17:15] ha [17:16] Probably gonna start at f00a [17:16] it does kind of sound like "be sick" though [17:16] or f00f like the pentium bug? [17:16] like "sick" as in, WAY SICK DOODZ [17:16] So help whichever net ends up on f00f ;) [17:17] * brycec spirals into the IPv6 "OMG SO MANY ADDRESSES" oblivion [17:17] 1c12 [17:17] ( i see one too) [17:17] actually I should just migrate my current HE prefixes [17:18] bryce: you only have 16 bits, it's not that many [17:18] you need the /64 for autoconfig [17:19] 16 bits is till pretty big [17:19] (And I know I can't really sub-divide the /64) [17:19] it never felt very big on pc's :) [17:19] dammn those 64k memory limits [17:20] it was a real pita [17:20] but yeah it's a lot better than like 1 or 8 or such [17:21] *** hazardous has joined #arpnetworks [17:28] *** laotzi has quit IRC (Quit: SIGQUIT) [17:28] for those running their own ntp server [17:28] "1. If you run ntpd, upgrading to the latest version, which removes the "monlist" command that is used for these attacks; alternately, disabling the monitoring function by adding "disable [17:28] +monitor" to your /etc/ntp.conf file." [17:28] we're getting LOTS of notices for NTP-based UDP amplification attacks [17:29] up_the_irons: Any way to forward those notices to the responsible party? [17:29] *parties [17:29] brycec: i am in the process of doing so, yes [17:29] a very big time suck [17:29] 39 notices based on IP. gotta lookup the IP, get email of customer, then foward. [17:30] *forward [17:30] *** dne has quit IRC (Ping timeout: 264 seconds) [17:30] *** Spitfire has quit IRC (Ping timeout: 264 seconds) [17:30] Bummer [17:31] *** Yamazaki-kun has quit IRC (Ping timeout: 245 seconds) [17:32] maybe i could write some filter.. [17:32] (Oh good, I was already secure) [17:33] procmail or something [17:35] *** Spitfire has joined #arpnetworks [17:37] *** dne has joined #arpnetworks [17:39] *** Yamazaki-kun has joined #arpnetworks [17:41] actually, would anyone *else* like to write something? I'll pay (obviously). Basic flow would be: 1) I get an abuse complaint, 2) i forward to some special address, 3) something / script on that address looks up IP with regex, 4) IP returns an email address (with our REST API), 5) forward that email [17:41] or, pointers to how this would be done would help [17:41] i can try to code something up [17:45] *** laotzi has joined #arpnetworks [17:46] *** Yamazaki-kun has quit IRC (Ping timeout: 245 seconds) [17:46] Seems straight-forward enough [17:48] up_the_irons: it very well could happen for dns too [17:48] mercutio: dns? [17:48] so maybe having an easy way to email ip's would be good [17:48] up_the_irons: the any thing, and open recursive are being hit on authorative and recursive a lot recently too [17:48] mercutio: if you mean the amplification attacks, yes, very much so [17:49] up_the_irons: what about having a sepcial domain you email with users ip@blah.arpnetworks.com [17:49] or such [17:50] and then it emails the right person, and a sepcial mailbox to keep note [17:50] which would just mean cutting and pasting the ip [17:50] which isn't automated, but is simpler to test, .. [17:50] mercutio: ah true [17:50] mercutio: i like it [17:51] I LIKE IT [17:51] can you map from ip to user with a mysql query or such? [17:52] more like a bit of ruby [17:52] 4) IP returns an email address (with our REST API)" [17:52] (obviously not a public REST API ;) ) [17:52] obviously :) [17:53] i wonder if for things like recursive dns there should be tests every now and then [17:54] *** Yamazaki-kun has joined #arpnetworks [17:54] but with a little script magic if such a system was setup it'd be easy to email effected users [17:54] err affected? [17:54] ^ [17:55] *** mhoran has joined #arpnetworks [17:55] *** ChanServ sets mode: +o mhoran [18:01] hmm as an addition could have some extra things to bounce to which would send automated message content that say how to fix open dns etc [18:01] or maybe just keep a list of the various things, and people can parse themselves. [18:02] for ntp i'm in favour of openntpd which doesn't listen by default [18:03] affected [18:21] are there any web based test tools for the NTP or DNS amplification attacks yet? [18:22] (I don't know how to exploit it offhand, but would like to verify my DNS and NTP servers are ok. [18:23] host -t any [18:24] the any thing is complicated [18:24] basically more providers need to do bcp38 to improve the situation [18:24] as the predominant issue is that it's valid to do an any request for a domain name. [18:25] yeah, that returns several records for all of my domains. [18:27] see that's normal [18:27] now the problem is someone can spoof an address [18:28] so that your response goes to another address [18:28] esp if one has lots of entries [18:28] like say host -t any microsoft.com has quite a bit of data [18:30] it's only like 4x amplification normally with that htough [18:30] but still if it's 10 megabit of requests that makes 40 megabit of response [18:30] yep yep [18:30] arp defaults to 5 megabit rate limit for udp, so you'll only be able to return 5 megabit [18:30] but that could impact other services.. [18:31] generally speaking most people seem to be ignoring the amplification attack and suggesting that it's the people sending spoofed requests that are the problem [18:31] heh. it's not the misconfigured SMTP servers, it's the spammers! [18:32] well udp will limit what response size normally [18:32] and tcp won't work [18:32] bcp38 means people can't spoof addresses as easily so it cna't work as easily [18:33] from memory comcast is the biggest provider with no protection [18:33] I'll have to read up on bcp38. not familiar with it. [18:33] https://www.nanog.org/sites/default/files/mon_general_weber_defeat_23.pdf [18:33] it only really matters for providers [18:34] basically it means that you can't send packets with my source ip address [18:34] which arp do btw [18:35] but basically if the any requests aren't terribly long it's probably mostly ok [18:45] Are there network anomalies at present? I am getting about 2% packetloss from Toronto [18:46] via ntt? [18:48] ntt -> verizon still seems lossy [18:49] acf had a smokeping [18:50] uhh [18:50] acf's smokeping was really good in the middle of the night [18:50] and his comcast gets better earlier [18:51] acf: did you check out your smpkeping? [18:57] via nlayer / mzima [18:57] oh prob diff issue then [18:57] 2% isn't so bad [18:57] thats the forward path [18:58] unless that's averaged over time [18:58] acf's issue was forward path from arp [18:58] err and not just arp [18:58] going via ntt in san jose was also broken [18:58] reverse path is: trit > he [18:58] i'd suspect that trit->he path [18:58] nlayer do heaps of icmp deprioritisation too [18:59] i'd do iperf in udp mode at low bandwidth [18:59] to check which direction [18:59] not that you can necessarily change anything [18:59] hence is the nature of the Internet / Inter webs [18:59] yeh [18:59] i feel better having more idea of where things are going wrong even if i can't change them :) [18:59] It surprising it actually works at all [19:00] try 20% packet loss [19:00] that is hell to use [19:00] one time i was playing dota and there was a ddos attack and had 50% packet loss [19:00] and the game was going terribly [19:00] so i cehcked with mtr etc [19:00] and then i thought it was doing well considering there was ilke 50% packet loss [19:01] with ssh if there's a bit of packet loss often typing another key can help things along [19:01] ilek if something's not appearing you can press backspace or something [19:02] but if it's completely broken often it's better not to touch anything at all [19:02] and have the connection time out / disconnect [19:03] *** DaCa has quit IRC (Ping timeout: 252 seconds) [19:15] I always use tmux anyway so the session stays alive [19:16] ahh yip [19:19] oh was that you that posted to outages@ acf? :) [19:21] who/what is acf? [19:22] the guy who brought up comcast/verizon packet loss before [19:22] or did i get it wrong? [19:22] http://kremvax.acfsys.net/smokeping.cgi?target=Remote.verizon-snloca [19:22] he linked that [19:23] well someone posted to outages@ [19:23] who uses arp [19:23] https://puck.nether.net/pipermail/outages/2014-February/006596.html [19:29] oh ok.. [19:30] I keep trying tmux, then switching back to screen. hehe [19:31] I like its default config, works out of the box [19:31] screen, I keep needing to paste in configs before I can use it [19:32] also the splitting of windows / panes is nice [19:33] I keep getting fumbled up by the default keybindings in tmux, so used to screen's [19:34] I need a basic "idiots guide to tmux" and just start using it with irssi. when I get more comfortable with it, then install it on all my machines with puppet. [19:35] http://www.amazon.com/tmux-Productive-Development-Brian-Hogan/dp/1934356964/ref=sr_1_1?ie=UTF8&qid=1392089683&sr=8-1&keywords=tmux [19:35] Amazon: "tmux: Productive Mouse-Free Development" [19:36] keybinding should be pretty basic to reconfigure [19:36] although you might not want to for sake of running screen within a tmux session [19:43] there are books on tmux? [19:43] wow [19:44] open resolver project is handy for identifying open resolvers on a network [19:44] http://openresolverproject.org [19:45] re: the ntpd thing, the default config for freebsd was changed when that vuln was identified, and freebsd10 ships with the modified config by default [19:47] http://www.itnews.com.au/News/372033,worlds-largest-ddos-strikes-us-europe.aspx [19:47] there was a ddos today [19:47] apparentyl 400 gigabit [19:47] oh and it was using ntp [19:48] did up_the_irons reports all come today? [19:49] whoa [19:49] jeez, that's enormous [19:49] so that could haev effected canda traffic [19:49] canada [19:50] cool, equiv of openresolverproject for ntpd [19:52] also, cloudflare's not on aws, but whatever, hahaha [19:52] i dunno [19:52] cloudflare is terrible [19:52] they may haev some stuff on aws [19:53] maybe staging or something, but they pride themselves on owning their hardware [19:53] i haven't found anything about this ddos on nanog yet [19:53] i was avoiding reading nanog to not get swamped :) [19:53] didn't see anything in nanog digests today [19:53] i don't know if 400 gigabit is actually the biggest ddos too [19:54] i been reading this carrier comparison [20:11] for some reason i can't find any other articles or mentioning of ddos [20:15] mercutio: all today, yeah [20:15] in fact, i dunno why i didn't look before, but like 30 minutes ago i noticed all our egress links are at like 300 Mbps! [20:15] ouch [20:16] lots of VPS' participating in the attacks (i'm sure innocent victims) [20:16] so i'm going to be blocking all NTP inbound [20:16] hmm [20:16] on all hosts [20:16] probably prudent [20:17] as a stop gap until people start fixing their setup [20:17] there's some debate whether it's a good idea to block all ntp [20:17] as some ntp like to use the same source/dest port [20:17] but yeah as stop-gap it makes a hell of a lot of sense [20:17] there's no debate in my mind when my network is hitting some target with > 1 Gbps of UDP [20:17] heh [20:18] well the debate was whether it shoudl be rate limited [20:18] or blocked ocmpletely [20:18] i reckon blocked completely [20:18] i'm kind of against rate limiting [20:18] rate limiting won't do shit [20:18] i mean, it will, but if 99% of the incoming is illegit traffic [20:18] your rate limit will effectively block all legit traffic too [20:19] so wtf [20:19] rather [20:19] it won't matter [20:19] hmm [20:19] won't people be hitting that 5 megabit udp rate limit anyway? [20:19] i shouldn't distract you [20:19] that's only in one direction [20:19] the wrong direction ;) [20:20] and yes, i'll take questions later :) [20:23] oof [20:24] for freebsd guests: http://www.freebsd.org/security/advisories/FreeBSD-SA-14:02.ntpd.asc [20:32] i still can't see anything on nanog [20:32] i wonder if tehre's another mailing list i should follow too [20:33] someone posted about it on nznog [20:59] *** DaCa has joined #arpnetworks [21:05] *** pcn has joined #arpnetworks [21:07] So I saw on some intertwitters about ntp blockage? [21:07] I don't have ntp running, but now that you mention it, is there an internal ntp server that can be peered with at the moment? [21:09] pcn: its inbound ntp requests, outbound as to get time from say pool.ntp.org should work just fine [21:24] OK [21:26] it's still valid question [21:26] i don't know of any [21:26] i think i just use pool.ntp.org [21:35] whats the nmap check or ntp check to ensure a host isnt configured incorrectly so as to be used in a UDP / ntp based DDOS attack? [21:35] uhh is saw something somewhere [21:36] [21:46] *** BryceBot has quit IRC (Excess Flood) [21:46] *** BryceBot has joined #arpnetworks [22:19] mnathani: either upgrade ntp or just disable monlist command [22:20] up_the_irons: that nmap thing checks for monlist [22:21] so you could port scan your ranges if you wanted to find out who is vulnerable to it [22:21] mercutio: oh sweet [22:21] which frmo your own ip could prob bypass any blocks [22:44] this is old, but still looks like it'd work: [22:44] http://railspikes.com/2007/6/1/rails-email-processing [22:45] no need to set up procmail or Postfix filter to fork into ruby process. just have a daemon check a special email box! [22:49] cool. [22:49] not that it really matters which way it is done [22:56] up_the_irons: roger [23:05] mercutio: yeah, I see that in the smokeping [23:06] after prodding NTT a bit more [23:06] http://paste.unixcube.org/k/246aaa [23:07] acf_: was it you that posted to outages@? [23:07] hmm? I emailed noc@us.ntt.net again [23:07] not technical in nature [23:07] oh i just saw that same address as you were saying on outages mailing list [23:08] how do i search scrollback? :) [23:08] peering disagreement or something likely [23:08] so it wasn't you that posted to oustages mailing list? [23:08] nope [23:08] https://puck.nether.net/pipermail/outages/2014-February/006596.html [23:09] maybe it not someone in irc even [23:09] any connection to the recent ddos news things you think? [23:09] i was wondering that [23:09] but i don't think it is [23:09] esp with your email response [23:10] it's ntt getting into messy situation like cogent [23:10] with not wanting to pay to send data [23:10] i imagine [23:10] wow. that guy sounds exactly like me [23:10] see how i wondered? [23:10] he even on arp :) [23:11] idk if "not technical in nature" means ntt/verizon is purpousely degrading connectivity [23:11] could be [23:11] or just that verizon/ntt have to negotiate bigger pipes to take the data [23:11] did you see the uhh [23:11] god damnit [23:11] i weant to find a way to find urls i pasted to irc :) [23:12] http://arstechnica.com/information-technology/2014/02/netflix-performance-on-verizon-and-comcast-has-been-dropping-for-months/ [23:12] Ars Technica: "Netflix performance on Verizon and Comcast has been dropping for months" [23:12] is that something bryce can do? [23:12] it's the same two providers [23:12] even if diff origin [23:13] i think ntt is generally considered tier 1 and cogent not though? [23:13] but they're both huge [23:13] yeah [23:13] cogent is usually considered tier crap afaik [23:13] heh i was reading on nanog about cogent [23:13] again :/ [23:14] but yeah i not a fan [23:14] godaddy is crap tier too [23:14] i still reckon up_the_irons should just route verizon/comcast a different way [23:14] maybe with max prefix limit [23:14] a bit of testing seems to reveal that rerouting through nlayer would still go through ntt [23:14] these things never seem to get fixed very quickly [23:14] and it usually gets worse before it gets better [23:15] we'll probably have to wait for level3 if up_the_irons wants to reroute [23:15] oh [23:15] idk much though [23:15] he has tata too [23:15] but i dunno [23:15] level3 is sure to fix it [23:15] how did you test via nlayer? [23:15] yeah, it doesn't look like we're near the end of this [23:15] oh from a lg? [23:16] i exepct level3 shouldn't take long to get connected up [23:16] i imagine it's just however long it takes to get a cross connect [23:16] which should be quicker for a big data centre you'd think [23:16] i mean they could probably turn it up tommorow if they felt like it [23:17] but if you tried to ask for tommorow they'd probably want to charge heaps for urgency [23:17] yeah. I certainly hope it's soon [23:17] well, nlayer-> verizon is direct [23:17] but he'd probably have to go there to plug in cross connect [23:17] but verizon-> nlayer is via ntt [23:17] oh [23:17] but it's -> verizon that is bad [23:18] judging by my routing via verizon being fine [23:18] so maybe that would fix it [23:18] i doubt reverse path is via verizon [23:18] but i dunno trace me ? [23:18] 202.49.67.22 [23:19] (from verizon) verizon->ntt->new zealand stuff [23:19] i actually think the routing side of things is somewhere the internet could really improve [23:19] so it's ntt return with no packet loss [23:19] which city is the verizon-> ntt in ? [23:19] oh of course california [23:19] is it san jose or los angeles? [23:19] or something else? [23:19] lax for me [23:20] and yeh we tested sending via ntt in la and sj [23:20] and both were bad [23:20] and sending via verizon in la was fine [23:20] yeah, no packet loss to you [23:20] you have nlayer path to verizon? [23:20] hmm [23:20] i misplaced your ip [23:20] i dunno what route it taking atm [23:20] 108.40.173.223 [23:21] yeh that forward path via verizon [23:21] uhh it's showing packet loss now [23:21] i wonder if it doesn't like two traceroutes at once [23:21] probably verizon just really sucks [23:21] i'll pingplotter from the other ip [23:22] yeah it looks fine [23:22] but i think it is rate limiting icmp too [23:22] oh no now some loss [23:23] this was all fine last night! [23:23] the internet is breaking down! [23:23] it does that [23:23] it's not as bad as before [23:23] it's 1.14% [23:23] and it was easily 7% usually going via ntt [23:24] http://kremvax.acfsys.net/smokeping.cgi?target=Remote.verizon-lsanca [23:24] yeh i saw that earlier today [23:24] the overnight thing is like wow [23:24] i can smokeping you from nz maybe? [23:24] that overnight thing was awesome [23:25] if it's clean it suggests that it's single direction [23:25] the latency dropped from 30-40ms. I wonder if the ARP route changed [23:25] i doubit it [23:25] it lpooks like congestion [23:25] but it's hard to know [23:25] it dropped off a bit sharply. does it do that? [23:25] you mind if i add you to my smokeping? [23:26] go for it [23:26] sometimes [23:26] i have curl testing too [23:26] but i assume you're not hosting any files on your dsl :) [23:26] nope [23:26] what was that comcast ip? [23:26] oh it was comcast.net [23:27] also 72.55.8.69 [23:27] btw my friend in sj on comcast cable wasn't packet loss [23:27] what's 72.55.8.69? [23:27] that's via level 3 forwaered route [23:27] the non-rate-limiting router in front of work's internet [23:27] work blocks icmp :0 [23:27] ahh [23:28] is it on level3? [23:28] that's comcast [23:28] normal comcast is via ntt [23:28] whereas this is via level3 [23:28] no ip's even say comcast.net [23:28] strange. I think they have some legacy IP address block, but I didn't think it would affect routing [23:29] it says comcast business [23:29] yeah, that's it [23:29] as13385 [23:29] wheras comcast.net is AS7922 [23:30] ok first syas comcast telecommunications, second says comcast cable [23:30] both go ntt-> comcast via tata over arp [23:30] not from here though [23:30] i'll do b oth [23:31] arp doesn't have level3 yet [23:31] it will be interesting to see the difference [23:31] maybe i shoudl do a subgroup [23:32] nah screw it [23:32] i wnat to be able to subgroup and not subgroup at the same time [23:32] it's handy scrolling through liwst