[02:25] Any2 for v4 is pretty much back (but lots of peers are down due to CoreSite's renumbering; just gotta wait for everyone to renumber) [03:15] hmm i still seeing as11799 [03:17] that's outgoing to two places though [03:23] incoming the same from one place [04:08] since the majority of peers are down due to renumbering, that is expected [07:42] http://openntpproject.org <- this also scans ip ranges (up to a /22 at a time) looking for ntpd w/insecure config [07:44] http://openresolverproject.org for open dns resolvers [09:39] Depends on client and whether you're logging... But you're probably thinking of "/lastlog" 23:06:39 < mercutio> how do i search scrollback? :) [09:40] Well you can try using @log_search but beyond that, I don't know of a good IRC interface for that sort of thing. (At least not one I'm willing to write :P) 23:10:20 < mercutio> i weant to find a way to find urls i pasted to irc :) [10:44] heh... my ARP IPv6 tunnel is about 1/2 the latency of my HE IPv6 tunnel. Yay ARP [10:45] yeah, about the same here - the closest HE tunnel ep was in LA, but it's way oversubscribed [10:45] and my latency to it was 80ms +/- 20ms [10:46] vs 30 ms +/- 2ms, haha [10:47] I'm pointed at Seattle, being just 300mi away and get ~100ms or so, and ARP (much further away) is closer to 50ms [10:47] huh. [10:47] (too busy to look at traceroutes, but needless to say it makes little sense and I did pick the shortest, quickest POP at the time) [10:47] weird [10:48] huh. looks like after that last 6500 reboot, my latency's actually 50ms instead of 30 [10:48] brycec: do you have a guide you could link to that describes how you setup the ipv6 tunnel using ARP? [10:48] oh well [10:49] mnathani: obviously depends on your OS... I followed m0unds's guide and realized that it's as simple as setting up matching (Debian) v4tunnel statements on either end. [10:50] m0unds' guide was for FreeBSD and Juniper SRX gear, but I got the gist [10:50] and you need to have the /48 enabled I assume [10:50] Note I just have the tunnel up, I don't have routing or /64 handoff setup yet [10:50] Yeah, though you could route /128s I guess? [10:50] I dunno, not an expert. [10:52] k [11:09] *** avj has joined #arpnetworks [11:32] *** DaCa has quit IRC (Ping timeout: 260 seconds) [12:16] up_the_irons: can't it be on both numbers at once? [12:21] maybe i shoujdl just log [12:22] oh i am logging it seems [12:24] that's better [12:44] *** PatrickINIZ has joined #arpnetworks [12:45] *** robonerd has quit IRC (Remote host closed the connection) [13:03] darnit [13:03] http://kremvax.acfsys.net/smokeping.cgi?target=Remote.voipms-lsanca [13:06] also, anything ipv4 on HE [13:07] you got blocked? [13:08] your dsl latency is starting :/ [13:08] no, arp -> anything through trit is broken [13:08] oh what [13:08] ping he.net [13:08] ping losangeles.voip.ms [13:08] maybe any2ix issue [13:09] *** PatrickINIZ has left "http://iniz.com" [13:09] yeh hmm [13:09] he having looking glass [13:10] oh it works from there [13:10] I'm confused [13:10] maybe need a diff trace point [13:11] did someone block icmp somewhere in one direction? [13:11] well this is traceroute so maybe udp [13:11] lg.he.net [13:11] actually reverse path filtering can look like that [13:11] sometimes [13:12] but it looked like it was coming in vl5.s1.lax.arpnetworks.com [13:12] unless that new box calls itself that [13:12] telnet on port 80 not working too [13:13] so, a ping to arpnetworks.com through the he lg works [13:13] well to my host [13:13] but a ping from 4or6.com to he.net breaks [13:13] yeh [13:13] but if there is linux rp_filter on the new router it won't allow a response to come back for ping [13:13] if it hasn't seen it go out on that router [13:14] and this outbound path is via trit.net [13:14] so of v5.s1 is thew new host [13:14] linux defaults to rp_filter set to 1 [13:14] and you need to set it to 0 [13:14] or it'll behave just like this [13:14] okay. I suppose that would explain it [13:14] and itg was just done last night [13:15] up_the_irons: you around? [13:15] so it may be that dns is wrong [13:15] do you think it was intentional? [13:15] the filter [13:15] nope [13:15] it's broken [13:15] it's not icmp onyl issue [13:15] port 80 doesn't work [13:15] ah yes [13:16] it may be connection tracking too [13:16] it's not necessarily rp_filter [13:16] but both can accomplish the same thing [13:17] http://kremvax.acfsys.net/smokeping.cgi?epoch_start=1392105600;hierarchy=;epoch_end=1392153357;target=Remote.voipms-lsanca;displaymode=n;start=2014-02-10%2024%3A00;end=now;Generate!=Generate! [13:17] the internet isn't normally symmetric [13:17] 5:00am ish [13:17] i think he was talking about making changes 11 hours ago [13:17] hmm taht 9 horus ago? [13:19] between 5:10am and 5:15am exactly [13:19] 15% packet loss on the last sample [13:20] i couldn't find any sites oging over any2ix las tnight [13:20] but i didn't try that hard [13:22] digitalwest.net [13:22] works [13:22] does it go over any2ix back? [13:22] idk, the lg has a password [13:22] it's not that it's going out trit.net, it's that it's coming back via any2ix [13:22] what [13:22] not for me [13:22] oh dw one [13:23] yeah [13:23] http shoudl be broken from he.net too [13:23] but they don't have any http tests [13:27] *** DaCa has joined #arpnetworks [13:33] looks like losangeles.voip.ms is @ quadranet [13:34] *** mike-burns has quit IRC (Read error: Connection reset by peer) [13:34] fwiw, i can't ping it from anything i have (arp, home, work, nada) [13:34] *** mike-burns has joined #arpnetworks [13:34] *** ChanServ sets mode: +o mike-burns [13:34] *** KDE_Perry has quit IRC (Ping timeout: 260 seconds) [13:35] m0unds: pings for me from TWC [13:35] PING losangeles.voip.ms (96.44.149.186) 56(84) bytes of data. [13:35] 64 bytes from 96.44.149.186.static.quadranet.com (96.44.149.186): icmp_seq=1 ttl=51 time=45.5 ms [13:35] *** KDE_Perry has joined #arpnetworks [13:35] And from comcast [13:35] I can ping that from my arpnetworks vps [13:35] I cannot ping it from ARP [13:36] I can ping it from Chunkhost though. [13:36] http://sprunge.us/JROF [13:36] On ARP, I cannot trace path coresite [13:36] *past [13:37] does 1gbit ports have a different v4 router? [13:37] toddf: maybe [13:37] I don't even see coresite [13:37] it's whether return path is coresite was the issue [13:37] (i think) [13:38] Mine on ARP: 1 174.136.103.129 (174.136.103.129) 23.764 ms 23.790 ms 24.034 ms [13:38] 2 v440.r6.lax2.trit.net (208.90.34.78) 0.603 ms 1.152 ms 1.147 ms [13:38] heh [13:38] well should be symmetric or not at all :) [13:38] did you guys look at my sprunge paste? I can clearly get to losangeles.voip.ms from my arpnetworks vps [13:39] can you telnet www.he.net on port 80 ? [13:39] anyone else here `testing' the 1gbps ports? [13:39] with vps? [13:39] yeh [13:39] I'm on a dedicated machine [13:39] cos that's really the best real test [13:39] same diff [13:39] it doesn't work on dedicated for me [13:39] I can hit www.he.net:80 both on v4 and v6 [13:39] it maybe some subnets are ok [13:40] toddf: why are you immune? :) [13:40] someone good with looking glass ? [13:40] just do a traceroute to your ip, see if it hits v5.. [13:40] if some subnets are working, its as if a bgp is not advertising all or something [13:40] mercutio: look at my sprunge post! http://sprunge.us/JROF [13:40] no 208.79... [13:40] no 174.136... [13:41] no 206.125... [13:41] i'm getting "permission denied" to www.he.net [13:41] m0unds: weird [13:41] permission denied sounds like a user running traceroute that requires root [13:41] you mean using telnet? [13:42] telnet: Unable to connect to remote host: Network is unreachable [13:42] i get that [13:42] this is me to he.net: [13:42] http://sprunge.us/BiPP [13:42] yea, it's throwing a 403 [13:42] when i try to curl it - might just be preventing curl from retrieving it [13:42] todd: mind telling us your ip? [13:42] 3.v.freedaemon.com ;-) [13:42] oh it doesn't even accept connection for me [13:43] i get nothing on ipv4, but i get 9ms on ipv6 via mtr to www.he.net [13:43] cool toddf you're on s7 [13:43] acf: how did you figure that out? [13:43] I'm on s1 [13:43] oh i see [13:43] yeh so am i [13:44] http://paste.unixcube.org/k/819449 [13:44] so yeh it's working for toddf cos he's on s7 [13:44] and yea, via v4 i'm going out over trit.net and it fails [13:44] m0unds: i think it's return path causing issues though [13:44] can't cut and paste that nicely [13:44] telent -4 3.v.freedaemon.com 1234 -> bounces you to v4 www.he.net just incase there's any confusion [13:44] for lg.he.net [13:45] heh [13:45] i think we have to wait for up_the_irons to look into it [14:51] * up_the_irons checks things out [14:51] oh cool [14:53] gimme a min to go through the scrollback [14:55] you must be quick reader :) [14:55] it looks like connectino tracking or rp_filter [14:55] i figure [14:55] but that's only if v5 is coresite [14:56] on the new box [15:02] well yes it is taking longer than aminute? [15:02] whoops [15:02] *minute ;) [15:03] mercutio: actually, i just thought connection tracking too from some support tickets i got. i just disabled it on r1.lax (should not have been on :( [15:03] let's see if that helps [15:03] (i see more traffic flowing now) [15:03] fwiw I can traceroute to losangeles.voip.ms from ARP, same route through coresite as before. Guess coresite got their act together. [15:04] ah cool [15:05] brycec: so that made a difference? [15:05] Maybe, or coresite fixed things for all I can tell. It's been ~2hrs since I tried and it failed :p [15:06] ok [15:07] www.he.net accepts connection on port 80 now [15:07] so yeh i think it fixed [15:08] up_the_irons: do you have a time in mind that level3 is coming up? [15:09] mercutio: they say by the end of the month i'll have an LoA for the x-conn, then like, a week after that, we turn up [15:09] oh yip [15:09] just this ntt->verizon issue seems like it might not be resolved until then [15:09] and then only if it goes via level3 outobund [15:11] it was affecting acf rather than me though [15:17] mercutio: yeah, the peers *could* be on both numbers at once, but since I was moving Any2 anyway to new gear, I decided to drop the old numbers [15:21] ahh ok [15:21] and there's that bgp collective fallback [15:21] and it helped minimise broken things :) [15:22] yeah, the next shortest path is generally The BGP Collective, so impact was just 1 extra hop [15:26] cool, i found different hosts on NLNOG ring that have inbound paths of: Trit, NTT, nLayer [15:26] but still trying to find one on an Any2 peer [15:26] (or rather, one that takes that path) [15:26] would help to save that one for future diagnostics [15:28] yeh but it only makes sense in the short term [15:28] in the longer term, there'll be way more options [15:29] like finding stuff that goes via bgp collective isn't hard [15:29] Oy vey... My smokeping slave config (the configuration pushed to each smokeping slave) is 248k (according to the http log) [15:33] should I see if I can play Eve over my VZW tether? [15:34] *** avj has quit IRC (Ping timeout: 245 seconds) [15:39] and the result: yeah, it works [15:39] hahaha [15:40] must be a low congestion vzw tower [15:40] their lte gear is so hideously oversubscribed in NM/CO it's absurd [15:45] *** eryc has quit IRC (Ping timeout: 245 seconds) [15:46] *** eryc has joined #arpnetworks [15:50] *** eryc has quit IRC (Ping timeout: 245 seconds) [15:50] *** acf_ has quit IRC (Ping timeout: 245 seconds) [15:50] *** acf_ has joined #arpnetworks [15:52] *** eryc has joined #arpnetworks [15:52] *** eryc has quit IRC (Changing host) [15:52] *** eryc has joined #arpnetworks [15:56] *** awyeah has joined #arpnetworks [15:57] Is there something blocking ntp traffic? [15:57] to and from the VPSs? [15:57] yes [15:57] yes [15:57] well, *to* the VPS [15:57] use a source port other than 123 [15:58] https://twitter.com/arpnetworks/status/433094185122414592 [15:58] TWITTER: We have blocked all incoming NTP traffic to VM hosts; many were unwittingly participating in UDP amplification attacks (Tue Feb 11 04:24:34 +0000 2014, retweeted 4 times) [15:58] ah [15:58] * brycec wittingly participates :P [15:58] But actually, i am just now applying a different filter [15:58] okely dokely [15:58] I am opening up NTP, but the misconfigured hosts will be blocked [15:59] yay [15:59] What constitutes misconfigured? [15:59] monlist [15:59] it participates in amplification attacks [15:59] lol [15:59] did you try that nmap cmd? [15:59] srsly, we had over 500 Mbps of traffic going out last night from misconfigured NTP servers [16:00] Holys iht. [16:00] mercutio: no, was having trouble getting all the dependencies [16:01] protip: When writing Smokeping targets, don't forget to include host= [16:03] Looking at my bandwidth, it's looking like my system was not participating, hopefully [16:03] it would be noticable [16:03] ahh [16:03] I've got this for my restrict statement: restrict default nomodify notrap noquery [16:03] up_the_irons: does arp have ntp servers? [16:03] Yep, you should be fine. Easy to test yourself though. [16:03] i want to see 1.2.3.<1-3> be anycast ntp servers [16:04] to go along with the proposal for 1.2.3.4 to be a standard anycast dns [16:04] mercutio: no [16:04] Ah, I see, it's the noquery that should take care of it. [16:04] *** BryceBot has quit IRC (Ping timeout: 245 seconds) [16:04] "disable monitor" is also an easy way to fix it [16:04] most people prob just use the pool anyway [16:05] You know what. That reminds me. [16:05] uh oh, come back BryceBot! [16:05] openntp also fixes it [16:06] does kvm actually require everyone run their own ntp clients? [16:06] i've kind of wondered that for a while [16:06] mercutio: host time tracking is independent of guest time tracking [16:06] so yes [16:06] Hey, cool, I'm talking ntp again. [16:06] so you can cronjob a command to set time against a remote system or you can use a ntpd [16:07] openntpd (I'm running it) defaults to client mode only, you have to explicitly uncomment the 'listen *' bit [16:07] just confirmed I am only a ntp client, so not likely contributing to the 500mbit of ntp traffic last night [16:08] it doesn't amplify even if it's listening too [16:10] removing '3.v.freedaemon.com:1234' redirect to he.net now that the problem it was in theory helping diagnose is now fixed [16:11] *** BryceBot has joined #arpnetworks [16:11] *** BryceBot has quit IRC (Changing host) [16:11] *** BryceBot has joined #arpnetworks [16:11] from the looks of the volume of vulnerable hosts that have been reported, it appears many hosts _default_ to the bad behavior [16:19] good thing i never run ntp! [16:19] i just occasionally hire a dwarf in a shoe to tweak the system clock [16:20] you can run ntp, it's when ntp /listens/ for requests that it's a problem [16:20] all you have to do is toggle of mon and it's fine, and it can sync to pool.ntp.org or time.nist.gov or whatever [16:20] off* mon [16:23] up_the_irons: i think at least freebsd 9 defaults to being vulnerable [16:23] it does until you run freebsd-update like you should do anyway [16:23] 8.3-9.2 all default to listening, run freebsd-update fetch & install and it's patched [16:24] it's been available as a patch since january [16:25] tehre was as huge ddos over new years [16:26] there was also a big one on like 12/25, which is when freebsd released the advisory to make config changes [16:26] someone even mentioned it in here that same day [16:28] at least i thought it was the same day [16:28] meh i'll just switch the fbsd box to openntpd [16:29] mercutio: damn.. we have SOOO many fbsd 9 hosts [16:29] and, big surprise, most people don't maintain their systems [16:30] when was this patched? [16:30] i linked the advisory from freebsd yesterday [16:30] up_the_irons: do you offer freebsd 10 yet? [16:30] january somesuch - they posted the original advisory in december [16:30] freebsd 10 adds zfs root support :) [16:30] http://www.freebsd.org/security/advisories/FreeBSD-SA-14:02.ntpd.asc [16:31] mercutio: ISO Only [16:31] http://svnweb.freebsd.org/base/head/etc/ntp.conf?view=log&pathrev=259973 original mention [16:32] yeah it's hard to keep systems up to date [16:32] there's an even bigger problem with routers and so on with ntp [16:32] as they're even less likely to be kept up to date [16:32] i did see that advisory, didn't read it >_> [16:32] i've been using openntpd for years though.. [16:35] i have a crontab set up to execute freebsd-update cron, which emails me if there are new updates [16:36] the problem is it's not people who are "reasonably connected" that are likely to be at fault [16:36] as much as people who have no idea [16:36] * m0unds shrugs [16:36] in 2014, it's sort of negligent to not maintain systems [16:37] people still don't do it, but i still think it's shitty regardless [16:37] s/negligent/common/ [16:37] in 2014, it's sort of common to not maintain systems [16:37] i can s/ your text :) [16:37] commonality and negligence aren't interchangeable [16:37] *** Nat_RH has joined #arpnetworks [16:38] it's what is vs what should be [16:39] i suppose i could add freebsd-update cron [16:41] it deos remind me though, i should follow freebsd security list [16:44] http://www.freebsd.org/security/advisories/FreeBSD-SA-14:01.bsnmpd.asc [16:44] that's also significant [16:45] How many were affected? pretty sure I modified mine correctly a few weeks back [16:49] quite a few [16:51] s/your/any/ [16:51] i can s/ any text :) [16:52] even ages back [16:53] About 20 lines or so [16:54] My smokeping data folder is 2.8GB :( [16:55] also - http://blogs.freebsdish.org/portmgr/2014/02/03/time-to-bid-farewell-to-the-old-pkg_-tools/ [16:55] What patchlevel was 9.2 patched? [16:55] s/ - pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng [16:55] brycec: what step size? [16:56] mercutio: still default [16:56] 2014-01-14 19:42:28 UTC (releng/9.2, 9.2-RELEASE-p3) [16:56] brycec: you must be doing a lot of probes :) [16:56] About 200 hosts now [16:56] and 5 slaves [16:56] if you're doing lots you may want to consider reducing the ping packet size [16:56] ah, I see I got an email about that a few days ago. time to updatge [16:56] i just started doing smokeping on arp [16:56] * brycec increases packet size to make up for up_the_irons' 500mbps [16:57] + FPing [16:57] binary = /usr/bin/fping [16:57] packetsize = 32 [16:57] i have that [16:57] cool [17:01] s/slaves/monitoring hosts (4 of which are slaves) [17:01] and 5 monitoring hosts (4 of which are slaves) [17:01] meh smokeping [17:01] back in a minute. [17:01] *** awyeah has quit IRC (Quit: EPIC5-1.1.7[1705] - amnesiac : Do the gene pool a service... Add a bucket of chlorine today!) [17:01] you don't like it? [17:01] not really no [17:02] i especially don't like the CGI webapp [17:02] i don't like it how it hides minimum/maximum [17:02] in the period [17:02] yeah, cgi makes me sad [17:02] mostly [17:02] it shows averages for the monitoring period [17:02] how it reloads all the time? [17:02] just don't like it in general [17:02] i find it useful [17:02] i run it on hardware directly at home [17:02] ^ [17:02] and i'm not going to write my own [17:02] yet [17:03] * staticsafe converts pkg db to pkg2ng [17:53] what a freakin' day (or week!).. and it's only the start... [17:54] up_the_irons: does your Bird setup support 4 byte AS numbers? [17:54] mnathani: i believe so [17:54] anything modern does :) [17:55] everything supports 4 byte asn these days [17:55] but some things use dot format [17:57] now i can't find it in the docs, bah [17:59] ah found it [17:59] so yes, my BIRD setup supports 4 byte ASNs [17:59] cool [18:03] is bird using dot format? [18:06] it's not using dot format [18:12] whats an example of a 4 byte only ASN? [18:14] 234567 [18:14] AS234567 has not been visible in the global routing table since March 09, 2011 [18:15] i meant it as an example [18:15] ahh [18:18] https://www.ietf.org/rfc/rfc5396.txt [18:20] for the diff bw asdot and asplain [18:23] i like asdot, but asplain is standard now pretty much [18:26] yea, i haven't seen asdot in a while [18:26] i don't really work with internet-connected systems a ton, though [18:36] i use openbgpd, which uses asdot notation [18:36] m0unds: is there an air gap between your systems and the internet? [18:36] and any new asn's now days are all 32 bit [18:39] Is it possible we might outgrow that limit on number of networks and need to expand to more than 4 byte ASNs [18:42] yes [18:42] but it's unlikely [18:42] *** robonerd has joined #arpnetworks [18:42] *** robonerd has quit IRC (Changing host) [18:42] *** robonerd has joined #arpnetworks [18:42] i think it's more prudent to replace bgp with something better [18:43] mnathani: what i mean is that i'm not a network engineer with internet-connected systems anymore [18:43] of* internet-connected* [18:43] there's a slow gradual shift to having routing decisions being made globally, rather than at every point in the network [18:43] so if a talks to b talks to c talks to d [18:43] then along at each hop it decides where to go next [18:43] as a hobbyist with virtual servers, i couldn't give two shits about which ASNs are which :) [18:43] so c might decide to talk to a and loop it all over again [18:44] global routing decisions sound as smart as software defined routing [18:44] ie, sounds bad [18:45] it's similar. [18:45] it's not necessarily a bad thing [18:45] but some kind of hybrid solution can be useful [18:45] can you give an example? [18:45] i had a kind of nifty idea of how things could work better, but a lot of decisions are motivated by large companies [18:46] and so you'r not really going to change them [18:46] what's the idea? [18:46] s/can/could/ [18:46] could you give an example? [18:46] well basically you pay to get traffic to a point near the user [18:46] forward only routing [18:46] so like you pay to get traffic to amsix [18:46]