Any2 for v4 is pretty much back (but lots of peers are down due to CoreSite's renumbering; just gotta wait for everyone to renumber) hmm i still seeing as11799 that's outgoing to two places though incoming the same from one place since the majority of peers are down due to renumbering, that is expected http://openntpproject.org <- this also scans ip ranges (up to a /22 at a time) looking for ntpd w/insecure config http://openresolverproject.org for open dns resolvers Depends on client and whether you're logging... But you're probably thinking of "/lastlog" 23:06:39 < mercutio> how do i search scrollback? :) Well you can try using @log_search but beyond that, I don't know of a good IRC interface for that sort of thing. (At least not one I'm willing to write :P) 23:10:20 < mercutio> i weant to find a way to find urls i pasted to irc :) heh... my ARP IPv6 tunnel is about 1/2 the latency of my HE IPv6 tunnel. Yay ARP yeah, about the same here - the closest HE tunnel ep was in LA, but it's way oversubscribed and my latency to it was 80ms +/- 20ms vs 30 ms +/- 2ms, haha I'm pointed at Seattle, being just 300mi away and get ~100ms or so, and ARP (much further away) is closer to 50ms huh. (too busy to look at traceroutes, but needless to say it makes little sense and I did pick the shortest, quickest POP at the time) weird huh. looks like after that last 6500 reboot, my latency's actually 50ms instead of 30 brycec: do you have a guide you could link to that describes how you setup the ipv6 tunnel using ARP? oh well mnathani: obviously depends on your OS... I followed m0unds's guide and realized that it's as simple as setting up matching (Debian) v4tunnel statements on either end. m0unds' guide was for FreeBSD and Juniper SRX gear, but I got the gist and you need to have the /48 enabled I assume Note I just have the tunnel up, I don't have routing or /64 handoff setup yet Yeah, though you could route /128s I guess? I dunno, not an expert. k up_the_irons: can't it be on both numbers at once? maybe i shoujdl just log oh i am logging it seems that's better darnit http://kremvax.acfsys.net/smokeping.cgi?target=Remote.voipms-lsanca also, anything ipv4 on HE you got blocked? your dsl latency is starting :/ no, arp -> anything through trit is broken oh what ping he.net ping losangeles.voip.ms maybe any2ix issue yeh hmm he having looking glass oh it works from there I'm confused maybe need a diff trace point did someone block icmp somewhere in one direction? well this is traceroute so maybe udp lg.he.net actually reverse path filtering can look like that sometimes but it looked like it was coming in vl5.s1.lax.arpnetworks.com unless that new box calls itself that telnet on port 80 not working too so, a ping to arpnetworks.com through the he lg works well to my host but a ping from 4or6.com to he.net breaks yeh but if there is linux rp_filter on the new router it won't allow a response to come back for ping if it hasn't seen it go out on that router and this outbound path is via trit.net so of v5.s1 is thew new host linux defaults to rp_filter set to 1 and you need to set it to 0 or it'll behave just like this okay. I suppose that would explain it and itg was just done last night up_the_irons: you around? so it may be that dns is wrong do you think it was intentional? the filter nope it's broken it's not icmp onyl issue port 80 doesn't work ah yes it may be connection tracking too it's not necessarily rp_filter but both can accomplish the same thing http://kremvax.acfsys.net/smokeping.cgi?epoch_start=1392105600;hierarchy=;epoch_end=1392153357;target=Remote.voipms-lsanca;displaymode=n;start=2014-02-10%2024%3A00;end=now;Generate!=Generate! the internet isn't normally symmetric 5:00am ish i think he was talking about making changes 11 hours ago hmm taht 9 horus ago? between 5:10am and 5:15am exactly 15% packet loss on the last sample i couldn't find any sites oging over any2ix las tnight but i didn't try that hard digitalwest.net works does it go over any2ix back? idk, the lg has a password it's not that it's going out trit.net, it's that it's coming back via any2ix what not for me oh dw one yeah http shoudl be broken from he.net too but they don't have any http tests looks like losangeles.voip.ms is @ quadranet fwiw, i can't ping it from anything i have (arp, home, work, nada) m0unds: pings for me from TWC PING losangeles.voip.ms (96.44.149.186) 56(84) bytes of data. 64 bytes from 96.44.149.186.static.quadranet.com (96.44.149.186): icmp_seq=1 ttl=51 time=45.5 ms And from comcast I can ping that from my arpnetworks vps I cannot ping it from ARP I can ping it from Chunkhost though. http://sprunge.us/JROF On ARP, I cannot trace path coresite *past does 1gbit ports have a different v4 router? toddf: maybe I don't even see coresite it's whether return path is coresite was the issue (i think) Mine on ARP: 1 174.136.103.129 (174.136.103.129) 23.764 ms 23.790 ms 24.034 ms 2 v440.r6.lax2.trit.net (208.90.34.78) 0.603 ms 1.152 ms 1.147 ms heh well should be symmetric or not at all :) did you guys look at my sprunge paste? I can clearly get to losangeles.voip.ms from my arpnetworks vps can you telnet www.he.net on port 80 ? anyone else here `testing' the 1gbps ports? with vps? yeh I'm on a dedicated machine cos that's really the best real test same diff it doesn't work on dedicated for me I can hit www.he.net:80 both on v4 and v6 it maybe some subnets are ok toddf: why are you immune? :) someone good with looking glass ? just do a traceroute to your ip, see if it hits v5.. if some subnets are working, its as if a bgp is not advertising all or something mercutio: look at my sprunge post! http://sprunge.us/JROF no 208.79... no 174.136... no 206.125... i'm getting "permission denied" to www.he.net m0unds: weird permission denied sounds like a user running traceroute that requires root you mean using telnet? telnet: Unable to connect to remote host: Network is unreachable i get that this is me to he.net: http://sprunge.us/BiPP yea, it's throwing a 403 when i try to curl it - might just be preventing curl from retrieving it todd: mind telling us your ip? 3.v.freedaemon.com ;-) oh it doesn't even accept connection for me i get nothing on ipv4, but i get 9ms on ipv6 via mtr to www.he.net cool toddf you're on s7 acf: how did you figure that out? I'm on s1 oh i see yeh so am i http://paste.unixcube.org/k/819449 so yeh it's working for toddf cos he's on s7 and yea, via v4 i'm going out over trit.net and it fails m0unds: i think it's return path causing issues though can't cut and paste that nicely telent -4 3.v.freedaemon.com 1234 -> bounces you to v4 www.he.net just incase there's any confusion for lg.he.net heh i think we have to wait for up_the_irons to look into it oh cool gimme a min to go through the scrollback you must be quick reader :) it looks like connectino tracking or rp_filter i figure but that's only if v5 is coresite on the new box well yes it is taking longer than aminute? whoops *minute ;) mercutio: actually, i just thought connection tracking too from some support tickets i got. i just disabled it on r1.lax (should not have been on :( let's see if that helps (i see more traffic flowing now) fwiw I can traceroute to losangeles.voip.ms from ARP, same route through coresite as before. Guess coresite got their act together. ah cool brycec: so that made a difference? Maybe, or coresite fixed things for all I can tell. It's been ~2hrs since I tried and it failed :p ok www.he.net accepts connection on port 80 now so yeh i think it fixed up_the_irons: do you have a time in mind that level3 is coming up? mercutio: they say by the end of the month i'll have an LoA for the x-conn, then like, a week after that, we turn up oh yip just this ntt->verizon issue seems like it might not be resolved until then and then only if it goes via level3 outobund it was affecting acf rather than me though mercutio: yeah, the peers *could* be on both numbers at once, but since I was moving Any2 anyway to new gear, I decided to drop the old numbers ahh ok and there's that bgp collective fallback and it helped minimise broken things :) yeah, the next shortest path is generally The BGP Collective, so impact was just 1 extra hop cool, i found different hosts on NLNOG ring that have inbound paths of: Trit, NTT, nLayer but still trying to find one on an Any2 peer (or rather, one that takes that path) would help to save that one for future diagnostics yeh but it only makes sense in the short term in the longer term, there'll be way more options like finding stuff that goes via bgp collective isn't hard Oy vey... My smokeping slave config (the configuration pushed to each smokeping slave) is 248k (according to the http log) should I see if I can play Eve over my VZW tether? and the result: yeah, it works hahaha must be a low congestion vzw tower their lte gear is so hideously oversubscribed in NM/CO it's absurd Is there something blocking ntp traffic? to and from the VPSs? yes yes well, *to* the VPS use a source port other than 123 https://twitter.com/arpnetworks/status/433094185122414592 TWITTER: We have blocked all incoming NTP traffic to VM hosts; many were unwittingly participating in UDP amplification attacks (Tue Feb 11 04:24:34 +0000 2014, retweeted 4 times) ah But actually, i am just now applying a different filter okely dokely I am opening up NTP, but the misconfigured hosts will be blocked yay What constitutes misconfigured? monlist it participates in amplification attacks lol did you try that nmap cmd? srsly, we had over 500 Mbps of traffic going out last night from misconfigured NTP servers Holys iht. mercutio: no, was having trouble getting all the dependencies protip: When writing Smokeping targets, don't forget to include host= Looking at my bandwidth, it's looking like my system was not participating, hopefully it would be noticable ahh I've got this for my restrict statement: restrict default nomodify notrap noquery up_the_irons: does arp have ntp servers? Yep, you should be fine. Easy to test yourself though. i want to see 1.2.3.<1-3> be anycast ntp servers to go along with the proposal for 1.2.3.4 to be a standard anycast dns mercutio: no Ah, I see, it's the noquery that should take care of it. "disable monitor" is also an easy way to fix it most people prob just use the pool anyway You know what. That reminds me. uh oh, come back BryceBot! openntp also fixes it does kvm actually require everyone run their own ntp clients? i've kind of wondered that for a while mercutio: host time tracking is independent of guest time tracking so yes Hey, cool, I'm talking ntp again. so you can cronjob a command to set time against a remote system or you can use a ntpd openntpd (I'm running it) defaults to client mode only, you have to explicitly uncomment the 'listen *' bit just confirmed I am only a ntp client, so not likely contributing to the 500mbit of ntp traffic last night it doesn't amplify even if it's listening too removing '3.v.freedaemon.com:1234' redirect to he.net now that the problem it was in theory helping diagnose is now fixed from the looks of the volume of vulnerable hosts that have been reported, it appears many hosts _default_ to the bad behavior good thing i never run ntp! i just occasionally hire a dwarf in a shoe to tweak the system clock you can run ntp, it's when ntp /listens/ for requests that it's a problem all you have to do is toggle of mon and it's fine, and it can sync to pool.ntp.org or time.nist.gov or whatever off* mon up_the_irons: i think at least freebsd 9 defaults to being vulnerable it does until you run freebsd-update like you should do anyway 8.3-9.2 all default to listening, run freebsd-update fetch & install and it's patched it's been available as a patch since january tehre was as huge ddos over new years there was also a big one on like 12/25, which is when freebsd released the advisory to make config changes someone even mentioned it in here that same day at least i thought it was the same day meh i'll just switch the fbsd box to openntpd mercutio: damn.. we have SOOO many fbsd 9 hosts and, big surprise, most people don't maintain their systems when was this patched? i linked the advisory from freebsd yesterday up_the_irons: do you offer freebsd 10 yet? january somesuch - they posted the original advisory in december freebsd 10 adds zfs root support :) http://www.freebsd.org/security/advisories/FreeBSD-SA-14:02.ntpd.asc mercutio: ISO Only http://svnweb.freebsd.org/base/head/etc/ntp.conf?view=log&pathrev=259973 original mention yeah it's hard to keep systems up to date there's an even bigger problem with routers and so on with ntp as they're even less likely to be kept up to date i did see that advisory, didn't read it >_> i've been using openntpd for years though.. i have a crontab set up to execute freebsd-update cron, which emails me if there are new updates the problem is it's not people who are "reasonably connected" that are likely to be at fault as much as people who have no idea in 2014, it's sort of negligent to not maintain systems people still don't do it, but i still think it's shitty regardless s/negligent/common/ in 2014, it's sort of common to not maintain systems i can s/ your text :) commonality and negligence aren't interchangeable it's what is vs what should be i suppose i could add freebsd-update cron it deos remind me though, i should follow freebsd security list http://www.freebsd.org/security/advisories/FreeBSD-SA-14:01.bsnmpd.asc that's also significant How many were affected? pretty sure I modified mine correctly a few weeks back quite a few s/your/any/ i can s/ any text :) even ages back About 20 lines or so My smokeping data folder is 2.8GB :( also - http://blogs.freebsdish.org/portmgr/2014/02/03/time-to-bid-farewell-to-the-old-pkg_-tools/ What patchlevel was 9.2 patched? s/ - pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng brycec: what step size? mercutio: still default 2014-01-14 19:42:28 UTC (releng/9.2, 9.2-RELEASE-p3) brycec: you must be doing a lot of probes :) About 200 hosts now and 5 slaves if you're doing lots you may want to consider reducing the ping packet size ah, I see I got an email about that a few days ago. time to updatge i just started doing smokeping on arp + FPing binary = /usr/bin/fping packetsize = 32 i have that cool s/slaves/monitoring hosts (4 of which are slaves) and 5 monitoring hosts (4 of which are slaves) meh smokeping back in a minute. you don't like it? not really no i especially don't like the CGI webapp i don't like it how it hides minimum/maximum in the period yeah, cgi makes me sad mostly it shows averages for the monitoring period how it reloads all the time? just don't like it in general i find it useful i run it on hardware directly at home ^ and i'm not going to write my own yet what a freakin' day (or week!).. and it's only the start... up_the_irons: does your Bird setup support 4 byte AS numbers? mnathani: i believe so anything modern does :) everything supports 4 byte asn these days but some things use dot format now i can't find it in the docs, bah ah found it so yes, my BIRD setup supports 4 byte ASNs cool is bird using dot format? it's not using dot format whats an example of a 4 byte only ASN? 234567 AS234567 has not been visible in the global routing table since March 09, 2011 i meant it as an example ahh https://www.ietf.org/rfc/rfc5396.txt for the diff bw asdot and asplain i like asdot, but asplain is standard now pretty much yea, i haven't seen asdot in a while i don't really work with internet-connected systems a ton, though i use openbgpd, which uses asdot notation m0unds: is there an air gap between your systems and the internet? and any new asn's now days are all 32 bit Is it possible we might outgrow that limit on number of networks and need to expand to more than 4 byte ASNs yes but it's unlikely i think it's more prudent to replace bgp with something better mnathani: what i mean is that i'm not a network engineer with internet-connected systems anymore of* internet-connected* there's a slow gradual shift to having routing decisions being made globally, rather than at every point in the network so if a talks to b talks to c talks to d then along at each hop it decides where to go next as a hobbyist with virtual servers, i couldn't give two shits about which ASNs are which :) so c might decide to talk to a and loop it all over again global routing decisions sound as smart as software defined routing ie, sounds bad it's similar. it's not necessarily a bad thing but some kind of hybrid solution can be useful can you give an example? i had a kind of nifty idea of how things could work better, but a lot of decisions are motivated by large companies and so you'r not really going to change them what's the idea? s/can/could/ could you give an example? well basically you pay to get traffic to a point near the user forward only routing so like you pay to get traffic to amsix from los agnels err los angeles and then the path between those two points can be varied and you have per minute charging or such and you can choose to take lower cost or lower latency/higher badnwidth paths and as more people choose the better paths the cost goes up like a stock exchange sounds like what internap did with their routing engine so when there's failures etc cost will tend to go up i think it's a great idea if we could get it on an open basis and when there is idle capcaity cost goes down so you might have a better path while it's cheaper, then shift to a cheaper path when cost goes up because you can't redally change how people send you traffic, only how you send them traffic yea damn, dis nigga be worn OUT i wrote a shitload of code today, but the biggest drain was 2 challenging problems/bugs i played video games and drank whiskey what kind of whiskey? (v games be damned) balcones brimstone i've not had that one yet http://www.balconesdistilling.com/products yea, looks worth trying how do you like it? the smoke is nice it's kinda sweet - first whiskey i've found that my wife will actually drink where does it lie? hm interesting it's pretty up front, smoke-wise almost like a firey nose to it much mellower than it smells though sweet and smokey, you know, that sounds just about right for texas bbq sauce and such yeah, haha well i'll keep an eye out for it http://www.youtube.com/watch?v=5tm23wDVU2U YouTube Education: "Grand Designs S09 E01 The Apprentice Store, Somerset SD ( Standard Definition )" by Roland Marginas (49m 3s), 27,742 views, 73 likes and 7 dislikes. Uploaded 2013-06-26T09:18:34.000Z. there's something for you took a while to find it locally - none of the bigger local liquor joints carried it m0unds: the main actual issue with implementing would be getting mpls connections cheaply on a usage basis or such, and getting people onboard to use it brb getting high err..? but i'm actually in favour of per-bit-charging rather than block pipe charging not sure if you meant to tag me there, haha because it encourages people to cull "bad" traffic as a way to save money rather than preserve performance. err i meant to tag robonerd it's an interesting idea, but i could see corporations figuring out ways to abuse it how so? it's kind of the way electricity works ehh, there are regulatory bodies that protect the cost of electricity delivery in the US dunno if that's the case abroad but PRCs prevent price gouging and stuff even to businesses? yep including big customers? yep. they can schedule pricing differently based on use here big customers can pay varaible power costs and get cheaper power. most of the time it can be dynamic depending on industry and consumption but as soon as like a power station goes or such prices jump heaps PRCs here require approval to raise rates but it means if you err if you're doing stuff that you can temp shut off when power use is highest, that uses a lot of power, then you can get cheaper power the rest of the time if it's reasonable, for instance, if you need to invest more money in delivery equipment or whatever, they can approve it pretty easily which happens for a few industrial type things. yea, they do that for things like arc furnaces for steel production and stuff yeh but that's how power works in general then on top of that are residential plans that offer smoother pricing they still have fixed rate schedules for large stuff in the US it's just a matter of whether it's high demand hours or not ahh ok, so it doesn't take outages into consideration? i started thinking about this more when there was that huge flooding incident in east coast US and some providers were completely screwed to europe ah didn't really see much local coverage of the extent of problems but reading overseas stuff it sounded like lots of datacentres did silly things like have their generators in baseemnts. so when there was flooding they couldn't run their generators. yeah - it sucks that there are so many facilities in areas that aren't well suited to modern stuff the thing is it's expesnive to fix these things not a ton of modern infrastructure, or stuff slapped together so if you want to move all of your generators to 4th floor from basement, it'll cost real money and when you say "what if there's a flood" people think it's like a biblical thing like noah's ark and not going to happen to them. until actual issues happen people don't tend to want to sepnd money yep even then with those that did, some people couldn't get fuel for generators. and "best advice" now seems to be that you should have 3 sources of fuel california has all the potential earthquake stuff going on there was a blog that was kept by some guys in a DC in louisiana during/after hurricane katrina and i'm sure most of the datacentres are pretty good for erathquake protection but if there's fibre breaks, there could be a long time to restore due to being in "dangerous" areas there may be some typhoon risk there too? http://interdictor.livejournal.com/2005/08/28/ ^ it was that blog there - intercosmos media group or something based in new orleans in CA? i think it's pretty limited typhoon risk not out of the realm of possibility, but i think earthquakes are more likely than typhoons by far ok well i'm far away so i don't really know the risks yeah power issues maybe socal has a super high demand for power and water i think water issues are very likely given an earthquake given that there is already water shortages If anybody is interested (mercutio, up_the_irons), I've increased my smokeping resolution to 1 minute. brycec: cool @smokeping https://smokeping.cobryce.com/ did you tweak your existing rrd thing? you have to when rrd has diff step size mercutio: I just nuked them Totally redesigned the rra's ok that works that's usually what i do :) I played with the idea, but I realized that the historical data isn't really that important whch reminds me i was going to see how verizon had been doing only 5% loss atm Which also played into the redesign of the rra's - I don't keep data beyond 6mos, and it's weekly averages past 1 wk interesting i sse the ping rising with forward path verizon, as well as forward path via ntt so i think there's dual issues, cos packet lsos doesn't happen when sending via verizon apparently another ddos is happening atm oh dear well arp shoudln't be contributing at least What proto is being used to attack? ntp OK, so same attack. yeh happened new years and xmas too brycec: cool Man, I wanna watch the olympics. It's the only time I've ever wished I had a VPS in some other country. :D i'm watching it every night, while coding / networking / bgp'ing ;) this is really cool. i've finally been able to enumerate some NLNOG hosts according to which incoming path they take to us: NTT - lchost01.ring.nlnog.net nLayer - doruknet01.ring.nlnog.net Trit - teamix01.ring.nlnog.net Mzima - inerail01.ring.nlnog.net Any2 IX - vocus01.ring.nlnog.net That should help greatly with diagnostics in the future cool.