[00:08] down again? [00:08] *** grepidemic has quit IRC (Ping timeout: 252 seconds) [00:09] mercutio: no alarms yet, but i'll check [00:10] it seems back again [00:10] it was short [00:10] short enough to not be able to debug properly :( [00:10] wouldn't be the s7.lax router cuz it doesn't reboot that fast ;) [00:10] yeh [00:10] i wonder if it was ntt [00:10] ya know, actually, my home twcable is down right now [00:10] it was up via another host [00:10] which is nlayer hmm [00:11] maybe related,d unno [00:11] well when i was first looking there was that small ntt packet loss [00:11] i'm using my android and tmobile [00:11] ahh ok [00:11] damn debugging is annoying :/ [00:11] srsly [00:11] i got alerts too though [00:11] i got alerts then debugged [00:12] s7.lax uptime is 1 hour, 27 minutes [00:12] woo [00:12] ;) [00:12] heh [00:12] (and the alerts are on diff network) [00:12] but i assume it was probably ntt issue [00:13] is your hoem connection over ntt? did it come back? [00:14] haven't checked it [00:15] looks like it didn't go out, but hit 90% packet loss [00:16] for about 5 minutes [00:16] but it wasn't so bad when it first hit [00:16] so i'm going with ddos [00:16] or ntt fault [00:17] oh interesting, my route to comcast from new zealand screwed up for same period too [00:18] might be co-incidental thoug [00:20] @wa why did the chicken cross the road [00:20] Why did the chicken cross the road?;To get to the other side., (ha, ha) [00:20] @wa why is the sky blue? [00:20] Why is the sky blue?;The sky\'s blue color is a result of the effect of Rayleigh scattering. Shorter-wavelength blue light is more strongly scattered in the earth\'s atmosphere than longer-wavelength red light; the human eye perceives the color blue when looking at the sky as a result. [00:20] @wa why do we exist? [00:21] Couldn't grab results from json stringified precioussss. [00:21] * mercutio wonders what issue happened an hnour ago [00:21] @wa when will ipv6 reach widespread adoption [00:21] Couldn't grab results from json stringified precioussss. [00:21] well it wasn't on arp [00:22] but from my local monitoring quite a few things experienced packet loss then without me noticing [00:22] @wa will there come a time when ISPs will hand out IPv6 addresses and tunnel IPv4 traffic over it [00:22] haha [00:22] Couldn't grab results from json stringified precioussss. [00:22] @wa tldr [00:22] TLDR (acronym);too long didn\'t read [00:23] * mercutio goes back to tv [00:40] *** Guest40141 is now known as easymac [00:41] *** easymac is now known as Guest45464 [00:50] *** alienresidents has quit IRC (Ping timeout: 240 seconds) [00:53] *** alienresidents has joined #arpnetworks [01:06] *** gizmoguy_ has quit IRC (Ping timeout: 245 seconds) [01:07] *** grepidemic has joined #arpnetworks [01:13] *** gizmoguy has joined #arpnetworks [01:17] mmm whisky [01:17] mercutio: nah, we've not done anything with netamp [01:17] we're looking to have a play with netfpga in the near future though [01:41] *** Guest45464 is now known as easymac [01:41] *** easymac is now known as Guest55282 [02:02] dunno what gave me the idea that you did hmm [02:06] we used to run a thing called nettest [02:06] and our main software we're developing at the moment is called amp (http://amp.wand.net.nz/) [02:07] nettest + amp = netamp? :P [02:15] nice [02:16] i want some nice way to trace from multiple difference places and record what networks it traverses [02:16] and identify where things break [02:17] when arp had issues with ntt and any2ix i didn't have much luck finding a path that didn't use those two in a short space of time manually [02:17] we can pretty much do that [02:17] cool. [02:18] assuming you build your test schedule properly [02:18] well routes can change over time [02:18] i see path length interesting [02:18] best way to visualise an outage like that would be our matrix - http://amp.wand.net.nz/matrix/absolute-latency/both/nzamp/nzamp/ [02:18] oh it still hard to tell if it changes by a little [02:19] the site is kind of slow from here [02:19] dunno if it backend or what [02:19] sorry for speed. i have a new major database improvement to push out shortly [02:19] ahh ok [02:19] lambda is broken? [02:20] lambda is in the US :) [02:20] how often does it update? [02:20] everything else is in NZ [02:20] oh red is < 300 msec and > 160 [02:20] rather than > 300 msec [02:20] if you look at relative latency rather than absolute, you'll get a better picture of changes [02:20] our postgres database is a bit broken at the moment [02:21] ~200million rows of data in a very naive schema [02:21] mada doesn't measure that well on that either [02:21] I've got all that fixed, just need to convert all the data and push out [02:21] ahh right [02:21] so does this need lots of resources to run? [02:22] actually the loss thing is prob fine [02:22] nah i see vocus is the borken one hah [02:22] vocus is really broken [02:22] I'm not sure what they do to our AMP monitor machine [02:22] but we see packet loss to everywhere [02:22] rate limit icmp? [02:22] forwarded ICMP? [02:22] maybe [02:22] as we see packet loss end to end [02:22] on icmp? [02:23] yeah [02:23] or tcp/udp/etc? [02:23] this is my favourite graph type - http://amp.wand.net.nz/view/amp-traceroute-rainbow/9238/1393323647/1393496447 [02:23] i saw a route going via cogent drop icmp completely once [02:23] from a host doign monitoring [02:23] shows up broken routing quite quickly [02:23] it was very bizzare :/ [02:23] trademe bounces between wgtn/akdl a lot [02:23] note - trademe have the worst high availablity setup ever [02:24] a lot? [02:24] contsantly [02:24] try every 10 minutes [02:24] http://202.49.71.24:24/smokeping/smokeping.fcgi?target=Wider.trademe [02:24] yes [02:24] i.e their TTL on their A record [02:24] 1 msec to akld 10 msec to wgtn [02:25] http://amp.wand.net.nz/view/amp-icmp/2415/1393384891/1393476724 [02:25] but it pretty consistent [02:25] 3i think it similar behaviour really [02:25] yeah, their DNS server just randomly selects from AKL or WLG whenever you query it [02:26] then it gets cached for the TTL [02:26] yeh [02:26] i know it's mental [02:26] they should just do anycast [02:26] imo [02:26] i'm amazed they can share state so well between their datacentres [02:26] if they have a real outage on one of them, then people will reload page anyway [02:26] they don't afaik [02:26] how do they keep you logged in? [02:26] it's basically just a proxy type setup [02:27] because it goes to the same backend constantly [02:27] just with a cookie i think [02:27] not based on ip or anything [02:27] yeah, but the cookie is a session id [02:27] that the backend needs to know about [02:27] hence they must do constant replication between each site to keep session IDs in sync [02:27] yeh well it goes back to the db probably [02:27] what do they need to keep state on anyway? [02:28] that's why it's generally advisable to direct the same group of people to the same set of frontend/backend servers [02:28] your session [02:28] hmm [02:28] because the cookie doesn't have your username/pass in it [02:28] it has some secondary auth (like a session ID) to verify you've logged in [02:28] maybe they replicate [02:28] yeah they will [02:28] i dunno really [02:28] very very often [02:29] i had a slow image loading issue on one of them once [02:29] it was fine on one bad on the other [02:29] i tried contacting them to no avail :/ [02:29] but i don't see why they don't just cdn their images [02:29] and direct all their normal web traffic to one location [02:29] yeah [02:30] did you watch any of the NZNOG videos btw? [02:30] they don't even really need to cdn their traffic probably [02:30] from janurary [02:30] a couple [02:30] the GCSB one? [02:30] i watched fincham's one on rpki [02:30] a+++++ for the GCSB one [02:30] so funny [02:30] and the apnic one [02:30] i dunno there was terrible video/sound quality [02:30] and most of the talks sounded pretty uninteresting :/ [02:30] yeah the richard naylor couldn't be there this year (the dude who normally does the streaming) [02:31] i liked the idea of 1.2.3.4 as standard anycast dns [02:31] the GCSB one was awesome [02:31] and i think the idea of rpki is slightly interesting [02:31] watch the Q&A [02:31] geoff houston's talk was awesome as usual too [02:31] was chatting to him at the pub the night before [02:31] i dunno bad audio quality is common for most talks [02:31] oh god [02:31] i watched some of the one by that guy about not keeping state [02:32] i think that's hwen i stopped heh [02:32] i already know about state issues etc [02:32] haha, Roland? [02:32] yes [02:32] I feel asleep during that one [02:32] dibbins or something? [02:32] heh [02:32] it was a good message [02:32] but it didn't need to go on for what felt like 60 minutes [02:32] well before hand i ntoiced he'd give the same exact talk before [02:32] and then i noticed the slides were the same [02:33] it was kind of amusing [02:33] he is mr stateless firewall [02:33] i'm in between [02:33] you need state for NAT :/ [02:33] i hate nat [02:33] but snyhcronising is a pita [02:33] and arp's route just changed [02:33] did ntt go down again [02:34] (i left mtr running) [02:34] any2ix and ntt are down it looks like [02:34] but it maybe up_the_irons doing ios things :/ [02:34] cos it didn't generate alerts etc. [02:35] *** notion has quit IRC (Read error: Connection reset by peer) [02:35] *** twobithacker has quit IRC (Read error: Connection reset by peer) [02:35] *** Guest1774 has quit IRC (Read error: Connection reset by peer) [02:35] so this monitoring thingy is it light on clients? [02:35] *** twobitha- has joined #arpnetworks [02:35] *** notion has joined #arpnetworks [02:35] netamp [02:36] and does it require lots of ram on server? [02:36] mercutio: yup [02:36] *** Guest1774 has joined #arpnetworks [02:36] we have plans to run on embedded devices [02:36] cool [02:36] it's just a little c program [02:36] sweet [02:36] and you can tack on rabbitmq if you want proper persistance (but requires 50mb of memory) [02:36] is it ready for testing now? [02:36] on the server or client? [02:37] the server is closed source at the moment (because of funding requirements / commercialisation potential) [02:37] but the client app is pretty open [02:37] oh but is it available for other people to run? [02:37] we try and run it on as much as we can [02:37] but we have a few trials soon on a few NZ ISPs [02:37] i suppose it prob not then [02:37] to trial monitoring a large residential ISP core [02:38] interesting [02:38] mercutio: http://wand.net.nz/amp/ [02:38] this is the link we gave out at NZNOG [02:38] for large ISPs we're giving them 1U dell servers to run as monitors [02:39] for smaller ISPs we're looking to give out embedded boxes to handle the monitoring [02:39] and we should have all the performance issues fixed up by the end of march [02:39] so it'll stick all of the isp's in a list? [02:39] we have a 4 year MBIE grant to develop the software to the stage of being able to monitor NZ's internet [02:39] rather than individuals being able to monitor different sites? [02:40] we're working towards ISPs being able to pick and choose what the monitor [02:40] tbh, nz internet seems better than overseas internet for domestic traffic [02:40] yeah, it's really not too bad [02:40] but i mean one user can't have 5 different sites that all monitor each other [02:40] though we find some pretty broken stuff now and then [02:41] other than silly people like you rate limiting to 10 megabit :/ [02:41] you mean like routing via australia? [02:41] So, the most recent funny one [02:41] i suppose there's less infrastructure to go wrong in nz [02:41] REANNZ <--> Callplus didn't work [02:41] that should be able to go over ape [02:41] their routers both said they should talk to each other over APE [02:41] in both directions [02:42] but on the APE layer 2, they couldn't talk [02:42] what [02:42] why not [02:42] so it was a blackhole between them [02:42] *** Guest55282 is now known as easymac [02:42] *** Guest1774 has quit IRC (Read error: Connection reset by peer) [02:42] *** twobitha- has quit IRC (Read error: Connection reset by peer) [02:42] we logged into reannz's APE router [02:42] *** twobithacker has joined #arpnetworks [02:42] bloody route servers [02:42] ran ping [callplus ape ip] [02:42] should directly connect :/ [02:42] and it started working :) [02:42] oh god [02:42] yeah.... [02:42] ubergroup had an issue like that before [02:42] except it'd start working [02:42] *** easymac is now known as Guest95370 [02:42] it hasn't fallen over again yet [02:42] they're running vpls or something from there? [02:43] probably because we run probes once every 30 seconds [02:43] and make sure the ARP table never expires :P [02:43] *** Guest1774 has joined #arpnetworks [02:43] ubergroup would drop the first few pings to them always [02:43] then start to work [02:43] also, callplus had a few ipv6 related issues we got fixed up [02:43] callplus have ipv6? [02:43] yeah [02:43] did you hear about the i217-lm issues? [02:43] nah? [02:43] apparently i217-lm have bugs [02:44] and disabling ipv6 makes them stop breaking switches [02:44] s/i217-lm/everything [02:44] apparently everything have bugs [02:44] and i217-lm are on standard hp and dell hosts these days [02:44] apparently disabling power saving stuff might fix it [02:45] http://www.edugeek.net/forums/hardware/132287-optiplex-9020-systems-spewing-ipv6-multicast-traffic-while-asleep-causing-havok.html [02:45] stuff like that [02:45] :( sigh [02:45] there's a few things floating around [02:45] we use a lot of dell at work [02:45] haven't hit that one yet [02:45] older ones probably [02:45] i217-lm is on haswell ones afaik [02:45] all our dells come with broadcom chips [02:45] but I buy proper intel server NICs for all of them [02:45] i217-v is integrated on consumer desktop boasrds with haswel onwards [02:45] broadcom isn't terible [02:46] it really is [02:46] it's a lot better than it used to be [02:46] for a desktop? [02:46] depends on the broadcom chipset I guess [02:46] my windows computer at home has onboard broadcom [02:46] we have some benchmarks of consumer broadcom vs consumer intel NICs [02:46] for iperf [02:46] i can do 972 megabit/sec [02:46] iirc [02:46] intel requires 20% less CPU usage (less spurious soft interrupts) for a given iperf run [02:46] it's something close to that if not that [02:47] hmm [02:47] mercutio: you probably didn't see: https://twitter.com/arpnetworks/status/438949834670612481 [02:47] TWITTER: We will be upgrading the IOS on our s7.lax router between 02:00-02:30 PST; expect NTT and Any2 IX to go down (Thu Feb 27 08:12:50 +0000 2014) [02:47] up_the_irons: it's only 12am here! :) [02:47] ahh [02:47] no i didn't up_the_irons [02:47] but i remember you said something about ios [02:47] yeah [02:47] and doing osmething when the first issue came up [02:47] happy to say s7.lax is now running a shiny new IOS [02:47] is it going to crash less now? :) [02:48] crash less or more, hard to say :) [02:48] need more data points [02:48] gizmoguy: 20% less cpu usage on iperf? [02:48] i give it 50 / 50 chance. this will rule out IOS issues, but there is still a possibility of bad hardware [02:48] 20% cpu usage on iperf for gigabit is pretty high [02:48] that's what my core2duo does :/ [02:48] mercutio: I believe it is because intel do a lot better with coalescing (http://www.intel.com/support/network/adapter/pro100/sb/CS-032546.htm) [02:48] i actually ordered a new sup for s7 today [02:48] gizmoguy: i dunno they both suck with udp traffic :/ [02:48] if everything ends up being OK, then it'll just be a spare [02:49] otherwise, it'll replace the current one [02:49] but yeah it could be [02:49] i went from 40% to 20% cpu usage from i7-3770 to i7-4770 for infiniband [02:49] well ip over infiniband [02:49] at over 10 gigabit [02:49] yeah [02:49] i was surprised at the difference [02:49] it is quite amazing [02:49] how much poor interrupt handling can screw you [02:49] but it's the same card in both machines [02:50] just better interrupt sharing? [02:50] i dunno, the intel coalescing is better though [02:50] yeh these infiniband careds are terrible for that [02:50] i can generate 80,000 interrupts a sec iirc [02:50] but at high interrupt loads different cpus/cache etc can mkae more diff [02:51] we tend to tune coalescing / MSI-X / irqbalance on any machine we care about doing throughput tests on [02:51] and tcp offload stuff is way better than udp offload stuff [02:51] i tend to disable irqbalance [02:51] what tuning do you do? [02:51] yeah it's all about sending irqs to the right CPU cores [02:51] my notes are at work [02:51] i was trying to figure out if there was anyway i could reduce cpu for infiniband [02:52] rather - the student I make tune all my machines is at work :) [02:52] yeah single cpu here [02:52] so it shouldn't matter afaik [02:52] so all the pci-e lanes on the same cpu [02:52] but yeah, i think i was going somewhere [02:52] but i can't remember where [02:53] gigabit isn't too complicated [02:53] it's 10 gigabit+ where things get complicated [02:53] anyway. i should sleep. got a meeting in 9 hours from now and I need my beauty sleep :) [02:53] heh [02:53] people who schedule meetings for 9am friday should be shot [02:53] what's that embedded monitoring device btw? [02:53] heh [02:53] we're trialing a few [02:54] i should stick app in :/ [02:54] I have one of these on my desk - http://www.freescale.com/webapp/sps/site/taxonomy.jsp?code=IMX6X_SERIES [02:54] but it'll be small [02:54] and we've also ordered one of those new intel embedded thingys [02:54] but it hasn't arrived yet [02:54] ahh ok [02:55] this is a better link for the freescale thing - http://www.element14.com/community/community/knode/single-board_computers/sabrelite [02:55] i dunno why can't just do a virtual machine [02:55] we're working on it [02:55] just have a few timing related issues to solve first [02:55] ahh ok [02:55] time drift is a big problem for us [02:55] and time drift occurs rather frequently in vm environments [02:55] i see [02:55] well if it's small it fine anyway [02:56] so, we're just building a list of common "correct time settings" for all the popular VM environments [02:56] to serve as a minimum set of requirements to host our VM image [02:56] i've only seen clock drift on vmware [02:56] and parallels [02:57] oh that the windows one [02:57] macosx :/ [02:57] we had some people running it [02:57] it was soo bad [02:57] haha [02:57] virtualbox! [02:57] i'll let you in on a secret (lambda is already AMP running on a VM) [02:57] ok [02:57] because I don't care about accurate timing myself [02:57] haha i guessed it would be [02:57] but the funders do [02:58] but unfortunately working for an academic instute, we get in trouble for not considering the 'science' aspect :P [02:58] i see [02:58] smokeping has way higher latency with ping to localhost on xen [02:58] so we just have to do some timing verification to prove it's not bad [02:58] then we can deploy it [02:58] it jumps from like 10 usec to 50 usec [02:59] yeah, generally if you use virtualised clock drives you're fine [02:59] not that 50 usec is a long time [02:59] but it's about as much as most ethernet coalescing is for [02:59] like KVMs pvclock - https://rwmj.wordpress.com/2010/10/15/kvm-pvclock/ [03:00] ahh ok [03:00] xen in pv mode is probably fine too? [03:00] i'll admit i've not looked at xen [03:00] i'd rather real hardware [03:01] with coalescing disabled [03:01] most embedded hardware doesn't even support coalescing though [03:01] http://wiki.xen.org/wiki/Xen_FAQ_DomU#How_can_i_synchronize_a_dom0_clock.3F [03:01] maybe? ^ [03:01] "I have problem with domU clock. It lose 30 minutes each day. How can i synchronize it with dom0 clock? " [03:01] yeah, this is why we want to run our tests... [03:01] maye it does have issues heh [03:01] some VM software is terrible at keeping time [03:01] i use ntp anyway [03:02] yeah NTP mostly insulates you [03:02] but i only care for second accuracy [03:02] for logs [03:02] mmm [03:02] ok [03:02]