[00:08] is maint window now [00:09] i assume so cos someone asked if it was over ipmi :) [00:23] https://twitter.com/arpnetworks/status/476982306163286017 was 22 minutes ago, so presumably the upgrade has bee going for the last 17 [00:23] TWITTER: Maintenance window on host kvr15 will begin in about 5 minutes (Thu Jun 12 07:00:18 +0000 2014) [00:27] Broadcast message from root (Thu Jun 12 03:05:31 2014): [00:27] The system is going down for system halt NOW! [00:30] :D [00:30] Because it doesn't make business sense. [00:31] They do have 24 hour drive throughs where it does make sense. [00:32] When I was at uni, there was a Starbucks open 'til 11pm [00:33] (the library was open 24hr, and the city scene were very active late into the night) [00:46] *** jm|laptop has joined #arpnetworks [00:46] did it pop? [01:05] there it is! [01:07] jm|laptop: Oh. Hello :) [01:07] Oh. Hi. :) [01:07] https://twitter.com/arpnetworks/status/476982306163286017 was it that? [01:08] TWITTER: Maintenance window on host kvr15 will begin in about 5 minutes (Thu Jun 12 07:00:18 +0000 2014) [01:08] Thu 12 Jun 09:08:07 BST 2014 [01:08] oh it seems backup but going slow [01:08] well taking ages to login [01:09] cuz all VMs doing fsck [01:09] hmm fresh login 2.89 load average [01:09] even with acpi shutdown? [01:09] oh uptime is only 5 minutes [01:09] ok :) [01:09] i thought it took about half an hour [01:10] so yeah, the fsck hasn't been done in x days thing [01:31] weee [01:31] I had done something wrong on that vps. Everything came up again after startup. :D [04:19] hmm i need to change to virtio now [04:20] well don't need but should [05:17] are there any known issues with network connectivity/routing? One of my CDN pops in Sydney doesn't seem able to reach my server [05:36] Can anyone else try to do a route from arp to 23.235.41.22? [05:46] *** z310 has quit IRC (*.net *.split) [06:01] *** jcv has quit IRC (Quit: leaving) [06:01] *** jcv has joined #arpnetworks [06:18] jbergstroem: http://pastebin.com/fjx9GiWC [06:21] plett: a while ago, seems to have fixed itself now: https://gist.github.com/jbergstroem/fd23d8ca83c9462ec807 [06:21] Gist: "https://gist.github.com/fd23d8ca83c9462ec807" [06:22] Cool. Glad it's working again then [06:22] just after i rerouted shit at my cdn's place :/ [06:26] *** z310 has joined #arpnetworks [09:59] Came across this long-running mtr that spans the Level3 turn-up, thought it might be interesting to some https://dl.dropboxusercontent.com/u/3167967/screenshot_2014-06-12_09-58-07.png [10:51] *** mus1cb0x has joined #arpnetworks [10:51] since 14 was upgraded this week, i've been getting noticably increased 'broken pipe' to it [10:51] i know it isn't my inet link because i also have a ssh session to a different arp server and it remains up [10:52] before this, my connection to 14 was rock solid. anyone else on 14 notice degradation in QoS? [10:52] mus1cb0x: which OS? [10:52] freebsd 9.2 [10:52] did you enable virtio? [10:53] where would that have been done? [10:53] in rc.conf or something? [10:53] i'm running stock -release kernel/base [11:02] mus1cb0x: have you made any changes to configuration since the maintenance event? [11:02] none [11:02] huh, weird [11:02] see anything in logs? [11:03] up_the_irons: any other reports of this? [11:03] which logs would you suggest i look at? [11:03] you could just try grep-ing the logs folder for your interface name [11:03] maybe look for watchdog as another item [11:06] do you know how i could grep all files in the current dir for str ? [11:07] em0 is the if [11:10] grep -rn em0 . [11:10] (the key here is -r) [11:13] ty brycec [11:14] http://pastebin.com/85d7sCwe is everything that matches em0 in /var/log [11:16] i don't know what "52:54:00:27:22:11" is, but it appears before the maintenance window so i doubt it's an issue [11:18] seems like there's a lot of promiscuous mode being enabled then disabled, not sure if that could be cleaned up [11:18] but again, appeared before maint. window also [11:22] mus1cb0x: You mean, em0: Ethernet address: 52:54:00:27:22:11 [11:22] It's the MAC [11:22] As for "promiscuous mode" you were most likely running tcpdump [12:02] *** httpduck1 has quit IRC (*.net *.split) [12:03] *** httpducks has joined #arpnetworks [12:07] brycec: that's really cool [12:07] the mtr [12:07] I see three paths? [12:08] ntt, level3, and any2 [12:12] ah [13:37] paste has been ermoved [13:37] i see none :) [13:42] https://dl.dropboxusercontent.com/u/3167967/screenshot_2014-06-12_09-58-07.png [13:42] mercutio ^ [13:46] *** tabthorpe has quit IRC (Ping timeout: 252 seconds) [13:46] *** tabthorpe has joined #arpnetworks [13:47] *** avj has quit IRC (Ping timeout: 264 seconds) [13:47] *** raptelan has quit IRC (Ping timeout: 264 seconds) [13:47] *** wallshot1 has joined #arpnetworks [13:47] *** CaZe has quit IRC (Ping timeout: 252 seconds) [13:47] *** mnathani has quit IRC (Ping timeout: 252 seconds) [13:47] *** jcv has quit IRC (Ping timeout: 264 seconds) [13:47] *** brycec has quit IRC (Ping timeout: 264 seconds) [13:47] *** acf_ has quit IRC (Ping timeout: 264 seconds) [13:47] *** eryc has quit IRC (Ping timeout: 264 seconds) [13:47] i'm guessing all those timeouts confirm what i was wondering [13:47] *** jlgaddis has joined #arpnetworks [13:48] network or vps host outage perhaps? i should try to vnc in and see what it says from the inside [13:48] internet b0rked? [13:48] wallshot1: Agreed. Me too. [13:48] yeah i had two vps' drop in the last three minutes [13:48] *** raptelan has joined #arpnetworks [13:49] *** acf_ has joined #arpnetworks [13:49] *** jcv has joined #arpnetworks [13:49] *** CaZe has joined #arpnetworks [13:50] I'm still connected to my VPS just fine [13:50] And some of those VPS->IRC connections are still good, but not VPS->Freenode(whatever server I'm on) [13:50] still connected to mine too [13:50] I can't ssh into my node. [13:50] ssh timeouts turned into ssh slow-but-successful [13:50] I'm seeing a bunch of OpenVPN connections flap thoguh [13:50] hm [13:50] hmm that was strange [13:51] Oh, there it goes. [13:51] So I'm thinking there's aroute having issues. [13:51] and pidgin is reconnected to my jabber server again, yay [13:51] and everything's back up lol [13:51] *** eryc has joined #arpnetworks [13:51] *** eryc has quit IRC (Changing host) [13:51] *** eryc has joined #arpnetworks [13:51] weird. didn't notice anything here at all. [13:52] mtr -4 asimov.freenode.net [13:52] │13:51:52 *status | Disconnected from IRC (Connection reset by peer). Reconnecting... [13:52] *** mnathani has joined #arpnetworks [13:52] oh it means the route changed [13:52] the two hops after s7.lax.arpnetworks.com are ??? [13:52] it may have b een withdrawn and readvertised [13:52] The route can change without destroying the connection... Unless the route changed to something unroutable. [13:53] well it's strange for 10.10.10.6 to appear in it [13:53] what are you tracing to mercutio? [13:53] acf: nothing, i'm lokoing at your mtr :) [13:53] oh I don't see 10.10.10.6... [13:53] yes you do [13:53] it goes via Level3 [13:53] hop 3 second number [13:54] err 2nd line [13:54] http://paste.unixcube.org/k/7b6ab9 [13:54] the screenshot [13:54] that kinud of thing doesn't show up in short mtr's normally [13:54] oh that [13:54] only when you leave mtr running in the backgruond [13:54] and catch a route dropping [13:54] or changing path [13:54] yes, that's brycec's trace from the Level3 switch over [13:54] it's cool [13:55] but when hop 10 says los angeles, it suggests it probably dropped [13:55] and looped a little [13:55] Yeah that's an old, long-running mtr [13:55] *** brycec has joined #arpnetworks [13:55] Oh look at me, I'm back [13:55] oh youre bryce [13:55] yeah [13:55] haha [13:55] am [13:55] am [13:55] i was wondering who it was [13:55] same here lol [13:55] i'm sure i'd seen the name before [13:56] This is a connection I leave up from another host, so... lurking [13:56] heh [13:56] freenode has mostly been stable for me recently [13:56] [mercutio] idle: 00 hours 00 minutes 03 seconds, signon at: Sat, 03 May 2014 06:31:08 [13:57] 14:57:02 freenode -- | [m0unds] idle: 00 hours 01 minute 25 seconds, signon at: Mon, 21 Apr 2014 17:26:55 [13:57] there's been a netsplit in that time [13:57] it used to be pretty unstable [13:58] *** mus1cb0x has left "WeeChat 0.4.2" [13:58] the vast majority of people in this channel seem to idle [13:58] yep [14:00] * dangel pets mosh. [14:27] i guess apd cornered an armed robbery suspect in our employee parking lot [14:28] haha [14:35] apd? [14:36] [something] police department, I would guess [14:36] yep [14:36] albuquerque [14:37] well - that explains it :) [14:56] hmm i left a mtr to amazon running [14:56] after it goes into amazon's network it hops around a lot [14:59] i'd expect that at any good sized org as clusters are put into and out of rotation for maintenance [15:00] with entire data centers going in and out of rotation at any given time [15:00] Even for simple load balancing [15:04] wall: sounds complicated [15:04] hop 14 has 8 differnet ip's [15:04] but they all start with 205.251 [15:05] it's 21 hops [15:05] and 11 hops to amazon [15:05] so 10 hops within their network [15:06] i probably should stop this mtr [15:06] id on't know where it's too :/ [15:06] that is indeed a few hops in their network before hitting the endpoint [15:06] ec2-54-214-15-46.us-west-2.compute.amazonaws.com [15:06] someone gave me that host to try us west? [15:07] http://54.214.15.46/videos/Yelle%20-%20Comme%20Un%20Enfant%20%28Freaks%20Remix%29.webm [15:07] what's that heh [15:07] watch it [15:07] the whole thing [15:07] ahh so it you acf? [15:07] yep [15:07] why are you on ec2 [15:08] it has super good transit [15:08] i don't know if i have a webm player [15:08] https://www.youtube.com/watch?v=hAiu04sUDMg [15:08] YouTube Music: "Yelle - Comme Un Enfant (Freaks Remix)" by Nathan Barnatt (5m 41s), 141,973 views, 4,515 likes and 29 dislikes. Uploaded 2014-01-03T21:48:54.000Z. [15:08] i dunno i have 2.7% packet loss to it :/ [15:08] and 0.2% to paix amazon [15:08] arp never has that high loss :/ [15:08] hmm, I don't see that [15:08] neither do i [15:09] what do you do when the mtr goes off the terminal? [15:09] too long [15:09] http://pastebin.com/epc89FBq [15:09] i resize the window [15:09] i did alt enter at first [15:10] my window won't go any bigger lol [15:10] then i realised it was way too wide to cut and paste [15:10] i'm using 1440p [15:10] ah, ok [15:10] 50pkts no loss [15:10] and it's over half my vertical screen size [15:10] so at 720p i'm sure you'd struggle. [15:10] i have 58,000 packets [15:10] :) [15:15] *** wallshot1 has quit IRC (Read error: No route to host) [15:15] No route to host? how does that happen? [15:16] depends [15:16] *** wallshot1 has joined #arpnetworks [16:08] Level3 seems kind of jittery since it dropped out earlier today [17:21] jbergstroem: plett : yeah i got some alerts of peering trouble earlier today [17:31] m0unds: whenever you mention apd it always reminds me of breaking bad, lol [17:31] up_the_irons: bahaha [17:33] And it probably will for years to come [17:33] One of these days I'm going to finish watching BB [17:33] yeah, it's either BB or USDOJ investigation [17:33] alternates between those two things, since all people know about ABQ is that breaking bad was shot here and that APD is under investigation for use of force [17:34] I was unaware of the latter, actually [17:34] So congrats on that [17:35] haha, i'm surprised. there was a shooting event where APD killed a batshit insane homeless guy and it made national news [17:35] and stupid anonymous targeted cabq.gov and apd's stuff [17:35] i was unaware of the latter too [17:36] well geez, i'm disappointed now [17:36] haha [17:36] lots of shootings by apd in the last 10 yrs [17:37] I've heard of DOJ investigating similar charges for LAPD and SPD (Seattle), but ABQ slipped under my very, very high radar. [17:37] haha, SPD was turned into a metro PD under recommendation by usdoj [17:37] o.o [17:37] or am i thinking of LVPD [17:37] or something [17:37] Ok. I have no idea what that "means [17:37] federally administered police dept [17:38] something like that [17:38] oh my [17:38] The town I live in has 6 cops :) [17:38] The PD is also administered (or governed, or something like that) by the county sheriff [17:39] yeah, lots of small agencies work that way [17:42] actually, seems like LV metro pd is that way too [17:42] and they're huge (officer count-wise) [17:42] up_the_irons: ok, thanks for letting me know [17:43] it's the jurisdiction radius, basically. counties are protected by sheriff's departments. within a county, the cities can have their own municipal police department (and many do, e.g. LAPD, BPD (Burbank), GPD (Glendale), etc... around my area) [17:43] cities that don't have their own PD get automatically covered by the sheriff [17:43] and lastly, some cities simply "outsource" their PD to the sheriff (e.g. West Hollywood) [17:44] those are called "contracted" cities [17:44] and is why you see sheriff cars in an otherwise LAPD controlled area [17:45] but i still don't know what "federally administered police dept" would be [17:45] either [17:45] up_the_irons: btw, next time i have similar issues should i just hit support@ ? [17:45] jbergstroem: yup [17:45] up_the_irons: done [17:45] up_the_irons: usdoj handles administrative duty and training stuff for a police agency [17:45] :) [17:45] m0unds: roger [17:48] *** siignal has joined #arpnetworks [17:49] haha, add qwest/centurylink to the list of ISPs who run their NTT peering way too hot [17:49] ugh [17:50] *** siignal has left [17:52] Verizon, AT&T, Comcast, CenturyLink/Qwest, TWC, Cox, Charter [17:52] centurylink announces bogons [17:52] yay [17:54] so does Level3 [17:54] according to bgp.he.net [17:54] and Verizon [17:55] even the Verizon <-> Comcast peering is hot [17:56] it's for the best - i don't want my house to burn down [19:31] acf: i kind of wish there was more of an international monitoring/graphing/record of this stuff [19:32] broken down as much as possible [19:32] yeah. it's kind of surprising that one doesn't exist actually [19:32] considering the magnitude of some of these problems [19:32] probably because it's hard [19:32] i mean you need 10,000+ peopel testing to really get good results. [19:33] I wonder if nanog ring could help any [19:33] you have access to test points from a bunch of different nsps/etc... [19:33] still isn't it like 20 locations? [19:33] you kind of want more ilke 50 locatiosn [19:33] oh it' smore complicated than that [19:34] like if you can select different providers in 50 locations.. [19:34] https://ring.nlnog.net/ [19:34] eggnog ring [19:34] https://ring.nlnog.net/participants/ [19:34] i got 50 from 7 countries and 6 places in each country + a bit [19:34] i don't relaly know good numbers [19:34] but i imagine some of them are near each other [19:35] nlnog ring is probably the best combined network monitoring project outside of perfsonar [19:35] our software used to be used to monitor the nlnog ring, not sure if it still is [19:35] so like in the US, you want to test at least dallas, los angeles, san jose, virginia, new york, chicago, phoenix, denver, miami, seattle, .. [19:35] i'm missing a few important network locations right? [19:36] i still like ripe atlas, despite the black box nature of it [19:36] and test the different providers in each of those regions [19:36] so nlnog is just like a big shell access exchange? [19:37] https://stats.es.net/perfSONAR/directorySearch.html [19:37] can you do throughput tesitng on nlonog? [19:37] nlnog ring [19:37] I suspect so [19:37] i imagine it's mostly about network health [19:37] what we did for them [19:38] oh cool [19:38] https://ring.nlnog.net/toolbox/ [19:38] is create a full mesh of all participants, and do regular traceroute/icmp [19:38] some people have a few, but some are jst one [19:38] in the mesh [19:38] and record networks traversed and jitter and so on? [19:39] naw they turned off our software [19:39] https://ring.nlnog.net/news/2012/10/root-cause-analysis-using-amp/ [19:39] damn [19:39] here's an old blog post [19:40] is it open source? [19:40] not at the moment [19:40] http://wand.net.nz/amp/ [19:41] we're in the middle of a 4 year project to rewrite the whole thing [19:41] i think the best way to do testing is to involve enough people that local issues on less well connected sitse don't matter [19:41] rather than having "well connected" test sites. [19:42] gizmoguy: do you ever look at geekzone forums? they were talking about congesting happening on 2.4 gigabit ONT's that go to 24 houses or such. [19:42] but a lot of people don't seem to really understand congestion well. [19:42] and likely places rather than unlikely places for it to happen [19:43] I try and stay well away from that place [19:43] haha good idea [19:43] i still think congestino is not widely understood [19:43] do you have a link to that forum post? [19:43] I have a few people I would like to share that with :) [19:43] it's on the 1 gigabit fibre thread [19:43] like the 24 houses are using up 2.4 gigabit of bandwidth? [19:44] http://www.geekzone.co.nz/forums.asp?forumid=49&topicid=147171&page_no=6#1064686 [19:44] somewhere in there [19:44] acf: gpon is 2.4/1.2 gigabit [19:44] 2.4 down 1.2 up [19:44] adn gpon supports oversubscription [19:44] so you can have 24 customers with gigabit connections. [19:44] the question is, is it likely to congest? [19:45] google fibre is the same diff [19:45] shoudl be calling it "google fiber" because it's only available in the US? [19:45] it's also probably less than 24 people on a node in practice [19:45] the iprovo thing is such a boondoggle [19:46] iprovo? [19:46] paid for by taxpayer money and the city sold it to google for $1 [19:46] oh [19:46] ir ead about it [19:46] https://transmission.xmission.com/2013/04/18/the-1-fiber-optic-network [19:46] but google are paying $30 million on construction loans [19:46] which is small potatoes [19:46] i think it sonuded reasonable enough to me [19:47] did you see the thing about them paying $5/month in taxes wehterhet or not they take it up? [19:47] i can't tyo. s/wehterhet/whether/ [19:47] hahaha [19:47] i can't type my mistypings either can it. [19:47] -t [19:47] argh [19:47] urgh myfullflavour [19:48] gah, i'm dropping keystrokes like crazy [19:48] gizmoguy: they're not around here, so dont' know much about them. [19:48] I designed and built their first network [19:48] they didn't pay me [19:48] i suppose you can see i posted a bit heh [19:48] eek [19:48] well they paid 1% of the bill or something [19:49] well apparently they're upgrading their core network now [19:49] lots ofp eople are upgrading core networks atm] [19:49] but yeah, i was trying to determine if any sites could push anythhing close to gigabit [19:50] the fastest speeds i've had are to you ai think gizmo :) [19:50] :) [19:51] but i imagine that server has less load than citylink's ftp mirror etc [19:51] and it's rare to have much nz content of any good size [19:51] http://ps02.reannz.co.nz/iso/ [19:51] see how fast you can pull that down at [19:51] it's not loading [19:51] same [19:52] it's reannz only isn't it? [19:52] so the question is [19:52] when everyone has google fibre [19:52] then what happens to our NTT congestion? [19:52] mercutio: oh yes right. sorry those machines aren't on the internet vrf [19:53] google use ntt i think? [19:53] hmm idk [19:53] its any2 from arp [19:53] acf: it's ape from me (nz peering exchange) [19:54] it's good they're well peered at least [19:54] they're peered directly to amazon too [19:54] and HE [19:54] via telecom? [19:54] err global gateway [19:54] ? [19:54] what ip [19:54] is it like 210.55 or something [19:55] http://paste.unixcube.org/k/2cad50 [19:55] top HE, bottom amazon [19:55] not sure what you're looking for [19:55] that's google [19:55] isnt' it? [19:56] teah [19:56] yeah [19:56] oh right [19:56] i thought you meant gizmo's server for some reason [19:56] sorry i'm slow [19:56] but that's google's web site [19:56] their fibre is different [19:56] i tracked down an ip ages ago [19:56] right [19:56] but i lost it [19:56] they don't have peering in kansas [19:56] at least not well peered [19:57] and their traffic goes via dallas/chicago [19:57] hmm I can see why that would be bad [19:57] can you buy transit from google fiber? [19:57] well kansas sucks for peering in general [19:57] it's one of the annoying places :/ [19:57] comcast and verizon suck for peering there too. [19:58] yeah, I haven't much experience with kansas [19:58] http://bgp.he.net/AS16591 [19:58] so what it looked like google were doing, is doing their own backhaul to places with peering [19:58] then buyign bandwidth adn peering there [19:58] that would make sense [19:58] i've got a friend in wichita [19:58] which is close enough [19:58] what kind of backhaul do they have? [19:58] he can download at 3 megabytes/sec from nz at least. [19:58] if everybody has gigabit... [19:59] on comcast i think that was [19:59]