is maint window now i assume so cos someone asked if it was over ipmi :) https://twitter.com/arpnetworks/status/476982306163286017 was 22 minutes ago, so presumably the upgrade has bee going for the last 17 TWITTER: Maintenance window on host kvr15 will begin in about 5 minutes (Thu Jun 12 07:00:18 +0000 2014) Broadcast message from root (Thu Jun 12 03:05:31 2014): The system is going down for system halt NOW! :D Because it doesn't make business sense. They do have 24 hour drive throughs where it does make sense. When I was at uni, there was a Starbucks open 'til 11pm (the library was open 24hr, and the city scene were very active late into the night) did it pop? there it is! jm|laptop: Oh. Hello :) Oh. Hi. :) https://twitter.com/arpnetworks/status/476982306163286017 was it that? TWITTER: Maintenance window on host kvr15 will begin in about 5 minutes (Thu Jun 12 07:00:18 +0000 2014) Thu 12 Jun 09:08:07 BST 2014 oh it seems backup but going slow well taking ages to login cuz all VMs doing fsck hmm fresh login 2.89 load average even with acpi shutdown? oh uptime is only 5 minutes ok :) i thought it took about half an hour so yeah, the fsck hasn't been done in x days thing weee I had done something wrong on that vps. Everything came up again after startup. :D hmm i need to change to virtio now well don't need but should are there any known issues with network connectivity/routing? One of my CDN pops in Sydney doesn't seem able to reach my server Can anyone else try to do a route from arp to 23.235.41.22? jbergstroem: http://pastebin.com/fjx9GiWC plett: a while ago, seems to have fixed itself now: https://gist.github.com/jbergstroem/fd23d8ca83c9462ec807 Gist: "https://gist.github.com/fd23d8ca83c9462ec807" Cool. Glad it's working again then just after i rerouted shit at my cdn's place :/ Came across this long-running mtr that spans the Level3 turn-up, thought it might be interesting to some https://dl.dropboxusercontent.com/u/3167967/screenshot_2014-06-12_09-58-07.png since 14 was upgraded this week, i've been getting noticably increased 'broken pipe' to it i know it isn't my inet link because i also have a ssh session to a different arp server and it remains up before this, my connection to 14 was rock solid. anyone else on 14 notice degradation in QoS? mus1cb0x: which OS? freebsd 9.2 did you enable virtio? where would that have been done? in rc.conf or something? i'm running stock -release kernel/base mus1cb0x: have you made any changes to configuration since the maintenance event? none huh, weird see anything in logs? up_the_irons: any other reports of this? which logs would you suggest i look at? you could just try grep-ing the logs folder for your interface name maybe look for watchdog as another item do you know how i could grep all files in the current dir for str ? em0 is the if grep -rn em0 . (the key here is -r) ty brycec http://pastebin.com/85d7sCwe is everything that matches em0 in /var/log i don't know what "52:54:00:27:22:11" is, but it appears before the maintenance window so i doubt it's an issue seems like there's a lot of promiscuous mode being enabled then disabled, not sure if that could be cleaned up but again, appeared before maint. window also mus1cb0x: You mean, em0: Ethernet address: 52:54:00:27:22:11 It's the MAC As for "promiscuous mode" you were most likely running tcpdump brycec: that's really cool the mtr I see three paths? ntt, level3, and any2 ah paste has been ermoved i see none :) https://dl.dropboxusercontent.com/u/3167967/screenshot_2014-06-12_09-58-07.png mercutio ^ i'm guessing all those timeouts confirm what i was wondering network or vps host outage perhaps? i should try to vnc in and see what it says from the inside internet b0rked? wallshot1: Agreed. Me too. yeah i had two vps' drop in the last three minutes I'm still connected to my VPS just fine And some of those VPS->IRC connections are still good, but not VPS->Freenode(whatever server I'm on) still connected to mine too I can't ssh into my node. ssh timeouts turned into ssh slow-but-successful I'm seeing a bunch of OpenVPN connections flap thoguh hm hmm that was strange Oh, there it goes. So I'm thinking there's aroute having issues. and pidgin is reconnected to my jabber server again, yay and everything's back up lol weird. didn't notice anything here at all. mtr -4 asimov.freenode.net │13:51:52 *status | Disconnected from IRC (Connection reset by peer). Reconnecting... oh it means the route changed the two hops after s7.lax.arpnetworks.com are ??? it may have b een withdrawn and readvertised The route can change without destroying the connection... Unless the route changed to something unroutable. well it's strange for 10.10.10.6 to appear in it what are you tracing to mercutio? acf: nothing, i'm lokoing at your mtr :) oh I don't see 10.10.10.6... yes you do it goes via Level3 hop 3 second number err 2nd line http://paste.unixcube.org/k/7b6ab9 the screenshot that kinud of thing doesn't show up in short mtr's normally oh that only when you leave mtr running in the backgruond and catch a route dropping or changing path yes, that's brycec's trace from the Level3 switch over it's cool but when hop 10 says los angeles, it suggests it probably dropped and looped a little Yeah that's an old, long-running mtr Oh look at me, I'm back oh youre bryce yeah haha am am i was wondering who it was same here lol i'm sure i'd seen the name before This is a connection I leave up from another host, so... lurking heh freenode has mostly been stable for me recently [mercutio] idle: 00 hours 00 minutes 03 seconds, signon at: Sat, 03 May 2014 06:31:08 14:57:02 freenode -- | [m0unds] idle: 00 hours 01 minute 25 seconds, signon at: Mon, 21 Apr 2014 17:26:55 there's been a netsplit in that time it used to be pretty unstable the vast majority of people in this channel seem to idle yep i guess apd cornered an armed robbery suspect in our employee parking lot haha apd? [something] police department, I would guess yep albuquerque well - that explains it :) hmm i left a mtr to amazon running after it goes into amazon's network it hops around a lot i'd expect that at any good sized org as clusters are put into and out of rotation for maintenance with entire data centers going in and out of rotation at any given time Even for simple load balancing wall: sounds complicated hop 14 has 8 differnet ip's but they all start with 205.251 it's 21 hops and 11 hops to amazon so 10 hops within their network i probably should stop this mtr id on't know where it's too :/ that is indeed a few hops in their network before hitting the endpoint ec2-54-214-15-46.us-west-2.compute.amazonaws.com someone gave me that host to try us west? http://54.214.15.46/videos/Yelle%20-%20Comme%20Un%20Enfant%20%28Freaks%20Remix%29.webm what's that heh watch it the whole thing ahh so it you acf? yep why are you on ec2 it has super good transit i don't know if i have a webm player https://www.youtube.com/watch?v=hAiu04sUDMg YouTube Music: "Yelle - Comme Un Enfant (Freaks Remix)" by Nathan Barnatt (5m 41s), 141,973 views, 4,515 likes and 29 dislikes. Uploaded 2014-01-03T21:48:54.000Z. i dunno i have 2.7% packet loss to it :/ and 0.2% to paix amazon arp never has that high loss :/ hmm, I don't see that neither do i what do you do when the mtr goes off the terminal? too long http://pastebin.com/epc89FBq i resize the window i did alt enter at first my window won't go any bigger lol then i realised it was way too wide to cut and paste i'm using 1440p ah, ok 50pkts no loss and it's over half my vertical screen size so at 720p i'm sure you'd struggle. i have 58,000 packets :) No route to host? how does that happen? depends Level3 seems kind of jittery since it dropped out earlier today jbergstroem: plett : yeah i got some alerts of peering trouble earlier today m0unds: whenever you mention apd it always reminds me of breaking bad, lol up_the_irons: bahaha And it probably will for years to come One of these days I'm going to finish watching BB yeah, it's either BB or USDOJ investigation alternates between those two things, since all people know about ABQ is that breaking bad was shot here and that APD is under investigation for use of force I was unaware of the latter, actually So congrats on that haha, i'm surprised. there was a shooting event where APD killed a batshit insane homeless guy and it made national news and stupid anonymous targeted cabq.gov and apd's stuff i was unaware of the latter too well geez, i'm disappointed now haha lots of shootings by apd in the last 10 yrs I've heard of DOJ investigating similar charges for LAPD and SPD (Seattle), but ABQ slipped under my very, very high radar. haha, SPD was turned into a metro PD under recommendation by usdoj o.o or am i thinking of LVPD or something Ok. I have no idea what that "means federally administered police dept something like that oh my The town I live in has 6 cops :) The PD is also administered (or governed, or something like that) by the county sheriff yeah, lots of small agencies work that way actually, seems like LV metro pd is that way too and they're huge (officer count-wise) up_the_irons: ok, thanks for letting me know it's the jurisdiction radius, basically. counties are protected by sheriff's departments. within a county, the cities can have their own municipal police department (and many do, e.g. LAPD, BPD (Burbank), GPD (Glendale), etc... around my area) cities that don't have their own PD get automatically covered by the sheriff and lastly, some cities simply "outsource" their PD to the sheriff (e.g. West Hollywood) those are called "contracted" cities and is why you see sheriff cars in an otherwise LAPD controlled area but i still don't know what "federally administered police dept" would be either up_the_irons: btw, next time i have similar issues should i just hit support@ ? jbergstroem: yup up_the_irons: done up_the_irons: usdoj handles administrative duty and training stuff for a police agency :) m0unds: roger haha, add qwest/centurylink to the list of ISPs who run their NTT peering way too hot ugh Verizon, AT&T, Comcast, CenturyLink/Qwest, TWC, Cox, Charter centurylink announces bogons yay so does Level3 according to bgp.he.net and Verizon even the Verizon <-> Comcast peering is hot it's for the best - i don't want my house to burn down acf: i kind of wish there was more of an international monitoring/graphing/record of this stuff broken down as much as possible yeah. it's kind of surprising that one doesn't exist actually considering the magnitude of some of these problems probably because it's hard i mean you need 10,000+ peopel testing to really get good results. I wonder if nanog ring could help any you have access to test points from a bunch of different nsps/etc... still isn't it like 20 locations? you kind of want more ilke 50 locatiosn oh it' smore complicated than that like if you can select different providers in 50 locations.. https://ring.nlnog.net/ eggnog ring https://ring.nlnog.net/participants/ i got 50 from 7 countries and 6 places in each country + a bit i don't relaly know good numbers but i imagine some of them are near each other nlnog ring is probably the best combined network monitoring project outside of perfsonar our software used to be used to monitor the nlnog ring, not sure if it still is so like in the US, you want to test at least dallas, los angeles, san jose, virginia, new york, chicago, phoenix, denver, miami, seattle, .. i'm missing a few important network locations right? i still like ripe atlas, despite the black box nature of it and test the different providers in each of those regions so nlnog is just like a big shell access exchange? https://stats.es.net/perfSONAR/directorySearch.html can you do throughput tesitng on nlonog? nlnog ring I suspect so i imagine it's mostly about network health what we did for them oh cool https://ring.nlnog.net/toolbox/ is create a full mesh of all participants, and do regular traceroute/icmp some people have a few, but some are jst one in the mesh and record networks traversed and jitter and so on? naw they turned off our software https://ring.nlnog.net/news/2012/10/root-cause-analysis-using-amp/ damn here's an old blog post is it open source? not at the moment http://wand.net.nz/amp/ we're in the middle of a 4 year project to rewrite the whole thing i think the best way to do testing is to involve enough people that local issues on less well connected sitse don't matter rather than having "well connected" test sites. gizmoguy: do you ever look at geekzone forums? they were talking about congesting happening on 2.4 gigabit ONT's that go to 24 houses or such. but a lot of people don't seem to really understand congestion well. and likely places rather than unlikely places for it to happen I try and stay well away from that place haha good idea i still think congestino is not widely understood do you have a link to that forum post? I have a few people I would like to share that with :) it's on the 1 gigabit fibre thread like the 24 houses are using up 2.4 gigabit of bandwidth? http://www.geekzone.co.nz/forums.asp?forumid=49&topicid=147171&page_no=6#1064686 somewhere in there acf: gpon is 2.4/1.2 gigabit 2.4 down 1.2 up adn gpon supports oversubscription so you can have 24 customers with gigabit connections. the question is, is it likely to congest? google fibre is the same diff shoudl be calling it "google fiber" because it's only available in the US? it's also probably less than 24 people on a node in practice the iprovo thing is such a boondoggle iprovo? paid for by taxpayer money and the city sold it to google for $1 oh ir ead about it https://transmission.xmission.com/2013/04/18/the-1-fiber-optic-network but google are paying $30 million on construction loans which is small potatoes i think it sonuded reasonable enough to me did you see the thing about them paying $5/month in taxes wehterhet or not they take it up? i can't tyo. s/wehterhet/whether/ hahaha i can't type my mistypings either can it. -t argh urgh myfullflavour gah, i'm dropping keystrokes like crazy gizmoguy: they're not around here, so dont' know much about them. I designed and built their first network they didn't pay me i suppose you can see i posted a bit heh eek well they paid 1% of the bill or something well apparently they're upgrading their core network now lots ofp eople are upgrading core networks atm] but yeah, i was trying to determine if any sites could push anythhing close to gigabit the fastest speeds i've had are to you ai think gizmo :) :) but i imagine that server has less load than citylink's ftp mirror etc and it's rare to have much nz content of any good size http://ps02.reannz.co.nz/iso/ see how fast you can pull that down at it's not loading same it's reannz only isn't it? so the question is when everyone has google fibre then what happens to our NTT congestion? mercutio: oh yes right. sorry those machines aren't on the internet vrf google use ntt i think? hmm idk its any2 from arp acf: it's ape from me (nz peering exchange) it's good they're well peered at least they're peered directly to amazon too and HE via telecom? err global gateway ? what ip is it like 210.55 or something http://paste.unixcube.org/k/2cad50 top HE, bottom amazon not sure what you're looking for that's google isnt' it? teah yeah oh right i thought you meant gizmo's server for some reason sorry i'm slow but that's google's web site their fibre is different i tracked down an ip ages ago right but i lost it they don't have peering in kansas at least not well peered and their traffic goes via dallas/chicago hmm I can see why that would be bad can you buy transit from google fiber? well kansas sucks for peering in general it's one of the annoying places :/ comcast and verizon suck for peering there too. yeah, I haven't much experience with kansas http://bgp.he.net/AS16591 so what it looked like google were doing, is doing their own backhaul to places with peering then buyign bandwidth adn peering there that would make sense i've got a friend in wichita which is close enough what kind of backhaul do they have? he can download at 3 megabytes/sec from nz at least. if everybody has gigabit... on comcast i think that was they don't all have gbit they have svc levels like anything else acf: they've probably got 10 gigabit links which they can upgrade to 40 gigabit when they congest. you have to have a lot of users to congest dual 10 gigabit linsk. I guess so they have 2 plans and it's more prudent to get links to more places than to make the pipes fatter. do you have a file to grab from something in nz? i'm curious m0unds: there's heaps of various files around :/ mercutio: ok I can see that http://202.49.71.24:24/10m that's a 10m test file on my server or potentially http://ftp.nz.debian.org/debian/ my rationality is that once files get big you're not waiting for them to download so it's better to test with files that aren't huge my experience with even domestic linux mirrors is that they're under pretty regular load and can vary wildly 0.9s download time on that mercutio ~500KB/sec gizmoguy: that seems slow oh hangon 0.9 isn't too bad that's 10mb/sec are you connected at 100 megabit? 11.0M/s in 0.9s yup 100mbit beacuse that's basically 100 megabit i can grab it faster from home than my arp vm hah a m0unds: haha it's ntt to arp i dunno if that says anything :/ outbound is any2 comcast it's level3 yeah comcast forward path is verizon i think level3 for me yeah verizon or fwd from you? forward from me 195ms 500kb/ssec is actually painfulyl slow :/ i wonder why it's going so slow verizon/comcast peering seems a bit congested atm through lax ^ could be that, because it routes me through lax yeah it will route via la or sj to nz the only time i see sjc for stuff out west is when things are screwy http://kremvax.acfsys.net/smokeping.cgi?displaymode=n;start=2014-06-12%2017:04;end=now;target=Remote.verizonnet~comcastep because it typically does abq > den > sjc > elsewhere or sometimes, abq > den > dfw > sjc > elsewhere but yeah, nz is pretty good for doing 100 megabit speeds but i really doubt that much is connected at any faster than 1 gigabit asnd like i'm pretty sure gizmo's being slowed down by a bsaically arbitary 100 megabit limit and that if he cared enough he coudl get it fixed to gigabit like one router or switch or such haha, those debian mirrors are using a cdn nz.archive.ubuntu.com does which is what i use for testing nz speeds normally but can't give you :( cos it is unfair cdns do complicate things yeah, the node it's giving me for the debian one is in san jose that one is the same actually haha 16. ubuntu.citylink.co.nz 0.0% 13 43.5 43.4 41.5 44.6 0.8 ftp://wits.cs.waikato.ac.nz/ispdsl/2/20100106-030946-0.dsl.erf.gz i found something on gizmo's server to link :/ but it's rated limited at 10 megabit on ipv4, and needs ipv6 to go faster and i can't do v6 because my v6 tunnel is to arp, which is congested atm the pseed isn't great from arp for that too 58.5KB/s eta 2h 50m 551k/sec average at 9.6mb ^ that's via the tunnel i tried to ^C around 10mb hahaha wow 604KB/s eta 11m 55s that was direct via v4 weird i can give you a big file on non-ratelimited connection if you want gizmo: how about a 10mb file? :) for science http://wand.net.nz/boot/mirror/isos/CentOS-6.4-x86_64-LiveDVD.iso oh 10 M sure you can just download my 10mb file it's /dev/urandom though dd if=/dev/urandom of=10m bs=1024k count=10 ] 16,915,140 3.70MB/s it's bouncing around, but highest speed was 5.7MB/sec there were some dodgy vps providers serving /dev/zero content over https http://wand.net.nz/boot/mirror/isos/10m https compresses such files and makes it faster 100 10.0M 100 10.0M 0 0 71.4M 0 --:--:-- --:--:-- --:--:-- 71.9M waikato netwokring is very broken at the moment unfortunately oh it was hitting proxy oosp 100 10.0M 100 10.0M 0 0 14.9M 0 --:--:-- --:--:-- --:--:-- 14.9M i hit 2.77MB/sec on that one curl -v http://wand.net.nz/boot/mirror/isos/10m > /dev/null 0.01s user 0.03s system 5% cpu 0.748 total hmm so it's not really a lot faster than your 100 megabit connectino hahaha i'm 7081mi from hamilton bah from home it hit proxy all the time :/ or almost 11400km this is why i use port 24 :) m0unds: that's pretty close that's if there was a cable draped from NM straight to nz haha so yeah, even if i had gigabit compared to 100 megabit, it'd only be about 20% faster to a location 5 msec away oh it's 2.5 msec actually i think gizmo is right about something being broken Might be off topic, but does gmail have problem yesterday or a day before? brycec: what season/episode are you on? brycec: of breaking bad anis: gmail has lots ofp roblems affecting some users but not others and some data centres but not others so it can easily be fine for other people and broken for you http://www.wired.com/2014/06/gmail-bug-could-have-exposed-every-users-address/ Wired: "Gmail Bug Could Have Exposed Every User’s Address | Threat Level | WIRED" Many people complaining about this since yesterday. But today seem ok. Yesterday and a day before. they dont received any emails. No errors my gmail spam folder has made me aware for years of just how many countless assholes have my email address weird And here too http://www.dailymail.co.uk/sciencetech/article-2655616/Massive-flaw-revealed-revealed-Gmail-address-EVERY-user-Googles-mail-service.html?ITO=1490&ns_mchannel=rss&ns_campaign=1490 my gmail spam mail box only gets false positives it happens quite frequently No wonder many friends yesterday dont receive the emails!! oh wow mail just came through for me quickly i just emadiled myself :0 Haven't noticed any gmail issues... mnathani: Don't remember exactly (but Netflix ought to), somewhere around season 3 or 4 (I think I last watched it back in December or thereabouts. But then life got busy, and...)