arpnetworks down?? nvm what happened? it seams at&t have blocked my connection to my own sites / services uverse well and arpnetworks.com entirely those dirty sluts man!! up_the_irons: what is arpnetworks ipv4 range? i wanna make sure they unblock it for everyone with at&t service in case others are having the same problem. People could be loosing potential business if no one on at&t uverse can access any sites hosted at arp not sure it's that wide spread, but can't hurt i dont think hmm, you can find out using IRR if he's been good about his object maintenance ;) my objects are up to date it's weird, I'm on twc, i can ping everything in my cage, but SSH, http, etc.. seems to be all blocked. started about an hour ago. I'm still in irc only because I'm using weechat android from my phone thats fucked oh wow.... so it's not just at&t... same symptoms here, can ping everything. but ssh, http, https all blocked lol this at&t rep wants me to reset my modem to factory. now i feel like an ass :P up_the_irons: can I try anything for you from Germany? ;) pyvpx : how about just arpnetworks. com? wyliebayes.com , nadasound.com 2 of mine pings and loads just fine I'm coming from 37.120.2.102 overall, my bandwidth graphs for the big pipes look pretty normal so most traffic is still getting out so weird about twc and att it's only residential uverse att tho. i can access via my att 4g network no problem what are the blocks you guys are coming from that can't reach arp? i am, 99-95-174-245.lightspeed.sndgca.sbcglobal.net pyvpx: would be weird bgp would effect all services, except ICMP oh, I meant from a policy stand point localize to a router or a routing event that could point to a config change it's easier to hit someone on the head at att when you are like "yeah at 4am this shit started" they go "wait we had a thing at 4am uhhhhh" ...sometimes i would be surprised if the "tech" im talking to in chat can even feed himself successfully from att sometimes if you play the act the ip noc folks can entertain you give you ticket numbers or other depts to call ya im trying, had to DEMAND i speak to his supervisor cause he tried to pawn it off on the hosting provider hehe by pasting me out of the book answers if resetting my modem didn't fix the problem, we have exhausted all possible solutions. LOL hmm the denver router on their looking glass had a thing 42 minutes ago probably nothing everything seen through verio hrmm and since icmp works it's hard to see where it might be failing i am in San Diego CA I'm playing around with shutting down certain transit sessions, to see if any of them are the culprit. nothing successful so far it's like only tcp doesn't work. i can still connect to my VPN (udp) and dns still works fine... wtf yeah, I’m connected over VPN but TCP is down so tcp on att uverse and twc?! weeeeird TWC here my vpn runs on tcp :( what do those have in common that we know of, only att and twc I can’t ping or SSH into the server even over the VPN but I can connect via the iKVM console ikvm console on different ip ranges? thx for the assist anyway pyvpx :) man this is weird yup, UDP works up_the_irons: yeah I'm kind of at a loss as to what I'd do yeah i always find the weird shit happening for me from comcast too shiz comcast as well? up_the_irons: you dont see these networks over any2, do you? fucking up something like all tcp traffic sounds like a coresite thing...on its face :p no, I'm small peanuts to them, they would not peer with me over any2 i can connect via ssh from devio.us not exactly sure where that is hosted if you guys do "mtr --tcp arpnetworks. com ", where does it stop? for me it is around nlayer same here dimenoc, somewhere in FL unfortunately mtr wants to crash on me today I’m conneted to a remote machine at 162.248.100.99 I can’t access the arpnetworks via TCP but the mtr command completes fine nmap -Pn -p22 -sU arpnetworks.com works even with --tcp? yes weird, doesn't finish for me and all my server monitors at nodeping.com have failed as well nmap comes back as closed for me, for 22. when it should be open for my own site shows filtered for arpnetworks.com no packet loss and 30-40ms pings using mtr so it doesn’t seem like it’s bandwidth actually I have no problems contacting arpnetworks.com at all from that server - just my own arpnetworks IP I'm trying from several nlnog ring nodes and everything works so far so mtr —tcp arpnetworks.com works mtr —tcp danconnor.com does not (failing at ge0-arpnet.cust.lax07.mzima.net) thestereobus : what's your server ip? 206.125.168.2 oh! that i know.. you're leaking traffic between your two uplinks and stp is shutting down your interface (completely unrelated issue) i am 206.125.175.0/29 i can re enable... done ah, ok. is that related to link aggregation? I enabled that recently most likely, yes ok. I enabled that recently I’ll turn it off for now carry on cool :) lol, finally back in weechat on my laptop (instead of phone). i just relayed through an nlnog node to my irc box.. sucks that i have to do that, but.... works for now my irssi session is still working fine, from my box to freenode lol just can't get into my box i have a feeling that this is a GTT/nLayer issue, so i just emailed their NOC word for me tcptraceroute gets to: 7 ae-5.r04.lsanca03.us.bb.gin.ntt.net (129.250.2.220) 28.686 ms 27.857 ms 28.676 ms then dies on udp next hop is: 7 ae-5.r04.lsanca03.us.bb.gin.ntt.net (129.250.2.220) 29.388 ms 29.274 ms 27.852 ms err 8 ge-0-7-0-24.r04.lsanca03.us.ce.gin.ntt.net (129.250.198.186) 29.820 ms 29.341 ms 29.662 ms up_the_irons: they are answering at the noc number.... but wont help me lol are you a direct customer of us.ntt.net ? so confirmed network issues i guess? im getting emails from people all around from here. it looks like the tcp reply never get backs to the client - server get my http request, and sends back reply - just don't reach the client some of the users i've been in touch with says that icmp ping doesn't reply either have yet to find a rig i control that can reproduce yeah, something verio I vaguely recall them having some mysterious IDS thing that did things like this occasionally but that was years ago but nlayer/gtt has used verio for ages. yeah, seems wicked atm. oh well, i'll wait some more - not working from .dk sites, but works from .de site enjoy the beer ;D =\ is Arp having some networking problems currently? I can ping my VPS, but SSH attempts time out, also cannot load arpnetworks.com in a browser. yes they are it appears only TCP traffic is not getting through, from pretty much every ISP in U.S. well, more like the more notable US access networks and a non-obvious number of european networks ok, I guess I don't have to go strangle somebody in our IT department then :P pyvpx: Only adding one data point, but ssh to my arp vps works fine for me in .uk (traffic goes via NTT for both v4 and v6) UK seems to have hit something magic - http://downdetector.co.uk/problems/ee-everything-everywhere/map/ works fine from .de well, my slice. BT outage chart dont look good :( http://downdetector.co.uk/problems/bt-british-telecom/map notion: I wouldn't put too much trust (i.e. any at all) in downdetector plett: ok no issues from .se either plett: my trafik through the UK link fails, but works perfect from my DE notion: From where to where? i'd trust that BT is terrible though Naturally, they are a defacto monopoly telco plett: UK link from: 2.111.95.123 to: 174.136.105.26 notion: That's from .dk to arp? Does that normally go through .uk for you? plett: yes my vps networking seems fine in fact the irssi instance i'm typing in now is running on it I still get reports from users having issues notion: Does a traceroute for that go through nlayer or GTT? up_the_irons said it might be a problem with them mostly "is the site down" plett: not sure who's who on the path - my traceroute: http://pastebin.com/754FDTGx notion: And ssh doesn't work for you? plett: ssh does not work plett: or http plett: ICMP does tho, odd notion: I can't connect to ssh on 174.136.105.26 (I'm probably firewalled), but HTTP is fine for me. My path looks very similar to yours http://pastebin.com/4Heg7P6B And here I was thinking it was Windows being dumb... Well I mean it still is, Windows is being dumb and refusing to ipv6 right now for whatever reason. plett: ok, thanks for checking - guess I'll wait some more and see how it goes But that just leads to one more data point: my ipv6 connection is still up and working and connected. But comcast ipv4 has shit the bed for me too - ping, but not tcp *gathering forgotte1: answering your hilight from another channel - yes I'm alive > But comcast ipv4 has shit the bed for me too - ping, but not tcp same here also, can't telnet underworld1.no.quakenet.org 6667 from ARP (hooray for openvpn/udp still working) also GRE Good to hear - hadn't tested that yet hmm... can't get to the IRC logger webpage how long has this been happening? I've been out of town... fwiw, comcast -> nlayer -> mzima ICMP acf_: for the bast 6-ish hours I remember trying to ssh in from my T-Mobile phone earlier in the week, and it not working... but I could ssh to Amazon EC2, and then from Amazon to ARP I'd offer mtr --tcp information, but on my FreeBSD box at home I get "bind(): No error: 0" and my OSX machine gives me "bind(): Undefined error: 0" so... sorry. forward path to underworld1 is via Level3 forward path to Comcast is via Level3 From the reports, this seems very widely spread, so more likely an ARP router issue than a provider. But that's just my $.02 distilled from the IRC history. Looks like the office TWC connection is getting through too yeah, other Level3 things seem to work... mtr --tcp and ICMP from Comcast: http://paste.debian.net/115125/ (The first reports were of TWC failing) Comcast Business gets through (dammit the only Linux box online at home is Debian, with mtr 0.82 that doesn't have --tcp) hmm... nothing in backports? acf_: Nope. https://packages.debian.org/wheezy/mtr-tiny I remember coming across that missing too... but Debian is/seems very closed-off when it comes to some guy that wants to say "hey, can you ___ this?" aww darn. I just run Debian testing on most stuff that's not critical. jessie has mtr 0.85 an mtr --tcp from arp to comcast and underworld1 is blank blank? how odd. ie, not hops but comcast business shows via Level3 as usual http://makeameme.org/media/created/how-odd.jpg It no doubt goes without saying, but up_the_irons, please fix :) comparison of Comcast business and residential paths: http://paste.debian.net/115137/ Unable to get to arpnetworks.com from comcast. traffic seems to be dropped after nlayer > mzima. So this basically affects all customers at arp? Im colo should that matter for some reason has anybody noticed IPv6 destinations being affected? I'm able to get to arpnetworks.com if I go through a server in Washington state (different ISP and different route) for me prob seems to be between ge0-arpnet.cust.lax07.mzima.net (67.199.135.102) and arpnetworks.. I can confirm that from a user I have in australia anyway.. enough whine for tonight. Any idea who can help at mzima.net? http://www.mzima.net gives 404. Was this up before? mzima is part of nlayer just read the IRC logs... it's all good. up_the irons knows about this already k, thanks mzima no longer exists but yeah, it is apart of GTT i am able to get to everything from an earthlink connection in San Diego. bummer i know dont judge me :( still going on? m0unds: pretty sure ya. im at work now and things are working, but diff isp. all my graphing and stuff is fine, has been all night via comcast in albuquerque hola anyone having issues connecting to arp from comcast? qbit: \o qbit: people be having issues from most major U.S. ISP's it's just tcp traffix o/ yeah - icmp is going fine well good to know it isn't just me fo sho :) Hi guys.. looks like I'm not the only one with issues I have clients in GA who can't connect to their stuff in ARP http://dpaste.com/0C59HJN traceroute makes it to the network fine yeah, tcp is not going only icmp er udp sexy Just scrolled way up and read the chatter.. looks like it's a fun one i discovered it at like 2am :P I'm having some issues as well. first nagios alert came at 3am CST Where can I get details of what's happened? I just joined the channel. Could someone copy the log to pastebin? can you get to http://irclogger.arpnetworks.com/ from somewhere? I can from the office TWC connection but that might be over v6 too yoberi: http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue i didn't see an AAAA record for it I can't seem to reach anything arpnetworks.com at the moment; yet my mobile phone can using the provider's data network Indeed there isn't. Yep, I can reach it over v4. (from twc business) acf: thanks! appears to be a huge problem for a lot of folks. Im sure up_the_irons is all over it. truf will we see a tweet from @arpnetworks or @bsdvps? Good point/idea I've had one of neil's employees also mention that can't get to his computers from some place in europe Is there an issue with some VPS Services and the website? invader: http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue acf_, Thank you Go acf_ ! Well thats fucked up. lol lol a wild up_the_irons appears :D Still not fixed :( Is something terrible happening right now? I can’t access the portal or my server. Yes What is it? bmacs: bet you can ping it, just not tcp yup http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue https://twitter.com/arpnetworks/status/499235055323996160 bmacs: somewhere, something is not passing tcp traffic. i see drops around nLayer, opened a ticket with them. They gave me a response that made me want to scream (basically, "Why are you using IPv4 to test an IPv6 host?". W.T.F. our hosts are dual homed dumbass, it has NOTHING to do with the issue) Okay, I jsut took a different route to my host and was able to access it. ugh yeah same here is it possible I'm having trouble getting through level3 ? erratic: are you seeing any drops around there? yeah when i trace to google or linkedin dropes at the 6th hop erratic: so from your vps to google? yes I can get through to other sites like yahoo seems to be back now lemme check yep Im back that was strange wait, so it works now? yes up_the_irons: btw, here's an mtr from a working network, both TCP and ICMP. I noted that the arpnet.cust.mzima router shows up in the ICMP but not the TCP - odd? http://sprunge.us/dfOK (I don't have a working mtr --tcp on a non-working network) brycec: yeah, i'm seeing the same I have no idea what it means though. And the fact that it's a working connection suggests that' s just a red herring. Whew; able to connect to my server from home, though not from work or cell phone. Hooray speaking of +o should someone announce the issue in the /topic too? (Thanks up_the_irons for tweeting it) (This is in Sweden, BTW.) any idea why a TCP traceroute would have zero hops? I still don't understand the /topic-sized announcement. "Some networks are down and we do not yet know why"? mike-burns: Just something that people would see when they /join and ask "is there a problem?" gotta make some calls, brb Something like "Upstream providers are dropping TCP traffic. We're working on it." good luck, up_the_irons Whew that's a long one. definitely when I do an mtr --tcp -4 underworld1.no.quakenet.org zero hops are displayed my arp metal box acts as a router, so when I do the same mtr from a machine routed through it, one hop is displayed hmm... looks like ARP routers don't show up in ICMP traceroutes anyway \o/ *TCP tracerouts this is kind of strange... whoa if I mtr --tcp -4 underworld1.no.quakenet.org, zero hops appear I can see in the ICMP mtr that 4.71.143.105 (Level3) is the first hop outside of ARP mtr --tcp -4 4.71.143.105 works fine so TCP traffic destined for that host must have been dropped before it reached 4.71.143.105 ? but TCP traffic directed to 4.71.143.105 is passed? that host == underworld1.no.quakenet.org hmm. my mtr doesn't have --tcp macports has only version 0.85 of mtr, it seems that's the one I'm using ahh - same as freebsd but it doesn't have --tcp :) hmm... mine definitely does must be a fork and sadly, without changing the version number to indicate that it's the Debian Jessie mtr package ahh - the github has --tcp wow... --tcp was added to the docs in 2013 maybe it's not supported on freebsd? maybe the code thinks that all the world is linux which of course, fails on the BSDs and OSX I noticed on FreeBSD's mtr, it simply craps out if I add "--tcp" (0 hops, wtf) anyone have a Ubuntu 14.04 host on our network that I can run some mtr's on? I'm always a version behind, so I don't have --tcp in my mtr yeah I also don't have a working mtr on a non-working host, lol up_the_irons: Linux mtr does the zero hops thing too I’ll make you a login I did a test with it earlier ^ acf_: hmm weird acf_: hey :) hru oh hi erratic. I'm alright, yourself? sitting here at home chilling and applying for some jobs RandalSchwartz: does it work in europe no sorry works great in US and CA though hey all.. any update? several clients beating down my path now :) Beat back. I think some of use at in the same boat. invader obviously ;) Just curious if there was any update (just getting into the office) If you ever dealt with Carriers its a bitch and a half. Give it time. up_the_irons, I jsut sent you an ssh login to my box if you need to run mtr on it invader I have, and am, just asking if there's an update up_the_irons: Is shutting down BGP to nlayer an option, if they can't work out how to fix it? Do we 100% know what Carriers are effected? level3 is affected maybe? seems the forward path to Comcast is via level3 pjs: no update yet and tcpdump says TCP packets don't arrive via the forward path plett: i shutdown all carriers in turn, last night, but no change :( i'm putting together a very detailed report and going to send to NTT, Level 3, GTT, etc... but TCP packets via the return path (nlayer) are fine brycec: does ARP to TWC go via Level3? Anything I can do from a carrier prospective? acf_: last I tried, yes. Also ARP to Comcast yeah. looking a lot like Level3 related then? same for underworld1.no.quakenet.org invader: which carrier perspective? Is invader a carrier? Can we just blame you? :D I work for a telco in WI. I multi homed to ntd twc and comcast. lol up_the_irons: Stupid question, because I'm sure you've already considered it, but are we sure it's not s7.lax having issues? invader: oh nice, can you do a "mtr -4rwc 10 --tcp arpnetworks.com" ? :) With a nick like invader, how can we not? haha brycec: well, s7.lax appears OK, but I could try to engineer it so we go out s1.lax instead. but first, gotta finish sending this report to carriers... Hmm not a damn thing. Its like the route is not even there.. Let me connect into our Verizon part of the network invader: so what does it show? no hops at all? nothing. Just verizon just works. one min up_the_irons: Best as I can tell, it's the single most common data point. And the fact that this TCP filtering would happen on multiple carriers, vs. a single router... As illogical as it seems, it's the most logical scenario from where I'm sitting. i tend to agree seems to me that Level3 is the only one affected lol acf_: but not universally... brycec: s1.lax uptime is 5 years, 25 weeks, 22 hours, 4 minutes brycec: could you point me to an example please? acf_: look in the scrollback for my sprunge.us link. :p acf_: In short, from work, which is TWC "Business Class" I have no issues to ARP over nlayer/mzima, and the return mtr (not included) comes over L3 level3 -> trit? trit not there (gathering mtr now) hmm yes... not all of level3 is affected fwiw the level3 looking glass routes through gblx -> trit -> arp acf_: http://sprunge.us/PXFD (ICMP because I have an old mtr on my VPS) ok thanks so it looks like some of Level3 isn't affected but are there any non-Level3 routed destinations that are affected? can't wget http://level3.com fwiw acf_: from? arp ditto up_the_irons: ^ (but I can from other networks including an "affected" comcast connection same hmm So, whether or not related to ARP's issue, (but it probably is), L3 has some split-braining going on ok i blew away level3 lots of people tweeting about level3 problems https://twitter.com/TVBroadcast/status/499243972221734912 TWITTER: Anyone aware of a "a large Internetnetwork issue is affecting ISPs across the country, like Comcast, Cogent, Level3, Time Warner, etc.?" (Tue Aug 12 17:20:13 +0000 2014, retweeted 1 times) https://twitter.com/factor1/status/499249059433943040 TWITTER: Good news is that our network issue seems to be clearing up as Comcast and Level3 are responding to the fix. (Tue Aug 12 17:40:26 +0000 2014, retweeted 1 times) up_the_irons: suddenly working whoa oh wait nevermind, was v6 hate that dualhoming biting me in the ass Heh. Yeah, all my users are complaining but v6 works just fine! hehe :D i've had v4 drop out in the office for hours at a time before I noticed (dhcp issue) Had someone try the mtr command but it produced no output. I don't have access to the box so I can't debug. http://downdetector.com/status/level3 ok seems Comcast goes via ntt now still no TCP packets going via Comcast -> ARP path Shit this explains the issues I am having in SunGard.. They just called L3 is broken and has been for the better part of a day. -_- Thanks for the call now.. Butt heads. mtr --tcp isn't working from my home FreeBSD box, trying to fix that so I can supply another data point return path still ok seeing the same problem from Comcast in NH though IPv6 works, fwiw up_the_irons: fwiw, curl level3.com from arp still not working, mtr shows it going ntt->level3, unable to ping same here. seems dropping level3 didn't fix the problem twobithacker: Yeah, my friends in MA can't connect via Comcast so probably the same route problem there. brycec: yeah it's taking the exact same path, just through NTT now ;) Whomp whomp. (And return path ARP->Comcast is over NTT->Comcast) yep that's still broken too though huh, I can connect just fine from Comcast Business though So Level3 is having problems... and Verizon has been lashing out at L3. I bet Verizon haxxored Level3!! ^ sarcasm twobithacker: yeah that's been observed ironically, my Verizon DSL line is the only residential connection I've seen unaffacted Seems home TWC and Comcast (among others) are affected, business customers are okay I'm fine on TWC. :) In NYC. Also Verizon FiOS at my office is fine. it's interesting that's it's just tcp Yeah. it seems to imply that somebody is discriminating tcp traffic in some way or another Again, I blame the whole Netflix/Verizon debacle :P "HOW DO YOU LIKE YOUR NETFLIX NOW, BITCHES?!?" "YOU THOUGHT WE WERE THROTTLING? NOOOOOOW WE'RE THROTTLING!" etc lol (except netflix.com is working for me) looking at just the forward path ARP -> Comcast residential (both v4 and v6) it's broken both with ntt and level3 Well I have other work to get done... Best of luck, #arpnetworks see you later brycec If it's just TCP, maybe some sort of traffic shaping failure of sorts? I was thinking about that... but where? brycec: LOL Yeah this is far more exciting than the admin tool I was just writing ... s/ail/feat/ If it's just TCP, maybe some sort of traffic shaping ffeature of sorts? hmm... traffic shaping, interesting.. http://www.secureworks.com/cyber-threat-intelligence/threats/bgp-hijacking-for-cryptocurrency-profit/ acf_: BryceBot mhoran erratic invader : see any change? checking first time using mtr testing ARP -> Comcast path via NTT ... still broken for tcp No dice. ok just talked with a friend on another connection says he can't connect to underworld1.no.quakenet.org either but it works from Comcast i still find it weird that level3.com is inaccessible yeah I haven't noticed anymore outage problems since earlier was having trouble getting to cloudfront, linkedin, google, things like that erratic: from what connection? just my vpn tunnel I have everything routed over my 206.125.168.65 addr oh when i did traceroutes on stuff it was timing out at like 6 hops in level 3 then it just started working again about 5 minutes later I wonder if theres a way I can continuously check with mtr like if I wanted to watch for changes I guess that wouldn't make a lot of sense now that I think about it mtr polls continuously, and should show changes as they happen weird sudo mtr --report -4 google.com it just runs once and exits for me probably the --report ah just shutdown Trit Networks BGP session, stab in the dark you better fix all these things outside of your control up_the_irons oh nice lol lol acf thx still appears broken for the Comcast thing im writing a blog post as we speak bad mouthing your services trit isn't in my comcast path fwiw np erratic yeah but sometimes... ya never know ;) lol: https://twitter.com/ellisinzion/status/499255527990239232 TWITTER: @Level3 Yes you do have a major core IP outage right now. Stop saying there is no issue. Multiple big voice providers are down cold. (Tue Aug 12 18:06:08 +0000 2014, in reply to @Level3) OL LOL ae-3-3505.ear1.Dallas1.Level3.net? I totally just tweetd about this neat so... ARP -> Comcast forward path via either NTT or Level3 is broken which kind of indicates that the issue doesn't lie entirely with NTT or Level3 acf_: But from .uk using NTT all the way to ARP works fine but other things (TWC) are broken too plett: but what is your return path yeah, pretty much all the NTT stuff I've seen works fine up_the_irons: I'll check up_the_irons: NTT as well ec2 (ntt both ways) is fine acf_: What's the path in both directions for the broken arp <-> comcast ? ARP -> Comcast is via NTT now, was Level3 earlier, both were broken Comcast -> ARP is via nlayer Ahh. nlayer I've been testing just the ARP -> Comcast part of the path the nlayer part looks fine acf_: you never receive acks? it's just TCP for everyone, yes? I do a tcpdump on the comcast side I try to connect from the arp side no packets come through UDP packets? or no packets going the other way works though I use tcpdump -ni eth3 port 6668 and telnet [hostname] 6668 hmmm... HRMMM. UDP packets are fine any change just now? i can actually check my email yep yup packets are getting through to comcast now Im up up_the_irons, comcast works now same with twc comcast is symmetric nlayer now the thing with NTT / Level3 is, a lot of NTT routes go through Level3 anyway is it possible that something is dropping things with asymmetric paths? so what i did was, lowered s7.lax default route local pref so most traffic goes out via s1.lax directly, not hitting s7.lax. s7.lax has Level 3 and NTT. acf_: that would take stateful filtering i believe and is pretty rare on regular transit links wow. everything is via nlayer now :P even any2ix stuff nlayer to the rescue! (surprisingly) yeah not going through s7.lax kills the rest of my network ;) (no peers, L3 or NTT) oooo while this works, it seems like a shitty solution unless the problem really is with Level3, in which case we can wait it out now I don't think it is... ARP -> NTT -> Comcast doesn't work... acf_: still? it's not NTT anymore, so not it works oh wait.. couldn't be cuz ARP -> NTT isn't possible right now * now it works it didn't work both when it was ARP -> NTT -> Comcast and ARP -> Level3 -> Comcast ok well w..t.f Im confuzled, how does it kill the rest of your network not that it matters just curious this stuff is interesting to me basically up_the irons took the s7 router out of the network right which is the connection between you and any2ix, ntt, and level3 I gathered that much does that mean none of the networks including mine will be reachable? so without s7, all you get is nlayer erratic: s7.lax is a bigger router and carries more routes. usually, all traffic goes to it. but now i'm diverting traffic to go directly out s1.lax. it just means they'll take a diffetent path and yeah, basically what acf_ said oh right like if nlayer goes down... i'm tempted to reboot s7 up_the_irons: so can you do testing with s7, static routes, etc... now that you've diverted traffic? acf_: well, technically yes, but hard to test when no traffic is going to it ;) just out of curiosity, does ios have a tcpdump equivilent? now that there isn't any traffic, would it be more realistic to debug with tcpdump? well, what i would do is use a SPAN port and then dump that port. i have one set up already. like switch port mirroring? yup that is handy yup, purely meant for diagnostics i can copy all traffic from any physical port or SVI (VLAN) and then just watch it from a regular Linux box with tcpdump Im not sure how much overhead that would be, probably too much but you could probably use it with snort assuming the switch or whatever can handle it, without creating a single point of failure yeah ok, s7.lax is getting a kick in the butt (reboot) sounds like a good idea :P womp womp I saw a momentary drop I presume that was a shuffling? .... wat I lost talk.google.com for like a sec ok yeah... my connection went dead for about 10 seconds yeah I think I lost everything for a secondlet laggggggg s7 reboot is done, i think i'm going to send traffic to it again and see what happens whats the verdict? :D Guilty. invader: see ^ and the sentense? looks like it's getting better how's it looking for u guys now? I need to consume this alcoholic beverage erratic: alllllcohol arp -> ntt -> comcast seems to work now glug so it was s7 being weird? then i'm going to throw fucking s7 into the dumpster, gaah! what sort of hardware is s7 up_the_irons: you just can't catch a break with that thing srsly but that's why shotguns exist erratic: Cisco 6506 w/ sup720-3bxl i've had a replacement Sup in the cage for months now :( Im just curious, couldn't you use a linux box to do that in the triage order of things, replacing it just wasn't high on the list. but it is now. up_the_irons: did you ever swap the sup, or is the one in the cage the one you got to replace it when it did that weird CEF barf thing? that thing looks heavy erratic: well you wouldn't get any hardware acceleration and the pps is really low in comparison. so basically, a linux box would melt under a DDoS m0unds: yeah it's the replacement for the other issue i'm going to re-enable level 3 now yeah Im reading the specs on it now lol why just tcp? man the ipv4 routing table is just getting too large :( and *some* tcp too? acf_: no. fucking. idea. ;) wow jeez im sure if it's working for others, it's working for at&t too i think i need to stop being married to the (now) old idea that carrying a full routing table is *that* much better Sorry I broke the Internet GUILTY! on a slightly unrelated note, are you planning to announce to level3 at some point? Webster: we have a lot of 6500 ciscos and have recently grown up to be able to get the ASR 9000s ooooooo ugh, ios-xr so, my first cisco class starts monday. and ends pretty much never if you take into account how esoteric it all is haha i hope you learn a lot about serial stuff, because that shit still exists in ios haha m0unds: just a sec my vmware class also starts monday that sounds like more fun indeed I'm only taking the cisco, cuz my job is making me. I don't find routing or switching to be intersting at all. I'd be far more interested in the security tract. m0unds: this is how I decided to upgrade my bios http://3.bp.blogspot.com/-GDEA1qZ0thk/U-d-u4aUM8I/AAAAAAAAPTo/tgcCHYdR5c4/s1600/Screenshot+-+03272013+-+05:40:47+AM.png just because lol y not hahahahah nice m0unds: hey i use serial all the time, the 2612's are great for out-of-band access to everything :) I have 32 serial ports with breakout cables on one of 'em. oh, i meant like frame relay serial :) oh lol i have an AS2516-RJ (iirc) that takes rollover cables terminated w/rj45s it's great as a terminal server for my crisco lab jpalmer: I have pretty much my lack of interest in anything except computers but more importantly the people I've surrounded myself with who have been willing to help to thank I got kicked out of school i got my employer to approve me bringing it to work so i wouldn't have to try and store all that gear at home :) haha yeah I love piling up my crap at work makes me feel comfortable to have crap around me i racked mine up and told my coworkers to learn something m0unds: definitely yeah I used so much of my own hardware at work since i'm hoping i'll be able to leave here in a year or so, and since they don't have another network engineer, they'll be utterly SOL if they don't well at my last good job I haven't had a decent job in a while finally starting to get interested in stuff again though m0unds: I do the same thing. and I teach them how to make and restore backups first.. so that if someone changes it when they are done, they can instantly revert it back to their last configuration. gives them a LOT of backup/restore practice, and makes sure everyone can use the lab without stepping on other peoples toes. Ive had crap jobs and a few short term contracts and stuff but nothing really my last job really spoiled my appetite for work was laid off a year ago today and Im just now like, ok I'm moving to a new place and I don't know anybody and I have no idea what to expect and I think it will be challenging for me socially erratic: where are you moving to? Netherlands Im in Greece atm thats.. a move, for sure. hehe ah, cool - my grandparents are from the netherlands my wife and I are talking about moving from florida to charlotte, NC NC isn't too bad charlotte is nice - my dad lives in burlington, just outside greensboro need a car for sure I like living in the city too much but I do appreciate living out in the country and I miss it sometimes we'd like in one of the outskirt areas, for sure. horses and such s/like/live/ we'd live in one of the outskirt areas, for sure. horses and such I lived in creedmoor outside of RD for awhile was like 16 or 17 long enough to make friends then move away again I really want to believe despite how much i hate redhat it would be really hard (definitely competitive) to work there but probably pretty interesting especially if you can get a good systems engineering job they pretty much birthed xen and now from what I've been reading a lot of the work on qemu/kvm looks pretty sweet and I'm running a lot of the more recent stuff actually found a vid this morning exactly what I have except the guy has much faster hardware https://www.youtube.com/watch?v=37D2bRsthfI YouTube Tech: "Qemu/KVM + vfio = Virtual machine for gaming with 95% of native performance = no more dual booting" by blu3bird84 (2m 8s), 18,965 views, 222 likes and 4 dislikes. Uploaded 2014-02-02T19:42:29.000Z. anyone seen any issues in the last 30 mins? things are working for me... wish i could test :( no issues here. been clear since the reboot roger that Yep. No issues here.. any issues with kvm 14 today? krv14, that is can someone put the topic back to the way it was :) i tried connecting several times a couple of hours ago without success seems the NTT comcast stuff is finally getting better mus1cb0x: was probably the general network issue we had just looked at my arp latency graph, and it's looking much healthier than a month ago ah, i see the topic how long was the outage? since like 1AM, but it was very selective on certain routes. Cacti showed most traffic was getting in/out. ouch, what was the cause? Looks good here. mus1cb0x: s7.lax being stupid (Had to get on a desktop for that one.) stupid? aka not smart sarcasm isn't helpful nor is "stupid" when discussing network outages i have to justify the outage and i'm not going to use stupid and risk my job this isn't an offical forum go put in a ticket heh, geez at least you're not op'd, because you would be the wrong person to represent arp play nice now up_the_irons: It does seem like there were more widespread issues than just with your router, though? I definitely experienced issues with non-ARP connectivity this morning. Perhaps all related in some way? e.g. a crazy routing table causing issues on routers across the nets? mus1cb0x: There was something strange with TCP traffic not being passed through to certain destinations over s7.lax. UDP was fine. I guess the fact that it was TCP only kind of makes my thought make no sense. Nor does that sentence make sense. Huzzah Internet working! mhoran: hmm.. that's a good point about the routing tables, maybe it is all related in some way oh boy, I missed a lot The TCP thing really sounds like someone played with acls. mhoran: I didn't realize you had non-ARP connectivity problems too did anyone else see the article about routing table hijacks lately to commandeer bitcoin mining? up_the_irons: Yeah I was having trouble with some things before I left from home this AM, over IPv4 from Time Warner. mus1cb0x: someone pasted that, yeah mhoran: hmm... interesting up_the_irons, L3 had an confirmed issue was a few asn's/ any chance that could be a cause? some type of BGP poisoning? I suppose it's possible http://www.reddit.com/r/cryptodev/comments/2d0bzx/bgp_hijacking_for_cryptocurrency_profit/ up_the_irons: we hit 512,000 active routes today, were you aware of that? up_the_irons, From the sounds of it, They only said "route issue" Getting a straight answer out of them is like try to date Misty Dawn. jpalmer: lol, someone just PM'd me, "related to the 512k routes issue?" perhaps that was it... i'm showing 499K right now i'm going to be doing a route haircut tonight maybe kill all /24's up_the_irons: lots of routers with the 512k limit have been having issues today. sup 720 should handle 1m routes, iirc Happy grooming might be thinking of the 2t though danke Sure should. what kind of gear you guys using at your edge? m0unds: jpalmer : i have the -3bxl, so it does 1M routes. however, i have two full tables, so right about there ohhhh gotcha CEF is actually bitching at me in the logs... lol haha i hate cef such a kludgy thing I could see CEF blowing up and causing all sorts of weird issues, in addition to the missing routes. Still don't understand why UDP would work, unless it took a different path somehow. yea, but it usually fails by dumping forwarding onto the cpu then the cpu shits its pants and the whole chassis stops passing traffic because of the load or crashes, depending haha http://status.livestream.com/ -- so it looks like my ex startup had routing troubles today, too. :p All the competent network techs left, so I'm not surprised their router crashed. Was also a 720-3bxl. That I doubt anyone has maintained in two years. well that was easy: s7.lax(config)#mls cef maximum-routes ip 768 Maximum routes set to 786432. Configuration will be effective on reboot. looks like another reboot will happen in the very near future... (tonight maybe) ;) do it now while only half of people think its fixed! :D hahahhah nice:) boo cef, boo and yeah, CEF blowing up means the router falls back to software switching, which will melt it yeah, i wonder if cef was loaded and it was unable to take on any flows actually s1.lax did that once, i hit the 128K limit. it went full on to CPU switching. some of you may remember that outage. but hey, it didn't crash, still 5+ year uptime! :) hahaha s1's a 45xx right? yup up_the_irons: so it was hitting the 512k limit? my buddy @ easystreet up in oregon had a similar thing w/a 45xx killing CEF and dying horribly overnight one night jpalmer: i don't think so, or else *nothing* would have gotten through Yeah, that's happened to me too, when I foolishly tried to load the entire routing table on a 4500. :p Upstream route filter was misconfigured and I didn't guard against that. ouch preemptive optimization doesn't exist in networking :P er, premature Heh. http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-major-outage-across-us-network-providers http://www.cidr-report.org/as2.0/#General_Status Number of ASes seems to be approaching a number that someone may have hardcoded into a system, too. :p yeah, haha I heard a rumor of a bgp apocalypse and I figured #arpnetworks would tell me whether or not to panic bgpocalypse wow. ziprecruiter.com has gotten about 5 times faster in the past few days for job searches. my efforts are paying off. m0unds: sjackso : lol so much fuss for so little and of course media blows it up hola :D so - are the weird issues fixed? from earlier alive! i seem to be having some dns issues - not sure if it's residual or not could also be from the upgrade i stupidly decided to do mid day :P aha host can resolve things fine.. but ping doesn't making me rage qbit: connectivity was resolved about 7 hours ago k it's totally fucked up - ping doesn't resolve some hosts.. chat.freenode.net.. etc - but it does for others and host works for all qbit: reboot? Seems like the host dns resolver cache is fubar'd but host does a fresh fetch iirc or at least a direct connection, bypassing the normal caching yeah - reboot does nothin to fix it RandalSchwartz: i meant, are you up to freebsd version 9 or 10 when did you "meant"? I'm still on 8.4 probably 10.x during some spare downtime in december after I first do it on insightcruises.com machines. :) fink? yo RandalSchwartz was answering your questions i said: fink: yo RandalSchwartz, how's it hanging. are you up to 9 or 10 with your vps? never saw that or maybe long ago yea, yesterday ;) so there's my answers i'm wary of moving my zfs on root from 8.4 to 10 Sounds exciting! yeah - there's definitely some possible issues there which is why I want to try it on $client's computers first :) there's some settings for internet in /etc/rc.conf that change, I'm told and if I get those wrong, I don't get back in to the box :) Yup, there's that. Though it's not too crazy. RandalSchwartz: i'm actually somewhat curious how much business/interest there is in insightful holiday packages i never really thought of it before https://supportforums.cisco.com/document/12202206/size-internet-global-routing-table-and-its-potential-side-effects lol http://bgr.com/2014/08/12/comcast-fcc-commissioner-clyburn-dinner/ lol curious in what way? these aren't holiday packages if you'd spend 4 days at a hotel in $random_city to attend a conf, why not 7 days on a cruise ship to do the same thing, but with better speakers? i dunno, i just never thought of that i assumed most techies would rather book their own stuff and explore like a local or maybe that's just me yeah, most people haven't, which is why Captain Neil makes some bucks forgot about the conf part primarily, these are conferences they just happen to be meeting on a ship instead of a hotel lots of advantages to that right all of my industry conferences are in vegas there's one that was in san diego once but never again and one big plus... the speakers are in the same BARS and same DINNER as you are unlike land conferences, where they all scatter. and our speakers are expected to mingle with the attendees at those. imagine the conferences you've attended where you could bump into one of the keynote speakers later at the bar... or even on a shore excursion... a few hours in the same small group with them so you have people actually interested in the subject there as opposed to forced to attend by corporate geez... you must have a twisted view of what we do! I don't even know how to answer that. "I am interested in X" "Oh look... a cruise where speakers will be talking about X" "I want to go" [if employed] "Employer, can you sponsor me to this conference?" [if retired] "Let's see if my travel budget covers this" That's how people get on Nobody is *forced* "Oh gawd... FORCED to go on a CRUISE!" :) see how silly that sounds now? i think hazardous was just pointing out that people attending a conf on a cruise are more likely to be going because they want to and not because someone forced them to at least that's how it read to me what was that command to run an mtr report with output to a text format for easy copy /paste have the upstream TCP transit issues been resolved? I was seeing intermittant failures earlier tonight but they didn’t last long mtr -4 --tcp -rc 10 mnathani: add -w so it doesn't cut off hostnames (wide format)