arpnetworks down??
nvm
what happened?
it seams at&t have blocked my connection to my own sites / services
uverse
well and arpnetworks.com entirely
those dirty sluts man!!
up_the_irons: what is arpnetworks ipv4 range?  i wanna make sure they unblock it for everyone with at&t service in case others are having the same problem.  People could be loosing potential business if no one on at&t uverse can access any sites hosted at arp
not sure it's that wide spread, but can't hurt i dont think
hmm, you can find out using IRR
if he's been good about his object maintenance ;)
my objects are up to date
it's weird, I'm on twc, i can ping everything in my cage, but SSH, http, etc..  seems to be all blocked. started about an hour ago.  I'm still in irc only because I'm using weechat android from my phone
thats fucked
oh wow....
so it's not just at&t...
same symptoms here, can ping everything. but ssh, http, https all blocked
lol this at&t rep wants me to reset my modem to factory. now i feel like an ass :P
up_the_irons: can I try anything for you from Germany? ;)
pyvpx : how about just arpnetworks. com?
wyliebayes.com  , nadasound.com
2 of mine
pings and loads just fine
I'm coming from 37.120.2.102
overall, my bandwidth graphs for the big pipes look pretty normal
so most traffic is still getting out
so weird about twc and att
it's only residential uverse att tho.  i can access via my att 4g network no problem
what are the blocks you guys are coming from that can't reach arp?
i am, 99-95-174-245.lightspeed.sndgca.sbcglobal.net
pyvpx: would be weird bgp would effect all services, except ICMP
oh, I meant from a policy stand point
localize to a router or a routing event that could point to a config change
it's easier to hit someone on the head at att when you are like
"yeah at 4am this shit started"
they go "wait we had a thing at 4am uhhhhh"
...sometimes
i would be surprised if the "tech" im talking to in chat can even feed himself successfully
from att
sometimes if you play the act
the ip noc folks can entertain you
give you ticket numbers or other depts to call
ya im trying, had to DEMAND i speak to his supervisor cause he tried to pawn it off on the hosting provider hehe
by pasting me out of the book answers
if resetting my modem didn't fix the problem, we have exhausted all possible solutions. LOL
hmm
the denver router on their looking glass had a thing 42 minutes ago
probably nothing
everything seen through verio
hrmm
and since icmp works it's hard to see where it might be failing
i am in San Diego CA
I'm playing around with shutting down certain transit sessions, to see if any of them are the culprit. nothing successful so far
it's like only tcp doesn't work. i can still connect to my VPN (udp) and dns still works fine... wtf
yeah, I’m connected over VPN but TCP is down
so tcp on att uverse and twc?!
weeeeird
TWC here
my vpn runs on tcp :(
what do those have in common
that we know of, only att and twc
I can’t ping or SSH into the server even over the VPN but I can connect via the iKVM console
ikvm console on different ip ranges?
thx for the assist anyway pyvpx :)
man this is weird
yup, UDP works
up_the_irons: yeah I'm kind of at a loss as to what I'd do
yeah
i always find the weird shit
happening for me from comcast too
shiz
comcast as well?
up_the_irons: you dont see these networks over any2, do you?
fucking up something like all tcp traffic sounds like a coresite thing...on its face :p
no, I'm small peanuts to them, they would not peer with me over any2
i can connect via ssh from devio.us
not exactly sure where that is hosted
if you guys do "mtr --tcp arpnetworks. com ", where does it stop? for me it is around nlayer
same here
dimenoc, somewhere in FL
unfortunately mtr wants to crash on me today
I’m conneted to a remote machine at 162.248.100.99
I can’t access the arpnetworks via TCP
but the mtr command completes fine
nmap -Pn -p22 -sU arpnetworks.com works
even with --tcp?
yes
weird, doesn't finish for me
and all my server monitors at nodeping.com have failed as well
nmap comes back as closed for me, for 22.  when it should be open
for my own site
shows filtered for arpnetworks.com
no packet loss and 30-40ms pings using mtr so it doesn’t seem like it’s bandwidth
actually I have no problems contacting arpnetworks.com at all from that server - just my own arpnetworks IP
I'm trying from several nlnog ring nodes and everything works so far
so mtr —tcp arpnetworks.com works mtr —tcp danconnor.com does not (failing at ge0-arpnet.cust.lax07.mzima.net)
thestereobus : what's your server ip?
206.125.168.2
oh!  that i know..   you're leaking traffic between your two uplinks and stp is shutting down your interface (completely unrelated issue)
i am 206.125.175.0/29
i can re enable...
done
ah, ok. is that related to link aggregation?
I enabled that recently
most likely, yes
ok. I enabled that recently
I’ll turn it off for now
carry on
cool :)
lol, finally back in weechat on my laptop (instead of phone).  i just relayed through an nlnog node to my irc box..
sucks that i have to do that, but....
works for now
my irssi session is still working fine, from my box to freenode lol
just can't get into my box
i have a feeling that this is a GTT/nLayer issue, so i just emailed their NOC
word
for me tcptraceroute gets to:   7  ae-5.r04.lsanca03.us.bb.gin.ntt.net (129.250.2.220)  28.686 ms  27.857 ms  28.676 ms
then dies
on udp next hop is:   7  ae-5.r04.lsanca03.us.bb.gin.ntt.net (129.250.2.220)  29.388 ms  29.274 ms  27.852 ms
err
 8  ge-0-7-0-24.r04.lsanca03.us.ce.gin.ntt.net (129.250.198.186)  29.820 ms  29.341 ms  29.662 ms
up_the_irons: they are answering at the noc number.... but wont help me lol
are you a direct customer of us.ntt.net ?
so confirmed network issues i guess?
im getting emails from people all around
from here. it looks like the tcp reply never get backs to the client - server get my http request, and sends back reply - just don't reach the client
some of the users i've been in touch with says that icmp ping doesn't reply either
have yet to find a rig i control that can reproduce
yeah, something verio
I vaguely recall them having some mysterious IDS thing that did things like this occasionally
but that was years ago
but nlayer/gtt has used verio for ages.
yeah, seems wicked atm. oh well, i'll wait some more - not working from .dk sites, but works from .de site
enjoy the beer ;D
=\
is Arp having some networking problems currently?
I can ping my VPS, but SSH attempts time out, also cannot load arpnetworks.com in a browser.
yes they are
it appears only TCP traffic is not getting through, from pretty much every ISP in U.S.
well, more like the more notable US access networks
and a non-obvious number of european networks
ok, I guess I don't have to go strangle somebody in our IT department then :P
pyvpx: Only adding one data point, but ssh to my arp vps works fine for me in .uk (traffic goes via NTT for both v4 and v6)
UK seems to have hit something magic - http://downdetector.co.uk/problems/ee-everything-everywhere/map/
works fine from .de
well, my slice.
BT outage chart dont look good :(
http://downdetector.co.uk/problems/bt-british-telecom/map
notion: I wouldn't put too much trust (i.e. any at all) in downdetector
plett: ok
no issues from .se either
plett: my trafik through the UK link fails, but works perfect from my DE
notion: From where to where?
i'd trust that BT is terrible though
Naturally, they are a defacto monopoly telco
plett: UK link from: 2.111.95.123 to: 174.136.105.26
notion: That's from .dk to arp? Does that normally go through .uk for you?
plett: yes
my vps networking seems fine
in fact the irssi instance i'm typing in now is running on it
I still get reports from users having issues
notion: Does a traceroute for that go through nlayer or GTT? up_the_irons said it might be a problem with them
mostly "is the site down"
plett: not sure who's who on the path - my traceroute: http://pastebin.com/754FDTGx
notion: And ssh doesn't work for you?
plett: ssh does not work
plett: or http
plett: ICMP does tho, odd
notion: I can't connect to ssh on 174.136.105.26 (I'm probably firewalled), but HTTP is fine for me. My path looks very similar to yours http://pastebin.com/4Heg7P6B
And here I was thinking it was Windows being dumb...
Well I mean it still is, Windows is being dumb and refusing to ipv6 right now for whatever reason.
plett: ok, thanks for checking - guess I'll wait some more and see how it goes
But that just leads to one more data point: my ipv6 connection is still up and working and connected. But comcast ipv4 has shit the bed for me too - ping, but not tcp
*gathering
forgotte1: answering your hilight from another channel - yes I'm alive
> But comcast ipv4 has shit the bed for me too - ping, but not tcp
same here
also, can't telnet underworld1.no.quakenet.org 6667 from ARP
(hooray for openvpn/udp still working)
also GRE
Good to hear - hadn't tested that yet
hmm... can't get to the IRC logger webpage
how long has this been happening? I've been out of town...
fwiw, comcast -> nlayer -> mzima ICMP
acf_: for the bast 6-ish hours
I remember trying to ssh in from my T-Mobile phone earlier in the week, and it not working...
but I could ssh to Amazon EC2, and then from Amazon to ARP
I'd offer mtr --tcp information, but on my FreeBSD box at home I get "bind(): No error: 0" and my OSX machine gives me "bind(): Undefined error: 0" so... sorry.
forward path to underworld1 is via Level3
forward path to Comcast is via Level3
From the reports, this seems very widely spread, so more likely an ARP router issue than a provider. But that's just my $.02 distilled from the IRC history.
Looks like the office TWC connection is getting through too
yeah, other Level3 things seem to work...
mtr --tcp and ICMP from Comcast: http://paste.debian.net/115125/
(The first reports were of TWC failing)
Comcast Business gets through
(dammit the only Linux box online at home is Debian, with mtr 0.82 that doesn't have --tcp)
hmm... nothing in backports?
acf_: Nope. https://packages.debian.org/wheezy/mtr-tiny
I remember coming across that missing too... but Debian is/seems very closed-off when it comes to some guy that wants to say "hey, can you ___ this?"
aww darn. I just run Debian testing on most stuff that's not critical. jessie has mtr 0.85
an mtr --tcp from arp to comcast and underworld1 is blank
blank? how odd.
ie, not hops
but comcast business shows via Level3 as usual
http://makeameme.org/media/created/how-odd.jpg
It no doubt goes without saying, but up_the_irons, please fix :)
comparison of Comcast business and residential paths: http://paste.debian.net/115137/
Unable to get to arpnetworks.com from comcast. traffic seems to be dropped after nlayer > mzima.
So this basically affects all customers at arp?
Im colo should that matter for some reason
has anybody noticed IPv6 destinations being affected?
I'm able to get to arpnetworks.com if I go through a server in Washington state (different ISP and different route)
for me prob seems to be between ge0-arpnet.cust.lax07.mzima.net (67.199.135.102) and arpnetworks..
I can confirm that from a user I have in australia
anyway.. enough whine for tonight.
Any idea who can help at mzima.net?
http://www.mzima.net gives 404. Was this up before?
mzima is part of nlayer
just read the IRC logs...
it's all good. up_the irons knows about this already
k, thanks
mzima no longer exists
but yeah, it is apart of GTT
i am able to get to everything from an earthlink connection in San Diego.
bummer
i know dont judge me :(
still going on?
m0unds: pretty sure ya.  im at work now and things are working, but diff isp.
all my graphing and stuff is fine, has been all night
via comcast in albuquerque
hola
anyone having issues connecting to arp from comcast?
qbit: \o
qbit: people be having issues from most major U.S. ISP's
it's just tcp traffix
o/
yeah - icmp is going fine
well good to know it isn't just me
fo sho :)
Hi guys.. looks like I'm not the only one with issues
I have clients in GA who can't connect to their stuff in ARP
http://dpaste.com/0C59HJN
traceroute makes it to the network fine
yeah, tcp is not going
only icmp
er udp
sexy
Just scrolled way up and read the chatter.. looks like it's a fun one
i discovered it at like 2am :P
I'm having some issues as well. first nagios alert came at 3am CST
Where can I get details of what's happened? I just joined the channel. Could someone copy the log to pastebin?
can you get to http://irclogger.arpnetworks.com/ from somewhere?
I can from the office TWC connection
but that might be over v6 too
yoberi: http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue
i didn't see an AAAA record for it
I can't seem to reach anything arpnetworks.com at the moment; yet my mobile phone can using the provider's data network
Indeed there isn't. Yep, I can reach it over v4. (from twc business)
acf: thanks!
appears to be a huge problem for a lot of folks.  Im sure up_the_irons is all over it.
truf
will we see a tweet from @arpnetworks or @bsdvps?
Good point/idea
I've had one of neil's employees also mention that
can't get to his computers from some place in europe
Is there an issue with some VPS Services and the website?
invader: http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue
acf_, Thank you
Go acf_ !
Well thats fucked up.
lol
lol
a wild up_the_irons appears :D
Still not fixed :(
Is something terrible happening right now? I can’t access the portal or my server.
Yes
What is it?
bmacs: bet you can ping it, just not tcp
yup
http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue https://twitter.com/arpnetworks/status/499235055323996160
bmacs: somewhere, something is not passing tcp traffic.  i see drops around nLayer, opened a ticket with them.  They gave me a response that made me want to scream (basically, "Why are you using IPv4 to test an IPv6 host?".  W.T.F.  our hosts are dual homed dumbass, it has NOTHING to do with the issue)
Okay, I jsut took a different route to my host and was able to access it.  ugh
yeah same here
is it possible I'm having trouble getting through level3 ?
erratic: are you seeing any drops around there?
yeah when i trace to google or linkedin
dropes at the 6th hop
erratic: so from your vps to google?
yes
I can get through to other sites like yahoo
seems to be back now lemme check
yep Im back
that was strange
wait, so it works now?
yes
up_the_irons: btw, here's an mtr from a working network, both TCP and ICMP. I noted that the arpnet.cust.mzima router shows up in the ICMP but not the TCP - odd? http://sprunge.us/dfOK
(I don't have a working mtr --tcp on a non-working network)
brycec: yeah, i'm seeing the same
I have no idea what it means though.
And the fact that it's a working connection suggests that'
s  just a red herring.
Whew; able to connect to my server from home, though not from work or cell phone.
Hooray
speaking of +o should someone announce the issue in the /topic too? (Thanks up_the_irons for tweeting it)
(This is in Sweden, BTW.)
any idea why a TCP traceroute would have zero hops?
I still don't understand the /topic-sized announcement. "Some networks are down and we do not yet know why"?
mike-burns: Just something that people would see when they /join and ask "is there a problem?"
gotta make some calls, brb
Something like "Upstream providers are dropping TCP traffic. We're working on it."
good luck, up_the_irons
Whew that's a long one.
definitely when I do an mtr --tcp -4 underworld1.no.quakenet.org zero hops are displayed
my arp metal box acts as a router, so
when I do the same mtr from a machine routed through it, one hop is displayed
hmm... looks like ARP routers don't show up in ICMP traceroutes anyway
\o/
*TCP tracerouts
this is kind of strange...
whoa
if I mtr --tcp -4 underworld1.no.quakenet.org, zero hops appear
I can see in the ICMP mtr that 4.71.143.105 (Level3) is the first hop outside of ARP
mtr --tcp -4 4.71.143.105 works fine
so TCP traffic destined for that host must have been dropped before it reached 4.71.143.105 ?
but TCP traffic directed to 4.71.143.105 is passed?
that host == underworld1.no.quakenet.org
hmm.  my mtr doesn't have --tcp
macports has only version 0.85 of mtr, it seems
that's the one I'm using
ahh - same as freebsd
but it doesn't have --tcp :)
hmm... mine definitely does
must be a fork
and sadly, without changing the version number to indicate that
it's the Debian Jessie mtr package
ahh - the github has --tcp
wow... --tcp was added to the docs in 2013
maybe it's not supported on freebsd?
maybe the code thinks that all the world is linux
which of course, fails on the BSDs and OSX
I noticed on FreeBSD's mtr, it simply craps out if I add "--tcp" (0 hops, wtf)
anyone have a Ubuntu 14.04 host on our network that I can run some mtr's on?
I'm always a version behind, so I don't have --tcp in my mtr
yeah
I also don't have a working mtr on a non-working host, lol
up_the_irons: Linux mtr does the zero hops thing too
I’ll make you a login
I did a test with it earlier ^
acf_: hmm weird
acf_: hey :) hru
oh hi erratic. I'm alright, yourself?
sitting here at home chilling and applying for some jobs
RandalSchwartz: does it work in europe
no sorry
works great in US and CA though
hey all.. any update? several clients beating down my path now :)
Beat back.
I think some of use at in the same boat.
invader obviously ;) Just curious if there was any update (just getting into the office)
If you ever dealt with Carriers its a bitch and a half. Give it time.
up_the_irons, I jsut sent you an ssh login to my box if you need to run mtr on it
invader I have, and am, just asking if there's an update
up_the_irons: Is shutting down BGP to nlayer an option, if they can't work out how to fix it?
Do we 100% know what Carriers are effected?
level3 is affected maybe?
seems the forward path to Comcast is via level3
pjs: no update yet
and tcpdump says TCP packets don't arrive via the forward path
plett: i shutdown all carriers in turn, last night, but no change :(
i'm putting together a very detailed report and going to send to NTT, Level 3, GTT, etc...
but TCP packets via the return path (nlayer) are fine
brycec: does ARP to TWC go via Level3?
Anything I can do from a carrier prospective?
acf_: last I tried, yes. Also ARP to Comcast
yeah. looking a lot like Level3 related then?
same for underworld1.no.quakenet.org
invader: which carrier perspective?
Is invader a carrier? Can we just blame you? :D
I work for a telco in WI. I multi homed to ntd twc and comcast.
lol
up_the_irons: Stupid question, because I'm sure you've already considered it, but are we sure it's not s7.lax having issues?
invader: oh nice, can you do a "mtr -4rwc 10 --tcp arpnetworks.com" ? :)
With a nick like invader, how can we not?
haha
brycec: well, s7.lax appears OK, but I could try to engineer it so we go out s1.lax instead.  but first, gotta finish sending this report to carriers...
Hmm not a damn thing.  Its like the route is not even there..
Let me connect into our Verizon part of the network
invader: so what does it show?  no hops at all?
nothing. Just verizon just works.
one min
up_the_irons: Best as I can tell, it's the single most common data point. And the fact that this TCP filtering would happen on multiple carriers, vs. a single router... As illogical as it seems, it's the most logical scenario from where I'm sitting.
i tend to agree
seems to me that Level3 is the only one affected
lol
acf_: but not universally...
brycec: s1.lax uptime is 5 years, 25 weeks, 22 hours, 4 minutes
brycec: could you point me to an example please?
acf_: look in the scrollback for my sprunge.us link. :p
acf_: In short, from work, which is TWC "Business Class" I have no issues to ARP over nlayer/mzima, and the return mtr (not included) comes over L3
level3 -> trit?
trit not there (gathering mtr now)
hmm yes... not all of level3 is affected
fwiw the level3 looking glass routes through gblx -> trit -> arp
acf_: http://sprunge.us/PXFD (ICMP because I have an old mtr on my VPS)
ok thanks
so it looks like some of Level3 isn't affected
but are there any non-Level3 routed destinations that are affected?
can't wget http://level3.com fwiw
acf_: from?
arp
ditto
up_the_irons: ^
(but I can from other networks
including an "affected" comcast connection
same
hmm
So, whether or not related to ARP's issue, (but it probably is), L3 has some split-braining going on
ok i blew away level3
lots of people tweeting about level3 problems
https://twitter.com/TVBroadcast/status/499243972221734912
TWITTER: Anyone aware of a "a large Internetnetwork issue is affecting ISPs across the country, like Comcast, Cogent, Level3, Time Warner, etc.?" (Tue Aug 12 17:20:13 +0000 2014, retweeted 1 times)
https://twitter.com/factor1/status/499249059433943040
TWITTER: Good news is that our network issue seems to be clearing up as Comcast and Level3 are responding to the fix. (Tue Aug 12 17:40:26 +0000 2014, retweeted 1 times)
up_the_irons: suddenly working
whoa
oh wait
nevermind, was v6
hate that
dualhoming biting me in the ass
Heh.
Yeah, all my users are complaining but v6 works just fine!
hehe :D
i've had v4 drop out in the office for hours at a time before I noticed
(dhcp issue)
Had someone try the mtr command but it produced no output. I don't have access to the box so I can't debug.
http://downdetector.com/status/level3
ok seems Comcast goes via ntt now
still no TCP packets going via Comcast -> ARP path
Shit this explains the issues I am having in SunGard.. They just called L3 is broken and has been for the better part of a day. -_- Thanks for the call now.. Butt heads.
mtr --tcp isn't working from my home FreeBSD box, trying to fix that so I can supply another data point
return path still ok
seeing the same problem from Comcast in NH though
IPv6 works, fwiw
up_the_irons: fwiw, curl level3.com from arp still not working, mtr shows it going ntt->level3, unable to ping
same here. seems dropping level3 didn't fix the problem
twobithacker: Yeah, my friends in MA can't connect via Comcast so probably the same route problem there.
brycec: yeah it's taking the exact same path, just through NTT now ;)
Whomp whomp.
(And return path ARP->Comcast is over NTT->Comcast)
yep
that's still broken too though
huh, I can connect just fine from Comcast Business though
So Level3 is having problems... and Verizon has been lashing out at L3. I bet Verizon haxxored Level3!!
^ sarcasm
twobithacker: yeah that's been observed
ironically, my Verizon DSL line is the only residential connection I've seen unaffacted
Seems home TWC and Comcast (among others) are affected, business customers are okay
I'm fine on TWC. :)
In NYC.
Also Verizon FiOS at my office is fine.
it's interesting that's it's just tcp
Yeah.
it seems to imply that somebody is discriminating tcp traffic in some way or another
Again, I blame the whole Netflix/Verizon debacle :P
"HOW DO YOU LIKE YOUR NETFLIX NOW, BITCHES?!?"
"YOU THOUGHT WE WERE THROTTLING? NOOOOOOW WE'RE THROTTLING!"
etc
lol
(except netflix.com is working for me)
looking at just the forward path ARP -> Comcast residential
(both v4 and v6)
it's broken
both with ntt and level3
Well I have other work to get done... Best of luck, #arpnetworks
see you later brycec
If it's just TCP, maybe some sort of traffic shaping failure of sorts?
I was thinking about that...
but where?
brycec: LOL
Yeah this is far more exciting than the admin tool I was just writing ...
s/ail/feat/
<mhoran> If it's just TCP, maybe some sort of traffic shaping ffeature of sorts?
hmm... traffic shaping, interesting..
http://www.secureworks.com/cyber-threat-intelligence/threats/bgp-hijacking-for-cryptocurrency-profit/
acf_: BryceBot mhoran erratic invader : see any change?
checking
first time using mtr
testing ARP -> Comcast path via NTT ... still broken for tcp
No dice.
ok
just talked with a friend on another connection
says he can't connect to underworld1.no.quakenet.org either
but it works from Comcast
i still find it weird that level3.com is inaccessible
yeah I haven't noticed anymore outage problems since earlier
was having trouble getting to cloudfront, linkedin, google, things like that
erratic: from what connection?
just my vpn tunnel
I have everything routed over my 206.125.168.65 addr
oh when i did traceroutes on stuff it was timing out at like 6 hops in level 3
then it just started working again about 5 minutes later
I wonder if theres a way I can continuously check with mtr
like if I wanted to watch for changes
I guess that wouldn't make a lot of sense now that I think about it
mtr polls continuously, and should show changes as they happen
weird sudo mtr --report -4 google.com
it just runs once and exits for me
probably the --report
ah
just shutdown Trit Networks BGP session, stab in the dark
you better fix all these things outside of your control up_the_irons
oh nice
lol
lol
acf thx
still appears broken for the Comcast thing
im writing a blog post as we speak bad mouthing your services
trit isn't in my comcast path fwiw
np erratic
yeah but sometimes... ya never know ;)
lol: https://twitter.com/ellisinzion/status/499255527990239232
TWITTER: @Level3 Yes you do have a major core IP outage right now. Stop saying there is no issue. Multiple big voice providers are down cold. (Tue Aug 12 18:06:08 +0000 2014, in reply to @Level3)
OL
LOL
ae-3-3505.ear1.Dallas1.Level3.net?
I totally just tweetd about this
neat
so... ARP -> Comcast forward path via either NTT or Level3 is broken
which kind of indicates that the issue doesn't lie entirely with NTT or Level3
acf_: But from .uk using NTT all the way to ARP works fine
but other things (TWC) are broken too
plett: but what is your return path
yeah, pretty much all the NTT stuff I've seen works fine
up_the_irons: I'll check
up_the_irons: NTT as well
ec2 (ntt both ways) is fine
acf_: What's the path in both directions for the broken arp <-> comcast ?
ARP -> Comcast is via NTT now, was Level3 earlier, both were broken
Comcast -> ARP is via nlayer
Ahh. nlayer
I've been testing just the ARP -> Comcast part of the path
the nlayer part looks fine
acf_: you never receive acks?
it's just TCP for everyone, yes?
I do a tcpdump on the comcast side
I try to connect from the arp side
no packets come through
UDP packets? or no packets
going the other way works though
I use tcpdump -ni eth3 port 6668
and telnet [hostname] 6668
hmmm...
HRMMM.
UDP packets are fine
any change just now?  i can actually check my email
yep
yup
packets are getting through to comcast now
Im up
up_the_irons, comcast works now
same with twc
comcast is symmetric nlayer now
the thing with NTT / Level3 is, a lot of NTT routes go through Level3 anyway
is it possible that something is dropping things with asymmetric paths?
so what i did was, lowered s7.lax default route local pref so most traffic goes out via s1.lax directly, not hitting s7.lax.  s7.lax has Level 3 and NTT.
acf_: that would take stateful filtering i believe and is pretty rare on regular transit links
wow. everything is via nlayer now :P
even any2ix stuff
nlayer to the rescue! (surprisingly)
yeah
not going through s7.lax kills the rest of my network ;)  (no peers, L3 or NTT)
oooo
while this works, it seems like a shitty solution
unless the problem really is with Level3, in which case we can wait it out now
I don't think it is... ARP -> NTT -> Comcast doesn't work...
acf_: still?
it's not NTT anymore, so not it works
oh wait.. couldn't be cuz ARP -> NTT isn't possible right now
* now it works
it didn't work both when it was ARP -> NTT -> Comcast and ARP -> Level3 -> Comcast
ok
well w..t.f
Im confuzled,
how does it kill the rest of your network
not that it matters just curious
this stuff is interesting to me
basically up_the irons took the s7 router out of the network
right
which is the connection between you and any2ix, ntt, and level3
I gathered that much does that mean none of the networks including mine will be reachable?
so without s7, all you get is nlayer
erratic: s7.lax is a bigger router and carries more routes.  usually, all traffic goes to it.  but now i'm diverting traffic to go directly out s1.lax.
it just means they'll take a diffetent path
and yeah, basically what acf_ said
oh right like if nlayer goes down...
i'm tempted to reboot s7
up_the_irons: so can you do testing with s7, static routes, etc... now that you've diverted traffic?
acf_: well, technically yes, but hard to test when no traffic is going to it ;)
just out of curiosity, does ios have a tcpdump equivilent?
now that there isn't any traffic, would it be more realistic to debug with tcpdump?
well, what i would do is use a SPAN port and then dump that port.  i have one set up already.
like switch port mirroring?
yup
that is handy
yup, purely meant for diagnostics
i can copy all traffic from any physical port or SVI (VLAN)
and then just watch it from a regular Linux box with tcpdump
Im not sure how much overhead that would be, probably too much
but
you could probably use it with snort
assuming the switch or whatever can handle it, without creating a single point of failure
yeah
ok, s7.lax is getting a kick in the butt (reboot)
sounds like a good idea :P
womp womp
I saw a momentary drop
I presume that was a shuffling?
....
wat
I lost talk.google.com for like a sec
ok
yeah... my connection went dead for about 10 seconds
yeah I think I lost everything for a secondlet
laggggggg
s7 reboot is done, i think i'm going to send traffic to it again and see what happens
whats the verdict? :D
Guilty.
invader: see ^
and the sentense?
looks like it's getting better
how's it looking for u guys now?
I need to consume this alcoholic beverage
erratic: alllllcohol
arp -> ntt -> comcast seems to work now
glug
so it was s7 being weird?
then i'm going to throw fucking s7 into the dumpster, gaah!
what sort of hardware is s7
up_the_irons: you just can't catch a break with that thing
srsly
but that's why shotguns exist
erratic: Cisco 6506 w/ sup720-3bxl
i've had a replacement Sup in the cage for months now :(
Im just curious, couldn't you use a linux box to do that
in the triage order of things, replacing it just wasn't high on the list.  but it is now.
up_the_irons: did you ever swap the sup, or is the one in the cage the one you got to replace it when it did that weird CEF barf thing?
that thing looks heavy
erratic: well you wouldn't get any hardware acceleration and the pps is really low in comparison.  so basically, a linux box would melt under a DDoS
m0unds: yeah it's the replacement for the other issue
i'm going to re-enable level 3 now
yeah Im reading the specs on it now lol
why just tcp?
man the ipv4 routing table is just getting too large :(
and *some* tcp too?
acf_: no. fucking. idea.  ;)
wow jeez
im sure if it's working for others, it's working for at&t too
i think i need to stop being married to the (now) old idea that carrying a full routing table is *that* much better
Sorry I broke the Internet
GUILTY!
on a slightly unrelated note, are you planning to announce to level3 at some point?
Webster:  we have a lot of 6500 ciscos and have recently grown up to be able to get the ASR 9000s
ooooooo
ugh, ios-xr
so,  my first cisco class starts monday.
and ends pretty much never if you take into account how esoteric it all is
haha
i hope you learn a lot about serial stuff, because that shit still exists in ios
haha
m0unds: just a sec
my vmware class also starts monday
that sounds like more fun
indeed
I'm only taking the cisco, cuz my job is making me.  I don't find routing or switching to be intersting at all.  I'd be far more interested in the security tract.
m0unds: this is how I decided to upgrade my bios http://3.bp.blogspot.com/-GDEA1qZ0thk/U-d-u4aUM8I/AAAAAAAAPTo/tgcCHYdR5c4/s1600/Screenshot+-+03272013+-+05:40:47+AM.png
just because lol y not
hahahahah
nice
m0unds: hey i use serial all the time, the 2612's are great for out-of-band access to everything :)  I have 32 serial ports with breakout cables on one of 'em.
oh, i meant like frame relay serial :)
oh
lol
i have an AS2516-RJ (iirc) that takes rollover cables terminated w/rj45s
it's great as a terminal server for my crisco lab
jpalmer: I have pretty much my lack of interest in anything except computers but more importantly the people I've surrounded myself with who have been willing to help to thank
I got kicked out of school
i got my employer to approve me bringing it to work so i wouldn't have to try and store all that gear at home :)
haha yeah I love piling up my crap at work
makes me feel comfortable to have crap around me
i racked mine up and told my coworkers to learn something
m0unds: definitely
yeah I used so much of my own hardware at work
since i'm hoping i'll be able to leave here in a year or so, and since they don't have another network engineer, they'll be utterly SOL if they don't
well at my last good job
I haven't had a decent job in a while
finally starting to get interested in stuff again though
m0unds: I do the same thing.   and I teach them how to make and restore backups first..  so that if someone changes it when they are done, they can instantly revert it back to their last configuration.  gives them a LOT of backup/restore practice,  and makes sure everyone can use the lab without stepping on other peoples toes.
Ive had crap jobs and a few short term contracts and stuff but nothing really
my last job really spoiled my appetite for work
was laid off a year ago today and Im just now like, ok I'm moving to a new place and I don't know anybody and I have no idea what to expect and I think it will be challenging for me socially
erratic: where are you moving to?
Netherlands
Im in Greece atm
thats..  a move, for sure.  hehe
ah, cool - my grandparents are from the netherlands
my wife and I are talking about moving from florida to charlotte, NC
NC isn't too bad
charlotte is nice - my dad lives in burlington, just outside greensboro
need a car for sure
I like living in the city too much
but I do appreciate living out in the country and I miss it sometimes
we'd like in one of the outskirt areas, for sure.   horses and such
s/like/live/
<jpalmer> we'd live in one of the outskirt areas, for sure.   horses and such
I lived in creedmoor outside of RD
for awhile
was like 16 or 17
long enough to make friends then move away again
I really want to believe despite how much i hate redhat it would be really hard (definitely competitive) to work there
but probably pretty interesting
especially if you can get a good systems engineering job
they pretty much birthed xen and now from what I've been reading a lot of the work on qemu/kvm looks pretty sweet
and I'm running a lot of the more recent stuff
actually found a vid this morning exactly what I have except the guy has much faster hardware
https://www.youtube.com/watch?v=37D2bRsthfI
YouTube Tech: "Qemu/KVM + vfio = Virtual machine for gaming with 95% of native performance = no more dual booting" by blu3bird84 (2m 8s), 18,965 views, 222 likes and 4 dislikes. Uploaded 2014-02-02T19:42:29.000Z.
anyone seen any issues in the last 30 mins?
things are working for me...
wish i could test :(
no issues here.  been clear since the reboot
roger that
Yep. No issues here..
any issues with kvm 14 today?
krv14, that is
can someone put the topic back to the way it was :)
i tried connecting several times a couple of hours ago without success
seems the NTT comcast stuff is finally getting better
mus1cb0x: was probably the general network issue we had
just looked at my arp latency graph, and it's looking much healthier than a month ago
ah, i see the topic
how long was the outage?
since like 1AM, but it was very selective on certain routes.  Cacti showed most traffic was getting in/out.
ouch, what was the cause?
Looks good here.
mus1cb0x: s7.lax being stupid
(Had to get on a desktop for that one.)
stupid?
aka not smart
sarcasm isn't helpful
nor is "stupid" when discussing network outages
i have to justify the outage and i'm not going to use stupid and risk my job
this isn't an offical forum
go put in a ticket
heh, geez
at least you're not op'd, because you would be the wrong person to represent arp
play nice now
up_the_irons: It does seem like there were more widespread issues than just with your router, though? I definitely experienced issues with non-ARP connectivity this morning.
Perhaps all related in some way?
e.g. a crazy routing table causing issues on routers across the nets?
mus1cb0x: There was something strange with TCP traffic not being passed through to certain destinations over s7.lax.  UDP was fine.
I guess the fact that it was TCP only kind of makes my thought make no sense.
Nor does that sentence make sense.
Huzzah Internet working!
mhoran: hmm.. that's a good point about the routing tables, maybe it is all related in some way
oh boy, I missed a lot
The TCP thing really sounds like someone played with acls.
mhoran: I didn't realize you had non-ARP connectivity problems too
did anyone else see the article about routing table hijacks lately to commandeer bitcoin mining?
up_the_irons: Yeah I was having trouble with some things before I left from home this AM, over IPv4 from Time Warner.
mus1cb0x: someone pasted that, yeah
mhoran: hmm... interesting
up_the_irons, L3 had an confirmed issue was a few asn's/
any chance that could be a cause?
some type of BGP poisoning?  I suppose it's possible
http://www.reddit.com/r/cryptodev/comments/2d0bzx/bgp_hijacking_for_cryptocurrency_profit/
up_the_irons: we hit 512,000 active routes today,  were you aware of that?
up_the_irons, From the sounds of it, They only said "route issue" Getting a straight answer out of them is like try to date Misty Dawn.
jpalmer: lol, someone just PM'd me, "related to the 512k routes issue?"
perhaps that was it...
i'm showing 499K right now
i'm going to be doing a route haircut tonight
maybe kill all /24's
up_the_irons: lots of routers with the 512k limit have been having issues today.
sup 720 should handle 1m routes, iirc
Happy grooming
might be thinking of the 2t though
danke
Sure should.
what kind of gear you guys using at your edge?
m0unds: jpalmer : i have the -3bxl, so it does 1M routes.  however, i have two full tables, so right about there
ohhhh
gotcha
CEF is actually bitching at me in the logs...  lol
haha
i hate cef
such a kludgy thing
I could see CEF blowing up and causing all sorts of weird issues, in addition to the missing routes.
Still don't understand why UDP would work, unless it took a different path somehow.
yea, but it usually fails by dumping forwarding onto the cpu
then the cpu shits its pants and the whole chassis stops passing traffic because of the load
or crashes, depending
haha
http://status.livestream.com/ -- so it looks like my ex startup had routing troubles today, too. :p
All the competent network techs left, so I'm not surprised their router crashed.
Was also a 720-3bxl.
That I doubt anyone has maintained in two years.
well that was easy:
s7.lax(config)#mls cef maximum-routes ip 768
Maximum routes set to 786432. Configuration will be effective on reboot.
looks like another reboot will happen in the very near future... (tonight maybe)
;)
do it now while only half of people think its fixed!
:D
hahahhah
nice:)
boo cef, boo
and yeah, CEF blowing up means the router falls back to software switching, which will melt it
yeah, i wonder if cef was loaded and it was unable to take on any flows
actually s1.lax did that once, i hit the 128K limit.  it went full on to CPU switching.  some of you may remember that outage.  but hey, it didn't crash, still 5+ year uptime! :)
hahaha
s1's a 45xx right?
yup
up_the_irons: so it was hitting the 512k limit?
my buddy @ easystreet up in oregon had a similar thing w/a 45xx killing CEF and dying horribly overnight one night
jpalmer: i don't think so, or else *nothing* would have gotten through
Yeah, that's happened to me too, when I foolishly tried to load the entire routing table on a 4500. :p
Upstream route filter was misconfigured and I didn't guard against that.
ouch
preemptive optimization doesn't exist in networking :P
er, premature
Heh.
http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-major-outage-across-us-network-providers
http://www.cidr-report.org/as2.0/#General_Status
Number of ASes seems to be approaching a number that someone may have hardcoded into a system, too. :p
yeah, haha
I heard a rumor of a bgp apocalypse and I figured #arpnetworks would tell me whether or not to panic
bgpocalypse
wow.  ziprecruiter.com has gotten about 5 times faster in the past few days for job searches.
my efforts are paying off.
m0unds: sjackso : lol
so much fuss for so little
and of course media blows it up
hola
:D
so - are the weird issues fixed?
from earlier
alive!
i seem to be having some dns issues - not sure if it's residual or not
could also be from the upgrade i stupidly decided to do mid day :P
aha
host can resolve things fine.. but ping doesn't
making me rage
qbit: connectivity was resolved about 7 hours ago
k
it's totally fucked up - ping doesn't resolve some hosts.. chat.freenode.net.. etc - but it does for others
and host works for all
qbit: reboot?
Seems like the host dns resolver cache is fubar'd
but host does a fresh fetch iirc
or at least a direct connection, bypassing the normal caching
yeah - reboot does nothin to fix it
RandalSchwartz: i meant, are you up to freebsd version 9 or 10
when did you "meant"?
I'm still on 8.4
probably 10.x during some spare downtime in december
after I first do it on insightcruises.com machines. :)
fink?
yo RandalSchwartz
was answering your questions
i said: fink: yo RandalSchwartz, how's it hanging. are you up to 9 or 10 with your vps?
never saw that
or maybe long ago
yea, yesterday ;)
so there's my answers
i'm wary of moving my zfs on root from 8.4 to 10
Sounds exciting!
yeah - there's definitely some possible issues there
which is why I want to try it on $client's computers first :)
there's some settings for internet in /etc/rc.conf that change, I'm told
and if I get those wrong, I don't get back in to the box :)
Yup, there's that.
Though it's not too crazy.
RandalSchwartz: i'm actually somewhat curious how much business/interest there is in insightful holiday packages
i never really thought of it before
https://supportforums.cisco.com/document/12202206/size-internet-global-routing-table-and-its-potential-side-effects lol
http://bgr.com/2014/08/12/comcast-fcc-commissioner-clyburn-dinner/ lol
curious in what way?
these aren't holiday packages
if you'd spend 4 days at a hotel in $random_city to attend a conf, why not 7 days on a cruise ship to do the same thing, but with better speakers?
i dunno, i just never thought of that
i assumed most techies would rather book their own stuff and explore like a local or maybe that's just me
yeah, most people haven't, which is why Captain Neil makes some bucks
forgot about the conf part
primarily, these are conferences
they just happen to be meeting on a ship
instead of a hotel
lots of advantages to that
right
all of my industry conferences are in vegas
there's one that was in san diego once
but never again
and one big plus... the speakers are in the same BARS and same DINNER as you are
unlike land conferences, where they all scatter.
and our speakers are expected to mingle with the attendees at those.
imagine the conferences you've attended where you could bump into one of the keynote speakers later at the bar...
or even on a shore excursion... a few hours in the same small group with them
so you have people actually interested in the subject there as opposed to forced to attend by corporate
geez... you must have a twisted view of what we do!
I don't even know how to answer that.
"I am interested in X"
"Oh look... a cruise where speakers will be talking about X"
"I want to go"
[if employed] "Employer, can you sponsor me to this conference?"
[if retired] "Let's see if my travel budget covers this"
That's how people get on
Nobody is *forced*
"Oh gawd... FORCED to go on a CRUISE!" :)
see how silly that sounds now?
i think hazardous was just pointing out that people attending a conf on a cruise are more likely to be going because they want to and not because someone forced them to
at least that's how it read to me
what was that command to run an mtr report with output to a text format for easy copy /paste
have the upstream TCP transit issues been resolved?
I was seeing intermittant failures earlier tonight but they didn’t last long
mtr -4 --tcp -rc 10 <your-IP>
mnathani: add -w so it doesn't cut off hostnames (wide format)