***: sga0 has quit IRC (Ping timeout: 246 seconds)
forgotte1 has joined #arpnetworks forgotte1: arpnetworks down??
nvm up_the_irons: what happened? forgotte1: it seams at&t have blocked my connection to my own sites / services
uverse
well and arpnetworks.com entirely
those dirty sluts man!!
up_the_irons: what is arpnetworks ipv4 range? i wanna make sure they unblock it for everyone with at&t service in case others are having the same problem. People could be loosing potential business if no one on at&t uverse can access any sites hosted at arp
not sure it's that wide spread, but can't hurt i dont think pyvpx: hmm, you can find out using IRR
if he's been good about his object maintenance ;) up_the_irons: my objects are up to date
it's weird, I'm on twc, i can ping everything in my cage, but SSH, http, etc.. seems to be all blocked. started about an hour ago. I'm still in irc only because I'm using weechat android from my phone pyvpx: thats fucked forgotte1: oh wow....
so it's not just at&t...
same symptoms here, can ping everything. but ssh, http, https all blocked
lol this at&t rep wants me to reset my modem to factory. now i feel like an ass :P pyvpx: up_the_irons: can I try anything for you from Germany? ;) up_the_irons: pyvpx : how about just arpnetworks. com? forgotte1: wyliebayes.com , nadasound.com
2 of mine pyvpx: pings and loads just fine
I'm coming from 37.120.2.102 up_the_irons: overall, my bandwidth graphs for the big pipes look pretty normal
so most traffic is still getting out
so weird about twc and att forgotte1: it's only residential uverse att tho. i can access via my att 4g network no problem pyvpx: what are the blocks you guys are coming from that can't reach arp? -: pyvpx looks to BGP forgotte1: i am, 99-95-174-245.lightspeed.sndgca.sbcglobal.net
pyvpx: would be weird bgp would effect all services, except ICMP pyvpx: oh, I meant from a policy stand point
localize to a router or a routing event that could point to a config change
it's easier to hit someone on the head at att when you are like
"yeah at 4am this shit started"
they go "wait we had a thing at 4am uhhhhh"
...sometimes forgotte1: i would be surprised if the "tech" im talking to in chat can even feed himself successfully
from att pyvpx: sometimes if you play the act
the ip noc folks can entertain you
give you ticket numbers or other depts to call -: pyvpx was never an ATT IP customer forgotte1: ya im trying, had to DEMAND i speak to his supervisor cause he tried to pawn it off on the hosting provider hehe
by pasting me out of the book answers
if resetting my modem didn't fix the problem, we have exhausted all possible solutions. LOL pyvpx: hmm
the denver router on their looking glass had a thing 42 minutes ago
probably nothing
everything seen through verio forgotte1: hrmm
and since icmp works it's hard to see where it might be failing
i am in San Diego CA ***: thestereobus has joined #arpnetworks
thestereobus has quit IRC (Ping timeout: 244 seconds)
thestereobus has joined #arpnetworks up_the_irons: I'm playing around with shutting down certain transit sessions, to see if any of them are the culprit. nothing successful so far
it's like only tcp doesn't work. i can still connect to my VPN (udp) and dns still works fine... wtf thestereobus: yeah, I’m connected over VPN but TCP is down pyvpx: so tcp on att uverse and twc?! forgotte1: weeeeird thestereobus: TWC here forgotte1: my vpn runs on tcp :( pyvpx: what do those have in common forgotte1: that we know of, only att and twc thestereobus: I can’t ping or SSH into the server even over the VPN but I can connect via the iKVM console pyvpx: ikvm console on different ip ranges? -: pyvpx isn't exactly a customer... forgotte1: thx for the assist anyway pyvpx :) up_the_irons: man this is weird thestereobus: yup, UDP works pyvpx: up_the_irons: yeah I'm kind of at a loss as to what I'd do up_the_irons: yeah forgotte1: i always find the weird shit KDE_Perry: happening for me from comcast too forgotte1: shiz pyvpx: comcast as well?
up_the_irons: you dont see these networks over any2, do you?
fucking up something like all tcp traffic sounds like a coresite thing...on its face :p up_the_irons: no, I'm small peanuts to them, they would not peer with me over any2 forgotte1: i can connect via ssh from devio.us
not exactly sure where that is hosted up_the_irons: if you guys do "mtr --tcp arpnetworks. com ", where does it stop? for me it is around nlayer KDE_Perry: same here forgotte1: dimenoc, somewhere in FL
unfortunately mtr wants to crash on me today thestereobus: I’m conneted to a remote machine at 162.248.100.99
I can’t access the arpnetworks via TCP
but the mtr command completes fine
nmap -Pn -p22 -sU arpnetworks.com works up_the_irons: even with --tcp? thestereobus: yes up_the_irons: weird, doesn't finish for me thestereobus: and all my server monitors at nodeping.com have failed as well forgotte1: nmap comes back as closed for me, for 22. when it should be open
for my own site
shows filtered for arpnetworks.com thestereobus: no packet loss and 30-40ms pings using mtr so it doesn’t seem like it’s bandwidth
actually I have no problems contacting arpnetworks.com at all from that server - just my own arpnetworks IP up_the_irons: I'm trying from several nlnog ring nodes and everything works so far thestereobus: so mtr —tcp arpnetworks.com works mtr —tcp danconnor.com does not (failing at ge0-arpnet.cust.lax07.mzima.net) up_the_irons: thestereobus : what's your server ip? thestereobus: 206.125.168.2 up_the_irons: oh! that i know.. you're leaking traffic between your two uplinks and stp is shutting down your interface (completely unrelated issue) forgotte1: i am 206.125.175.0/29 up_the_irons: i can re enable...
done thestereobus: ah, ok. is that related to link aggregation?
I enabled that recently up_the_irons: most likely, yes thestereobus: ok. I enabled that recently
I’ll turn it off for now
carry on up_the_irons: cool :)
lol, finally back in weechat on my laptop (instead of phone). i just relayed through an nlnog node to my irc box..
sucks that i have to do that, but....
works for now forgotte1: my irssi session is still working fine, from my box to freenode lol
just can't get into my box ***: thestereobus_ has joined #arpnetworks
thestereobus has quit IRC (Ping timeout: 255 seconds)
thestereobus_ is now known as thestereobus up_the_irons: i have a feeling that this is a GTT/nLayer issue, so i just emailed their NOC forgotte1: word
for me tcptraceroute gets to: 7 ae-5.r04.lsanca03.us.bb.gin.ntt.net (129.250.2.220) 28.686 ms 27.857 ms 28.676 ms
then dies
on udp next hop is: 7 ae-5.r04.lsanca03.us.bb.gin.ntt.net (129.250.2.220) 29.388 ms 29.274 ms 27.852 ms
err
8 ge-0-7-0-24.r04.lsanca03.us.ce.gin.ntt.net (129.250.198.186) 29.820 ms 29.341 ms 29.662 ms
up_the_irons: they are answering at the noc number.... but wont help me lol
are you a direct customer of us.ntt.net ? jbergstroem: so confirmed network issues i guess?
im getting emails from people all around notion: from here. it looks like the tcp reply never get backs to the client - server get my http request, and sends back reply - just don't reach the client jbergstroem: some of the users i've been in touch with says that icmp ping doesn't reply either
have yet to find a rig i control that can reproduce pyvpx: yeah, something verio
I vaguely recall them having some mysterious IDS thing that did things like this occasionally
but that was years ago
but nlayer/gtt has used verio for ages. -: pyvpx has more beer than technical details
pyvpx opens another one notion: yeah, seems wicked atm. oh well, i'll wait some more - not working from .dk sites, but works from .de site
enjoy the beer ;D forgotte1: =[06:28] *** raptelan_ has joined #arpnetworks raptelan_: is Arp having some networking problems currently?
I can ping my VPS, but SSH attempts time out, also cannot load arpnetworks.com in a browser. forgotte1: yes they are
it appears only TCP traffic is not getting through, from pretty much every ISP in U.S. pyvpx: well, more like the more notable US access networks
and a non-obvious number of european networks raptelan_: ok, I guess I don't have to go strangle somebody in our IT department then :P plett: pyvpx: Only adding one data point, but ssh to my arp vps works fine for me in .uk (traffic goes via NTT for both v4 and v6) notion: UK seems to have hit something magic - http://downdetector.co.uk/problems/ee-everything-everywhere/map/ ***: fink has joined #arpnetworks pyvpx: works fine from .de
well, my slice. notion: BT outage chart dont look good :(
http://downdetector.co.uk/problems/bt-british-telecom/map plett: notion: I wouldn't put too much trust (i.e. any at all) in downdetector notion: plett: ok dne: no issues from .se either notion: plett: my trafik through the UK link fails, but works perfect from my DE plett: notion: From where to where? hazardous: i'd trust that BT is terrible though plett: Naturally, they are a defacto monopoly telco notion: plett: UK link from: 2.111.95.123 to: 174.136.105.26 plett: notion: That's from .dk to arp? Does that normally go through .uk for you? notion: plett: yes jcv: my vps networking seems fine
in fact the irssi instance i'm typing in now is running on it jbergstroem: I still get reports from users having issues plett: notion: Does a traceroute for that go through nlayer or GTT? up_the_irons said it might be a problem with them jbergstroem: mostly "is the site down" notion: plett: not sure who's who on the path - my traceroute: http://pastebin.com/754FDTGx plett: notion: And ssh doesn't work for you? notion: plett: ssh does not work
plett: or http ***: acf_ has joined #arpnetworks notion: plett: ICMP does tho, odd plett: notion: I can't connect to ssh on 174.136.105.26 (I'm probably firewalled), but HTTP is fine for me. My path looks very similar to yours http://pastebin.com/4Heg7P6B brycec: And here I was thinking it was Windows being dumb...
Well I mean it still is, Windows is being dumb and refusing to ipv6 right now for whatever reason. notion: plett: ok, thanks for checking - guess I'll wait some more and see how it goes brycec: But that just leads to one more data point: my ipv6 connection is still up and working and connected. But comcast ipv4 has shit the bed for me too - ping, but not tcp -: brycec continues gathring data... brycec: *gathering
forgotte1: answering your hilight from another channel - yes I'm alive acf_: > But comcast ipv4 has shit the bed for me too - ping, but not tcp
same here
also, can't telnet underworld1.no.quakenet.org 6667 from ARP brycec: (hooray for openvpn/udp still working) acf_: also GRE brycec: Good to hear - hadn't tested that yet acf_: hmm... can't get to the IRC logger webpage
how long has this been happening? I've been out of town... brycec: fwiw, comcast -> nlayer -> mzima ICMP
acf_: for the bast 6-ish hours acf_: I remember trying to ssh in from my T-Mobile phone earlier in the week, and it not working...
but I could ssh to Amazon EC2, and then from Amazon to ARP brycec: I'd offer mtr --tcp information, but on my FreeBSD box at home I get "bind(): No error: 0" and my OSX machine gives me "bind(): Undefined error: 0" so... sorry. acf_: forward path to underworld1 is via Level3
forward path to Comcast is via Level3 brycec: From the reports, this seems very widely spread, so more likely an ARP router issue than a provider. But that's just my $.02 distilled from the IRC history.
Looks like the office TWC connection is getting through too acf_: yeah, other Level3 things seem to work...
mtr --tcp and ICMP from Comcast: http://paste.debian.net/115125/ brycec: (The first reports were of TWC failing) acf_: Comcast Business gets through brycec: (dammit the only Linux box online at home is Debian, with mtr 0.82 that doesn't have --tcp) acf_: hmm... nothing in backports? brycec: acf_: Nope. https://packages.debian.org/wheezy/mtr-tiny
I remember coming across that missing too... but Debian is/seems very closed-off when it comes to some guy that wants to say "hey, can you ___ this?" acf_: aww darn. I just run Debian testing on most stuff that's not critical. jessie has mtr 0.85
an mtr --tcp from arp to comcast and underworld1 is blank brycec: blank? how odd. acf_: ie, not hops
but comcast business shows via Level3 as usual raptelan_: http://makeameme.org/media/created/how-odd.jpg brycec: It no doubt goes without saying, but up_the_irons, please fix :) ***: ess has joined #arpnetworks acf_: comparison of Comcast business and residential paths: http://paste.debian.net/115137/ ess: Unable to get to arpnetworks.com from comcast. traffic seems to be dropped after nlayer > mzima. jbergstroem: So this basically affects all customers at arp?
Im colo should that matter for some reason acf_: has anybody noticed IPv6 destinations being affected? ess: I'm able to get to arpnetworks.com if I go through a server in Washington state (different ISP and different route)
for me prob seems to be between ge0-arpnet.cust.lax07.mzima.net (67.199.135.102) and arpnetworks.. jbergstroem: I can confirm that from a user I have in australia
anyway.. enough whine for tonight. ess: Any idea who can help at mzima.net?
http://www.mzima.net gives 404. Was this up before? acf_: mzima is part of nlayer
just read the IRC logs...
it's all good. up_the irons knows about this already ess: k, thanks ***: fink has quit IRC (Quit: fink)
dj_goku has joined #arpnetworks
dj_goku_ has quit IRC (Read error: Connection reset by peer)
forgotten has joined #arpnetworks pyvpx: mzima no longer exists -: pyvpx pours one out pyvpx: but yeah, it is apart of GTT forgotten: i am able to get to everything from an earthlink connection in San Diego. m0unds: bummer forgotte1: i know dont judge me :( m0unds: still going on? forgotte1: m0unds: pretty sure ya. im at work now and things are working, but diff isp. m0unds: all my graphing and stuff is fine, has been all night
via comcast in albuquerque -: m0unds shrugs qbit: hola
anyone having issues connecting to arp from comcast? forgotte1: qbit: o
qbit: people be having issues from most major U.S. ISP's
it's just tcp traffix qbit: o/
yeah - icmp is going fine
well good to know it isn't just me forgotte1: fo sho :) ***: yoberi has joined #arpnetworks pjs: Hi guys.. looks like I'm not the only one with issues
I have clients in GA who can't connect to their stuff in ARP
http://dpaste.com/0C59HJN
traceroute makes it to the network fine qbit: yeah, tcp is not going
only icmp
er udp pjs: sexy
Just scrolled way up and read the chatter.. looks like it's a fun one forgotte1: i discovered it at like 2am :P yoberi: I'm having some issues as well. first nagios alert came at 3am CST
Where can I get details of what's happened? I just joined the channel. Could someone copy the log to pastebin? acf_: can you get to http://irclogger.arpnetworks.com/ from somewhere? ***: milki_ has joined #arpnetworks brycec: I can from the office TWC connection
but that might be over v6 too acf_: yoberi: http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue m0unds: i didn't see an AAAA record for it yoberi: I can't seem to reach anything arpnetworks.com at the moment; yet my mobile phone can using the provider's data network brycec: Indeed there isn't. Yep, I can reach it over v4. (from twc business) yoberi: acf: thanks! forgotten: appears to be a huge problem for a lot of folks. Im sure up_the_irons is all over it. qbit: truf milki_: will we see a tweet from @arpnetworks or @bsdvps? brycec: Good point/idea RandalSchwartz: I've had one of neil's employees also mention that
can't get to his computers from some place in europe ***: thestereobus_ has joined #arpnetworks
milki_ is now known as milki^
thestereobus_ has quit IRC (Client Quit)
thestereobus has quit IRC (Ping timeout: 264 seconds)
befrank has joined #arpnetworks
invader has joined #arpnetworks invader: Is there an issue with some VPS Services and the website? acf_: invader: http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue invader: acf_, Thank you brycec: Go acf_ ! invader: Well thats fucked up. forgotten: lol up_the_irons: lol brycec: a wild up_the_irons appears :D
Still not fixed :( ***: bmacs has joined #arpnetworks bmacs: Is something terrible happening right now? I can’t access the portal or my server. brycec: Yes bmacs: What is it? up_the_irons: bmacs: bet you can ping it, just not tcp bmacs: yup brycec: http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue https://twitter.com/arpnetworks/status/499235055323996160 up_the_irons: bmacs: somewhere, something is not passing tcp traffic. i see drops around nLayer, opened a ticket with them. They gave me a response that made me want to scream (basically, "Why are you using IPv4 to test an IPv6 host?". W.T.F. our hosts are dual homed dumbass, it has NOTHING to do with the issue) bmacs: Okay, I jsut took a different route to my host and was able to access it. ugh up_the_irons: yeah same here erratic: is it possible I'm having trouble getting through level3 ? up_the_irons: erratic: are you seeing any drops around there? erratic: yeah when i trace to google or linkedin
dropes at the 6th hop up_the_irons: erratic: so from your vps to google? erratic: yes
I can get through to other sites like yahoo
seems to be back now lemme check
yep Im back
that was strange up_the_irons: wait, so it works now? erratic: yes brycec: up_the_irons: btw, here's an mtr from a working network, both TCP and ICMP. I noted that the arpnet.cust.mzima router shows up in the ICMP but not the TCP - odd? http://sprunge.us/dfOK
(I don't have a working mtr --tcp on a non-working network) up_the_irons: brycec: yeah, i'm seeing the same brycec: I have no idea what it means though.
And the fact that it's a working connection suggests that'
s just a red herring. mike-burns: Whew; able to connect to my server from home, though not from work or cell phone. brycec: Hooray
speaking of +o should someone announce the issue in the /topic too? (Thanks up_the_irons for tweeting it) mike-burns: (This is in Sweden, BTW.) acf_: any idea why a TCP traceroute would have zero hops? mike-burns: I still don't understand the /topic-sized announcement. "Some networks are down and we do not yet know why"? brycec: mike-burns: Just something that people would see when they /join and ask "is there a problem?" up_the_irons: gotta make some calls, brb brycec: Something like "Upstream providers are dropping TCP traffic. We're working on it."
good luck, up_the_irons ***: mike-burns changes topic to: Upstream providers are dropping TCP traffic. We're working on it. | VPS Services: http://www.arpnetworks.com/vps | FAQ: http://support.arpnetworks.com/faqs | Wiki: http://wiki.arpnetworks.com | Twitter: http://twitter.com/bsdvps | @up_the_irons is staff, other ops are long-time customers | Channel Log: http://is.gd/eWA5u mike-burns: Whew that's a long one. acf_: definitely when I do an mtr --tcp -4 underworld1.no.quakenet.org zero hops are displayed
my arp metal box acts as a router, so
when I do the same mtr from a machine routed through it, one hop is displayed
hmm... looks like ARP routers don't show up in ICMP traceroutes anyway brycec: o/ acf_: *TCP tracerouts
this is kind of strange... qbit: whoa acf_: if I mtr --tcp -4 underworld1.no.quakenet.org, zero hops appear
I can see in the ICMP mtr that 4.71.143.105 (Level3) is the first hop outside of ARP
mtr --tcp -4 4.71.143.105 works fine
so TCP traffic destined for that host must have been dropped before it reached 4.71.143.105 ?
but TCP traffic directed to 4.71.143.105 is passed?
that host == underworld1.no.quakenet.org -: brycec has no idea RandalSchwartz: hmm. my mtr doesn't have --tcp
macports has only version 0.85 of mtr, it seems acf_: that's the one I'm using RandalSchwartz: ahh - same as freebsd
but it doesn't have --tcp :) acf_: hmm... mine definitely does RandalSchwartz: must be a fork
and sadly, without changing the version number to indicate that acf_: it's the Debian Jessie mtr package RandalSchwartz: ahh - the github has --tcp
wow... --tcp was added to the docs in 2013 acf_: maybe it's not supported on freebsd? RandalSchwartz: maybe the code thinks that all the world is linux
which of course, fails on the BSDs and OSX up_the_irons: I noticed on FreeBSD's mtr, it simply craps out if I add "--tcp" (0 hops, wtf)
anyone have a Ubuntu 14.04 host on our network that I can run some mtr's on?
I'm always a version behind, so I don't have --tcp in my mtr ***: thestereobus has joined #arpnetworks bmacs: yeah up_the_irons: I also don't have a working mtr on a non-working host, lol acf_: up_the_irons: Linux mtr does the zero hops thing too bmacs: I’ll make you a login acf_: I did a test with it earlier ^ up_the_irons: acf_: hmm weird erratic: acf_: hey :) hru acf_: oh hi erratic. I'm alright, yourself? erratic: sitting here at home chilling and applying for some jobs -: RandalSchwartz mumbles "ziprecruiter.com" ) erratic: RandalSchwartz: does it work in europe RandalSchwartz: no sorry
works great in US and CA though pjs: hey all.. any update? several clients beating down my path now :) invader: Beat back.
I think some of use at in the same boat. pjs: invader obviously ;) Just curious if there was any update (just getting into the office) invader: If you ever dealt with Carriers its a bitch and a half. Give it time. bmacs: up_the_irons, I jsut sent you an ssh login to my box if you need to run mtr on it pjs: invader I have, and am, just asking if there's an update plett: up_the_irons: Is shutting down BGP to nlayer an option, if they can't work out how to fix it? invader: Do we 100% know what Carriers are effected? acf_: level3 is affected maybe?
seems the forward path to Comcast is via level3 up_the_irons: pjs: no update yet acf_: and tcpdump says TCP packets don't arrive via the forward path up_the_irons: plett: i shutdown all carriers in turn, last night, but no change :(
i'm putting together a very detailed report and going to send to NTT, Level 3, GTT, etc... acf_: but TCP packets via the return path (nlayer) are fine
brycec: does ARP to TWC go via Level3? invader: Anything I can do from a carrier prospective? brycec: acf_: last I tried, yes. Also ARP to Comcast acf_: yeah. looking a lot like Level3 related then?
same for underworld1.no.quakenet.org up_the_irons: invader: which carrier perspective? brycec: Is invader a carrier? Can we just blame you? :D invader: I work for a telco in WI. I multi homed to ntd twc and comcast. -: brycec blames invader up_the_irons: lol -: invader crys
forgotten 2nds the vote to blame invader brycec: up_the_irons: Stupid question, because I'm sure you've already considered it, but are we sure it's not s7.lax having issues? up_the_irons: invader: oh nice, can you do a "mtr -4rwc 10 --tcp arpnetworks.com" ? :) brycec: With a nick like invader, how can we not? up_the_irons: haha ***: thestereobus has quit IRC (Quit: thestereobus) up_the_irons: brycec: well, s7.lax appears OK, but I could try to engineer it so we go out s1.lax instead. but first, gotta finish sending this report to carriers... invader: Hmm not a damn thing. Its like the route is not even there..
Let me connect into our Verizon part of the network up_the_irons: invader: so what does it show? no hops at all? invader: nothing. Just verizon just works.
one min brycec: up_the_irons: Best as I can tell, it's the single most common data point. And the fact that this TCP filtering would happen on multiple carriers, vs. a single router... As illogical as it seems, it's the most logical scenario from where I'm sitting. up_the_irons: i tend to agree acf_: seems to me that Level3 is the only one affected -: brycec wonders what s1's uptime is up_the_irons: lol brycec: acf_: but not universally... up_the_irons: brycec: s1.lax uptime is 5 years, 25 weeks, 22 hours, 4 minutes acf_: brycec: could you point me to an example please? brycec: acf_: look in the scrollback for my sprunge.us link. :p
acf_: In short, from work, which is TWC "Business Class" I have no issues to ARP over nlayer/mzima, and the return mtr (not included) comes over L3 ***: essact has joined #arpnetworks acf_: level3 -> trit? brycec: trit not there (gathering mtr now) acf_: hmm yes... not all of level3 is affected ***: essact has quit IRC (Client Quit) acf_: fwiw the level3 looking glass routes through gblx -> trit -> arp brycec: acf_: http://sprunge.us/PXFD (ICMP because I have an old mtr on my VPS) acf_: ok thanks
so it looks like some of Level3 isn't affected
but are there any non-Level3 routed destinations that are affected?
can't wget http://level3.com fwiw brycec: acf_: from? acf_: arp brycec: ditto
up_the_irons: ^
(but I can from other networks
including an "affected" comcast connection acf_: same up_the_irons: hmm brycec: So, whether or not related to ARP's issue, (but it probably is), L3 has some split-braining going on -: brycec waits for up_the_irons to shut off the l3 session up_the_irons: ok i blew away level3 bmacs: lots of people tweeting about level3 problems
https://twitter.com/TVBroadcast/status/499243972221734912 BryceBot: TWITTER: Anyone aware of a "a large Internetnetwork issue is affecting ISPs across the country, like Comcast, Cogent, Level3, Time Warner, etc.?" (Tue Aug 12 17:20:13 +0000 2014, retweeted 1 times) bmacs: https://twitter.com/factor1/status/499249059433943040 BryceBot: TWITTER: Good news is that our network issue seems to be clearing up as Comcast and Level3 are responding to the fix. (Tue Aug 12 17:40:26 +0000 2014, retweeted 1 times) brycec: up_the_irons: suddenly working up_the_irons: whoa brycec: oh wait
nevermind, was v6 up_the_irons: hate that brycec: dualhoming biting me in the ass mhoran: Heh.
Yeah, all my users are complaining but v6 works just fine! brycec: hehe :D
i've had v4 drop out in the office for hours at a time before I noticed
(dhcp issue) mhoran: Had someone try the mtr command but it produced no output. I don't have access to the box so I can't debug. bmacs: http://downdetector.com/status/level3 acf_: ok seems Comcast goes via ntt now
still no TCP packets going via Comcast -> ARP path invader: Shit this explains the issues I am having in SunGard.. They just called L3 is broken and has been for the better part of a day. -_- Thanks for the call now.. Butt heads. twobithacker: mtr --tcp isn't working from my home FreeBSD box, trying to fix that so I can supply another data point acf_: return path still ok twobithacker: seeing the same problem from Comcast in NH though
IPv6 works, fwiw brycec: up_the_irons: fwiw, curl level3.com from arp still not working, mtr shows it going ntt->level3, unable to ping acf_: same here. seems dropping level3 didn't fix the problem mhoran: twobithacker: Yeah, my friends in MA can't connect via Comcast so probably the same route problem there. up_the_irons: brycec: yeah it's taking the exact same path, just through NTT now ;) mhoran: Whomp whomp. brycec: (And return path ARP->Comcast is over NTT->Comcast) acf_: yep
that's still broken too though twobithacker: huh, I can connect just fine from Comcast Business though brycec: So Level3 is having problems... and Verizon has been lashing out at L3. I bet Verizon haxxored Level3!!
^ sarcasm
twobithacker: yeah that's been observed acf_: ironically, my Verizon DSL line is the only residential connection I've seen unaffacted brycec: Seems home TWC and Comcast (among others) are affected, business customers are okay mhoran: I'm fine on TWC. :)
In NYC.
Also Verizon FiOS at my office is fine. acf_: it's interesting that's it's just tcp -: brycec nods mhoran: Yeah. acf_: it seems to imply that somebody is discriminating tcp traffic in some way or another brycec: Again, I blame the whole Netflix/Verizon debacle :P
"HOW DO YOU LIKE YOUR NETFLIX NOW, BITCHES?!?"
"YOU THOUGHT WE WERE THROTTLING? NOOOOOOW WE'RE THROTTLING!"
etc acf_: lol brycec: (except netflix.com is working for me) acf_: looking at just the forward path ARP -> Comcast residential brycec: (both v4 and v6) acf_: it's broken
both with ntt and level3 brycec: Well I have other work to get done... Best of luck, #arpnetworks acf_: see you later brycec mhoran: If it's just TCP, maybe some sort of traffic shaping failure of sorts? acf_: I was thinking about that...
but where? up_the_irons: brycec: LOL mhoran: Yeah this is far more exciting than the admin tool I was just writing ... brycec: s/ail/feat/ BryceBot: <mhoran> If it's just TCP, maybe some sort of traffic shaping ffeature of sorts? up_the_irons: hmm... traffic shaping, interesting.. erratic: http://www.secureworks.com/cyber-threat-intelligence/threats/bgp-hijacking-for-cryptocurrency-profit/ up_the_irons: acf_: BryceBot mhoran erratic invader : see any change? -: mhoran checks. erratic: checking
first time using mtr acf_: testing ARP -> Comcast path via NTT ... still broken for tcp mhoran: No dice. up_the_irons: ok acf_: just talked with a friend on another connection
says he can't connect to underworld1.no.quakenet.org either
but it works from Comcast up_the_irons: i still find it weird that level3.com is inaccessible erratic: yeah I haven't noticed anymore outage problems since earlier
was having trouble getting to cloudfront, linkedin, google, things like that acf_: erratic: from what connection? erratic: just my vpn tunnel
I have everything routed over my 206.125.168.65 addr
oh when i did traceroutes on stuff it was timing out at like 6 hops in level 3
then it just started working again about 5 minutes later
I wonder if theres a way I can continuously check with mtr
like if I wanted to watch for changes
I guess that wouldn't make a lot of sense now that I think about it acf_: mtr polls continuously, and should show changes as they happen erratic: weird sudo mtr --report -4 google.com
it just runs once and exits for me acf_: probably the --report erratic: ah up_the_irons: just shutdown Trit Networks BGP session, stab in the dark forgotten: you better fix all these things outside of your control up_the_irons erratic: oh nice
lol up_the_irons: lol erratic: acf thx acf_: still appears broken for the Comcast thing forgotten: im writing a blog post as we speak bad mouthing your services acf_: trit isn't in my comcast path fwiw
np erratic up_the_irons: yeah but sometimes... ya never know ;) bmacs: lol: https://twitter.com/ellisinzion/status/499255527990239232 BryceBot: TWITTER: @Level3 Yes you do have a major core IP outage right now. Stop saying there is no issue. Multiple big voice providers are down cold. (Tue Aug 12 18:06:08 +0000 2014, in reply to @Level3) up_the_irons: OL
LOL erratic: ae-3-3505.ear1.Dallas1.Level3.net? pyvpx: I totally just tweetd about this
neat acf_: so... ARP -> Comcast forward path via either NTT or Level3 is broken
which kind of indicates that the issue doesn't lie entirely with NTT or Level3 plett: acf_: But from .uk using NTT all the way to ARP works fine acf_: but other things (TWC) are broken too up_the_irons: plett: but what is your return path acf_: yeah, pretty much all the NTT stuff I've seen works fine plett: up_the_irons: I'll check
up_the_irons: NTT as well acf_: ec2 (ntt both ways) is fine plett: acf_: What's the path in both directions for the broken arp <-> comcast ? acf_: ARP -> Comcast is via NTT now, was Level3 earlier, both were broken
Comcast -> ARP is via nlayer plett: Ahh. nlayer acf_: I've been testing just the ARP -> Comcast part of the path
the nlayer part looks fine pyvpx: acf_: you never receive acks?
it's just TCP for everyone, yes? acf_: I do a tcpdump on the comcast side
I try to connect from the arp side
no packets come through pyvpx: UDP packets? or no packets acf_: going the other way works though
I use tcpdump -ni eth3 port 6668
and telnet [hostname] 6668 up_the_irons: hmmm...
HRMMM. acf_: UDP packets are fine up_the_irons: any change just now? i can actually check my email acf_: yep bmacs: yup acf_: packets are getting through to comcast now bmacs: Im up invader: up_the_irons, comcast works now
same with twc acf_: comcast is symmetric nlayer now up_the_irons: the thing with NTT / Level3 is, a lot of NTT routes go through Level3 anyway acf_: is it possible that something is dropping things with asymmetric paths? up_the_irons: so what i did was, lowered s7.lax default route local pref so most traffic goes out via s1.lax directly, not hitting s7.lax. s7.lax has Level 3 and NTT.
acf_: that would take stateful filtering i believe and is pretty rare on regular transit links acf_: wow. everything is via nlayer now :P
even any2ix stuff
nlayer to the rescue! (surprisingly) up_the_irons: yeah
not going through s7.lax kills the rest of my network ;) (no peers, L3 or NTT) erratic: oooo up_the_irons: while this works, it seems like a shitty solution
unless the problem really is with Level3, in which case we can wait it out now acf_: I don't think it is... ARP -> NTT -> Comcast doesn't work... up_the_irons: acf_: still? acf_: it's not NTT anymore, so not it works up_the_irons: oh wait.. couldn't be cuz ARP -> NTT isn't possible right now acf_: * now it works
it didn't work both when it was ARP -> NTT -> Comcast and ARP -> Level3 -> Comcast up_the_irons: ok
well w..t.f erratic: Im confuzled,
how does it kill the rest of your network
not that it matters just curious
this stuff is interesting to me acf_: basically up_the irons took the s7 router out of the network erratic: right acf_: which is the connection between you and any2ix, ntt, and level3 erratic: I gathered that much does that mean none of the networks including mine will be reachable? acf_: so without s7, all you get is nlayer up_the_irons: erratic: s7.lax is a bigger router and carries more routes. usually, all traffic goes to it. but now i'm diverting traffic to go directly out s1.lax. acf_: it just means they'll take a diffetent path up_the_irons: and yeah, basically what acf_ said erratic: oh right like if nlayer goes down... up_the_irons: i'm tempted to reboot s7 acf_: up_the_irons: so can you do testing with s7, static routes, etc... now that you've diverted traffic? up_the_irons: acf_: well, technically yes, but hard to test when no traffic is going to it ;) acf_: just out of curiosity, does ios have a tcpdump equivilent?
now that there isn't any traffic, would it be more realistic to debug with tcpdump? up_the_irons: well, what i would do is use a SPAN port and then dump that port. i have one set up already. erratic: like switch port mirroring? up_the_irons: yup erratic: that is handy up_the_irons: yup, purely meant for diagnostics
i can copy all traffic from any physical port or SVI (VLAN)
and then just watch it from a regular Linux box with tcpdump erratic: Im not sure how much overhead that would be, probably too much
but
you could probably use it with snort
assuming the switch or whatever can handle it, without creating a single point of failure up_the_irons: yeah
ok, s7.lax is getting a kick in the butt (reboot) acf_: sounds like a good idea :P m0unds: womp womp RandalSchwartz: I saw a momentary drop
I presume that was a shuffling? ***: fink has joined #arpnetworks up_the_irons: .... ***: neish has quit IRC (Read error: Connection reset by peer)
neish has joined #arpnetworks neish: wat erratic: I lost talk.google.com for like a sec RandalSchwartz: ok
yeah... my connection went dead for about 10 seconds erratic: yeah I think I lost everything for a secondlet
laggggggg up_the_irons: s7 reboot is done, i think i'm going to send traffic to it again and see what happens forgotten: whats the verdict? :D mhoran: Guilty. forgotten: invader: see ^
and the sentense? pjs: looks like it's getting better up_the_irons: how's it looking for u guys now? erratic: I need to consume this alcoholic beverage m0unds: erratic: alllllcohol acf_: arp -> ntt -> comcast seems to work now erratic: glug acf_: so it was s7 being weird? up_the_irons: then i'm going to throw fucking s7 into the dumpster, gaah! erratic: what sort of hardware is s7 m0unds: up_the_irons: you just can't catch a break with that thing up_the_irons: srsly m0unds: but that's why shotguns exist up_the_irons: erratic: Cisco 6506 w/ sup720-3bxl
i've had a replacement Sup in the cage for months now :( erratic: Im just curious, couldn't you use a linux box to do that up_the_irons: in the triage order of things, replacing it just wasn't high on the list. but it is now. m0unds: up_the_irons: did you ever swap the sup, or is the one in the cage the one you got to replace it when it did that weird CEF barf thing? erratic: that thing looks heavy up_the_irons: erratic: well you wouldn't get any hardware acceleration and the pps is really low in comparison. so basically, a linux box would melt under a DDoS
m0unds: yeah it's the replacement for the other issue
i'm going to re-enable level 3 now erratic: yeah Im reading the specs on it now lol acf_: why just tcp? up_the_irons: man the ipv4 routing table is just getting too large :( acf_: and *some* tcp too? up_the_irons: acf_: no. fucking. idea. ;) erratic: wow jeez forgotten: im sure if it's working for others, it's working for at&t too up_the_irons: i think i need to stop being married to the (now) old idea that carrying a full routing table is *that* much better invader: Sorry I broke the Internet forgotten: GUILTY! acf_: on a slightly unrelated note, are you planning to announce to level3 at some point? erratic: Webster: we have a lot of 6500 ciscos and have recently grown up to be able to get the ASR 9000s
ooooooo m0unds: ugh, ios-xr jpalmer: so, my first cisco class starts monday. erratic: and ends pretty much never if you take into account how esoteric it all is m0unds: haha
i hope you learn a lot about serial stuff, because that shit still exists in ios
haha erratic: m0unds: just a sec jpalmer: my vmware class also starts monday m0unds: that sounds like more fun jpalmer: indeed
I'm only taking the cisco, cuz my job is making me. I don't find routing or switching to be intersting at all. I'd be far more interested in the security tract. erratic: m0unds: this is how I decided to upgrade my bios http://3.bp.blogspot.com/-GDEA1qZ0thk/U-d-u4aUM8I/AAAAAAAAPTo/tgcCHYdR5c4/s1600/Screenshot+-+03272013+-+05:40:47+AM.png
just because lol y not m0unds: hahahahah
nice up_the_irons: m0unds: hey i use serial all the time, the 2612's are great for out-of-band access to everything :) I have 32 serial ports with breakout cables on one of 'em. m0unds: oh, i meant like frame relay serial :) up_the_irons: oh
lol m0unds: i have an AS2516-RJ (iirc) that takes rollover cables terminated w/rj45s
it's great as a terminal server for my crisco lab erratic: jpalmer: I have pretty much my lack of interest in anything except computers but more importantly the people I've surrounded myself with who have been willing to help to thank
I got kicked out of school m0unds: i got my employer to approve me bringing it to work so i wouldn't have to try and store all that gear at home :) erratic: haha yeah I love piling up my crap at work
makes me feel comfortable to have crap around me m0unds: i racked mine up and told my coworkers to learn something erratic: m0unds: definitely
yeah I used so much of my own hardware at work m0unds: since i'm hoping i'll be able to leave here in a year or so, and since they don't have another network engineer, they'll be utterly SOL if they don't erratic: well at my last good job
I haven't had a decent job in a while
finally starting to get interested in stuff again though jpalmer: m0unds: I do the same thing. and I teach them how to make and restore backups first.. so that if someone changes it when they are done, they can instantly revert it back to their last configuration. gives them a LOT of backup/restore practice, and makes sure everyone can use the lab without stepping on other peoples toes. erratic: Ive had crap jobs and a few short term contracts and stuff but nothing really
my last job really spoiled my appetite for work
was laid off a year ago today and Im just now like, ok I'm moving to a new place and I don't know anybody and I have no idea what to expect and I think it will be challenging for me socially jpalmer: erratic: where are you moving to? erratic: Netherlands
Im in Greece atm jpalmer: thats.. a move, for sure. hehe m0unds: ah, cool - my grandparents are from the netherlands jpalmer: my wife and I are talking about moving from florida to charlotte, NC erratic: NC isn't too bad m0unds: charlotte is nice - my dad lives in burlington, just outside greensboro erratic: need a car for sure
I like living in the city too much
but I do appreciate living out in the country and I miss it sometimes jpalmer: we'd like in one of the outskirt areas, for sure. horses and such
s/like/live/ BryceBot: <jpalmer> we'd live in one of the outskirt areas, for sure. horses and such erratic: I lived in creedmoor outside of RD
for awhile
was like 16 or 17
long enough to make friends then move away again
I really want to believe despite how much i hate redhat it would be really hard (definitely competitive) to work there
but probably pretty interesting
especially if you can get a good systems engineering job
they pretty much birthed xen and now from what I've been reading a lot of the work on qemu/kvm looks pretty sweet
and I'm running a lot of the more recent stuff ***: invader has quit IRC (Remote host closed the connection) erratic: actually found a vid this morning exactly what I have except the guy has much faster hardware
https://www.youtube.com/watch?v=37D2bRsthfI BryceBot: YouTube Tech: "Qemu/KVM + vfio = Virtual machine for gaming with 95% of native performance = no more dual booting" by blu3bird84 (2m 8s), 18,965 views, 222 likes and 4 dislikes. Uploaded 2014-02-02T19:42:29.000Z. up_the_irons: anyone seen any issues in the last 30 mins?
things are working for me... forgotten: wish i could test :( bmacs: no issues here. been clear since the reboot up_the_irons: roger that ess: Yep. No issues here.. ***: mus1cb0x has joined #arpnetworks mus1cb0x: any issues with kvm 14 today?
krv14, that is up_the_irons: can someone put the topic back to the way it was :) mus1cb0x: i tried connecting several times a couple of hours ago without success m0unds: seems the NTT comcast stuff is finally getting better up_the_irons: mus1cb0x: was probably the general network issue we had m0unds: just looked at my arp latency graph, and it's looking much healthier than a month ago mus1cb0x: ah, i see the topic
how long was the outage? up_the_irons: since like 1AM, but it was very selective on certain routes. Cacti showed most traffic was getting in/out. mus1cb0x: ouch, what was the cause? mhoran: Looks good here. ***: invader has joined #arpnetworks
fink has quit IRC (Quit: fink) up_the_irons: mus1cb0x: s7.lax being stupid ***: mhoran changes topic to: VPS Services: http://www.arpnetworks.com/vps | FAQ: http://support.arpnetworks.com/faqs | Wiki: http://wiki.arpnetworks.com | Twitter: http://twitter.com/bsdvps | @up_the_irons is staff, other ops are long-time customers | Channel Log: http://is.gd/eWA5u mhoran: (Had to get on a desktop for that one.) mus1cb0x: stupid? forgotten: aka not smart mus1cb0x: sarcasm isn't helpful
nor is "stupid" when discussing network outages
i have to justify the outage and i'm not going to use stupid and risk my job forgotten: this isn't an offical forum
go put in a ticket mus1cb0x: heh, geez
at least you're not op'd, because you would be the wrong person to represent arp -: forgotten takes a bow up_the_irons: play nice now mhoran: up_the_irons: It does seem like there were more widespread issues than just with your router, though? I definitely experienced issues with non-ARP connectivity this morning.
Perhaps all related in some way?
e.g. a crazy routing table causing issues on routers across the nets? up_the_irons: mus1cb0x: There was something strange with TCP traffic not being passed through to certain destinations over s7.lax. UDP was fine. mhoran: I guess the fact that it was TCP only kind of makes my thought make no sense.
Nor does that sentence make sense. brycec: Huzzah Internet working! up_the_irons: mhoran: hmm.. that's a good point about the routing tables, maybe it is all related in some way brycec: oh boy, I missed a lot invader: The TCP thing really sounds like someone played with acls. up_the_irons: mhoran: I didn't realize you had non-ARP connectivity problems too mus1cb0x: did anyone else see the article about routing table hijacks lately to commandeer bitcoin mining? mhoran: up_the_irons: Yeah I was having trouble with some things before I left from home this AM, over IPv4 from Time Warner. up_the_irons: mus1cb0x: someone pasted that, yeah
mhoran: hmm... interesting invader: up_the_irons, L3 had an confirmed issue was a few asn's/ mus1cb0x: any chance that could be a cause? up_the_irons: some type of BGP poisoning? I suppose it's possible mus1cb0x: http://www.reddit.com/r/cryptodev/comments/2d0bzx/bgp_hijacking_for_cryptocurrency_profit/ jpalmer: up_the_irons: we hit 512,000 active routes today, were you aware of that? invader: up_the_irons, From the sounds of it, They only said "route issue" Getting a straight answer out of them is like try to date Misty Dawn. up_the_irons: jpalmer: lol, someone just PM'd me, "related to the 512k routes issue?"
perhaps that was it...
i'm showing 499K right now
i'm going to be doing a route haircut tonight
maybe kill all /24's jpalmer: up_the_irons: lots of routers with the 512k limit have been having issues today. m0unds: sup 720 should handle 1m routes, iirc -: invader hands up_the_irons a scissors invader: Happy grooming m0unds: might be thinking of the 2t though up_the_irons: danke mhoran: Sure should. invader: what kind of gear you guys using at your edge? up_the_irons: m0unds: jpalmer : i have the -3bxl, so it does 1M routes. however, i have two full tables, so right about there m0unds: ohhhh
gotcha up_the_irons: CEF is actually bitching at me in the logs... lol m0unds: haha
i hate cef
such a kludgy thing mhoran: I could see CEF blowing up and causing all sorts of weird issues, in addition to the missing routes.
Still don't understand why UDP would work, unless it took a different path somehow. m0unds: yea, but it usually fails by dumping forwarding onto the cpu
then the cpu shits its pants and the whole chassis stops passing traffic because of the load
or crashes, depending
haha mhoran: http://status.livestream.com/ -- so it looks like my ex startup had routing troubles today, too. :p
All the competent network techs left, so I'm not surprised their router crashed.
Was also a 720-3bxl.
That I doubt anyone has maintained in two years. up_the_irons: well that was easy:
s7.lax(config)#mls cef maximum-routes ip 768
Maximum routes set to 786432. Configuration will be effective on reboot.
looks like another reboot will happen in the very near future... (tonight maybe)
;) forgotten: do it now while only half of people think its fixed!
:D up_the_irons: hahahhah
nice:) m0unds: boo cef, boo up_the_irons: and yeah, CEF blowing up means the router falls back to software switching, which will melt it m0unds: yeah, i wonder if cef was loaded and it was unable to take on any flows up_the_irons: actually s1.lax did that once, i hit the 128K limit. it went full on to CPU switching. some of you may remember that outage. but hey, it didn't crash, still 5+ year uptime! :) m0unds: hahaha
s1's a 45xx right? up_the_irons: yup -: brycec remembers, he thinks jpalmer: up_the_irons: so it was hitting the 512k limit? m0unds: my buddy @ easystreet up in oregon had a similar thing w/a 45xx killing CEF and dying horribly overnight one night up_the_irons: jpalmer: i don't think so, or else *nothing* would have gotten through mhoran: Yeah, that's happened to me too, when I foolishly tried to load the entire routing table on a 4500. :p
Upstream route filter was misconfigured and I didn't guard against that. up_the_irons: ouch ***: raptelan_ has quit IRC (Quit: Page closed) mus1cb0x: preemptive optimization doesn't exist in networking :P
er, premature mhoran: Heh. forgotten: http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-major-outage-across-us-network-providers ***: fink has joined #arpnetworks
invader has quit IRC (Ping timeout: 245 seconds)
thestereobus has joined #arpnetworks
qbit has quit IRC (Quit: leaving)
ess has quit IRC (Quit: Page closed)
thestereobus has quit IRC (Ping timeout: 250 seconds)
sjackso has joined #arpnetworks
thestereobus has joined #arpnetworks
thestereobus has quit IRC (Quit: thestereobus) m0unds: http://www.cidr-report.org/as2.0/#General_Status mhoran: Number of ASes seems to be approaching a number that someone may have hardcoded into a system, too. :p m0unds: yeah, haha sjackso: I heard a rumor of a bgp apocalypse and I figured #arpnetworks would tell me whether or not to panic m0unds: bgpocalypse ***: fink has quit IRC (Quit: fink) RandalSchwartz: wow. ziprecruiter.com has gotten about 5 times faster in the past few days for job searches.
my efforts are paying off. ***: qbit has joined #arpnetworks
qbit is now known as Guest18868
Guest18868 is now known as qbit_
qbit_ has quit IRC (Quit: leaving) up_the_irons: m0unds: sjackso : lol staticsafe: so much fuss for so little
and of course media blows it up ***: qbit has joined #arpnetworks
qbit is now known as Guest87335
Guest87335 is now known as qbit qbit: hola
:D
so - are the weird issues fixed? ***: qbit is now known as Guest67943 Guest67943: from earlier milki: alive! ***: Guest67943 is now known as qbit qbit: i seem to be having some dns issues - not sure if it's residual or not
could also be from the upgrade i stupidly decided to do mid day :P milki: aha qbit: host can resolve things fine.. but ping doesn't
making me rage brycec: qbit: connectivity was resolved about 7 hours ago qbit: k
it's totally fucked up - ping doesn't resolve some hosts.. chat.freenode.net.. etc - but it does for others
and host works for all brycec: qbit: reboot?
Seems like the host dns resolver cache is fubar'd
but host does a fresh fetch iirc
or at least a direct connection, bypassing the normal caching qbit: yeah - reboot does nothin to fix it ***: qbit has quit IRC (Quit: leaving)
fink has joined #arpnetworks fink: RandalSchwartz: i meant, are you up to freebsd version 9 or 10 RandalSchwartz: when did you "meant"?
I'm still on 8.4
probably 10.x during some spare downtime in december
after I first do it on insightcruises.com machines. :)
fink? fink: yo RandalSchwartz RandalSchwartz: was answering your questions fink: i said: fink: yo RandalSchwartz, how's it hanging. are you up to 9 or 10 with your vps? RandalSchwartz: never saw that
or maybe long ago fink: yea, yesterday ;) RandalSchwartz: so there's my answers fink: i'm wary of moving my zfs on root from 8.4 to 10 mhoran: Sounds exciting! ***: qbit has joined #arpnetworks
qbit is now known as Guest13111
Guest13111 is now known as qbit RandalSchwartz: yeah - there's definitely some possible issues there
which is why I want to try it on $client's computers first :) ***: fink has quit IRC (Quit: fink) RandalSchwartz: there's some settings for internet in /etc/rc.conf that change, I'm told
and if I get those wrong, I don't get back in to the box :) mhoran: Yup, there's that.
Though it's not too crazy. hazardous: RandalSchwartz: i'm actually somewhat curious how much business/interest there is in insightful holiday packages
i never really thought of it before m0unds: https://supportforums.cisco.com/document/12202206/size-internet-global-routing-table-and-its-potential-side-effects lol hazardous: http://bgr.com/2014/08/12/comcast-fcc-commissioner-clyburn-dinner/ lol RandalSchwartz: curious in what way?
these aren't holiday packages
if you'd spend 4 days at a hotel in $random_city to attend a conf, why not 7 days on a cruise ship to do the same thing, but with better speakers? hazardous: i dunno, i just never thought of that
i assumed most techies would rather book their own stuff and explore like a local or maybe that's just me RandalSchwartz: yeah, most people haven't, which is why Captain Neil makes some bucks hazardous: forgot about the conf part RandalSchwartz: primarily, these are conferences
they just happen to be meeting on a ship
instead of a hotel
lots of advantages to that -: m0unds is sick of conferences in vegas RandalSchwartz: right m0unds: all of my industry conferences are in vegas
there's one that was in san diego once
but never again RandalSchwartz: and one big plus... the speakers are in the same BARS and same DINNER as you are
unlike land conferences, where they all scatter.
and our speakers are expected to mingle with the attendees at those.
imagine the conferences you've attended where you could bump into one of the keynote speakers later at the bar...
or even on a shore excursion... a few hours in the same small group with them ***: r0b has joined #arpnetworks hazardous: so you have people actually interested in the subject there as opposed to forced to attend by corporate RandalSchwartz: geez... you must have a twisted view of what we do!
I don't even know how to answer that.
"I am interested in X"
"Oh look... a cruise where speakers will be talking about X"
"I want to go"
[if employed] "Employer, can you sponsor me to this conference?"
[if retired] "Let's see if my travel budget covers this"
That's how people get on
Nobody is *forced*
"Oh gawd... FORCED to go on a CRUISE!" :)
see how silly that sounds now? -: RandalSchwartz wanders off for a bit while relocating IRL m0unds: i think hazardous was just pointing out that people attending a conf on a cruise are more likely to be going because they want to and not because someone forced them to
at least that's how it read to me ***: r0b has quit IRC (Ping timeout: 255 seconds)
r0b has joined #arpnetworks
r0b has quit IRC (Ping timeout: 240 seconds)
thestereobus has joined #arpnetworks
thestereobus_ has joined #arpnetworks
thestereobus has quit IRC (Ping timeout: 264 seconds)
thestereobus_ is now known as thestereobus mnathani: what was that command to run an mtr report with output to a text format for easy copy /paste
have the upstream TCP transit issues been resolved? thestereobus: I was seeing intermittant failures earlier tonight but they didn’t last long
mtr -4 --tcp -rc 10 <your-IP> ***: thestereobus has quit IRC (Ping timeout: 246 seconds) up_the_irons: mnathani: add -w so it doesn't cut off hostnames (wide format) ***: thestereobus has joined #arpnetworks
befrank has quit IRC (Quit: Page closed)