| ↑back Search ←Prev date Next date→ Show only urls | (Click on time to select a line by its url) |
| Who | What | When |
|---|---|---|
| *** | sga0_ has joined #arpnetworks
sga0 has quit IRC (Ping timeout: 246 seconds) | [00:50] |
| ......... (idle for 42mn) | ||
| forgotte1 has joined #arpnetworks | [01:35] | |
| forgotte1 | arpnetworks down?? | [01:35] |
| nvm | [01:40] | |
| up_the_irons | what happened? | [01:46] |
| forgotte1 | it seams at&t have blocked my connection to my own sites / services
uverse well and arpnetworks.com entirely | [01:56] |
| those dirty sluts man!!
up_the_irons: what is arpnetworks ipv4 range? i wanna make sure they unblock it for everyone with at&t service in case others are having the same problem. People could be loosing potential business if no one on at&t uverse can access any sites hosted at arp not sure it's that wide spread, but can't hurt i dont think | [02:09] | |
| pyvpx | hmm, you can find out using IRR
if he's been good about his object maintenance ;) | [02:19] |
| up_the_irons | my objects are up to date
it's weird, I'm on twc, i can ping everything in my cage, but SSH, http, etc.. seems to be all blocked. started about an hour ago. I'm still in irc only because I'm using weechat android from my phone | [02:20] |
| pyvpx | thats fucked | [02:23] |
| forgotte1 | oh wow....
so it's not just at&t... same symptoms here, can ping everything. but ssh, http, https all blocked lol this at&t rep wants me to reset my modem to factory. now i feel like an ass :P | [02:26] |
| pyvpx | up_the_irons: can I try anything for you from Germany? ;) | [02:30] |
| up_the_irons | pyvpx : how about just arpnetworks. com? | [02:32] |
| forgotte1 | wyliebayes.com , nadasound.com
2 of mine | [02:33] |
| pyvpx | pings and loads just fine
I'm coming from 37.120.2.102 | [02:34] |
| up_the_irons | overall, my bandwidth graphs for the big pipes look pretty normal
so most traffic is still getting out so weird about twc and att | [02:34] |
| forgotte1 | it's only residential uverse att tho. i can access via my att 4g network no problem | [02:41] |
| pyvpx | what are the blocks you guys are coming from that can't reach arp?
pyvpx looks to BGP | [02:42] |
| forgotte1 | i am, 99-95-174-245.lightspeed.sndgca.sbcglobal.net
pyvpx: would be weird bgp would effect all services, except ICMP | [02:43] |
| pyvpx | oh, I meant from a policy stand point
localize to a router or a routing event that could point to a config change it's easier to hit someone on the head at att when you are like "yeah at 4am this shit started" they go "wait we had a thing at 4am uhhhhh" ...sometimes | [02:46] |
| forgotte1 | i would be surprised if the "tech" im talking to in chat can even feed himself successfully
from att | [02:48] |
| pyvpx | sometimes if you play the act
the ip noc folks can entertain you give you ticket numbers or other depts to call pyvpx was never an ATT IP customer | [02:48] |
| forgotte1 | ya im trying, had to DEMAND i speak to his supervisor cause he tried to pawn it off on the hosting provider hehe
by pasting me out of the book answers if resetting my modem didn't fix the problem, we have exhausted all possible solutions. LOL | [02:49] |
| pyvpx | hmm
the denver router on their looking glass had a thing 42 minutes ago probably nothing everything seen through verio | [02:53] |
| forgotte1 | hrmm
and since icmp works it's hard to see where it might be failing i am in San Diego CA | [02:54] |
| *** | thestereobus has joined #arpnetworks
thestereobus has quit IRC (Ping timeout: 244 seconds) thestereobus has joined #arpnetworks | [02:56] |
| up_the_irons | I'm playing around with shutting down certain transit sessions, to see if any of them are the culprit. nothing successful so far
it's like only tcp doesn't work. i can still connect to my VPN (udp) and dns still works fine... wtf | [03:02] |
| thestereobus | yeah, I’m connected over VPN but TCP is down | [03:04] |
| pyvpx | so tcp on att uverse and twc?! | [03:04] |
| forgotte1 | weeeeird | [03:04] |
| thestereobus | TWC here | [03:04] |
| forgotte1 | my vpn runs on tcp :( | [03:04] |
| pyvpx | what do those have in common | [03:04] |
| forgotte1 | that we know of, only att and twc | [03:05] |
| thestereobus | I can’t ping or SSH into the server even over the VPN but I can connect via the iKVM console | [03:06] |
| pyvpx | ikvm console on different ip ranges?
pyvpx isn't exactly a customer... | [03:08] |
| forgotte1 | thx for the assist anyway pyvpx :) | [03:11] |
| up_the_irons | man this is weird | [03:19] |
| thestereobus | yup, UDP works | [03:19] |
| pyvpx | up_the_irons: yeah I'm kind of at a loss as to what I'd do | [03:21] |
| up_the_irons | yeah | [03:22] |
| forgotte1 | i always find the weird shit | [03:24] |
| KDE_Perry | happening for me from comcast too | [03:25] |
| forgotte1 | shiz | [03:27] |
| pyvpx | comcast as well?
up_the_irons: you dont see these networks over any2, do you? fucking up something like all tcp traffic sounds like a coresite thing...on its face :p | [03:27] |
| up_the_irons | no, I'm small peanuts to them, they would not peer with me over any2 | [03:28] |
| forgotte1 | i can connect via ssh from devio.us
not exactly sure where that is hosted | [03:29] |
| up_the_irons | if you guys do "mtr --tcp arpnetworks. com ", where does it stop? for me it is around nlayer | [03:30] |
| KDE_Perry | same here | [03:31] |
| forgotte1 | dimenoc, somewhere in FL
unfortunately mtr wants to crash on me today | [03:31] |
| thestereobus | I’m conneted to a remote machine at 162.248.100.99
I can’t access the arpnetworks via TCP but the mtr command completes fine nmap -Pn -p22 -sU arpnetworks.com works | [03:36] |
| up_the_irons | even with --tcp? | [03:36] |
| thestereobus | yes | [03:37] |
| up_the_irons | weird, doesn't finish for me | [03:37] |
| thestereobus | and all my server monitors at nodeping.com have failed as well | [03:39] |
| forgotte1 | nmap comes back as closed for me, for 22. when it should be open
for my own site shows filtered for arpnetworks.com | [03:40] |
| thestereobus | no packet loss and 30-40ms pings using mtr so it doesn’t seem like it’s bandwidth
actually I have no problems contacting arpnetworks.com at all from that server - just my own arpnetworks IP | [03:41] |
| up_the_irons | I'm trying from several nlnog ring nodes and everything works so far | [03:43] |
| thestereobus | so mtr —tcp arpnetworks.com works mtr —tcp danconnor.com does not (failing at ge0-arpnet.cust.lax07.mzima.net) | [03:43] |
| up_the_irons | thestereobus : what's your server ip? | [03:44] |
| thestereobus | 206.125.168.2 | [03:44] |
| up_the_irons | oh! that i know.. you're leaking traffic between your two uplinks and stp is shutting down your interface (completely unrelated issue) | [03:45] |
| forgotte1 | i am 206.125.175.0/29 | [03:46] |
| up_the_irons | i can re enable...
done | [03:46] |
| thestereobus | ah, ok. is that related to link aggregation?
I enabled that recently | [03:46] |
| up_the_irons | most likely, yes | [03:47] |
| thestereobus | ok. I enabled that recently
I’ll turn it off for now carry on | [03:47] |
| up_the_irons | cool :) | [03:47] |
| lol, finally back in weechat on my laptop (instead of phone). i just relayed through an nlnog node to my irc box..
sucks that i have to do that, but.... works for now | [03:57] | |
| forgotte1 | my irssi session is still working fine, from my box to freenode lol
just can't get into my box | [03:58] |
| *** | thestereobus_ has joined #arpnetworks
thestereobus has quit IRC (Ping timeout: 255 seconds) thestereobus_ is now known as thestereobus | [03:59] |
| .... (idle for 16mn) | ||
| up_the_irons | i have a feeling that this is a GTT/nLayer issue, so i just emailed their NOC | [04:18] |
| forgotte1 | word | [04:26] |
| .... (idle for 16mn) | ||
| for me tcptraceroute gets to: 7 ae-5.r04.lsanca03.us.bb.gin.ntt.net (129.250.2.220) 28.686 ms 27.857 ms 28.676 ms
then dies on udp next hop is: 7 ae-5.r04.lsanca03.us.bb.gin.ntt.net (129.250.2.220) 29.388 ms 29.274 ms 27.852 ms err 8 ge-0-7-0-24.r04.lsanca03.us.ce.gin.ntt.net (129.250.198.186) 29.820 ms 29.341 ms 29.662 ms | [04:42] | |
| up_the_irons: they are answering at the noc number.... but wont help me lol
are you a direct customer of us.ntt.net ? | [04:50] | |
| .... (idle for 16mn) | ||
| jbergstroem | so confirmed network issues i guess?
im getting emails from people all around | [05:06] |
| notion | from here. it looks like the tcp reply never get backs to the client - server get my http request, and sends back reply - just don't reach the client | [05:09] |
| jbergstroem | some of the users i've been in touch with says that icmp ping doesn't reply either
have yet to find a rig i control that can reproduce | [05:10] |
| pyvpx | yeah, something verio
I vaguely recall them having some mysterious IDS thing that did things like this occasionally but that was years ago but nlayer/gtt has used verio for ages. pyvpx has more beer than technical details pyvpx opens another one | [05:16] |
| notion | yeah, seems wicked atm. oh well, i'll wait some more - not working from .dk sites, but works from .de site
enjoy the beer ;D | [05:19] |
| forgotte1 | =[06:28] *** raptelan_ has joined #arpnetworks | [05:28] |
| ............. (idle for 1h1mn) | ||
| raptelan_ | is Arp having some networking problems currently?
I can ping my VPS, but SSH attempts time out, also cannot load arpnetworks.com in a browser. | [06:29] |
| forgotte1 | yes they are
it appears only TCP traffic is not getting through, from pretty much every ISP in U.S. | [06:32] |
| pyvpx | well, more like the more notable US access networks
and a non-obvious number of european networks | [06:33] |
| raptelan_ | ok, I guess I don't have to go strangle somebody in our IT department then :P | [06:34] |
| plett | pyvpx: Only adding one data point, but ssh to my arp vps works fine for me in .uk (traffic goes via NTT for both v4 and v6) | [06:35] |
| notion | UK seems to have hit something magic - http://downdetector.co.uk/problems/ee-everything-everywhere/map/ | [06:35] |
| *** | fink has joined #arpnetworks | [06:36] |
| pyvpx | works fine from .de
well, my slice. | [06:36] |
| notion | BT outage chart dont look good :(
http://downdetector.co.uk/problems/bt-british-telecom/map | [06:36] |
| plett | notion: I wouldn't put too much trust (i.e. any at all) in downdetector | [06:38] |
| notion | plett: ok | [06:38] |
| dne | no issues from .se either | [06:38] |
| notion | plett: my trafik through the UK link fails, but works perfect from my DE | [06:43] |
| plett | notion: From where to where? | [06:43] |
| hazardous | i'd trust that BT is terrible though | [06:43] |
| plett | Naturally, they are a defacto monopoly telco | [06:44] |
| notion | plett: UK link from: 2.111.95.123 to: 174.136.105.26 | [06:44] |
| plett | notion: That's from .dk to arp? Does that normally go through .uk for you? | [06:45] |
| notion | plett: yes | [06:45] |
| jcv | my vps networking seems fine
in fact the irssi instance i'm typing in now is running on it | [06:45] |
| jbergstroem | I still get reports from users having issues | [06:46] |
| plett | notion: Does a traceroute for that go through nlayer or GTT? up_the_irons said it might be a problem with them | [06:46] |
| jbergstroem | mostly "is the site down" | [06:46] |
| notion | plett: not sure who's who on the path - my traceroute: http://pastebin.com/754FDTGx | [06:47] |
| plett | notion: And ssh doesn't work for you? | [06:48] |
| notion | plett: ssh does not work
plett: or http | [06:48] |
| *** | acf_ has joined #arpnetworks | [06:49] |
| notion | plett: ICMP does tho, odd | [06:50] |
| plett | notion: I can't connect to ssh on 174.136.105.26 (I'm probably firewalled), but HTTP is fine for me. My path looks very similar to yours http://pastebin.com/4Heg7P6B | [06:51] |
| brycec | And here I was thinking it was Windows being dumb...
Well I mean it still is, Windows is being dumb and refusing to ipv6 right now for whatever reason. | [06:52] |
| notion | plett: ok, thanks for checking - guess I'll wait some more and see how it goes | [06:53] |
| brycec | But that just leads to one more data point: my ipv6 connection is still up and working and connected. But comcast ipv4 has shit the bed for me too - ping, but not tcp
brycec continues gathring data... *gathering forgotte1: answering your hilight from another channel - yes I'm alive | [06:54] |
| acf_ | > But comcast ipv4 has shit the bed for me too - ping, but not tcp
same here also, can't telnet underworld1.no.quakenet.org 6667 from ARP | [06:58] |
| brycec | (hooray for openvpn/udp still working) | [06:59] |
| acf_ | also GRE | [06:59] |
| brycec | Good to hear - hadn't tested that yet | [07:00] |
| acf_ | hmm... can't get to the IRC logger webpage
how long has this been happening? I've been out of town... | [07:00] |
| brycec | fwiw, comcast -> nlayer -> mzima ICMP
acf_: for the bast 6-ish hours | [07:02] |
| acf_ | I remember trying to ssh in from my T-Mobile phone earlier in the week, and it not working...
but I could ssh to Amazon EC2, and then from Amazon to ARP | [07:03] |
| brycec | I'd offer mtr --tcp information, but on my FreeBSD box at home I get "bind(): No error: 0" and my OSX machine gives me "bind(): Undefined error: 0" so... sorry. | [07:05] |
| acf_ | forward path to underworld1 is via Level3
forward path to Comcast is via Level3 | [07:06] |
| brycec | From the reports, this seems very widely spread, so more likely an ARP router issue than a provider. But that's just my $.02 distilled from the IRC history.
Looks like the office TWC connection is getting through too | [07:08] |
| acf_ | yeah, other Level3 things seem to work...
mtr --tcp and ICMP from Comcast: http://paste.debian.net/115125/ | [07:09] |
| brycec | (The first reports were of TWC failing) | [07:09] |
| acf_ | Comcast Business gets through | [07:10] |
| brycec | (dammit the only Linux box online at home is Debian, with mtr 0.82 that doesn't have --tcp) | [07:10] |
| acf_ | hmm... nothing in backports? | [07:10] |
| brycec | acf_: Nope. https://packages.debian.org/wheezy/mtr-tiny
I remember coming across that missing too... but Debian is/seems very closed-off when it comes to some guy that wants to say "hey, can you ___ this?" | [07:12] |
| acf_ | aww darn. I just run Debian testing on most stuff that's not critical. jessie has mtr 0.85
an mtr --tcp from arp to comcast and underworld1 is blank | [07:14] |
| brycec | blank? how odd. | [07:15] |
| acf_ | ie, not hops
but comcast business shows via Level3 as usual | [07:15] |
| raptelan_ | http://makeameme.org/media/created/how-odd.jpg | [07:15] |
| brycec | It no doubt goes without saying, but up_the_irons, please fix :) | [07:16] |
| *** | ess has joined #arpnetworks | [07:17] |
| acf_ | comparison of Comcast business and residential paths: http://paste.debian.net/115137/ | [07:19] |
| ess | Unable to get to arpnetworks.com from comcast. traffic seems to be dropped after nlayer > mzima. | [07:19] |
| jbergstroem | So this basically affects all customers at arp?
Im colo should that matter for some reason | [07:21] |
| acf_ | has anybody noticed IPv6 destinations being affected? | [07:22] |
| ess | I'm able to get to arpnetworks.com if I go through a server in Washington state (different ISP and different route)
for me prob seems to be between ge0-arpnet.cust.lax07.mzima.net (67.199.135.102) and arpnetworks.. | [07:23] |
| jbergstroem | I can confirm that from a user I have in australia | [07:26] |
| anyway.. enough whine for tonight. | [07:32] | |
| ess | Any idea who can help at mzima.net? | [07:34] |
| http://www.mzima.net gives 404. Was this up before? | [07:44] | |
| acf_ | mzima is part of nlayer
just read the IRC logs... it's all good. up_the irons knows about this already | [07:47] |
| ess | k, thanks | [07:47] |
| *** | fink has quit IRC (Quit: fink)
dj_goku has joined #arpnetworks dj_goku_ has quit IRC (Read error: Connection reset by peer) forgotten has joined #arpnetworks | [07:49] |
| pyvpx | mzima no longer exists
pyvpx pours one out but yeah, it is apart of GTT | [07:57] |
| forgotten | i am able to get to everything from an earthlink connection in San Diego. | [07:58] |
| m0unds | bummer | [08:06] |
| forgotte1 | i know dont judge me :( | [08:09] |
| m0unds | still going on? | [08:09] |
| forgotte1 | m0unds: pretty sure ya. im at work now and things are working, but diff isp. | [08:09] |
| m0unds | all my graphing and stuff is fine, has been all night
via comcast in albuquerque m0unds shrugs | [08:10] |
| qbit | hola
anyone having issues connecting to arp from comcast? | [08:11] |
| forgotte1 | qbit: o
qbit: people be having issues from most major U.S. ISP's it's just tcp traffix | [08:11] |
| qbit | o/
yeah - icmp is going fine well good to know it isn't just me | [08:12] |
| forgotte1 | fo sho :) | [08:14] |
| *** | yoberi has joined #arpnetworks | [08:24] |
| pjs | Hi guys.. looks like I'm not the only one with issues
I have clients in GA who can't connect to their stuff in ARP http://dpaste.com/0C59HJN traceroute makes it to the network fine | [08:30] |
| qbit | yeah, tcp is not going
only icmp er udp | [08:32] |
| pjs | sexy
Just scrolled way up and read the chatter.. looks like it's a fun one | [08:33] |
| forgotte1 | i discovered it at like 2am :P | [08:34] |
| yoberi | I'm having some issues as well. first nagios alert came at 3am CST
Where can I get details of what's happened? I just joined the channel. Could someone copy the log to pastebin? | [08:35] |
| acf_ | can you get to http://irclogger.arpnetworks.com/ from somewhere? | [08:37] |
| *** | milki_ has joined #arpnetworks | [08:37] |
| brycec | I can from the office TWC connection
but that might be over v6 too | [08:37] |
| acf_ | yoberi: http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue | [08:38] |
| m0unds | i didn't see an AAAA record for it | [08:38] |
| yoberi | I can't seem to reach anything arpnetworks.com at the moment; yet my mobile phone can using the provider's data network | [08:38] |
| brycec | Indeed there isn't. Yep, I can reach it over v4. (from twc business) | [08:38] |
| yoberi | acf: thanks! | [08:38] |
| forgotten | appears to be a huge problem for a lot of folks. Im sure up_the_irons is all over it. | [08:52] |
| qbit | truf | [08:54] |
| milki_ | will we see a tweet from @arpnetworks or @bsdvps? | [08:55] |
| brycec | Good point/idea | [08:55] |
| RandalSchwartz | I've had one of neil's employees also mention that
can't get to his computers from some place in europe | [09:03] |
| *** | thestereobus_ has joined #arpnetworks
milki_ is now known as milki^ thestereobus_ has quit IRC (Client Quit) thestereobus has quit IRC (Ping timeout: 264 seconds) befrank has joined #arpnetworks | [09:05] |
| .... (idle for 16mn) | ||
| invader has joined #arpnetworks | [09:27] | |
| invader | Is there an issue with some VPS Services and the website? | [09:28] |
| acf_ | invader: http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue | [09:29] |
| invader | acf_, Thank you | [09:29] |
| brycec | Go acf_ ! | [09:30] |
| invader | Well thats fucked up. | [09:30] |
| forgotten | lol | [09:39] |
| up_the_irons | lol | [09:41] |
| brycec | a wild up_the_irons appears :D
Still not fixed :( | [09:41] |
| *** | bmacs has joined #arpnetworks | [09:42] |
| bmacs | Is something terrible happening right now? I can’t access the portal or my server. | [09:44] |
| brycec | Yes | [09:45] |
| bmacs | What is it? | [09:45] |
| up_the_irons | bmacs: bet you can ping it, just not tcp | [09:45] |
| bmacs | yup | [09:45] |
| brycec | http://anonymouse.org/cgi-bin/anon-www.cgi/http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-08-12,Tue https://twitter.com/arpnetworks/status/499235055323996160 | [09:46] |
| up_the_irons | bmacs: somewhere, something is not passing tcp traffic. i see drops around nLayer, opened a ticket with them. They gave me a response that made me want to scream (basically, "Why are you using IPv4 to test an IPv6 host?". W.T.F. our hosts are dual homed dumbass, it has NOTHING to do with the issue) | [09:47] |
| bmacs | Okay, I jsut took a different route to my host and was able to access it. ugh | [09:49] |
| up_the_irons | yeah same here | [09:50] |
| erratic | is it possible I'm having trouble getting through level3 ? | [09:50] |
| up_the_irons | erratic: are you seeing any drops around there? | [09:51] |
| erratic | yeah when i trace to google or linkedin
dropes at the 6th hop | [09:51] |
| up_the_irons | erratic: so from your vps to google? | [09:51] |
| erratic | yes
I can get through to other sites like yahoo seems to be back now lemme check yep Im back that was strange | [09:52] |
| up_the_irons | wait, so it works now? | [09:53] |
| erratic | yes | [09:53] |
| brycec | up_the_irons: btw, here's an mtr from a working network, both TCP and ICMP. I noted that the arpnet.cust.mzima router shows up in the ICMP but not the TCP - odd? http://sprunge.us/dfOK
(I don't have a working mtr --tcp on a non-working network) | [09:54] |
| up_the_irons | brycec: yeah, i'm seeing the same | [09:55] |
| brycec | I have no idea what it means though.
And the fact that it's a working connection suggests that' s just a red herring. | [09:55] |
| mike-burns | Whew; able to connect to my server from home, though not from work or cell phone. | [09:57] |
| brycec | Hooray
speaking of +o should someone announce the issue in the /topic too? (Thanks up_the_irons for tweeting it) | [09:57] |
| mike-burns | (This is in Sweden, BTW.) | [09:57] |
| acf_ | any idea why a TCP traceroute would have zero hops? | [09:58] |
| mike-burns | I still don't understand the /topic-sized announcement. "Some networks are down and we do not yet know why"? | [09:58] |
| brycec | mike-burns: Just something that people would see when they /join and ask "is there a problem?" | [09:59] |
| up_the_irons | gotta make some calls, brb | [09:59] |
| brycec | Something like "Upstream providers are dropping TCP traffic. We're working on it."
good luck, up_the_irons | [09:59] |
| *** | mike-burns changes topic to: Upstream providers are dropping TCP traffic. We're working on it. | VPS Services: http://www.arpnetworks.com/vps | FAQ: http://support.arpnetworks.com/faqs | Wiki: http://wiki.arpnetworks.com | Twitter: http://twitter.com/bsdvps | @up_the_irons is staff, other ops are long-time customers | Channel Log: http://is.gd/eWA5u | [09:59] |
| mike-burns | Whew that's a long one. | [09:59] |
| acf_ | definitely when I do an mtr --tcp -4 underworld1.no.quakenet.org zero hops are displayed
my arp metal box acts as a router, so when I do the same mtr from a machine routed through it, one hop is displayed hmm... looks like ARP routers don't show up in ICMP traceroutes anyway | [10:00] |
| brycec | o/ | [10:01] |
| acf_ | *TCP tracerouts
this is kind of strange... | [10:01] |
| qbit | whoa | [10:04] |
| acf_ | if I mtr --tcp -4 underworld1.no.quakenet.org, zero hops appear
I can see in the ICMP mtr that 4.71.143.105 (Level3) is the first hop outside of ARP mtr --tcp -4 4.71.143.105 works fine so TCP traffic destined for that host must have been dropped before it reached 4.71.143.105 ? but TCP traffic directed to 4.71.143.105 is passed? that host == underworld1.no.quakenet.org | [10:04] |
| brycec | brycec has no idea | [10:08] |
| RandalSchwartz | hmm. my mtr doesn't have --tcp
macports has only version 0.85 of mtr, it seems | [10:13] |
| acf_ | that's the one I'm using | [10:14] |
| RandalSchwartz | ahh - same as freebsd
but it doesn't have --tcp :) | [10:14] |
| acf_ | hmm... mine definitely does | [10:15] |
| RandalSchwartz | must be a fork
and sadly, without changing the version number to indicate that | [10:15] |
| acf_ | it's the Debian Jessie mtr package | [10:15] |
| RandalSchwartz | ahh - the github has --tcp
wow... --tcp was added to the docs in 2013 | [10:16] |
| acf_ | maybe it's not supported on freebsd? | [10:19] |
| RandalSchwartz | maybe the code thinks that all the world is linux
which of course, fails on the BSDs and OSX | [10:21] |
| up_the_irons | I noticed on FreeBSD's mtr, it simply craps out if I add "--tcp" (0 hops, wtf)
anyone have a Ubuntu 14.04 host on our network that I can run some mtr's on? I'm always a version behind, so I don't have --tcp in my mtr | [10:21] |
| *** | thestereobus has joined #arpnetworks | [10:22] |
| bmacs | yeah | [10:22] |
| up_the_irons | I also don't have a working mtr on a non-working host, lol | [10:23] |
| acf_ | up_the_irons: Linux mtr does the zero hops thing too | [10:23] |
| bmacs | I’ll make you a login | [10:23] |
| acf_ | I did a test with it earlier ^ | [10:23] |
| up_the_irons | acf_: hmm weird | [10:24] |
| erratic | acf_: hey :) hru | [10:24] |
| acf_ | oh hi erratic. I'm alright, yourself? | [10:24] |
| erratic | sitting here at home chilling and applying for some jobs | [10:24] |
| RandalSchwartz | RandalSchwartz mumbles "ziprecruiter.com" ) | [10:25] |
| erratic | RandalSchwartz: does it work in europe | [10:25] |
| RandalSchwartz | no sorry
works great in US and CA though | [10:25] |
| pjs | hey all.. any update? several clients beating down my path now :) | [10:26] |
| invader | Beat back.
I think some of use at in the same boat. | [10:26] |
| pjs | invader obviously ;) Just curious if there was any update (just getting into the office) | [10:27] |
| invader | If you ever dealt with Carriers its a bitch and a half. Give it time. | [10:27] |
| bmacs | up_the_irons, I jsut sent you an ssh login to my box if you need to run mtr on it | [10:27] |
| pjs | invader I have, and am, just asking if there's an update | [10:27] |
| plett | up_the_irons: Is shutting down BGP to nlayer an option, if they can't work out how to fix it? | [10:28] |
| invader | Do we 100% know what Carriers are effected? | [10:28] |
| acf_ | level3 is affected maybe?
seems the forward path to Comcast is via level3 | [10:31] |
| up_the_irons | pjs: no update yet | [10:32] |
| acf_ | and tcpdump says TCP packets don't arrive via the forward path | [10:32] |
| up_the_irons | plett: i shutdown all carriers in turn, last night, but no change :(
i'm putting together a very detailed report and going to send to NTT, Level 3, GTT, etc... | [10:32] |
| acf_ | but TCP packets via the return path (nlayer) are fine
brycec: does ARP to TWC go via Level3? | [10:33] |
| invader | Anything I can do from a carrier prospective? | [10:35] |
| brycec | acf_: last I tried, yes. Also ARP to Comcast | [10:35] |
| acf_ | yeah. looking a lot like Level3 related then?
same for underworld1.no.quakenet.org | [10:35] |
| up_the_irons | invader: which carrier perspective? | [10:36] |
| brycec | Is invader a carrier? Can we just blame you? :D | [10:37] |
| invader | I work for a telco in WI. I multi homed to ntd twc and comcast. | [10:37] |
| brycec | brycec blames invader | [10:37] |
| up_the_irons | lol | [10:37] |
| invader | invader crys | [10:37] |
| forgotten | forgotten 2nds the vote to blame invader | [10:37] |
| brycec | up_the_irons: Stupid question, because I'm sure you've already considered it, but are we sure it's not s7.lax having issues? | [10:38] |
| up_the_irons | invader: oh nice, can you do a "mtr -4rwc 10 --tcp arpnetworks.com" ? :) | [10:38] |
| brycec | With a nick like invader, how can we not? | [10:38] |
| up_the_irons | haha | [10:38] |
| *** | thestereobus has quit IRC (Quit: thestereobus) | [10:39] |
| up_the_irons | brycec: well, s7.lax appears OK, but I could try to engineer it so we go out s1.lax instead. but first, gotta finish sending this report to carriers... | [10:39] |
| invader | Hmm not a damn thing. Its like the route is not even there..
Let me connect into our Verizon part of the network | [10:39] |
| up_the_irons | invader: so what does it show? no hops at all? | [10:40] |
| invader | nothing. Just verizon just works.
one min | [10:41] |
| brycec | up_the_irons: Best as I can tell, it's the single most common data point. And the fact that this TCP filtering would happen on multiple carriers, vs. a single router... As illogical as it seems, it's the most logical scenario from where I'm sitting. | [10:41] |
| up_the_irons | i tend to agree | [10:42] |
| acf_ | seems to me that Level3 is the only one affected | [10:42] |
| brycec | brycec wonders what s1's uptime is | [10:42] |
| up_the_irons | lol | [10:42] |
| brycec | acf_: but not universally... | [10:42] |
| up_the_irons | brycec: s1.lax uptime is 5 years, 25 weeks, 22 hours, 4 minutes | [10:42] |
| acf_ | brycec: could you point me to an example please? | [10:43] |
| brycec | acf_: look in the scrollback for my sprunge.us link. :p
acf_: In short, from work, which is TWC "Business Class" I have no issues to ARP over nlayer/mzima, and the return mtr (not included) comes over L3 | [10:43] |
| *** | essact has joined #arpnetworks | [10:44] |
| acf_ | level3 -> trit? | [10:44] |
| brycec | trit not there (gathering mtr now) | [10:45] |
| acf_ | hmm yes... not all of level3 is affected | [10:45] |
| *** | essact has quit IRC (Client Quit) | [10:45] |
| acf_ | fwiw the level3 looking glass routes through gblx -> trit -> arp | [10:45] |
| brycec | acf_: http://sprunge.us/PXFD (ICMP because I have an old mtr on my VPS) | [10:47] |
| acf_ | ok thanks
so it looks like some of Level3 isn't affected but are there any non-Level3 routed destinations that are affected? can't wget http://level3.com fwiw | [10:47] |
| brycec | acf_: from? | [10:50] |
| acf_ | arp | [10:50] |
| brycec | ditto
up_the_irons: ^ (but I can from other networks including an "affected" comcast connection | [10:50] |
| acf_ | same | [10:51] |
| up_the_irons | hmm | [10:51] |
| brycec | So, whether or not related to ARP's issue, (but it probably is), L3 has some split-braining going on
brycec waits for up_the_irons to shut off the l3 session | [10:51] |
| up_the_irons | ok i blew away level3 | [10:53] |
| bmacs | lots of people tweeting about level3 problems
https://twitter.com/TVBroadcast/status/499243972221734912 | [10:53] |
| BryceBot | TWITTER: Anyone aware of a "a large Internetnetwork issue is affecting ISPs across the country, like Comcast, Cogent, Level3, Time Warner, etc.?" (Tue Aug 12 17:20:13 +0000 2014, retweeted 1 times) | [10:53] |
| bmacs | https://twitter.com/factor1/status/499249059433943040 | [10:53] |
| BryceBot | TWITTER: Good news is that our network issue seems to be clearing up as Comcast and Level3 are responding to the fix. (Tue Aug 12 17:40:26 +0000 2014, retweeted 1 times) | [10:53] |
| brycec | up_the_irons: suddenly working | [10:53] |
| up_the_irons | whoa | [10:54] |
| brycec | oh wait
nevermind, was v6 | [10:54] |
| up_the_irons | hate that | [10:54] |
| brycec | dualhoming biting me in the ass | [10:54] |
| mhoran | Heh.
Yeah, all my users are complaining but v6 works just fine! | [10:54] |
| brycec | hehe :D
i've had v4 drop out in the office for hours at a time before I noticed (dhcp issue) | [10:54] |
| mhoran | Had someone try the mtr command but it produced no output. I don't have access to the box so I can't debug. | [10:55] |
| bmacs | http://downdetector.com/status/level3 | [10:55] |
| acf_ | ok seems Comcast goes via ntt now
still no TCP packets going via Comcast -> ARP path | [10:55] |
| invader | Shit this explains the issues I am having in SunGard.. They just called L3 is broken and has been for the better part of a day. -_- Thanks for the call now.. Butt heads. | [10:55] |
| twobithacker | mtr --tcp isn't working from my home FreeBSD box, trying to fix that so I can supply another data point | [10:55] |
| acf_ | return path still ok | [10:56] |
| twobithacker | seeing the same problem from Comcast in NH though
IPv6 works, fwiw | [10:56] |
| brycec | up_the_irons: fwiw, curl level3.com from arp still not working, mtr shows it going ntt->level3, unable to ping | [10:56] |
| acf_ | same here. seems dropping level3 didn't fix the problem | [10:57] |
| mhoran | twobithacker: Yeah, my friends in MA can't connect via Comcast so probably the same route problem there. | [10:57] |
| up_the_irons | brycec: yeah it's taking the exact same path, just through NTT now ;) | [10:57] |
| mhoran | Whomp whomp. | [10:57] |
| brycec | (And return path ARP->Comcast is over NTT->Comcast) | [10:58] |
| acf_ | yep
that's still broken too though | [10:58] |
| twobithacker | huh, I can connect just fine from Comcast Business though | [10:58] |
| brycec | So Level3 is having problems... and Verizon has been lashing out at L3. I bet Verizon haxxored Level3!!
^ sarcasm twobithacker: yeah that's been observed | [10:59] |
| acf_ | ironically, my Verizon DSL line is the only residential connection I've seen unaffacted | [10:59] |
| brycec | Seems home TWC and Comcast (among others) are affected, business customers are okay | [10:59] |
| mhoran | I'm fine on TWC. :)
In NYC. Also Verizon FiOS at my office is fine. | [11:00] |
| acf_ | it's interesting that's it's just tcp | [11:00] |
| brycec | brycec nods | [11:01] |
| mhoran | Yeah. | [11:01] |
| acf_ | it seems to imply that somebody is discriminating tcp traffic in some way or another | [11:01] |
| brycec | Again, I blame the whole Netflix/Verizon debacle :P
"HOW DO YOU LIKE YOUR NETFLIX NOW, BITCHES?!?" "YOU THOUGHT WE WERE THROTTLING? NOOOOOOW WE'RE THROTTLING!" etc | [11:01] |
| acf_ | lol | [11:02] |
| brycec | (except netflix.com is working for me) | [11:03] |
| acf_ | looking at just the forward path ARP -> Comcast residential | [11:03] |
| brycec | (both v4 and v6) | [11:03] |
| acf_ | it's broken
both with ntt and level3 | [11:03] |
| brycec | Well I have other work to get done... Best of luck, #arpnetworks | [11:04] |
| acf_ | see you later brycec | [11:04] |
| mhoran | If it's just TCP, maybe some sort of traffic shaping failure of sorts? | [11:04] |
| acf_ | I was thinking about that...
but where? | [11:05] |
| up_the_irons | brycec: LOL | [11:05] |
| mhoran | Yeah this is far more exciting than the admin tool I was just writing ... | [11:05] |
| brycec | s/ail/feat/ | [11:05] |
| BryceBot | <mhoran> If it's just TCP, maybe some sort of traffic shaping ffeature of sorts? | [11:05] |
| up_the_irons | hmm... traffic shaping, interesting.. | [11:05] |
| erratic | http://www.secureworks.com/cyber-threat-intelligence/threats/bgp-hijacking-for-cryptocurrency-profit/ | [11:06] |
| up_the_irons | acf_: BryceBot mhoran erratic invader : see any change? | [11:10] |
| mhoran | mhoran checks. | [11:10] |
| erratic | checking
first time using mtr | [11:11] |
| acf_ | testing ARP -> Comcast path via NTT ... still broken for tcp | [11:11] |
| mhoran | No dice. | [11:11] |
| up_the_irons | ok | [11:11] |
| acf_ | just talked with a friend on another connection
says he can't connect to underworld1.no.quakenet.org either but it works from Comcast | [11:12] |
| up_the_irons | i still find it weird that level3.com is inaccessible | [11:13] |
| erratic | yeah I haven't noticed anymore outage problems since earlier
was having trouble getting to cloudfront, linkedin, google, things like that | [11:14] |
| acf_ | erratic: from what connection? | [11:15] |
| erratic | just my vpn tunnel
I have everything routed over my 206.125.168.65 addr oh when i did traceroutes on stuff it was timing out at like 6 hops in level 3 then it just started working again about 5 minutes later I wonder if theres a way I can continuously check with mtr like if I wanted to watch for changes I guess that wouldn't make a lot of sense now that I think about it | [11:16] |
| acf_ | mtr polls continuously, and should show changes as they happen | [11:20] |
| erratic | weird sudo mtr --report -4 google.com
it just runs once and exits for me | [11:21] |
| acf_ | probably the --report | [11:21] |
| erratic | ah | [11:21] |
| up_the_irons | just shutdown Trit Networks BGP session, stab in the dark | [11:21] |
| forgotten | you better fix all these things outside of your control up_the_irons | [11:22] |
| erratic | oh nice
lol | [11:22] |
| up_the_irons | lol | [11:22] |
| erratic | acf thx | [11:22] |
| acf_ | still appears broken for the Comcast thing | [11:23] |
| forgotten | im writing a blog post as we speak bad mouthing your services | [11:23] |
| acf_ | trit isn't in my comcast path fwiw
np erratic | [11:23] |
| up_the_irons | yeah but sometimes... ya never know ;) | [11:23] |
| bmacs | lol: https://twitter.com/ellisinzion/status/499255527990239232 | [11:26] |
| BryceBot | TWITTER: @Level3 Yes you do have a major core IP outage right now. Stop saying there is no issue. Multiple big voice providers are down cold. (Tue Aug 12 18:06:08 +0000 2014, in reply to @Level3) | [11:26] |
| up_the_irons | OL
LOL | [11:27] |
| erratic | ae-3-3505.ear1.Dallas1.Level3.net? | [11:27] |
| pyvpx | I totally just tweetd about this
neat | [11:29] |
| acf_ | so... ARP -> Comcast forward path via either NTT or Level3 is broken
which kind of indicates that the issue doesn't lie entirely with NTT or Level3 | [11:30] |
| plett | acf_: But from .uk using NTT all the way to ARP works fine | [11:30] |
| acf_ | but other things (TWC) are broken too | [11:30] |
| up_the_irons | plett: but what is your return path | [11:31] |
| acf_ | yeah, pretty much all the NTT stuff I've seen works fine | [11:31] |
| plett | up_the_irons: I'll check
up_the_irons: NTT as well | [11:31] |
| acf_ | ec2 (ntt both ways) is fine | [11:31] |
| plett | acf_: What's the path in both directions for the broken arp <-> comcast ? | [11:33] |
| acf_ | ARP -> Comcast is via NTT now, was Level3 earlier, both were broken
Comcast -> ARP is via nlayer | [11:33] |
| plett | Ahh. nlayer | [11:33] |
| acf_ | I've been testing just the ARP -> Comcast part of the path
the nlayer part looks fine | [11:33] |
| pyvpx | acf_: you never receive acks?
it's just TCP for everyone, yes? | [11:34] |
| acf_ | I do a tcpdump on the comcast side
I try to connect from the arp side no packets come through | [11:34] |
| pyvpx | UDP packets? or no packets | [11:34] |
| acf_ | going the other way works though
I use tcpdump -ni eth3 port 6668 and telnet [hostname] 6668 | [11:34] |
| up_the_irons | hmmm...
HRMMM. | [11:35] |
| acf_ | UDP packets are fine | [11:35] |
| up_the_irons | any change just now? i can actually check my email | [11:35] |
| acf_ | yep | [11:35] |
| bmacs | yup | [11:35] |
| acf_ | packets are getting through to comcast now | [11:35] |
| bmacs | Im up | [11:35] |
| invader | up_the_irons, comcast works now
same with twc | [11:36] |
| acf_ | comcast is symmetric nlayer now | [11:36] |
| up_the_irons | the thing with NTT / Level3 is, a lot of NTT routes go through Level3 anyway | [11:36] |
| acf_ | is it possible that something is dropping things with asymmetric paths? | [11:37] |
| up_the_irons | so what i did was, lowered s7.lax default route local pref so most traffic goes out via s1.lax directly, not hitting s7.lax. s7.lax has Level 3 and NTT.
acf_: that would take stateful filtering i believe and is pretty rare on regular transit links | [11:37] |
| acf_ | wow. everything is via nlayer now :P
even any2ix stuff nlayer to the rescue! (surprisingly) | [11:39] |
| up_the_irons | yeah
not going through s7.lax kills the rest of my network ;) (no peers, L3 or NTT) | [11:40] |
| erratic | oooo | [11:41] |
| up_the_irons | while this works, it seems like a shitty solution
unless the problem really is with Level3, in which case we can wait it out now | [11:42] |
| acf_ | I don't think it is... ARP -> NTT -> Comcast doesn't work... | [11:43] |
| up_the_irons | acf_: still? | [11:43] |
| acf_ | it's not NTT anymore, so not it works | [11:44] |
| up_the_irons | oh wait.. couldn't be cuz ARP -> NTT isn't possible right now | [11:44] |
| acf_ | * now it works
it didn't work both when it was ARP -> NTT -> Comcast and ARP -> Level3 -> Comcast | [11:44] |
| up_the_irons | ok
well w..t.f | [11:44] |
| erratic | Im confuzled,
how does it kill the rest of your network not that it matters just curious this stuff is interesting to me | [11:46] |
| acf_ | basically up_the irons took the s7 router out of the network | [11:47] |
| erratic | right | [11:47] |
| acf_ | which is the connection between you and any2ix, ntt, and level3 | [11:47] |
| erratic | I gathered that much does that mean none of the networks including mine will be reachable? | [11:47] |
| acf_ | so without s7, all you get is nlayer | [11:47] |
| up_the_irons | erratic: s7.lax is a bigger router and carries more routes. usually, all traffic goes to it. but now i'm diverting traffic to go directly out s1.lax. | [11:47] |
| acf_ | it just means they'll take a diffetent path | [11:48] |
| up_the_irons | and yeah, basically what acf_ said | [11:48] |
| erratic | oh right like if nlayer goes down... | [11:48] |
| up_the_irons | i'm tempted to reboot s7 | [11:49] |
| acf_ | up_the_irons: so can you do testing with s7, static routes, etc... now that you've diverted traffic? | [11:49] |
| up_the_irons | acf_: well, technically yes, but hard to test when no traffic is going to it ;) | [11:50] |
| acf_ | just out of curiosity, does ios have a tcpdump equivilent?
now that there isn't any traffic, would it be more realistic to debug with tcpdump? | [11:53] |
| up_the_irons | well, what i would do is use a SPAN port and then dump that port. i have one set up already. | [11:54] |
| erratic | like switch port mirroring? | [11:56] |
| up_the_irons | yup | [11:56] |
| erratic | that is handy | [11:56] |
| up_the_irons | yup, purely meant for diagnostics
i can copy all traffic from any physical port or SVI (VLAN) and then just watch it from a regular Linux box with tcpdump | [11:56] |
| erratic | Im not sure how much overhead that would be, probably too much
but you could probably use it with snort assuming the switch or whatever can handle it, without creating a single point of failure | [11:57] |
| up_the_irons | yeah
ok, s7.lax is getting a kick in the butt (reboot) | [11:57] |
| acf_ | sounds like a good idea :P | [11:58] |
| m0unds | womp womp | [11:58] |
| RandalSchwartz | I saw a momentary drop
I presume that was a shuffling? | [11:58] |
| *** | fink has joined #arpnetworks | [11:59] |
| up_the_irons | .... | [12:00] |
| *** | neish has quit IRC (Read error: Connection reset by peer)
neish has joined #arpnetworks | [12:00] |
| neish | wat | [12:01] |
| erratic | I lost talk.google.com for like a sec | [12:03] |
| RandalSchwartz | ok
yeah... my connection went dead for about 10 seconds | [12:03] |
| erratic | yeah I think I lost everything for a secondlet
laggggggg | [12:04] |
| up_the_irons | s7 reboot is done, i think i'm going to send traffic to it again and see what happens | [12:09] |
| forgotten | whats the verdict? :D | [12:12] |
| mhoran | Guilty. | [12:12] |
| forgotten | invader: see ^
and the sentense? | [12:13] |
| pjs | looks like it's getting better | [12:14] |
| up_the_irons | how's it looking for u guys now? | [12:14] |
| erratic | I need to consume this alcoholic beverage | [12:14] |
| m0unds | erratic: alllllcohol | [12:14] |
| acf_ | arp -> ntt -> comcast seems to work now | [12:15] |
| erratic | glug | [12:16] |
| acf_ | so it was s7 being weird? | [12:16] |
| up_the_irons | then i'm going to throw fucking s7 into the dumpster, gaah! | [12:16] |
| erratic | what sort of hardware is s7 | [12:16] |
| m0unds | up_the_irons: you just can't catch a break with that thing | [12:17] |
| up_the_irons | srsly | [12:17] |
| m0unds | but that's why shotguns exist | [12:17] |
| up_the_irons | erratic: Cisco 6506 w/ sup720-3bxl
i've had a replacement Sup in the cage for months now :( | [12:17] |
| erratic | Im just curious, couldn't you use a linux box to do that | [12:18] |
| up_the_irons | in the triage order of things, replacing it just wasn't high on the list. but it is now. | [12:18] |
| m0unds | up_the_irons: did you ever swap the sup, or is the one in the cage the one you got to replace it when it did that weird CEF barf thing? | [12:18] |
| erratic | that thing looks heavy | [12:19] |
| up_the_irons | erratic: well you wouldn't get any hardware acceleration and the pps is really low in comparison. so basically, a linux box would melt under a DDoS
m0unds: yeah it's the replacement for the other issue i'm going to re-enable level 3 now | [12:19] |
| erratic | yeah Im reading the specs on it now lol | [12:20] |
| acf_ | why just tcp? | [12:21] |
| up_the_irons | man the ipv4 routing table is just getting too large :( | [12:21] |
| acf_ | and *some* tcp too? | [12:21] |
| up_the_irons | acf_: no. fucking. idea. ;) | [12:21] |
| erratic | wow jeez | [12:21] |
| forgotten | im sure if it's working for others, it's working for at&t too | [12:22] |
| up_the_irons | i think i need to stop being married to the (now) old idea that carrying a full routing table is *that* much better | [12:22] |
| invader | Sorry I broke the Internet | [12:23] |
| forgotten | GUILTY! | [12:24] |
| acf_ | on a slightly unrelated note, are you planning to announce to level3 at some point? | [12:24] |
| erratic | Webster: we have a lot of 6500 ciscos and have recently grown up to be able to get the ASR 9000s
ooooooo | [12:27] |
| m0unds | ugh, ios-xr | [12:29] |
| jpalmer | so, my first cisco class starts monday. | [12:31] |
| erratic | and ends pretty much never if you take into account how esoteric it all is | [12:32] |
| m0unds | haha
i hope you learn a lot about serial stuff, because that shit still exists in ios haha | [12:32] |
| erratic | m0unds: just a sec | [12:32] |
| jpalmer | my vmware class also starts monday | [12:33] |
| m0unds | that sounds like more fun | [12:33] |
| jpalmer | indeed
I'm only taking the cisco, cuz my job is making me. I don't find routing or switching to be intersting at all. I'd be far more interested in the security tract. | [12:33] |
| erratic | m0unds: this is how I decided to upgrade my bios http://3.bp.blogspot.com/-GDEA1qZ0thk/U-d-u4aUM8I/AAAAAAAAPTo/tgcCHYdR5c4/s1600/Screenshot+-+03272013+-+05:40:47+AM.png
just because lol y not | [12:34] |
| m0unds | hahahahah
nice | [12:35] |
| up_the_irons | m0unds: hey i use serial all the time, the 2612's are great for out-of-band access to everything :) I have 32 serial ports with breakout cables on one of 'em. | [12:37] |
| m0unds | oh, i meant like frame relay serial :) | [12:37] |
| up_the_irons | oh
lol | [12:37] |
| m0unds | i have an AS2516-RJ (iirc) that takes rollover cables terminated w/rj45s
it's great as a terminal server for my crisco lab | [12:38] |
| erratic | jpalmer: I have pretty much my lack of interest in anything except computers but more importantly the people I've surrounded myself with who have been willing to help to thank
I got kicked out of school | [12:38] |
| m0unds | i got my employer to approve me bringing it to work so i wouldn't have to try and store all that gear at home :) | [12:39] |
| erratic | haha yeah I love piling up my crap at work
makes me feel comfortable to have crap around me | [12:39] |
| m0unds | i racked mine up and told my coworkers to learn something | [12:40] |
| erratic | m0unds: definitely
yeah I used so much of my own hardware at work | [12:40] |
| m0unds | since i'm hoping i'll be able to leave here in a year or so, and since they don't have another network engineer, they'll be utterly SOL if they don't | [12:40] |
| erratic | well at my last good job
I haven't had a decent job in a while finally starting to get interested in stuff again though | [12:41] |
| jpalmer | m0unds: I do the same thing. and I teach them how to make and restore backups first.. so that if someone changes it when they are done, they can instantly revert it back to their last configuration. gives them a LOT of backup/restore practice, and makes sure everyone can use the lab without stepping on other peoples toes. | [12:42] |
| erratic | Ive had crap jobs and a few short term contracts and stuff but nothing really
my last job really spoiled my appetite for work was laid off a year ago today and Im just now like, ok I'm moving to a new place and I don't know anybody and I have no idea what to expect and I think it will be challenging for me socially | [12:42] |
| jpalmer | erratic: where are you moving to? | [12:44] |
| erratic | Netherlands
Im in Greece atm | [12:44] |
| jpalmer | thats.. a move, for sure. hehe | [12:44] |
| m0unds | ah, cool - my grandparents are from the netherlands | [12:44] |
| jpalmer | my wife and I are talking about moving from florida to charlotte, NC | [12:45] |
| erratic | NC isn't too bad | [12:45] |
| m0unds | charlotte is nice - my dad lives in burlington, just outside greensboro | [12:45] |
| erratic | need a car for sure
I like living in the city too much but I do appreciate living out in the country and I miss it sometimes | [12:45] |
| jpalmer | we'd like in one of the outskirt areas, for sure. horses and such
s/like/live/ | [12:46] |
| BryceBot | <jpalmer> we'd live in one of the outskirt areas, for sure. horses and such | [12:46] |
| erratic | I lived in creedmoor outside of RD
for awhile was like 16 or 17 long enough to make friends then move away again I really want to believe despite how much i hate redhat it would be really hard (definitely competitive) to work there but probably pretty interesting especially if you can get a good systems engineering job they pretty much birthed xen and now from what I've been reading a lot of the work on qemu/kvm looks pretty sweet and I'm running a lot of the more recent stuff | [12:46] |
| *** | invader has quit IRC (Remote host closed the connection) | [12:50] |
| erratic | actually found a vid this morning exactly what I have except the guy has much faster hardware
https://www.youtube.com/watch?v=37D2bRsthfI | [12:51] |
| BryceBot | YouTube Tech: "Qemu/KVM + vfio = Virtual machine for gaming with 95% of native performance = no more dual booting" by blu3bird84 (2m 8s), 18,965 views, 222 likes and 4 dislikes. Uploaded 2014-02-02T19:42:29.000Z. | [12:51] |
| up_the_irons | anyone seen any issues in the last 30 mins?
things are working for me... | [12:57] |
| forgotten | wish i could test :( | [13:01] |
| bmacs | no issues here. been clear since the reboot | [13:01] |
| up_the_irons | roger that | [13:02] |
| ess | Yep. No issues here.. | [13:06] |
| *** | mus1cb0x has joined #arpnetworks | [13:07] |
| mus1cb0x | any issues with kvm 14 today?
krv14, that is | [13:07] |
| up_the_irons | can someone put the topic back to the way it was :) | [13:08] |
| mus1cb0x | i tried connecting several times a couple of hours ago without success | [13:08] |
| m0unds | seems the NTT comcast stuff is finally getting better | [13:08] |
| up_the_irons | mus1cb0x: was probably the general network issue we had | [13:08] |
| m0unds | just looked at my arp latency graph, and it's looking much healthier than a month ago | [13:08] |
| mus1cb0x | ah, i see the topic
how long was the outage? | [13:08] |
| up_the_irons | since like 1AM, but it was very selective on certain routes. Cacti showed most traffic was getting in/out. | [13:09] |
| mus1cb0x | ouch, what was the cause? | [13:09] |
| mhoran | Looks good here. | [13:10] |
| *** | invader has joined #arpnetworks
fink has quit IRC (Quit: fink) | [13:11] |
| up_the_irons | mus1cb0x: s7.lax being stupid | [13:12] |
| *** | mhoran changes topic to: VPS Services: http://www.arpnetworks.com/vps | FAQ: http://support.arpnetworks.com/faqs | Wiki: http://wiki.arpnetworks.com | Twitter: http://twitter.com/bsdvps | @up_the_irons is staff, other ops are long-time customers | Channel Log: http://is.gd/eWA5u | [13:13] |
| mhoran | (Had to get on a desktop for that one.) | [13:13] |
| mus1cb0x | stupid? | [13:13] |
| forgotten | aka not smart | [13:14] |
| mus1cb0x | sarcasm isn't helpful
nor is "stupid" when discussing network outages i have to justify the outage and i'm not going to use stupid and risk my job | [13:14] |
| forgotten | this isn't an offical forum
go put in a ticket | [13:14] |
| mus1cb0x | heh, geez
at least you're not op'd, because you would be the wrong person to represent arp | [13:14] |
| forgotten | forgotten takes a bow | [13:15] |
| up_the_irons | play nice now | [13:16] |
| mhoran | up_the_irons: It does seem like there were more widespread issues than just with your router, though? I definitely experienced issues with non-ARP connectivity this morning.
Perhaps all related in some way? e.g. a crazy routing table causing issues on routers across the nets? | [13:16] |
| up_the_irons | mus1cb0x: There was something strange with TCP traffic not being passed through to certain destinations over s7.lax. UDP was fine. | [13:17] |
| mhoran | I guess the fact that it was TCP only kind of makes my thought make no sense.
Nor does that sentence make sense. | [13:17] |
| brycec | Huzzah Internet working! | [13:17] |
| up_the_irons | mhoran: hmm.. that's a good point about the routing tables, maybe it is all related in some way | [13:17] |
| brycec | oh boy, I missed a lot | [13:17] |
| invader | The TCP thing really sounds like someone played with acls. | [13:18] |
| up_the_irons | mhoran: I didn't realize you had non-ARP connectivity problems too | [13:18] |
| mus1cb0x | did anyone else see the article about routing table hijacks lately to commandeer bitcoin mining? | [13:18] |
| mhoran | up_the_irons: Yeah I was having trouble with some things before I left from home this AM, over IPv4 from Time Warner. | [13:18] |
| up_the_irons | mus1cb0x: someone pasted that, yeah
mhoran: hmm... interesting | [13:18] |
| invader | up_the_irons, L3 had an confirmed issue was a few asn's/ | [13:18] |
| mus1cb0x | any chance that could be a cause? | [13:18] |
| up_the_irons | some type of BGP poisoning? I suppose it's possible | [13:19] |
| mus1cb0x | http://www.reddit.com/r/cryptodev/comments/2d0bzx/bgp_hijacking_for_cryptocurrency_profit/ | [13:19] |
| jpalmer | up_the_irons: we hit 512,000 active routes today, were you aware of that? | [13:21] |
| invader | up_the_irons, From the sounds of it, They only said "route issue" Getting a straight answer out of them is like try to date Misty Dawn. | [13:22] |
| up_the_irons | jpalmer: lol, someone just PM'd me, "related to the 512k routes issue?"
perhaps that was it... i'm showing 499K right now i'm going to be doing a route haircut tonight maybe kill all /24's | [13:24] |
| jpalmer | up_the_irons: lots of routers with the 512k limit have been having issues today. | [13:25] |
| m0unds | sup 720 should handle 1m routes, iirc | [13:25] |
| invader | invader hands up_the_irons a scissors
Happy grooming | [13:25] |
| m0unds | might be thinking of the 2t though | [13:25] |
| up_the_irons | danke | [13:25] |
| mhoran | Sure should. | [13:26] |
| invader | what kind of gear you guys using at your edge? | [13:26] |
| up_the_irons | m0unds: jpalmer : i have the -3bxl, so it does 1M routes. however, i have two full tables, so right about there | [13:26] |
| m0unds | ohhhh
gotcha | [13:26] |
| up_the_irons | CEF is actually bitching at me in the logs... lol | [13:26] |
| m0unds | haha
i hate cef such a kludgy thing | [13:27] |
| mhoran | I could see CEF blowing up and causing all sorts of weird issues, in addition to the missing routes.
Still don't understand why UDP would work, unless it took a different path somehow. | [13:29] |
| m0unds | yea, but it usually fails by dumping forwarding onto the cpu
then the cpu shits its pants and the whole chassis stops passing traffic because of the load or crashes, depending haha | [13:29] |
| mhoran | http://status.livestream.com/ -- so it looks like my ex startup had routing troubles today, too. :p
All the competent network techs left, so I'm not surprised their router crashed. Was also a 720-3bxl. That I doubt anyone has maintained in two years. | [13:30] |
| up_the_irons | well that was easy:
s7.lax(config)#mls cef maximum-routes ip 768 Maximum routes set to 786432. Configuration will be effective on reboot. looks like another reboot will happen in the very near future... (tonight maybe) ;) | [13:36] |
| forgotten | do it now while only half of people think its fixed!
:D | [13:36] |
| up_the_irons | hahahhah
nice:) | [13:37] |
| m0unds | boo cef, boo | [13:38] |
| up_the_irons | and yeah, CEF blowing up means the router falls back to software switching, which will melt it | [13:38] |
| m0unds | yeah, i wonder if cef was loaded and it was unable to take on any flows | [13:38] |
| up_the_irons | actually s1.lax did that once, i hit the 128K limit. it went full on to CPU switching. some of you may remember that outage. but hey, it didn't crash, still 5+ year uptime! :) | [13:39] |
| m0unds | hahaha
s1's a 45xx right? | [13:39] |
| up_the_irons | yup | [13:39] |
| brycec | brycec remembers, he thinks | [13:39] |
| jpalmer | up_the_irons: so it was hitting the 512k limit? | [13:39] |
| m0unds | my buddy @ easystreet up in oregon had a similar thing w/a 45xx killing CEF and dying horribly overnight one night | [13:39] |
| up_the_irons | jpalmer: i don't think so, or else *nothing* would have gotten through | [13:40] |
| mhoran | Yeah, that's happened to me too, when I foolishly tried to load the entire routing table on a 4500. :p
Upstream route filter was misconfigured and I didn't guard against that. | [13:42] |
| up_the_irons | ouch | [13:44] |
| *** | raptelan_ has quit IRC (Quit: Page closed) | [13:46] |
| mus1cb0x | preemptive optimization doesn't exist in networking :P
er, premature | [13:48] |
| mhoran | Heh. | [13:56] |
| forgotten | http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-major-outage-across-us-network-providers | [14:01] |
| *** | fink has joined #arpnetworks
invader has quit IRC (Ping timeout: 245 seconds) | [14:01] |
| ...... (idle for 29mn) | ||
| thestereobus has joined #arpnetworks
qbit has quit IRC (Quit: leaving) | [14:30] | |
| ess has quit IRC (Quit: Page closed) | [14:41] | |
| .... (idle for 19mn) | ||
| thestereobus has quit IRC (Ping timeout: 250 seconds)
sjackso has joined #arpnetworks | [15:00] | |
| thestereobus has joined #arpnetworks | [15:07] | |
| thestereobus has quit IRC (Quit: thestereobus) | [15:15] | |
| .... (idle for 16mn) | ||
| m0unds | http://www.cidr-report.org/as2.0/#General_Status | [15:31] |
| mhoran | Number of ASes seems to be approaching a number that someone may have hardcoded into a system, too. :p | [15:38] |
| m0unds | yeah, haha | [15:39] |
| sjackso | I heard a rumor of a bgp apocalypse and I figured #arpnetworks would tell me whether or not to panic | [15:40] |
| m0unds | bgpocalypse | [15:48] |
| ....... (idle for 31mn) | ||
| *** | fink has quit IRC (Quit: fink) | [16:19] |
| ............. (idle for 1h4mn) | ||
| RandalSchwartz | wow. ziprecruiter.com has gotten about 5 times faster in the past few days for job searches.
my efforts are paying off. | [17:23] |
| ...... (idle for 26mn) | ||
| *** | qbit has joined #arpnetworks
qbit is now known as Guest18868 Guest18868 is now known as qbit_ | [17:50] |
| qbit_ has quit IRC (Quit: leaving) | [18:00] | |
| ....... (idle for 34mn) | ||
| up_the_irons | m0unds: sjackso : lol | [18:34] |
| staticsafe | so much fuss for so little
and of course media blows it up | [18:38] |
| *** | qbit has joined #arpnetworks
qbit is now known as Guest87335 Guest87335 is now known as qbit | [18:43] |
| qbit | hola
:D so - are the weird issues fixed? | [18:45] |
| *** | qbit is now known as Guest67943 | [18:46] |
| Guest67943 | from earlier | [18:46] |
| milki | alive! | [18:46] |
| *** | Guest67943 is now known as qbit | [18:47] |
| qbit | i seem to be having some dns issues - not sure if it's residual or not
could also be from the upgrade i stupidly decided to do mid day :P | [18:49] |
| milki | aha | [18:49] |
| qbit | host can resolve things fine.. but ping doesn't
making me rage | [18:49] |
| brycec | qbit: connectivity was resolved about 7 hours ago | [19:00] |
| qbit | k
it's totally fucked up - ping doesn't resolve some hosts.. chat.freenode.net.. etc - but it does for others and host works for all | [19:01] |
| brycec | qbit: reboot?
Seems like the host dns resolver cache is fubar'd but host does a fresh fetch iirc or at least a direct connection, bypassing the normal caching | [19:01] |
| qbit | yeah - reboot does nothin to fix it | [19:02] |
| ..... (idle for 21mn) | ||
| *** | qbit has quit IRC (Quit: leaving)
fink has joined #arpnetworks | [19:23] |
| fink | RandalSchwartz: i meant, are you up to freebsd version 9 or 10 | [19:26] |
| RandalSchwartz | when did you "meant"?
I'm still on 8.4 probably 10.x during some spare downtime in december after I first do it on insightcruises.com machines. :) | [19:33] |
| fink? | [19:41] | |
| fink | yo RandalSchwartz | [19:42] |
| RandalSchwartz | was answering your questions | [19:42] |
| fink | i said: fink: yo RandalSchwartz, how's it hanging. are you up to 9 or 10 with your vps? | [19:42] |
| RandalSchwartz | never saw that
or maybe long ago | [19:42] |
| fink | yea, yesterday ;) | [19:42] |
| RandalSchwartz | so there's my answers | [19:42] |
| fink | i'm wary of moving my zfs on root from 8.4 to 10 | [19:43] |
| mhoran | Sounds exciting! | [19:43] |
| *** | qbit has joined #arpnetworks
qbit is now known as Guest13111 Guest13111 is now known as qbit | [19:50] |
| RandalSchwartz | yeah - there's definitely some possible issues there
which is why I want to try it on $client's computers first :) | [19:55] |
| *** | fink has quit IRC (Quit: fink) | [19:56] |
| RandalSchwartz | there's some settings for internet in /etc/rc.conf that change, I'm told
and if I get those wrong, I don't get back in to the box :) | [19:56] |
| mhoran | Yup, there's that.
Though it's not too crazy. | [20:07] |
| hazardous | RandalSchwartz: i'm actually somewhat curious how much business/interest there is in insightful holiday packages
i never really thought of it before | [20:08] |
| m0unds | https://supportforums.cisco.com/document/12202206/size-internet-global-routing-table-and-its-potential-side-effects lol | [20:09] |
| hazardous | http://bgr.com/2014/08/12/comcast-fcc-commissioner-clyburn-dinner/ lol | [20:10] |
| RandalSchwartz | curious in what way?
these aren't holiday packages if you'd spend 4 days at a hotel in $random_city to attend a conf, why not 7 days on a cruise ship to do the same thing, but with better speakers? | [20:12] |
| hazardous | i dunno, i just never thought of that
i assumed most techies would rather book their own stuff and explore like a local or maybe that's just me | [20:13] |
| RandalSchwartz | yeah, most people haven't, which is why Captain Neil makes some bucks | [20:14] |
| hazardous | forgot about the conf part | [20:14] |
| RandalSchwartz | primarily, these are conferences
they just happen to be meeting on a ship instead of a hotel lots of advantages to that | [20:14] |
| m0unds | m0unds is sick of conferences in vegas | [20:14] |
| RandalSchwartz | right | [20:15] |
| m0unds | all of my industry conferences are in vegas
there's one that was in san diego once but never again | [20:15] |
| RandalSchwartz | and one big plus... the speakers are in the same BARS and same DINNER as you are
unlike land conferences, where they all scatter. and our speakers are expected to mingle with the attendees at those. imagine the conferences you've attended where you could bump into one of the keynote speakers later at the bar... or even on a shore excursion... a few hours in the same small group with them | [20:15] |
| *** | r0b has joined #arpnetworks | [20:17] |
| hazardous | so you have people actually interested in the subject there as opposed to forced to attend by corporate | [20:17] |
| RandalSchwartz | geez... you must have a twisted view of what we do!
I don't even know how to answer that. "I am interested in X" "Oh look... a cruise where speakers will be talking about X" "I want to go" [if employed] "Employer, can you sponsor me to this conference?" [if retired] "Let's see if my travel budget covers this" That's how people get on Nobody is *forced* "Oh gawd... FORCED to go on a CRUISE!" :) see how silly that sounds now? RandalSchwartz wanders off for a bit while relocating IRL | [20:19] |
| m0unds | i think hazardous was just pointing out that people attending a conf on a cruise are more likely to be going because they want to and not because someone forced them to
at least that's how it read to me | [20:25] |
| ...... (idle for 27mn) | ||
| *** | r0b has quit IRC (Ping timeout: 255 seconds)
r0b has joined #arpnetworks | [20:52] |
| r0b has quit IRC (Ping timeout: 240 seconds) | [21:02] | |
| ......... (idle for 41mn) | ||
| thestereobus has joined #arpnetworks | [21:43] | |
| thestereobus_ has joined #arpnetworks
thestereobus has quit IRC (Ping timeout: 264 seconds) thestereobus_ is now known as thestereobus | [21:53] | |
| ................. (idle for 1h23mn) | ||
| mnathani | what was that command to run an mtr report with output to a text format for easy copy /paste
have the upstream TCP transit issues been resolved? | [23:18] |
| thestereobus | I was seeing intermittant failures earlier tonight but they didn’t last long
mtr -4 --tcp -rc 10 <your-IP> | [23:20] |
| *** | thestereobus has quit IRC (Ping timeout: 246 seconds) | [23:31] |
| up_the_irons | mnathani: add -w so it doesn't cut off hostnames (wide format) | [23:34] |
| *** | thestereobus has joined #arpnetworks | [23:41] |
| befrank has quit IRC (Quit: Page closed) | [23:46] | |
| ↑back Search ←Prev date Next date→ Show only urls | (Click on time to select a line by its url) |
