#arpnetworks/ 2015-09-29,Tue

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***qbit has quit IRC (Quit: Lost terminal) [00:38]
.... (idle for 16mn)
mkb has quit IRC (Remote host closed the connection) [00:54]
mrsaintany outage now?
my traceroute ends up at
9 GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86) 203.131 ms 202.774 ms 202.614 ms
10 * * *
and my node down :(
[00:59]
mercutiois your host on kvr04? [01:00]
mrsaintno idea. :) [01:00]
mercutioi'd say it probably is [01:01]
mrsaintyeah :) [01:01]
mercutioplanned maintenance
it should be back in a few minutes
literally
[01:01]
mrsaintI cant see it in the portal where it is located :)
aah now I see. Yes its on kvr04 if its the same as vnc that I would expect it to be :)
[01:02]
mercutiois it back now?
yes it is
[01:03]
mrsaintyes [01:03]
mercutiocool [01:04]
mrsaintlet me see if my reboots worked :)
do actually seems soo :)
[01:04]
mercutiosweet [01:05]
mrsaintbut somehow quagga starts on reboots on empty conf [01:06]
mercutiochange to bird [01:06]
mrsaintso I have to paste conf into it again soo wierd.. :)
Might do in future :)
[01:06]
mercutiois it using Quagga.conf or whatever [01:06]
***tabthorpe has quit IRC (Remote host closed the connection) [01:06]
mrsaintJust that I have 10 nodes with same setup now and it all is good. [01:06]
mercutioquagga's config is messy [01:06]
mrsaintYes.. it is..
but it starts on empty conf.. soo I paste bgp+routemaps/accesslist from .sav file and it works. :)
[01:06]
mercutioweird [01:07]
mrsaintyes.. :) [01:07]
mercutioit used to have each daemon with it's own config file
then Zebra.conf
[01:07]
mrsaintbut anyways its all good :) [01:07]
mercutiofrom memory
and Zebra.conf would be like a master file
but it meant there was this weird kind of double up
[01:07]
mrsaintyeah.. but I use vtysh [01:07]
mercutioyeah
which puts it into the master one
sorry i'm a little rusty
but i seem to recall that master worked better
[01:08]
mrsaintnp. :)
I do have it on my 10g routers working just fine.. :)
its just my virtual setup that things not always stay on reboots.
but as I know it its a easy fix.
[01:08]
mercutiobbs
gl :)
[01:09]
mrsaint: did you get the e-mail about the maintenance? [01:18]
mrsaintmercutio: yeah but didnt reflect much at it :) [01:21]
mercutioahh right [01:21]
mrsaintguess im constantly tiried with small baby girl at home and way to much work. :) [01:21]
mercutioheh [01:22]
........ (idle for 37mn)
plettmrsaint: Out of interest, what are you using the BGP for? Internet facing, or internal routing inside vpn tunnels? [01:59]
mrsaintplett: announce /24 for anycast. [02:00]
plettDNS? [02:00]
mrsaintin this case on many locations with vps :)
yes dns.. and trying some web :)
[02:00]
plettI can't imagine web traffic will work quite as well [02:01]
mrsaintyeah it actually does :)
try www.mainstreamhosting.net :)
see if it works
[02:01]
plettFirefox can't establish a connection to the server at www.mainstreamhosting.net.
(yes, for real, not pulling your leg)
[02:01]
mrsaintoh thats interesting.
whre did you end up? show me last 3 jumps in route?
coz it works for me
[02:01]
plett6.|-- linx-10ge.lon1.uk.portlane.net 0.0% 10 3.9 10.7 3.4 39.1 13.8
7.|-- v-router-lon.mainstreamhosting.net 0.0% 10 3.4 3.6 3.4 3.9 0.0
8.|-- web-anycast.mainstreamhosting.net 0.0% 10 3.9 3.6 3.3 3.9 0.0
[02:02]
mrsaintoh.. let me check the london box
try now
[02:03]
plettYep. Works now :) [02:03]
mrsaintYeah.. varnish had died.
its a dynamic page with mysql on the backend soo its quite interesting how well it does work.
[02:04]
plettMore nagios'ing required? [02:04]
mrsaintYeah
should work :) as long as backend is reachable and same route is active atleast :)
but now time for lunch here.. ttl
[02:04]
plettWhy are you doing anycast for the web traffic? Just because you can, or for DoS resilience etc? [02:06]
mrsaintJust coz I can :) [02:06]
plettFair enough. That's a good answer :) [02:06]
mrsaintand I wasted /24 subnet for 1 service? dns.. why not put my own page on a second ip :)
:)
[02:06]
plettI would probably have used the anycast DNS to return geo-ip CNAMEs for the web servers [02:07]
mrsaintvarnish do cache alot and the content is rather static.
yeah will play with that to at later stage.
[02:07]
plettAs you just know that at some point someone will want a long-running http session, some websockets thing or a big download, and BGP will move them when they don't want it [02:08]
mrsaintYeah know. But I dont have much on my page. its static content. hehe.
but now off to food.
[02:08]
plettNow, if you were doing something even crazier like sharing pfsync state between all the boxes so that any node could just carry on with the existing TCP session, that would be awesome :)
Cool. Bye :)
[02:09]
mrsainthehe :) wont do that.. from eu to thai to japan and us :) [02:11]
mercutioplett: there's ways to make that less of an issue
but ime if web sites load slow most people click reload
of course there are probably morons who wait forever :)
but if a connection doesn't even do a tcp reset, then people will reload
or they'll go to another page
if you do a reset for unknown tcp connections it might speed some people up
then they can just click reload
plett.. i'm actually interested in this sharing state between locations thing
plett.. you don't have any idea of some easy way to do that with linux do you? it would also need to not reset connections it doesn't knwo about yet.
so that asymmetric routing would work normally
it's part of my transparent tcp proxy idea
[02:14]
plettmercutio: It wouldn't actually work. It isn't just the IP traffic which needs state syncing, you would need the web servers on each node to share state about every connection. [02:21]
mercutiothat should be doable [02:21]
plettEven if you did write a custom web server to do that, the overheads in syncing all that would be crazy. You /might/ get away with it between two machines in the same room, but doing it between continents would be crazy [02:22]
mercutiothe biggest problem i see is that if traffic comes in before it's received the state information
you have to delay sending reject packets.
not necessarily plett.. if it's many connections you could do compression of sorts.
my idea is slightly different though
i want a mitm transparent tcp proxy
and to be able to send outgoing traffic out whatever path, and have bgp in multiple locations that takes traffic in
and when traffic comes in it can create a mitm tcp proxy
and in that situation, if you have more than one way to communicate, if a connection moves you could always copy the state
like request the state of the connection
and then you can just have multiple paths
and protect against network issues rather than server outages.
for local server outages you could copy all state information between two hosts too
[02:22]
plettI'm now wondering if you could do something like syncookies where another node could identify that it was valid traffic and know what to do with it based on the contents of the traffic itself, without having any previous knowledge of the session. [02:25]
mercutiowell my idea is for a fully transparent tcp proxy
so if necessary it could always just tunnel all of the data as is closer to the source origin
if something else has state
[02:25]
plettI think that /might/ be possible, but now we're either into complete batshit insane territory or industry-changing retire-at-40-to-the-bahamas territory
mercutio: What's the use case? What's the underlying problem you're trying to solve?
[02:26]
mercutioso like the idea is you have connection a that connects to c, a sends to c, b is in between the two and recieves from c, so it creates a new local tcp connection and bounces all the traffic b<>c back to a, as in b<>a
plett: well if there's packet loss rtt really kills performance
and performance goes up with shorter rtt
so having a tcp proxy half way in between cna improve performance
basically it's about tcp acceleration
so like, new zealand to california is around 130 msec, california to uk is around 130 msec
new zealand to uk is around 260msec
so transparently tcp proxying in california improves performance a lot
especially if there's loss
but you're very unlikely over the normal internet to get 200 megabit between uk and nz
[02:27]
plettFor a small number of users, I'd just host a SOCKS proxy in california [02:29]
mercutiothen you have a california ip :)
i wrote something non transparent already
although it was just for web, and using squid.
and it made web browsing around 20% faster.
[02:29]
plettmercutio: That sounds like a challenge. Have you got a web server in .nz which can serve 200 Mb to me? I can test it from here in .uk ;) [02:30]
mercutioyeh sure [02:30]
plettmercutio: 2015-09-29 10:30:00 (2.37 MB/s) - `testfile.zip' saved [209715200/209715200]
mercutio: From arp: 2015-09-29 10:31:25 (4.75 MB/s) - `testfile.zip' saved [209715200/209715200]
[02:32]
mercutioweird
i got faster than that with arp
oh from dedicated though
[02:34]
plettMine's just a shared vps, but I can sustain a faster speed than that back to the UK (I do each night when I back it up) [02:35]
mercutioyeah dunno what's with that
i didn't download the whole thing
but at 90 megabytes it was at 16.7megabytes/sec
so you see, mitm in california could easily improve rate
you can transparently tcp proxy incoming traffic easily
but it's outgoing traffic that's the challenge
so it's pretty easy to see that it can benefit
but if you want seamless transparent tcp proxying then it's a bit of a challenge
[02:35]
plettmercutio: Hmm. Host a SOCKS proxy in cali, but using a .nz IP announced by BGP? [02:39]
mercutioyeah i want the IP to stay the same though
i got some first hop transparent proxying going btw
there's a few gotchas with that even
was running into state issues with iptables
but that hardly made any difference tbh
and still has the issue that you don't get tcp connection timed out, connection refused etc
so what i want to do is intercept after connection is established
it's all just ideas atm though
[02:39]
plettmercutio: Do you need to support failover between nodes during a connection?
i.e. half way through a large file download, BGP changes its mind and directs the client to another node
If the client will accept a RST at that point and re-connect and carry on from where it left off, then you don't need anything fancy
[02:42]
mercutioplett: not doing failover at first
i want to do proof of concept first
[02:44]
plettmercutio: Then a bsd box running relayd at each node might work well enough [02:45]
mercutiofunny you mention that, i tried relayd early on
the problem is like say to japan, often can send out good paths here
but traffic will come back in via the US
so you don't want to send out the US
but what if for some reason traffic came back to US before NZ had sent state information to US
like say bufferbloat issue or such
also bsd's tcp stack is worse than linux for performance
i think really would need to do kernel modifications to do it well
but i haven't looked into it yet
if there's some way for userland to hook state information and what to do about rejecting connections etc it'd be good
i dunno if there is or not though
and really where i want to start is making tcp smoother :)
err ssh smoother
cos it's easy to benchmark a lot without using heaps of bandwidth
[02:45]
the idea makes sense to you though right? it's just hard.. [02:55]
plettI can see why you'd want to do it. If it's for file distribution via HTTP though, building your own CDN with some varnish nodes around the globe and adding some geo-ip dns on top of it is probably much easier than making TCP behave differently [03:07]
mercutioyeh i want to do it for user experience
it's much easier from a cdn pov
i prefer apache traffic server to varnish, but accelerating web sites is much easier.
[03:07]
..... (idle for 21mn)
mrsaintplett: pretty much my idea to have static content sites on varnish with nginx backend locally and backup remotely. [03:31]
mercutiomrsaint: do you have any database stuff for your site? [03:32]
mrsaintand just make sure I have stable providers of my hosts.. most of them is on my network anyways. [03:32]
mercutiothat seems to be the biggest issue, i suppose you can create static content from dynamic database etc. [03:32]
mrsaintmercutio: I do.. :) but as its joomla currently and I dont change content its pretty much stable static data [03:32]
mercutioi love push caching
i've been using cloudflare for a bit for some personal stuff, and there's actually quite a bit of delay
[03:33]
mrsaintI just like to test it out.. I have a customer with 2 static page sites that Im gonna distribute out on the servers on thursday and see how it goes :)
www.mainstreamhosting.se is on cloudflare it also has mysql backend.
[03:34]
mercutiowith things like cloudflare when you test from random locations with low accessed site it will often need to pull [03:34]
mrsaintyes. [03:35]
mercutioand it doesn't pull from it's other caches
it pulls straight from origin
i think mixing pushing/pull is good if lots of content
i mean ideally i'd like to see lots of pull caches everywhere
and push caches reasonably nearby
[03:35]
mrsaintJust need to find more VPS providers in the US that supports bgp :)
they dont really grow on trees :)
[03:38]
mercutiothere's not many i looked ages ago [03:42]
mrsaintmaybe colo/box rental places with bgp support? :) [03:43]
mercutioi reckon having a few good nodes is better than heaps of them though [03:43]
mrsaintyeah but atleast two in the us I would like to have.
redundancy per region :)
[03:43]
mercutioyeah i understand [03:43]
BryceBotThat's what she said!! [03:43]
mrsainthehe [03:44]
............................................ (idle for 3h35mn)
***qbit has joined #arpnetworks
qbit is now known as Guest16723
Guest16723 is now known as qbit
[07:19]
........... (idle for 54mn)
mkb has joined #arpnetworks [08:14]
mkbhmm my server turned off in the night [08:15]
mkb must have been on that server that was upgraded [08:22]
***hycer- is now known as hycer [08:36]
............... (idle for 1h10mn)
mrsaintcan someone do hostlookup for www.geo.mainstreamhosting.net ? [09:46]
brycecwww.geo.mainstreamhosting.net is an alias for web-anycast-lon.mainstreamhosting.net.
web-anycast-lon.mainstreamhosting.net has address 46.246.65.37
[09:59]
mrsaintinteresting :)
where you located?
[09:59]
brycecThat was from North Carolina, US
From Washington state, US, I get anycast-arp.mainstreamhosting.net.
[09:59]
mrsaintah yeah.. I dont do real geoip.. as I do have 1000 domains in the dns.. and views today I will have to load all the zones in all views.
and I cant anymore load same zonefile multiple times and I dont want to load 20k domains or soo due to diff views :)
thats nice :)
but atleast I can keep the sessions to the "same" box now..
[10:00]
.................................. (idle for 2h48mn)
***m0unds has quit IRC (Quit: WeeChat 1.1.1) [12:49]
mrsainttweaked it.. host www.geo.mainstreamhosting.net what does it give you guys anyone? [12:50]
plettmrsaint: www.geo.mainstreamhosting.net is an alias for web-anycast-lon.mainstreamhosting.net.
Which makes sense for me
[12:53]
mrsaintok dns located there?
whats your dns you lookup on host -t ns geo.mainstreamhosting.net ?
could still be my old
[12:54]
plettgeo.mainstreamhosting.net name server dns1.mainstreamhosting.net. [12:54]
mrsaintah good.. then its correct
testing out geo acls
but its good then I dare launching my customer website on thursday :)
[12:55]
plett$ dig +short chaos txt hostname.bind @dns-anycast.mainstreamhosting.net.
"dns-anycast-lon.mainstreamhosting.net"
[12:56]
mrsaintyeah well its not the server it should be looking up against.
I think I need to change my soa
I do have dns1 in soa
[12:58]
plettNo, but it would look there to find the ns record for geo.mainstreamhosting.net , so that's presumably where the anycasting is happening [12:59]
mrsainthehe. yeah well its just lookup the subzone geo [12:59]
plettOr rather, the geo-ip [12:59]
mrsaintthat is pointed to a specific server where I do geo magic
I deligate geo IN NS dns1 in mainstreamhosting.net zone.
im gonna make sure the traffic from canada and us ends up in arp server. if possible. that is my maingoal. The customer has bought traffic for a site in canada. ( gaming site) and uk and nl
soo I want to direct the traffic to those specific mirrors.
would be fun if it worked.
[12:59]
.............................. (idle for 2h26mn)
mercutioi have done geo direction with tinydns
it mostly worked
doing the /8s to various places. the main issue was universitys etc.
if you anycast dns then you could answer with different results depending on what dns server it hit
[15:31]
............ (idle for 59mn)
***pyvpx_ has joined #arpnetworks
ben2 has joined #arpnetworks
toeshred has quit IRC (Ping timeout: 246 seconds)
pyvpx has quit IRC (Ping timeout: 246 seconds)
mercutio has quit IRC (Ping timeout: 246 seconds)
toeshred has joined #arpnetworks
[16:33]
mnathani_https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919 [16:38]
***ben2 is now known as mercutio [16:39]
........ (idle for 36mn)
gizmoguymnathani_: isn't that from 2010? [17:15]
staticsafeit is
i just shared that in another channel too rofl
[17:15]
gizmoguyI did as well
and then checked the date
[17:16]
mercutiohah [17:28]
......... (idle for 44mn)
mnathani_lol
my bad
someone posted it on NANOG
[18:12]
mercutioheh
nanog is getting more and more average
[18:13]
staticsafei've been ignoring it for the last couple of weeks, doesn't look like I missed much [18:16]
....................... (idle for 1h50mn)
mjp__nanog
whats that
[20:06]
.... (idle for 15mn)
staticsafenanog.org [20:22]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)