***: mkb has quit IRC (Remote host closed the connection) mrsaint: any outage now?
my traceroute ends up at
9 GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86) 203.131 ms 202.774 ms 202.614 ms
10 * * *
and my node down :( mercutio: is your host on kvr04? mrsaint: no idea. :) mercutio: i'd say it probably is mrsaint: yeah :) mercutio: planned maintenance
it should be back in a few minutes
literally mrsaint: I cant see it in the portal where it is located :)
aah now I see. Yes its on kvr04 if its the same as vnc that I would expect it to be :) mercutio: is it back now?
yes it is mrsaint: yes mercutio: cool mrsaint: let me see if my reboots worked :)
do actually seems soo :) mercutio: sweet mrsaint: but somehow quagga starts on reboots on empty conf mercutio: change to bird mrsaint: so I have to paste conf into it again soo wierd.. :)
Might do in future :) mercutio: is it using Quagga.conf or whatever ***: tabthorpe has quit IRC (Remote host closed the connection) mrsaint: Just that I have 10 nodes with same setup now and it all is good. mercutio: quagga's config is messy mrsaint: Yes.. it is..
but it starts on empty conf.. soo I paste bgp+routemaps/accesslist from .sav file and it works. :) mercutio: weird mrsaint: yes.. :) mercutio: it used to have each daemon with it's own config file
then Zebra.conf mrsaint: but anyways its all good :) mercutio: from memory
and Zebra.conf would be like a master file
but it meant there was this weird kind of double up mrsaint: yeah.. but I use vtysh mercutio: yeah
which puts it into the master one
sorry i'm a little rusty
but i seem to recall that master worked better mrsaint: np. :)
I do have it on my 10g routers working just fine.. :)
its just my virtual setup that things not always stay on reboots.
but as I know it its a easy fix. mercutio: bbs
gl :)
mrsaint: did you get the e-mail about the maintenance? mrsaint: mercutio: yeah but didnt reflect much at it :) mercutio: ahh right mrsaint: guess im constantly tiried with small baby girl at home and way to much work. :) mercutio: heh plett: mrsaint: Out of interest, what are you using the BGP for? Internet facing, or internal routing inside vpn tunnels? mrsaint: plett: announce /24 for anycast. plett: DNS? mrsaint: in this case on many locations with vps :)
yes dns.. and trying some web :) plett: I can't imagine web traffic will work quite as well mrsaint: yeah it actually does :)
try www.mainstreamhosting.net :)
see if it works plett: Firefox can't establish a connection to the server at www.mainstreamhosting.net.
(yes, for real, not pulling your leg) mrsaint: oh thats interesting.
whre did you end up? show me last 3 jumps in route?
coz it works for me plett: 6.|-- linx-10ge.lon1.uk.portlane.net 0.0% 10 3.9 10.7 3.4 39.1 13.8
7.|-- v-router-lon.mainstreamhosting.net 0.0% 10 3.4 3.6 3.4 3.9 0.0
8.|-- web-anycast.mainstreamhosting.net 0.0% 10 3.9 3.6 3.3 3.9 0.0 mrsaint: oh.. let me check the london box
try now plett: Yep. Works now :) mrsaint: Yeah.. varnish had died.
its a dynamic page with mysql on the backend soo its quite interesting how well it does work. plett: More nagios'ing required? mrsaint: Yeah
should work :) as long as backend is reachable and same route is active atleast :)
but now time for lunch here.. ttl plett: Why are you doing anycast for the web traffic? Just because you can, or for DoS resilience etc? mrsaint: Just coz I can :) plett: Fair enough. That's a good answer :) mrsaint: and I wasted /24 subnet for 1 service? dns.. why not put my own page on a second ip :)
:) plett: I would probably have used the anycast DNS to return geo-ip CNAMEs for the web servers mrsaint: varnish do cache alot and the content is rather static.
yeah will play with that to at later stage. plett: As you just know that at some point someone will want a long-running http session, some websockets thing or a big download, and BGP will move them when they don't want it mrsaint: Yeah know. But I dont have much on my page. its static content. hehe.
but now off to food. plett: Now, if you were doing something even crazier like sharing pfsync state between all the boxes so that any node could just carry on with the existing TCP session, that would be awesome :)
Cool. Bye :) mrsaint: hehe :) wont do that.. from eu to thai to japan and us :) mercutio: plett: there's ways to make that less of an issue
but ime if web sites load slow most people click reload
of course there are probably morons who wait forever :)
but if a connection doesn't even do a tcp reset, then people will reload
or they'll go to another page
if you do a reset for unknown tcp connections it might speed some people up
then they can just click reload
plett.. i'm actually interested in this sharing state between locations thing
plett.. you don't have any idea of some easy way to do that with linux do you? it would also need to not reset connections it doesn't knwo about yet.
so that asymmetric routing would work normally
it's part of my transparent tcp proxy idea plett: mercutio: It wouldn't actually work. It isn't just the IP traffic which needs state syncing, you would need the web servers on each node to share state about every connection. mercutio: that should be doable plett: Even if you did write a custom web server to do that, the overheads in syncing all that would be crazy. You /might/ get away with it between two machines in the same room, but doing it between continents would be crazy mercutio: the biggest problem i see is that if traffic comes in before it's received the state information
you have to delay sending reject packets.
not necessarily plett.. if it's many connections you could do compression of sorts.
my idea is slightly different though
i want a mitm transparent tcp proxy
and to be able to send outgoing traffic out whatever path, and have bgp in multiple locations that takes traffic in
and when traffic comes in it can create a mitm tcp proxy
and in that situation, if you have more than one way to communicate, if a connection moves you could always copy the state
like request the state of the connection
and then you can just have multiple paths
and protect against network issues rather than server outages.
for local server outages you could copy all state information between two hosts too plett: I'm now wondering if you could do something like syncookies where another node could identify that it was valid traffic and know what to do with it based on the contents of the traffic itself, without having any previous knowledge of the session. mercutio: well my idea is for a fully transparent tcp proxy
so if necessary it could always just tunnel all of the data as is closer to the source origin
if something else has state plett: I think that /might/ be possible, but now we're either into complete batshit insane territory or industry-changing retire-at-40-to-the-bahamas territory
mercutio: What's the use case? What's the underlying problem you're trying to solve? mercutio: so like the idea is you have connection a that connects to c, a sends to c, b is in between the two and recieves from c, so it creates a new local tcp connection and bounces all the traffic b<>c back to a, as in b<>a
plett: well if there's packet loss rtt really kills performance
and performance goes up with shorter rtt
so having a tcp proxy half way in between cna improve performance
basically it's about tcp acceleration
so like, new zealand to california is around 130 msec, california to uk is around 130 msec
new zealand to uk is around 260msec
so transparently tcp proxying in california improves performance a lot
especially if there's loss
but you're very unlikely over the normal internet to get 200 megabit between uk and nz plett: For a small number of users, I'd just host a SOCKS proxy in california mercutio: then you have a california ip :)
i wrote something non transparent already
although it was just for web, and using squid.
and it made web browsing around 20% faster. plett: mercutio: That sounds like a challenge. Have you got a web server in .nz which can serve 200 Mb to me? I can test it from here in .uk ;) mercutio: yeh sure plett: mercutio: 2015-09-29 10:30:00 (2.37 MB/s) - `testfile.zip' saved [209715200/209715200]
mercutio: From arp: 2015-09-29 10:31:25 (4.75 MB/s) - `testfile.zip' saved [209715200/209715200] mercutio: weird
i got faster than that with arp
oh from dedicated though plett: Mine's just a shared vps, but I can sustain a faster speed than that back to the UK (I do each night when I back it up) mercutio: yeah dunno what's with that
i didn't download the whole thing
but at 90 megabytes it was at 16.7megabytes/sec
so you see, mitm in california could easily improve rate
you can transparently tcp proxy incoming traffic easily
but it's outgoing traffic that's the challenge
so it's pretty easy to see that it can benefit
but if you want seamless transparent tcp proxying then it's a bit of a challenge plett: mercutio: Hmm. Host a SOCKS proxy in cali, but using a .nz IP announced by BGP? mercutio: yeah i want the IP to stay the same though
i got some first hop transparent proxying going btw
there's a few gotchas with that even
was running into state issues with iptables
but that hardly made any difference tbh
and still has the issue that you don't get tcp connection timed out, connection refused etc
so what i want to do is intercept after connection is established
it's all just ideas atm though plett: mercutio: Do you need to support failover between nodes during a connection?
i.e. half way through a large file download, BGP changes its mind and directs the client to another node
If the client will accept a RST at that point and re-connect and carry on from where it left off, then you don't need anything fancy mercutio: plett: not doing failover at first
i want to do proof of concept first plett: mercutio: Then a bsd box running relayd at each node might work well enough mercutio: funny you mention that, i tried relayd early on
the problem is like say to japan, often can send out good paths here
but traffic will come back in via the US
so you don't want to send out the US
but what if for some reason traffic came back to US before NZ had sent state information to US
like say bufferbloat issue or such
also bsd's tcp stack is worse than linux for performance
i think really would need to do kernel modifications to do it well
but i haven't looked into it yet
if there's some way for userland to hook state information and what to do about rejecting connections etc it'd be good
i dunno if there is or not though
and really where i want to start is making tcp smoother :)
err ssh smoother
cos it's easy to benchmark a lot without using heaps of bandwidth
the idea makes sense to you though right? it's just hard.. plett: I can see why you'd want to do it. If it's for file distribution via HTTP though, building your own CDN with some varnish nodes around the globe and adding some geo-ip dns on top of it is probably much easier than making TCP behave differently mercutio: yeh i want to do it for user experience
it's much easier from a cdn pov
i prefer apache traffic server to varnish, but accelerating web sites is much easier. mrsaint: plett: pretty much my idea to have static content sites on varnish with nginx backend locally and backup remotely. mercutio: mrsaint: do you have any database stuff for your site? mrsaint: and just make sure I have stable providers of my hosts.. most of them is on my network anyways. mercutio: that seems to be the biggest issue, i suppose you can create static content from dynamic database etc. mrsaint: mercutio: I do.. :) but as its joomla currently and I dont change content its pretty much stable static data mercutio: i love push caching
i've been using cloudflare for a bit for some personal stuff, and there's actually quite a bit of delay mrsaint: I just like to test it out.. I have a customer with 2 static page sites that Im gonna distribute out on the servers on thursday and see how it goes :)
www.mainstreamhosting.se is on cloudflare it also has mysql backend. mercutio: with things like cloudflare when you test from random locations with low accessed site it will often need to pull mrsaint: yes. mercutio: and it doesn't pull from it's other caches
it pulls straight from origin
i think mixing pushing/pull is good if lots of content
i mean ideally i'd like to see lots of pull caches everywhere
and push caches reasonably nearby mrsaint: Just need to find more VPS providers in the US that supports bgp :)
they dont really grow on trees :) mercutio: there's not many i looked ages ago mrsaint: maybe colo/box rental places with bgp support? :) mercutio: i reckon having a few good nodes is better than heaps of them though mrsaint: yeah but atleast two in the us I would like to have.
redundancy per region :) mercutio: yeah i understand BryceBot: That's what she said!! mrsaint: hehe ***: qbit has joined #arpnetworks
qbit is now known as Guest16723
Guest16723 is now known as qbit
mkb has joined #arpnetworks mkb: hmm my server turned off in the night -: mkb must have been on that server that was upgraded ***: hycer- is now known as hycer mrsaint: can someone do hostlookup for www.geo.mainstreamhosting.net ? brycec: www.geo.mainstreamhosting.net is an alias for web-anycast-lon.mainstreamhosting.net.
web-anycast-lon.mainstreamhosting.net has address 46.246.65.37 mrsaint: interesting :)
where you located? brycec: That was from North Carolina, US
From Washington state, US, I get anycast-arp.mainstreamhosting.net. mrsaint: ah yeah.. I dont do real geoip.. as I do have 1000 domains in the dns.. and views today I will have to load all the zones in all views.
and I cant anymore load same zonefile multiple times and I dont want to load 20k domains or soo due to diff views :)
thats nice :)
but atleast I can keep the sessions to the "same" box now.. ***: m0unds has quit IRC (Quit: WeeChat 1.1.1) mrsaint: tweaked it.. host www.geo.mainstreamhosting.net what does it give you guys anyone? plett: mrsaint: www.geo.mainstreamhosting.net is an alias for web-anycast-lon.mainstreamhosting.net.
Which makes sense for me mrsaint: ok dns located there?
whats your dns you lookup on host -t ns geo.mainstreamhosting.net ?
could still be my old plett: geo.mainstreamhosting.net name server dns1.mainstreamhosting.net. mrsaint: ah good.. then its correct
testing out geo acls
but its good then I dare launching my customer website on thursday :) plett: $ dig +short chaos txt hostname.bind @dns-anycast.mainstreamhosting.net.
"dns-anycast-lon.mainstreamhosting.net" mrsaint: yeah well its not the server it should be looking up against.
I think I need to change my soa
I do have dns1 in soa plett: No, but it would look there to find the ns record for geo.mainstreamhosting.net , so that's presumably where the anycasting is happening mrsaint: hehe. yeah well its just lookup the subzone geo plett: Or rather, the geo-ip mrsaint: that is pointed to a specific server where I do geo magic
I deligate geo IN NS dns1 in mainstreamhosting.net zone.
im gonna make sure the traffic from canada and us ends up in arp server. if possible. that is my maingoal. The customer has bought traffic for a site in canada. ( gaming site) and uk and nl
soo I want to direct the traffic to those specific mirrors.
would be fun if it worked. mercutio: i have done geo direction with tinydns
it mostly worked
doing the /8s to various places. the main issue was universitys etc.
if you anycast dns then you could answer with different results depending on what dns server it hit ***: pyvpx_ has joined #arpnetworks
ben2 has joined #arpnetworks
toeshred has quit IRC (Ping timeout: 246 seconds)
pyvpx has quit IRC (Ping timeout: 246 seconds)
mercutio has quit IRC (Ping timeout: 246 seconds)
toeshred has joined #arpnetworks mnathani_: https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919 ***: ben2 is now known as mercutio gizmoguy: mnathani_: isn't that from 2010? staticsafe: it is
i just shared that in another channel too rofl gizmoguy: I did as well
and then checked the date mercutio: hah mnathani_: lol
my bad
someone posted it on NANOG mercutio: heh
nanog is getting more and more average staticsafe: i've been ignoring it for the last couple of weeks, doesn't look like I missed much mjp__: nanog
whats that staticsafe: nanog.org