***: nixbag has joined #arpnetworks
nixbag has quit IRC (Quit: leaving)
nixbag has joined #arpnetworks
amdprophet has joined #arpnetworks up_the_irons: toorop: it'll be run again automatically, but if you'd like to let someone know, just reply to the email
like brycec said ***: dj_goku has quit IRC (Ping timeout: 256 seconds)
heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer
heavysixer has quit IRC (Quit: heavysixer)
dj_goku has joined #arpnetworks
dj_goku has quit IRC (Ping timeout: 255 seconds)
raptelan has joined #arpnetworks raptelan: *grumble grumble* not having a server out on the net is really inconvenient sometimes.
time to sign up for arp again :) ***: LEMONed has joined #arpnetworks
LEMONed has quit IRC (Ping timeout: 260 seconds)
raptelan has quit IRC (Quit: leaving)
raptelan has joined #arpnetworks
WaREX has quit IRC (Quit: leaving) brycec: agreed, do it ***: raptelan has quit IRC (Ping timeout: 245 seconds)
raptelan has joined #arpnetworks
raptelan has quit IRC (Ping timeout: 260 seconds)
raptelan has joined #arpnetworks
raptelan has quit IRC (Ping timeout: 245 seconds)
phlux has quit IRC (Quit: WeeChat 0.4.1-dev)
phlux has joined #arpnetworks
heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer -: up_the_irons nods
brycec nods mercutio: has the arpnetworks had network issues in the last 3 hours?
i've noticed smokeping has lots of loss, but it's fine atm brycec: mercutio: my smokeping is quiet mercutio: i have 4.11% average loss for last 3 hours, 44.42% max
it was worse from 3 hours to 2 hours ago
but it seems to alternate between rather broken and completely fine brycec: mercutio: Here are my last 8hrs, v4 and v6 http://imgur.com/jI587nI,PJg054g
think I'm on kvr07 mercutio: that's close
but yeah you have no issues :) brycec: (yep, kvro7) mercutio: it's usually fine -: brycec wonders why he bothers smokepinging localhost mercutio: heh
i do too
http://imgur.com/a/OvsG5#8qe0RNg
does that work? brycec: I think it was in the default config
loads yes mercutio: see how it's fine some of the time completely
and screwed other times ?
and of course it's fine now when i noticed brycec: heh
Well I'm going to blame the other end of your smokeping
since... I can mercutio: i'm monitoring other points too
it looked like just arp effected
but it could be pccw
hardly anything goes via pccw except arp brycec: (My smokeping runs on a Chunkhost vps, looking inward at ARP) mercutio: it's like pccw, trit, arp brycec: mercutio: bummer
I go chunkhost - peer1 - arp
literally 1 host between networks
(never noticed that before, sweet) mercutio: that's all peering traffic i imagine
well there's someone else before pccw too
http://i.imgur.com/PaJdJf0.png
that's another jost in san jose
but yeah not a global issue
oh hmm, i do have monitoring of arp from somewhere else too brycec: hm yeah guess it's not a pccw issue, at least not in general. mercutio: i should check brycec: lol mercutio: it could be the node or it could be transit provider etc
yeh it's fine from chicago up_the_irons: i think i peer with peer1 brycec: up_the_irons: yeah
you do
2 10ge-ten1-1.la-600w-cor-1.peer1.net (216.187.88.145) 0.371 ms 0.389 ms 0.418 ms mercutio: chicago is via mzima brycec: 3 any2-ix.la.arpnetworks.com (206.223.143.166) 2.031 ms 2.391 ms 2.559 ms up_the_irons: must be through the route server cuz i don't see them in my official peering list mercutio: in both directions up_the_irons: sweet mercutio: i'm going to assume it's some pccw/trit issue
i dunno what else goes via pccw/trit
and it's nlayer on the way back from arp
and if was nlayer issue more people would have issues i imagine staticsafe: yes mercutio: oh, and because pccw/trit has had minor issues before staticsafe: http://smokeping.asininetech.com/smokeping.cgi?target=IRC.scruffy looking good mercutio: oh hmm, i should be able to find some other host using pccw/trit
6. i-0-0-0-0.eqla01.bi.telstraglobal.net 0.0% 1979 174.0 125.8 125.5 204.1 5.2
7. 63-218-51-149.static.pccwglobal.net 0.2% 1979 132.6 149.8 130.2 503.5 41.1
my long running mtr found at least a little packet loss
but not to destination
it's really difficult to find good ways to check in between networks
when tehre are so many of them
err
static
did you not notice hte packet loss at the end? :)
oh
to multiple destinations
it'e the same to farnsworth/scruffy so local issue most likely
and bender and amy too
oh packet loss started again
6. i-0-0-0-0.eqla01.bi.telstraglobal.net 0.0% 2402 125.9 125.7 125.5 204.1 4.9 staticsafe: hmm mercutio: 7. 63-218-51-149.static.pccwglobal.net 0.2% 2402 130.5 150.0 130.2 503.5 41.4
8. 63-218-212-14.static.pccwglobal.net 2.1% 2401 221.2 159.4 157.4 354.4 11.0
9. cxc.r6.lax2.trit.net 1.6% 2401 195.8 134.5 133.5 197.5 7.4
10. arpnetworks-lax2-gw.cust.trit.net 2.2% 2401 195.5 133.0 126.2 472.0 26.0
11. arp 1.7% 2401 192.4 131.4 130.5 194.3 7.2
so yeh looks like a pccw issue to me
the tarceroute from arp has that annoying nlayer icmp deprioritisation
curiosuly the trace back from arp has this low/high alternating packet loss
and so does the forward but not as obviously
so i think ther'es some load balancing there
8. i-0-1-0.hptw-core01.bx.telstraglobal.net 0.8% 625 327.0 349.9 135.2 621.4 108.0
9. unknown.telstraglobal.net 1.3% 625 130.4 131.4 129.7 170.6 4.6
10. xe-0-3-0-1037.mdr.core.snap.net.nz 5.3% 625 131.2 134.5 130.6 222.4 12.9
11. 202-49-70-162.plain.net.nz 0.8% 625 131.9 131.3 130.4 288.2 6.9
12. 202.36.174.250 5.4% 625 131.3 132.5 130.5 195.3 10.4
but it heavily suggests the issue is sending to arp not receiving from arp
which was already my hunch anyway jlgaddis: < mercutio> the tarceroute from arp has that annoying nlayer icmp deprioritisation <--- so you finally watched that video, huh? ;) mercutio: jlgaddis: i've always known about icmp deprioritisation
thing is, that it's actually rather an annoying thing
as there can be packet loss carried through too
like hop 11/12 aren't deprioritising icmp
but it makes it look like load balcnign issue up_the_irons: icmp replies must be generated by the RP of a switch/router, so it is often rate limited, to save CPU resources
now if all u kids would stop tracerouting all the time, maybe we wouldn't have to do that... ;) staticsafe: heh jbergstroem: script kid if i may mercutio: wow, in the whole bgp table there is only arpnetworks using pccw/trit combination
so much for finding anothe rhost to monitor up_the_irons: there's a Trit node in Amsterdam, but that's all i know... ;) mercutio: this is from new zealand
and pccw/trit interconnection is happening in los angeles
oh actually i did find another it's just not preferred
199.249.180.0/23
but that's cogent/pccw/trit
or xeex up_the_irons: btw, I signed with NTT last week, so that circuit will be up soon mercutio: oh cool
what AS is ntt? -: mercutio wonders if it'll route via it mercutio: it seems trit is mostly pccw
using the not-very-accurate bgp.he.net/AS40193 jlgaddis: ntt america is 2914 mercutio: just found that
searching for asn owrks beter than AS
as number
hmm wonder who AS703 is staticsafe: Verizon mercutio: ahh verizon
so yeh it shoudl be abl to go via as703 to ntt
i dunno if that is good or bad :)
actually i thought there was already NTT routes before
oh i think mzima had NTT too
to at&t for example
mzima/nlayer/at&t
err mzima/nlayer/ntt/at&t
so will make it more direct to at&t
oh well, so pccw issues should be abolished
up_the_irons: how are you finding trit in general? jlgaddis: sh ip bgp regexp ? mercutio: i';m using openbgpd
and transit-as
7. 63-218-51-149.static.pccwglobal.net 0.0% 254 132.7 151.0 130.2 399.0 43.6
8. 63-218-212-14.static.pccwglobal.net 10.6% 254 159.6 164.7 157.4 222.3 18.3
heh it's still happening jlgaddis: btw, "wget ftp://ftp.arin.net/info/asn.txt", then you can just "grep -i whatever asn.txt" mercutio: i normaly do whois AS<blah>
in openbsd it seems to lack a few though
but yeah i suppose that would make it easier to search name jlgaddis: that works if you know the asn :P -: jlgaddis nods mercutio: true good point
openbsd doesn't have AS701 or AS703
i think linux does
oh weird
AS701 and AS703 are both UUNET...
maybe they're not in some AS database that openbsd uses with whois cos they're legacy
and i think AS701 is alter.net ?
in traceroutes jlgaddis: VZ mercutio: yeh but it's show up as alter.net? jlgaddis: some places still refer to it as uunet mercutio: oh that's a nice text file
mtr 161.159.0.1
like tracing to that shows alter.net
actually not from arp
arp shows ntt jlgaddis: does your traceroute support -a
? mercutio: i'll check
nope jlgaddis: -a == "show asn's", basically mercutio: not even the linux one does jlgaddis: debian/ubuntu: apt-get install traceroute-nanog mercutio: oh hangon
openbsd has -A up_the_irons: whoa
you learn something new every day mercutio: yeh this is working
4 0.xo-1-0-0.XT4.AKL1.ALTER.NET (210.80.35.17) [AS703] 1.776 ms 1.776 ms 1.769 ms
5 0.so-0-0-0.IL2.LAX12.Alter.Net (210.80.50.89) [AS703] 193.172 ms 193.180 ms 193.397 ms
6 0.xe-3-3-0.IL4.LAX9.ALTER.NET (152.63.114.17) [AS701] 149.355 ms 152.660 ms 151.895 ms up_the_irons: mercutio: trit is alright mercutio: up_the_irons: does much go over it? up_the_irons: the owner is my friend and we exchange traffic ***: heavysixer has quit IRC (Quit: heavysixer) jlgaddis: ahhh yeah, looks like traceroute-nanog is -A also. -a on os x. mercutio: ahh up_the_irons: mercutio: maybe 10% of my traffic; basically just the pccw routes mercutio: heh
it's the pccw routes i'm having issues with :) up_the_irons: yeah u need to bitch at pccw for that :) mercutio: i'm not pccw customer up_the_irons: doesn't matter, neither am i mercutio: i wonder if they'd listen
arne't they like a china provider or something up_the_irons: yeah
most NOCs are responsive to problems, whether you are a customer or not. cuz you just amplify what their customers are probably already telling them. mercutio: ahh
usnoc@pccwglobal.com
seems to be addresse
-e staticsafe: im glad that ARP peers with Vocus, I have a Australian ircd node linked to it mercutio: static: that should make things more stable staticsafe: http://bgp.he.net/AS9297#_asinfo my VPS provider in .au is in this AS mercutio: ok
i emailed them
wonder if they'll say anything
static: i know someone who has vocus transit in NZ
it seems very on/off performance wise
i think a lot of their problems come from using he.net staticsafe: I've only had it for a short time, but I've not noticed any significant issues yet mercutio: but even running two downloads one after the other of a file can be a 10x speed diff
i think it was like 4 or 5mb/sec to arp
and like 200 to 2000k/sec to europe staticsafe: I have it linked over v6 so if Vocus dies for whatever reason, its gone mercutio: hmm it's only 13.5 megabit/sec to arp atm
vocus seem to be growing quick
will be interesting to see if they try and be bottom of the barrrel, or offer a higher level of service
i think AU market is like NZ market where people are cheap
oh up_the_irons are NTT doing you ipv6 too?
wow
ipv6 is way faster to arp with vocus
err from arp
it's not faster to arp
oh it is faster to too
bizzare, seems to be performance issue with vocus to arp atm even with peering
the diff in performance between ipv6 and ipv6 can be bizzare
like even within one provider
[ 3] 0.0-10.4 sec 9.25 MBytes 7.49 Mbits/sec
[ 3] 0.0-10.2 sec 22.9 MBytes 18.7 Mbits/sec
like that's ipv4 versus ipv6
vocus for both ipv4 and ipv6
and download was like 25 to 35 megabit
which don't make a lot of sense either
cos would be less change of congestion uploading to arp staticsafe: farnsworth.entropynet.net, how does it look from .nz? mercutio: 113.9 msec
over ipv6
54 msec over ipv4
via vocus
48 msec via other provider
which is still via vocus
the vocus one is 6 msec away from auckland though staticsafe: the v6 difference there is interesting mercutio: so yeh it's good via ipv4
it's vocus all the way
6. ten-0-2-0-300.bdr02.akl01.akl.VOCUS.net.au 0.0% 20 7.7 8.3 7.6 12.0 1.0
7. 2402:7800:110:1::a 0.0% 20 31.7 31.9 30.7 38.1 1.6
8. ten-0-3-0-911.bdr02.per02.wa.VOCUS.net.au 0.0% 20 79.0 80.0 78.9 83.5 1.2
9. 2402:7800:40:1::d 0.0% 20 80.7 79.9 79.0 81.7 0.7
10. 2402:7800:30:1::15 0.0% 20 112.3 112.8 112.3 115.2 0.7
it has another vocus route at 11 with rdns up_the_irons: mercutio: yeah, ipv6 too. will finally have a non-HE ipv6 provider. mercutio: up_the_irons: yay :) up_the_irons: m/ mercutio: i thought vocus dual homed all their routers on ipv4/ipv6
so it's weird when performance is diff
actually the ipv4 route goes over ape which is auckland peering exchange
and ipv6 is going over transit
static: you could do a reverse trace to 2403:b700:1::6 jlgaddis: up_the_irons: provider or peer? mercutio: but i suspect the issue is routing via perth up_the_irons: jlgaddis: provider, full transit jlgaddis: nice up_the_irons: yep:) mercutio: and you're in adelaide? staticsafe: correct mercutio: they should be able to reduce latency from adelaide to sydney staticsafe: soon mercutio: looking on a map, it seems that it's liekly it goes via melbourne etc staticsafe: WAIA is in Adelaide now and Colocity is provisioning mercutio: so your normal pings to sydney are 21 to 24 msec?
and auckland to sydney is about 24
hmm by road it's 1410km
and distance between sydney/auckland is 2153km by bird.
so i imagine they can get latency down to 15/16 msec
i suppose that doesn't make a lot of diff staticsafe: it doesn't mercutio: i always underestimate how big australia is staticsafe: inter-Australia should be much better once the WAIA provisioning is done
heh mercutio: NZ to australia can suck at times
well it's fine to sydney generally
but once you hit domestic routing you can get stuff like adelaide going via perth
i heard that some isp's have really broken routing internally too
so the same happens to dsl users staticsafe: oh yes mercutio: and like perth will go via sydney for internet staticsafe: case in point - AT&T/Bellsouth DSL
in USA mercutio: and brisbane
and darwin
i also heard that a whole lot of users in australia can still only get adsl1 with 1.5 megabit/384 kbit or something
which uhh, means no high def youtube
terrible congestion etc
then there are transit providers like pipe networks with real high latency going via guam staticsafe: http://pastie.org/pastes/6385091/text?key=knxpf1n79zzlljnhufpqnq its not too bad for me from home mercutio: i can't even find guam on the map atm
i see hawaii
isn't it near hawaii? jlgaddis: just a bit west mercutio: oh you have to zoom in
it's like between papa new guinea and japan jlgaddis: it's probably about as near to hawaii as LA is to NYC =) mercutio: oh you're not in australia static?
jlg: it's closer to hawaii than it is to japan to us staticsafe: mercutio: no, I'm in Canada mercutio: err it's closer to US going via guam/hawaii than guam/japan
268 is really high ping
that's what i get to UK staticsafe: well yes mercutio: it's enough that it can be frustrating
you have 165 msec from san jose to australia
oh but it's mpls crap staticsafe: its only doing ircd, I don't even need to touch it except for any security updates mercutio: with keeping the ping high from start point to end point
suppose that's actually pretty good
yeh
not touching much is fine for that ping
but if you used tmux/irc itd' be annoying staticsafe: as long as it stays linked in a stable manner, I'm happy mercutio: heh
what irc network is that?
back years ago undernet used to have so many splits staticsafe: http://www.entropynet.net/ my small little thing mercutio: now freenode is like undernet was ***: heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer mercutio: Please provide the PCCW Global Circuit ID or Service ID.
i told them not direct customer. staticsafe: heh mercutio: err in my initial mail
it does seem to have calmed down now though -: up_the_irons goes and grabs In n' Out burgers ***: heavysixer has quit IRC (Quit: heavysixer) milki: burgers plural o.O staticsafe: mm burgers brycec: milki: well up_the_irons is married with kids... milki: o, so its not all just for him brycec: maybe ;) jlgaddis: i miss in 'n out -: brycec doesn't... it's just 2miaway mercutio: up_the_irons: is that new router coming in with the NTT link? up_the_irons: mercutio: new router is up (just not doing anything). NTT link will go in it, yes. all VM hosts will then have cable to the new router. and finally, the ability to route in / out either one will be complete.
brycec: milki : ok, well, one burger (double double, no onions, protein style) mercutio: up_the_irons: ahh sweet, was wondering how that was going up_the_irons: yeah, slowly but surely.. :)
biggest part was getting the new upstream (research, negotiations, and finally signing) mercutio: yeh i thought was two seperate things though :)
but i suppose tying together isn't stupid
how's fallback going to work? vrrp? ***: heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer up_the_irons: mercutio: it will be a manual failover. shutting down vlan on one router and up'ing it on the other would effectively 'switch' a customer from one router to another. up'ing the vlan sends a gratuituous ARP, so the VMs would automatically pick up the new gateway MAC address.
mercutio: no automatic fallback method was viable in my testing. they all caused more problems than they cured. failover should be an extremely rare situation, if it even occurs at all (we've already gone 4+ years on the first router) mercutio: ok
yeh i had a quick think through n my head and i couldn't think of any easy ways to do it
best answer in my head i came up with was two /31s instead of /30
but that wouldn't just work either
and if you have two /31s on private addresses going to internet /32
then that's a bit of a pita too, and there'd be shared link state up_the_irons: mercutio: you can't do vrrp with /30's without hoops that make it not worth it; i wouldn't be surprised if those hoops would crash Cisco IOS
mercutio: those who have /29's however, perhaps i could offer vrrp as a service, who knows... like, elective auto failover mercutio: up_the_irons: ok
yeh, well i'd rather just two bgp sessions myself...
but i realise that you're not going to have bgp sessions to everyone. up_the_irons: right mercutio: and you can't just go using /24s or such, because of everyone having their own vlan
which means that they wouldn't be able to reach people on the same /24 easily up_the_irons: yeah i can't stick entire /24's to one or the other
well, i could, but why..
the inbound traffic will come in through *both* routers
and ibgp (or perhaps ospf) will decide where to take it from there mercutio: i mean /24 with vrrp etc
3 ip addresses
you'd need bridging between vlans or such though
so then it gets more complicated
and it may remove some peoples assumptions about how the network works
ie would be change in design
cos i assume volume comes from vlan rather than from whether it's routed/bridged up_the_irons: yeah vrrp was not designed for that mercutio: well that'd just be to enable access within the /24
you'd also need ip/mac enforcement
so yaeh more complicated
but could save ip's.
but yeah i understand completely reluctance to go down that path up_the_irons: oh, that, no.. i decided against that common design long long ago
one customer == one vlan, solves so many practical problems
i would never do it another way :) mercutio: yeh you can still do the one customer to one vlan
but you could bridge between vlans for the same subnet
if wanting to use /24s rather than /30s
so like customer 1, vlan 4321, customer 2 vlan 4432, internal vlan 105 up_the_irons: i think i just barfed a little in my mouth mercutio: oops customer 2 4322 up_the_irons: ;) mercutio: so vlan4321 can bridge to vlan4322
ok up_the_irons: but what would bridge it? mercutio: but router is still on vlan105
the switch up_the_irons: customer vlan's terminate directly on the core (one day, perhaps an edge) mercutio: hmm up_the_irons: i'm not sure where the "internal" vlan would live mercutio: ok the router then up_the_irons: draw it out and show me if u wanna discuss more :) mercutio: there's also proxyarp
*cough*
i may be missing osmething, i'll try and have a further think about it in a few hours -: up_the_irons wanders off ***: amdprophet has quit IRC (Ping timeout: 245 seconds)
ant has quit IRC (Ping timeout: 264 seconds)
ant has joined #arpnetworks
heavysixer has quit IRC (Quit: heavysixer)
vin has joined #arpnetworks
vin has quit IRC (Quit: Killed by BlackJac (Requested by panasync))
amdprophet has joined #arpnetworks
mtve has quit IRC (Ping timeout: 255 seconds)
Webhostbudd_ has joined #arpnetworks
Webhostbudd has quit IRC (Ping timeout: 255 seconds)
HighJinx has quit IRC (Ping timeout: 252 seconds)
HighJinx has joined #arpnetworks
LEMONerd has joined #arpnetworks
LEMONerd is now known as LEMONed LEMONed: Garry there?