#arpnetworks 2013-03-04,Mon

i have 4.11% average loss for last 3 hours, 44.42% max
it was worse from 3 hours to 2 hours ago
but it seems to alternate between rather broken and completely fine

[14:13]

brycec

mercutio: Here are my last 8hrs, v4 and v6 http://imgur.com/jI587nI,PJg054g
think I'm on kvr07

[14:14]

mercutio

that's close
but yeah you have no issues :)

[14:14]

brycec

(yep, kvro7)

[14:15]

mercutio

it's usually fine

[14:16]

brycec

brycec wonders why he bothers smokepinging localhost

[14:16]

mercutio

heh
i do too
http://imgur.com/a/OvsG5#8qe0RNg
does that work?

[14:16]

brycec

I think it was in the default config
loads yes

[14:18]

mercutio

see how it's fine some of the time completely
and screwed other times ?
and of course it's fine now when i noticed

[14:18]

brycec

heh
Well I'm going to blame the other end of your smokeping
since... I can

[14:18]

mercutio

i'm monitoring other points too
it looked like just arp effected
but it could be pccw
hardly anything goes via pccw except arp

[14:19]

brycec

(My smokeping runs on a Chunkhost vps, looking inward at ARP)

[14:20]

mercutio

it's like pccw, trit, arp

[14:20]

brycec

mercutio: bummer
I go chunkhost - peer1 - arp
literally 1 host between networks
(never noticed that before, sweet)

[14:21]

mercutio

that's all peering traffic i imagine
well there's someone else before pccw too

http://i.imgur.com/PaJdJf0.png

that's another jost in san jose
but yeah not a global issue
oh hmm, i do have monitoring of arp from somewhere else too

[14:21]

brycec

hm yeah guess it's not a pccw issue, at least not in general.

[14:23]

mercutio

i should check

[14:23]

brycec

lol

[14:23]

mercutio

it could be the node or it could be transit provider etc
yeh it's fine from chicago

[14:23]

up_the_irons

i think i peer with peer1

[14:24]

brycec

up_the_irons: yeah
you do
2 10ge-ten1-1.la-600w-cor-1.peer1.net (216.187.88.145) 0.371 ms 0.389 ms 0.418 ms

[14:25]

mercutio

chicago is via mzima

[14:25]

brycec

3 any2-ix.la.arpnetworks.com (206.223.143.166) 2.031 ms 2.391 ms 2.559 ms

[14:25]

up_the_irons

must be through the route server cuz i don't see them in my official peering list

[14:25]

mercutio

in both directions

[14:25]

up_the_irons

sweet

[14:25]

mercutio

i'm going to assume it's some pccw/trit issue
i dunno what else goes via pccw/trit
and it's nlayer on the way back from arp
and if was nlayer issue more people would have issues i imagine

[14:26]

staticsafe

yes

[14:27]

mercutio

oh, and because pccw/trit has had minor issues before

[14:27]

staticsafe

http://smokeping.asininetech.com/smokeping.cgi?target=IRC.scruffy looking good

[14:27]

mercutio

oh hmm, i should be able to find some other host using pccw/trit
6. i-0-0-0-0.eqla01.bi.telstraglobal.net 0.0% 1979 174.0 125.8 125.5 204.1 5.2
7. 63-218-51-149.static.pccwglobal.net 0.2% 1979 132.6 149.8 130.2 503.5 41.1
my long running mtr found at least a little packet loss
but not to destination
it's really difficult to find good ways to check in between networks
when tehre are so many of them
err
static
did you not notice hte packet loss at the end? :)
oh
to multiple destinations
it'e the same to farnsworth/scruffy so local issue most likely
and bender and amy too
oh packet loss started again
6. i-0-0-0-0.eqla01.bi.telstraglobal.net 0.0% 2402 125.9 125.7 125.5 204.1 4.9

[14:28]

staticsafe

hmm

[14:31]

mercutio

7. 63-218-51-149.static.pccwglobal.net 0.2% 2402 130.5 150.0 130.2 503.5 41.4
8. 63-218-212-14.static.pccwglobal.net 2.1% 2401 221.2 159.4 157.4 354.4 11.0
9. cxc.r6.lax2.trit.net 1.6% 2401 195.8 134.5 133.5 197.5 7.4
10. arpnetworks-lax2-gw.cust.trit.net 2.2% 2401 195.5 133.0 126.2 472.0 26.0
11. arp 1.7% 2401 192.4 131.4 130.5 194.3 7.2
so yeh looks like a pccw issue to me
the tarceroute from arp has that annoying nlayer icmp deprioritisation
curiosuly the trace back from arp has this low/high alternating packet loss
and so does the forward but not as obviously
so i think ther'es some load balancing there
8. i-0-1-0.hptw-core01.bx.telstraglobal.net 0.8% 625 327.0 349.9 135.2 621.4 108.0
9. unknown.telstraglobal.net 1.3% 625 130.4 131.4 129.7 170.6 4.6
10. xe-0-3-0-1037.mdr.core.snap.net.nz 5.3% 625 131.2 134.5 130.6 222.4 12.9
11. 202-49-70-162.plain.net.nz 0.8% 625 131.9 131.3 130.4 288.2 6.9
12. 202.36.174.250 5.4% 625 131.3 132.5 130.5 195.3 10.4
but it heavily suggests the issue is sending to arp not receiving from arp
which was already my hunch anyway

[14:31]

jlgaddis

< mercutio> the tarceroute from arp has that annoying nlayer icmp deprioritisation <--- so you finally watched that video, huh? ;)

[14:43]

mercutio

jlgaddis: i've always known about icmp deprioritisation
thing is, that it's actually rather an annoying thing
as there can be packet loss carried through too
like hop 11/12 aren't deprioritising icmp
but it makes it look like load balcnign issue

[14:45]

up_the_irons

icmp replies must be generated by the RP of a switch/router, so it is often rate limited, to save CPU resources
now if all u kids would stop tracerouting all the time, maybe we wouldn't have to do that... ;)

[14:52]

staticsafe

heh

[14:52]

jbergstroem

script kid if i may

[14:53]

mercutio

wow, in the whole bgp table there is only arpnetworks using pccw/trit combination
so much for finding anothe rhost to monitor

[14:53]

up_the_irons

there's a Trit node in Amsterdam, but that's all i know... ;)

[14:54]

mercutio

this is from new zealand
and pccw/trit interconnection is happening in los angeles
oh actually i did find another it's just not preferred
199.249.180.0/23
but that's cogent/pccw/trit
or xeex

[14:54]

up_the_irons

btw, I signed with NTT last week, so that circuit will be up soon

[14:58]

mercutio

oh cool
what AS is ntt?
mercutio wonders if it'll route via it
it seems trit is mostly pccw
using the not-very-accurate bgp.he.net/AS40193

[14:58]

jlgaddis

ntt america is 2914

[15:01]

mercutio

just found that
searching for asn owrks beter than AS
as number
hmm wonder who AS703 is

[15:01]

staticsafe

Verizon

[15:02]

mercutio

ahh verizon
so yeh it shoudl be abl to go via as703 to ntt
i dunno if that is good or bad :)
actually i thought there was already NTT routes before
oh i think mzima had NTT too
to at&t for example
mzima/nlayer/at&t
err mzima/nlayer/ntt/at&t
so will make it more direct to at&t
oh well, so pccw issues should be abolished
up_the_irons: how are you finding trit in general?

[15:03]

jlgaddis

sh ip bgp regexp ?

[15:05]

mercutio

i';m using openbgpd
and transit-as
7. 63-218-51-149.static.pccwglobal.net 0.0% 254 132.7 151.0 130.2 399.0 43.6
8. 63-218-212-14.static.pccwglobal.net 10.6% 254 159.6 164.7 157.4 222.3 18.3
heh it's still happening

[15:05]

jlgaddis

btw, "wget ftp://ftp.arin.net/info/asn.txt", then you can just "grep -i whatever asn.txt"

[15:06]

mercutio

i normaly do whois AS<blah>
in openbsd it seems to lack a few though
but yeah i suppose that would make it easier to search name

[15:06]

jlgaddis

that works if you know the asn :P
jlgaddis nods

[15:06]

mercutio

true good point
openbsd doesn't have AS701 or AS703
i think linux does
oh weird
AS701 and AS703 are both UUNET...
maybe they're not in some AS database that openbsd uses with whois cos they're legacy
and i think AS701 is alter.net ?
in traceroutes

[15:06]

jlgaddis

[15:08]

mercutio

yeh but it's show up as alter.net?

[15:08]

jlgaddis

some places still refer to it as uunet

[15:09]

mercutio

oh that's a nice text file
mtr 161.159.0.1
like tracing to that shows alter.net
actually not from arp
arp shows ntt

[15:09]

jlgaddis

does your traceroute support -a
?

[15:10]

mercutio

i'll check
nope

[15:10]

jlgaddis

-a == "show asn's", basically

[15:10]

mercutio

not even the linux one does

[15:10]

jlgaddis

debian/ubuntu: apt-get install traceroute-nanog

[15:10]

mercutio

oh hangon
openbsd has -A

[15:11]

up_the_irons

whoa
you learn something new every day

[15:11]

mercutio

yeh this is working
4 0.xo-1-0-0.XT4.AKL1.ALTER.NET (210.80.35.17) [AS703] 1.776 ms 1.776 ms 1.769 ms
5 0.so-0-0-0.IL2.LAX12.Alter.Net (210.80.50.89) [AS703] 193.172 ms 193.180 ms 193.397 ms
6 0.xe-3-3-0.IL4.LAX9.ALTER.NET (152.63.114.17) [AS701] 149.355 ms 152.660 ms 151.895 ms

[15:11]

up_the_irons

mercutio: trit is alright

[15:11]

mercutio

up_the_irons: does much go over it?

[15:11]

up_the_irons

the owner is my friend and we exchange traffic

[15:12]

***

heavysixer has quit IRC (Quit: heavysixer)

[15:12]

jlgaddis

ahhh yeah, looks like traceroute-nanog is -A also. -a on os x.

[15:12]

mercutio

ahh

[15:12]

up_the_irons

mercutio: maybe 10% of my traffic; basically just the pccw routes

[15:12]

mercutio

heh
it's the pccw routes i'm having issues with :)

[15:12]

up_the_irons

yeah u need to bitch at pccw for that :)

[15:13]

mercutio

i'm not pccw customer

[15:13]

up_the_irons

doesn't matter, neither am i

[15:13]

mercutio

i wonder if they'd listen
arne't they like a china provider or something

[15:13]

up_the_irons

yeah
most NOCs are responsive to problems, whether you are a customer or not. cuz you just amplify what their customers are probably already telling them.

[15:14]

mercutio

ahh
usnoc@pccwglobal.com
seems to be addresse
-e

[15:15]

staticsafe

im glad that ARP peers with Vocus, I have a Australian ircd node linked to it

[15:16]

mercutio

static: that should make things more stable

[15:17]

staticsafe

http://bgp.he.net/AS9297#_asinfo my VPS provider in .au is in this AS

[15:17]

mercutio

ok
i emailed them
wonder if they'll say anything
static: i know someone who has vocus transit in NZ
it seems very on/off performance wise
i think a lot of their problems come from using he.net

[15:18]

staticsafe

I've only had it for a short time, but I've not noticed any significant issues yet

[15:19]

mercutio

but even running two downloads one after the other of a file can be a 10x speed diff
i think it was like 4 or 5mb/sec to arp
and like 200 to 2000k/sec to europe

[15:19]

staticsafe

I have it linked over v6 so if Vocus dies for whatever reason, its gone

[15:21]

mercutio

hmm it's only 13.5 megabit/sec to arp atm
vocus seem to be growing quick
will be interesting to see if they try and be bottom of the barrrel, or offer a higher level of service
i think AU market is like NZ market where people are cheap
oh up_the_irons are NTT doing you ipv6 too?
wow
ipv6 is way faster to arp with vocus
err from arp
it's not faster to arp
oh it is faster to too
bizzare, seems to be performance issue with vocus to arp atm even with peering
the diff in performance between ipv6 and ipv6 can be bizzare
like even within one provider
[ 3] 0.0-10.4 sec 9.25 MBytes 7.49 Mbits/sec
[ 3] 0.0-10.2 sec 22.9 MBytes 18.7 Mbits/sec
like that's ipv4 versus ipv6
vocus for both ipv4 and ipv6
and download was like 25 to 35 megabit
which don't make a lot of sense either
cos would be less change of congestion uploading to arp

[15:21]

staticsafe

farnsworth.entropynet.net, how does it look from .nz?

[15:33]

mercutio

113.9 msec
over ipv6
54 msec over ipv4
via vocus
48 msec via other provider
which is still via vocus
the vocus one is 6 msec away from auckland though

[15:35]

staticsafe

the v6 difference there is interesting

[15:36]

mercutio

so yeh it's good via ipv4
it's vocus all the way
6. ten-0-2-0-300.bdr02.akl01.akl.VOCUS.net.au 0.0% 20 7.7 8.3 7.6 12.0 1.0
7. 2402:7800:110:1::a 0.0% 20 31.7 31.9 30.7 38.1 1.6
8. ten-0-3-0-911.bdr02.per02.wa.VOCUS.net.au 0.0% 20 79.0 80.0 78.9 83.5 1.2
9. 2402:7800:40:1::d 0.0% 20 80.7 79.9 79.0 81.7 0.7
10. 2402:7800:30:1::15 0.0% 20 112.3 112.8 112.3 115.2 0.7
it has another vocus route at 11 with rdns

[15:36]

up_the_irons

mercutio: yeah, ipv6 too. will finally have a non-HE ipv6 provider.

[15:37]

mercutio

up_the_irons: yay :)

[15:37]

up_the_irons

[15:37]

mercutio

i thought vocus dual homed all their routers on ipv4/ipv6
so it's weird when performance is diff
actually the ipv4 route goes over ape which is auckland peering exchange
and ipv6 is going over transit
static: you could do a reverse trace to 2403:b700:1::6

[15:37]

jlgaddis

up_the_irons: provider or peer?

[15:40]

mercutio

but i suspect the issue is routing via perth

[15:40]

up_the_irons

jlgaddis: provider, full transit

[15:40]

jlgaddis

nice

[15:40]

up_the_irons

yep:)

[15:40]

mercutio

and you're in adelaide?

[15:41]

staticsafe

correct

[15:41]

mercutio

they should be able to reduce latency from adelaide to sydney

[15:41]

staticsafe

soon

[15:41]

mercutio

looking on a map, it seems that it's liekly it goes via melbourne etc

[15:42]

staticsafe

WAIA is in Adelaide now and Colocity is provisioning

[15:42]

mercutio

so your normal pings to sydney are 21 to 24 msec?
and auckland to sydney is about 24
hmm by road it's 1410km
and distance between sydney/auckland is 2153km by bird.
so i imagine they can get latency down to 15/16 msec
i suppose that doesn't make a lot of diff

[15:42]

staticsafe

it doesn't

[15:48]

mercutio

i always underestimate how big australia is

[15:48]

staticsafe

inter-Australia should be much better once the WAIA provisioning is done
heh

[15:48]

mercutio

NZ to australia can suck at times
well it's fine to sydney generally
but once you hit domestic routing you can get stuff like adelaide going via perth
i heard that some isp's have really broken routing internally too
so the same happens to dsl users

[15:48]

staticsafe

oh yes

[15:49]

mercutio

and like perth will go via sydney for internet

[15:49]

staticsafe

case in point - AT&T/Bellsouth DSL
in USA

[15:49]

mercutio

and brisbane
and darwin
i also heard that a whole lot of users in australia can still only get adsl1 with 1.5 megabit/384 kbit or something
which uhh, means no high def youtube
terrible congestion etc
then there are transit providers like pipe networks with real high latency going via guam

[15:49]

staticsafe

http://pastie.org/pastes/6385091/text?key=knxpf1n79zzlljnhufpqnq its not too bad for me from home

[15:51]

mercutio

i can't even find guam on the map atm
i see hawaii
isn't it near hawaii?

[15:51]

jlgaddis

just a bit west

[15:52]

mercutio

oh you have to zoom in
it's like between papa new guinea and japan

[15:52]

jlgaddis

it's probably about as near to hawaii as LA is to NYC =)

[15:52]

mercutio

oh you're not in australia static?
jlg: it's closer to hawaii than it is to japan to us

[15:52]

staticsafe

mercutio: no, I'm in Canada

[15:53]

mercutio

err it's closer to US going via guam/hawaii than guam/japan
268 is really high ping
that's what i get to UK

[15:53]

staticsafe

well yes

[15:53]

mercutio

it's enough that it can be frustrating
you have 165 msec from san jose to australia
oh but it's mpls crap

[15:53]

staticsafe

its only doing ircd, I don't even need to touch it except for any security updates

[15:54]

mercutio

with keeping the ping high from start point to end point
suppose that's actually pretty good
yeh
not touching much is fine for that ping
but if you used tmux/irc itd' be annoying

[15:54]

staticsafe

as long as it stays linked in a stable manner, I'm happy

[15:55]

mercutio

heh
what irc network is that?
back years ago undernet used to have so many splits

[15:55]

staticsafe

http://www.entropynet.net/ my small little thing

[15:56]

mercutio

now freenode is like undernet was

[15:56]

...... (idle for 26mn)

***

heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer

[16:22]

mercutio

Please provide the PCCW Global Circuit ID or Service ID.
i told them not direct customer.

[16:27]

staticsafe

heh

[16:28]

..... (idle for 24mn)

mercutio

err in my initial mail
it does seem to have calmed down now though

[16:52]

up_the_irons

up_the_irons goes and grabs In n' Out burgers

[16:53]

***

heavysixer has quit IRC (Quit: heavysixer)

[16:54]

milki

burgers plural o.O

[16:57]

staticsafe

mm burgers

[16:58]

brycec

milki: well up_the_irons is married with kids...

[16:58]

milki

o, so its not all just for him

[16:59]

brycec

maybe ;)

[16:59]

....... (idle for 30mn)

jlgaddis

i miss in 'n out

[17:29]

brycec

brycec doesn't... it's just 2miaway

[17:31]

mercutio

up_the_irons: is that new router coming in with the NTT link?

[17:33]

......... (idle for 41mn)

up_the_irons

mercutio: new router is up (just not doing anything). NTT link will go in it, yes. all VM hosts will then have cable to the new router. and finally, the ability to route in / out either one will be complete.
brycec: milki : ok, well, one burger (double double, no onions, protein style)

[18:14]

mercutio

up_the_irons: ahh sweet, was wondering how that was going

[18:17]

up_the_irons

yeah, slowly but surely.. :)
biggest part was getting the new upstream (research, negotiations, and finally signing)

[18:17]

mercutio

yeh i thought was two seperate things though :)
but i suppose tying together isn't stupid
how's fallback going to work? vrrp?

[18:26]

..... (idle for 22mn)

***

heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer

[18:48]

up_the_irons

mercutio: it will be a manual failover. shutting down vlan on one router and up'ing it on the other would effectively 'switch' a customer from one router to another. up'ing the vlan sends a gratuituous ARP, so the VMs would automatically pick up the new gateway MAC address.
mercutio: no automatic fallback method was viable in my testing. they all caused more problems than they cured. failover should be an extremely rare situation, if it even occurs at all (we've already gone 4+ years on the first router)

[18:50]

mercutio

ok
yeh i had a quick think through n my head and i couldn't think of any easy ways to do it
best answer in my head i came up with was two /31s instead of /30
but that wouldn't just work either
and if you have two /31s on private addresses going to internet /32
then that's a bit of a pita too, and there'd be shared link state

[18:53]

up_the_irons

mercutio: you can't do vrrp with /30's without hoops that make it not worth it; i wouldn't be surprised if those hoops would crash Cisco IOS
mercutio: those who have /29's however, perhaps i could offer vrrp as a service, who knows... like, elective auto failover

[18:56]

mercutio

up_the_irons: ok
yeh, well i'd rather just two bgp sessions myself...
but i realise that you're not going to have bgp sessions to everyone.

[19:02]

up_the_irons

right

[19:03]

mercutio

and you can't just go using /24s or such, because of everyone having their own vlan
which means that they wouldn't be able to reach people on the same /24 easily

[19:04]

up_the_irons

yeah i can't stick entire /24's to one or the other
well, i could, but why..
the inbound traffic will come in through *both* routers
and ibgp (or perhaps ospf) will decide where to take it from there

[19:05]

mercutio

i mean /24 with vrrp etc
3 ip addresses
you'd need bridging between vlans or such though
so then it gets more complicated
and it may remove some peoples assumptions about how the network works
ie would be change in design
cos i assume volume comes from vlan rather than from whether it's routed/bridged

[19:12]

up_the_irons

yeah vrrp was not designed for that

[19:14]

mercutio

well that'd just be to enable access within the /24
you'd also need ip/mac enforcement
so yaeh more complicated
but could save ip's.
but yeah i understand completely reluctance to go down that path

[19:14]

up_the_irons

oh, that, no.. i decided against that common design long long ago
one customer == one vlan, solves so many practical problems
i would never do it another way :)

[19:15]

mercutio

yeh you can still do the one customer to one vlan
but you could bridge between vlans for the same subnet
if wanting to use /24s rather than /30s
so like customer 1, vlan 4321, customer 2 vlan 4432, internal vlan 105

[19:15]

up_the_irons

i think i just barfed a little in my mouth

[19:16]

mercutio

oops customer 2 4322

[19:16]

up_the_irons

;)

[19:17]

mercutio

so vlan4321 can bridge to vlan4322
ok