toorop: it'll be run again automatically, but if you'd like to let someone know, just reply to the email like brycec said *grumble grumble* not having a server out on the net is really inconvenient sometimes. time to sign up for arp again :) agreed, do it has the arpnetworks had network issues in the last 3 hours? i've noticed smokeping has lots of loss, but it's fine atm mercutio: my smokeping is quiet i have 4.11% average loss for last 3 hours, 44.42% max it was worse from 3 hours to 2 hours ago but it seems to alternate between rather broken and completely fine mercutio: Here are my last 8hrs, v4 and v6 http://imgur.com/jI587nI,PJg054g think I'm on kvr07 that's close but yeah you have no issues :) (yep, kvro7) it's usually fine heh i do too http://imgur.com/a/OvsG5#8qe0RNg does that work? I think it was in the default config loads yes see how it's fine some of the time completely and screwed other times ? and of course it's fine now when i noticed heh Well I'm going to blame the other end of your smokeping since... I can i'm monitoring other points too it looked like just arp effected but it could be pccw hardly anything goes via pccw except arp (My smokeping runs on a Chunkhost vps, looking inward at ARP) it's like pccw, trit, arp mercutio: bummer I go chunkhost - peer1 - arp literally 1 host between networks (never noticed that before, sweet) that's all peering traffic i imagine well there's someone else before pccw too http://i.imgur.com/PaJdJf0.png that's another jost in san jose but yeah not a global issue oh hmm, i do have monitoring of arp from somewhere else too hm yeah guess it's not a pccw issue, at least not in general. i should check lol it could be the node or it could be transit provider etc yeh it's fine from chicago i think i peer with peer1 up_the_irons: yeah you do 2 10ge-ten1-1.la-600w-cor-1.peer1.net (216.187.88.145) 0.371 ms 0.389 ms 0.418 ms chicago is via mzima 3 any2-ix.la.arpnetworks.com (206.223.143.166) 2.031 ms 2.391 ms 2.559 ms must be through the route server cuz i don't see them in my official peering list in both directions sweet i'm going to assume it's some pccw/trit issue i dunno what else goes via pccw/trit and it's nlayer on the way back from arp and if was nlayer issue more people would have issues i imagine yes oh, and because pccw/trit has had minor issues before http://smokeping.asininetech.com/smokeping.cgi?target=IRC.scruffy looking good oh hmm, i should be able to find some other host using pccw/trit 6. i-0-0-0-0.eqla01.bi.telstraglobal.net 0.0% 1979 174.0 125.8 125.5 204.1 5.2 7. 63-218-51-149.static.pccwglobal.net 0.2% 1979 132.6 149.8 130.2 503.5 41.1 my long running mtr found at least a little packet loss but not to destination it's really difficult to find good ways to check in between networks when tehre are so many of them err static did you not notice hte packet loss at the end? :) oh to multiple destinations it'e the same to farnsworth/scruffy so local issue most likely and bender and amy too oh packet loss started again 6. i-0-0-0-0.eqla01.bi.telstraglobal.net 0.0% 2402 125.9 125.7 125.5 204.1 4.9 hmm 7. 63-218-51-149.static.pccwglobal.net 0.2% 2402 130.5 150.0 130.2 503.5 41.4 8. 63-218-212-14.static.pccwglobal.net 2.1% 2401 221.2 159.4 157.4 354.4 11.0 9. cxc.r6.lax2.trit.net 1.6% 2401 195.8 134.5 133.5 197.5 7.4 10. arpnetworks-lax2-gw.cust.trit.net 2.2% 2401 195.5 133.0 126.2 472.0 26.0 11. arp 1.7% 2401 192.4 131.4 130.5 194.3 7.2 so yeh looks like a pccw issue to me the tarceroute from arp has that annoying nlayer icmp deprioritisation curiosuly the trace back from arp has this low/high alternating packet loss and so does the forward but not as obviously so i think ther'es some load balancing there 8. i-0-1-0.hptw-core01.bx.telstraglobal.net 0.8% 625 327.0 349.9 135.2 621.4 108.0 9. unknown.telstraglobal.net 1.3% 625 130.4 131.4 129.7 170.6 4.6 10. xe-0-3-0-1037.mdr.core.snap.net.nz 5.3% 625 131.2 134.5 130.6 222.4 12.9 11. 202-49-70-162.plain.net.nz 0.8% 625 131.9 131.3 130.4 288.2 6.9 12. 202.36.174.250 5.4% 625 131.3 132.5 130.5 195.3 10.4 but it heavily suggests the issue is sending to arp not receiving from arp which was already my hunch anyway < mercutio> the tarceroute from arp has that annoying nlayer icmp deprioritisation <--- so you finally watched that video, huh? ;) jlgaddis: i've always known about icmp deprioritisation thing is, that it's actually rather an annoying thing as there can be packet loss carried through too like hop 11/12 aren't deprioritising icmp but it makes it look like load balcnign issue icmp replies must be generated by the RP of a switch/router, so it is often rate limited, to save CPU resources now if all u kids would stop tracerouting all the time, maybe we wouldn't have to do that... ;) heh script kid if i may wow, in the whole bgp table there is only arpnetworks using pccw/trit combination so much for finding anothe rhost to monitor there's a Trit node in Amsterdam, but that's all i know... ;) this is from new zealand and pccw/trit interconnection is happening in los angeles oh actually i did find another it's just not preferred 199.249.180.0/23 but that's cogent/pccw/trit or xeex btw, I signed with NTT last week, so that circuit will be up soon oh cool what AS is ntt? it seems trit is mostly pccw using the not-very-accurate bgp.he.net/AS40193 ntt america is 2914 just found that searching for asn owrks beter than AS as number hmm wonder who AS703 is Verizon ahh verizon so yeh it shoudl be abl to go via as703 to ntt i dunno if that is good or bad :) actually i thought there was already NTT routes before oh i think mzima had NTT too to at&t for example mzima/nlayer/at&t err mzima/nlayer/ntt/at&t so will make it more direct to at&t oh well, so pccw issues should be abolished up_the_irons: how are you finding trit in general? sh ip bgp regexp ? i';m using openbgpd and transit-as 7. 63-218-51-149.static.pccwglobal.net 0.0% 254 132.7 151.0 130.2 399.0 43.6 8. 63-218-212-14.static.pccwglobal.net 10.6% 254 159.6 164.7 157.4 222.3 18.3 heh it's still happening btw, "wget ftp://ftp.arin.net/info/asn.txt", then you can just "grep -i whatever asn.txt" i normaly do whois AS in openbsd it seems to lack a few though but yeah i suppose that would make it easier to search name that works if you know the asn :P true good point openbsd doesn't have AS701 or AS703 i think linux does oh weird AS701 and AS703 are both UUNET... maybe they're not in some AS database that openbsd uses with whois cos they're legacy and i think AS701 is alter.net ? in traceroutes VZ yeh but it's show up as alter.net? some places still refer to it as uunet oh that's a nice text file mtr 161.159.0.1 like tracing to that shows alter.net actually not from arp arp shows ntt does your traceroute support -a ? i'll check nope -a == "show asn's", basically not even the linux one does debian/ubuntu: apt-get install traceroute-nanog oh hangon openbsd has -A whoa you learn something new every day yeh this is working 4 0.xo-1-0-0.XT4.AKL1.ALTER.NET (210.80.35.17) [AS703] 1.776 ms 1.776 ms 1.769 ms 5 0.so-0-0-0.IL2.LAX12.Alter.Net (210.80.50.89) [AS703] 193.172 ms 193.180 ms 193.397 ms 6 0.xe-3-3-0.IL4.LAX9.ALTER.NET (152.63.114.17) [AS701] 149.355 ms 152.660 ms 151.895 ms mercutio: trit is alright up_the_irons: does much go over it? the owner is my friend and we exchange traffic ahhh yeah, looks like traceroute-nanog is -A also. -a on os x. ahh mercutio: maybe 10% of my traffic; basically just the pccw routes heh it's the pccw routes i'm having issues with :) yeah u need to bitch at pccw for that :) i'm not pccw customer doesn't matter, neither am i i wonder if they'd listen arne't they like a china provider or something yeah most NOCs are responsive to problems, whether you are a customer or not. cuz you just amplify what their customers are probably already telling them. ahh usnoc@pccwglobal.com seems to be addresse -e im glad that ARP peers with Vocus, I have a Australian ircd node linked to it static: that should make things more stable http://bgp.he.net/AS9297#_asinfo my VPS provider in .au is in this AS ok i emailed them wonder if they'll say anything static: i know someone who has vocus transit in NZ it seems very on/off performance wise i think a lot of their problems come from using he.net I've only had it for a short time, but I've not noticed any significant issues yet but even running two downloads one after the other of a file can be a 10x speed diff i think it was like 4 or 5mb/sec to arp and like 200 to 2000k/sec to europe I have it linked over v6 so if Vocus dies for whatever reason, its gone hmm it's only 13.5 megabit/sec to arp atm vocus seem to be growing quick will be interesting to see if they try and be bottom of the barrrel, or offer a higher level of service i think AU market is like NZ market where people are cheap oh up_the_irons are NTT doing you ipv6 too? wow ipv6 is way faster to arp with vocus err from arp it's not faster to arp oh it is faster to too bizzare, seems to be performance issue with vocus to arp atm even with peering the diff in performance between ipv6 and ipv6 can be bizzare like even within one provider [ 3] 0.0-10.4 sec 9.25 MBytes 7.49 Mbits/sec [ 3] 0.0-10.2 sec 22.9 MBytes 18.7 Mbits/sec like that's ipv4 versus ipv6 vocus for both ipv4 and ipv6 and download was like 25 to 35 megabit which don't make a lot of sense either cos would be less change of congestion uploading to arp farnsworth.entropynet.net, how does it look from .nz? 113.9 msec over ipv6 54 msec over ipv4 via vocus 48 msec via other provider which is still via vocus the vocus one is 6 msec away from auckland though the v6 difference there is interesting so yeh it's good via ipv4 it's vocus all the way 6. ten-0-2-0-300.bdr02.akl01.akl.VOCUS.net.au 0.0% 20 7.7 8.3 7.6 12.0 1.0 7. 2402:7800:110:1::a 0.0% 20 31.7 31.9 30.7 38.1 1.6 8. ten-0-3-0-911.bdr02.per02.wa.VOCUS.net.au 0.0% 20 79.0 80.0 78.9 83.5 1.2 9. 2402:7800:40:1::d 0.0% 20 80.7 79.9 79.0 81.7 0.7 10. 2402:7800:30:1::15 0.0% 20 112.3 112.8 112.3 115.2 0.7 it has another vocus route at 11 with rdns mercutio: yeah, ipv6 too. will finally have a non-HE ipv6 provider. up_the_irons: yay :) \m/ i thought vocus dual homed all their routers on ipv4/ipv6 so it's weird when performance is diff actually the ipv4 route goes over ape which is auckland peering exchange and ipv6 is going over transit static: you could do a reverse trace to 2403:b700:1::6 up_the_irons: provider or peer? but i suspect the issue is routing via perth jlgaddis: provider, full transit nice yep:) and you're in adelaide? correct they should be able to reduce latency from adelaide to sydney soon looking on a map, it seems that it's liekly it goes via melbourne etc WAIA is in Adelaide now and Colocity is provisioning so your normal pings to sydney are 21 to 24 msec? and auckland to sydney is about 24 hmm by road it's 1410km and distance between sydney/auckland is 2153km by bird. so i imagine they can get latency down to 15/16 msec i suppose that doesn't make a lot of diff it doesn't i always underestimate how big australia is inter-Australia should be much better once the WAIA provisioning is done heh NZ to australia can suck at times well it's fine to sydney generally but once you hit domestic routing you can get stuff like adelaide going via perth i heard that some isp's have really broken routing internally too so the same happens to dsl users oh yes and like perth will go via sydney for internet case in point - AT&T/Bellsouth DSL in USA and brisbane and darwin i also heard that a whole lot of users in australia can still only get adsl1 with 1.5 megabit/384 kbit or something which uhh, means no high def youtube terrible congestion etc then there are transit providers like pipe networks with real high latency going via guam http://pastie.org/pastes/6385091/text?key=knxpf1n79zzlljnhufpqnq its not too bad for me from home i can't even find guam on the map atm i see hawaii isn't it near hawaii? just a bit west oh you have to zoom in it's like between papa new guinea and japan it's probably about as near to hawaii as LA is to NYC =) oh you're not in australia static? jlg: it's closer to hawaii than it is to japan to us mercutio: no, I'm in Canada err it's closer to US going via guam/hawaii than guam/japan 268 is really high ping that's what i get to UK well yes it's enough that it can be frustrating you have 165 msec from san jose to australia oh but it's mpls crap its only doing ircd, I don't even need to touch it except for any security updates with keeping the ping high from start point to end point suppose that's actually pretty good yeh not touching much is fine for that ping but if you used tmux/irc itd' be annoying as long as it stays linked in a stable manner, I'm happy heh what irc network is that? back years ago undernet used to have so many splits http://www.entropynet.net/ my small little thing now freenode is like undernet was Please provide the PCCW Global Circuit ID or Service ID. i told them not direct customer. heh err in my initial mail it does seem to have calmed down now though burgers plural o.O mm burgers milki: well up_the_irons is married with kids... o, so its not all just for him maybe ;) i miss in 'n out up_the_irons: is that new router coming in with the NTT link? mercutio: new router is up (just not doing anything). NTT link will go in it, yes. all VM hosts will then have cable to the new router. and finally, the ability to route in / out either one will be complete. brycec: milki : ok, well, one burger (double double, no onions, protein style) up_the_irons: ahh sweet, was wondering how that was going yeah, slowly but surely.. :) biggest part was getting the new upstream (research, negotiations, and finally signing) yeh i thought was two seperate things though :) but i suppose tying together isn't stupid how's fallback going to work? vrrp? mercutio: it will be a manual failover. shutting down vlan on one router and up'ing it on the other would effectively 'switch' a customer from one router to another. up'ing the vlan sends a gratuituous ARP, so the VMs would automatically pick up the new gateway MAC address. mercutio: no automatic fallback method was viable in my testing. they all caused more problems than they cured. failover should be an extremely rare situation, if it even occurs at all (we've already gone 4+ years on the first router) ok yeh i had a quick think through n my head and i couldn't think of any easy ways to do it best answer in my head i came up with was two /31s instead of /30 but that wouldn't just work either and if you have two /31s on private addresses going to internet /32 then that's a bit of a pita too, and there'd be shared link state mercutio: you can't do vrrp with /30's without hoops that make it not worth it; i wouldn't be surprised if those hoops would crash Cisco IOS mercutio: those who have /29's however, perhaps i could offer vrrp as a service, who knows... like, elective auto failover up_the_irons: ok yeh, well i'd rather just two bgp sessions myself... but i realise that you're not going to have bgp sessions to everyone. right and you can't just go using /24s or such, because of everyone having their own vlan which means that they wouldn't be able to reach people on the same /24 easily yeah i can't stick entire /24's to one or the other well, i could, but why.. the inbound traffic will come in through *both* routers and ibgp (or perhaps ospf) will decide where to take it from there i mean /24 with vrrp etc 3 ip addresses you'd need bridging between vlans or such though so then it gets more complicated and it may remove some peoples assumptions about how the network works ie would be change in design cos i assume volume comes from vlan rather than from whether it's routed/bridged yeah vrrp was not designed for that well that'd just be to enable access within the /24 you'd also need ip/mac enforcement so yaeh more complicated but could save ip's. but yeah i understand completely reluctance to go down that path oh, that, no.. i decided against that common design long long ago one customer == one vlan, solves so many practical problems i would never do it another way :) yeh you can still do the one customer to one vlan but you could bridge between vlans for the same subnet if wanting to use /24s rather than /30s so like customer 1, vlan 4321, customer 2 vlan 4432, internal vlan 105 i think i just barfed a little in my mouth oops customer 2 4322 ;) so vlan4321 can bridge to vlan4322 ok but what would bridge it? but router is still on vlan105 the switch customer vlan's terminate directly on the core (one day, perhaps an edge) hmm i'm not sure where the "internal" vlan would live ok the router then draw it out and show me if u wanna discuss more :) there's also proxyarp *cough* i may be missing osmething, i'll try and have a further think about it in a few hours Garry there?