[00:18] *** nixbag has quit IRC (Quit: leaving) [00:20] *** nixbag has joined #arpnetworks [00:43] *** nixbag has quit IRC (Quit: leaving) [00:50] *** nixbag has joined #arpnetworks [01:29] *** amdprophet has joined #arpnetworks [02:08] toorop: it'll be run again automatically, but if you'd like to let someone know, just reply to the email [02:08] like brycec said [04:47] *** dj_goku has quit IRC (Ping timeout: 256 seconds) [05:29] *** heavysixer has joined #arpnetworks [05:29] *** ChanServ sets mode: +o heavysixer [05:42] *** heavysixer has quit IRC (Quit: heavysixer) [05:46] *** dj_goku has joined #arpnetworks [06:54] *** dj_goku has quit IRC (Ping timeout: 255 seconds) [07:10] *** raptelan has joined #arpnetworks [07:10] *grumble grumble* not having a server out on the net is really inconvenient sometimes. [07:10] time to sign up for arp again :) [07:17] *** LEMONed has joined #arpnetworks [07:27] *** LEMONed has quit IRC (Ping timeout: 260 seconds) [07:50] *** raptelan has quit IRC (Quit: leaving) [08:12] *** raptelan has joined #arpnetworks [08:26] *** WaREX has quit IRC (Quit: leaving) [08:30] agreed, do it [09:55] *** raptelan has quit IRC (Ping timeout: 245 seconds) [09:56] *** raptelan has joined #arpnetworks [10:07] *** raptelan has quit IRC (Ping timeout: 260 seconds) [10:07] *** raptelan has joined #arpnetworks [10:21] *** raptelan has quit IRC (Ping timeout: 245 seconds) [11:51] *** phlux has quit IRC (Quit: WeeChat 0.4.1-dev) [11:52] *** phlux has joined #arpnetworks [13:27] *** heavysixer has joined #arpnetworks [13:27] *** ChanServ sets mode: +o heavysixer [14:10] * up_the_irons nods [14:11] * brycec nods [14:12] has the arpnetworks had network issues in the last 3 hours? [14:13] i've noticed smokeping has lots of loss, but it's fine atm [14:13] mercutio: my smokeping is quiet [14:13] i have 4.11% average loss for last 3 hours, 44.42% max [14:14] it was worse from 3 hours to 2 hours ago [14:14] but it seems to alternate between rather broken and completely fine [14:14] mercutio: Here are my last 8hrs, v4 and v6 http://imgur.com/jI587nI,PJg054g [14:14] think I'm on kvr07 [14:14] that's close [14:14] but yeah you have no issues :) [14:15] (yep, kvro7) [14:16] it's usually fine [14:16] * brycec wonders why he bothers smokepinging localhost [14:16] heh [14:16] i do too [14:17] http://imgur.com/a/OvsG5#8qe0RNg [14:18] does that work? [14:18] I think it was in the default config [14:18] loads yes [14:18] see how it's fine some of the time completely [14:18] and screwed other times ? [14:18] and of course it's fine now when i noticed [14:18] heh [14:19] Well I'm going to blame the other end of your smokeping [14:19] since... I can [14:19] i'm monitoring other points too [14:19] it looked like just arp effected [14:19] but it could be pccw [14:19] hardly anything goes via pccw except arp [14:20] (My smokeping runs on a Chunkhost vps, looking inward at ARP) [14:20] it's like pccw, trit, arp [14:21] mercutio: bummer [14:21] I go chunkhost - peer1 - arp [14:21] literally 1 host between networks [14:21] (never noticed that before, sweet) [14:21] that's all peering traffic i imagine [14:21] well there's someone else before pccw too [14:22] http://i.imgur.com/PaJdJf0.png [14:22] that's another jost in san jose [14:22] but yeah not a global issue [14:23] oh hmm, i do have monitoring of arp from somewhere else too [14:23] hm yeah guess it's not a pccw issue, at least not in general. [14:23] i should check [14:23] lol [14:23] it could be the node or it could be transit provider etc [14:23] yeh it's fine from chicago [14:24] i think i peer with peer1 [14:25] up_the_irons: yeah [14:25] you do [14:25] 2 10ge-ten1-1.la-600w-cor-1.peer1.net (216.187.88.145) 0.371 ms 0.389 ms 0.418 ms [14:25] chicago is via mzima [14:25] 3 any2-ix.la.arpnetworks.com (206.223.143.166) 2.031 ms 2.391 ms 2.559 ms [14:25] must be through the route server cuz i don't see them in my official peering list [14:25] in both directions [14:25] sweet [14:26] i'm going to assume it's some pccw/trit issue [14:26] i dunno what else goes via pccw/trit [14:26] and it's nlayer on the way back from arp [14:26] and if was nlayer issue more people would have issues i imagine [14:27] yes [14:27] oh, and because pccw/trit has had minor issues before [14:27] http://smokeping.asininetech.com/smokeping.cgi?target=IRC.scruffy looking good [14:28] oh hmm, i should be able to find some other host using pccw/trit [14:28] 6. i-0-0-0-0.eqla01.bi.telstraglobal.net 0.0% 1979 174.0 125.8 125.5 204.1 5.2 [14:28] 7. 63-218-51-149.static.pccwglobal.net 0.2% 1979 132.6 149.8 130.2 503.5 41.1 [14:28] my long running mtr found at least a little packet loss [14:28] but not to destination [14:29] it's really difficult to find good ways to check in between networks [14:29] when tehre are so many of them [14:29] err [14:29] static [14:29] did you not notice hte packet loss at the end? :) [14:30] oh [14:30] to multiple destinations [14:30] it'e the same to farnsworth/scruffy so local issue most likely [14:30] and bender and amy too [14:31] oh packet loss started again [14:31] 6. i-0-0-0-0.eqla01.bi.telstraglobal.net 0.0% 2402 125.9 125.7 125.5 204.1 4.9 [14:31] hmm [14:31] 7. 63-218-51-149.static.pccwglobal.net 0.2% 2402 130.5 150.0 130.2 503.5 41.4 [14:31] 8. 63-218-212-14.static.pccwglobal.net 2.1% 2401 221.2 159.4 157.4 354.4 11.0 [14:31] 9. cxc.r6.lax2.trit.net 1.6% 2401 195.8 134.5 133.5 197.5 7.4 [14:32] 10. arpnetworks-lax2-gw.cust.trit.net 2.2% 2401 195.5 133.0 126.2 472.0 26.0 [14:32] 11. arp 1.7% 2401 192.4 131.4 130.5 194.3 7.2 [14:32] so yeh looks like a pccw issue to me [14:32] the tarceroute from arp has that annoying nlayer icmp deprioritisation [14:35] curiosuly the trace back from arp has this low/high alternating packet loss [14:35] and so does the forward but not as obviously [14:35] so i think ther'es some load balancing there [14:36] 8. i-0-1-0.hptw-core01.bx.telstraglobal.net 0.8% 625 327.0 349.9 135.2 621.4 108.0 [14:36] 9. unknown.telstraglobal.net 1.3% 625 130.4 131.4 129.7 170.6 4.6 [14:36] 10. xe-0-3-0-1037.mdr.core.snap.net.nz 5.3% 625 131.2 134.5 130.6 222.4 12.9 [14:36] 11. 202-49-70-162.plain.net.nz 0.8% 625 131.9 131.3 130.4 288.2 6.9 [14:36] 12. 202.36.174.250 5.4% 625 131.3 132.5 130.5 195.3 10.4 [14:36] but it heavily suggests the issue is sending to arp not receiving from arp [14:36] which was already my hunch anyway [14:43] < mercutio> the tarceroute from arp has that annoying nlayer icmp deprioritisation <--- so you finally watched that video, huh? ;) [14:45] jlgaddis: i've always known about icmp deprioritisation [14:45] thing is, that it's actually rather an annoying thing [14:46] as there can be packet loss carried through too [14:46] like hop 11/12 aren't deprioritising icmp [14:46] but it makes it look like load balcnign issue [14:52] icmp replies must be generated by the RP of a switch/router, so it is often rate limited, to save CPU resources [14:52] now if all u kids would stop tracerouting all the time, maybe we wouldn't have to do that... ;) [14:52] heh [14:53] script kid if i may [14:53] wow, in the whole bgp table there is only arpnetworks using pccw/trit combination [14:53] so much for finding anothe rhost to monitor [14:54] there's a Trit node in Amsterdam, but that's all i know... ;) [14:54] this is from new zealand [14:54] and pccw/trit interconnection is happening in los angeles [14:55] oh actually i did find another it's just not preferred [14:55] 199.249.180.0/23 [14:56] but that's cogent/pccw/trit [14:56] or xeex [14:58] btw, I signed with NTT last week, so that circuit will be up soon [14:58] oh cool [14:58] what AS is ntt? [14:58] * mercutio wonders if it'll route via it [15:00] it seems trit is mostly pccw [15:01] using the not-very-accurate bgp.he.net/AS40193 [15:01] ntt america is 2914 [15:01] just found that [15:02] searching for asn owrks beter than AS [15:02] as number [15:02] hmm wonder who AS703 is [15:02] Verizon [15:03] ahh verizon [15:03] so yeh it shoudl be abl to go via as703 to ntt [15:03] i dunno if that is good or bad :) [15:04] actually i thought there was already NTT routes before [15:04] oh i think mzima had NTT too [15:04] to at&t for example [15:04] mzima/nlayer/at&t [15:04] err mzima/nlayer/ntt/at&t [15:04] so will make it more direct to at&t [15:05] oh well, so pccw issues should be abolished [15:05] up_the_irons: how are you finding trit in general? [15:05] sh ip bgp regexp ? [15:05] i';m using openbgpd [15:05] and transit-as [15:05] 7. 63-218-51-149.static.pccwglobal.net 0.0% 254 132.7 151.0 130.2 399.0 43.6 [15:05] 8. 63-218-212-14.static.pccwglobal.net 10.6% 254 159.6 164.7 157.4 222.3 18.3 [15:05] heh it's still happening [15:06] btw, "wget ftp://ftp.arin.net/info/asn.txt", then you can just "grep -i whatever asn.txt" [15:06] i normaly do whois AS [15:06] in openbsd it seems to lack a few though [15:06] but yeah i suppose that would make it easier to search name [15:06] that works if you know the asn :P [15:06] * jlgaddis nods [15:06] true good point [15:07] openbsd doesn't have AS701 or AS703 [15:07] i think linux does [15:07] oh weird [15:07] AS701 and AS703 are both UUNET... [15:08] maybe they're not in some AS database that openbsd uses with whois cos they're legacy [15:08] and i think AS701 is alter.net ? [15:08] in traceroutes [15:08] VZ [15:08] yeh but it's show up as alter.net? [15:09] some places still refer to it as uunet [15:09] oh that's a nice text file [15:09] mtr 161.159.0.1 [15:09] like tracing to that shows alter.net [15:10] actually not from arp [15:10] arp shows ntt [15:10] does your traceroute support -a [15:10] ? [15:10] i'll check [15:10] nope [15:10] -a == "show asn's", basically [15:10] not even the linux one does [15:10] debian/ubuntu: apt-get install traceroute-nanog [15:11] oh hangon [15:11] openbsd has -A [15:11] whoa [15:11] you learn something new every day [15:11] yeh this is working [15:11] 4 0.xo-1-0-0.XT4.AKL1.ALTER.NET (210.80.35.17) [AS703] 1.776 ms 1.776 ms 1.769 ms [15:11] 5 0.so-0-0-0.IL2.LAX12.Alter.Net (210.80.50.89) [AS703] 193.172 ms 193.180 ms 193.397 ms [15:11] 6 0.xe-3-3-0.IL4.LAX9.ALTER.NET (152.63.114.17) [AS701] 149.355 ms 152.660 ms 151.895 ms [15:11] mercutio: trit is alright [15:11] up_the_irons: does much go over it? [15:12] the owner is my friend and we exchange traffic [15:12] *** heavysixer has quit IRC (Quit: heavysixer) [15:12] ahhh yeah, looks like traceroute-nanog is -A also. -a on os x. [15:12] ahh [15:12] mercutio: maybe 10% of my traffic; basically just the pccw routes [15:12] heh [15:12] it's the pccw routes i'm having issues with :) [15:13] yeah u need to bitch at pccw for that :) [15:13] i'm not pccw customer [15:13] doesn't matter, neither am i [15:13] i wonder if they'd listen [15:13] arne't they like a china provider or something [15:14] yeah [15:14] most NOCs are responsive to problems, whether you are a customer or not. cuz you just amplify what their customers are probably already telling them. [15:15] ahh [15:15] usnoc@pccwglobal.com [15:15] seems to be addresse [15:15] -e [15:16] im glad that ARP peers with Vocus, I have a Australian ircd node linked to it [15:17] static: that should make things more stable [15:17] http://bgp.he.net/AS9297#_asinfo my VPS provider in .au is in this AS [15:18] ok [15:18] i emailed them [15:18] wonder if they'll say anything [15:19] static: i know someone who has vocus transit in NZ [15:19] it seems very on/off performance wise [15:19] i think a lot of their problems come from using he.net [15:19] I've only had it for a short time, but I've not noticed any significant issues yet [15:19] but even running two downloads one after the other of a file can be a 10x speed diff [15:20] i think it was like 4 or 5mb/sec to arp [15:20] and like 200 to 2000k/sec to europe [15:21] I have it linked over v6 so if Vocus dies for whatever reason, its gone [15:21] hmm it's only 13.5 megabit/sec to arp atm [15:22] vocus seem to be growing quick [15:22] will be interesting to see if they try and be bottom of the barrrel, or offer a higher level of service [15:22] i think AU market is like NZ market where people are cheap [15:23] oh up_the_irons are NTT doing you ipv6 too? [15:25] wow [15:25] ipv6 is way faster to arp with vocus [15:25] err from arp [15:25] it's not faster to arp [15:26] oh it is faster to too [15:26] bizzare, seems to be performance issue with vocus to arp atm even with peering [15:27] the diff in performance between ipv6 and ipv6 can be bizzare [15:27] like even within one provider [15:29] [ 3] 0.0-10.4 sec 9.25 MBytes 7.49 Mbits/sec [15:29] [ 3] 0.0-10.2 sec 22.9 MBytes 18.7 Mbits/sec [15:29] like that's ipv4 versus ipv6 [15:29] vocus for both ipv4 and ipv6 [15:30] and download was like 25 to 35 megabit [15:30] which don't make a lot of sense either [15:31] cos would be less change of congestion uploading to arp [15:33] farnsworth.entropynet.net, how does it look from .nz? [15:35] 113.9 msec [15:35] over ipv6 [15:35] 54 msec over ipv4 [15:35] via vocus [15:36] 48 msec via other provider [15:36] which is still via vocus [15:36] the vocus one is 6 msec away from auckland though [15:36] the v6 difference there is interesting [15:36] so yeh it's good via ipv4 [15:36] it's vocus all the way [15:37] 6. ten-0-2-0-300.bdr02.akl01.akl.VOCUS.net.au 0.0% 20 7.7 8.3 7.6 12.0 1.0 [15:37] 7. 2402:7800:110:1::a 0.0% 20 31.7 31.9 30.7 38.1 1.6 [15:37] 8. ten-0-3-0-911.bdr02.per02.wa.VOCUS.net.au 0.0% 20 79.0 80.0 78.9 83.5 1.2 [15:37] 9. 2402:7800:40:1::d 0.0% 20 80.7 79.9 79.0 81.7 0.7 [15:37] 10. 2402:7800:30:1::15 0.0% 20 112.3 112.8 112.3 115.2 0.7 [15:37] it has another vocus route at 11 with rdns [15:37] mercutio: yeah, ipv6 too. will finally have a non-HE ipv6 provider. [15:37] up_the_irons: yay :) [15:37] \m/ [15:37] i thought vocus dual homed all their routers on ipv4/ipv6 [15:38] so it's weird when performance is diff [15:38] actually the ipv4 route goes over ape which is auckland peering exchange [15:39] and ipv6 is going over transit [15:39] static: you could do a reverse trace to 2403:b700:1::6 [15:40] up_the_irons: provider or peer? [15:40] but i suspect the issue is routing via perth [15:40] jlgaddis: provider, full transit [15:40] nice [15:40] yep:) [15:41] and you're in adelaide? [15:41] correct [15:41] they should be able to reduce latency from adelaide to sydney [15:41] soon [15:42] looking on a map, it seems that it's liekly it goes via melbourne etc [15:42] WAIA is in Adelaide now and Colocity is provisioning [15:42] so your normal pings to sydney are 21 to 24 msec? [15:42] and auckland to sydney is about 24 [15:43] hmm by road it's 1410km [15:44] and distance between sydney/auckland is 2153km by bird. [15:44] so i imagine they can get latency down to 15/16 msec [15:45] i suppose that doesn't make a lot of diff [15:48] it doesn't [15:48] i always underestimate how big australia is [15:48] inter-Australia should be much better once the WAIA provisioning is done [15:48] heh [15:48] NZ to australia can suck at times [15:48] well it's fine to sydney generally [15:49] but once you hit domestic routing you can get stuff like adelaide going via perth [15:49] i heard that some isp's have really broken routing internally too [15:49] so the same happens to dsl users [15:49] oh yes [15:49] and like perth will go via sydney for internet [15:49] case in point - AT&T/Bellsouth DSL [15:49] in USA [15:49] and brisbane [15:49] and darwin [15:50] i also heard that a whole lot of users in australia can still only get adsl1 with 1.5 megabit/384 kbit or something [15:50] which uhh, means no high def youtube [15:50] terrible congestion etc [15:51] then there are transit providers like pipe networks with real high latency going via guam [15:51] http://pastie.org/pastes/6385091/text?key=knxpf1n79zzlljnhufpqnq its not too bad for me from home [15:51] i can't even find guam on the map atm [15:51] i see hawaii [15:51] isn't it near hawaii? [15:52] just a bit west [15:52] oh you have to zoom in [15:52] it's like between papa new guinea and japan [15:52] it's probably about as near to hawaii as LA is to NYC =) [15:52] oh you're not in australia static? [15:53] jlg: it's closer to hawaii than it is to japan to us [15:53] mercutio: no, I'm in Canada [15:53] err it's closer to US going via guam/hawaii than guam/japan [15:53] 268 is really high ping [15:53] that's what i get to UK [15:53] well yes [15:53] it's enough that it can be frustrating [15:54] you have 165 msec from san jose to australia [15:54] oh but it's mpls crap [15:54] its only doing ircd, I don't even need to touch it except for any security updates [15:54] with keeping the ping high from start point to end point [15:54] suppose that's actually pretty good [15:54] yeh [15:55] not touching much is fine for that ping [15:55] but if you used tmux/irc itd' be annoying [15:55] as long as it stays linked in a stable manner, I'm happy [15:55] heh [15:55] what irc network is that? [15:56] back years ago undernet used to have so many splits [15:56] http://www.entropynet.net/ my small little thing [15:56] now freenode is like undernet was [16:22] *** heavysixer has joined #arpnetworks [16:22] *** ChanServ sets mode: +o heavysixer [16:27] Please provide the PCCW Global Circuit ID or Service ID. [16:28] i told them not direct customer. [16:28] heh [16:52] err in my initial mail [16:52] it does seem to have calmed down now though [16:53] * up_the_irons goes and grabs In n' Out burgers [16:54] *** heavysixer has quit IRC (Quit: heavysixer) [16:57] burgers plural o.O [16:58] mm burgers [16:58] milki: well up_the_irons is married with kids... [16:59] o, so its not all just for him [16:59] maybe ;) [17:29] i miss in 'n out [17:31] * brycec doesn't... it's just 2miaway [17:33] up_the_irons: is that new router coming in with the NTT link? [18:14] mercutio: new router is up (just not doing anything). NTT link will go in it, yes. all VM hosts will then have cable to the new router. and finally, the ability to route in / out either one will be complete. [18:15] brycec: milki : ok, well, one burger (double double, no onions, protein style) [18:17] up_the_irons: ahh sweet, was wondering how that was going [18:17] yeah, slowly but surely.. :) [18:18] biggest part was getting the new upstream (research, negotiations, and finally signing) [18:26] yeh i thought was two seperate things though :) [18:26] but i suppose tying together isn't stupid [18:26] how's fallback going to work? vrrp? [18:48] *** heavysixer has joined #arpnetworks [18:48] *** ChanServ sets mode: +o heavysixer [18:50] mercutio: it will be a manual failover. shutting down vlan on one router and up'ing it on the other would effectively 'switch' a customer from one router to another. up'ing the vlan sends a gratuituous ARP, so the VMs would automatically pick up the new gateway MAC address. [18:52] mercutio: no automatic fallback method was viable in my testing. they all caused more problems than they cured. failover should be an extremely rare situation, if it even occurs at all (we've already gone 4+ years on the first router) [18:53] ok [18:54] yeh i had a quick think through n my head and i couldn't think of any easy ways to do it [18:54] best answer in my head i came up with was two /31s instead of /30 [18:54] but that wouldn't just work either [18:55] and if you have two /31s on private addresses going to internet /32 [18:56] then that's a bit of a pita too, and there'd be shared link state [18:56] mercutio: you can't do vrrp with /30's without hoops that make it not worth it; i wouldn't be surprised if those hoops would crash Cisco IOS [18:57] mercutio: those who have /29's however, perhaps i could offer vrrp as a service, who knows... like, elective auto failover [19:02] up_the_irons: ok [19:02] yeh, well i'd rather just two bgp sessions myself... [19:03] but i realise that you're not going to have bgp sessions to everyone. [19:03] right [19:04] and you can't just go using /24s or such, because of everyone having their own vlan [19:04] which means that they wouldn't be able to reach people on the same /24 easily [19:05] yeah i can't stick entire /24's to one or the other [19:05] well, i could, but why.. [19:05] the inbound traffic will come in through *both* routers [19:05] and ibgp (or perhaps ospf) will decide where to take it from there [19:12] i mean /24 with vrrp etc [19:13] 3 ip addresses [19:13] you'd need bridging between vlans or such though [19:13] so then it gets more complicated [19:13] and it may remove some peoples assumptions about how the network works [19:14] ie would be change in design [19:14] cos i assume volume comes from vlan rather than from whether it's routed/bridged [19:14] yeah vrrp was not designed for that [19:14] well that'd just be to enable access within the /24 [19:14] you'd also need ip/mac enforcement [19:15] so yaeh more complicated [19:15] but could save ip's. [19:15] but yeah i understand completely reluctance to go down that path [19:15] oh, that, no.. i decided against that common design long long ago [19:15] one customer == one vlan, solves so many practical problems [19:15] i would never do it another way :) [19:15] yeh you can still do the one customer to one vlan [19:16] but you could bridge between vlans for the same subnet [19:16] if wanting to use /24s rather than /30s [19:16] so like customer 1, vlan 4321, customer 2 vlan 4432, internal vlan 105 [19:16] i think i just barfed a little in my mouth [19:16] oops customer 2 4322 [19:17] ;) [19:17] so vlan4321 can bridge to vlan4322 [19:17] ok [19:17] but what would bridge it? [19:17] but router is still on vlan105 [19:17] the switch [19:17] customer vlan's terminate directly on the core (one day, perhaps an edge) [19:18] hmm [19:18] i'm not sure where the "internal" vlan would live [19:18] ok the router then [19:18]