#arpnetworks 2014-06-07,Sat

mercutio

heh

m0unds

i think NTT is just screwy - this only happened in the last few days
i can see it from work (centurylink) as well

mercutio

uhh it was happening months ago

m0unds

i'm talking about the thing with comcast and me
not the thing with verizon

mercutio

m0unds

well, comcast and centurylink and me

mercutio

well whenever we talked about it last time with some of us, verizon and comast were both a bit screwy
and ntt had had some los angeles routing oddities on top of thagt
where ntt via san jose wasn't as bad as ntt via los angeles
or osmething ilke that

m0unds

yeah, but i wasn't seeing 20+% packet loss and 200% higher latency

mercutio

oh
but there was 5% loss or something
based on time of day

m0unds

yeah

mercutio

reoccuring

m0unds

hadn't seen that in months via comcast here though

mercutio

oh ok
there was 20 msec higher latency before or something i think?

m0unds

with the exception of that weird thing late last week, when kvr26 was experiencing tons of pkt loss

mercutio

i don't actually even remember what month this was :)
yeh that wasn't ntt's fault

m0unds

well, it's weird - one of my addresses ordinarily pings ~30ms and is routed via LAX directly off comcast to arp
but other ones are routed to SJC via denver and ping at 50ms or so

[06:44]

mercutio

have you tried iperf?

[06:44]

m0unds

nah
throughput is fine - i tunnel ipv6 traffic through this vm

[06:44]

mercutio

ahh ok
you can figure out direction of loss

[06:45]

m0unds

i'm reasonably sure it's NTT, because my CL route is NTT bidirectionally
but comcast is only NTT outbound from the vm
and the CL one has the same hop both directions

[06:45]

mercutio

yeah i think it's outbound via ntt that is the issue too

[06:46]

m0unds

ae-3.r05.lsanca03.us.bb.gin.ntt.net <- that one

[06:46]

mercutio

bnut that's from last time rather than testing

[06:47]

m0unds

right

[06:47]

mercutio

bah i'll see if i have any affected hosts :/
i think i didn't
oh
it's too late at night now
isn't it

[06:47]

m0unds

yeah
http://i.imgur.com/zPKFrvm.png
that's on one of the low-latency IP addresses via LAX

mercutio

so what 4 hours too late?

m0unds

mercutio

m0unds

stops at around midnight gmt -0700

[06:49]

mercutio

it's nearly 2 am here
i should be in bed heh
but yeah means it finishes about 7 pm here
which is about when it's easy to do testing hah

[06:49]

m0unds

haha

[06:49]

mercutio

oh it's only 5% loss
it was way worse than that what i saw?

m0unds

yeah, i think the vz one is much worse

mercutio

ahh

m0unds

speed congestion and latency are what i'm seeing here
just coincides with the vz thing, just less severe

mercutio

yeah it's complicated to determine impact

m0unds

usually the graph is nice and flat 24/7

mercutio

i normalyl reckon 0.5% to 5% loss is degraded
and over 5% is severely degraded

m0unds

29ms +/- .3ms right now

mercutio

and 20%+ loss is unusable.
but i thikn loss should be below 0.05%

http://internetpulse.net/

m0unds

and ordinarily it is, there's just some wonky stuff somewhere

[06:52]

mercutio

for transit links etc
well i imagine comcast may be resolvable
ie they may actually take notice if you email them
whereas i think verizon is long standing dispute territory?

[06:52]

m0unds

yeah, they're terrible
i've got a buddy who's an engineer w/comcast, but he's out of town. i figured if it's persisting when he gets back, i'll drop him a line

[06:53]

mercutio

i doubt much will happen in a timely manner
but i've been wrong in the past
oh hey i'm seeing latency as well
http://202.49.71.24:24/smokeping/smokeping.fcgi?displaymode=n;start=2014-06-06%2019:53;end=now;target=other.comcastnet
from nz, via verizon to comcast
it's probably ntt return path
it originates from 202.49.71.24
can you traceroute from comcast back to that ip?
uhh it looks like level3
oh it's ntt forward path
ok it verizon forward path again
will see what it liketommorow

[06:56]

............... (idle for 1h14mn)

brycec

lol professional pingers....

[08:16]

.... (idle for 16mn)

m0unds

yeah, we're pro-pingers

[08:32]

.......... (idle for 46mn)

jpalmer

up_the_irons: I realized yesterday that my card on file, had expired and I was two months behind. Thanks for not killing my account! I've updated the card info, and you can run it whenever you'd like.

[09:18]

brycec

Wouldn't up_the_irons have emailed on payment failure?
I agree - thanks for not killing the account - but I think there should've been a warning/notification

[09:29]

jpalmer

brycec: I get a ton of emails, I probably just overlooked one, if it came in.

[09:31]

brycec

[09:36]

.......................................... (idle for 3h25mn)

acf_

ok, just sent NTT an email about Comcast

[13:01]

Alan,
We are currently seeing no issues on the NTT network. Please contact
your network provider to have the issue investigated further.
lol

[13:14]

m0unds

haha, i guess give it 2 hours

[13:15]

acf_

yeah, if you look at my Verizon email
it went the same way
I have a modified version of the reply: http://paste.unixcube.org/k/eec77a
with s/Verizon/Comcast/

[13:20]

m0unds

hahaha, nice

[13:21]

...... (idle for 25mn)

***

toddf has quit IRC (Ping timeout: 245 seconds)

[13:46]

........................ (idle for 1h56mn)

pseudorandom has joined #arpnetworks

[15:42]

pseudodoge has joined #arpnetworks
pseudorandom has quit IRC (Ping timeout: 240 seconds)

[15:47]

pseudodoge has quit IRC (Quit: Leaving)

[15:55]

........ (idle for 38mn)

m0unds

haha, here it comes

[16:33]

anisfarhana

Love the arpnetworksssss!!!!!
Running psybnc on arpnetworks box, idle time is almost 800 minutes now! :D
Very stable
Thanks arpnetworks!

[16:40]

..................... (idle for 1h41mn)

mercutio

anis: that's not very high :/
heh
acf: i wonder where people are meant to cpomlain about these things
there's that outages mailing list but they probably won't notice
there's nanong :/
but its' not really relevant

[18:22]

m0unds

i added that particularly flaky hop to my monitoring

[18:24]

mercutio

could email nanong where to discuss such things?
i'd love to see some kind of proper reporting of interconnects
the only one that's around doesn't actually seem very good

[18:24]

m0unds

stupid wind. ugh.

[18:25]

mercutio

[18:25]

m0unds

@wx
!wx

[18:25]

brycec

?
m0unds: You looking for @weather ?

[18:26]

m0unds

wasn't sure what the trigger was, figured it'd be one of those two
haha

[18:26]

brycec

wx is kinda uncommon... weather makes way more sense :p

[18:27]

m0unds

moar chars
@weather kaeg

[18:27]

BryceBot

There are 3 weather alerts in effect for your area! There is a Tornado Watch. There is a Areal Flood Advisory. There is a Wind Advisory.
Double Eagle Ii, NM: Partly Cloudy ☁ 79°F (26°C), Humidity: 19%, Wind: From the West at 14 MPH Gusting to 12 MPH -- For more details including the forecast and almanac, see http://www.wunderground.com/cgi-bin/findweather/getForecast?query=, or re-request this with: @weather -v kaeg

[18:27]

staticsafe

tornado watch o_o

m0unds

yep, north of me
@weather 87113

BryceBot

There are 3 weather alerts in effect for your area! There is a Tornado Watch. There is a Areal Flood Advisory. There is a Wind Advisory.
Albuquerque, NM: Partly Cloudy ☁ 86°F (30°C), Humidity: 23%, Wind: From the SE at 9.4 MPH -- For more details including the forecast and almanac, see http://www.wunderground.com/cgi-bin/findweather/getForecast?query=35.169689,-106.567902 or re-request this with: @weather -v 87113

brycec

Nice weather m0unds

acf_

is that a unicode cloud? ☁

m0unds

wind seems off, it's shaking light poles and making the building creak

mercutio

erk
that doesn't sound good :)

brycec

Yes acf_

mercutio

does that include gust speeds for wind?

m0unds

yeah, tornado thing is north of here - they just implicated the county albuquerque is in for whatever reason

mercutio

tonado watch doesn't mean there's tornado

m0unds

nope, but conditions are favorable

mercutio

winds do move :)

m0unds

they had golfball sized hail up in santa fe a bit ago (where the tornado watch is)

mercutio

ouch
lots of broken windows etc?

[18:30]

m0unds

no idea, someone on failbuque just said there was a lot of hail

[18:30]

mercutio

@weather auckland

[18:36]

BryceBot

Auckland, New Zealand: Mostly Cloudy ☁ 67°F (19°C), Humidity: 91%, Wind: From the NE at 4.5 MPH Gusting to 12.1 MPH -- For more details including the forecast and almanac, see http://www.wunderground.com/cgi-bin/findweather/getForecast?query=-36.973896,174.878021 or re-request this with: @weather -v auckland

[18:36]

m0unds

67 sounds nice

brycec

@wx 99019

BryceBot

Liberty Lake, WA: Scattered Clouds 74°F (23°C), Humidity: 27%, Wind: Calm -- For more details including the forecast and almanac, see http://www.wunderground.com/cgi-bin/findweather/getForecast?query=47.666508,-117.100792 or re-request this with: @wx -v 99019

m0unds

bah

mercutio

what is wind calm
haha
does that mean below 5mph or something?
so how does one get oen of these personal meter thingys

[18:39]

brycec

@wiki Beaufort scale

[18:40]

BryceBot

Beaufort scale :: The Beaufort scale /ˈboʊfərt/ is an empirical measure that relates wind speed to observed conditions at sea or on land. Its full name is the Beaufort wind force scale, although it is a measure of wind speed and not of force in the scientific sense. History The scale was devised in 1805 by Francis Beaufort (later Rear Admiral Sir Francis Beaufort), an Irish... http://en.wikipedia.org/wiki/Beaufort%20scale

[18:40]

brycec

mercutio: ^

[18:40]

m0unds

weather underground had a thing outlining hardware requirements for PWS data feeding
with some suggestions for vendors in the us and whatnot

mercutio

oh it's less than 1km/h

m0unds

there's one like 500ft from my house

mercutio

eerie still
m0unds: there's quite a few around it seems
so i imagine it's not extermely expensive

http://www.wunderground.com/personal-weather-station/dashboard?ID=KNMALBUQ167#history

m0unds

[18:42]

brycec

http://www.wunderground.com/weatherstation/about.asp

[18:42]

m0unds

that's the one right near my house
yea, my tunnel is shit slow right now so i didn't dare to try and dig around the site
haha

[18:42]

mercutio

you need to use windows?

brycec

No?

mercutio

WeatherLink suonded windowsy

brycec

It's self-contained, just plug in Ethernet
"The RainWise RapidFire(tm) enabled weather station doesn't need a PC to upload to us."
"WeatherBridge is an IP WiFi appliance that connects to weather stations and updates to Weather Underground real-time, providing up to the second data without the use of a PC."
etc
all non-PC

mercutio

$1271
that's expesnive :/

[18:44]

brycec

Starting at $250 tho

mercutio

you also need to wire it up etc

brycec

some of them http://www.ambientweather.com/amws1000wifi.html

mercutio

the $250 are the windows ones?

brycec

No?
http://rainwise.com/wunderground/ $139

mercutio

oh so they are all self contained except the usb ones?

brycec

(And still no PC required)
What USB ones?

http://www.davisnet.com/weather/products/weather_product.asp?pnum=06510USB

mercutio

brycec

Oh you clicked on the top of the line one :p

mercutio

i googled?
you save $139
it's $1350 :/

[18:47]

m0unds

i can do too many fun things with $1350 to tie it up in weather stations
unless i had interesting weather or something
but it's sunny something like 300 days a year in ABQ

[18:47]

mercutio

exactly
haha
but it's windy

m0unds

wind just means it's either dusty or cold usually
(right now it's dusty)

mercutio

http://www.ambientweather.com/amws1000wifi.html
so this one is good?

brycec

m0unds: If you ran an airport, for instance, you'd care about weather details such as the $$ will get you

mercutio

airports have weather stations :/
that upload to these weather thingys

[18:49]

brycec

mercutio: Yes, they buy them from somewhere...

[18:49]

m0unds

if i ran an airport, it'd matter

brycec

(small airports, air strips, etc)

m0unds

i live near a small civil aviation airport
DOUBLE EAGLE Ii (the second i is lowercase for some reason)

brycec

Yeah m0unds but the idea seemed to go over mercutio's head

m0unds

lol

[18:51]

brycec

And mercutio I have no idea if it's "good" - I only know that it was linked from the wunderground URL I provided.

[18:51]

mercutio

it didn't go over my head
i just saw there were like 20 personal stations in my city
and figured that lots of them wouldn't be spending lots of money
there are two airport official ones in the city too

[18:51]

brycec

Airport weather rarely matches the weather a few miles/km away

[18:53]

mercutio

oh?
well lots of sample sets give better ideas anyway
this city doesnt' have the same waether all over :/

[18:53]

brycec

Either on account of the airport's distance away from the city, or the microclimate caused by jet engines.
(The airport here is atop a tall hill, 30 minutes away)
(for example)

[18:53]

mercutio

kind of annoying when driving, because you have to turn wipers on/off/on/off

brycec

brycec has automatic wipers :D

mercutio

that sounds more annoying :)

brycec

I don't see how

mercutio

i find automatic cars annoying too :/
it'sk ind of disturbing when somethnng else is decding when to do stuff

[18:55]

brycec

Starts raining, wipers start. Rains harder, wipers speed up to match.

[18:55]

m0unds

my truck does that and the lights turn on when it gets dark too
MAGIC

[18:55]

brycec

(me too)

[18:56]

m0unds

magic!

[18:56]

brycec

lol

[18:56]

mercutio

heh

[18:59]

m0unds

i gotta go into work super early on wednesday
have to leave at 0300. bleck.

[18:59]

brycec

And honestly, the magic wipers have come in handy in some very inclement weather; I could keep my hands on the wheel at all times (necessary) and not worry about tweaking the speed.

[19:00]

m0unds

SHAZAAM

[19:02]

z310

ouch, m0unds
What time are you going to go to bed?

[19:08]

m0unds

probably 2100 or so
get to be half asleep while i take down our recording and operator environment so a vendor can fix UPS'

[19:08]

z310

and you'll wake up, i imagine, at 0200?

[19:08]

m0unds

yeah, something like that probably
haha

[19:09]

z310

>_<
I haven't done anything like that since college

[19:09]

m0unds

pair of 80kVA eaton powerware units that our facility engineer decided didn't need maintenance bypasses :)

[19:09]

z310

oh dear

[19:10]

m0unds

oh cool. the air pressure differential caused the roof covering over the convention center at work to balloon up, tear, then flop down like a deflated balloon

[19:19]

gizmoguy

m0unds: you guys probably saved a couple hundred bucks, you should say congrats to your power engineer :P

[19:20]

m0unds

gizmoguy: ikr
it was a $381 option for the two UPS' i specced as replacements late last year (they won't give me the money)
EVERYTHING MUST STAY UP. wait. you want money? whoa, whoa whoa. watch it.

[19:20]

gizmoguy

why are you guys even buying UPSs
they are pretty expensive

m0unds

hahaha

gizmoguy

and the maintainence of replacing batteries etc

m0unds

business continuity, brah

gizmoguy

is the rest of the casino on generators?

m0unds

that's the funny part
i think i told you about that whole lightning/flooding mess we had in july, right?

gizmoguy

yeah

m0unds

we have N+1 gensets for the facility, total generating capacity is like 7MW
the ones (two of them) that backed our gear had no maintenance records, dead batteries, dead battery chargers, so they failed to start when we lost utility power due to the transformer exploding

gizmoguy

rofl

[19:24]

m0unds

so they decided that the way to fix it was to throw more money at new UPS subsystems to keep us up longer if the gens fail to fire up

[19:24]

gizmoguy

see, way better off if you didn't have them :)
why don't you buy laptops for servers

[19:24]

m0unds

lol
YUS

[19:25]

gizmoguy

I hear they have built in UPSs

[19:25]

m0unds

so, the old men with the big checkbooks wanted us to get new, better UPS' (which i would like, because the ones we have are 13 years old)
they have battery maintenance and stuff, but they're old and don't have maintenance bypasses >:|
so they asked me for a ballpark price range for what we wanted, so i specced 4 80kVA 9390s w/vycon flywheel frontends as an option to reduce battery wear and extended battery cabinets rated for 15 mins at full load (because that's the magic number they had in their minds)
it was gonna be something like $385k installed, with transfer switches and everything
but that was too much
but they just approved $280k for us to wire up a temporary parking lot for camera coverage and stuff while a parking garage is built :)

[19:25]

gizmoguy

buy second hand
ebay probably has a lot of UPSs

[19:29]

m0unds

i'd rather just buy a conex container full of 1000VA rackmount ups' and dump them in the fountain
the flywheels are neat though
http://vyconenergy.com/pq/pages_pq/pqprod.htm

[19:30]

.... (idle for 19mn)

***

toddf has joined #arpnetworks
ChanServ sets mode: +o toddf

[19:50]

.... (idle for 18mn)

m0unds

well, the packet loss + latency thing is back in full force again. seeing a 30ms differential between ae10-50g.cr1.lax2.us.nlayer.net and ge0-arpnet.cust.lax07.mzima.net

mercutio

gizmo

m0unds

and 10-15% pkt loss
haha

acf_

ok sending mail

mercutio

you know that some people do actually advocate buying laptops for servers
worst idea ever :/

m0unds

mercutio: you're too late, i already bought 60 at his advice
they'll be here friday
s/at/on

BryceBot

<m0unds> mercutio: you're too lone, i already bought 60 on his advice

m0unds

hahahah
lone
m0unds gives up

mercutio

oh i tried smokeping last night
i should check it

acf_

yep :/
http://kremvax.acfsys.net/smokeping.cgi?target=Local.unixcube

mercutio

oh that was to comcast

acf_

yeah

mercutio

and yeah via verizon to comcast no loss
like /no/ loss
today, and loss yesterday

m0unds

acf_: i've got a 30ms differential between ntt and comcast, and a 30ms differential between arp and nlayer

mercutio

ntt vs verizon forward route
level3 return route
so yeah it's ntt's forward path that sucks most likely
arp and nlayer could be about return path too

m0unds

just hard to diagnose
haha

[20:11]

acf_

if you look at the mtr for ARP -> Comcast

m0unds

i'm doing both now

acf_

the packet loss/latency clearly starts right between NTT and Comcast

m0unds

yeah, there's a hop w/no rdns

acf_

wow that's some sucky packet loss
the whois says Comcast

m0unds

yup

mercutio

well ntt aren't going to help
it soundedl ike :/

acf_

I know. I just want to see what they say
if they'll tell me if it's peering again this time

mercutio

need lots of people cmoplaining really
complaining

acf_

not even sure if that will help

mercutio

yeah might not

acf_

I'm sure they *want* to fix it

mercutio

it probably helps if you pay them money

acf_

just Comcast, etc... is being unreasonable / they're being unreasonable

gizmoguy

mercutio: :D

mercutio

did you try complaining to comcast as well?

acf_

no. I just figure I'll get the Verizon-type response

gizmoguy

I like to walk the line of ironic

http://pastebin.com/xrywYeLq

mercutio

well anyway i'm vonvinced that if the outbound path gets fixed it'll be a lot better
based on changing outbound path on my nz host: http://202.49.71.24:24/smokeping/smokeping.fcgi?target=other.comcastnet

[20:15]

m0unds

[20:15]

mercutio

and it going to 0 loss
when it didn't go via ntt
it had like 4 bad days
and some in between stuff before that

[20:15]

acf_

http://paste.unixcube.org/k/91504f
with visible loss

[20:16]

mercutio

that nlayer to mzima hop is return path most likely
nlayer and mzima are the same company

[20:16]

m0unds

one side is arp's, if i was to bet i'd say lax07 = arp switch 07
s7.lax.arpnetworks.com

[20:17]

mercutio

that's the new server i think
and s1 is the old
s/server/router/

[20:17]

BryceBot

<mercutio> that's the new router i think

[20:17]

acf_

I would think that lax07 would be controlled by nlayer/mzima

m0unds

just guessing that particular hop is probably the mzima port

acf_

oh ok

m0unds

could be wrong

acf_

yeah that's almost definitely s7.lax.arpnetworks.com
it would definitely explain the packet loss
if the return path was via ntt

http://www.comcast.com/peering/

m0unds

and just because it's so odd, http://pastebin.com/xABATfn9
that is on the same VM, just a different IP addr
but the path to it is completely different
haha
abq -> denver -> sjc -> lax -> arp
vs abq -> lax -> arp

[20:21]

acf_

hmm strange
Comcast's paths seem to change a lot
at one point I was going through Chicago

[20:21]

mercutio

well basically up_the_irons can complain to ntt himself, or can maek exception to not wanting to route around issues.

[20:22]

acf_

but I'm in CA
I don't think NTT can do anything even if up_the_irons complains

[20:22]

m0unds

yeah, it seems like that particular path bw comcast and ntt is hot
just oversubscribed or something

[20:22]

mercutio

it's been hot in the past too
it looked like i started monitoring it in march, so i imagine it's march when we discussed last

[20:23]

m0unds

at least it's not tata
tata is comcast's cesspool for traffic

[20:23]

acf_

arp traffic used to go via NTT <-> TATA <-> Comcast
I think

[20:23]

m0unds

it used to be nlayer both directions for me

[20:24]

mercutio

i wonder if routes are being advertised to level3 yet

[20:24]

acf_

level3 looking glass traceroute says no

[20:25]

mercutio

i found a comcast looking glass last night
but it was in canada

[20:25]

acf_

oh cool
at least it's on their asn

[20:25]

mercutio

i didn't even know that comcast was in canada

[20:26]

acf_

I didn't either
Verizon hasn't responded to my request for looking glass :(

[20:26]

m0unds

[20:28]

acf_

Dear Verizon Enterprise Solutions Customer,
We have received your correspondence and forwarded it to the appropriate personnel for processing.
that's cool

[20:28]

m0unds

oh, here you go
telnet://route-server.newyork.ny.ibone.comcast.net
^ looking glass
haha
oh boo
authenticated
haha
actually, rviews is the username, no pw
http://pastebin.com/H0K9vMDw

[20:31]

acf_

only route is via NTT?
I don't know how to read those...

[20:37]

m0unds

4436 is nlayer

[20:41]

acf_

oh ok I didn't look up the AS
so, only route out is via nlayer
does that mean NTT peers only one way?
that doesn't make sense

[20:42]

m0unds

well, from comcast's perspective, nlayer is the preferred route to arp
which we can see when we traceroute from comcast to an arp address

[20:43]

acf_

ok, right

[20:44]

m0unds

haha, nice, ntt is using juniper gear

[20:46]

gizmoguy

yeah, they use some fun load balancers too
ping ntt.net and see if you notice anything strange

[20:47]

m0unds

just saw the junos syntax in their lg when i had it show me bgp
haha, funny
dupe responses

[20:47]

gizmoguy

yup

[20:48]

m0unds

i get why they'd only return terse output, but i think that also makes them jerks

[20:49]

acf_

so, Verizon and Comcast are doing the paid peering thing
and NTT is refusing to cooperate?

[20:50]

m0unds

comcast makes lots of stipulations for settlement-free peering - you have to have a big network with reasonable route announcements, you have to receive as much as you send, etc

[20:52]

...... (idle for 26mn)

mercutio

ntt are probably in that situation
ntt is tier 1 i think
they're not meant to pay

[21:18]

acf_

yeah

[21:18]

mercutio

think is cogent is huge
and is tier 2

acf_

level3 payed up I think

mercutio

it gets messy

acf_

that cogent HE ipv6 peering thing is still a problem

mercutio

it doesn't erally matter if it's paid or free peering
it's whether links are congested or not

acf_

it seems to them it matters :)

mercutio

from an operational rather than political pov

acf_

yeah

mercutio

i am actually pro paid peering
i just think it should be at reasonable cost
as whenever things are free people don't take them as seriously

acf_

it seems like the idea of settlement-free peering requires that the traffic ratios be roughly equal
because both sides would be paying each other the same amount

[21:21]

mercutio

well it depends
i mean for big transit providers they should
normally sender pays

[21:22]

acf_

right

mercutio

so if you transmit 200 megabit/sec and receive 100 megabit/sec
how much shuold you pay?

acf_

50% of your transmit volume?

mercutio

and say you want to be able to send 2000 megabit/sec
and so you want to have aggregated ethernet or 10 gigabit
how much shoudl you pay
i imagine paid peering for 2 gigabit on 10 gigabit is around $1000/month or more
but i could be wrong

acf_

that doesn't seem unreasonable
for NTT who has big bucks to spend on that sort of thing

[21:25]

mercutio

but if it is that kind of cost
than people shoudl be able to justify it
unless people want to cost it out at ilke $10,000/month
but
if you're only transmitting 200 megabit normally it's hard to justify
so say instead it's 200 megabit average, but 800 megabit peak
during the evenings
on a gigabit link
as soon as any link fails, that traffic would instead go over that link
you're screwwed
but the cost goes up generally going above 1gigabit
leaving either upgrading to 10 gigabit, or doubling up gigabit link
s
or rerouting traffic
but if the other link was only peaking at 300 megabit in evings
it might have been culled
because it wasn't worthwhile
but of course that puts more pressure on the first link
of course don't know all the details, but it's a complicated problem

[21:25]

acf_

just it seems like network quality would be more important than a few dollars to NTT and the like

[21:28]

mercutio

and the simplest solution is to way overprovision
so if your peak traffic is over 500 megabit

[21:28]

acf_

and most effective I would imagine

[21:29]

mercutio

then you should have another link
if you have two gigabit links, that peak at over 1 gigabit combined, you shoudl have a third link etc
but basically the way internet changes are moving towards is make it cheaper to go for 10 gigabit
s/changes/exchanges/

[21:29]

BryceBot

<mercutio> but basically the way internet exchanges are moving towards is make it cheaper to go for 10 gigabit

acf_

do the Tier 1 providers still use gigabit links at all?

mercutio

because that also reduces impact of ddos attacks.

acf_

for interconnect

mercutio

if it says ge in a traceroute it's one gigabit
if it says xe it's 10 gigabit

acf_

like is NTT <-> Verizon/Comcast really gigabit?

mercutio

in juniper speaker
it's 10 gigabit i think
te is cisco 10 gigabit

acf_

my mtrs all say ae
which is aggregated

mercutio

ae is hard to know
it's juniper aggregated ethernet
yeh
it could be either

acf_

more likely gigabit though?

mercutio

well it means it's at least 2 gigabit or at least 20 gigabit
hard to know

acf_

would they have more than 10 gigabit?

mercutio

could do
40 gigabit is common now

acf_

wow

mercutio

what is he
comcast are using he and te
i imagine he is moer than 10 gigabit

[21:34]

acf_

I think it's old 10 gigabit interfaces

[21:34]

mercutio

http://t1rex.blogspot.co.nz/2013/01/comcasts-100-gbps-core-network-supports.html
so comcast is shifting from 40 to 100 gigabit
with multiple links probably

[21:34]

acf_

what
he- 100 Gigabit Ethernet (Cisco)

[21:35]

mercutio

hmm i think denver can go la->phoenix>denver
or la->sj>denver
yeah anyway, ntt talk to comcast at 10 gigabit
or come into their netowrk at 10 gigbit
it's hard to know

[21:35]

acf_

so Verizon still likely uses SONET/OC-192 for their backbone?

[21:36]

mercutio

arp talks to ntt at gigabit

[21:36]

acf_

yeah, I see that in the mtr

[21:37]

mercutio

verizon do internationally i'm pretty sure
i imagine they have lots of legacy gear
and it doesn't make sense to update it

[21:37]

acf_

why would it make sense for Comcast and not Verizon?

[21:37]

mercutio

because comcast have national gear not international?
their network is newer/less established

acf_

I guess

mercutio

and their users mostly talk to netflix etc :/
verizon also have slas
with regards to latency, packet loss etc.
which sonet makes it easier to maange

acf_

but can they really stay with 10 gigabit forever?

mercutio

10 gigabit is heaps

[21:39]

acf_

I suppose
but as you said, other networks are going 40/100 gigabit

[21:39]

mercutio

just becasue it's 10 gigabit doesn't mean it's only one 10 gigabit
and 10 gigabit interconnects between cities are probably normally fine
comcast is the biggest provider int eh US afaik
but even in NZ there's one provider doing 100 gigabit

[21:39]

acf_

because they don't suck nearly as bad as Verizon

mercutio

the thing is how much is 100 gigabit

acf_

is it the same there?

mercutio

like comcast seem to have 100 gigabit between san jose and denver

acf_

the dominate provider is dominate because their service doesnt suck?

mercutio

i imagine most cities are 10 gigabit though
we have this regulated deregulation system
uhh
the country sold the telephone network
then they imposed limits of them
then the telephone company split in to two companies
and one of them is wholesale and one is consumer
and the wholesale seels to the consumer
within the two split companies
and then the government is paying money to different companies to build their own fibre networks
which impact the only cable company here
the cable company bought out the third biggest provider or something

[21:41]

m0unds

can't copy / paste because im on a tablet, but comcast's own ibone backbone is still relatively new, so the comparison of vz w legacy gear vs comcast's new network is an apt one

[21:43]

mercutio

so now there are two major companies with cellphone/broadband.

[21:43]

m0unds

they used to rely on at&t for coast to coast stuff

[21:44]

mercutio

but there are lots of smaller companies too
yaeh i imagine verizon has a lot of legacy cruft
didn't uunet go bust?
and was bailed out by government?

[21:44]

acf_

they are verizon
now

mercutio

then acquired by verizon?

acf_

yep
so-2-1-0-0.DFW03-CORE-RTR1.veriz
mtr verizon.net

mercutio

companies that go bust often have more legavy stuff prior :)
as there's less desire to invest in modernising
and companies that are bought
well people don't understand :)
lots of staff leave etc
management dilemas etc
it's really common to have billnig issues when companies are bought out
and billing systems always take ages to integrated
-d

acf_

verizon seems like a giant stew of bought up telecoms
that has no desire to integrate its components

mercutio

haha
joy

acf_

or do anything other than mainain the status quo

mercutio

is cox good?

acf_

and get out of the landline and dsl market
I've heard it is

mercutio

it's just not anywhere that comcast is?

acf_

it doesn't tend to have the peering congestion we see on Comcast and Verizon
right
it's a regional monopoly thing

mercutio

because it's like a drug cartel, and they can't compete.
did i show you guys the youtube video

acf_

don't think so

mercutio

https://t.co/wD3MiiXOSM

https://t.co/wD3MiiXOSM -> https://www.youtube.com/watch?v=fpbOEoRrHyU

BryceBot

mercutio

i thought it was humourous

brycec

@youtube fpbOEoRrHyU

BryceBot

<http://youtu.be/fpbOEoRrHyU> YouTube Entertainment: "Last Week Tonight with John Oliver (HBO): Net Neutrality" by LastWeekTonight (13m 18s), 2,856,851 views, 44,185 likes and 428 dislikes. Uploaded 2014-06-02T06:30:01.000Z.