#arpnetworks 2014-02-10,Mon

BryceBot

Ars Technica: "Netflix performance on Verizon and Comcast has been dropping for months"

mercutio

it's interesting that verizon and comcast were the two destinations ntt were haveing issues too
s/too/to/

BryceBot

<mercutio> it's interesting that verizon and comcast were the two destinations ntt were haveing issues to

Guest58998

Hey. I could use some help on an ipv6 /48 ubuntu configuration. No matter the search query in google I can't seem to find anyone that describes it the way arp networks does. Someone know how to set up the /48 on a single Ubuntu VPS?

[14:06]

toddf

at one point it was routed to your vps. at one point the lowest /64 was an ethernet segment and the rest was avilable on a support ticket request basis for routing. I'm not sure what the defaults are at this point. if you're a recent customer, just try setting the lowest /64 subnet on your ethernet segment and see how that goes .. try fe80::1 and <yourv6network>::1 for a default router, one of those should work. perhaps there's a ...
... wiki page I'm unaware of. hope that helps.

[14:09]

Guest58998

I'm told it has been routed to link-local and that I should set my side to fe80::2/64
I'm not exactly sure what they mean by "my side".. default gateway, local address or?

[14:10]

brycec

http://wiki.arpnetworks.com/wiki/48%20IPv6%20on%20OpenBSD is good reference
So "set your side to..." means set the IP on the interface to fe80::2
the default gateway is fe80::1
(because ARP's side is fe80::1 and routing the /48 to fe80::2)
(Also: Requisite "if you don't know how to do this stuff, then you probably shouldn't be messing with it.")

[14:11]

mercutio

you shouldn't need a /48

[14:13]

Guest58998

Oh I know it's expert only ;) But I have other servers where /48 is routed differently (I think)
but I do

[14:13]

mercutio

basically

[14:13]

Guest58998

but yeah I know that I shouldn't

[14:14]

brycec

ARP's method of routing is actually pretty common too, fwiw.
Though the majority of tutorials and howtos are written for people with HE tunnels and the like, so I can see how that drowns out the useful information.

[14:16]

Guest58998

I dont doubt it. It's just really hard to find it described that way anywhere else
yep, it's mostly two lines about native ipv6 and then 4 pages about tunnels

[14:17]

mercutio

native ipv6 is easy though

[14:18]

brycec

Yep.
And the /48 too once you realize it's two lines or so

[14:19]

Guest58998

I got some ipv6 connectivity now. Thanks a lot for your help

[14:20]

***

Guest58998 has quit IRC (Quit: Leaving)

[14:23]

toddf

arp's default config is great for a single vps. if you have multiple, you have to route v6 to the others from your first vps, or ask arpnetworks for changes. I opted for plan b *grin*, one /64 on the ethernet segment.

[14:26]

.... (idle for 19mn)

m0unds

yea, i wrote a post about configuring SRX devices with a roll-your-own ipv6 tunnel in flow mode because so many of the HE tunnel broker tutorials are silly and tell you to switch off flow mode on your appliance and stuff
hopefully it'll help someone sooner or later - same with working srcnat for xbox live, since it seems people way overthink that stuff

[14:45]

....... (idle for 30mn)

brycec

heh
m0unds: link?

[15:15]

....... (idle for 30mn)

up_the_irons

mercutio: not yet
toddf: for the record, our default is no routing at all, just /64 on your VLAN, so no single vps is a point of failure

[15:45]

***

up_the_irons has quit IRC (Read error: Operation timed out)

[15:47]

toddf

up_the_irons: ah. I've been around too long to know what the actual current default is, hope I made that clear above ;-)

[15:47]

***

up_the_irons has joined #arpnetworks
ChanServ sets mode: +o up_the_irons
mhoran has quit IRC (Ping timeout: 246 seconds)

[15:48]

brycec

lol
sine up_the_irons missed it: 15:46:39 <@toddf> up_the_irons: ah. I've been around too long to know what the actual current default is, hope I made that clear above ;-)
*since

[15:49]

up_the_irons

brycec: tnx!

[15:50]

brycec

If you had multiple VPS'en and a /48, I suppose you could always CARP them all, but routing would be annoying/tricky.

[15:51]

up_the_irons

yeah

[15:51]

brycec

I still haven't worked out a good way to give my CARP backup IPv6 access to an HE tunnel :/ Not without watching for the state change and scripting route changes anyways.
(It's also not high on the priority list)

[15:52]

toddf

brycec: convince he.net it needs to do ospf6 with you and have two tunnels one to each router?
not always doable because some people only have a single ip, carp can be done in this case, but v4 connectivity is always fun in the backup router instance

[16:03]

brycec

Two tunnels but same subnet?
Not to worry, both routers have public v4 IPs
plus one shared

[16:04]

toddf

you'd need two tunnels and ospf6 should handle routing of the same subnet yes

[16:04]

brycec

Well all that's left is to convince HE of anything, lol

[16:05]

toddf

(note I've never heard of anyone doing it, but if you want to avoid scripting and wish to do it up proper...)

[16:05]

brycec

Yeah that would be proper. But given how much I'm paying them... I don't expect them to do anything "for me"

[16:06]

toddf

you could of course get two vps'en from arpnetworks and do ospf6 across two gif tunnels to your home for full redundancy on your side ..
they do permit bgp6 over a tunnel for a fee, if I read their website properly

[16:06]

brycec

That sounds like fun :) And I'm still meaning to move my IPv6 tunnels to ARP. However lately, HE's reliability has been > ARP :(
toddf: Actually I can request a BGP tunnel for free
But first I'd need an ASN...

[16:06]

toddf

details

[16:08]

brycec

And only 7 POPs support it
(As in: not my closest POP)

[16:08]

........ (idle for 35mn)

***

KDE_Perry has quit IRC (Ping timeout: 246 seconds)
KDE_Perry has joined #arpnetworks

[16:44]

m0unds

brycec: http://chris.vanvoro.us/2013/12/26/fun-with-ipv6/

[16:46]

brycec

Thanks m0unds

[16:47]

m0unds

sure, it's not the best, but it's better than most of what i'd read, haha

[16:47]

staticsafe

that your site?

[16:47]

m0unds

yep, terribleness that it is
octopress + nginx
code repo on bitbucket

[16:49]

staticsafe

i was too lazy to octopress so i just went back to wordpress

[16:50]

m0unds

i started using nitrous.io as a quick IDE for posting
i'm hosting a friend's wordpress site - it's the only reason i still have mysql and php running on my vps

[16:50]

***

laotzi has quit IRC (Remote host closed the connection)

[16:59]

brycec

Man I have no idea why I thought this would be more difficult... Using my ARP VPS as a v6 tunnel endpoint accomplished! (Still need to setup routing and firewalling, but that's all)
thanks for the kick in the butt m0unds

[17:09]

m0unds

you betcha!
i sat on mine for 6 mos before i did it

[17:09]

brycec

I'm over 1yr now
i think

m0unds

then got bored at work and went 'meh' and just did it

mercutio

apparently cogent -> verizon is even more broken than normal

brycec

tl;dr just need matching gif/v4tunnel/etc sections on both ends, that's it

m0unds

the srx part was what i hung up on initially though because i was on junos 11.4, which doesn't support ipv6 in flow mode

mercutio

cogent issues have been going on for something like two years now?

m0unds

so when i updated to the final build for my srx (discontinued model) it fixed it

[17:11]

brycec

cool, congrats
(my oldest invoice seems to be Nov 2012. Over a year now, woo)

[17:11]

mercutio

brycec: yeh it pretty simple to tunnel ipv6
you may have to mss clamp if you're forwarding traffic though

[17:11]

brycec

Hardest part now is deciding on address allocations

mercutio

fac3 ?

brycec

I'll keep that in mind, thanks mercutio
lol

mercutio

i dunno :)
err face would work too

brycec

face:b00c is pretty well-known ;)

mercutio

1337 ?
yeh i know

brycec

Yeah there are a bunch of "clever" ones out there. I'm far more practical.... But I can't just start at "1"

mercutio

you only have 16 bits to play with

brycec

(0 is already in use)

m0unds

hahaha
i started at 2

brycec

16 bits?

***

laotzi has joined #arpnetworks

mercutio

48 to 64

m0unds

my clients at home are 4, iirc

brycec

Oh sure, duh

mercutio

bcec ?
removing r and y from your nick, that don't map to hex :)

[17:15]

brycec

ha
Probably gonna start at f00a

[17:15]

mercutio

it does kind of sound like "be sick" though
or f00f like the pentium bug?

[17:16]

m0unds

like "sick" as in, WAY SICK DOODZ

[17:16]

brycec

So help whichever net ends up on f00f ;)
brycec spirals into the IPv6 "OMG SO MANY ADDRESSES" oblivion

[17:16]

mercutio

1c12
( i see one too)

[17:17]

brycec

actually I should just migrate my current HE prefixes

[17:17]

mercutio

bryce: you only have 16 bits, it's not that many
you need the /64 for autoconfig

[17:18]

brycec

16 bits is till pretty big
(And I know I can't really sub-divide the /64)

[17:19]

mercutio

it never felt very big on pc's :)
dammn those 64k memory limits
it was a real pita
but yeah it's a lot better than like 1 or 8 or such

[17:19]

***

hazardous has joined #arpnetworks

[17:21]

laotzi has quit IRC (Quit: SIGQUIT)

[17:28]

up_the_irons

for those running their own ntp server
"1. If you run ntpd, upgrading to the latest version, which removes the "monlist" command that is used for these attacks; alternately, disabling the monitoring function by adding "disable
+monitor" to your /etc/ntp.conf file."
we're getting LOTS of notices for NTP-based UDP amplification attacks

[17:28]

brycec

up_the_irons: Any way to forward those notices to the responsible party?
*parties

[17:29]

up_the_irons

brycec: i am in the process of doing so, yes
a very big time suck
39 notices based on IP. gotta lookup the IP, get email of customer, then foward.
*forward

[17:29]

***

dne has quit IRC (Ping timeout: 264 seconds)
Spitfire has quit IRC (Ping timeout: 264 seconds)

[17:30]

brycec

Bummer

[17:30]

***

Yamazaki-kun has quit IRC (Ping timeout: 245 seconds)

[17:31]

up_the_irons

maybe i could write some filter..

[17:32]

brycec

(Oh good, I was already secure)

[17:32]

up_the_irons

procmail or something

[17:33]

***

Spitfire has joined #arpnetworks
dne has joined #arpnetworks
Yamazaki-kun has joined #arpnetworks

[17:35]

up_the_irons

actually, would anyone *else* like to write something? I'll pay (obviously). Basic flow would be: 1) I get an abuse complaint, 2) i forward to some special address, 3) something / script on that address looks up IP with regex, 4) IP returns an email address (with our REST API), 5) forward that email
or, pointers to how this would be done would help
i can try to code something up

[17:41]

***

laotzi has joined #arpnetworks
Yamazaki-kun has quit IRC (Ping timeout: 245 seconds)

[17:45]

brycec

Seems straight-forward enough

[17:46]

mercutio

up_the_irons: it very well could happen for dns too

up_the_irons

mercutio: dns?

mercutio

so maybe having an easy way to email ip's would be good
up_the_irons: the any thing, and open recursive are being hit on authorative and recursive a lot recently too

up_the_irons

mercutio: if you mean the amplification attacks, yes, very much so

http://www.amazon.com/tmux-Productive-Development-Brian-Hogan/dp/1934356964/ref=sr_1_1?ie=UTF8&qid=1392089683&sr=8-1&keywords=tmux

mercutio

up_the_irons: what about having a sepcial domain you email with users ip@blah.arpnetworks.com
or such
and then it emails the right person, and a sepcial mailbox to keep note
which would just mean cutting and pasting the ip
which isn't automated, but is simpler to test, ..

[17:49]

up_the_irons

mercutio: ah true
mercutio: i like it
I LIKE IT

[17:50]

mercutio

can you map from ip to user with a mysql query or such?

[17:51]

up_the_irons

more like a bit of ruby

[17:52]

brycec

4) IP returns an email address (with our REST API)"
(obviously not a public REST API ;) )

[17:52]

up_the_irons

obviously :)

[17:52]

mercutio

i wonder if for things like recursive dns there should be tests every now and then

[17:53]

***

Yamazaki-kun has joined #arpnetworks

[17:54]

mercutio

but with a little script magic if such a system was setup it'd be easy to email effected users
err affected?

[17:54]

brycec

[17:54]

***

mhoran has joined #arpnetworks
ChanServ sets mode: +o mhoran

[17:55]

mercutio

hmm as an addition could have some extra things to bounce to which would send automated message content that say how to fix open dns etc
or maybe just keep a list of the various things, and people can parse themselves.
for ntp i'm in favour of openntpd which doesn't listen by default

[18:01]

up_the_irons

affected

[18:03]

.... (idle for 18mn)

jpalmer

are there any web based test tools for the NTP or DNS amplification attacks yet?
(I don't know how to exploit it offhand, but would like to verify my DNS and NTP servers are ok.

[18:21]

mercutio

host -t any <your domain name>
the any thing is complicated
basically more providers need to do bcp38 to improve the situation
as the predominant issue is that it's valid to do an any request for a domain name.

[18:23]

jpalmer

yeah, that returns several records for all of my domains.

[18:25]

mercutio

see that's normal
now the problem is someone can spoof an address
so that your response goes to another address
esp if one has lots of entries
like say host -t any microsoft.com has quite a bit of data
it's only like 4x amplification normally with that htough
but still if it's 10 megabit of requests that makes 40 megabit of response

[18:27]

jpalmer

yep yep

[18:30]

mercutio

arp defaults to 5 megabit rate limit for udp, so you'll only be able to return 5 megabit
but that could impact other services..
generally speaking most people seem to be ignoring the amplification attack and suggesting that it's the people sending spoofed requests that are the problem

[18:30]

jpalmer

heh. it's not the misconfigured SMTP servers, it's the spammers!

[18:31]

mercutio

well udp will limit what response size normally
and tcp won't work
bcp38 means people can't spoof addresses as easily so it cna't work as easily
from memory comcast is the biggest provider with no protection

[18:32]

jpalmer

I'll have to read up on bcp38. not familiar with it.

[18:33]

mercutio

https://www.nanog.org/sites/default/files/mon_general_weber_defeat_23.pdf
it only really matters for providers
basically it means that you can't send packets with my source ip address
which arp do btw
but basically if the any requests aren't terribly long it's probably mostly ok

[18:33]

mnathani

Are there network anomalies at present? I am getting about 2% packetloss from Toronto

[18:45]

mercutio

via ntt?
ntt -> verizon still seems lossy
acf had a smokeping
uhh
acf's smokeping was really good in the middle of the night
and his comcast gets better earlier
acf: did you check out your smpkeping?

[18:46]

mnathani

via nlayer / mzima

[18:57]

mercutio

oh prob diff issue then
2% isn't so bad

[18:57]

mnathani

thats the forward path

[18:57]

mercutio

unless that's averaged over time
acf's issue was forward path from arp
err and not just arp
going via ntt in san jose was also broken

[18:58]

mnathani

reverse path is: trit > he

[18:58]

mercutio

i'd suspect that trit->he path
nlayer do heaps of icmp deprioritisation too
i'd do iperf in udp mode at low bandwidth
to check which direction
not that you can necessarily change anything

[18:58]

mnathani

hence is the nature of the Internet / Inter webs

[18:59]

mercutio

yeh
i feel better having more idea of where things are going wrong even if i can't change them :)

[18:59]

mnathani

It surprising it actually works at all

[18:59]

mercutio

try 20% packet loss
that is hell to use
one time i was playing dota and there was a ddos attack and had 50% packet loss
and the game was going terribly
so i cehcked with mtr etc
and then i thought it was doing well considering there was ilke 50% packet loss
with ssh if there's a bit of packet loss often typing another key can help things along
ilek if something's not appearing you can press backspace or something
but if it's completely broken often it's better not to touch anything at all
and have the connection time out / disconnect

[19:00]

***

DaCa has quit IRC (Ping timeout: 252 seconds)

[19:03]

mnathani

I always use tmux anyway so the session stays alive

[19:15]

mercutio

ahh yip
oh was that you that posted to outages@ acf? :)

[19:16]

mnathani

who/what is acf?

[19:21]

mercutio

the guy who brought up comcast/verizon packet loss before
or did i get it wrong?
http://kremvax.acfsys.net/smokeping.cgi?target=Remote.verizon-snloca
he linked that
well someone posted to outages@
who uses arp
https://puck.nether.net/pipermail/outages/2014-February/006596.html

[19:22]

mnathani

oh ok..

[19:29]

jpalmer

I keep trying tmux, then switching back to screen. hehe

[19:30]

mnathani

I like its default config, works out of the box
screen, I keep needing to paste in configs before I can use it
also the splitting of windows / panes is nice

[19:31]

jpalmer

I keep getting fumbled up by the default keybindings in tmux, so used to screen's
I need a basic "idiots guide to tmux" and just start using it with irssi. when I get more comfortable with it, then install it on all my machines with puppet.

[19:33]

mnathani

[19:35]

BryceBot

Amazon: "tmux: Productive Mouse-Free Development"

[19:35]

mnathani

keybinding should be pretty basic to reconfigure
although you might not want to for sake of running screen within a tmux session

[19:36]

mercutio

there are books on tmux?
wow

[19:43]

m0unds

open resolver project is handy for identifying open resolvers on a network
http://openresolverproject.org
re: the ntpd thing, the default config for freebsd was changed when that vuln was identified, and freebsd10 ships with the modified config by default

[19:44]

mercutio

http://www.itnews.com.au/News/372033,worlds-largest-ddos-strikes-us-europe.aspx
there was a ddos today
apparentyl 400 gigabit
oh and it was using ntp
did up_the_irons reports all come today?

[19:47]

m0unds

whoa
jeez, that's enormous

[19:49]

mercutio

so that could haev effected canda traffic
canada

[19:49]

m0unds

cool, equiv of openresolverproject for ntpd
also, cloudflare's not on aws, but whatever, hahaha

[19:50]

mercutio

i dunno
cloudflare is terrible
they may haev some stuff on aws

[19:52]

m0unds

maybe staging or something, but they pride themselves on owning their hardware

mercutio

i haven't found anything about this ddos on nanog yet
i was avoiding reading nanog to not get swamped :)

m0unds

didn't see anything in nanog digests today

mercutio

i don't know if 400 gigabit is actually the biggest ddos too
i been reading this carrier comparison

.... (idle for 17mn)

for some reason i can't find any other articles or mentioning of ddos

[20:11]

up_the_irons

mercutio: all today, yeah
in fact, i dunno why i didn't look before, but like 30 minutes ago i noticed all our egress links are at like 300 Mbps!

[20:15]

mercutio

ouch

[20:15]

up_the_irons

lots of VPS' participating in the attacks (i'm sure innocent victims)
so i'm going to be blocking all NTP inbound

mercutio

hmm

up_the_irons

on all hosts

mercutio

probably prudent

up_the_irons

as a stop gap until people start fixing their setup

mercutio

there's some debate whether it's a good idea to block all ntp
as some ntp like to use the same source/dest port
but yeah as stop-gap it makes a hell of a lot of sense

up_the_irons

there's no debate in my mind when my network is hitting some target with > 1 Gbps of UDP

mercutio

heh
well the debate was whether it shoudl be rate limited
or blocked ocmpletely
i reckon blocked completely
i'm kind of against rate limiting

http://railspikes.com/2007/6/1/rails-email-processing

up_the_irons

rate limiting won't do shit
i mean, it will, but if 99% of the incoming is illegit traffic
your rate limit will effectively block all legit traffic too
so wtf
rather
it won't matter

[20:18]

mercutio

hmm
won't people be hitting that 5 megabit udp rate limit anyway?
i shouldn't distract you

[20:19]

up_the_irons

that's only in one direction
the wrong direction ;)
and yes, i'll take questions later :)

[20:19]

m0unds

oof
for freebsd guests: http://www.freebsd.org/security/advisories/FreeBSD-SA-14:02.ntpd.asc

[20:23]

mercutio

i still can't see anything on nanog
i wonder if tehre's another mailing list i should follow too
someone posted about it on nznog

[20:32]

...... (idle for 26mn)

***

DaCa has joined #arpnetworks

[20:59]

pcn has joined #arpnetworks

[21:05]

pcn

So I saw on some intertwitters about ntp blockage?
I don't have ntp running, but now that you mention it, is there an internal ntp server that can be peered with at the moment?

[21:07]

mnathani

pcn: its inbound ntp requests, outbound as to get time from say pool.ntp.org should work just fine

[21:09]

.... (idle for 15mn)

pcn

[21:24]

mercutio

it's still valid question
i don't know of any
i think i just use pool.ntp.org

[21:26]

mnathani

whats the nmap check or ntp check to ensure a host isnt configured incorrectly so as to be used in a UDP / ntp based DDOS attack?

[21:35]

mercutio

uhh is saw something somewhere
<http://nmap.org/nsedoc/scripts/ntp-monlist.html>

[21:35]

***

BryceBot has quit IRC (Excess Flood)
BryceBot has joined #arpnetworks

[21:46]

....... (idle for 33mn)

up_the_irons

mnathani: either upgrade ntp or just disable monlist command

[22:19]

mercutio

up_the_irons: that nmap thing checks for monlist
so you could port scan your ranges if you wanted to find out who is vulnerable to it

[22:20]

up_the_irons

mercutio: oh sweet

[22:21]

mercutio

which frmo your own ip could prob bypass any blocks

[22:21]

..... (idle for 23mn)

up_the_irons

this is old, but still looks like it'd work:

no need to set up procmail or Postfix filter to fork into ruby process. just have a daemon check a special email box!

[22:44]

mercutio

cool.
not that it really matters which way it is done

[22:49]

mnathani

up_the_irons: roger

[22:56]

acf_

mercutio: yeah, I see that in the smokeping
after prodding NTT a bit more
http://paste.unixcube.org/k/246aaa

[23:05]

mercutio

acf_: was it you that posted to outages@?

[23:07]

acf_

hmm? I emailed noc@us.ntt.net again

[23:07]

mercutio

not technical in nature
oh i just saw that same address as you were saying on outages mailing list
how do i search scrollback? :)

[23:07]

acf_

peering disagreement or something likely

mercutio

so it wasn't you that posted to oustages mailing list?

acf_

nope

mercutio

https://puck.nether.net/pipermail/outages/2014-February/006596.html
maybe it not someone in irc even

acf_

any connection to the recent ddos news things you think?

[23:09]

mercutio

i was wondering that
but i don't think it is
esp with your email response
it's ntt getting into messy situation like cogent
with not wanting to pay to send data
i imagine

[23:09]

acf_

wow. that guy sounds exactly like me

[23:10]

mercutio

see how i wondered?
he even on arp :)

[23:10]

acf_

idk if "not technical in nature" means ntt/verizon is purpousely degrading connectivity

mercutio

could be

acf_

or just that verizon/ntt have to negotiate bigger pipes to take the data

mercutio

did you see the uhh
god damnit
i weant to find a way to find urls i pasted to irc :)
http://arstechnica.com/information-technology/2014/02/netflix-performance-on-verizon-and-comcast-has-been-dropping-for-months/

BryceBot

Ars Technica: "Netflix performance on Verizon and Comcast has been dropping for months"

[23:12]

mercutio

is that something bryce can do?
it's the same two providers
even if diff origin
i think ntt is generally considered tier 1 and cogent not though?
but they're both huge

[23:12]

acf_

yeah
cogent is usually considered tier crap afaik

[23:13]

mercutio

heh i was reading on nanog about cogent
again :/
but yeah i not a fan

[23:13]

mnathani

godaddy is crap tier too

mercutio

i still reckon up_the_irons should just route verizon/comcast a different way
maybe with max prefix limit

acf_

a bit of testing seems to reveal that rerouting through nlayer would still go through ntt

mercutio

these things never seem to get fixed very quickly
and it usually gets worse before it gets better

acf_

we'll probably have to wait for level3 if up_the_irons wants to reroute

mercutio

acf_

idk much though

mercutio

he has tata too
but i dunno
level3 is sure to fix it
how did you test via nlayer?

acf_

yeah, it doesn't look like we're near the end of this

mercutio

oh from a lg?
i exepct level3 shouldn't take long to get connected up
i imagine it's just however long it takes to get a cross connect
which should be quicker for a big data centre you'd think
i mean they could probably turn it up tommorow if they felt like it
but if you tried to ask for tommorow they'd probably want to charge heaps for urgency

acf_

yeah. I certainly hope it's soon
well, nlayer-> verizon is direct

mercutio

but he'd probably have to go there to plug in cross connect

acf_

but verizon-> nlayer is via ntt

mercutio

oh
but it's -> verizon that is bad
judging by my routing via verizon being fine

acf_

so maybe that would fix it

[23:18]

mercutio

i doubt reverse path is via verizon
but i dunno trace me ?
202.49.67.22

[23:18]

acf_

(from verizon) verizon->ntt->new zealand stuff

[23:19]

mercutio

i actually think the routing side of things is somewhere the internet could really improve
so it's ntt return with no packet loss
which city is the verizon-> ntt in ?
oh of course california
is it san jose or los angeles?
or something else?

[23:19]

acf_

lax for me

[23:19]

mercutio

and yeh we tested sending via ntt in la and sj
and both were bad
and sending via verizon in la was fine

acf_

yeah, no packet loss to you
you have nlayer path to verizon?

mercutio

hmm
i misplaced your ip
i dunno what route it taking atm

acf_

108.40.173.223

http://kremvax.acfsys.net/smokeping.cgi?target=Remote.verizon-lsanca

mercutio

yeh that forward path via verizon
uhh it's showing packet loss now
i wonder if it doesn't like two traceroutes at once

[23:21]

acf_

probably verizon just really sucks

[23:21]

mercutio

i'll pingplotter from the other ip
yeah it looks fine
but i think it is rate limiting icmp too
oh no now some loss
this was all fine last night!

[23:21]

acf_

the internet is breaking down!

[23:23]

mercutio

it does that
it's not as bad as before
it's 1.14%
and it was easily 7% usually going via ntt

[23:23]

acf_

[23:24]

mercutio

yeh i saw that earlier today
the overnight thing is like wow
i can smokeping you from nz maybe?

[23:24]

acf_

that overnight thing was awesome

[23:24]

mercutio

if it's clean it suggests that it's single direction

acf_

the latency dropped from 30-40ms. I wonder if the ARP route changed

mercutio

i doubit it
it lpooks like congestion
but it's hard to know

acf_

it dropped off a bit sharply. does it do that?

mercutio

you mind if i add you to my smokeping?

acf_

go for it

mercutio

sometimes
i have curl testing too
but i assume you're not hosting any files on your dsl :)

acf_

nope

mercutio

what was that comcast ip?
oh it was comcast.net

acf_

also 72.55.8.69

mercutio

btw my friend in sj on comcast cable wasn't packet loss
what's 72.55.8.69?
that's via level 3 forwaered route

acf_

the non-rate-limiting router in front of work's internet
work blocks icmp :0

mercutio

ahh
is it on level3?