yeah i know but there are politics involved...
if it's not tweaking every little site but only the bigger providers then it's simpler config wise
and if it's that the as path length is the same for both, it could hop randomly if the weight is the same
up_the_irons: 14.04 here, but mine are in enforce mode
up_the_irons: fwiw, I emailed the NTT NOC a long time ago
idk if you've seen it
http://paste.unixcube.org/k/bb55b1
... http://dogeon.org/
it seems that the return path of level3 (level3 -> arp) goes through gblx and trit
http://paste.unixcube.org/k/e0c7ac
mercutio: I would think that since ARP and level3 are peered that it wouldn't go through trit
acf_: i've seen people mention that it's only outbound l3 right now
http://pastebin.com/k3meCyuJ acf_ - from a couple days ago
cool, thanks, didn't see that
sure thing, had to dig through my buffer to find it. wondered if i'd just imagined seeing it :)
I too recall up_the_irons stating it's outbound only now, that ARP isn't yet announcing.
A nice gradual turn-up
http://urbanvivant.com/wp-content/uploads/2010/12/turnip.jpg
yay. my headphone amp was repaired and is on its way back to me.
onboard audio makes me sad
m0unds: turns out i had the security driver set to "none" in the config; so commenting that out fixed the prob
acf_: i haven't announced our routes to level3 yet
it's all outbound
oh, all you guys already said outbound...
:)
acf_: read your email; so i take it Verizon didn't respond when you contacted them?
up_the_irons: ah, gotcha
up_the_irons: noticed any difference in CPU load on the updated hosts?
slightly lower
I was working to counter that last night with a git-gc on a 1.5GB repo... Then again, i'm not on an updated host yet
haha
up_the_irons: yep. basically Verizon won't talk to you unless you're a transit customer or something
that's been my experience with them anyway
aholes
yeah
I don't think they could/would have told me any more than NTT anyway though
up_the_irons: how do the as path lengths compare with verizon/comcast/etc?
mercutio: as path lengths of what?
i do wonder if a lot are the samw
verizon ip's
umm
whatrever acf is tracing to
comcast.net is a good comcast example apparently
i can't remember acf's traceroute site :)
http://kremvax.acfsys.net/smokeping.cgi?target=Remote.verizon-lsanca
that's a router
verizon.net
you shouldn't really make assumptions about routers cos of deprioritisation etc
verizon.net actually takes a different return path
hmm
if you look at the last 360 days graph
but yeah i do wonder if as path length is the same
hmm i have some verizon ip i'm smokepinging from nz
hmm there's loss atm
there's always loss
from arp
but not from nz
yep
oh
yeah from nz it's going via alter.net
which is verizon right?
yep
what is going via alter.net?
from my home
oh sweet kvr15 update is happening :)
i wonder whose ip this was
 ae-0.verizonbusiness.lsanca03.us.bb.gin.ntt.net
the packet loss seems to start there
same from arp
this is from arp
from home there's no loss :)
ah, ok
how can i find out if my vps has acpi enabled?
i think they all do
unless it's dos or something
what OS?
fbsd 9.2
in openbsd it shows up in dmesg
yeh that should be fine
did you get emailed too? :)
yeh
i guess we're server brothers
heh
LOL
he did say he was doing two servers
i'm on 14
i'm actually doing three now
im'm on 15
oh
i have some extra time next week, so might as well bust out the upgrades
up_the_irons: is it getting easier?
14 > 15
neighbors then
mercutio: it IS actually..  last few upgrades have went very smoothly, even faster than my estimated time in the maint advisory
i always overestimate times myself
just in case anything goes wrong
and because i usually underestimate by default
yeah
we can't use virtio until there's a knob though right?
well without manual support request
you can modify the host OS without it affecting the disk partitions of the vpss?
mus1cb0x: that's normal for virtualisation hosts.
the two are disconnected
i wonder how deep you can nest that before things go weird
imaging a tight loop programmatically creating nested guest environments
some will run openvz inside kvm
qemu
ad infinitium i imagine
you could send a chinese hacker into a mirrorverse of insanity
you'd probably run out of memory though
the extra hw abstraction can help you work around bad hw support in kernels compatible w/openvz
qemu doesn't need privileges for emulation
it also supports non intel cpus
yes, i run qemu on an opteron server
probably quite slowly.
m0unds: no, i mean like it can run on powerpc etc
and can emulate 68k etc
oh, non x86
xen is using qemu too
but qemu is actually supporeting lots of weird stuff, it's damn cool in a way
not a big fan of xen-pv
bhyve is getting some nice upgrades this summer
xen has a whole lot of new modes now
like hybrid stuff
modern cpus are not so quick in pv mode
yeah, but lots of hosts that run xen environments won't run w/hvm or whatever
yeah there's somethign in between too
so you're stuck with linux or obnoxiously unstable bsd
which is better
well yeh i chose arp becauase of kvm and openbsd
err
kvm and ipv6
yep
because i wanted to run openbsd
by openbsd i assume you mean freebsd
right
i don't use ipv6 though
;>
oh location was theo ther reason
haha
i'm in new zealand, los angeles and san jose are the only two fsat places from here really
i have a silly amount of unused referral credit w/another provider that i won't use until i can get bsd stable in their env
well low latency
i've run freebsd on xen fwiw
it runs fine in hvm
and openbsd on vmware
it runs in pv mode too
it's terrible in xen-pv when your only option is pvgrub to bootstrap bsd
it's awful
but it runs better in hvm mode
i was using pygrub i think?
it was on opensolaris
which doesn't have pvgrub , and is xen 3.4
oh having it's own cp is good too
i am kind of anti solusvm
yeah, solusvm is awful
the way it handles assigning ipv6 addresses is hilarious
you don't get a routable /64 or whatever, you get X number of addresses
so...8 or 16
or whatever the host chooses
mercutio: i'd like to visit nz some day
m0unds: sounds very mcdonalds
mus1cb0x: heh
m0unds: it's also got lots of funky mysql and iptables stuff
if you want to do thigns like limit people to their own ip it's complicated by solusvm getting in the way with iptables
which means most solusvm hosts allow spoofing
most are just huge bridge domains too
so you get arps for a whole lot of random ip's
Comcast cable kind of does the same thing
you see lots of arps for random IPs
can you spoof?
i think that's just the nature of cable networks
no
no
how many /24s is it?
well, my netmask is /24
cable network in nz overloaded
and if I try to ping things in the same /24
and routers couldn't handle the number of arp requests
there was no standard router provided
the cable network gateway will spoof its mac address for the ARP replies
but it meant lots of people's internet died completely in evening
yeah i had /24 on cable
it was nice that you could just plug it into a pc
same with Verizon DSL here kind of
set an ip and gateway and you were off
verizon is vdsl now right?
except Verizon caches your MAC address for DHCP, so you can't change computers without calling them first
or spoofing the MAC of course
with some shitty router that everyone puts their own one after
no, ADSL
2
oh weird
i'm sure i had a friend who went comcast -> verizon -> comcast
comcast was higher latency apparently
that's kind of hard to believe
and only like 30 megabit
he was in kansas
Verizon DSL sucks crap
i think vz can offer vdsl2 if you're close to their gear
oh
like..sub 1000ft loop length
i meant verizon was oops
sorry!
I'm less than that from their CO
and comcast was 50 megabit i think?
old dslams maybe in that facility
where in the US are you?
central coast of california
93455
old GTE territory
huh, weird - business i used to work at had vdsl service in huntington beach area ca
heh i have friend in san jose, and he said comcast was pretty bad
but it's basically comcast or at&t
i suppose comcast is better than at&t
comcast is in the middle of doubling residential speeds again
at least they actually upgrade their network sometimes
he has heaps of jitter
and really high pings to cogent
it's about 50 msec pings to san jose from san jose
over cogent
peering congestion you think?
this was a while ago
the routing is probably goofy
before the netflix peering
routing looked fine
it's probably the netflix/cogent/comcast dispute
wouldn't have surprised me if it went SJC>LAX>Cogent>SJC or something
for some reasno people call it the netflix issue
but it affects other traffic too
my mtr to cogentco.com: http://paste.unixcube.org/k/efacc8
nah it didn't
yeah
it did over 3mb/sec to los angeles though
on 25 megabit plan
mci
ok it wasn't that broken
hahaha
i've seen that from cogent to at&t
oh that's to cogent's website
try sj.meh.net.nz
via nlayer
oh
http://paste.unixcube.org/k/ea49f6
ok that's bbetter
it was going via comcast before
cogent denver pop -> albuquerque, nm (460mi) is 55ms
maybe they stopped sending via cogent when netflix paid money
cogent LAX pop -> albuquerque = 27ms
800mi
hmm that ping is still kind of high though
what do you get to arp?
28ms
he got lower ping than that to arp
http://paste.unixcube.org/k/7cabf5
i don't really know what santamaria is
he was in santa clara?
right next to santa barbara
but they prob not close
right in the middle of lax and sfo
nice area of ca
oh
my Verizon traffic goes via lax
i've spent a fair amount of time in ventura & carpinteria for work
so the issue is that there's no peering there
and it has to go to sj/la
or both
and my Comcast trafic goes via sj
yeah
weird google says santa marisa is in crazil
santa maria brazil
i can't type
http://binged.it/1i9ZuWW
http://binged.it/1i9ZuWW -> https://www.bing.com/maps/?v=2&where1=Santa Maria, CA&form=LMLTSN&cp=34.95372009277341~-120.4355697631836&lvl=12&sty=r&encType=1
oh i see where it is now
that does look nice
it looks samll though
i like that area of ca a lot
it's pretty nice :)
last time i went was last june, it was 65 every day
so nice
does it rain there?
we're in a pretty bad drought right now
so not much
that's the most disturbing thing about california to me
yeah, it is pretty bothersome
it was a lot better some years ago
wow
it rains over 100 times more here than there
a good friend of mine used to work for the UCSB NOC a few years back
1,212.4 inches per year, compared to 14.01 inches per year
haha
so yeah it seems a ltitle disoncerting :)
http://goo.gl/Bw1jei
the local CO
http://goo.gl/Bw1jei -> https://www.google.com/maps/@34.883954,-120.418628,3a,75y,91.38h,85.04t/data=!3m4!1e1!3m2!1scvaM7PWqCG7UQ9E4kfPquQ!2e0
wait..
1,212 inches = 100ft of rain
northern california rains more?
haha
tahts' per year
mm
wikipedia shows inches in parenthesis
1212.4mm
oh it's precipitation vs rainfall
is that the same thing
yes
the parenthesis is 47.732"
oh i fucked up didn't i
yes
backwards, lol
they swapped it aruond
i was trying to use inches for US audience :/
yeah, i think because you're metric and we're lame
sorry
well i expected wikipedia to be constant
ok that difference isnt' so bad
it's just 4x as much
we get 9.45 in/yr
i'd rather 14 :/
you and i both
i dunno
haha
haha
i'm also in the desert
nice
when i was younger i thought it'd be cool to get a job in the US
but you need work visa and to work for one company for ages :/
my house is at ~1800m above sea level
my friend did it by shifting within the same company to a diff country
but it ook him two attempts
yeah, it's a pain
there wsa some paperwork error or something
i have a friend who's finishing up her fellowships in the US as a pediatric orthopedic surgeon
and they're still not going to let her stay, she has to return to brazil at the end of the year
and you can't fix it you have to wait until next year?
yeah, it's a bureaucratic mess
damn
legal immigration into the us is gross
i wonder how he managed it
another friend got married in canada, and it took 6 months for his wife to legally immigrate to the US
and ~$5k in attorney fees
ouch
he must have done it properly because he bought a house
and hours upon hours at US consulates for meetings with immigration clerks and stuff
apparently there's ths really exepensive grooming thing for houses when they sell
and therew as like a video walk through and stuff
that doesn't happen here at all
but it was actualyl cheaper than i thought it'd be
huh
they show this video of the house
going through all the rooms
with music
and panning
hahaha, yeah, i've seen that before
and everything looks so neat and tidy
and sterile
yeah, they "stage" the house
sometimes they do it empty, sometimes they rent furniture and set it up so you can get an idea of what it'd look like if you lived there
oh that's the term yes
oh, great
my smokeping graph looks like acf_'s verizon one
hahaha
yeah, seeing roughly 2x the latency at this hop 10. ge0-arpnet.cust.lax07.mzima.net
looks like it's been like this daily since the 3rd
https://twitter.com/j_smedley/status/475023041106169856 :O
TWITTER: looking good for a Waterson and Mattherson merge. (Fri Jun 06 21:14:53 +0000 2014, retweeted 50 times)
did anyone not on kvr15 get a maintenance announcement for kvr15 ?
i didnt notice anything
kvr14 is goiung down too
and somethjing else
m0unds: mind revealing the source destination network/host?
/s/source destination/source/
(i'm on kvr15 so don't know if they all say that)
m0unds are you on Comcast?
http://kremvax.acfsys.net/smokeping.cgi?target=Local.phoenix
that is rather upsetting
crap now comcast <-> nlayer is bad...
or is it comcast <-> ntt?
my forward path is via nlayer
and return is via ntt
from Comcast's perspective
hmm if you're seeing loss at ge0-arpnet.cust.lax07.mzima.net
it makes me think ntt
because ntt becomes the return path at that hop
definitely NTT: http://paste.unixcube.org/k/65f903
sorry, was commuting home - i'm on comcast biz
it's nlayer inbound from comcast -> arp
and return path is via ntt
and i think it's ntt
it stops being problematic at precisely midnight mdt
more peering problems I assume...
guessing so
starts at the same time
ends at the same time
I'll email NTT again
not that it'll help any
are you seeing this stuff starting at around 1530ish PDT?
it starts for me at 1630 mdt, ends at midnight +/- 5minutes
yepyep
http://kremvax.acfsys.net/smokeping.cgi?epoch_start=1402038000;hierarchy=;epoch_end=1402124340;target=Local.phoenix;displaymode=n;start=2014-06-06%2000%3A00;end=2014-06-06%2023%3A59;Generate%21=Generate%21
mine's just as smooth before that too, haha
very little jitter
I've been noticing problems with voip lately
probably that
ugh
up_the_irons: maybe you should see this
strange
Level3 appears to have a shorter AS path to Comcast than NTT
http://paste.unixcube.org/k/83497d
vs
http://paste.unixcube.org/k/6f24b0
or it's the same... I think I misread
but Level3 looks like NTT when it's congestion-free
acf: i found something from last year on google with ntt/verizon :/
it's probably the same
meaning the issue has been ongoing for at least a year?
that's not as path
the AS path should end in 7922
with the network prior, like 3356 appearing
where 3356 is level3
like i can see routes for 701 and 3356
can't see what arp sees :/
hmm ok
and 701/703 is verizon
need a looking glass
well don't need, but would mean coudl see what they see
hmm
so verizon have 703 and 701 in the AS Path
and level 3 just has 3356
but it's interesting that it's both comcast and verizon having issues
i wonder if they're working together
because they both had netflix issues too
in a similar fashion