why is the fbi logging this channel?
kvr27 (beta) server going down for one last reboot...
crap, one more time...
heh
what are you doing to it? :)
just curiosity not waiting on it or anything
oh it's up anyway
says uptime of 6 minutes
mercutio: kernel patches
while we may still need some openbsd / freebsd testing for the beta VMs, my cacti server VM has been running on it for over two months without issues, so I'm declaring kvr27 production ready.  I have provisioned the first Linux customer on it.
(cacti server runs Linux if that wasn't obvious ;)
mercutio: can you re-run the test you mentioned was "it's quite reproducable"
i found the RAID controller write cache was off; this was one reason iowait was high; now fixed
no rush, just whenever...
Do you have any process by which you evaluate a guest OS on a beta system?
(features to try, stress testing, etc?)
I know some issues I've had in FreeBSD VMWare guests that seem fairly reproducable under higher IO situations.
They seem to crop up more frequently when ZFS is involved as well.
easymac: i want to see 1) no guest panic, 2) clean network throughput, 3) decent disk performance
i've seen frequent guest freezing with ZFS and load
less so if a lot of RAM is involed
zfs necessarily requires lots of ram for optimimum performance (I know this and I only admin'ed a freebsd system that used it for a very short time)
yeah
recommendations on a good Intel NIC for FreeBSD? (dual port, gigabit)
so that it uses the em0 driver
igb0 is driving me insane
INSANE
http://www.freebsd.org/cgi/man.cgi?query=em&apropos=0&sektion=0&manpath=FreeBSD+9.0-RELEASE&arch=default&format=html
I'd presume that would be a good list to work from
I can't say I have hardware knowledge from a freebsd perspective ;-)
up_the_irons: don't look at me at 6:47PM i'll be gone :-)
up_the_irons: but yes i'll help you rock that app.
is there an ipv6 routing issue im not aware of?
up_the_irons: you there?
my traceroutes are hitting 120 hops
that definitely can't be right
the packets just never get there
looooop
exactly
that's what im thinking
does arp control this router?
13  2607:f2f8:0:102::1 (2607:f2f8:0:102::1)  105.860 ms  105.698 ms  105.629 ms
after that it looks like it loops
Webhostbudd: do you have a sample traceroute with source / destination
I'm reaching google.com via ipv6 with no problem
RandalScwartz: same here
from the vps
however, from my home ipv6 to the vps
i get that loop
just a sec on the full trace
and a few others I tried
ahh, yes, let me try from my home to here.
http://sprunge.us/AFAY
I can ping6 from home to red.stonehenge.com
traceroute is good too
what's your host?
2607:f2f8:a9e4::2
it looks like it is one of the intermediate routers
because it can ping out fine
could i somehow be causing this?
I'm checking...
I'm getting stars after a bunch of hops.
let me try from red.
yea
Yeah - I can't get to yours from even nearby
I think it
is your end perhaps.
weird
alright
I cant ping 2607:f2f8:a9e4::2 from home
oddly enough i can ping red just fine
I'd still file a support request in case something is broke
perhaps ICMP are blocked?
mnathani: nope
it has an almost identical pf ruleset to my other ipv6 box
which can be pinged just fine
pass inet6 proto icmp6 all icmp6-type {echoreq unreach neighbradv neighbrsol routeradv}
maybe your default route is busted?
Well even the ipv6 ping sourced from Sprints looking glass : Success rate is 0 percent (0/5)
maybe
hmmmmm
try pinging my home IP: 2001:470:b148::25
it just has to be something on my end
works fine from the box
no problem reaching you from red
(your home, that is)
see, it's weird, i can ping out and get response
so routing may be working?
sounds like wonky firewall then
yes it does
by the way, if you haven't seen mtr, get it.
it's far slicker than ping or traceroute
from my ARP VPS: 18 packets transmitted, 0 received, 100% packet loss, time 17582ms
i just turned off pf
still nothing
... http://en.wikipedia.org/wiki/MTR_(software)
yea i've used it
and long since forgotten
=p
my rc.conf is pretty simple
ipv6_enable="YES"
ipv6_defaultrouter="2607:f2f8:a8e4::1"
ifconfig_em0_ipv6="2607:f2f8:a8e4::2/64"
nothing?
hmmmm
I'm using the 0 address for my box. :)
red.stonehenge.com has address 208.79.95.2 red.stonehenge.com has IPv6 address 2607:f2f8:3080::
shorter to type
=p
yea
i should actually use the 0
anyway
ill see what i can try and change
I also have ipv6_gateway_enable=YES
I think that was to try openvpn over ipv6 though
up_the_irons: well the test runs a lot faster :)
round-trip min/avg/max/std-dev = 0.559/1.208/7.285/1.409 ms
and that's not as bad
up_the_irons: speed increase of roughly 4x for my test
where it's twice as fast  without the sync commands
anyone else seeing packet loss to their VPS?
twobithacker: not noticing anything here, mtr?
yeah, it's clean to ge0-arpnet.cust.lax07.mzima.net then about 25~30% packet loss
twobithacker: hostname/IP?
Comcast on the way there, trying to get a trace back now
174.136.99.74 is my vps, I'm on 17216.146.45.2343
err 216.146.45.243
IRCing from my VPS too, the packet loss is more than mosh can deal with nicely
ah yes
I have loss over v4 as well
0% loss from Toronto
ah, v6 is much better
looks like it's the return path, packet loss as soon as it hits Comcast
maybe someone saturating a link :/
go comcast
i think the loss just got worse o_o
heh
i lost connection to freenode
which seems to happen to lots of people frequently
http://pastie.org/private/agllzswbtezlpyyjtz2a
but i was just thinking yesterday how i seem to avoid it
from both sides
wow
I think thats why my friend was complaining about not being able to reach his znc session
that looks like congestion to mzima in genreal
which most outgoing routes go over
until up_the_irons get's his new router
znc is still connected to IRC because its over IPv6
hmm
happsne here too it seems
ge0-15.as01.lax07.mzima.net
as starting hop
ahh that's same as you
yea seems like mzima is having issues
im sshing in through another VPS via IPv6
notice how it doesn't seem consistent between hops
i feel like i have slight lag
but smooth
hmm it's routing via trit
oh hey
how do i do a paste?
pastie.org
http://pastie.org/5075292
I'm guessing v6 is mostly over peering with HE
who's saturating my linkz?!
i think it's an incoming issue not outgoing
based on that
but the other things i'd found were screwed in both directions
err i mean packet loss all the way
telstra must have a different return path that's hitting an unsaturated link
man i am gettin lagielagg
i can do the reverse path from that ip
oh wow 1 Gbps of traffic, who will i shit can today...
http://pastie.org/5075303
up_the_irons: O_o
twobithacker: not telstra return path but yeah
gah
this wind is creepy
let's hope the power doesn't go out again
up_the_irons: who? Thought you limited us to 100Mbps, so that would be 10 users to shitcan... and grammatically speaking, "whom"
things look clearer now
brycec: it's prob incoming ddos
which can go way over that
probs
some loser kid in his basement i imagine
brycec: you can't limit incoming though
target has been identified and null routed
yeah, that's much better from here
swoot
it does look like it cleared up
yeah I know up_the_irons (unless you either have friends on their end, or like borking the BGP periodically)
up_the_irons: that write cache thing doubled my performance with and without sync
err actually i think it was 4x
up_the_irons: outgoing packet flood? :P
static: i think it just happened incoming was the same as outgoing for the hops that showed all loss
brycec: we basically give the target IP a null route community and certain important peers / transits cease forwarding traffic to this IP
bingo
mercutio: yeah i'm not surprised, that write cache rocks
up_the_irons: well, it use to be slower than the old node...
but not by a lot
and a bit variable
mercutio: would you say the performance is on par with the regular VPS' ?
and now it's obviously faster :)
up_the_irons: i /think/ so
staticsafe: incoming :)
ah
but i ssh into the old vps to get into the new vps
or via ipv6
ipv6 is longer route though
mercutio: oh so it's faster now?
tar definitely seems faster
nice
i was disconcerted by the high network ping times when untarring before htough
mercutio: do those high network ping times still occur?
and vmstat was showing very few interrupts when that was happening
up_the_irons: well highest was 7 msec
in my untar
i assume it's probably a bug somewehre where it's running out of queue slots or something
and spinning in the inrerrupt handler
and not receiving network packets while that happens
so teh write cache thing really might alleviate it, and it may not happen at lower load anyway
but yeah it was over 200 msec before the write cache peak
mercutio: roger
oh the other weird thing is that ping times in general are higher than old vps
like if i ping the ipv6 gateway
but that may be going through an extra switch or something
mercutio: nah same switch
mercutio: how much higher are we talking about
about 100%
err 50%
so double?
yeh
but
that's still like half a msec
--- 2607:f2f8:add0::1 ping6 statistics ---
29 packets transmitted, 29 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.394/0.622/1.158/0.178 ms
that's old one
yeah i don't think i'm gonna be able to do much about
it may be ethernet coalescing setting
--- 2607:f2f8:add0::1 ping6 statistics ---
27 packets transmitted, 27 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.696/0.928/1.821/0.263 ms
that's new one
btw, this recent UDP flood was targetting an IP on kvr26, and I recently used my new NOTRACK rules (that we talked about) on that host.  Man, even with 1 Gbps incoming, shell over SSH to kvr26 was clean as a whistle, no delays at all.
sweet
up_the_irons: nice
my beta kvm is on kvr27 isn't it?
mercutio: yup
does that have notrack too?
mercutio: yeah looks like double
mercutio: try IPv4, not IPv6 though.  IPv4 is hardware accelerated
up_the_irons: any specific target IP on kvr26?
i first noticed it when thinking it was weird that gateway had much lower ping than the other vm
yeah kvr27 has notrack
yeh well the ipv4 gateway is congested atm
solj: yeah, the target was identified as a single IP
err i mean pinging next hop is deprioritised
and the ipv4 from the new vps goes through the first vm
so would need to contrast it against something
solj: if you can still reach your vps, it wasn't you :)
mercutio: ah right
oh up_the_irons ... is kvr27 on 100 megabit atm btw?
yes
cool
well ipv6 performance to another host in los angeles was basically the same
cool
up_the_irons: awesome, thanks!
solj: np
oh, and your ethernet card issues on freebsd would probably be alleviated by Intel® PRO/1000 PT Server Adapter
they're standard pci-e cards
without the fancy stuff
err withotu the really fancy stuff
latency to another LAX VPS from my arpnetworks vps - http://sprunge.us/iVFI
:)
heh under 1 msec is nice static
yep
we interviewed the KVM guys today on FLOSS Weekly
pretty active chat room
that sounds like a dentist magazine
oooh, and the audio is already out
Webhostbudd: nothing on ipv6 changed btw.  looping is a sign that my router can't see a route to you
FLOSS = free libre open source software
hmm
my podcast that gets about 50k downloads a week
mercutio: yeah after a bit of research i also found the PT card and NewEgg'd it already :)
latency over v6 is a bit higher cause the other LA host does not have native v6 and is using a HE tunnel
is it audio only?
up_the_irons: oh cool
RandalSchwartz: nice!  i'll have to listen to that one
static: he.net tunnels aren't likely to be that slow actually
it isn't
for just los angeles traffic
staticsafe: nice traceroute!  that VPS at Cyberverse is about 1 mile down the street, so yeah, pings are way low :)
:D
thats my friend's box with Chunkhost
mercutio: http://sprunge.us/iRTH
a lot of that latency's coming from the arp ipv6 router having higher latency
if i trace from another host it's 0.8 msec to that destination
although more hops
http://pastie.org/5075448
gonna get some air
why is it so hot in LA now?  It's almost november!
mercutio: interesting
welcome to global warming
i'm listening to this web cast
about kvm
twit.tv/floss ?
yeh
yeah.  that's what I sound like :)
so kvm was made for windows
no - there's a version of it for windows
ok this is too basic for me :)
no - wait... we get deeper
I just had to give gradual overview
i skipped ahead :0
gah i need more coffee
the developer is hard to understand/follow for me
probably partially accent
Yeah - they failed to follow our instructions about audio
it's also partially i think cos of being a geek :)
and they're israeli
rather than public talker
ahh ok
My instructions are to have a mic that is no further than six inches from the mouth.  they shared a single mic between them that was 2-3 feet away
that makes the sound very muddy
oh
it reminds me of skype
but with skype you have feedback loop
i use skype on my laptop rather than desktop with some inbuilt mic
it *is* skype
so it probably sounds terrible
oh real
yeah
skype isn't great for audio quality
it is if you have enough bandwidth
really?
i only have 1 megabit upload  i suppsoe
in fact, it's better than nearly anything else
but i've never noticed skype being that great
the SILKv3 codec is amazing
i find voip better
better than G.729
voip doesn't use g729
voip uses g711
or g722
uh - I have voip that goes 729
it depends on your clients
.. http://voip.about.com/od/voipbasics/a/voipcodecs.htm
yeh some people do voip with g729
g729 sucks
gsm is even worse
g729 is used in some call centres
that's why the music sounds terrible
er
g729 on *good* bandwidth sounds great
I'll just let you keep digging a hole for yourself though
what he said.
I've been to astricon.  I've hung out on vuc.me a number of times.
I've even had "the asterisk voice lady" make a promo for me for FLOSS Weekly. :)
I haven't, but I do have vanity commits in Asterisk, for whatever that's worth ;)
RandalSchwartz: that's pretty cool.
g729 isn't bad
but you notice it on music
gsm is bad
but g711 is only like 80kbit with sip
... http://soundcloud.com/randal-l-schwartz/allison-smith-floss-weekly
there it is
mercutio: and when you only run one channel of voice, that's wonderful.
well if you run lots of channels you can get a e1/t1 cheap anyway
i use voip over adsl with single channel for my normal claling
it does depend on the client a bit
it's pretty good with my linksys phone
jdoe - have you seen https://www.tropo.com/home.jsp ?
as long as you don't upload you're sweet without evne having qos
they have a develop-for-free setup... I have a couple of demo apps :)
RandalSchwartz: heard of it, I use twilio though.
unfortunately I don't really have a business use for it, I just somehow scored ~$80 in credit, so I use it for personal things.
tropo is more flexible, I think
could be... same idea though, looks like. Programmable voice/sms stuff?
basically, it binds together any of POTS, VOIP, SMS, and Jabber.
along with voice recognition and menu systems
ah. Jabber is neat, I don't think twilio does that.
though nothing prevents you from writing some glue between twilio and jabber, I guess.
what do you apps do?
... http://www.quora.com/What-if-anything-separates-Twilio-from-its-competitors
that top answer makes me cringe
well, the third answer down or so is interesting
tropo seems to be more of a glue
skype and AIM and twitter I forgot about :)
the Colin one? yeah.
wow, that guy's name is almost the same as mine
RandalSchwartz: yeah that's reasonable. I dunno, I think if that was something I was super concerned about I might just DIY the glue bit... You've got me thinking about gluing twilio to my jabber bot now :P
... but right now, I just use it for stupid shit.
like I can text myself a number, and it calls a script which prods asterisk to find me, call the number, and bridge.
call (424) 235-1666
that was one of the demos, but it's still cute
up_the_irons: any idea why your router might not "see" my node?
up_the_irons: my ipv6 config is the provided config
up_the_irons: so I'm not really sure what I'm doing wrong here
Webhostbudd: just a thought - try using a different address? Also might help if you provided your network settings file (pastebin, or whatever).
i actually did above
but ill repost
Thanks (too much backlog to sift through)
http://sprunge.us/UUaW
Webhostbudd: /128? Pretty sure you were given a /64
ifconfig_em0_alias0="inet6 2607:f2f8:a8e4::2/128"
yea, but aliases should be a single host
always
in bsd
unless ipv6 is different than ipv4
heh tbh never setup aliases on bsd
*bsd
(well that's not quite true... I've done it on OpenBSD. But this isn't OpenBSD)
but i mean
Right you are, it would be given a 128
i can't even connect to the original IP
the one with /64
Yeah I would recommend trying a different IP, just to test
eg ::2
i did that before
didn't work
im pretty stumped
what's weird is that i can use ipv6 on outbound connections, but no one can connect to me
i even tried disabling pf, nothing
it looks like there is a loop in routing to my ip for new connections
which makes no sense
im baffled
I can ping ::0
64 bytes from 2607:f2f8:a8e4::: icmp_seq=0 ttl=59 time=26.9 ms
wat
you are the only person who can
heh
from an HE tunnel
no way....
no one else can
WTF
im soooo confused
http://sprunge.us/FeSX
it just loops
and hits max hops
Webhostbudd: do you have any services open?
such as?
80, 22, something I can try and open a connection to?
ssh
22
doesn't connect :/
it should
so I must not be hitting you, or your traffic isn't making the return journey
shouldnt&
i just changed the ip
need to update sshd
sec
ssh: connect to host 2607:f2f8:a8e4:: port 22: Connection timed out
yea
it won't
sec
np
now
still nada, timeout
hmmmm
but you can ping it?
yes
traceroute hits HE, coresite, then you
I can ping you from my ARP vps too
see
my packets get stuck at a higher up router
13  2607:f2f8:0:102::1 (2607:f2f8:0:102::1)  99.083 ms  102.662 ms  103.600 ms
that's the last hop that works
I don't even have that hop...
exactly
that's probably why yours works
and based on address, that would be one of up_the_irons' boxes
mhmmm
it gets to the gateway fine
what's weird
is that requests to the gateway ip
2607:f2f8:a8e4::1
don't go through that box
does chrome seriously not work with ipv6
now it routes
woah
Yes CHrome does ipv6
but i can't type in the url
http://2607:f2f8:a8e4::
it tries to search it
ipv6 urls need to be in []
always
wat
really
since the browser can't tell what port you're trying.
ohhhh
good call
ha
just for fun, Webhostbudd... do you have a link-local address on that adapter? (it should be assigned one automatically, just checking)
yes
what's odd
is that it now magically responds to pings
it's not looping at that router anymore
heh definitely some weird routing going on
hmmm yep everything checks out okay on my end. I get responses from pinging your address, but absolutely no response if I try and ssh.
I'm stumped.
me too
hmmm
I dare say that maybe the routing holes earlier were a red herring, but can't say for certain.
yea, who knows
Webhostbudd: do you see any hits from me in your tcpdump?
let me see
(I've got a ping running on vps3)
If you don't see it, then I'm not really hitting you, and that would explain routing anomalies (and who the hell's IP do you have anyways) :p
getting the pings
damn. GOing to try ssh now
tbh was hoping you didn't see the pings... would've made life simpler, problems would've been upstream of you.
ping timed out. retrying...
seeing them?
serc
sec
nope
awesome...
next test I guess - try hitting vps3.cobryce.com
ping or ssh
PING6(56=40+8+8 bytes) 2607:f2f8:a8e4:: --> 2607:f2f8:a650::3
16 bytes from 2607:f2f8:a650::3, icmp_seq=0 hlim=63 time=3.072 ms
16 bytes from 2607:f2f8:a650::3, icmp_seq=1 hlim=63 time=1.152 ms
16 bytes from 2607:f2f8:a650::3, icmp_seq=2 hlim=63 time=1.137 ms
^C
see your pings
--- vps3.cobryce.com ping6 statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 1.137/1.787/3.072/0.909 ms
now ssh?
looks like you made it
im baffled by this
i get no ssh packets
getting lots of http packets
Webhostbudd: traceroute says ICMP reach you, but NO tcp traffic does.
http is hitting
 2  * * *
if you say so - I'm not getting it back
but then that might just be traceroute
yes, my firewall should be rejecting too
(it's traceroute...)
but if the firewall dropped the packets
you would just get losses
"set block-policy return"
so i would hope it's actually properly rejecting
try hitting me with ssh traffic again?
hitting
not returning
getting no dumps
imma try port 80...
wait, now i got some
16:42:50.583604 IP6 vps3.cobryce.com.47136 > 2607:f2f8:a8e4::.ssh: Flags [S], seq 3526079872, win 5760, options [mss 1440,sackOK,TS val 177719508 ecr 0,nop,wscale 6], length 0
16:42:53.582536 IP6 vps3.cobryce.com.47136 > 2607:f2f8:a8e4::.ssh: Flags [S], seq 3526079872, win 5760, options [mss 1440,sackOK,TS val 177720258 ecr 0,nop,wscale 6], length 0
16:42:59.582439 IP6 vps3.cobryce.com.47136 > 2607:f2f8:a8e4::.ssh: Flags [S], seq 3526079872, win 5760, options [mss 1440,sackOK,TS val 177721758 ecr 0,nop,wscale 6], length 0
super late though
(once I install curl)
trying port 80
is tcpdump super delayed
or is this router just having trouble?
nope, never been retarded for me
because i get some packets
but they should up almost a minute late
and all sorts of weird shit
do you get synack?
im getting nothing right now
so... traceroute6 -T on ARP got responses back from you Webhostbudd
now it looks like it might be doing stuff
i'll see if i can connect to your ssh port
can frmo arp
and can from another location
getting immediate connection refused now from Webhostbudd on 22
as well as a 3rd location, so i think it's working now?
oh maybe i'm connecting to wrong locaiton?
i was going to: 2607:f2f8:a650::3
hmmm
mercutio: that's me
a8e4 is Webhostbudd
oh
:1?
srsly though, both ARP and HE I get immediate refusals from Webhostbudd
mercutio: ::0
2607:f2f8:a8e4::
now it works
lol
wtf
# telnet 2607:f2f8:a834:: 22
Trying 2607:f2f8:a834::...
like that?
it's not working
mercutio: you typoed
it's not even working from arp
oh
oh e not 3
:)
oh my god
SSH-2.0-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503
i think i just found out whaty it was
LOL
Webhostbudd: eh?
works from 3 locations
2607:f2f8:a9e4::
Webhostbudd: another dead hooker clogging the Internet pipes?
what was it webhost?
is what my dns records say
2607:f2f8:a8e4::
is correct
oh heh
holy
webhost typo'ed too
i think i misread between 3/e
ugggh ipv6
the addresses aren't very memorable
That doesn't make sense though... we're all haxing with addresses here. DNS shouldn't have played the slightest role in any of this.
oh god
all of my firewall rules are wrong too
they have the same ip
CRAP
oh well that could explain it some... though you said you'd tried disabling pf, so that should have ruled it out
yes, but i was using my domain name
to connect after disabling pf
and they were both wrong
LAWL
okay
SOB
C+P error
mysterIES  solved
 /dcc Webhostbudd send beer
there there, it's okay
hahah
up_the_irons: Webhostbudd's IPv6 woes solved. User error, typo.
up_the_irons: Bug report: IPv6 addresses are haaaard and easily goofed up.
yep
=p
and impossible to see
I figure he'd appreciate knowing things are resolved... and how he could improve them.
well, i wish non-existant hosts would not be available
but maybe a9e4 is something
at any rate, if that guy doesn't exist, then it should really give a unreachable error
that would have been a huge red flag
in pf
is there any way to do $prefix"2"
so it appends the string inside a rule
darn
heh, fyodor's girlfriend just gave me her phone number
I don't think it counts if it's written on the bathroom wall...
nmap?
"fyodor" is the pseudonym of the guy who wrote it
heh I meant that I was talking smack about strangers, something I was raised to know better than to do.
regardless, lol
in the absence of those who raised brycec
:D
:(
how about
heh
BRYCE NO MIDDLE INITIAL C
WHAT DID WE TELL YOU
Man... I want to name my kids "no middle initial" but I'd shorten it to just NMI... and nobody would understand
or they would and hate the kid'd parents ;)
well some people might... and I'd carry around cookies for those people. pretty stale  I imagine.
haha
hot