kellytk: Normally what you've described is resolved by set keep-state
(I tried looking at your pf.conf but it's been removed already)
*set keep state
i think
oh my bad, it's this:
block return
pass
That establishes the "keep state" for connections
(Overall my point is that the default pf.conf, at least in OpenBSD, has no problems)
fsck my cc company, gonna have to use bloody paypal mc instead
they got real assy after i bought things on play
bummer
brycec: http://pastebin.com/6uGA28JM is my ruleset
FYI, it's for FBSD's port of pf
Well assuming it's similar enough to OpenBSD's, then yeah your block by default isn't helping ;P
(it's blocking everything including the return acks)
ime it's nearly the same as openbsd's pf
(same)
i love pf
Differs when it comes to specifics like queueing
yeah
but its common features are the same
yup
like most languages with regional dialects :p
bryce, which line?
"block log all"?
Is the traffic being blocked in http://pastebin.com/kzSv01i5 important?  It looks like it's related to DNSSEC resolution but I'm not positive
You blocked the response from a DNS server. It happens to be a request for a DNSKEY record, but I don't think that's why it was blocked. It was blocked by rule 1, "block log all"
heh
(you can confirm which active with "pfctl -sr" of course)
brycec: What I would like to do is block all, and selectively pass
udp is nasty
That's what she said!!
well at least if you want to send out udp packets and get them back
Yeah, states and UDP...
http://pastebin.com/afQv0gj5 is the output of pfctl -sr
if you allow all responses from port 53, then people can taget any udp ports on your host just by using a source port of 53
there's no direction with udp, ...
there are helpers, but they can have issues too
and i don't think pf supports any of those fancy helper things.
(tftp-proxy...)
(siproxd...)
tftp-proxy is transparent isn't it?
err i mean, you redirect the port to a local host
rather than inline
i suppose it makes no diff
anyway, if you use the same external recursive dns you can allow source/destination ip with all ports for udp
I'm not finding a way to flush Unbound's cache totally (http://unbound.net/documentation/unbound-control.html)  Am I missing something?
 reload Reload the server. This flushes the cache and reads  the  config
              file fresh.
^
I just found that, thank you :-)
While running `host update.freebsd.org` the states are http://pastebin.com/bHVa3GDL
unbound-control flush *
or that
hahaha
staticsafe: Are you sure?
That gave odd output when I tried it
root@lasciel:~# unbound-control flush *
ok
staticsafe: http://pastebin.com/BKvQCAMc
thats a shell interpretation problem
I use tcsh
try quoting the whole thing
What do you mean by whole thing?
"/usr/local/sbin/unbound-control flush *"
/usr/local/sbin/unbound-control flush "*" worked
or that
Thanks for the tip staticsafe
np
So I'm back to the firewall not allowing name resolution
i didn't realise reload flushes the cache on unbound
that's kind of sub-optimal
that would not be valid :P  staticsafe | "/usr/local/sbin/unbound-control flush *"
i have found that reload tends to crash out though, so i've been doing restarts...
(Unless you have an executable named "unbound-control flush *" of course)
ah true
which is also sub-optimal
that's with ubuntu trusty, i haven't checked to see if it's got better since then.
http://pastebin.com/1kZ66MPk is a summary of the ruleset problem I'm having
I'm getting the impression /usr/local/sbin/unbound-control flush "*" doesn't actually flush.  Results return immediately, whereas after a `service unbound restart` results take a moment
Something else interesting is `host google.com` returns with the firewall up, however `host update.freebsd.org` does not
possible you are dropping EDNS queries at the firewall
i would suggest adjusting your DNS rules
flush the cache, do queries for google.com and freebsd.org and check firewall log
staticsafe: Is http://pastebin.com/J3x6PgQA what you mean?
I recall having to add the last line in the past, but when I've looked for recent info on it I only found mailing lists, no docs
edns0 (since glibc 2.6)
                     sets RES_USE_EDNSO in _res.options.  This enables support for the DNS extensions described in RFC 2671.
that is on linux
Which man page?
man resolv.conf
FreeBSD's resolve.conf man page doesn't include an explanation of the option unfortunately
its probably not a valid option then
the difference between google.com and freebsd.org is that freebsd.org is DNSSEC signed which requires EDNS queries to validate
Ok that's what I suspected.  So is it likely that the pf ruleset is blocking DNSSEC, but not regular DNS?
its breaking EDNS in some way yes
That's what she said!!
kellytk: http://lists.freebsd.org/pipermail/freebsd-net/2007-May/014190.html
BryceBot: no
Oh, okay... I'm sorry. 'its breaking EDNS in some way yes'
staticsafe: IPv6 isn't necessary for this, correct?
no
pf is dropping the fragments
pass out quick on $pub_if inet proto udp from $pub_if to any port $out_udp_services keep state > pass out quick on $pub_if inet proto udp from $pub_if to any port $out_udp_services keep state keep frag?
staticsafe: What are fragments?
http://www.dnssec-deployment.org/tag/udp-fragments/
Unbound has a edns-buffer-size configuration option to help, however is it correct to think that the better solution is to modify the pf ruleset to allow fragments?
staticsafe: Thoughts on using scrub fragment reassemble?
This is strange.  Two identically configured FreeBSD boxes on my LAN, each having "scrub fragment reassemble" added to pf.conf, one can resolve update.freebsd.org and the other cannot
i do not know, i don't have experience with pf
Which firewall do you use?
Two identically configured boxes on the LAN (except for differing pf.conf), working pf.conf http://pastie.org/private/o6exhdd0wgyofhf0htcq and the broken pf.conf http://pastie.org/private/paf0wnaik0i49l2q0cxyyq
Ok this is odd, when I drop the firewall on the broken box and rerun `host update.freebsd.org`, it still returns "Host update.freebsd.org not found: 3(NXDOMAIN)"
up_the_irons brycec mercutio : I emailed softlayer yesterday about the domain they hadn't registered. No response and it is still not registered.
mnathani_: hahahaha
Register it and redirect to a lolcats?
well you've given them fair warning
direct it to ovh
lolol
would be embarassing for them
or worse, GoDaddy
set the nameserver expiry times insanely long
and direct to goatse
i dunno it depends how much you want to stir :)
I would rather they fix it
yeah :)
as one of my clients is about to become a customer of theirs
That's what she said!!
BryceBot: nbo
BryceBot: no
Oh, okay... I'm sorry. 'as one of my clients is about to become a customer of theirs'
you could just register it and set the name registration to their name servers
you could register it, and say that you got no response and go public
and say you're willing to give it to them at cost
I wouldnt want to risk a lawsuit
but going public without registering it first would be irresponsible
i thought you were in canada for some reaosn
I am in Canada
they have a datacenter here also
oh, i thought that protected you from US lawsuits for the most part.
at least frivilous ones.
apparently there's a big el nino thing happening soon
close to Mexico?
and july was the hottest recorded month on average around the world.
across huge areas afaik
across pacific ocean it seems
i'm trying to find something more moderate and balanced rather than alarming
not to much avail
http://www.thedailybeast.com/articles/2015/09/01/we-re-worse-off-than-ever-for-el-ni-o.html
this seems better than most
it's still a bit alarming though.
kellytk: i use iptables for the most part
i'd love to see some fear mongering about when we don't have el nino patterns
there's the non el nino pattern too
i know
but no fear mongering
hmm, apparently el nino may bring rain to california
SW US enjoys el nino because it means we get rain
and it means we have fewer forest fires
heh
winter here hasn't been neraly as wet or stormy as last year.
.w
i wonder what would happen if someone registered that softlayer domain
hm what was the trigger
@weather
mercutio: Fetching weather for your previous query (akl).
staticsafe: A greater oddity has arisen.  With firewalls disabled, one server correctly resolves `host update.freebsd.org` whereas the other server returns ";; connection timed out; no servers could be reached"
Auckland International, New Zealand: Mostly Cloudy ☁ 57°F (14°C), Humidity: 82%, Wind: From the West at 28 MPH -- For more details including the forecast and almanac, see http://www.wunderground.com/cgi-bin/findweather/getForecast?query=-37.00805664,174.79167175 or re-request this with: @weather -v
m0unds: i don't want to know enough to risk a lawsuit
well
staticsafe: lol
UDRP process maybe
maybe brycebot can register it
be a sport
eh even if somebody did, Softlayer has certainly enough money and lawyers to file a UDRP
which would be decided in their favour
"The selection and placement of stories on this page were determined automatically by a computer programme. "
does google news spell program as programme for other people too?
probably localization
or is it trying to use US vs UK spelling.
e.g. you're in a place where that might be normal
here it's normal to call computer programs, programs.
but if you have an event or something you may have a programme
would you otherwise say UK english is pretty typical?
yeah.
ok
What's sorta weird is news.google.com shows "program" even though my language is en-UK
Perhaps because this is the "U.S. edition"
i don't even remember seeing that statement before.
it says programme on news.google.co.uk
it says program on news.google.com w/en_us
It has said it for a long time
brycec: maybe it is IP geo-locating you
:P
i'm on www.google.co.nz with news tab
it says program on co.nz
for me
lol I switched edition to UK and now it's programme https://news.google.com/?edchanged=1&ned=uk&authuser=0
google is mysterious
brycec: probably just didn't notice
hahaha
i don't usually scroll down all the way
And when I switch it to France edition, it's all in french, including programmae
*programme
we can't profile you if we don't know where you are and what language you want
how do you switch editions
i tried appending &ned=us and it's still programme
"But Google, I care about the world and speak multiple languages!"
mercutio: there's a drop-down for me
based on your search history, we can confirm you're a liar
https://dl.dropboxusercontent.com/u/3167967/screenshot_2015-09-02_16-23-29.png
redirecting to pig latin edition
oh that looks totally different
i don't even have the top stories on the left
ahh goign to news.google.com is different
And just for completeness https://dl.dropboxusercontent.com/u/3167967/screenshot_2015-09-02_16-24-39.png
the american news is more disturbing
for some strange reason australia tells me about another china chemical explosion.
i wonder how they decide what's important for different regions.
Australia? like the whole entire country at once, shouting it across the sea? :D
australia google news dition
there's actually no china english edition
Whatever, I liked my mental image better.
australia just shouting about a chemical explosion
what would australia sound like?
not sure
@google site:youtube.com Australians shouting
2,490 total results returned for 'site:youtube.com Australians shouting', here's 3
A century worth shouting about. 100 years of the Royal Australian ... (http://www.youtube.com/redirect?event=stream_redirect&q=http%3A%2F%2Fwww.insidehistory.com.au%2F2013%2F10%2Fa-century-worth-shouting-about%2F&usg=VVFYdiLaFnMweikWVjKABUmaEh4=) Oct 3, 2013 ... Lindsey Shaw, formerly a Senior Curator at the Australian National Maritime  Museum, starts a series of four articles on the history of the Royal ...
Are you God? Crazy guy shouting on Australian Train - YouTube (http://www.youtube.com/watch?v=uq5DzvqJma0) Dec 19, 2013 ... Are you God? Crazy guy shouting on Australian Train. ... Are you God? Crazy guy  shouting on Australian Train. MCARDLEPRODUCTIONS.
Construction Workers Shouting Catcalls Women Can Appreciate ... (http://www.youtube.com/redirect?event=stream_redirect&q=http%3A%2F%2Fwallstreetinsanity.com%2Fconstruction-workers-shouting-catcalls-women-can-appreciate-video%2F&usg=4V43l-ajdmbsmA1yXc9ZQTPShKc=) Mar 27, 2014 ... Snickers has released a new ad in Australia that has good intentions, ... The  builders then shouted loud, empowering statements at the women ...
are you god video sounds like it might be a winner
well then
I am suitably amused.
Especially the part where they try and push him off/down
yeah
imagining him screaming about chemical explosions
Needs more female voices shouting too though
haha
Java based IPMIs make me sad
the ipmi isn't java based
it's the kvm that is java
yeah thats what I meant
you can use ipmitool and serial console to get around it
and you can reboot etc with ipmitool too
but yeah java isn't even supported in chrome anymore :(
and it never really seemed that great.
https://www.snellman.net/blog/archive/2015-09-01-the-most-obsolete-infrastructure-money-could-buy/
I figured out the Unbound resolution issue.  After removing the search domain, all became well.  It's an imperfect solution as I made use of the search domain feature however
Has anyone seen "Could not establish a chain of trust to keys for ntp.org. DNSKEY IN" in unbound.log?  In the course of research it seems to be possibly related to pf ruleset + UDP fragmentation, however my pf ruleset should handle frags with its `scrub fragment reassemble` option, so I'm confused
dnssec is probably going to occur over TCP
do you handle tcp fragmentation?
That's what she said!!
gizmoguy: http://pastie.org/private/imat8lhakzvxkt0fbytmla is my entire pf.conf
I don't believe I do
FWIW I'm using the FreeBSD pf port.  Can you suggest any improvements to my ruleset?
you shouldn't really have to handle fragmentation differently
That's what she said!!
also I can't say I've used pf before..
hold up
is ntp.org even signed?
no it's not
I don't know
I would suspect that's why DNSSEC to ntp.org fails
So that failure is normal?
maybe?
BryceBot: no
Oh, okay... I'm sorry. 'you shouldn't really have to handle fragmentation differently'
What is the purpose of that bot BTW?
gross packet loss
@last m0unds_
gizmoguy, I last saw m0unds_ 4 sec ago saying in a channel: gross packet loss.
can't even stay connected to my VM via ipv6
Oh
ipv6 is for losers
I've switched to IPv9.
block log quick inet6 all
mike-burns: how is v9? do your pakkitz travel at least 15% faster than the speed of light?
i run chimiak-enhanced-ipv4
they arrive before they were transmitted
Yes but that makes them very loud.
best ipv4
https://tools.ietf.org/html/draft-chimiak-enhanced-ipv4-00
hahaha
basically he removes some cruft from the ipv4 header and lets you use 64bit ipv4 addresses
for some reason it didn't take off
funny
ah yes, NTT return path shittiness
just saw 50% packet loss at s3, then my session died
sweet
gizmoguy: that sounds like a good idea
mercutio: anything going on w/ipv6?
m0unds: nothing diff from usual that i know about
i thought it was ntt being stupid, but i keep seeing packet loss at s3 incrementing, then my ssh session drops when it hits 50%
wow
i'm seeing something funky
with ntt too hah
wtf
it's not even all ntt, ..
hmm and i trace again and it's fine
yeah, it's fine right now
yeah i was tracing to www.kame.net
give it a minute, it'll get weird again
it's getting worse now
oh it's going funky again
yeah
haha
and it hits japan ok
then it hits another router in japan and starts dropping
me -> arp via ipv6 goes comcast -> he -> arp
in both directions?
nah, outbound to arp only
return is ntt
outbound to arp is worse
ntt is just regular old flaky ntt
i'm not well situated for ipv6 test sites atm
it seems like it's just v6 that's acting up though, for sure
vultr in sydney seems fine atm
because i'm still connected via v4
but i'll keep it going
and that goes level3, not ntt
bah late hops on vult just screwed up
and of course there's no reverse lookups and 12 hops...
hahaha
just hit...75% loss and dropped
toggling asn info isn't working
if you press z does it tell you asn's in mtr?
negative, it's not doing it
used to
damnit
it's working on my vm
maybe it only ever worked with ipv4
what version of mtr are you?
0.86
i have .86 on fbsd and .82 on deb
oh
on openbsd
hm
and 0.85 on linux
neither are working
it's working on freebsd but not debian on an rpi
weird
hahaha
it's showing loss from vultr in the same way
vultr mostly use ntt afaik
oh, -z isn't a flag on .82 that's why
2402:7800
i'm pressing inside the app
hmm 2402:7800 is vocus
so vultr's screwing up on vocus
before hitting arp even
vultr is vocus in both directions
not ntt
although i'm not sure what 2001:504:13::210:136 is
it's probably coresite though
coresite
yea
this is whack though
i'm mtr'ing in both directions, and one way is showing much more loss than the other
and my smokeping has been broken for 40 minutes too
wtf
40 minutes ago it got TERM signals
so i have no ipv4 smokepings to look at
but if i look at sydney's smokeping stuff to arp there was some loss a couple of hours ago
so there may be concurrent vocus and ntt issues
i'm struggling to determine any consistent patterns
it's only www.kame.net i saw the severe loss pattern too
oh another bind crash vulnerability
yup
did coresite die?
looks like the route changed, outbound from me to arp changed from he to ntt, and return path is still ntt
lol, he's lg at one wilshire looks awful
awful to arp or awful in general?
awful in general
hmm
yeah not sure what's happening tbh
800ms to me from lax @ coresite
hahaha
ouch
vs 35ms to me from equinix
pinging arpnetworks.com via coresite lg = 750ms
it seems a lot of disparate failures at once
yeah
so i'm wondering what the connection is
it may fibre cut
there was fibre cut in san francisco the other day
maybe there were more
i think it's up to like 13 in the last year?
of reported cuts around there
yea
but they seem to cluster a bit
oh well, weird as hell
time for planetside
it does make me think i should setup better ipv6 monitoring though :)
yea, i have just long interval ping monitoring via uptimerobot
Am I the only one getting horribly network activity?
i worded that badly
oh a quick skim of the backlog is ffffffasfl;jksadjkladljkasdjkl;sdjkl;asjkl;asjkl;asasdfjkasdfjkasdf[
it hung again ^
I'll have to get more info, but looks like I haven't been alone
brycec: i liked your mental image better too
Thanks. When a country can work together as one voice, it's always great.
Now, wtf is up with my connection???. I have too much shit to get done to debug this stuff.
https://smokeping.cobryce.com/?target=ARP shows some nasty IPv6 latency and spikes since 5pm
(inside ARP)
And it's really fucking with my SSH session.
I feel so dirty, connected to my VPS over IPv4
but hopefully it's smoother
(Hm an mtr I've left running for awhile from my VPS to an ipv6 host shows 3% packet loss starting at the second hop 2001:504:13::1a, that would be the first hop beyond ARP.
Aw I had 30 days connected to this Freenode server too, lost due to the network issues I was seeing.
aha
2001:504:13::1a is an Any2 IX peer
At this very second, it's dropping packets for me
Just started flowing
dropping
flowing
(that was 45 seconds dropping)
dropping
flowing after 36 seconds
dropping
brycec: mine was working via v4
v6 was terribad for a long while
flowing
(I also dropped 2 packets to ARP's router :O)
that's what it was doing for me too - it was bad when my v6 route was via he
(that was another 42 seconds of dropped packets)
but it seemed to change the last time i tracerouted and it was using ntt instead
dropping...
Wow
the coresite he lg was hosed - 900ms to itself, 900ms to arp, 900ms to other stuff
This is...
That's what she said!!
flowing
hahaha
52 seconds, and again 2 dropped @ ARP
dropping...
flowing, 52 seconds agin
this is cray cray
looks like it drops every 90 seconds or so for about 52 seconds
(I should point out that HE is involved in all directions and destinations to which I have access - I can't mtr from a non-HE address besides ARP)
Well it's not the cleanest way to share two mtr's but it works :P Issue is that he.net->ntt.net handoff it looks like https://dl.dropboxusercontent.com/u/3167967/screenshot_2015-09-02_21-35-05.png
Dear up_the_irons please to be fixing upstream's issue, kthx
brycec: there were issues with just ntt in both directions too
and there were issues with vocus/any2ix
brycec: did it come right?
Still craptastic
Dropped up to a few seconds even
*a few seconds ago
And there it goes dropping again
flowing again
(but it's not worth flooding the channel, and I have better things to do.)
got an ip address can trace to to reproduce?
2607:f2f8:a650::3
from arp i mean :)
2001:470:4:2a5::feed:dead
cool
that coresite hop having high pings suggests the router is under heavy cpu load
I'm happy to say in the last 60 seconds, I've only dropped 1 packet in mtr.
Agreed.
(I figure it will sort itself out soon enough)
aka "eventually"
yeah i was thinking that a couple of hours ago
even across any2ix direct it does that
knock on wood but it's looking more stable right now.
i'm seeing around 0.7% loss
11/500 packets dropped
that's like 2% loss
i have 3 out of 519 dropped
(% without context can be a bit hard to grasp. 50% of 2 packets vs 500 can indicate very different things :P)
yeah
can be different if they're all dropped in a row etc too
sounds better
Running Unbound, is there a reason why a fresh start up is often met with a random number of failures (0-~5) to resolve update.freebsd.org, but not google.com?  I suspect the former being signed and the latter not has something to do with it