#arpnetworks 2014-02-13,Thu

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***gizmoguy has quit IRC (Ping timeout: 245 seconds)
gizmoguy has joined #arpnetworks
Hien has quit IRC (Ping timeout: 264 seconds)
solj has quit IRC (Ping timeout: 264 seconds)
phlux has quit IRC (Ping timeout: 264 seconds)
staticsafe-znc has quit IRC (Ping timeout: 264 seconds)
SpeedBus has quit IRC (Ping timeout: 245 seconds)
staticsafe-znc has joined #arpnetworks
Hien has joined #arpnetworks
gizmoguy has quit IRC (Ping timeout: 264 seconds)
gizmoguy has joined #arpnetworks
solj has joined #arpnetworks
SpeedBus has joined #arpnetworks
phlux has joined #arpnetworks
SpeedBus has quit IRC (Ping timeout: 245 seconds)
SpeedBus has joined #arpnetworks
[00:10]
.... (idle for 19mn)
robonerd has quit IRC (Read error: Connection reset by peer)
robonerd has joined #arpnetworks
robonerd has quit IRC (Read error: Connection reset by peer)
robonerd has joined #arpnetworks
[00:46]
..... (idle for 20mn)
LT has joined #arpnetworks [01:12]
........ (idle for 36mn)
robonerd has quit IRC (Read error: Connection reset by peer)
robonerd has joined #arpnetworks
[01:48]
.......................... (idle for 2h6mn)
staticsafe-znc has quit IRC (Ping timeout: 265 seconds)
staticsafe has quit IRC (Ping timeout: 265 seconds)
xales has quit IRC (Ping timeout: 246 seconds)
staticsafe has joined #arpnetworks
staticsafe-znc has joined #arpnetworks
[03:55]
........... (idle for 51mn)
up_the_ironsmnathani: no i mean VPS customers [04:51]
....................... (idle for 1h54mn)
m0undsi tried it in a vps once, it works fine - alias configuration is a PITA vs freebsd or openbsd though [06:45]
......... (idle for 42mn)
***dj_goku_ has quit IRC (Ping timeout: 248 seconds) [07:27]
.... (idle for 15mn)
mjp_ has quit IRC (Ping timeout: 265 seconds)
abthorpet has joined #arpnetworks
mhoran1 has joined #arpnetworks
ChanServ sets mode: +o mhoran1
[FBI] has quit IRC (Ping timeout: 265 seconds)
[FBI] starts logging #arpnetworks at Thu Feb 13 07:44:23 2014
[FBI] has joined #arpnetworks
mhoran has quit IRC (Ping timeout: 265 seconds)
tabthorpe has quit IRC (Ping timeout: 265 seconds)
[07:42]
.............. (idle for 1h9mn)
brycecWhat do you mean? Clicking buttons in a GUI versus just typing it in pf.conf? 06:43:51 < m0unds> i tried it in a vps once, it works fine - alias configuration is a PITA vs freebsd or openbsd though
In which case, I'd say that writing rules is similarly PITA :p
[08:54]
m0undsbrycec: no, when i messed with it somewhat recently, interface aliases don't work the way i expected them to (the way they work in freebsd or openbsd) [09:01]
brycecDo you mean "interface groups" as they're termed in pfSense? [09:02]
m0undsthe UI element does some weird stuff with aliases that wasn't clear
i don't remember, but it felt kinda counterproductive
at any rate, i only looked because i hadn't used it since like 2007
[09:02]
brycecheh
Well I'm happy to say pfSense has come a long, long way in 7 years :p
[09:02]
m0undsand the thing that seemed like it should be alises wasn't
well, they're still on teh same release tree and the ui has some awful 90s UX to it that they need to get rid of asap :P
the notification thing at the top in teh default theme is awful
s/teh/the
[09:03]
BryceBot<m0unds> the notification thing at the top in the default theme is awful [09:03]
m0undsi looked at m0n0 last night because i hadn't used it since probably 2003, and they just moved up to freebsd 8.3 last month, haha [09:04]
brycecI don't think I follow what you mean by same release tree - were you expecting them to just scrap everything and start over?
Nice, they're catching up to pfSense
[09:04]
m0undsno, they were working on "2.0" forever [09:04]
brycecIIRC that 8.3 is due to the NanoBSD base.
True, but 1.2.3 was stable up until ~2 years ago
*stable/current
That 2.0 was effectively a total rewrite
[09:04]
m0undsyeah, so i guess it was 2 years ago that i messed with it, haha
because it was 1.2.3 (i still have the disk image on my kvm server at home)
[09:06]
brycec"Version 1.0 of the software was released on October 4, 2006.[5] Version 2.0 was released on September 17, 2011,[6] with updates 2.0.1 to 2.0.3 between then and 2013, and version 2.1 was released on September 15, 2013."
Or you can peruse http://www.pfsense.org/about-pfsense/versions.html
1.2.3 released Dec 2009, 2.0.1 released Dec 2011
at least they're somewhat consistent
[09:07]
Good news, up_the_irons, ARP was not on the list of the top 24 networks http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack
It didn't make the list at all! https://docs.google.com/spreadsheet/ccc?key=0AhuvvqAkGlindHFtS0pJa0lYZGNlLXNONWtlY01qanc&usp=sharing#gid=0
[09:17]
m0undsactually, i did try it more recently - i was trying to figure out how i knew about the thing at the top. i tried it on my small arp vps after i migrated stuff to the bigger arp vps
i do have a 1.2.3 disk image locally though
which i don't remember doing anything with
at any rate, maybe it was a quirk with the vps having a single if or something, or i could just be remembering something that doesn't exist, hahaha
haha, chinanet. go figure.
[09:30]
brycecsrsly
and ovh
[09:33]
m0undsyeah
color me surprised
[09:33]
brycecbrycec colours in m0unds [09:33]
m0undswhat color is surprised? [09:33]
***LT has quit IRC (Quit: Leaving) [09:34]
brycecm0unds: mauve [09:35]
m0undseeewwwww
at least it's not taupe
or chartreuse
[09:35]
brycecOh sweet, I'm getting hit with buttloads of DNS queries... [09:35]
m0undsm0unds throws DNS queries at brycec [09:35]
brycecI'm guessing my own DOS reflection
an attack of my very own. On a cable modem. woo
[09:36]
m0undssweet [09:36]
brycecI wonder how this IP was choosen... I don't really use it for anything, and it's not in DNS anywhere [09:36]
m0undsare you connected to IRC w/it?
or bot connected from it or something?
the only time i ever had a box get ddosed was when i was connected to a provider's IRC channel w/v4 address and some kid decided to start ddosing people connected to the channel
[09:37]
brycecnope and nope
It's the external IP of a router, but all traffic goes out a different IP
[09:37]
m0undshuh.
is the address SWIP'd to another business or something?
[09:38]
brycecdun think so [09:38]
m0undsyou must just be lucky [09:39]
brycecholy crap, TWC actually filled in the business information
(protip: always smart to whois your own addresses once in awhile)
ANY? .
that's the query
whee
brycec wonders who to complain at
[09:39]
hazardousthis is probably odd question but does arp ratelimit dns or anything [09:45]
brycecUDP inbound as I recall
to 5mbps
[09:45]
hazardous$ time host lightning.net >/dev/null
real 0m8.197s
$ time host nac.net >/dev/null
real 0m2.327s
[09:46]
brycecouch
real 0m0.946s
[09:46]
hazardoususing the official arp resolvers, and i don't get this kind of response time elsewhere even hitting the same authoritative ns
close to 1 second seems incredibly weird/bad as is
[09:46]
brycechazardous: for google.com real 0m0.161s [09:47]
hazardousyeah that seems cached maybe, idk [09:47]
brycecAnd nac.net now gives me real 0m0.027s [09:47]
hazardousi sometimes have periods of time when it's somewhat acceptable and periods of time where anything takes forever [09:47]
brycechttp://support.arpnetworks.com/kb/main/is-there-a-firewall-filter-rate-limit-or-similar-device-applied-to-my-traffic
outbound UDP traffic is rate-limited to 5mbps
I wonder if up_the_irons / ARP's resolvers are exempt from that, and/or being overwhelming it
wow that was terrible grammar
[09:48]
m0undstry again
hahah
[09:49]
hazardousyeah i dunno, i just did host google.com and the A records returned instantly
then it hung for 3-4 seconds before returning the mx part
[09:49]
m0undshuh.
was just gonna say maybe it wasn't cached but if it's hanging on mx recs, dunno
the prev query, i mean
since it returned google quickly
[09:50]
hazardousyeah i have absolutely no idea
http://pastebin.aquilenet.fr/?1328915b0bfb2488#y8PHFMpIH159tKAd5vFItNbggiBNrLCLq4fJmGtx7oE=
wat
[09:52]
brycecoh man this dns ddos is terrible, how will i deal with the 50kbps of traffic pouring in???
Silliest. DOS. Ever.
[10:05]
...... (idle for 26mn)
m0undsomgz, you better mitigate it [10:33]
..... (idle for 24mn)
up_the_ironsbrycec: <phew> ARP wasn't in the list ;) [10:57]
mnathaniWhat is the purpose of running pfSense (a firewall) on a VPS with only one interface? [11:01]
m0undsm0unds shrugs
i did it to screw around; you can still use it as a gateway or firewall in front of another vps if you really wanted to
you don't /have/ to have separate interfaces unless you're gonna nat
[11:02]
brycecAnd even then, there are vlans ;) (or vlans inside of vlans, on ARP) [11:16]
m0undsyes [11:16]
.......... (idle for 47mn)
toddfvlans inside of vlans on ARP only work if you use svlan(4) (IEEE 802.1AD)
and then only if you're unblocked and permitted to do so
I know from experience!
now don't try to do svlan(4) on vio(4) on current, something about cksums kills it
current openbsd that is
[12:03]
.......... (idle for 49mn)
mnathaniDoes anyone know how much it costs to Akamaize a website? (Serve it using Akaamai's CDN) ? [12:53]
m0undsi've never seen $ amounts, but i've also never heard 'affordable' mentioned alongside their name
might look into edgecast or cachefly too
[12:58]
mnathaniOfcourse it would depend on the size of the site / complexity beign served as well as actual traffic / bandwidth
s/beign/being
[12:59]
BryceBot<mnathani> Ofcourse it would depend on the size of the site / complexity being served as well as actual traffic / bandwidth [13:00]
m0undsi googled and found pricing from an akamai partner showing a $4/GB x 500GB/mo commit
and that was 'cheap'
[13:01]
mnathaniso approx $2000/month for that setup? [13:02]
m0undsyea
later clarification showed $200/mo @ per GB
and $375/mo for 500GB
http://www.cachefly.com/pricing.html
[13:02]
mnathanim0unds: thans
s/thans/thanks
[13:11]
BryceBot<mnathani> m0unds: thanks [13:11]
RandalSchwartzcloudflare.com is free if you don't need SSL [13:15]
m0undsyeah, cloudflare
's free stuff is good if you just want the cdn features
[13:21]
toddfindeed it does sound good [13:22]
......... (idle for 40mn)
up_the_ironsi'm not sure what i like better
allow(Mail).to receive(:all).and_return(@msgs)
allow(Mail).to receive(:all) { @msgs }
[14:02]
staticsafehttp://www.pantz.org/software/pf/use_freebsd_10_as_a_pf_firewall.html [14:03]
mike-burnsI like the #and_return. It's more clear what it's doing. [14:05]
mercutiom0unds: i prefer cahefly to akamai as a user [14:06]
mike-burnsI save the block syntax for when I really need a block. [14:06]
mercutioakamai is often terrible performance with "cache misses"
akamai has closer nodes to me than cachefly, but cachefly's average performance is way up.
[14:06]
up_the_ironsmike-burns: ah [14:07]
mercutioit's harder to test cache miss performance though. you used to be able to send ?1 ?2 ?3 ?4 etc to get an uncached version of stuff to test... but that seemed to stop working
and with the number of akamai nodes, unless you're huge there are likely to be lots of cache misses
cloudflrae is terrible i reckon
[14:07]
up_the_ironsCloudFlare should get on Any2 so I can peer with them
Akamai is, so is EdgeCast
[14:14]
mercutioi don't think it'd make meuch difference?
cloud flare is in san jose atm isn't it?
i can't think of any domains that use it off hand
[14:14]
up_the_ironsNO, peering with ARP makes ALL THE DIFFERENCE IN THE WORLD. Get it right mercutio ;) [14:15]
mercutiowell i doubt there's much traffic being pushed to there, and with multiple upstreams ... incoming ins't likely to saturate
and hardly any users on arp are likely to be pulling large files off cloudflare
[14:15]
up_the_ironsbut cloudflare pulls lots of files from arp [14:16]
mercutiooh
ok i didn't realise that
[14:16]
up_the_irons:)
some dedi customers use CF heavily
[14:16]
RandalSchwartzmercutio - anyone hosted at (mt) is using cloudflare by default [14:17]
up_the_ironsyeah, shared hosting co's are starting to do that
we have some here
[14:17]
mercutiomt? [14:18]
up_the_ironsmedia temple [14:19]
RandalSchwartz(mt)® is Media Temple. Not sure what "mt" is. :) [14:21]
brycecme either. But I'm pretty sure a RandalSchwartz is a smartass :P [14:22]
mercutioi've only noticed cloudflare when sites have problems
i suppose that's one of the problems with those things
[14:22]
brycecditto [14:22]
mercutiosome sites only shift to cloudflare when they're getting ddos'ed
or have load issues
and if a site's going slow and you trace and it says cloudflare..
[14:22]
RandalSchwartzinsightcruises.com had it enabled for a week, then they screwed up the DNS, and different people were getting different pages or even A records pointing to nowhere [14:23]
staticsafeO_o [14:24]
RandalSchwartzso we ripped it all out, and haven't gone back
might have been early growing pains
[14:24]
mercutiowho's using it right now? [14:24]
RandalSchwartzpresume these guys: https://www.cloudflare.com/case-studies [14:25]
up_the_ironsRandalSchwartz: you coming to any js.la meetups soon? it's kinda hard to believe the last one where we had a chance to chat was over a year ago! (christmas before last) [14:25]
mercutioimgur will do [14:25]
up_the_ironsRandalSchwartz: or perhaps that was devops.la... hmmm
*ladevops
[14:26]
RandalSchwartzinteresting - http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack [14:26]
up_the_ironsup_the_irons brain is dying
RandalSchwartz: that was pasted earlier
[14:26]
RandalSchwartzmy client before this one was directly across the street from my hotel and a bar. I stopped going to meetups. :)
now I'm working in santa monica, and I have to actually drive around, so I can start doing meetups again.
plus, it'll give me an excuse to drink a bit less. I was a bit out of control because I wasn't driving for 5 days at a time.
[14:27]
staticsafeo_o [14:27]
mercutiooh cloudflare do have node in los angeles [14:28]
staticsafethey would be stupid not to [14:28]
mercutioand the ?1 ?2 trick works [14:28]
m0undsthey've got lots of pops [14:30]
mercutiocache hit time in la, 20 msec, cache miss time 500 msec
cache hit time in nz, 85 msec, cache miss time 530 msec
[14:30]
brycecnot bad
pretty consistent anyways
[14:31]
mercutioyeh i seen way worse from akamai
with > 1 second.
[14:31]
m0undsgross [14:31]
mercutioit's only 15k file though
it seemed to get more consistent when i did it more
[14:31]
m0undsmy friend put their blog on the free cloudflare plan and saw spam comments just stop altogether - i think just as a side effect of cloudflare's anti-bot stuff [14:32]
mercutioimgur still loaded images slow
main imgur.com site isn't cdn'ed, jsut their images
[14:32]
m0undsright
img.imgur.com
[14:33]
mercutionah i.imgur.com [14:33]
m0undsimg.imgur.com is too [14:33]
mercutiooh ok [14:33]
staticsafeimgur is quite reliable
imgur++
[14:33]
mercutioi don't normaly look at it
http://imgur.com/gallery/oxn6ZMh
but this is funny
[14:34]
staticsafeheh [14:34]
mercutiodamn now i want to test akamai
but they using https
akamai is 56 msec hit, 1 second miss for apple 56k image
and 17 msec hit, 260 msec miss from la
i suppose that not so bad, i suppose it depends wehre origin servers are
still even for 56k 1 second miss seems kind of bad to me
[14:37]
mnathaniJust setup my first site to site vpn in a lab, but I can't seem to ping from a subnet behind the vpn to a subnet on the other site. [14:53]
mercutiodid you enable forwarding? [14:53]
mnathaniThe reverse ping works just fine. (pfSense with Openvpn btw) [14:53]
brycecDid you set the routing?
Non-overlapping subnets?
Firewall?
[14:53]
mnathanifirewall - allow any/any
non-overlapping subnets yup
[14:54]
brycectraceroute, tcpdump [14:54]
mnathanithe server config section had local subnets and remote subnets [14:54]
mercutioi assume pfsense would enable forwarding :) [14:54]
mnathaniclient only had remote subnets
client side can ping server side, but not reverse
how foolish of me, twice in as many days now, Windows Firewall got the better of me and dropped my icmp packets
[14:54]
mercutiooh that's normal [15:00]
bryceclolol [15:03]
mnathaninow I need to redo it to verify if all that config I did was necessary [15:03]
brycec@mnathani [15:04]
BryceBotHave you checked whether Windows Firewall is enabled and dropping packets? [15:04]
mnathaniadding extra route statement etc [15:04]
BryceBot425 results found. Here's #265 Dec 19, 2013 16:43:30 <mnathani > requests are going through, but not being cached I dont think [15:04]
mnathanilol BryceBot / brycec [15:05]
brycecYou know what they say, if you use something 3 times, you sohuld buy it/script it.
Well, this was only twice but I'm proactive ;)
[15:05]
RandalSchwartzUgh - had some beans and rice (normally skipped) with my mexican dish for lunch
now I'm in a carb crash fog :(
forgotten how bad these are
[15:10]
up_the_ironsRandalSchwartz: yeah they suck [15:11]
m0undsyeah, rice tends to knock me on my ass too
i think i'm gonna adjust my pf ruleset a bit - i got lazy and didn't define macros or anything so the rules are ugly as hell
[15:11]
toddfwhen done.. var=..nvar=..nntable..ntable..nnmatch..nmatch..nnblock..npass..n done! ;-) [15:15]
mercutioy'll are lazy
oh i don't quite know how the y thing works
i don't use macros with pf either
[15:19]
brycecy'all are lazy'
well you get the idea I think... You need 2 or 3 separators
it's a regex like any other
[15:20]
mercutiooh [15:21]
RandalSchwartzugh - this office has horrible "free" stuff in the kitchen. either sweetened beverages ("vitamin water", ugh), or decaffinated teas.
and all sorts of "hearthealthywholegrain" snacks
it's sad how much I've found out that this stuff is all crap in the past 18 months
[15:22]
brycec(well not exactly a regex... it's a PCRE verb, with sed expression syntax) [15:23]
RandalSchwartzPCRE is a misnomer :) [15:23]
m0undsi like macros because it keeps things readable - granted, i'm likely the only one who will ever see it, it's still nice to keep it concise
i do the same w/network hw configs on gear that actually supports macro type functionality (junos)
[15:26]
***m0unds has quit IRC (Quit: reloading) [15:39]
mercutiohmm cloudflare aren't on coresite [15:40]
***mnathani_web has joined #arpnetworks [15:41]
mercutioso best hope is for bgp collective to spread to equinix [15:41]
mnathani_webare there network issues atm
I cant ping arpnetworks.com or my VPS
[15:42]
mercutioweird
traceroute?
[15:42]
mnathani_websec. [15:43]
toddfI cannot ping arpnetworks.com [15:43]
mnathani_webhttp://pastebin.com/wZehQBW7
here is mtr
[15:43]
toddfI can ping 2.v.freedaemon.com (which is on the 1gbps ports) [15:43]
mercutioblame nlayer? [15:44]
mnathani_webtraceroute is taking a while [15:44]
meingtslai can't resolve arpnetworks.com, weird
oh wait yes i can
[15:44]
mercutio208.79.89.243
i'm on arpnewtorks fine if anyone wants a traceroute in other direciton
vl5.s1.lax.arpnetworks.com is giving packet loss
[15:44]
toddfarpnetworks v6 is working, arpnetworks v4 is not for ns1 and ns2.arpnetworks.com [15:44]
meingtslaah [15:45]
toddfjust started responding [15:45]
mercutioge0-arpnet.cust.lax07.mzima.net gives packet lsos too
i wonder what happend
[15:45]
mnathani_webits back now [15:46]
mercutiooh
i never lost connectivity :)
[15:46]
RandalSchwartzlsos! [15:47]
brycecAlso showing IPv4 loss from INSIDE ARP https://smokeping.cobryce.com/?target=ARP.ThisGW
more accurately, packet loss on my own vlan to the router
guessing router crash?
[15:48]
mercutionice
but not for me
are you on gigabit?
[15:49]
brycecDon't think so? [15:49]
mnathani_webcould it be BIRD related? [15:49]
mercutioi'm not pinging the link address
don't think so
hardly anyone goes over any2ix on the new range
i think it's proably nlayer
[15:49]
brycecfwiw I can ping arpnetworks.com and my VPS both. As far as I can tell from smokeping, there _was_ a brief outage but it's back up now. [15:50]
mercutioactually why would link address stop working then [15:50]
bryceconly likely cause, something unplugged or reset [15:51]
mercutiowow you have a lot of sites in smokeping :) [15:53]
brycecAbout 250 probes [15:54]
mercutioi like the overlayed thing
how much bandwidth does that use?
[15:54]
brycec212 hosts to be precise
overlayed... When there's a whole bunch of hosts on one graph? like for the IRC networks?
[15:54]
mercutioyeah [15:55]
brycecbrycec asks graphs.arpnetworks.com how much b/w is used
A surprising amount.
Reportedly 6.71GB in the last 24 horus
[15:55]
mercutio20 ping every 60 seconds [15:56]
staticsafeall that ICMP [15:56]
mercutio212 sites [15:56]
brycecmercutio: 500byte packet size too [15:57]
staticsafebrycec: would you mind sharing your hosts file for smokeping? [15:57]
mercutioahy are you doing 500 byte packet size
i was going to ask that but he has some local stuff in it too
[15:57]
brycecmercutio: Because it's better than the default 5000 :P [15:57]
mercutiofrguly isn't 500
err 5000 it's like 72
[15:57]
brycecAccording to the docs it's 5000 [15:58]
mercutiowhere [15:58]
brycechttp://oss.oetiker.ch/smokeping/doc/smokeping_config.en.html [15:58]
mercutiooh wtf
that's example value
which isnt' the default, but why are they suggesting that
[15:58]
brycecer http://oss.oetiker.ch/smokeping/probe/FPing.en.html [15:58]
mercutioi'd suggest trying 32
and see if graphs look bsaically the same
[15:59]
brycecYou have in the past :P [15:59]
mercutioand identify issues just the same [15:59]
brycecAnd if the docs don't say "default" but do say "example" what am I to assume? [15:59]
mercutiowell i thought i'd do it again
ikr
[15:59]
brycecThey aren't the world's best docs :/
Change made
just in time for 4pm on the dot
[16:00]
mercutiohmm.. up_the_irons have you considered making graphs of transit links visible to users? [16:02]
brycecAh, 56 is the default. RTFS :/ [16:02]
mercutiohmm
i wonder why i saw 72 then
oih maybe it was 76
i htink 56 doesn't include the IP header size of 20 bytes
[16:02]
brycecprobably
and actually it leaves it up to fping now that I read right
[16:03]
mercutiojust set it to 32 :) [16:03]
brycec"Default is 56, as in ping."
mercutio: 15:58:49 < brycec> Change made
[16:03]
mercutioless load on the network
oh you did
[16:03]
brycecBut I like chewing up ~7GB/day [16:03]
mercutioheh [16:04]
brycec~2MB/minute, from 5 monitoring hosts [16:04]
mercutiothing is it's every destination network
and the more people use larger packets etc the more people think it's a good idea to block icmp
which is annyoing
[16:04]
brycecI thought I'd dialed back the default to 10%, I was a Good Guy :p
dumb docs
[16:05]
staticsafesome of the nlnog ring nodes seem to have stopped responding [16:05]
brycecstaticsafe: if you're talking about my graphs, some never started responding :P (hurricane, I'm looking at you...) [16:05]
staticsafeah [16:05]
brycecI wanted to give them time to fix themselves :p [16:06]
***m0unds has joined #arpnetworks [16:06]
mercutioyour smokeping responds so slowly
i dunno if it's cos it's swapping, or because you have so many hosts
[16:06]
brycecwhat do you mean? [16:06]
mercutiolike clicking things is slow [16:06]
brycecAnd no swapping [16:07]
mercutiooh maybe cos it's https [16:07]
staticsafeI like the idea of the NLNOG ring [16:07]
brycecme too
(mmm first page load after I restar the fastcgi, now THAT is slow)
[16:07]
mercutiomine are slower than they could be on arp cos not using fastcgi [16:09]
m0undsbrycec: it was pretty quick for me via comcast fwiw [16:10]
mercutioit seems your pages take ~2.5 seconds to generate [16:10]
brycecThanks m0unds [16:10]
m0undshalf a second or less for me [16:10]
mercutioit was 2.6 seconds from arp with curl
weird
time curl -v 'https://smokeping.cobryce.com/?target=Internet.NLNOGRING' > /dev/null
is what i was doing
it's a bit up and down, now it's 1.4 seconds from arp
[16:10]
brycec2.47 to generate for me [16:11]
mercutiofrom arp? [16:11]
m0unds.947s for me from comcast in nm [16:11]
brycecimages only 250ms each [16:11]
m0undshaha [16:11]
brycecmercutio: from my desk [16:11]
mercutioi just got a bad gateway error [16:11]
brycecdamnit who broke smokeping [16:11]
mercutiofrom nginx [16:11]
m0undsahahahahah [16:11]
brycecSometimes the fastcgi crashes... [16:11]
mercutiohmm mayb eit is cos too many people using it [16:11]
m0undswomp womp [16:11]
brycec(And yes from nginx, because nginx is servingup the fastcgi) [16:12]
mercutiohow many fastcgi processes does it run? [16:12]
m0unds1100 [16:12]
brycecmercutio: Just the one process, because just the one cgi
Unless you're asking total on the system... In which case still just one.
[16:12]
mercutioi think i have two
yeh two running as www-data
and one running as smpokeping
i wonder if that slows it down more for me
cos i so far away
oh images don't go via smokeping
[16:13]
brycec(graphs is showing a reduction in b/w woo)
Yes. Those images are built on page load
[16:14]
up_the_ironsmercutio: not considered it [16:17]
mercutioi understand there may be some reluctance to
up_the_irons: any idea what happened aerlier?
[16:17]
up_the_ironsbrycec: on v4 it would not be a router crash
s1.lax#sh ver
...
s1.lax uptime is 4 years, 51 weeks, 4 days, 4 hours, 33 minutes
[16:18]
m0undslol [16:19]
up_the_ironsmercutio: no idea [16:19]
m0undswhat chassis is that one? [16:19]
up_the_irons4500 [16:19]
m0undscool [16:19]
staticsafedat uptime [16:21]
m0undsmy vss pair (2x6509E) is at 3y49w2d5h37m [16:21]
up_the_ironsnice :) [16:22]
m0undsyup, pretty solid
our IS dept did a core upgrade to redundant 7009s - those have been horrible
[16:22]
mercutioi think the longer you deal with IT the more you decide that everything is terrible [16:23]
m0unds15 years in and i can't say i have that opinion at all [16:23]
mercutioreally? [16:23]
m0undssome stuff is really bad, some stuff isn't as bad, some stuff is good [16:23]
up_the_ironsyeah me either [16:23]
mercutiosmtp, spam [16:23]
up_the_ironsi've found some stuff to be very good [16:24]
mercutiooh right, but the conditions
get worse and worse
bugs get more obscure etc
i suppsoe the problem with things like routers is traffic volume goes up and up
[16:24]
m0undsright [16:25]
mercutioif 100 gigabit connectinos were standard it wouldn't be such an issue
but tehy're new, and require new investment etc
and heaps of "background" stuff to make it work
i suppose one advantage of bandwidth going up over time is it's going to get harder and harder to sniff traffic
[16:25]
brycecHeh, exactly at 4pm, my usage drops considerably https://dl.dropboxusercontent.com/u/3167967/screenshot_2014-02-13_16-26-15.png [16:28]
mercutiodo the graphs look the same? [16:28]
brycecbrycec hasn't looked
probably, yes
[16:28]
mercutioyeh basically [16:29]
brycecI don't have side-by-side 500vs32, but seems right [16:29]
mercutioalthough you'd havd a few outages
to this tunnel
[16:29]
brycechuh?
English please.
[16:29]
mercutiother's gaps in your grpahs [16:29]
brycecyes when I restart smokeping
Well 15:39 was the ARP outage
[16:29]
mercutioi don't think it's that
https://smokeping.cobryce.com/images/Internet/NLNOGRING/doruknet01ringnlnognetv4~vps1_last_3600.png
it looks like what smokeping does when it can't keep up
[16:30]
brycecThat's chunkhost though... those have always been shitty [16:30]
mercutiooh
it doesn't say loss though
[16:30]
brycecI know [16:30]
mercutioare you doing dns lookups? [16:30]
brycecFor whatever reason, that slave just doesn't keep up
some
[16:30]
mercutiosmokeping can have issues with graphs if it doesn't do all the hosts in the right time period [16:31]
brycecYeah I know [16:31]
mercutioand i've seen it happen due to partial issues with connectivity to some hosts, making dns timeout before [16:31]
brycecbrycec wishes smokeping bad better, or any logging [16:31]
mercutioit has logging
it tells you hwen it can't keep up
[16:31]
brycecI only see logs when I start/stop [16:32]
mercutioweird
are you using rrdcached?
i was just checking my logs and saw some stuff from rrdcached
[16:32]
brycecLast I heard, smokeping doesn't support rrdcached [16:33]
staticsafeyea it doesn't [16:33]
mercutiohmm maybe i was using that for cacti [16:34]
brycec(I use rrdcached for munin stuffs though, but that's isolated) [16:34]
staticsafestaticsafe needs to completely redo his smokeping setup
i got rid of munin entirely
I just use zabbix now
[16:34]
brycecAh that's why you wanted my configs
staticsafe: there's nothing special or unique to my Targets config, it's just long
[16:34]
mercutioi'd like to see a standard set of test sites myself [16:34]
staticsafebrycec: yea, just want it to because I'm lazy :P [16:35]
brycecsure I'll sanitize and toss it on sprunge.us [16:35]
staticsafety [16:35]
brycecmercutio: that's my dream with nlnog
That I can use them as standard test sites
[16:35]
mercutioahh [16:36]
staticsafebrycec: my config isn't even monitoring v6 atm [16:36]
mercutiowell i hvae google, gmail facebook twitter slashdot, nytimes, anandtech bbc, guardian godaddy, ubuntu archive wikipedia, [16:36]
staticsafewhich is unacceptable [16:36]
mercutiowhich are some goodish sites to test
but like guardian hops around
ubuntu archive always give packet loss when there is a new release
and hmm i didn't comma right at all
so it can make sense to do from multipel sources, to check whether it's the destination site or an in general thing
[16:36]
brycecstaticsafe: As requested http://sprunge.us/ZOhN [16:38]
staticsafebrycec: ty [16:38]
brycecIt needs cleanup... I was inebriated at one point and forgot how Config::Grammar inherits [16:38]
staticsafeheh
config inheritance is my favorite thing
nagios...
[16:38]
brycecbrycec was thinking fall-through, hence calling SlavesV4/6 over and over and over
(but it's actually parent/child inheritance)
[16:39]
staticsafe# Boy this is getting annoying.
hehe
[16:40]
brycecit really was! [16:40]
mercutioheh
so basically you're scraping
[16:42]
brycecscraping?
referring to my regex?
[16:42]
mercutioto get the destination sites [16:42]
brycecAs opposed to inventing sites? :P [16:43]
mercutioor did you manually do mirrors?
yeh
there's a lot of text
[16:43]
brycecThe scraping was not automated [16:43]
mercutiolooks like you're relying on dns too [16:43]
staticsafestaticsafe dumps all existing smokeping data [16:43]
mercutioi wnoder if fpign can have a cache for dns
and use prior data if it cant' do a lookup
[16:44]
brycecmercutio: I'd copy from a webpage into my text editor, apply the regex to mold into target configs, and paste that into Targets [16:44]
staticsafemercutio: run a local resolver? [16:44]
brycecmercutio: Yes, some sites don't have fixed IP's. [16:44]
mercutiostaticsafe: you could still hit expiring ttls [16:44]
staticsafeyep [16:44]
mercutiounless you use unbound with prefetch hmm [16:44]
staticsafewhich is fine [16:44]
brycec(or are known to change their IP from time to time) [16:44]
mercutioyou could still hit it though
static: not if the site goes down
the problem is you don't want to fping to wait forever for dns
s/to//
[16:44]
BryceBot<mercutio> the problem is you don't want fping wait forever for dns [16:45]
mercutiooh
i just wanted to kill the first one
[16:45]
brycectoo bad :P [16:46]
mercutioso fping can lose all the results
does it assume /g ?
[16:46]
brycecyes
Because that's what PHP's preg_replace does
[16:46]
staticsafelooking at my traffic graphs and my machines are so idle :( [16:47]
mercutioheh
you should host wikileaks mirror
[16:49]
brycecIs wikileaks still a "thing"? :p [16:50]
mercutiono idea [16:50]
staticsafemercutio: or a tor relay :P [16:50]
mercutiogah no not tor
argh
[16:50]
brycecup_the_irons has stated that he drops you at the first abuse report [16:51]
mercutioweird my vm just spiked to 20kbit bnadwidth [16:51]
brycecTor seems liek an unnecessary risk [16:51]
mercutioit looks funyn on the graph [16:51]
staticsafebrycec: only exits get abuse reports
not relays
[16:51]
mercutiobut then 20kbit isn't much [16:51]
brycectrue [16:51]
staticsafebut i wouldn't run it on ARP anyways [16:51]
mercutioi average 2kbit/sec inbound
and 1.68kbit/sec outbound
[16:52]
staticsafeseed some linux ISOs too I guess [16:54]
mercutioi suspect most vm users don't use much bandwidth [16:54]
staticsafeagreed [16:55]
mercutiootherwise the ntp thing wouldn't ahve been as obvious
it's probably like 10% of the users use 90% of the bandwidth kind of thing
[16:55]
staticsafestaticsafe nods [16:56]
mnathaniAnother pfSense issue, I have 4 interfaces: WAN, LAN, OPT1 and OPT2
LAN and OPT1 can ping 8.8.8.8
but OPT2 can not
identical firewall rules
OPT2 can ping LAN and OPT1 though
[16:56]
brycec@mnathani [16:57]
BryceBotmnathani: Have you checked whether Windows Firewall is enabled and dropping packets?
432 results found. Here's #204 Dec 06, 2013 21:22:38 <mnathani > Anyone know of a method to Auto-BCC a copy of all outgoing mail to a specific address from within the Gmail Web Interface?
[16:57]
mnathanipacket capture shows pings leaving, but no reply
lol dont think 8.8.8.8 is behind a windows firewall
lol though
[16:57]
brycecSuuuure
opt1 and opt2 setup identically? routable subnets?
[16:58]
mnathaniyup [16:58]
brycecAnd just for kicks, swap (assign) them and see what happens [16:58]
mnathaniafter swapping, the vm that could ping can no longer ping and the vm that could not ping can ping now [17:01]
brycecGood
Just wanted to confirm it wasn't something besides configuration
and that the vm's were confi'd right
[17:02]
mnathaniwould you reset pfsense interface assignments at this point [17:02]
brycecIf it didn't matter, I'd leave that and focus on the issue [17:03]
mnathanifound it
it had a gateway assigned to it that needed to be removed
from within the pfsense interface
[17:07]
brycecawesome
lol why would you set the gateway on the internal interface? :p
[17:07]
mnathani"It made sense at the time" ... [17:09]
............................ (idle for 2h15mn)
***dj_goku has joined #arpnetworks [19:24]
.... (idle for 15mn)
dj_goku has quit IRC (Ping timeout: 260 seconds)
dj_goku has joined #arpnetworks
[19:39]
jcv has quit IRC (Ping timeout: 265 seconds)
jcv has joined #arpnetworks
[19:50]
......... (idle for 40mn)
brycec(Oh good, I just checked and the attempted DNS DOS on me subsided :D) [20:30]
m0undswhew, that was close
http://1drv.ms/1c4utnA

i had a live wasp crawling its way across the floor in my server room
[20:30]
brycecwasps are a "NOPE!" for me
practically send me running
[20:33]
m0undshaha, i captured it in a cup and took it outside
this guy was super lethargic because of how cold it is in that room
[20:34]
brycecoh shit... I just discovered my carp backup's ntpd is open. damnit damnit. [20:37]
m0undsYOU'RE PART OF THE PROBLEM [20:38]
brycecnot according to monlist
at least, a very tiny part
[20:38]
m0undshaha
someone in another channel said they had a supermicro board w/ntpd + monlist running on its ipmi interface
[20:38]
brycecbrycec decides it's easier to firewall rather than edit the conf
lolol
[20:39]
mercutiobrycec: you mean it tells you the time? [20:44]
brycecI mean it lists its recent peers, but it's a short list [20:46]
mercutiooh right
all the open to monlist hosts got blocked
even the ones with shorter lists
[20:46]
brycecI'm talking about a non-ARP system [20:47]
mercutioyeh on arp i mean [20:47]
m0unds(we were here yesterday when up_the_irons said he was gonna do it) [20:47]
mercutioi dunno if people trying to ddos differntiate
oh right
was that eysterday
[20:47]
m0undsyea, or the day before
the days all sort of blur together
[20:48]
brycecbrycec shakes his fist at pfSense [20:48]
mercutioopenntpd hasn't taken off as much as openssh did [20:49]
brycec(pfSense has "pass in quick" rules to explicitly allow NTP access on all interfaces at the top :( grr) [20:50]
m0undswhoa, really? [20:51]
brycecyep
pfctl -sr http://sprunge.us/Vijb
Ahahaha http://translate.google.com/#auto/en/%E8%87%AA%E5%8B%95%E3%82%A2%E3%82%AF%E3%82%BB%E3%82%B9%E3%81%AE%E5%A0%B4%E5%90%88%E3%81%AF%E3%80%81%E9%9B%BB%E5%AD%90%E3%83%A1%E3%83%BC%E3%83%AB%E3%82%92%20ntp-scan%40puck.nether.net%20%E3%81%B8%E3%81%8F%E3%81%A0%E3%81%95%E3%81%84
"For automatic access, please fart ntp-scan@puck.nether.net e-mail"
[20:51]
Reviewing some tcpdumps, I see people trying to use my (correctly configured) router to NTP DDOS. Yay for it being setup properly at least
wow, quite a bit in fact
[21:01]
mnathaniwhat can you do using the command line / ssh with pfSense that can't be done from the web interface? [21:02]
brycecdd? :P
well even that can be done from the web
On account of Diagnostics->Command
mnathani: I find it easier to dump pfctl info, run tcpdumps and other diag tools
[21:02]
mnathaniI guess I meant in terms of managing / configuring the firewall [21:03]
brycecI leave all management and configuration to the web ui, unless I lock myself out. Since all changes get made to an XML config, best to leave it in the capable, tested, properly-formatting hands of the GUI
Though I suppose watching pflog could count as managing
[21:04]
mnathaniwhat distro is it easiest to setup / resolve dependencies of smokeping? [21:07]
brycecmnathani: Debian, I'd say
At least for slaves...
I just launch an VZ container, apt-get install --no-install-recommends smokeping ; service smokeping stop ; $EDITOR /etc/default/smokeping /etc/smokeping/secrets ; set permissions ; service start smokeping
19... 1000 packets dumped, 19 different "sources"
brycec takes a larger sample
[21:08]
mnathanibrycec: thanks [21:14]
brycecnp
also, wtf my VPS load spiked to 33
christ on a cracker
[21:15]
mnathani00:20:50 up 6 days, 8:50, 1 user, load average: 0.00, 0.00, 0.00
00:21:09 up 17 days, 0 min, 1 user, load average: 0.08, 0.10, 0.07
my 2 ARP VPSen
[21:22]
brycec@uptime host [21:22]
BryceBothost uptime: 140 days, 22 hours, 35 minutes, and 48.429999999702 seconds. [21:22]
brycecMy one
(and loadavg is settling back to the .2 range
[21:22]
mercutiobrycec: so much more bw you doing an hour with smaller ping size? [21:32]
brycecmercutio: Funny you should ask, I was just looking at that https://dl.dropboxusercontent.com/u/3167967/screenshot_2014-02-13_21-31-44.png
You can clearly see when I added a bunch of hosts with size=500, and when I dropped that back down
[21:33]
mercutioiyou're probably still doing a lot of pps
were you at defaults prior?
i didn't quite get that
whther you'd shifted to 500 byte packets today
or ages ago
there are still gaps
i don't think fping does have a way to cache dns lookups
you can use ps to find out what command line it's calling and cut and paste the fping
and do one ping, to figure out long it's waiting on dns
but would need to do it during some kind of outage to know for eusre how much it impacting
comcast.net if you ping that has dns ttl of 30 seconds for instance
so isn't likely to be cached between polls
[21:36]
........ (idle for 37mn)
***kevr1 has joined #arpnetworks [22:20]
kevr1 has quit IRC (Quit: WeeChat 0.4.3)
mnathani_web has quit IRC (Quit: Page closed)
[22:25]
......... (idle for 44mn)
xales has joined #arpnetworks [23:12]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)