#arpnetworks 2014-02-11,Tue

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
up_the_ironsAny2 for v4 is pretty much back (but lots of peers are down due to CoreSite's renumbering; just gotta wait for everyone to renumber) [02:25]
........... (idle for 50mn)
mercutiohmm i still seeing as11799
that's outgoing to two places though
[03:15]
incoming the same from one place [03:23]
.......... (idle for 45mn)
up_the_ironssince the majority of peers are down due to renumbering, that is expected [04:08]
........................................... (idle for 3h34mn)
m0undshttp://openntpproject.org <- this also scans ip ranges (up to a /22 at a time) looking for ntpd w/insecure config
http://openresolverproject.org for open dns resolvers
[07:42]
........................ (idle for 1h55mn)
brycecDepends on client and whether you're logging... But you're probably thinking of "/lastlog" 23:06:39 < mercutio> how do i search scrollback? :)
Well you can try using @log_search <terms> but beyond that, I don't know of a good IRC interface for that sort of thing. (At least not one I'm willing to write :P) 23:10:20 < mercutio> i weant to find a way to find urls i pasted to irc :)
[09:39]
............. (idle for 1h4mn)
heh... my ARP IPv6 tunnel is about 1/2 the latency of my HE IPv6 tunnel. Yay ARP [10:44]
m0undsyeah, about the same here - the closest HE tunnel ep was in LA, but it's way oversubscribed
and my latency to it was 80ms +/- 20ms
vs 30 ms +/- 2ms, haha
[10:45]
brycecI'm pointed at Seattle, being just 300mi away and get ~100ms or so, and ARP (much further away) is closer to 50ms [10:47]
m0undshuh. [10:47]
brycec(too busy to look at traceroutes, but needless to say it makes little sense and I did pick the shortest, quickest POP at the time) [10:47]
m0undsweird
huh. looks like after that last 6500 reboot, my latency's actually 50ms instead of 30
[10:47]
mnathanibrycec: do you have a guide you could link to that describes how you setup the ipv6 tunnel using ARP? [10:48]
m0undsoh well [10:48]
brycecmnathani: obviously depends on your OS... I followed m0unds's guide and realized that it's as simple as setting up matching (Debian) v4tunnel statements on either end.
m0unds' guide was for FreeBSD and Juniper SRX gear, but I got the gist
[10:49]
mnathaniand you need to have the /48 enabled I assume [10:50]
brycecNote I just have the tunnel up, I don't have routing or /64 handoff setup yet
Yeah, though you could route /128s I guess?
I dunno, not an expert.
[10:50]
mnathanik [10:52]
.... (idle for 17mn)
***avj has joined #arpnetworks [11:09]
..... (idle for 23mn)
DaCa has quit IRC (Ping timeout: 260 seconds) [11:32]
......... (idle for 44mn)
mercutioup_the_irons: can't it be on both numbers at once? [12:16]
maybe i shoujdl just log
oh i am logging it seems
that's better
[12:21]
..... (idle for 20mn)
***PatrickINIZ has joined #arpnetworks
robonerd has quit IRC (Remote host closed the connection)
[12:44]
.... (idle for 18mn)
acf_darnit
http://kremvax.acfsys.net/smokeping.cgi?target=Remote.voipms-lsanca
also, anything ipv4 on HE
[13:03]
mercutioyou got blocked?
your dsl latency is starting :/
[13:07]
acf_no, arp -> anything through trit is broken [13:08]
mercutiooh what [13:08]
acf_ping he.net
ping losangeles.voip.ms
[13:08]
mercutiomaybe any2ix issue [13:08]
***PatrickINIZ has left "http://iniz.com" [13:09]
mercutioyeh hmm
he having looking glass
oh it works from there
[13:09]
acf_I'm confused [13:10]
mercutiomaybe need a diff trace point [13:10]
acf_did someone block icmp somewhere in one direction? [13:11]
mercutiowell this is traceroute so maybe udp
lg.he.net
actually reverse path filtering can look like that
sometimes
but it looked like it was coming in vl5.s1.lax.arpnetworks.com
unless that new box calls itself that
telnet on port 80 not working too
[13:11]
acf_so, a ping to arpnetworks.com through the he lg works [13:13]
mercutiowell to my host [13:13]
acf_but a ping from 4or6.com to he.net breaks [13:13]
mercutioyeh
but if there is linux rp_filter on the new router it won't allow a response to come back for ping
if it hasn't seen it go out on that router
and this outbound path is via trit.net
so of v5.s1 is thew new host
linux defaults to rp_filter set to 1
and you need to set it to 0
or it'll behave just like this
[13:13]
acf_okay. I suppose that would explain it [13:14]
mercutioand itg was just done last night
up_the_irons: you around?
so it may be that dns is wrong
[13:14]
acf_do you think it was intentional?
the filter
[13:15]
mercutionope
it's broken
it's not icmp onyl issue
port 80 doesn't work
[13:15]
acf_ah yes [13:15]
mercutioit may be connection tracking too
it's not necessarily rp_filter
but both can accomplish the same thing
[13:16]
acf_http://kremvax.acfsys.net/smokeping.cgi?epoch_start=1392105600;hierarchy=;epoch_end=1392153357;target=Remote.voipms-lsanca;displaymode=n;start=2014-02-10%2024%3A00;end=now;Generate!=Generate! [13:17]
mercutiothe internet isn't normally symmetric [13:17]
acf_5:00am ish [13:17]
mercutioi think he was talking about making changes 11 hours ago
hmm taht 9 horus ago?
[13:17]
acf_between 5:10am and 5:15am exactly
15% packet loss on the last sample
[13:19]
mercutioi couldn't find any sites oging over any2ix las tnight
but i didn't try that hard
[13:20]
acf_digitalwest.net
works
[13:22]
mercutiodoes it go over any2ix back? [13:22]
acf_idk, the lg has a password [13:22]
mercutioit's not that it's going out trit.net, it's that it's coming back via any2ix
what
not for me
oh dw one
[13:22]
acf_yeah [13:23]
mercutiohttp shoudl be broken from he.net too
but they don't have any http tests
[13:23]
***DaCa has joined #arpnetworks [13:27]
m0undslooks like losangeles.voip.ms is @ quadranet [13:33]
***mike-burns has quit IRC (Read error: Connection reset by peer) [13:34]
m0undsfwiw, i can't ping it from anything i have (arp, home, work, nada) [13:34]
***mike-burns has joined #arpnetworks
ChanServ sets mode: +o mike-burns
KDE_Perry has quit IRC (Ping timeout: 260 seconds)
[13:34]
brycecm0unds: pings for me from TWC
PING losangeles.voip.ms (96.44.149.186) 56(84) bytes of data.
64 bytes from 96.44.149.186.static.quadranet.com (96.44.149.186): icmp_seq=1 ttl=51 time=45.5 ms
[13:35]
***KDE_Perry has joined #arpnetworks [13:35]
brycecAnd from comcast [13:35]
toddfI can ping that from my arpnetworks vps [13:35]
brycecI cannot ping it from ARP
I can ping it from Chunkhost though.
[13:35]
toddfhttp://sprunge.us/JROF [13:36]
brycecOn ARP, I cannot trace path coresite
*past
[13:36]
toddfdoes 1gbit ports have a different v4 router? [13:37]
mercutiotoddf: maybe [13:37]
toddfI don't even see coresite [13:37]
mercutioit's whether return path is coresite was the issue
(i think)
[13:37]
brycecMine on ARP: 1 174.136.103.129 (174.136.103.129) 23.764 ms 23.790 ms 24.034 ms
2 v440.r6.lax2.trit.net (208.90.34.78) 0.603 ms 1.152 ms 1.147 ms
heh
[13:38]
mercutiowell should be symmetric or not at all :) [13:38]
toddfdid you guys look at my sprunge paste? I can clearly get to losangeles.voip.ms from my arpnetworks vps [13:38]
mercutiocan you telnet www.he.net on port 80 ? [13:39]
toddfanyone else here `testing' the 1gbps ports? [13:39]
acf_with vps? [13:39]
mercutioyeh [13:39]
acf_I'm on a dedicated machine [13:39]
mercutiocos that's really the best real test
same diff
it doesn't work on dedicated for me
[13:39]
toddfI can hit www.he.net:80 both on v4 and v6 [13:39]
mercutioit maybe some subnets are ok
toddf: why are you immune? :)
[13:39]
toddfsomeone good with looking glass ? [13:40]
mercutiojust do a traceroute to your ip, see if it hits v5.. [13:40]
toddfif some subnets are working, its as if a bgp is not advertising all or something
mercutio: look at my sprunge post! http://sprunge.us/JROF
[13:40]
acf_no 208.79...
no 174.136...
no 206.125...
[13:40]
m0undsi'm getting "permission denied" to www.he.net [13:41]
mercutiom0unds: weird [13:41]
toddfpermission denied sounds like a user running traceroute that requires root [13:41]
mercutioyou mean using telnet?
telnet: Unable to connect to remote host: Network is unreachable
i get that
[13:41]
toddfthis is me to he.net:
http://sprunge.us/BiPP
[13:42]
m0undsyea, it's throwing a 403
when i try to curl it - might just be preventing curl from retrieving it
[13:42]
mercutiotodd: mind telling us your ip? [13:42]
toddf3.v.freedaemon.com ;-) [13:42]
mercutiooh it doesn't even accept connection for me [13:42]
m0undsi get nothing on ipv4, but i get 9ms on ipv6 via mtr to www.he.net [13:43]
acf_cool toddf you're on s7 [13:43]
mercutioacf: how did you figure that out? [13:43]
acf_I'm on s1 [13:43]
mercutiooh i see
yeh so am i
[13:43]
acf_http://paste.unixcube.org/k/819449 [13:44]
mercutioso yeh it's working for toddf cos he's on s7 [13:44]
m0undsand yea, via v4 i'm going out over trit.net and it fails [13:44]
mercutiom0unds: i think it's return path causing issues though
can't cut and paste that nicely
[13:44]
toddftelent -4 3.v.freedaemon.com 1234 -> bounces you to v4 www.he.net just incase there's any confusion [13:44]
mercutiofor lg.he.net
heh
i think we have to wait for up_the_irons to look into it
[13:44]
.............. (idle for 1h6mn)
up_the_ironsup_the_irons checks things out [14:51]
mercutiooh cool [14:51]
up_the_ironsgimme a min to go through the scrollback [14:53]
mercutioyou must be quick reader :)
it looks like connectino tracking or rp_filter
i figure
but that's only if v5 is coresite
on the new box
[14:55]
up_the_ironswell yes it is taking longer than aminute?
whoops
*minute ;)
mercutio: actually, i just thought connection tracking too from some support tickets i got. i just disabled it on r1.lax (should not have been on :(
let's see if that helps
(i see more traffic flowing now)
[15:02]
brycecfwiw I can traceroute to losangeles.voip.ms from ARP, same route through coresite as before. Guess coresite got their act together.
ah cool
[15:03]
up_the_ironsbrycec: so that made a difference? [15:05]
brycecMaybe, or coresite fixed things for all I can tell. It's been ~2hrs since I tried and it failed :p [15:05]
up_the_ironsok [15:06]
mercutiowww.he.net accepts connection on port 80 now
so yeh i think it fixed
up_the_irons: do you have a time in mind that level3 is coming up?
[15:07]
up_the_ironsmercutio: they say by the end of the month i'll have an LoA for the x-conn, then like, a week after that, we turn up [15:09]
mercutiooh yip
just this ntt->verizon issue seems like it might not be resolved until then
and then only if it goes via level3 outobund
it was affecting acf rather than me though
[15:09]
up_the_ironsmercutio: yeah, the peers *could* be on both numbers at once, but since I was moving Any2 anyway to new gear, I decided to drop the old numbers [15:17]
mercutioahh ok
and there's that bgp collective fallback
and it helped minimise broken things :)
[15:21]
up_the_ironsyeah, the next shortest path is generally The BGP Collective, so impact was just 1 extra hop
cool, i found different hosts on NLNOG ring that have inbound paths of: Trit, NTT, nLayer
but still trying to find one on an Any2 peer
(or rather, one that takes that path)
would help to save that one for future diagnostics
[15:22]
mercutioyeh but it only makes sense in the short term
in the longer term, there'll be way more options
like finding stuff that goes via bgp collective isn't hard
[15:28]
brycecOy vey... My smokeping slave config (the configuration pushed to each smokeping slave) is 248k (according to the http log) [15:29]
Yamazaki-kunshould I see if I can play Eve over my VZW tether? [15:33]
***avj has quit IRC (Ping timeout: 245 seconds) [15:34]
Yamazaki-kunand the result: yeah, it works [15:39]
m0undshahaha
must be a low congestion vzw tower
their lte gear is so hideously oversubscribed in NM/CO it's absurd
[15:39]
***eryc has quit IRC (Ping timeout: 245 seconds)
eryc has joined #arpnetworks
eryc has quit IRC (Ping timeout: 245 seconds)
acf_ has quit IRC (Ping timeout: 245 seconds)
acf_ has joined #arpnetworks
eryc has joined #arpnetworks
eryc has quit IRC (Changing host)
eryc has joined #arpnetworks
awyeah has joined #arpnetworks
[15:45]
awyeahIs there something blocking ntp traffic?
to and from the VPSs?
[15:57]
mercutioyes [15:57]
brycecyes
well, *to* the VPS
[15:57]
mercutiouse a source port other than 123 [15:57]
up_the_ironshttps://twitter.com/arpnetworks/status/433094185122414592 [15:58]
BryceBotTWITTER: We have blocked all incoming NTP traffic to VM hosts; many were unwittingly participating in UDP amplification attacks (Tue Feb 11 04:24:34 +0000 2014, retweeted 4 times) [15:58]
awyeahah [15:58]
brycecbrycec wittingly participates :P [15:58]
up_the_ironsBut actually, i am just now applying a different filter [15:58]
awyeahokely dokely [15:58]
up_the_ironsI am opening up NTP, but the misconfigured hosts will be blocked [15:58]
brycecyay [15:59]
awyeahWhat constitutes misconfigured? [15:59]
mercutiomonlist [15:59]
up_the_ironsit participates in amplification attacks [15:59]
awyeahlol [15:59]
mercutiodid you try that nmap cmd? [15:59]
up_the_ironssrsly, we had over 500 Mbps of traffic going out last night from misconfigured NTP servers [15:59]
awyeahHolys iht. [16:00]
up_the_ironsmercutio: no, was having trouble getting all the dependencies [16:00]
brycecprotip: When writing Smokeping targets, don't forget to include host= [16:01]
awyeahLooking at my bandwidth, it's looking like my system was not participating, hopefully [16:03]
up_the_ironsit would be noticable [16:03]
mercutioahh [16:03]
awyeahI've got this for my restrict statement: restrict default nomodify notrap noquery [16:03]
mercutioup_the_irons: does arp have ntp servers? [16:03]
brycecYep, you should be fine. Easy to test yourself though. [16:03]
mercutioi want to see 1.2.3.<1-3> be anycast ntp servers
to go along with the proposal for 1.2.3.4 to be a standard anycast dns
[16:03]
up_the_ironsmercutio: no [16:04]
awyeahAh, I see, it's the noquery that should take care of it. [16:04]
***BryceBot has quit IRC (Ping timeout: 245 seconds) [16:04]
up_the_irons"disable monitor" is also an easy way to fix it [16:04]
mercutiomost people prob just use the pool anyway [16:04]
awyeahYou know what. That reminds me. [16:05]
brycecuh oh, come back BryceBot! [16:05]
mercutioopenntp also fixes it
does kvm actually require everyone run their own ntp clients?
i've kind of wondered that for a while
[16:05]
toddfmercutio: host time tracking is independent of guest time tracking [16:06]
mercutioso yes [16:06]
awyeahHey, cool, I'm talking ntp again. [16:06]
toddfso you can cronjob a command to set time against a remote system or you can use a ntpd
openntpd (I'm running it) defaults to client mode only, you have to explicitly uncomment the 'listen *' bit
just confirmed I am only a ntp client, so not likely contributing to the 500mbit of ntp traffic last night
[16:06]
mercutioit doesn't amplify even if it's listening too [16:08]
toddfremoving '3.v.freedaemon.com:1234' redirect to he.net now that the problem it was in theory helping diagnose is now fixed [16:10]
***BryceBot has joined #arpnetworks
BryceBot has quit IRC (Changing host)
BryceBot has joined #arpnetworks
[16:11]
up_the_ironsfrom the looks of the volume of vulnerable hosts that have been reported, it appears many hosts _default_ to the bad behavior [16:11]
hazardousgood thing i never run ntp!
i just occasionally hire a dwarf in a shoe to tweak the system clock
[16:19]
m0undsyou can run ntp, it's when ntp /listens/ for requests that it's a problem
all you have to do is toggle of mon and it's fine, and it can sync to pool.ntp.org or time.nist.gov or whatever
off* mon
[16:20]
mercutioup_the_irons: i think at least freebsd 9 defaults to being vulnerable [16:23]
m0undsit does until you run freebsd-update like you should do anyway
8.3-9.2 all default to listening, run freebsd-update fetch & install and it's patched
it's been available as a patch since january
[16:23]
mercutiotehre was as huge ddos over new years [16:25]
m0undsthere was also a big one on like 12/25, which is when freebsd released the advisory to make config changes
someone even mentioned it in here that same day
at least i thought it was the same day
[16:26]
staticsafemeh i'll just switch the fbsd box to openntpd [16:28]
up_the_ironsmercutio: damn.. we have SOOO many fbsd 9 hosts
and, big surprise, most people don't maintain their systems
[16:29]
staticsafewhen was this patched? [16:30]
m0undsi linked the advisory from freebsd yesterday [16:30]
mercutioup_the_irons: do you offer freebsd 10 yet? [16:30]
m0undsjanuary somesuch - they posted the original advisory in december [16:30]
mercutiofreebsd 10 adds zfs root support :) [16:30]
m0undshttp://www.freebsd.org/security/advisories/FreeBSD-SA-14:02.ntpd.asc [16:30]
up_the_ironsmercutio: ISO Only [16:31]
m0undshttp://svnweb.freebsd.org/base/head/etc/ntp.conf?view=log&pathrev=259973 original mention [16:31]
mercutioyeah it's hard to keep systems up to date
there's an even bigger problem with routers and so on with ntp
as they're even less likely to be kept up to date
[16:32]
staticsafei did see that advisory, didn't read it >_> [16:32]
mercutioi've been using openntpd for years though.. [16:32]
m0undsi have a crontab set up to execute freebsd-update cron, which emails me if there are new updates [16:35]
mercutiothe problem is it's not people who are "reasonably connected" that are likely to be at fault
as much as people who have no idea
[16:36]
m0undsm0unds shrugs
in 2014, it's sort of negligent to not maintain systems
people still don't do it, but i still think it's shitty regardless
[16:36]
mercutios/negligent/common/ [16:37]
BryceBot<m0unds> in 2014, it's sort of common to not maintain systems [16:37]
mercutioi can s/ your text :) [16:37]
m0undscommonality and negligence aren't interchangeable [16:37]
***Nat_RH has joined #arpnetworks [16:37]
mercutioit's what is vs what should be [16:38]
staticsafei suppose i could add freebsd-update cron [16:39]
mercutioit deos remind me though, i should follow freebsd security list
http://www.freebsd.org/security/advisories/FreeBSD-SA-14:01.bsnmpd.asc

that's also significant
[16:41]
Nat_RHHow many were affected? pretty sure I modified mine correctly a few weeks back [16:45]
mercutioquite a few [16:49]
brycecs/your/any/ [16:51]
BryceBot<mercutio> i can s/ any text :) [16:51]
mercutioeven ages back [16:52]
brycecAbout 20 lines or so
My smokeping data folder is 2.8GB :(
[16:53]
staticsafealso - http://blogs.freebsdish.org/portmgr/2014/02/03/time-to-bid-farewell-to-the-old-pkg_-tools/ [16:55]
awyeahWhat patchlevel was 9.2 patched? [16:55]
brycecs/ - pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng [16:55]
mercutiobrycec: what step size? [16:55]
brycecmercutio: still default [16:56]
staticsafe2014-01-14 19:42:28 UTC (releng/9.2, 9.2-RELEASE-p3) [16:56]
mercutiobrycec: you must be doing a lot of probes :) [16:56]
brycecAbout 200 hosts now
and 5 slaves
[16:56]
mercutioif you're doing lots you may want to consider reducing the ping packet size [16:56]
awyeahah, I see I got an email about that a few days ago. time to updatge [16:56]
mercutioi just started doing smokeping on arp [16:56]
brycecbrycec increases packet size to make up for up_the_irons' 500mbps [16:56]
mercutio+ FPing
binary = /usr/bin/fping
packetsize = 32
i have that
[16:57]
bryceccool
s/slaves/monitoring hosts (4 of which are slaves)
[16:57]
BryceBot<brycec> and 5 monitoring hosts (4 of which are slaves) [17:01]
staticsafemeh smokeping [17:01]
awyeahback in a minute. [17:01]
***awyeah has quit IRC (Quit: EPIC5-1.1.7[1705] - amnesiac : Do the gene pool a service... Add a bucket of chlorine today!) [17:01]
mercutioyou don't like it? [17:01]
staticsafenot really no
i especially don't like the CGI webapp
[17:01]
mercutioi don't like it how it hides minimum/maximum
in the period
[17:02]
m0undsyeah, cgi makes me sad [17:02]
mercutiomostly
it shows averages for the monitoring period
how it reloads all the time?
[17:02]
m0undsjust don't like it in general [17:02]
mercutioi find it useful [17:02]
m0undsi run it on hardware directly at home [17:02]
brycec^ [17:02]
mercutioand i'm not going to write my own
yet
[17:02]
staticsafestaticsafe converts pkg db to pkg2ng [17:03]
........... (idle for 50mn)
up_the_ironswhat a freakin' day (or week!).. and it's only the start... [17:53]
mnathaniup_the_irons: does your Bird setup support 4 byte AS numbers? [17:54]
up_the_ironsmnathani: i believe so
anything modern does :)
[17:54]
mercutioeverything supports 4 byte asn these days
but some things use dot format
[17:55]
up_the_ironsnow i can't find it in the docs, bah
ah found it
so yes, my BIRD setup supports 4 byte ASNs
[17:57]
mnathanicool [17:59]
mercutiois bird using dot format?
it's not using dot format
[18:03]
mnathaniwhats an example of a 4 byte only ASN? [18:12]
m0unds234567 [18:14]
mnathaniAS234567 has not been visible in the global routing table since March 09, 2011 [18:14]
m0undsi meant it as an example [18:15]
mnathaniahh [18:15]
m0undshttps://www.ietf.org/rfc/rfc5396.txt
for the diff bw asdot and asplain
[18:18]
mercutioi like asdot, but asplain is standard now pretty much [18:23]
m0undsyea, i haven't seen asdot in a while
i don't really work with internet-connected systems a ton, though
[18:26]
mercutioi use openbgpd, which uses asdot notation [18:36]
mnathanim0unds: is there an air gap between your systems and the internet? [18:36]
mercutioand any new asn's now days are all 32 bit [18:36]
mnathaniIs it possible we might outgrow that limit on number of networks and need to expand to more than 4 byte ASNs [18:39]
mercutioyes
but it's unlikely
[18:42]
***robonerd has joined #arpnetworks
robonerd has quit IRC (Changing host)
robonerd has joined #arpnetworks
[18:42]
mercutioi think it's more prudent to replace bgp with something better [18:42]
m0undsmnathani: what i mean is that i'm not a network engineer with internet-connected systems anymore
of* internet-connected*
[18:43]
mercutiothere's a slow gradual shift to having routing decisions being made globally, rather than at every point in the network
so if a talks to b talks to c talks to d
then along at each hop it decides where to go next
[18:43]
m0undsas a hobbyist with virtual servers, i couldn't give two shits about which ASNs are which :) [18:43]
mercutioso c might decide to talk to a and loop it all over again [18:43]
robonerdglobal routing decisions sound as smart as software defined routing
ie, sounds bad
[18:44]
mercutioit's similar.
it's not necessarily a bad thing
but some kind of hybrid solution can be useful
[18:45]
robonerdcan you give an example? [18:45]
mercutioi had a kind of nifty idea of how things could work better, but a lot of decisions are motivated by large companies
and so you'r not really going to change them
[18:45]
robonerdwhat's the idea? [18:46]
mercutios/can/could/ [18:46]
BryceBot<robonerd> could you give an example? [18:46]
mercutiowell basically you pay to get traffic to a point near the user
forward only routing
so like you pay to get traffic to amsix
from los agnels
err los angeles
and then the path between those two points can be varied
and you have per minute charging or such
and you can choose to take lower cost or lower latency/higher badnwidth paths
[18:46]
***pcn has quit IRC (Ping timeout: 245 seconds) [18:47]
mercutioand as more people choose the better paths the cost goes up like a stock exchange [18:47]
robonerdsounds like what internap did with their routing engine [18:47]
mercutioso when there's failures etc
cost will tend to go up
[18:48]
robonerdi think it's a great idea if we could get it on an open basis [18:48]
mercutioand when there is idle capcaity cost goes down
so you might have a better path while it's cheaper, then shift to a cheaper path when cost goes up
because you can't redally change how people send you traffic, only how you send them traffic
[18:48]
***pcn has joined #arpnetworks [18:49]
robonerdyea
damn, dis nigga be worn OUT
i wrote a shitload of code today, but the biggest drain was 2 challenging problems/bugs
[18:51]
m0undsi played video games and drank whiskey [18:52]
robonerdwhat kind of whiskey?
(v games be damned)
[18:53]
m0undsbalcones brimstone [18:53]
robonerdi've not had that one yet [18:53]
m0undshttp://www.balconesdistilling.com/products [18:53]
robonerdyea, looks worth trying
how do you like it?
[18:54]
m0undsthe smoke is nice
it's kinda sweet - first whiskey i've found that my wife will actually drink
[18:54]
robonerdwhere does it lie?
hm interesting
[18:55]
m0undsit's pretty up front, smoke-wise
almost like a firey nose to it
much mellower than it smells though
[18:55]
robonerdsweet and smokey, you know, that sounds just about right for texas
bbq sauce and such
[18:55]
m0undsyeah, haha [18:55]
robonerdwell i'll keep an eye out for it
http://www.youtube.com/watch?v=5tm23wDVU2U
[18:55]
BryceBotYouTube Education: "Grand Designs S09 E01 The Apprentice Store, Somerset SD ( Standard Definition )" by Roland Marginas (49m 3s), 27,742 views, 73 likes and 7 dislikes. Uploaded 2013-06-26T09:18:34.000Z. [18:55]
robonerdthere's something for you [18:56]
m0undstook a while to find it locally - none of the bigger local liquor joints carried it [18:56]
mercutiom0unds: the main actual issue with implementing would be getting mpls connections cheaply on a usage basis or such, and getting people onboard to use it [18:56]
robonerdbrb getting high [18:56]
m0undserr..? [18:56]
mercutiobut i'm actually in favour of per-bit-charging rather than block pipe charging [18:57]
m0undsnot sure if you meant to tag me there, haha [18:57]
mercutiobecause it encourages people to cull "bad" traffic as a way to save money
rather than preserve performance.
err i meant to tag robonerd
[18:57]
m0undsit's an interesting idea, but i could see corporations figuring out ways to abuse it [18:58]
mercutiohow so?
it's kind of the way electricity works
[18:58]
m0undsehh, there are regulatory bodies that protect the cost of electricity delivery in the US
dunno if that's the case abroad
but PRCs prevent price gouging and stuff
[18:59]
mercutioeven to businesses? [18:59]
m0undsyep [18:59]
mercutioincluding big customers? [18:59]
m0undsyep. they can schedule pricing differently based on use [18:59]
mercutiohere big customers can pay varaible power costs
and get cheaper power.
most of the time
[18:59]
m0undsit can be dynamic depending on industry and consumption [18:59]
mercutiobut as soon as like a power station goes or such prices jump heaps [19:00]
m0undsPRCs here require approval to raise rates [19:00]
mercutiobut it means if you
err if you're doing stuff that you can temp shut off when power use is highest, that uses a lot of power, then you can get cheaper power the rest of the time
[19:00]
m0undsif it's reasonable, for instance, if you need to invest more money in delivery equipment or whatever, they can approve it pretty easily [19:00]
mercutiowhich happens for a few industrial type things. [19:01]
m0undsyea, they do that for things like arc furnaces for steel production and stuff [19:01]
mercutioyeh
but that's how power works in general
then on top of that are residential plans that offer smoother pricing
[19:01]
m0undsthey still have fixed rate schedules for large stuff in the US
it's just a matter of whether it's high demand hours or not
[19:03]
mercutioahh ok, so it doesn't take outages into consideration?
i started thinking about this more when there was that huge flooding incident in east coast US
and some providers were completely screwed to europe
[19:04]
m0undsah [19:05]
mercutiodidn't really see much local coverage of the extent of problems
but reading overseas stuff it sounded like lots of datacentres did silly things like have their generators in baseemnts.
so when there was flooding they couldn't run their generators.
[19:06]
m0undsyeah - it sucks that there are so many facilities in areas that aren't well suited to modern stuff [19:07]
mercutiothe thing is it's expesnive to fix these things [19:07]
m0undsnot a ton of modern infrastructure, or stuff slapped together [19:07]
mercutioso if you want to move all of your generators to 4th floor from basement, it'll cost real money
and when you say "what if there's a flood"
people think it's like a biblical thing like noah's ark
and not going to happen to them.
until actual issues happen people don't tend to want to sepnd money
[19:07]
m0undsyep [19:08]
mercutioeven then with those that did, some people couldn't get fuel for generators.
and "best advice" now seems to be that you should have 3 sources of fuel
california has all the potential earthquake stuff going on
[19:08]
m0undsthere was a blog that was kept by some guys in a DC in louisiana during/after hurricane katrina [19:09]
mercutioand i'm sure most of the datacentres are pretty good for erathquake protection
but if there's fibre breaks, there could be a long time to restore
due to being in "dangerous" areas
there may be some typhoon risk there too?
[19:09]
m0undshttp://interdictor.livejournal.com/2005/08/28/
^ it was that blog there - intercosmos media group or something based in new orleans
in CA? i think it's pretty limited typhoon risk
not out of the realm of possibility, but i think earthquakes are more likely than typhoons by far
[19:12]
mercutiook
well i'm far away so i don't really know the risks
[19:14]
m0undsyeah
power issues maybe
socal has a super high demand for power and water
[19:14]
mercutioi think water issues are very likely
given an earthquake
given that there is already water shortages
[19:23]
..... (idle for 23mn)
brycecIf anybody is interested (mercutio, up_the_irons), I've increased my smokeping resolution to 1 minute. [19:47]
mnathanibrycec: cool
@smokeping
[19:49]
BryceBothttps://smokeping.cobryce.com/ [19:49]
mercutiodid you tweak your existing rrd thing?
you have to when rrd has diff step size
[19:50]
brycecmercutio: I just nuked them
Totally redesigned the rra's
[19:50]
mercutiook
that works
that's usually what i do :)
[19:50]
brycecI played with the idea, but I realized that the historical data isn't really that important [19:50]
mercutiowhch reminds me i was going to see how verizon had been doing
only 5% loss atm
[19:51]
brycecWhich also played into the redesign of the rra's - I don't keep data beyond 6mos, and it's weekly averages past 1 wk [19:51]
mercutiointeresting i sse the ping rising with forward path verizon, as well as forward path via ntt
so i think there's dual issues, cos packet lsos doesn't happen when sending via verizon
[19:52]
.... (idle for 15mn)
apparently another ddos is happening atm [20:07]
brycecoh dear [20:07]
mercutiowell arp shoudln't be contributing at least [20:08]
pcnWhat proto is being used to attack? [20:11]
mercutiontp [20:11]
pcnOK, so same attack. [20:22]
mercutioyeh
happened new years and xmas too
[20:23]
..................... (idle for 1h44mn)
up_the_ironsbrycec: cool [22:07]
CaZeMan, I wanna watch the olympics.
It's the only time I've ever wished I had a VPS in some other country. :D
[22:18]
up_the_ironsi'm watching it every night, while coding / networking / bgp'ing ;) [22:19]
.... (idle for 17mn)
this is really cool. i've finally been able to enumerate some NLNOG hosts according to which incoming path they take to us:
NTT - lchost01.ring.nlnog.net
nLayer - doruknet01.ring.nlnog.net
Trit - teamix01.ring.nlnog.net
Mzima - inerail01.ring.nlnog.net
Any2 IX - vocus01.ring.nlnog.net
That should help greatly with diagnostics in the future
[22:36]
........ (idle for 37mn)
mercutiocool. [23:13]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)