mercutio: hmm i still seeing as11799
that's outgoing to two places though
incoming the same from one place up_the_irons: since the majority of peers are down due to renumbering, that is expected m0unds: http://openntpproject.org <- this also scans ip ranges (up to a /22 at a time) looking for ntpd w/insecure config
http://openresolverproject.org for open dns resolvers brycec: Depends on client and whether you're logging... But you're probably thinking of "/lastlog" 23:06:39 < mercutio> how do i search scrollback? :)
Well you can try using @log_search <terms> but beyond that, I don't know of a good IRC interface for that sort of thing. (At least not one I'm willing to write :P) 23:10:20 < mercutio> i weant to find a way to find urls i pasted to irc :)
heh... my ARP IPv6 tunnel is about 1/2 the latency of my HE IPv6 tunnel. Yay ARP m0unds: yeah, about the same here - the closest HE tunnel ep was in LA, but it's way oversubscribed
and my latency to it was 80ms +/- 20ms
vs 30 ms +/- 2ms, haha brycec: I'm pointed at Seattle, being just 300mi away and get ~100ms or so, and ARP (much further away) is closer to 50ms m0unds: huh. brycec: (too busy to look at traceroutes, but needless to say it makes little sense and I did pick the shortest, quickest POP at the time) m0unds: weird
huh. looks like after that last 6500 reboot, my latency's actually 50ms instead of 30 mnathani: brycec: do you have a guide you could link to that describes how you setup the ipv6 tunnel using ARP? m0unds: oh well brycec: mnathani: obviously depends on your OS... I followed m0unds's guide and realized that it's as simple as setting up matching (Debian) v4tunnel statements on either end.
m0unds' guide was for FreeBSD and Juniper SRX gear, but I got the gist mnathani: and you need to have the /48 enabled I assume brycec: Note I just have the tunnel up, I don't have routing or /64 handoff setup yet
Yeah, though you could route /128s I guess?
I dunno, not an expert. mnathani: k ***: avj has joined #arpnetworks
DaCa has quit IRC (Ping timeout: 260 seconds) mercutio: up_the_irons: can't it be on both numbers at once?
maybe i shoujdl just log
oh i am logging it seems
that's better ***: PatrickINIZ has joined #arpnetworks
robonerd has quit IRC (Remote host closed the connection) acf_: darnit
http://kremvax.acfsys.net/smokeping.cgi?target=Remote.voipms-lsanca
also, anything ipv4 on HE mercutio: you got blocked?
your dsl latency is starting :/ acf_: no, arp -> anything through trit is broken mercutio: oh what acf_: ping he.net
ping losangeles.voip.ms mercutio: maybe any2ix issue ***: PatrickINIZ has left "http://iniz.com" mercutio: yeh hmm
he having looking glass
oh it works from there acf_: I'm confused mercutio: maybe need a diff trace point acf_: did someone block icmp somewhere in one direction? mercutio: well this is traceroute so maybe udp
lg.he.net
actually reverse path filtering can look like that
sometimes
but it looked like it was coming in vl5.s1.lax.arpnetworks.com
unless that new box calls itself that
telnet on port 80 not working too acf_: so, a ping to arpnetworks.com through the he lg works mercutio: well to my host acf_: but a ping from 4or6.com to he.net breaks mercutio: yeh
but if there is linux rp_filter on the new router it won't allow a response to come back for ping
if it hasn't seen it go out on that router
and this outbound path is via trit.net
so of v5.s1 is thew new host
linux defaults to rp_filter set to 1
and you need to set it to 0
or it'll behave just like this acf_: okay. I suppose that would explain it mercutio: and itg was just done last night
up_the_irons: you around?
so it may be that dns is wrong acf_: do you think it was intentional?
the filter mercutio: nope
it's broken
it's not icmp onyl issue
port 80 doesn't work acf_: ah yes mercutio: it may be connection tracking too
it's not necessarily rp_filter
but both can accomplish the same thing acf_: http://kremvax.acfsys.net/smokeping.cgi?epoch_start=1392105600;hierarchy=;epoch_end=1392153357;target=Remote.voipms-lsanca;displaymode=n;start=2014-02-10%2024%3A00;end=now;Generate!=Generate! mercutio: the internet isn't normally symmetric acf_: 5:00am ish mercutio: i think he was talking about making changes 11 hours ago
hmm taht 9 horus ago? acf_: between 5:10am and 5:15am exactly
15% packet loss on the last sample mercutio: i couldn't find any sites oging over any2ix las tnight
but i didn't try that hard acf_: digitalwest.net
works mercutio: does it go over any2ix back? acf_: idk, the lg has a password mercutio: it's not that it's going out trit.net, it's that it's coming back via any2ix
what
not for me
oh dw one acf_: yeah mercutio: http shoudl be broken from he.net too
but they don't have any http tests ***: DaCa has joined #arpnetworks m0unds: looks like losangeles.voip.ms is @ quadranet ***: mike-burns has quit IRC (Read error: Connection reset by peer) m0unds: fwiw, i can't ping it from anything i have (arp, home, work, nada) ***: mike-burns has joined #arpnetworks
ChanServ sets mode: +o mike-burns
KDE_Perry has quit IRC (Ping timeout: 260 seconds) brycec: m0unds: pings for me from TWC
PING losangeles.voip.ms (96.44.149.186) 56(84) bytes of data.
64 bytes from 96.44.149.186.static.quadranet.com (96.44.149.186): icmp_seq=1 ttl=51 time=45.5 ms ***: KDE_Perry has joined #arpnetworks brycec: And from comcast toddf: I can ping that from my arpnetworks vps brycec: I cannot ping it from ARP
I can ping it from Chunkhost though. toddf: http://sprunge.us/JROF brycec: On ARP, I cannot trace path coresite
*past toddf: does 1gbit ports have a different v4 router? mercutio: toddf: maybe toddf: I don't even see coresite mercutio: it's whether return path is coresite was the issue
(i think) brycec: Mine on ARP: 1 174.136.103.129 (174.136.103.129) 23.764 ms 23.790 ms 24.034 ms
2 v440.r6.lax2.trit.net (208.90.34.78) 0.603 ms 1.152 ms 1.147 ms
heh mercutio: well should be symmetric or not at all :) toddf: did you guys look at my sprunge paste? I can clearly get to losangeles.voip.ms from my arpnetworks vps mercutio: can you telnet www.he.net on port 80 ? toddf: anyone else here `testing' the 1gbps ports? acf_: with vps? mercutio: yeh acf_: I'm on a dedicated machine mercutio: cos that's really the best real test
same diff
it doesn't work on dedicated for me toddf: I can hit www.he.net:80 both on v4 and v6 mercutio: it maybe some subnets are ok
toddf: why are you immune? :) toddf: someone good with looking glass ? mercutio: just do a traceroute to your ip, see if it hits v5.. toddf: if some subnets are working, its as if a bgp is not advertising all or something
mercutio: look at my sprunge post! http://sprunge.us/JROF acf_: no 208.79...
no 174.136...
no 206.125... m0unds: i'm getting "permission denied" to www.he.net mercutio: m0unds: weird toddf: permission denied sounds like a user running traceroute that requires root mercutio: you mean using telnet?
telnet: Unable to connect to remote host: Network is unreachable
i get that toddf: this is me to he.net:
http://sprunge.us/BiPP m0unds: yea, it's throwing a 403
when i try to curl it - might just be preventing curl from retrieving it mercutio: todd: mind telling us your ip? toddf: 3.v.freedaemon.com ;-) mercutio: oh it doesn't even accept connection for me m0unds: i get nothing on ipv4, but i get 9ms on ipv6 via mtr to www.he.net acf_: cool toddf you're on s7 mercutio: acf: how did you figure that out? acf_: I'm on s1 mercutio: oh i see
yeh so am i acf_: http://paste.unixcube.org/k/819449 mercutio: so yeh it's working for toddf cos he's on s7 m0unds: and yea, via v4 i'm going out over trit.net and it fails mercutio: m0unds: i think it's return path causing issues though
can't cut and paste that nicely toddf: telent -4 3.v.freedaemon.com 1234 -> bounces you to v4 www.he.net just incase there's any confusion mercutio: for lg.he.net
heh
i think we have to wait for up_the_irons to look into it -: up_the_irons checks things out mercutio: oh cool up_the_irons: gimme a min to go through the scrollback mercutio: you must be quick reader :)
it looks like connectino tracking or rp_filter
i figure
but that's only if v5 is coresite
on the new box up_the_irons: well yes it is taking longer than aminute?
whoops
*minute ;)
mercutio: actually, i just thought connection tracking too from some support tickets i got. i just disabled it on r1.lax (should not have been on :(
let's see if that helps
(i see more traffic flowing now) brycec: fwiw I can traceroute to losangeles.voip.ms from ARP, same route through coresite as before. Guess coresite got their act together.
ah cool up_the_irons: brycec: so that made a difference? brycec: Maybe, or coresite fixed things for all I can tell. It's been ~2hrs since I tried and it failed :p up_the_irons: ok mercutio: www.he.net accepts connection on port 80 now
so yeh i think it fixed
up_the_irons: do you have a time in mind that level3 is coming up? up_the_irons: mercutio: they say by the end of the month i'll have an LoA for the x-conn, then like, a week after that, we turn up mercutio: oh yip
just this ntt->verizon issue seems like it might not be resolved until then
and then only if it goes via level3 outobund
it was affecting acf rather than me though up_the_irons: mercutio: yeah, the peers *could* be on both numbers at once, but since I was moving Any2 anyway to new gear, I decided to drop the old numbers mercutio: ahh ok
and there's that bgp collective fallback
and it helped minimise broken things :) up_the_irons: yeah, the next shortest path is generally The BGP Collective, so impact was just 1 extra hop
cool, i found different hosts on NLNOG ring that have inbound paths of: Trit, NTT, nLayer
but still trying to find one on an Any2 peer
(or rather, one that takes that path)
would help to save that one for future diagnostics mercutio: yeh but it only makes sense in the short term
in the longer term, there'll be way more options
like finding stuff that goes via bgp collective isn't hard brycec: Oy vey... My smokeping slave config (the configuration pushed to each smokeping slave) is 248k (according to the http log) Yamazaki-kun: should I see if I can play Eve over my VZW tether? ***: avj has quit IRC (Ping timeout: 245 seconds) Yamazaki-kun: and the result: yeah, it works m0unds: hahaha
must be a low congestion vzw tower
their lte gear is so hideously oversubscribed in NM/CO it's absurd ***: eryc has quit IRC (Ping timeout: 245 seconds)
eryc has joined #arpnetworks
eryc has quit IRC (Ping timeout: 245 seconds)
acf_ has quit IRC (Ping timeout: 245 seconds)
acf_ has joined #arpnetworks
eryc has joined #arpnetworks
eryc has quit IRC (Changing host)
eryc has joined #arpnetworks
awyeah has joined #arpnetworks awyeah: Is there something blocking ntp traffic?
to and from the VPSs? mercutio: yes brycec: yes
well, *to* the VPS mercutio: use a source port other than 123 up_the_irons: https://twitter.com/arpnetworks/status/433094185122414592 BryceBot: TWITTER: We have blocked all incoming NTP traffic to VM hosts; many were unwittingly participating in UDP amplification attacks (Tue Feb 11 04:24:34 +0000 2014, retweeted 4 times) awyeah: ah -: brycec wittingly participates :P up_the_irons: But actually, i am just now applying a different filter awyeah: okely dokely up_the_irons: I am opening up NTP, but the misconfigured hosts will be blocked brycec: yay awyeah: What constitutes misconfigured? mercutio: monlist up_the_irons: it participates in amplification attacks awyeah: lol mercutio: did you try that nmap cmd? up_the_irons: srsly, we had over 500 Mbps of traffic going out last night from misconfigured NTP servers awyeah: Holys iht. up_the_irons: mercutio: no, was having trouble getting all the dependencies brycec: protip: When writing Smokeping targets, don't forget to include host= awyeah: Looking at my bandwidth, it's looking like my system was not participating, hopefully up_the_irons: it would be noticable mercutio: ahh awyeah: I've got this for my restrict statement: restrict default nomodify notrap noquery mercutio: up_the_irons: does arp have ntp servers? brycec: Yep, you should be fine. Easy to test yourself though. mercutio: i want to see 1.2.3.<1-3> be anycast ntp servers
to go along with the proposal for 1.2.3.4 to be a standard anycast dns up_the_irons: mercutio: no awyeah: Ah, I see, it's the noquery that should take care of it. ***: BryceBot has quit IRC (Ping timeout: 245 seconds) up_the_irons: "disable monitor" is also an easy way to fix it mercutio: most people prob just use the pool anyway awyeah: You know what. That reminds me. brycec: uh oh, come back BryceBot! mercutio: openntp also fixes it
does kvm actually require everyone run their own ntp clients?
i've kind of wondered that for a while toddf: mercutio: host time tracking is independent of guest time tracking mercutio: so yes awyeah: Hey, cool, I'm talking ntp again. toddf: so you can cronjob a command to set time against a remote system or you can use a ntpd
openntpd (I'm running it) defaults to client mode only, you have to explicitly uncomment the 'listen *' bit
just confirmed I am only a ntp client, so not likely contributing to the 500mbit of ntp traffic last night mercutio: it doesn't amplify even if it's listening too toddf: removing '3.v.freedaemon.com:1234' redirect to he.net now that the problem it was in theory helping diagnose is now fixed ***: BryceBot has joined #arpnetworks
BryceBot has quit IRC (Changing host)
BryceBot has joined #arpnetworks up_the_irons: from the looks of the volume of vulnerable hosts that have been reported, it appears many hosts _default_ to the bad behavior hazardous: good thing i never run ntp!
i just occasionally hire a dwarf in a shoe to tweak the system clock m0unds: you can run ntp, it's when ntp /listens/ for requests that it's a problem
all you have to do is toggle of mon and it's fine, and it can sync to pool.ntp.org or time.nist.gov or whatever
off* mon mercutio: up_the_irons: i think at least freebsd 9 defaults to being vulnerable m0unds: it does until you run freebsd-update like you should do anyway
8.3-9.2 all default to listening, run freebsd-update fetch & install and it's patched
it's been available as a patch since january mercutio: tehre was as huge ddos over new years m0unds: there was also a big one on like 12/25, which is when freebsd released the advisory to make config changes
someone even mentioned it in here that same day
at least i thought it was the same day staticsafe: meh i'll just switch the fbsd box to openntpd up_the_irons: mercutio: damn.. we have SOOO many fbsd 9 hosts
and, big surprise, most people don't maintain their systems staticsafe: when was this patched? m0unds: i linked the advisory from freebsd yesterday mercutio: up_the_irons: do you offer freebsd 10 yet? m0unds: january somesuch - they posted the original advisory in december mercutio: freebsd 10 adds zfs root support :) m0unds: http://www.freebsd.org/security/advisories/FreeBSD-SA-14:02.ntpd.asc up_the_irons: mercutio: ISO Only m0unds: http://svnweb.freebsd.org/base/head/etc/ntp.conf?view=log&pathrev=259973 original mention mercutio: yeah it's hard to keep systems up to date
there's an even bigger problem with routers and so on with ntp
as they're even less likely to be kept up to date staticsafe: i did see that advisory, didn't read it >_> mercutio: i've been using openntpd for years though.. m0unds: i have a crontab set up to execute freebsd-update cron, which emails me if there are new updates mercutio: the problem is it's not people who are "reasonably connected" that are likely to be at fault
as much as people who have no idea -: m0unds shrugs m0unds: in 2014, it's sort of negligent to not maintain systems
people still don't do it, but i still think it's shitty regardless mercutio: s/negligent/common/ BryceBot: <m0unds> in 2014, it's sort of common to not maintain systems mercutio: i can s/ your text :) m0unds: commonality and negligence aren't interchangeable ***: Nat_RH has joined #arpnetworks mercutio: it's what is vs what should be staticsafe: i suppose i could add freebsd-update cron mercutio: it deos remind me though, i should follow freebsd security list
http://www.freebsd.org/security/advisories/FreeBSD-SA-14:01.bsnmpd.asc
that's also significant Nat_RH: How many were affected? pretty sure I modified mine correctly a few weeks back mercutio: quite a few brycec: s/your/any/ BryceBot: <mercutio> i can s/ any text :) mercutio: even ages back brycec: About 20 lines or so
My smokeping data folder is 2.8GB :( staticsafe: also - http://blogs.freebsdish.org/portmgr/2014/02/03/time-to-bid-farewell-to-the-old-pkg_-tools/ awyeah: What patchlevel was 9.2 patched? brycec: s/ - pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng mercutio: brycec: what step size? brycec: mercutio: still default staticsafe: 2014-01-14 19:42:28 UTC (releng/9.2, 9.2-RELEASE-p3) mercutio: brycec: you must be doing a lot of probes :) brycec: About 200 hosts now
and 5 slaves mercutio: if you're doing lots you may want to consider reducing the ping packet size awyeah: ah, I see I got an email about that a few days ago. time to updatge mercutio: i just started doing smokeping on arp -: brycec increases packet size to make up for up_the_irons' 500mbps mercutio: + FPing
binary = /usr/bin/fping
packetsize = 32
i have that brycec: cool
s/slaves/monitoring hosts (4 of which are slaves) BryceBot: <brycec> and 5 monitoring hosts (4 of which are slaves) staticsafe: meh smokeping awyeah: back in a minute. ***: awyeah has quit IRC (Quit: EPIC5-1.1.7[1705] - amnesiac : Do the gene pool a service... Add a bucket of chlorine today!) mercutio: you don't like it? staticsafe: not really no
i especially don't like the CGI webapp mercutio: i don't like it how it hides minimum/maximum
in the period m0unds: yeah, cgi makes me sad mercutio: mostly
it shows averages for the monitoring period
how it reloads all the time? m0unds: just don't like it in general mercutio: i find it useful m0unds: i run it on hardware directly at home brycec: ^ mercutio: and i'm not going to write my own
yet -: staticsafe converts pkg db to pkg2ng up_the_irons: what a freakin' day (or week!).. and it's only the start... mnathani: up_the_irons: does your Bird setup support 4 byte AS numbers? up_the_irons: mnathani: i believe so
anything modern does :) mercutio: everything supports 4 byte asn these days
but some things use dot format up_the_irons: now i can't find it in the docs, bah
ah found it
so yes, my BIRD setup supports 4 byte ASNs mnathani: cool mercutio: is bird using dot format?
it's not using dot format mnathani: whats an example of a 4 byte only ASN? m0unds: 234567 mnathani: AS234567 has not been visible in the global routing table since March 09, 2011 m0unds: i meant it as an example mnathani: ahh m0unds: https://www.ietf.org/rfc/rfc5396.txt
for the diff bw asdot and asplain mercutio: i like asdot, but asplain is standard now pretty much m0unds: yea, i haven't seen asdot in a while
i don't really work with internet-connected systems a ton, though mercutio: i use openbgpd, which uses asdot notation mnathani: m0unds: is there an air gap between your systems and the internet? mercutio: and any new asn's now days are all 32 bit mnathani: Is it possible we might outgrow that limit on number of networks and need to expand to more than 4 byte ASNs mercutio: yes
but it's unlikely ***: robonerd has joined #arpnetworks
robonerd has quit IRC (Changing host)
robonerd has joined #arpnetworks mercutio: i think it's more prudent to replace bgp with something better m0unds: mnathani: what i mean is that i'm not a network engineer with internet-connected systems anymore
of* internet-connected* mercutio: there's a slow gradual shift to having routing decisions being made globally, rather than at every point in the network
so if a talks to b talks to c talks to d
then along at each hop it decides where to go next m0unds: as a hobbyist with virtual servers, i couldn't give two shits about which ASNs are which :) mercutio: so c might decide to talk to a and loop it all over again robonerd: global routing decisions sound as smart as software defined routing
ie, sounds bad mercutio: it's similar.
it's not necessarily a bad thing
but some kind of hybrid solution can be useful robonerd: can you give an example? mercutio: i had a kind of nifty idea of how things could work better, but a lot of decisions are motivated by large companies
and so you'r not really going to change them robonerd: what's the idea? mercutio: s/can/could/ BryceBot: <robonerd> could you give an example? mercutio: well basically you pay to get traffic to a point near the user
forward only routing
so like you pay to get traffic to amsix
from los agnels
err los angeles
and then the path between those two points can be varied
and you have per minute charging or such
and you can choose to take lower cost or lower latency/higher badnwidth paths ***: pcn has quit IRC (Ping timeout: 245 seconds) mercutio: and as more people choose the better paths the cost goes up like a stock exchange robonerd: sounds like what internap did with their routing engine mercutio: so when there's failures etc
cost will tend to go up robonerd: i think it's a great idea if we could get it on an open basis mercutio: and when there is idle capcaity cost goes down
so you might have a better path while it's cheaper, then shift to a cheaper path when cost goes up
because you can't redally change how people send you traffic, only how you send them traffic ***: pcn has joined #arpnetworks robonerd: yea
damn, dis nigga be worn OUT
i wrote a shitload of code today, but the biggest drain was 2 challenging problems/bugs m0unds: i played video games and drank whiskey robonerd: what kind of whiskey?
(v games be damned) m0unds: balcones brimstone robonerd: i've not had that one yet m0unds: http://www.balconesdistilling.com/products robonerd: yea, looks worth trying
how do you like it? m0unds: the smoke is nice
it's kinda sweet - first whiskey i've found that my wife will actually drink robonerd: where does it lie?
hm interesting m0unds: it's pretty up front, smoke-wise
almost like a firey nose to it
much mellower than it smells though robonerd: sweet and smokey, you know, that sounds just about right for texas
bbq sauce and such m0unds: yeah, haha robonerd: well i'll keep an eye out for it
http://www.youtube.com/watch?v=5tm23wDVU2U BryceBot: YouTube Education: "Grand Designs S09 E01 The Apprentice Store, Somerset SD ( Standard Definition )" by Roland Marginas (49m 3s), 27,742 views, 73 likes and 7 dislikes. Uploaded 2013-06-26T09:18:34.000Z. robonerd: there's something for you m0unds: took a while to find it locally - none of the bigger local liquor joints carried it mercutio: m0unds: the main actual issue with implementing would be getting mpls connections cheaply on a usage basis or such, and getting people onboard to use it robonerd: brb getting high m0unds: err..? mercutio: but i'm actually in favour of per-bit-charging rather than block pipe charging m0unds: not sure if you meant to tag me there, haha mercutio: because it encourages people to cull "bad" traffic as a way to save money
rather than preserve performance.
err i meant to tag robonerd m0unds: it's an interesting idea, but i could see corporations figuring out ways to abuse it mercutio: how so?
it's kind of the way electricity works m0unds: ehh, there are regulatory bodies that protect the cost of electricity delivery in the US
dunno if that's the case abroad
but PRCs prevent price gouging and stuff mercutio: even to businesses? m0unds: yep mercutio: including big customers? m0unds: yep. they can schedule pricing differently based on use mercutio: here big customers can pay varaible power costs
and get cheaper power.
most of the time m0unds: it can be dynamic depending on industry and consumption mercutio: but as soon as like a power station goes or such prices jump heaps m0unds: PRCs here require approval to raise rates mercutio: but it means if you
err if you're doing stuff that you can temp shut off when power use is highest, that uses a lot of power, then you can get cheaper power the rest of the time m0unds: if it's reasonable, for instance, if you need to invest more money in delivery equipment or whatever, they can approve it pretty easily mercutio: which happens for a few industrial type things. m0unds: yea, they do that for things like arc furnaces for steel production and stuff mercutio: yeh
but that's how power works in general
then on top of that are residential plans that offer smoother pricing m0unds: they still have fixed rate schedules for large stuff in the US
it's just a matter of whether it's high demand hours or not mercutio: ahh ok, so it doesn't take outages into consideration?
i started thinking about this more when there was that huge flooding incident in east coast US
and some providers were completely screwed to europe m0unds: ah mercutio: didn't really see much local coverage of the extent of problems
but reading overseas stuff it sounded like lots of datacentres did silly things like have their generators in baseemnts.
so when there was flooding they couldn't run their generators. m0unds: yeah - it sucks that there are so many facilities in areas that aren't well suited to modern stuff mercutio: the thing is it's expesnive to fix these things m0unds: not a ton of modern infrastructure, or stuff slapped together mercutio: so if you want to move all of your generators to 4th floor from basement, it'll cost real money
and when you say "what if there's a flood"
people think it's like a biblical thing like noah's ark
and not going to happen to them.
until actual issues happen people don't tend to want to sepnd money m0unds: yep mercutio: even then with those that did, some people couldn't get fuel for generators.
and "best advice" now seems to be that you should have 3 sources of fuel
california has all the potential earthquake stuff going on m0unds: there was a blog that was kept by some guys in a DC in louisiana during/after hurricane katrina mercutio: and i'm sure most of the datacentres are pretty good for erathquake protection
but if there's fibre breaks, there could be a long time to restore
due to being in "dangerous" areas
there may be some typhoon risk there too? m0unds: http://interdictor.livejournal.com/2005/08/28/
^ it was that blog there - intercosmos media group or something based in new orleans
in CA? i think it's pretty limited typhoon risk
not out of the realm of possibility, but i think earthquakes are more likely than typhoons by far mercutio: ok
well i'm far away so i don't really know the risks m0unds: yeah
power issues maybe
socal has a super high demand for power and water mercutio: i think water issues are very likely
given an earthquake
given that there is already water shortages brycec: If anybody is interested (mercutio, up_the_irons), I've increased my smokeping resolution to 1 minute. mnathani: brycec: cool
@smokeping BryceBot: https://smokeping.cobryce.com/ mercutio: did you tweak your existing rrd thing?
you have to when rrd has diff step size brycec: mercutio: I just nuked them
Totally redesigned the rra's mercutio: ok
that works
that's usually what i do :) brycec: I played with the idea, but I realized that the historical data isn't really that important mercutio: whch reminds me i was going to see how verizon had been doing
only 5% loss atm brycec: Which also played into the redesign of the rra's - I don't keep data beyond 6mos, and it's weekly averages past 1 wk mercutio: interesting i sse the ping rising with forward path verizon, as well as forward path via ntt
so i think there's dual issues, cos packet lsos doesn't happen when sending via verizon
apparently another ddos is happening atm brycec: oh dear mercutio: well arp shoudln't be contributing at least pcn: What proto is being used to attack? mercutio: ntp pcn: OK, so same attack. mercutio: yeh
happened new years and xmas too up_the_irons: brycec: cool CaZe: Man, I wanna watch the olympics.
It's the only time I've ever wished I had a VPS in some other country. :D up_the_irons: i'm watching it every night, while coding / networking / bgp'ing ;)
this is really cool. i've finally been able to enumerate some NLNOG hosts according to which incoming path they take to us:
NTT - lchost01.ring.nlnog.net
nLayer - doruknet01.ring.nlnog.net
Trit - teamix01.ring.nlnog.net
Mzima - inerail01.ring.nlnog.net
Any2 IX - vocus01.ring.nlnog.net
That should help greatly with diagnostics in the future mercutio: cool.