oooook
oh, didn't realize the routing to arp via comcast went back to normal on the 12th
Yes, about the same PPS as when I had 200+ targets with 500B pings. (86pps or so)
^ 21:35:19 < mercutio> iyou're probably still doing a lot of pps
I don't recall when I added packetsize=500 but it wasn't yesterday. Might have been two days ago when I added a bunch more hosts, might have been earlier. 21:35:26 < mercutio> were you at defaults prior?
Today (now yesterday) went from 500B to 32B pings 21:36:03 < mercutio> whther you'd shifted to 500 byte packets today
And yes, I'll have to figure out the DNS woes. Someday.
But for now, we're settling for using better DNS servers. Hopefully we'll see improvement.
haha
i just use IPs for smokeping stuff and an external monitor that uses dns to resolve the hostname
I would too. However, some of the hosts I monitor are known to change their IPs
mercutio: One possible solution for pseudo-caching would be a script that [daily, hourly, whatever] did the DNS lookups and dumped the results in /etc/hosts
ah, gotcha
At the very least, it wouldn't hold up FPing
m0unds: Things like Tunnelr nodes, HE endpoints, my gateways and DNS servers are all easily static. But mirror sites and nlnog ring nodes may change
Or at least, there's no amount of guarantee they won't change.
:( better DNS servers did not fix Chunkhost's gaps
Oh cool! A useful log message! FPing: WARNING: smokeping took 85 seconds to complete 1 round of polling. It should complete polling in 60 seconds. You may have unresponsive devices in your setup.
It doesn't tell me WHAT or anything :/
But it's a start
I think I'll stop monitoring mirrors.
They were just there to provide more geographical data, which nlnogring is doing too
nlnogring?
oh, that
heh
https://atlas.ripe.net/
^ eh, up_the_irons?
that project seems less skeevy to me than a shared shell
but i'm paranoid
haha
not to say that anyone /would/ abuse it, but meh
I like the transparency of NLNOG. I don't know *what* I'm hooking up to my network with RIPE
And if the RIPE device/network is compromised in some fashion??
At lest with the shell, I have eyes on it, I can monitor it etc
And I suspect it's easier to kill a VM than to shutoff a switch port in some remote DC
well
Speaking as someone that admins a shared shell, with the right configuration, you've nothing to worry about.
conf t >> int gi1/0/1 >> shut
assuming you have access to the switch in question
which, if you're a network op, i'd assume you do :)
Of course you also have to connect to the switch :p
that's a given
Or if you're up_the_irons, I'm sure he has some management portal up and ready to click the right name.
(Switch: you also have to figure out the right port)
(and login)
But I digress. Hooking up a little black box to my network makes me nervous.
Also nlnog++ for just requiring a VM, and not requiring you to rack hardware
request remote hands to cut the cable, CUT IT
DO IT NAO
the old gen ripe dongle is 2x3x1"
http://probev2.ripe.net/
lolol
"If you find this device and are unaware of its purpose"
http://probev3.ripe.net/
shit if I came across that in a closet, I'd be *pissed* (at least until I knew what it was
yeah - NSA probably loves those little embedded linux server dinguses
http://www.lantronix.com/device-networking/embedded-device-servers/xport-pro.html
That is insane... The same fping takes 23s on ARP, takes 59s on Chunkhost.
(oh 61s that time)
i had a chunkhost box once
it was some kind of slow, horrid abomination
lol
In the very beginning, they were cool
i think
they did some free server giveaway
ended up getting shit on because they didn't actually give any out apart from one or two
then ended up getting shit on more whe nthey spammed the poeple they didn't gvie anything out to
chunk host...vomit host? sweet.
I'm unaware of that... They did offer (do still?) a "free" tier though, a way to pack host machines fuller, the instance was culled if detected idle for too long, and I think it was v6 only.
they paid for this ad http://www.reddit.com/comments/1lcxo4/get_a_superfast_cloud_server_with_ssd_hard_drives/
And I do know people who took advantage of that (briefly)
it was crapped on immediately
because it was a giant, blatant lie
heh
the title was SSD hard drives and 8GB of RAM for just $9/month. WHAT?! (chunkhost.com)
it was $9/monht and something like $2500 setup fee
Yep, you "purchase" the hardware
or some crap like that
except you dont' actually own any of it
and it's still shared
so that's a huge douchemove
Yep.
Unique idea, but only becausepeople don't usually share the bad ideas.
what i really want to know is
(Okay, it's not DNS that's slowing fping down)
what the hell was chunkhost thinking when they decided to put a blatant lie for an ad in r/programming and r/sysadmin
like, maybe minecraft or something with less competent people would work
but come on
https://scontent-a.xx.fbcdn.net/hphotos-ash3/t1/1656358_278087589008207_569417404_n.jpg one of hte positive aspects of working @ a tribal enterprise
what is that
indian tac!
*taco
i see beans and .. not sure if that's pita
well, in the long term, is $2500 setup fee and $9/mo cheaper than $100/mo or so for equivalent elsewhere?
Fried tortilla
vegeterian taco?
frybread
yeah
toddf: well it's an 8 gb vps, so yes
sounds like someone is playing with the pricing models to see who will bite at what pricepoints
and it's also oversold to death and complete crap
what is frybread
m0unds: looks like you got valentine's fruitsnacks too?
oversold, well, thats kindof par for the course with chunkhost right?
i wiki'd it but i'm not sure what its equivalent is
yeah, hahaha
http://en.wikipedia.org/wiki/Frybread
Frybread :: Frybread (also spelled fry bread) is a flat dough fried or deep-fried in oil, shortening, or lard. The dough is generally leavened by yeast or baking powder.[citation needed] Frybread can be eaten alone or with various toppings such as honey, jam, or hot beef. Frybread can also be made into tacos, like Indian tacos. It is a simple complement to meals.      History  According to Navajo tradition, frybread was created in...
is it like those things at county fairs that have tons of powdered sugar on them
or is that called something else
m0unds: I work in a 1-man office. Now I feel very alone :(
* nb: i've never travelled outside of ultra-urban areas anywhere in the US
hazardous: Ever been to a carnival/fair?
brycec: too bad the valentines fruit snacks didn't have TO: and FROM: written in :<
bummer. i rarely eat anything people bring, except today because i had no idea frybread was egg-free (i'm allergic to eggs)
brycec: nope, never
hazardous: so basically I have to put this in terms of taco bell, mcdonalds, and wendy's?
hahah
pretty much, yeah
i've seen a parade or two but those msotly consist of naked gay people and the sisters of perpetual indulgence here
m0unds: Okay, easy. IT"S TASTY.
m0unds: awesome, very lucky. I bet you get lunch for free too :(
like i'm trying to picture fried bread, but the only thing i can think of is french toast
i've had frybread before, but before i realized i was allergic to eggs (i grew up in CO/NM)
m0unds: wow, that must .. suck..
there was red chile too, but it had meat in it
doesn't that exclude almost everything made commercially
yeah
i remember seeing a lot of WARNING: MADE IN A FACILITY THAT ALSO PROCESSESS x,y,z,a
and it's something like nuts, wheat, eggs, gluten?
yeah, eggs are a common ingredient in stuff. it just means i eat less shit than most people, since crappy food tends to have a lot of egg in it
crappy as in lacking in any nutritional value
so we tend to buy vegan stuff mostly, as it doesn't require diligent label checking
i wish i could eat cilantro, but it tastes like soap and it sucks because i like mexican food and they never remember to leave it off when i ask
it's not an allergy or anything but it just tastes vomit inducing and revolting
maybe something is wrong with my tongue
yeah, cilantro is great
hahaha
DEFECTIVE
hazardous: a lot of people think cilantro tastes like soap..
they're WRONG WRONG WRONG
m0unds: no, it's actually in how they process the flavor
their processing of the flavor is wrong
cilantro is over-used as a crutch by crappy and wannabe "mexican" places
m0unds++
it is a genetic thing no?
it could be
http://www.huffingtonpost.com/2012/09/20/cilantro-aversion-gene-study_n_1901124.html
http://www.nytimes.com/2010/04/14/dining/14curious.html?_r=0
staticsafe: there is no formal proof that it is genetic, but that is what they are studying.  they suspect it might be.
I often hear "soap"  along with chemicals, hand lotion, etc.
(ie, it smells or tastes like chemicals)
A Japanese study ... suggested that crushing the leaves will give leaf enzymes the chance to gradually convert the aldehydes into other substances with no aroma.
cilantro pesto as a way to enjoy cilantro
oh man. http://www.huffingtonpost.com/2011/10/27/roasted-vegetable-tofu-a_n_1057197.html
m0unds: you're veegan?
food wise, yea
lifestyle stuff makes my head hurt
brycec: you can judge some dns servers to set minimum ttl
s/servers/recursors/
<mercutio> brycec: you can judge some dns recursors to set minimum ttl
s/judge/set/
<mercutio> brycec: you can set some dns servers to set minimum ttl
oh i need both heh
i shouldn't be on irc before coffee
wow ubuntu is going to change to systemd apparently
it's not april 1st hmm
lol
mercutio: But I've determined that it's not the DNS lookup choking that one host, so yay that.
Tested with time fping... a whole bunch of IP's, still took much longer on Chunkhost for no discernible reason (same results)
weird
you want to pastebin me the command?
i could see if  i can figure out what causing it
i found a ntt looking glass
I could, sure, but I don't think you'll see anything. I run smokeping on 5 hosts, and Chunkhost is the only one being poopy. </technicalterm>
but it seems to be going amazingly slow
http://www.us.ntt.net/support/looking-glass/  ?
yeh
@ddg ntt looking glass
m0unds: I'm sorry, the DDG Zero-Click API returned no results. As this is a Zero-click Info API, most deep queries (non topic names) will be blank. Quite frankly, I suck cocks.
Sorry, the Looking Glass is currently unavailable. Please try again later.
oh, that' sright
i thought it may show up in more detail the ntt issues
chunkhost along is making it take more than 60 seconds?
alone
as a destination?
mercutio: source
running http://sprunge.us/eAhL takes forever
^ on a chunkhost VM
forever in the linked case is 61s
oh that was from a chunkhost vm not to chunkhost
(And yes that's one with hostnames, it was quick/easy for me to post)
what do you get on arp?  from NZ it took 37 seconds
real    0m23.240s
nice
i should try agian
I try it every 60 seconds :P
yeh
so more are cached i meant
heh, in my debugging earlier, I tcpdumped the DNS lookups, didn't take more than 3s
^ On Chunk
oh curious
22.7 seconds second time in nz
I then tcpdumped icmp... that was a mistake :P
so dns definitely makes a diff when a whole lot of uncached stuff
haha
i use strace sometimes
Yeah, but only 50% in your case.
I did that too, didn't show anything obvious
you're doing 20 pings
But strace doesn't have timing
that means it'll take 20 seconds for the pings at least
way less diff between cached/uncached on arp
and 22.7 sconds again heh
nice
so chunkhost is blowing chunks?
what dns server are you using on chunkhost?
and what location is it?
mercutio: I was using HE's DNS earlier, switched to using Google's DNS. No difference.
mercutio: LAX1
(It's vps1.cobryce.com)
ahh
8.8.8.8 isn't close in la :/
maybe try 4.2.2.2?
i think he is close but not very good
Again: DNS lag has been taken out of the equation completely. Tcpdump says DNS only took ~3s. Trying without DNS lookups at all still takes 50+ seconds.
i did namebench ages ago
mercutio: ^^
(Oh my bad, I'm using Chunkhost's own DNS right now)
which ARE close :P
sbc global is the other one that was good
heh
maybe they throttle you
it's not openvz is it?
Must be something like that.
mercutio: Xen
that nz host was xen
xen should be fine normally
uhh
add -c 10
or something
mercutio: yes
but later
because I have important shit to get done when I'm on the clock ;p
cos ping -f could be considered abusive behaviour
ok
Network throttling is the only thing that makes sense at this point, but I don't have a good way to test for that.
smokeping? :)
umm usually http download and look at packet dumps
cos http has back off behaviour
it is recent version riht?
recent version of?
fping
there were soem updates at some point
just reading changelog now
And I was thinking they're only throttling icmp
i doubt it
good question, it's Debian Squeeze so no
fping sohuld timeout
is it 3.2+ ?
  * Performance optimization for big select timeouts (#10, Andrey Bondarenko)
i wonder if that make sa diff
fping: Version 2.4b2_to $Date: 2002/01/16 00:33:42 $
hmm
is arp using same version?
arp is 3.2
as are my other nodes
hmm...
i'll tackle that later
you can probably just install the later debian package
but good call
if it breaks it only breaks fping :)
newer fping not in backports, so there may be a reason
that someone didn't get around to it
you can often just wget the package and dpkg -i it
for single programs with no dependencies
real    0m34.709s
quite an improvement
heh
i like clean graphs :)
ok so fping bug :)
is mirrors.arpnetworks.com down?
weird it works from another location
ha! mercutio was slightly wrong, it did break fping slightly :P  ERROR: fping ('/usr/bin/fping6 -C 1 ::1') could not be run: /usr/bin/fping6: /lib/libc.so.6: version LIBC_2.15' not found (required by /usr/bin/fping6)
what version was that
glibc can be the biggest bitch
apt-get source fping may be necessary :(
if that's 3.2
fuck, nvm... In spite of dpkg giving an error when I tried installing fping 3.8, it went ahead and installed it.
it's whether smokeping minds
3.2 back installed and working
Fun fact: http://sprunge.us/LZAT doesn't mean that it didn't install the package anyways.
oh
i thought you were going to 3.2
3.2 should be fine
mercutio: I did, then I tried 3.8 for shits and giggles.
ahh ok
It shat more than he giggled
it works with ubuntu from debian packages too sometimes
https://github.com/schweikert/fping/blob/master/ChangeLog
I get the libc thing... I just figured "ERROR installing" meant "I didn't actually install it, you still have 3.2"
things are less interesting from 3.2 to 3.8
yip
but when you do other updates it can be a bitch
It's squeeze, there will never be another update to fping on it :P
no ot other packages
what does apt-get -f install say atm?
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
oh cool
it not complaining
ok
the real question is did smokeping fix itself
Once 3.2 is on, yes
3 minutes' data and counting
cool
m0unds: brycec i like the NLNOG; since you get an actual shell, you have access to more tools; way better than a looking glass
up_the_irons: what about ARP hosting a RIPE probe?
(I'm skeptical, but worth asking)
s/worth asking/curious)
<brycec> (I'm skeptical, but curious))
brycec: a possibility
https://atlas.ripe.net/targets/ringnodes/map/
whoa
there's already one at ARP!
that's from nlnog
ohduh
it's the probe target map
so RIPE hits NLNOG?
yeah, you can select those as user-configured probe destinations
or targets
they do
it's kinda buried
it's right at the top
Like RIPE has their own list of nodes
lol
it's the third line of text above the map "visit nlnog project" or something
hahaha
Yeah, but it's only ONE
the whole thing should be links
Why did RIPE make their own list and own map?
why should someone click off the site to look at the map when it's right there?
heh okay so you understand my point :p
BAD UX BRAH
why should someone take the time to scrape(?) another site for information when you could just send the user to the source?
BAD BRAH
heh
doubt there's scraping going on
maybe some shovelling
or they're paying some bloke to update it :(
the coordinator for nlnog is a commitee member at ripe
and he works for atrato
croneyism!
MUST BE
also, i'm guessing he's trying to get exposure for nlnog
but mostly cronyism
whoa, we're already in the Atlas target list
yeah, nlnog ring target
cool
brycec: there's a way to get output of root-servers locations
with dns
 dig nodes.l.root-servers.org txt +tcp |
brycec: just put in your Targets file with some modifications, smokeping takes a while to start up heh
s/|$//
s/\|$//
gah
but how's that for a list bryce
smooooooooooooke
it's ok to ping them i seem to recall too
i can't find confirmation of that though
it's kind of crazy how well the root serrvers are overprovisioned these days
i think it's pretty unlikely for the root servers to not be reachable, while the internet works now.
Fri Feb 14 15:51:44 2014 - FPing: probing 158 targets with step 300 s and offset 249 s.
Fri Feb 14 15:51:44 2014 - FPing6: probing 47 targets with step 300 s and offset 181 s.
hm
zomg
welcome to the smokeping age, staticsafe
i wonder how much icmp traffic is on arp from people running smokeping :)
mercutio: nice idea
hahaha
brycec: i was surprised that was out there.
it's all isc infrastructure afaik
lol well I'm currently responsible for about 150kb/s of ICMP traffic :P
TRIPLE IT
hm the smokeping webapp is pretty damn slow now
m0unds: You may recall I did more than triple it...
what is the secret to your speed brycec?
staticsafe: cgi? fastcgi? other?
staticsafe: fastcgi
im passing it through fcgiwrap
nginx
I'm not familiar with fcgiwrap
I'm running it from spawn-fcgi
I was responsible for at least 1mbps of ICMP traffic :P 15:54:42 < m0unds> TRIPLE IT
s'kb/'kbp
<brycec> lol well I'm currently responsible for about 150kbps of ICMP traffic :P
s/kbps/gbps
TRIPLE IT AGAIN
I'd rather not, thanks. I came awful close to 20% of my monthly bandwidth at that rate
And by awful close, I do mean still far from
brycec: boooooooo
staticsafe: here's mine http://sprunge.us/POVj
But I was on track to hitting 200GB/mo which is quite a bit more than I normally do
jeez
i got my burst usage up quite high once, which disconcerted me, as i wnated to stay under 200gb/month.  but it didn't last.
up_the_irons: "feature request" https://graphs.arpnetworks.com/ ought to redirect to https://graphs.arpnetworks.com/cacti/graph_view.php instead of giving a Forbidden error. Unless that's intentional?
i've used...140GB since the middle of nov
254.9GB in the past 12mo
s/9/69
<brycec> 254.69GB in the past 12mo
64.67GB according to vnstat. Hey what's the deal with this, is up_the_irons trying to rob me of my bandwidth? :p
... https://twitter.com/ScanBC/status/434469138325831680
TWITTER: #Coquitlam LaFarge park RCMP investigating a report of someone attempting to steal a duck (Fri Feb 14 23:28:09 +0000 2014, retweeted 123 times)
DAMNIT THAT'S MY DUCK
UNHAND MR. MALLARD
SIR QUACKSALOT WILL NEVER BE TAKEN ALIVE
http://brewvival.com/beer/whiskey-chub?utm_source=twitterfeed&utm_medium=twitter
want.
whee spawn-fcgi works much better brycec
brycec: well i don't think there's anything at graphs.arp*; everyone's Cacti graphs link in the Portal shows a complete URL
congrats staticsafe
up_the_irons: Maybe I'm not following a link from the portal ;)
Maybe I'm typing in the URL and expect to be greeted with the login
brycec: then put '/cacti' on it
maybe I will!
brycec: there are other things on graphs that are not meant for customers :)
heh, I thought that might be the case
but since i was JUST in that apache config updating the SSL, maybe...
lol
IT'S BEEN TOO LONG
redirect ^/$ /cacti/
I think
(because it's been so long since I've had to much with Apache. NGINX FOREVER. Also lighttpd is okay.)
nginx por vida
staticsafe: The reason fcgiwrap was slow is because it acts as fastcgi to nginx, but it's still executing the cgi for each and every request.
buddy of mine works for the utah MLS (real estate MLS) and lighttpd really shines at delivering static content on their web frontend
it was nearly 15% faster than nginx w/the same content
staticsafe: whereas fastcgi loads and runs once and leaves it running for multiple queries
faster than nginx? Wow
yeah, that's the only load where that was the case too
brycec: i think that would also rewrite other things
he uses nginx on everything else
up_the_irons: the pattern would match requests to / only
up_the_irons: would not match /pr0nstash for instance
oh
m0unds: Is it "dynamic" static content? ie. something that could pass back X-SendFile? I've found that to be wicked fast in nginx.
*yoink*
E_CONNREFUSED looks like graphs is down and up_the_irons has lost is /pr0nstash
*his
brycec: i have no idea
Well, good for lighttpd anyways.
When I abandoned Apache, I went to lighttpd. It was a learning curve... But I liked it and its configurability over Apache immensely. When I upgraded to Wheezy, I moved to nginx and I'm instantly in love. I can't explain it, but I love nginx.
brycec: there, it works
up_the_irons: yep the redirect works. And now I have a Cacti "access denied" page
i like contextual configs a lot
that's all the Apache work I will do for a week
And clicking login gives me access denied again
stupid Cacti
nginx configs remind me of junos, so i like them
hahaha
lol
brycec: hmm.. i can login
up_the_irons: going directly to my graphs.php URL works fine
up_the_irons: So I'm gussing it's user permissions
the only time i used lighttpd was just as an exercise to see how it worked, and that was just to handle static images for a site
"bryce is not allowed to see the Cacti index"
lighttpd and nginx configs are similar, which is good
i returned to my graph url and i'm seeing: GRAPH DOES NOT EXIST
(by no means interchangeable... but similar, esp. comared to Apache)
lol m0unds srsly?
yeah, just reloaded it
haha
ok, logged out and in and it's ok
haha
i dunno if you're able to go to graphs.php directly
 /graph_view.php
is the one that works
and is what you get redirected to after login
argh, lastpass broke form filling again
dammit
i'll leave it this way for a bit and if i get complaints, i'll just put it back
to the way it was
Yeah must be cacti perms. If I logout, I'm greeted with the ARP login page for Cacti at "/cacti/", if I'm already logged in and visit "/cacti/" Cacti gives me access denied and links me to login.
and brycec will simply have to type "/cacti" ;)
but that's so much work!
i've had cacti permission issues on multiple different cacti instances.  i think cacti has some kind of cookie or such bug.
(mmmm cookies)
sometimes if i login to cacti, and click on a graph, it will bring up the login again.
and then it'll work
brycec: i can't imagine it would be any different from before if you hit '/cacti' directly.  i mean, the redirect just redirects to that way...
*anyway
it worked fine for me - logged out of the portal, then logged in and went through the links to get to graphs that way
logged in to cacti, saw my vlan and my graphs
up_the_irons: I'm sure it's exactly the same as if I hit /cacti directly.
k
heh /home/garry/www/cacti/rra/
now we know where the RRDs are kept
Can anyone else from an ARP IP ping hitchcock.freenode.net ?
According to https://smokeping.cobryce.com/?target=Internet.FreeNode.hitchcock4 it has dropped off the map for two hosts, both hosts traceroute through NTT. (And another working host traceroutes through peer1+init7)
nada here
can't get to dest
i can't get to it from home either
it's in bulgaria?
it may be the destination
can't telnet on port 6667 too
i'd say the host is just offline maybe due to ddos or hardware failure
maybe, but I can reach it from a TWC and Comcast connection :p
well i'll still blame them
actually maybe it ntt
quite a bit of stuff to me returns via ntt
yeah, it's fine via comcast for me
via level3
well my forward route from home is via cogent :/
Morelikely NTT, http://sprunge.us/aWhZ
but dunno what reverse route is
What's important to me is that it's not my Smokeping that's broken :p
ok
well i don't think it is
I didn't either - that's the purpose to having a diverse set of slaves - but I wanted to check anyways
SLAVES
Hooray for slaves
There, I said it.
For history and ignominy
@log_search slaves
11 results found. Here's #6 Feb 14, 2014 16:49:29 <brycec > I didn't either - that's the purpose to having a diverse set of slaves - but I wanted to check anyways
i can reach hitchcock.freenode.net from my smokeping master but not from my 2 slaves
ntt is unable to reach it via their sofia pop (which is the last one i see in my traceroute before it fails)
ntt looking glass is working again?
yep
Ahaha I wonder if Comcast is doing this intentionally (and yes it's only the Comcast slave) https://smokeping.cobryce.com/images/__navcache/139242549325735_1392425493_1392382260.png
it's gross htough
though
heh
hahahah
"it's that brycec guy again.."
it's not packet lossy
It's actually pretty regular... https://smokeping.cobryce.com/images/__navcache/139242558525735_1392425585_1391820660.png
hahaha, awesome
To be clear, I was accusing Comcast of "throttling" Netflix
looks like verizon graphs :/
brycec: accuse netflix of not having enough capacity
actually verizon changed a little
it seems shorter now
i don't think comcast participates in that cdn program they have though
mercutio: I would, but it's perfectly stable on TWC, Orbit, ARP, and Chunkhost
Actually, I think they must
why does spawn-fcgi just keep dying? o_o
so is verizon on non ntt forward path :/
staticsafe: because aids?
you have to feed it or it starves
you can't just spawn things willy nilly
they'll die without care
it just depends if you hit some congested link or not
lol m0unds
but at least verizon got better partially
"better" is a relative term w/vzw
err
vz
probably vzw too
i think things will get worse over tiem not better
yeah, but i didn't want to confuse anyone
ddos attacks are becmoing more and more common
vzw is garbage in the SW US because they're the dominant carrier and are super oversubscribed
no reason to change?
that'd cost money
VZW is dominant in the SW?
welcome to net neutrality
In the PHX area, it was not.
now you can have degredation to everywhere
it is in CO, UT, NM, and NV
for sure
no idea about PHX specifically, but i know more people and businesses that use vzw in flagstaff and tucson than anything else
My impression growing up in PHX was that ATT was dominant
a ton of their subs in NM and CO are former alltel subs
Oh well that's sortof cheating :p
"I'll become dominant, not by people choosing me, but buy buying up those that people chose.)
it's how the world works
unfortunately
(Reviewing more of my Smokeping graphs, it's clear that Comcast has a few saturated routes. It's not limited to Netflix.)
Comcast<->Cogent, Comcast<->Tata
heh
i had an idea about that btw bryce..
brycec: it's because alltel's rural network was much larger than VZW or Sprint's, and VZW was having to eat tons of roaming charges (their customers just see everything as native)
if ttl changes it probably means route changes
so if you don't want to do heaps of traceroutes, you can prioritise when ttl changes
it wouldn't always work
heh, I could...
For now, I plan to run traceroutes hourly
and would do some false positives
i been planning to do my own monitoring thing
what was the ping target for that ramped comcast graph?
but there's a lot of work involved...
so i mostly just been thinking about it so far
m0unds: netflix's nlnog node
Just need to come up with a decent UI, and fix the config parser.
i want to start by doing passive monitoring of normal traffic
active monitoring doesn't necessarily hit the things that are failing
and it can be hard to know what to monitor
good luck :)
i think one problem with heaps of hosts is it's hard to get an overview of the state of things
if it can come up with something like "paths out via ntt seem broken"
it could be useful
and so if traceroute starts feeding out bound paths, you can start collecting information
and have some intelligence
So it would need to have some intelligence to udnertand the paths being taken
but return paths is still a pain unless you have multiple targets doing the same in reverse
yeh
traceroute -A may be the simplest..
i want something that can work in a couple of minutes
when there are isuses
and looking at smokeping for that is hard
so i'll probably start with just measuring at the time
using a mesh system to be able to communicate with hosts even if they're unreachable by one route...
http://krebsonsecurity.com/2014/02/the-new-normal-200-400-gbps-ddos-attacks/
there's also snmp problems
that aren't being exploited yet
i suspect if ntp gets widely patched they'll move on
there's probably some unknown vector too
i'm surprised there aren't bots that try and buy things on amazon or such from peoples hosts using generated credit cards or such
Isn't there always an unknown vector?
yeh
it seems most attacks target networks atm rather than hosts
that have lots of db activity etc
it's easier to block network attacks atm i expect
more people are ready and waiting to respond
whereas application level stuff could lead to longer outages.
amazon probably has people on hand though
but i imagine lots of banks don't for instance
well not at short notice at 3 am etc
nah, they do
all of them?
that's silly
i suppose large ones will
but what about those little in between things
i know people who work at local banks and national banks, and both have people on call to respond to that stuff
they don't necessarily have people with expertise to be able to figure out who is real and who isn't
and they can do application level filtering?
maybe i underestimate
you do
absolutely
that country where banking is really strong got hit by ddos's hard
slovenia?
somewehre near there at least
sweden?
it was somewhere with really high internet banking penetration
so probably somewhere cold, but i don't think it was sweden
they had government enforced id cards i know
that had crytographic pgp type keys
for identifying
so it's like everyone of their population had a pgp key
Lithuania
maybe have been there
https://www.lb.lt/cyber_attacks_were_used_to_interrupt_the_provision_of_the_bank_of_lithuanias_online_services
maybe it was that
not finding out much in search but that my country has hackable transport cards :)
ahh estonia
http://en.wikipedia.org/wiki/Estonian_ID_card
Estonian ID card :: The Estonian identity card (Estonian: Isikutunnistus) is a chipped picture ID issued in the Republic of Estonia by the Citizenship and Migration Board of the Ministry of Internal Affairs. It is officially a primary-picture ID in Estonia, and is therefore recognised by all member states of the European Union or the Schengen Area and some other European countries as an official travel document. For travel outside the EU...
Today 99.6% of banking transactions are done electronically and the number of users of the Estonian Internet banks is more than 1.8 million clients, a bit more than the whole population of Estonia ??? 1.3 million.
99.6%?!
uhh
most older people don't seem to like electronic banking
my in-laws only recently started using any online banking
exactly
apparnetly only 75% of people are on the internet there
but i suppose 99.6% includes non-internet electronic too
and 75% is still damn high
it's a country i know very little about
but they seem to be quite advanced techonology wise
i imagine it's not as hard to push tech when your country's population is that tiny
not necessarily
i wonder if they're rich
it'd likely be easier to push stuff like that into service in a country w/1.3m population than it would be to push it into service in a country with half a billion poeple
people
takes just as much effort to create
but distribution is easier
heh
probably more that there's less things in the way
exactly
apparently they're the richest east european country
cool, tallest midget
i never really hear about east surope
europe
GDP of estonia is (according to 2012 data) 20bn USD
is that high or load
nominal
they're near east europe so they can prob get some stuff cheap
nz is 140 billion
with like 4 millionpopulation
my state is 80bn with 2m pop
http://www.numbeo.com/cost-of-living/country_result.jsp?country=Estonia
it doesnt' loko cheap
but i don'w know value of euros exact
change the exch rate
hmm internet is like $24usd/month
err, exch currency
oh cool
these sites are often not very accurate i've found
when i look at my own country
"And everybody looks like a hobbit"
https://www.youtube.com/watch?v=AGF5ROpjRAU
YouTube Music: "Leonard Nimoy - The Ballad of Bilbo Baggins [FULL VERSION] - best quality" by SputnikMonkey (2m 21s), 1,147,193 views, 9,488 likes and 105 dislikes. Uploaded 2011-09-07T06:23:34.000Z.
it's a bit hit and miss
they have amazingly cheap rent epr month here compraed to what is normal
so some people look like hobbits and some don't?
no hobbits here
m0unds++
sigh. http://krqe.com/2014/02/07/recent-nm-skinwalker-photo-ignites-fear/
Stay New Mexican, New Mexico.
yeah.
like the kinda shit that you'd hear about on coast to coast
heh
https://fbcdn-sphotos-a-a.akamaihd.net/hphotos-ak-prn2/t1/1891280_486670144770622_1185801086_n.jpg
hahaha
Seems uncomfortable
yeah probably not pratical
but if it's just to scare your neighbours :)
i'd have to do it with red paint because we haven't been getting any snow
sigh
damn
it doesn't snow here
we usually get a bunch of snow from dec-feb
we just get rain
but it's gonna be 24C tomorrow
i don't particularly like the snow used to get in my old city anyway
there was never enough to be fun, just enough to turn to slosh
24c in winter?
yeah
nice
ridiculous
it's 24c here right now
@weather ABQ
There is 1 weather alert in effect for your area! There is a Fire Weather Watch.
Albuquerque International Sunport, NM: Mostly Cloudy ☁ 62°F (17°C), Humidity: 23%, Wind: From the NNW at 12 MPH -- For more details including the forecast and almanac, see http://www.wunderground.com/cgi-bin/findweather/getForecast?query=35.04166794,-106.61472321 or re-request this with: @weather -v ABQ
@weather Kiwi-land
oh i read that at first as a "fine weather watch"
Error, No cities match your search query
@weather auckland
Auckland, New Zealand: Clear 72°F (22°C), Humidity: 60%, Wind: From the SW at 16 MPH -- For more details including the forecast and almanac, see http://www.wunderground.com/cgi-bin/findweather/getForecast?query=-37.00805664,174.79167175 or re-request this with: @weather -v auckland
oh it says 22c.
i am sure the wind is not 16mph
@eather waitakere
@weather waitakere
Waitakere, New Zealand: Clear 74°F (23°C), Humidity: 62%, Wind: From the NE at 1.0 MPH Gusting to 4.0 MPH -- For more details including the forecast and almanac, see http://www.wunderground.com/cgi-bin/findweather/getForecast?query=-36.857498,174.520554 or re-request this with: @weather -v waitakere
lol
windy in the big city
it's not really a separate city, it's just a part of the city.
@weather 99019
There is 1 weather alert in effect for your area! There is a Areal Flood Warning.
Liberty Lake, WA: Clear 44°F (7°C), Humidity: 72%, Wind: From the ESE at 6.9 MPH Gusting to 10.1 MPH -- For more details including the forecast and almanac, see http://www.wunderground.com/cgi-bin/findweather/getForecast?query=47.666508,-117.100792 or re-request this with: @weather -v 99019
i read that as "waithere"
i think it's wai-tak-a-rear-e
s/wai/why/
<mercutio> i think it's why-tak-a-rear-e
Don't you know? Aren't you there?
i think wai is probably thought of as whey normally
You know, just for that I'm going to put on Flight of the Conchords
i'm not from around here
i live here
lol
lots of places are miscomonly pronounced here
err commonly
gah commounly mispronounced
my brain don't work right
weather underground is forecasting a much more (still not terribly) reasonable high tomorrow
of 72F vs 77F from accuweather
Why-tahk-ah-reh)
see i was wrong
it's becuase they use native words, and where i was from didn't haev natives
http://www.shelleymunro.com/blog/2010/03/04/place-names-that-roll-off-the-tongue/
how's that for confusing
I'm sure it's perfectly easy once you're accustomed to it.
Hawaiian names are similarly frightening to see, but easily picked up
i dunno, news reporters here mispronuce placenames
god i'm mistyping words even today
some place names have actually hard to say names, because they use uncommon speech
there's a lot of wonky tiwa (rio grande valley-area dialect) native names near here
p'oswaugueh, tesuque, ohkay owingueh, tlur pa, ba shi'ie
pee-o-swaga?
no idea about second
tee-suck?
teh-soo-kay
tee-sook?
ahh
i like the third one
okay oh-winga?
okay ohwingay
oh-ween-gay
i'm terrible at this
the spanish version of the first one is pojoaque, po-wah-kay
tlur-pa is t-lur pah, last one is bah shee-eh
http://www.itnews.com.au/News/372345,cisco-stung-by-switch-memory-glitch.aspx
uhh
sounds nasty
i wonder if that is at all related to up_the_irons issue :/
(cos it showing some memory issue, it hard to know if OS bug or memory problem though)
hmm
HMMMMMMMM
it's kind of scary, 7 years of memory problems
yeah
i still think it was an IOS bug.  after removing the Any2 port, no more crashes.  with the Any2 port, crashed 3 days in a row within 5 minutes of 05:30 PST.  WAY too much of a coincidence for memory problems.
that said most of these problems seem to be triggered by power cycling
yeh i think it probably ios bug too
i wonder who cisco uses for memory
omfg.
that's the issue i had with 3 whole stacks of 2960s in july
good thing i don't power cycle my cisco's much ;)
m0unds: ouch
all had multiple-years' worth of uptime, all went down because of an enormous systemic power failure during a huge storm
m0unds: looks like you'd get free replacement memory / gear now :)
do you still have them?
i suppose you just claimed insurance?
we still have 30 of them in service - i revived 9 out of 12 of them
nah, they just did RMAs
or does that not work there
it was just a power down, no voltage craziness or anything
ahh
2960 is confusing, there are multiple versions of them :)
through many years
yeah
when i think 2960 i think old
but they're still selling new ones
mine are....WS-C2960S-48FPS-L
3 years old
s is newer than g i think
yeah g is 2005
m0unds: so they powered down b/c of the power failure then refused to come back up properly?
that is f'in scary
it was a pre-emptive shut down?
or just protected feed?
cos you can still get spikes sometimes that are short
managed shutdown
eek
prior to UPS gear running out of juice (generator failed to start, it cooked its battery harnesses)
and yeah, powered down and when it came back up, it would dump the second IOS finished booting
full on crash and boot loop
up_the_irons: are you using 2960s?
mercutio: no
just straight into router?
mercutio: yeah our topology is very flat
nice
i don't like 2960s :)
VM -> VM Host -> (trunk port) -> s1.lax (or s7.lax for the GigE ports)
so when there's two ports on dedicated servers they both go to s1?
directly in?
or do they have switches?
oh, no, the dedi's have an edge
s8.lax, s9.lax for dedi's (each have dual 1 Gbps, to each switch), then to either s1 or s7
ahh
thanks so much for linking that stuff (the blog post and whatnot)
just passed it on to our casino ops IT director and neteng guys
cool.
it sounded like another of those times when buying things at different periods of times or from different vendors could help
it's like when peopld do raid with mirrors, and all their hard-disks come from the same lot
and they all fail at once
yeah
i use different vendors ;)
but it's a pita to multivendor
1x Hitachi, 1x WD RE4
yep, especially with a time-constrained large project
(pair)
ahh cool.
also, we got our 2960s from 3 different vendors
i didn't realise you were doing that
i think our 6500s came from two different vendors too
but same timeframe
it can be hard to get diff lots.
yep
i have my ssd's with different lots :/
well at least diff looking model numbers
we had a bunch of cisco-brand SFPs just croak this year too
it's been a rough year for cisco at my facility
haha
why use cisco sfp?
oh you cisco shop?
on top of our IT/ops guys having tons of issues with their new 7000-series stuff
because it was what was specced
heh
if it was up to me, we wouldn't have used any cisco gear, but our VMS vendor prefers cisco and won't certify installs that don't use cisco gear
cisco tax is quite a lot on sfp's though
yep, it's absurd
m0unds: i heard bad stuff about the 7k series too
i mean i'm more ok with cisco where it matters.
not that i necssarily agree iwth it, but i can understand not wanthing to change.  buf sfp isn't going to be a problem with non cisco
up_the_irons: i've been hearing they're already trying to forget about it in favor of the newer 6xxx chassis'
have they considered changing vendors?
well, cisco actually changed suppliers for their sfps because of rampant failures
apparently huwai are trying quite hard to compete with cisco now days. not that i know anything about them
but juniper have qutie a bit of market share now
huawei tried really hard to reverse engineer a lot of competitors' products because their own os and stuff was garbage
apparently juniper doesn't have smp support yet
m0unds: the newer 6xxx being the 65xx-E or..?
i was surprised.
oh and apparently openbsd is adding smp support in kernel now
they done audio, now doing disks and network interfaces i think, before making the network and disk interfaces smp..
cos one of juniper's present-day issues is being slow at taking bgp tables, reconverging etc.
however much cisco may have issues, for the cpus they use they have a damn fast bgp implementation.
i mean you were saying bird is fast up_the_irons .. but the cpu on that bird machine i bet is way faster than the cisco.
mercutio: well that's true too :)
rpki is going to slow bgp down soon too if people start implementing it
that and when ipv6 takes off more
up_the_irons: 6800
m0unds: ah
kinda weird considering the nexus stuff should just be better :/
should
they are different platform
but such horrible teething problems with it on the software side have wrecked credibility
would be nice if this stuff could be open source :)
wher epeople still might pay to get issues fixed
or for support
facebook was doing some open switch thing, i think that's as close as it gets
hmm ok probably expensive, it's 48 10 gigabit ports and 4 40 gigabit ports
shuttleworth announced ubuntu will now go with systemd.
i heard of a juniper (or was it cisco?) version of that
grepidemic: godamnit
http://www.opencompute.org/blog/progress-in-opening-up-the-network-/
so i guess upstart is gonna end
grepidemic: i actually said this earlier today :)
but yeah, it looks legit
well, i didn't like upstart much either.  so we're going from "meh" to "meh"
i won't panic
it is on shuttleworth's b;pg
i assume they don't want to focus on maintaining upstart
too busy with mir and ubuntu phone
there's mellanox intel boradcom and cumlus networks all doing open soruce switches
i don't know much about mir. i am guessing it is mostly for mobile linux?
it's for desktop too
it's like wayland but different
with a lot of ambiguity about why
https://github.com/Mellanox/SwitchX-interfaces/tree/master/SwitchX-2/source
the only things i know about wayland are from a conference video 1 hour. the guy is mostly talking about how it is better than X.
grepidemic: yeh i heard about better than X stuff being necessary for 15 years
but X is still around, and doesn't seem to be struggling that badly
i have no problems with X. but maybe i'll disagree with myself after using wayland.
X can be kind of bad over high latency links
but for local desktop it's fine
it may be more efficient whatever they're doing
but i assume they'll cover that all up in layers of abstraction.
there were some security concerns about applications being able to impede other applications once upon a time, i assume they're still valid.
i know netscape 4.7 used to sometimes lock X up until you sshed into from remote host nad killed it.
i think wayland will have something similar to "ssh -X" which will be more efficient than X11 forwarding.
i was playing with X forwarding the other day.  skype seems to resize slowly when using it.
but other than that it seems just like being local.
but that's on lan.
on wan it still sucks.
but i think people moving to web for that stuff now anyway
maybe desktop OS's will just become a client for web apps.
like chromeos?
and everything will be cloud.
haha
now i know you taking the mickey.
the new MS guy is a cloud person.
my condolences.
heh
well private cloud is fine
i never used chromeos, but i'm not really too fond of cloud services. i like using a vps for storage and running my own services, but i don't like the idea of EVERYTHING being stored over the internet.
it seems even technology focused companies are sometimes using things like AWS now.
grepidemic: i kind of do.. but..
as soon as your net goes down you lose everything.
bbl
the only cloud thing i use (that im aware of) is Steam client.
haha. http://t.co/S3ZKrxcf2g
http://t.co/S3ZKrxcf2g -> http://twitter.com/HistoricalPics/status/434488522704441344/photo/1
steam's cloud sync is terribly unreliable
but their normal stuff is just straight cdn i'm prettty sure
origin does a pretty good job of detecting and resolving local/remote save conflicts
steam just says "there's a conflict, if you start the game you'll lose your save"
How can there be a conflict? It's not like you can play the game in two places at once
like..you were playing and the game crashed, and steam didn't sync
so your save is different than the one in TEH KLAUD
Huh. Odd.
yeah, only game i've seen it with was the walking dead (telltale)
it ended up eating my save so i had to start over
It's always appeared as the sync was part of Steam, not the game, and when Steam detects the game has closed, it syncs the relevant data. But that's total speculation based on what I've seen (notably, the "sync" popup is Steam's, and it shows in the Steam "Downloads" page briefly)
Bummer
yeah, i think that's how it does it
it has like a local glob of save data
and it pushes it up when the title closes
but when your game mangles it or it gets mangled in transit somehow, then you have that weird condition that might lead to losing the save altogether
i just flipped off steam cloud saves for that game and set spideroak to back it up since it versions files
m0unds: not ime
s/versions files/saves versions of files/
<m0unds> i just flipped off steam cloud saves for that game and set spideroak to back it up since it saves versions of files
but i dunno my games are stuck in not syncing
dunno what to tell you
blow the hobbits out of your cpe
i dunno i just choose don't sync
then next time i choose don't sync again
the syncing is just key bindings etc for things like dota2
i did kind of want my key bindings, and once it actually worked.
steam has various issues
like i used to find every time i started steam it wanted to download a huge update to do with big picture
which i don't even use.
lol, a nuissance but not an "issue" :p
they just keep their program up to date
now i play dota2 heaps i use steam way more, so steam updates seem less frequent, but dota2 updates can still be multiple updates a week
well it's like 80mb+ download i think
i like it how chrome got their update sizes down.
i'm not necssarily opposed to updates, but when they're huge, and you have to wait for them at startup, they're kind of annoying
chrome is good, because it doesn't make you wait for the update, and they're small.
https://twitter.com/bsdvps/status/434532130480082945
TWITTER: OpenBSD 5.4 is now available to order (Sat Feb 15 03:38:27 +0000 2014, retweeted 1 times)
*finally*
now if only ubuntu would do dif udpates
Just as 5.5 is on the verge
5.5 is going to break things
Yes :D
Can't wait
it's suggested that people wanting to update soon go to snapshots atm
which is curious
(if i really couldn't wait, I'd install a snap...)
i'm using 5.3
Why not 5.4?
why 5.4?
 4:50PM  up 257 days,  4:15, 1 user, load averages: 0.19, 0.17, 0.16
Because you're s'posed'a upgrade! :P
Also 5.3 will be dropped from most mirrors once 5.5 is out
i dunno i have one host much further behind
not to mention "support"
only if you have problems
now i have to do FreeBSD 10
i quite like freebsd 10
FreeBSD doesn't really matter... :P
how do you find out openbsd release?
oh right uname -a
ok this host is openbsd 4.8
uptime of over 1000 days
going to replace hardware and shift to freebsd 10 though
the mirrors thing is intresting
i haven't installed anything on it recently but id idn't have issues getting packages last time i did
curiously where i was installing packages has 4.9 packages but not 4.8
PKG_PATH=http://mirror.internode.on.net/pub/OpenBSD/4.8/packages/i386/
oosp
PKG_PATH=http://mirror.internode.on.net/pub/OpenBSD/4.8/packages/i386/
gah
cut and paste issues
http://ftp.vim.org/OpenBSD/
that has old openbsd to 2.0
freebsd 10 has been good so far - only issue i had was w/pkgng which i don't really use too frequently
i reading openbsd 5.4 changes
nothign really strikes out as requiring me to update
i liked pkgng
i hit a compile time bug with trafficserver
due to clang
but that was fixable, and due to bad code
http://www.openbsd.org/cgi-bin/man.cgi?query=ppoll&sektion=2
the only pkg i've installed was to work around a compile time issue with something, i don't remember what
actually that looks cool
and is new in 5.4
i might jump to a snapshot of 5.5 now
hmm em0 watchdog timeout
disable mpbios
it's in the installer
is it just mpbios that causes that?
Oh, weird.
i'm pretty sure i had it a while back without mpbios
Ive never had it hapen in the installer.
maybe it was doing it before, and i didn't do enough network traffic
I just installed a snapshot a few weeks ago.
i'm downloading from mirrors.arpnetworks.com
ahh cool, i doing snapshot
figure it's not worth jumping to 5.4
i normally don't do update using bsd.rd
but 5.5 requires it, so i figure may as well.
err encourages it heavily.
I always use bsd.rd.
it varies for me
I can't think of a reson not to, as long as you have a bootloader installed, and network access.
hmm got another on boot
and some pf syntax errors
damnit, it wont' login
first time i've had an issue actually
oh 5.5 i smeant to not actually run earlier binaries iirc
so it prboably doesn't liek my zsh
zsh is for wizards
YER A WIZARD m0unds
ahahahah
i love zsh
heh so do i
damn
still not working
so it's working for you CaZe ?
ohhh
my fault
i didn't read instructions, apparently you have to do some stuff upgrade doesn't do
network isn't working at all hmm
caze: are you on virtio or legacy?
@smokeping
https://smokeping.cobryce.com/
@uptime
Bot uptime: 15 days, 21 hours, 45 minutes, and 17 seconds.
@mnathani
mnathani: Have you checked whether Windows Firewall is enabled and dropping packets?
451 results found. Here's #366 Feb 09, 2014 13:06:24 <mnathani > 5 packets transmitted, 0 received, 100% packet loss, time 4101ms (  0.v.freedaemon.com )
@uptime host
host uptime: 141 days, 21 hours, 58 minutes, and 44.800000000745 seconds.
oh that was just needing to disable mpbios
precision
almost get to walk a quarter mile to my car. muahahaha.
i'm half tempted to shfit to freebsd now :)
apparently downgrading to 5.4 isn't supported too
doh
Downgrading is never supported by OpenBSD. Might be "possible" but never "supported."
i'm hopeful i can make it work anyway
i think it's something to do with the password database
step by step..
it seems i wasn't forcing shell right
woot got it
mercutio: legacy
it was just mpbios making ethernet fail in the end
i've used self-compiled kernels for ages so haven't really noticed
i dunno why installer works
and editing /etc/passwd and /etc/master.passwd didn't seem to work to change shell from zsh
in the end i just upgraded zsh
You have to regenerate the db.
i did
that's the strange thing
otherwise i think things wouldn't work at all?
i did pwd_mkdb i think
yeh pwd_mkdb
Well, there's vipw.
weird, now it works
bbl
i ran it again and it set it to zsh
yeh i normally do vipw
i think openbsd 5.5 is slightly quicker