[03:21] *** phlux has quit IRC (Ping timeout: 264 seconds) [03:28] *** phlux has joined #arpnetworks [06:59] *** mhoran1 is now known as mhoran [07:03] *** avj has joined #arpnetworks [07:34] oooook [07:35] oh, didn't realize the routing to arp via comcast went back to normal on the 12th [09:05] Yes, about the same PPS as when I had 200+ targets with 500B pings. (86pps or so) [09:05] ^ 21:35:19 < mercutio> iyou're probably still doing a lot of pps [09:06] I don't recall when I added packetsize=500 but it wasn't yesterday. Might have been two days ago when I added a bunch more hosts, might have been earlier. 21:35:26 < mercutio> were you at defaults prior? [09:07] Today (now yesterday) went from 500B to 32B pings 21:36:03 < mercutio> whther you'd shifted to 500 byte packets today [09:07] And yes, I'll have to figure out the DNS woes. Someday. [09:08] But for now, we're settling for using better DNS servers. Hopefully we'll see improvement. [10:37] haha [10:37] i just use IPs for smokeping stuff and an external monitor that uses dns to resolve the hostname [10:39] I would too. However, some of the hosts I monitor are known to change their IPs [10:40] mercutio: One possible solution for pseudo-caching would be a script that [daily, hourly, whatever] did the DNS lookups and dumped the results in /etc/hosts [10:40] ah, gotcha [10:40] At the very least, it wouldn't hold up FPing [10:41] m0unds: Things like Tunnelr nodes, HE endpoints, my gateways and DNS servers are all easily static. But mirror sites and nlnog ring nodes may change [10:41] Or at least, there's no amount of guarantee they won't change. [10:42] :( better DNS servers did not fix Chunkhost's gaps [10:49] Oh cool! A useful log message! FPing: WARNING: smokeping took 85 seconds to complete 1 round of polling. It should complete polling in 60 seconds. You may have unresponsive devices in your setup. [10:49] It doesn't tell me WHAT or anything :/ [10:50] But it's a start [10:53] I think I'll stop monitoring mirrors. [10:53] They were just there to provide more geographical data, which nlnogring is doing too [10:54] nlnogring? [10:55] oh, that [10:55] heh [10:57] https://atlas.ripe.net/ [10:58] ^ eh, up_the_irons? [10:58] that project seems less skeevy to me than a shared shell [10:58] but i'm paranoid [10:59] haha [10:59] not to say that anyone /would/ abuse it, but meh [10:59] I like the transparency of NLNOG. I don't know *what* I'm hooking up to my network with RIPE [10:59] And if the RIPE device/network is compromised in some fashion?? [10:59] At lest with the shell, I have eyes on it, I can monitor it etc [11:00] And I suspect it's easier to kill a VM than to shutoff a switch port in some remote DC [11:00] well [11:01] Speaking as someone that admins a shared shell, with the right configuration, you've nothing to worry about. [11:01] conf t >> int gi1/0/1 >> shut [11:01] assuming you have access to the switch in question [11:01] which, if you're a network op, i'd assume you do :) [11:01] Of course you also have to connect to the switch :p [11:01] that's a given [11:01] Or if you're up_the_irons, I'm sure he has some management portal up and ready to click the right name. [11:02] (Switch: you also have to figure out the right port) [11:02] (and login) [11:02] But I digress. Hooking up a little black box to my network makes me nervous. [11:03] Also nlnog++ for just requiring a VM, and not requiring you to rack hardware [11:03] request remote hands to cut the cable, CUT IT [11:04] DO IT NAO [11:10] the old gen ripe dongle is 2x3x1" [11:11] http://probev2.ripe.net/ [11:11] lolol [11:12] "If you find this device and are unaware of its purpose" [11:12] http://probev3.ripe.net/ [11:12] shit if I came across that in a closet, I'd be *pissed* (at least until I knew what it was [11:12] yeah - NSA probably loves those little embedded linux server dinguses [11:12] http://www.lantronix.com/device-networking/embedded-device-servers/xport-pro.html [11:18] That is insane... The same fping takes 23s on ARP, takes 59s on Chunkhost. [11:18] (oh 61s that time) [11:22] i had a chunkhost box once [11:22] it was some kind of slow, horrid abomination [11:22] lol [11:23] In the very beginning, they were cool [11:23] i think [11:23] they did some free server giveaway [11:23] ended up getting shit on because they didn't actually give any out apart from one or two [11:23] then ended up getting shit on more whe nthey spammed the poeple they didn't gvie anything out to [11:24] chunk host...vomit host? sweet. [11:24] I'm unaware of that... They did offer (do still?) a "free" tier though, a way to pack host machines fuller, the instance was culled if detected idle for too long, and I think it was v6 only. [11:25] they paid for this ad http://www.reddit.com/comments/1lcxo4/get_a_superfast_cloud_server_with_ssd_hard_drives/ [11:25] And I do know people who took advantage of that (briefly) [11:25] it was crapped on immediately [11:25] because it was a giant, blatant lie [11:25] heh [11:25] the title was SSD hard drives and 8GB of RAM for just $9/month. WHAT?! (chunkhost.com) [11:25] it was $9/monht and something like $2500 setup fee [11:26] Yep, you "purchase" the hardware [11:26] or some crap like that [11:26] except you dont' actually own any of it [11:26] and it's still shared [11:26] so that's a huge douchemove [11:26] Yep. [11:26] Unique idea, but only becausepeople don't usually share the bad ideas. [11:26] what i really want to know is [11:27] (Okay, it's not DNS that's slowing fping down) [11:27] what the hell was chunkhost thinking when they decided to put a blatant lie for an ad in r/programming and r/sysadmin [11:27] like, maybe minecraft or something with less competent people would work [11:27] but come on [11:28] https://scontent-a.xx.fbcdn.net/hphotos-ash3/t1/1656358_278087589008207_569417404_n.jpg one of hte positive aspects of working @ a tribal enterprise [11:28] what is that [11:28] indian tac! [11:28] *taco [11:28] i see beans and .. not sure if that's pita [11:28] well, in the long term, is $2500 setup fee and $9/mo cheaper than $100/mo or so for equivalent elsewhere? [11:28] Fried tortilla [11:28] vegeterian taco? [11:28] frybread [11:28] yeah [11:28] toddf: well it's an 8 gb vps, so yes [11:28] sounds like someone is playing with the pricing models to see who will bite at what pricepoints [11:28] and it's also oversold to death and complete crap [11:29] what is frybread [11:29] m0unds: looks like you got valentine's fruitsnacks too? [11:29] oversold, well, thats kindof par for the course with chunkhost right? [11:29] i wiki'd it but i'm not sure what its equivalent is [11:29] yeah, hahaha [11:29] http://en.wikipedia.org/wiki/Frybread [11:29] Frybread :: Frybread (also spelled fry bread) is a flat dough fried or deep-fried in oil, shortening, or lard. The dough is generally leavened by yeast or baking powder.[citation needed] Frybread can be eaten alone or with various toppings such as honey, jam, or hot beef. Frybread can also be made into tacos, like Indian tacos. It is a simple complement to meals. History According to Navajo tradition, frybread was created in... [11:29] is it like those things at county fairs that have tons of powdered sugar on them [11:29] or is that called something else [11:29] m0unds: I work in a 1-man office. Now I feel very alone :( [11:29] * nb: i've never travelled outside of ultra-urban areas anywhere in the US [11:29] hazardous: Ever been to a carnival/fair? [11:30] brycec: too bad the valentines fruit snacks didn't have TO: and FROM: written in :< [11:30] bummer. i rarely eat anything people bring, except today because i had no idea frybread was egg-free (i'm allergic to eggs) [11:30] brycec: nope, never [11:30] *** jpalmer has quit IRC (Quit: leaving) [11:30] hazardous: so basically I have to put this in terms of taco bell, mcdonalds, and wendy's? [11:30] hahah [11:30] pretty much, yeah [11:30] *** jpalmer has joined #arpnetworks [11:30] i've seen a parade or two but those msotly consist of naked gay people and the sisters of perpetual indulgence here [11:30] m0unds: Okay, easy. IT"S TASTY. [11:31] m0unds: awesome, very lucky. I bet you get lunch for free too :( [11:31] like i'm trying to picture fried bread, but the only thing i can think of is french toast [11:31] i've had frybread before, but before i realized i was allergic to eggs (i grew up in CO/NM) [11:31] m0unds: wow, that must .. suck.. [11:31] there was red chile too, but it had meat in it [11:31] doesn't that exclude almost everything made commercially [11:31] yeah [11:32] i remember seeing a lot of WARNING: MADE IN A FACILITY THAT ALSO PROCESSESS x,y,z,a [11:32] and it's something like nuts, wheat, eggs, gluten? [11:32] yeah, eggs are a common ingredient in stuff. it just means i eat less shit than most people, since crappy food tends to have a lot of egg in it [11:32] crappy as in lacking in any nutritional value [11:33] so we tend to buy vegan stuff mostly, as it doesn't require diligent label checking [11:33] i wish i could eat cilantro, but it tastes like soap and it sucks because i like mexican food and they never remember to leave it off when i ask [11:33] it's not an allergy or anything but it just tastes vomit inducing and revolting [11:33] maybe something is wrong with my tongue [11:33] yeah, cilantro is great [11:33] hahaha [11:33] DEFECTIVE [11:33] hazardous: a lot of people think cilantro tastes like soap.. [11:34] they're WRONG WRONG WRONG [11:34] m0unds: no, it's actually in how they process the flavor [11:34] their processing of the flavor is wrong [11:34] cilantro is over-used as a crutch by crappy and wannabe "mexican" places [11:34] m0unds++ [11:35] it is a genetic thing no? [11:35] it could be [11:35] http://www.huffingtonpost.com/2012/09/20/cilantro-aversion-gene-study_n_1901124.html [11:36] http://www.nytimes.com/2010/04/14/dining/14curious.html?_r=0 [11:36] staticsafe: there is no formal proof that it is genetic, but that is what they are studying. they suspect it might be. [11:36] * staticsafe nods [11:37] I often hear "soap" along with chemicals, hand lotion, etc. [11:37] (ie, it smells or tastes like chemicals) [11:38] A Japanese study ... suggested that crushing the leaves will give leaf enzymes the chance to gradually convert the aldehydes into other substances with no aroma. [11:38] cilantro pesto as a way to enjoy cilantro [11:40] oh man. http://www.huffingtonpost.com/2011/10/27/roasted-vegetable-tofu-a_n_1057197.html [11:57] m0unds: you're veegan? [12:15] food wise, yea [12:15] lifestyle stuff makes my head hurt [12:52] brycec: you can judge some dns servers to set minimum ttl [13:05] s/servers/recursors/ [13:05] brycec: you can judge some dns recursors to set minimum ttl [13:05] s/judge/set/ [13:05] brycec: you can set some dns servers to set minimum ttl [13:05] oh i need both heh [13:05] i shouldn't be on irc before coffee [13:08] wow ubuntu is going to change to systemd apparently [13:08] it's not april 1st hmm [13:20] lol [13:20] mercutio: But I've determined that it's not the DNS lookup choking that one host, so yay that. [13:21] Tested with time fping... a whole bunch of IP's, still took much longer on Chunkhost for no discernible reason (same results) [13:25] weird [13:25] you want to pastebin me the command? [13:26] i could see if i can figure out what causing it [13:27] i found a ntt looking glass [13:28] I could, sure, but I don't think you'll see anything. I run smokeping on 5 hosts, and Chunkhost is the only one being poopy. [13:28] but it seems to be going amazingly slow [13:28] http://www.us.ntt.net/support/looking-glass/ ? [13:28] yeh [13:28] @ddg ntt looking glass [13:28] m0unds: I'm sorry, the DDG Zero-Click API returned no results. As this is a Zero-click Info API, most deep queries (non topic names) will be blank. Quite frankly, I suck cocks. [13:29] Sorry, the Looking Glass is currently unavailable. Please try again later. [13:29] oh, that' sright [13:29] i thought it may show up in more detail the ntt issues [13:29] chunkhost along is making it take more than 60 seconds? [13:30] alone [13:30] as a destination? [13:31] mercutio: source [13:32] running http://sprunge.us/eAhL takes forever [13:32] ^ on a chunkhost VM [13:32] forever in the linked case is 61s [13:32] oh that was from a chunkhost vm not to chunkhost [13:32] (And yes that's one with hostnames, it was quick/easy for me to post) [13:33] what do you get on arp? from NZ it took 37 seconds [13:34] real 0m23.240s [13:34] nice [13:34] i should try agian [13:34] I try it every 60 seconds :P [13:34] yeh [13:34] so more are cached i meant [13:35] heh, in my debugging earlier, I tcpdumped the DNS lookups, didn't take more than 3s [13:35] ^ On Chunk [13:35] oh curious [13:35] 22.7 seconds second time in nz [13:35] I then tcpdumped icmp... that was a mistake :P [13:35] so dns definitely makes a diff when a whole lot of uncached stuff [13:36] haha [13:36] i use strace sometimes [13:36] Yeah, but only 50% in your case. [13:36] I did that too, didn't show anything obvious [13:36] you're doing 20 pings [13:36] But strace doesn't have timing [13:36] that means it'll take 20 seconds for the pings at least [13:36] way less diff between cached/uncached on arp [13:37] and 22.7 sconds again heh [13:38] nice [13:38] so chunkhost is blowing chunks? [13:38] what dns server are you using on chunkhost? [13:38] and what location is it? [13:38] mercutio: I was using HE's DNS earlier, switched to using Google's DNS. No difference. [13:39] mercutio: LAX1 [13:39] (It's vps1.cobryce.com) [13:39] ahh [13:39] 8.8.8.8 isn't close in la :/ [13:39] maybe try 4.2.2.2? [13:39] i think he is close but not very good [13:39] Again: DNS lag has been taken out of the equation completely. Tcpdump says DNS only took ~3s. Trying without DNS lookups at all still takes 50+ seconds. [13:39] i did namebench ages ago [13:40] mercutio: ^^ [13:40] (Oh my bad, I'm using Chunkhost's own DNS right now) [13:40] which ARE close :P [13:40] sbc global is the other one that was good [13:40] heh [13:40] maybe they throttle you [13:40] it's not openvz is it? [13:41] Must be something like that. [13:41] mercutio: Xen [13:41] that nz host was xen [13:41] xen should be fine normally [13:41] * brycec is going to throw a ping -f at things and see how it does. [13:41] uhh [13:41] add -c 10 [13:41] or something [13:41] mercutio: yes [13:41] but later [13:42] because I have important shit to get done when I'm on the clock ;p [13:42] cos ping -f could be considered abusive behaviour [13:42] ok [13:42] Network throttling is the only thing that makes sense at this point, but I don't have a good way to test for that. [13:43] *** kevr has quit IRC (Ping timeout: 252 seconds) [13:43] smokeping? :) [13:43] umm usually http download and look at packet dumps [13:43] cos http has back off behaviour [13:44] it is recent version riht? [13:44] recent version of? [13:45] fping [13:45] there were soem updates at some point [13:45] just reading changelog now [13:45] And I was thinking they're only throttling icmp [13:45] i doubt it [13:45] good question, it's Debian Squeeze so no [13:45] fping sohuld timeout [13:45] is it 3.2+ ? [13:46] * Performance optimization for big select timeouts (#10, Andrey Bondarenko) [13:46] i wonder if that make sa diff [13:46] fping: Version 2.4b2_to $Date: 2002/01/16 00:33:42 $ [13:46] hmm [13:46] is arp using same version? [13:46] arp is 3.2 [13:46] as are my other nodes [13:46] hmm... [13:46] i'll tackle that later [13:47] you can probably just install the later debian package [13:47] but good call [13:47] if it breaks it only breaks fping :) [13:47] newer fping not in backports, so there may be a reason [13:48] that someone didn't get around to it [13:48] you can often just wget the package and dpkg -i it [13:48] *** kevr has joined #arpnetworks [13:49] for single programs with no dependencies [13:49] * brycec thinks mercutio is way more invested in this than he [13:51] real 0m34.709s [13:51] quite an improvement [13:56] heh [13:56] i like clean graphs :) [13:56] ok so fping bug :) [13:57] is mirrors.arpnetworks.com down? [13:57] weird it works from another location [14:01] ha! mercutio was slightly wrong, it did break fping slightly :P ERROR: fping ('/usr/bin/fping6 -C 1 ::1') could not be run: /usr/bin/fping6: /lib/libc.so.6: version LIBC_2.15' not found (required by /usr/bin/fping6) [14:01] what version was that [14:01] glibc can be the biggest bitch [14:02] apt-get source fping may be necessary :( [14:02] if that's 3.2 [14:02] fuck, nvm... In spite of dpkg giving an error when I tried installing fping 3.8, it went ahead and installed it. [14:02] it's whether smokeping minds [14:02] 3.2 back installed and working [14:02] * brycec shakes a fist at dpkg [14:03] *** kevr has quit IRC (Read error: Operation timed out) [14:04] Fun fact: http://sprunge.us/LZAT doesn't mean that it didn't install the package anyways. [14:04] oh [14:04] i thought you were going to 3.2 [14:04] 3.2 should be fine [14:04] mercutio: I did, then I tried 3.8 for shits and giggles. [14:05] ahh ok [14:05] It shat more than he giggled [14:05] it works with ubuntu from debian packages too sometimes [14:05] https://github.com/schweikert/fping/blob/master/ChangeLog [14:05] I get the libc thing... I just figured "ERROR installing" meant "I didn't actually install it, you still have 3.2" [14:05] things are less interesting from 3.2 to 3.8 [14:05] yip [14:05] but when you do other updates it can be a bitch [14:06] It's squeeze, there will never be another update to fping on it :P [14:06] no ot other packages [14:06] what does apt-get -f install say atm? [14:06] 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. [14:06] oh cool [14:06] it not complaining [14:07] ok [14:07] the real question is did smokeping fix itself [14:07] Once 3.2 is on, yes [14:07] 3 minutes' data and counting [14:07] cool [14:08] *** kevr has joined #arpnetworks [14:52] m0unds: brycec i like the NLNOG; since you get an actual shell, you have access to more tools; way better than a looking glass [14:58] up_the_irons: what about ARP hosting a RIPE probe? [14:58] (I'm skeptical, but worth asking) [14:58] s/worth asking/curious) [14:58] (I'm skeptical, but curious)) [14:59] brycec: a possibility [15:09] https://atlas.ripe.net/targets/ringnodes/map/ [15:10] whoa [15:10] there's already one at ARP! [15:10] that's from nlnog [15:10] ohduh [15:10] it's the probe target map [15:10] so RIPE hits NLNOG? [15:10] yeah, you can select those as user-configured probe destinations [15:11] or targets [15:11] * brycec wonders why RIPE doesn't just link to NLNOG's site? [15:11] they do [15:11] it's kinda buried [15:11] it's right at the top [15:11] Like RIPE has their own list of nodes [15:12] * brycec should stop browsing with zoom=50% [15:12] lol [15:12] it's the third line of text above the map "visit nlnog project" or something [15:12] hahaha [15:12] Yeah, but it's only ONE [15:12] the whole thing should be links [15:12] Why did RIPE make their own list and own map? [15:12] why should someone click off the site to look at the map when it's right there? [15:12] heh okay so you understand my point :p [15:12] BAD UX BRAH [15:13] why should someone take the time to scrape(?) another site for information when you could just send the user to the source? [15:13] BAD BRAH [15:13] heh [15:13] doubt there's scraping going on [15:14] maybe some shovelling [15:14] or they're paying some bloke to update it :( [15:22] the coordinator for nlnog is a commitee member at ripe [15:22] and he works for atrato [15:25] croneyism! [15:25] MUST BE [15:25] also, i'm guessing he's trying to get exposure for nlnog [15:25] but mostly cronyism [15:25] whoa, we're already in the Atlas target list [15:26] yeah, nlnog ring target [15:26] cool [15:43] brycec: there's a way to get output of root-servers locations [15:43] with dns [15:45] dig nodes.l.root-servers.org txt +tcp | [15:45] brycec: just put in your Targets file with some modifications, smokeping takes a while to start up heh [15:45] s/|$// [15:46] s/\|$// [15:46] gah [15:46] but how's that for a list bryce [15:47] smooooooooooooke [15:47] it's ok to ping them i seem to recall too [15:47] i can't find confirmation of that though [15:49] it's kind of crazy how well the root serrvers are overprovisioned these days [15:49] i think it's pretty unlikely for the root servers to not be reachable, while the internet works now. [15:53] Fri Feb 14 15:51:44 2014 - FPing: probing 158 targets with step 300 s and offset 249 s. [15:53] Fri Feb 14 15:51:44 2014 - FPing6: probing 47 targets with step 300 s and offset 181 s. [15:53] hm [15:53] zomg [15:54] welcome to the smokeping age, staticsafe [15:54] i wonder how much icmp traffic is on arp from people running smokeping :) [15:54] mercutio: nice idea [15:55] hahaha [15:55] brycec: i was surprised that was out there. [15:55] it's all isc infrastructure afaik [15:55] lol well I'm currently responsible for about 150kb/s of ICMP traffic :P [15:56] TRIPLE IT [15:56] hm the smokeping webapp is pretty damn slow now [15:56] m0unds: You may recall I did more than triple it... [15:56] what is the secret to your speed brycec? [15:56] staticsafe: cgi? fastcgi? other? [15:56] staticsafe: fastcgi [15:56] im passing it through fcgiwrap [15:56] nginx [15:57] I'm not familiar with fcgiwrap [15:57] I'm running it from spawn-fcgi [15:57] I was responsible for at least 1mbps of ICMP traffic :P 15:54:42 < m0unds> TRIPLE IT [15:58] s'kb/'kbp [15:58] lol well I'm currently responsible for about 150kbps of ICMP traffic :P [15:58] s/kbps/gbps [15:58]