[06:04] *** m0unds has quit IRC (Ping timeout: 265 seconds) [06:05] *** m0unds has joined #arpnetworks [06:19] anyone else seeing network issues? [07:16] *** m0unds has quit IRC (Ping timeout: 252 seconds) [07:19] *** m0unds has joined #arpnetworks [07:25] *** m0unds has quit IRC (Ping timeout: 272 seconds) [07:25] *** m0unds has joined #arpnetworks [08:35] *** jlgaddis has quit IRC (Ping timeout: 250 seconds) [08:36] *** jlgaddis has joined #arpnetworks [08:36] *** jlgaddis has quit IRC (Changing host) [08:36] *** jlgaddis has joined #arpnetworks [10:02] *** phlux has quit IRC (Quit: WeeChat 1.1-rc1) [12:16] nope, but it's a little later than your original query [12:18] My smokeping looks mostly clear, m0unds. There was a bit of jitter on ARP's ipv6 router about 3 hours before your question, but nothing lost and nothing anomalous. [12:31] *** phlux has joined #arpnetworks [13:11] *** thestereobus has joined #arpnetworks [13:12] anyone seeing intermittent network issues? [13:19] *** thestereobus has quit IRC (Quit: thestereobus) [13:27] i saw ipv6 take dump and had multiple v4 monitors trigger, but i was in and out of airports so i couldnt look into it [13:28] seems fine since i asked earlier though [13:49] Okay, brain-trust, I could use a bit of guidance. A buddy of mine is looking for some kind of document portal, something he can link to, possibly do some user/password authentication for sensitive documents and downloads. The only solution I know of is Drupal, and its security track record is a big concern to me. Any other ideas/suggestions? [14:04] hmm there has een a few ping sikes rcently [14:05] well ping and 50% packet loss [14:05] mercutio: yea, I saw those yesterday [14:05] I saw them over Level3 [14:06] yeh but the weird thing is my route is to/from coresite/any2ix [14:06] oh hmm [14:06] yea [14:06] it's affecting other things too [14:06] so i think it's ddos related. [14:06] http://kremvax.acfsys.net/smokeping.cgi?target=Remote.peeringarp [14:07] http://kremvax.acfsys.net/smokeping.cgi?target=Remote.s7laxarpnetworks [14:07] you get 50% too [14:07] also over v4 or v6 to arpnetworks.com [14:09] hmm you don't have r1 in your list. [14:09] what does r1 do? [14:09] peering [14:09] is that different than 10.10.10.6? [14:09] i have no idea what 10.10.10.6 is [14:10] some peering box at arp I think [14:10] s1 -> s7 -> 10.10.10.6 for me [14:12] I used to see it in traceroutes [14:12] apparently not anymore? [14:12] do you see r1 now? [14:12] yea. maybe r1 replaced 10.10.10.6 [14:12] or addressing just changed [14:13] your smokeping is going slow :( [14:13] oh [14:13] probably the packet loss [14:13] there's some more loss again [14:13] yeah [14:16] nmap indicates that r1 and 10.10.10.6 are different [14:23] http://irclogger.arpnetworks.com/irclogger_log/arpnetworks?date=2014-06-11,Wed&sel=252#l248 [14:23] 10.10.10.6 is a peering box running BIRD [14:25] yeah it may be another ip on r1 [15:02] if i set up a vlan on my arp vps (only need it to test something) it wont interfere with anythingnon the virtual switch will it? [15:03] woukdnt want to accidently push outbtagged vlans and it send alarm bells ringing or somethingbcrazy [15:03] probably not [15:04] what are you using VLANs for? [15:04] doesn't seem like you'd want to configure them on eth0 or whatever anyway [15:05] grody: ports aren't tagged, so you'll be doing a vlan inside a vlan [15:06] you will probably find that your mtu has to go down too [15:06] how do you intend to combine tagged and non tagged? [15:07] theorising an install of pfsense as it needs 2 interfaces to be configured, until i get it rigged, using a vlan will solve that [15:07] it needs two for install, damn [15:07] you could make a dummy tap interface or something [15:08] if you just need an interface which does nothing [15:08] acf: in the installer? [15:08] yea fakenanvlan for lan, solves it.. log in via wan and setup vpns etc [15:08] probably there is a shortcut to open a console? [15:08] its possible [15:08] so daylight saving in the US just changed right? :) [15:09] so it's 3 pm in los angeles, and plus a few to east coast? [15:09] yea [15:18] 50% loss again :( [15:18] cool [15:18] Yes -> 15:07:55 ⤷ | so it's 3 pm in los angeles, and plus a few to east coast [15:18] i think s1 to s7 is where issues are coming [15:19] from arp to world it goes s1 s7 r1 for peering, and r1 s1 for incoming [15:20] and tracing over peering from out to arp, it shows s1 as having no loss, but destination site having loss. [15:20] The pfSense *installer* (the component that copies files to disk) doesn't require any interfaces, however initial boot configuration does insist upon configuring at least 1 interface. [15:20] s1 itself is showing some loss on smokeping [15:20] grody: Not sure where you're seeing pfSense requiring *two* interfaces to setup. That was done away with around 2.0 [15:20] yeah some loss, but not much, and icmp is deprioritised. [15:20] (Personally, I set it up with one, pfctl -d to disable the wan filtering, login and play.) [15:21] arpnetworks.com is also showing some loss [15:21] (being connected to s1) [15:21] i've done 17404 packets to end destination, and 184 dropped to destination host, and none dropped from s1 [15:22] well it could also be switch that's into s1, that hosts are connected to. [15:22] but i'm seeing 0 loss to s1 [15:22] but this is from dedicated server not vps, so it may be different. [15:22] also it could be something like flow table overflow [15:23] where things like smokeping that start and stop could be worse [15:23] I saw ~1% to s1 [15:23] i'm just leaving mtr running. [15:23] and ~6% to arpnetworks.com [15:23] I'm also testing from a dedicated box fwiw [15:23] hmm [15:23] that was a very short sample (75 packets) [15:23] i wonder when up_the_irons will make an appearance :) [15:24] oh was it the first packet dropped? [15:24] * brycec wonders too, there's a ticket with his name on it... [15:24] it is a sunday there, i realise... [15:25] the last two haven't been as bad as the 3 prior [15:26] has anyone put a ticket in? [15:29] usually I just wait for up_the_irons to show up here :P [15:31] heh [15:36] i imagine monitoring systems got triggered. [15:45] my monitoring systems got triggered :P [15:45] i hate it when monitoring systems get triggered int he middle of the night for an intermittent issue [15:47] i got a trigger like 4 hours ago, but it looked this issue happende 3 hours ago [15:54] *** mnathani_ has quit IRC (Ping timeout: 240 seconds) [15:56] *** mnathani_ has joined #arpnetworks [15:56] *** thestereobus has joined #arpnetworks [16:21] weird i accidentally left a mtr running to arp a day or so ago, and it's been showing these weird unrelated hosts, that it definitely wouldn't have gone through [16:21] mtr must be buggy [16:22] it does suggest s1 has had less loss than destination, but more loss the hop a couple of hops prior [16:22] still not much though. [16:26] I'm actually not seeing anything weird on the graphs. Just host kvr12 is having issues. [16:26] up_the irons, nothing at all? [16:26] http://kremvax.acfsys.net/smokeping.cgi?target=Remote.googledns [16:27] http://kremvax.acfsys.net/smokeping.cgi?target=Remote.peeringarp [16:28] the last 30 hours graphs show well [16:29] peering route has a few of these: [16:29] Mar 8 08:54:06 r1 kernel: [33692309.054249] nf_conntrack: table full, dropping packet. [16:29] but only at that time (around 9am pst) [16:29] strange [16:30] I was seeing problems over Level3 also [16:30] http://kremvax.acfsys.net/smokeping.cgi?target=Remote.l3dns1 [16:30] problems to everywhere really, even arpnetworks.com [16:30] maybe it was small packet ddos of low volume? [16:30] err to random destinations or something, overflowing tables of where to send stuff [16:31] i tend not to keep state for that reason [16:31] yeah [16:32] acf_: the peering box running bird is r1, fyi [16:33] ah, ok [16:34] is that also 10.10.10.6 then? [16:35] yeah [16:35] --- kvr12.arpnetworks.com ping statistics --- [16:36] 690 packets transmitted, 690 received, 0% packet loss, time 689956ms [16:36] rtt min/avg/max/mdev = 14.570/22.440/290.342/21.077 ms [16:36] so i'm not seeing any loss from my house [16:36] it's on/off [16:37] roger [16:43] there it goes again [16:44] also IPv6 is affected [16:45] yeah _now_ i see it. Like > 1 Gbps spikes incoming Level 3 [16:46] oh fun [16:46] or wait, is it outgoing... [16:54] oh [16:57] on kvr12? :) [16:57] i think it probably is outgoing, as incoming to s1 looked fine [17:01] mercutio: negative. somewhere on s8.lax (so dedi), but i don't see the traffic on any of the downstreams [17:01] i must be missing a graph... [17:01] damn [17:02] so it's not split evenlyish [17:02] oh graph to customer [17:02] cos level3 was showing it [17:32] brycec, really? when i fresh installed 2.1.5 it needed one for WAN and one for LAN [17:32] 2.2 is latest, but i've ran into issues with it [17:33] the way around was the create a vlan for WAN usually and allocate LAN to IP range you want/can access [17:33] i dunno why people don't just do openbsd [17:33] i dont get on with it [17:33] freebsd though [17:33] it's pretty easy to setup for firewall if you're not scared of the cli [17:33] * grody is a console junkie [17:33] even right now im ssh'd into an openbsd shell to use irssi [17:33] then just go with openbsd :) [17:34] it's too cool for me [17:34] im a freebsd whore [17:34] they make epic firewalls [17:34] well freebsd has old pf, but still does work :) [17:34] was my first homegrown back in 4.3 days [17:34] used to use smoothwall/mandrake snf before [17:35] i'm using linux for my nat [17:35] im actually running freebsd again on this lappie [17:35] it runs soo smooth [17:35] linux/ferm is sort of ok [17:35] but it doubles up as my file server etc. [17:35] and linux desktop :) [17:35] im using openwrt (linux) on my border as a pure router, no tracking/firewall and a pfsense for firewalling [17:35] sounds complicated. [17:36] stupidly easy really [17:36] i've actually been thinking about shifting to terminating pppoe on linux from modems. [17:36] internet > | networks [17:36] ya im using pppoe on a vdsl link [17:36] but it's being held off by not wanting to reboot, and wanting to put the extra ram in first. [17:37] i have both adsl and vdsl with two modems [17:37] and openwrt is the most stable OS for my embedded router that supports PPP minijumbos [17:37] with a /29 and two gateways. [17:37] i have a few blocks from my ISP [17:37] yeah i been trying openwrt on a wireless ac router [17:37] i can't stand it tbh :) [17:37] a /48 of 6 in /52's (i have multi lines with them), and a few small blocks of 4's [17:37] but it seemed stable [17:37] eek [17:38] i hate openwrt for wifi [17:38] i think it sucks [17:38] for anything but... [17:38] well i am on the development branch [17:38] ddwrt on these routers works the wifi so much better [17:38] as it was broken in the stable branch [17:38] i have archer c7 [17:38] but ddwrt lacks ipv6 and the minijumbos [17:38] 802.11ac atheros [17:38] tplink 841nd [17:38] thats g/a/n [17:38] i have a tp-link 4300 too which is in between those two [17:39] and that's running gargoyle [17:39] and it sucks much less :/ [17:39] have another tplink 5GHz in bridge mode [17:39] yeah they're all in bridge mode :/ [17:39] gargoyle makes bridge mode nice [17:39] my network is actually a mess @home, physically speaking [17:40] this is why i'm trying to use wireless bridging :/ [17:40] it works sound as a network.. but my setup for the rig is shoddy [17:40] never tinkered with that [17:40] tplink's software is ok [17:40] i havce two erthernet cables and fibre across the room [17:40] but if i can dd or open them, even better [17:40] i been thinking about running it by the ceiling... [17:40] but i have no idea what the nicest way of doing that is... [17:41] i figured i could just shift more stuff over there, and wireless bridge. [17:41] hehe i used to run a media converter (ether to fibre) in my old place [17:41] was a proper old skewl 100 mbit/s effort but was fun [17:41] my switch has fibre, and i have 10 gigabit ethernet cards that were cheap on ebay. [17:41] these tplinks have gigabit switches, but only run 10/100 ports [17:41] weird, this is gigabit fire [17:41] almost full port speed per port [17:41] err fibre [17:42] it's really expensive to get 10 gigabit tranceivers. [17:42] i dont have fibre anywhere here now [17:42] and i'd need a 10 gigabit capable switch. [17:42] all cat5/6 or wifi and the nte5 box for vdsl [17:42] should go gigabit, but only have a handful of gadgets that will utilize it [17:43] heh [17:43] i have infiniband between windows/linux [17:43] and when i tried using just gigabit again it was so slow :/ [17:43] do have trunking going on from one switch to another to get more speed from the file server, but i only use 140mbps max off that [17:43] but i'm using ssd's.. [17:44] i get about 400 to 500 megabytes/sec normally [17:44] so 100 megabytes/sec on gigabit felt slow [17:44] the wifi on these ddwrt's can managed a total (phy) speed of around 190mbps [17:44] if you only look at it one way [17:44] like shifting large volumes of data from ssd to server [17:44] i can do 600 megabit wireless with archer c7.. [17:44] did a client from one into AP on another and got those speeds between [17:45] hehe [17:45] that's same room though [17:45] for internet it doesn't really make a difference [17:45] yet it's still faster putting an 8GB pendrive on a pigeon to send it from spain to UK than it is to upload the 8GBs over ther internet [17:45] but for rsync etc it can [17:46] i did 150gb upload on vdsl [17:46] took like a day [17:46] my rule is if my wifi can saturate my internet, my wifi is working optimally [17:46] which it does :) [17:46] but it's good to have off site back [17:46] but yeah, my main data is only like 150gb. [17:46] that would have seemed huge 10 years ago [17:47] i get about 8/9MB/s on wifi sending via sshfs/sftp either way, 11 on ethernet (not sure why that is) - but unencrypted stuff i hit a usualy 10MB/s [17:48] network wise thats enough for me, even backing up my laptop only took as long as watching a movie [17:48] backing up from UK my VPS takes a while, usually about 8/9GB [17:48] that runs from anywhere from 10mbit/s to 30/40 [17:49] heh my box plugged into the tv installs packages faster than my desktop [17:49] even using sshfs back and forth is efficient enough [17:49] because i'm using a web cache, and so it gets like 15mb/sec [17:49] when my net is more lik4 mb/sec [17:49] err more like 4mb/sec [17:49] ouch [17:49] it's got a hard-disk though [17:49] i mean megabytes not megabit [17:50] i used to run a large web proxy in transparent mode once [17:50] used to help a shyte load on slower links [17:50] well it doesn't help much [17:50] i dont see the point these days, i get full speed 24/7 on my ISP [17:50] but it makes package installs so fast when you have multiple computers with the same os [17:50] in fact, i have to shape locally in order to prevent hosts from taking to peter [17:50] i have fq_codel on my connection [17:51] whats that? [17:51] it doesn't buffer bloat or get loss when maxing it out [17:51] it's a aqm thing that gives fare sharing [17:51] ahh [17:51] http://www.bufferbloat.net/projects/codel/wiki/Wiki/ [17:51] it's kind of magical [17:51] im using software based stuff in pfsense [17:51] yeah i'm doing it at the isp end [17:51] works really well in fact [17:51] so it doesn't even hit my network and get capped at my end [17:51] yea shaping is kinda pointless when the packets are already there at the link [17:52] yeah you have to go further under your connection [17:52] it even raised my download speeds [17:52] damnit i'm such a geek :/ [17:52] tc class add dev $DEV parent 1: classid 1:1 htb overhead 8 rate 36.9mbit ceil 37mbit burst 6k [17:52] tc qdisc add dev $DEV parent 1:1 fq_codel [17:53] the idea for this is to match the rate limiting my ISP can offer (96% under the synch speed) - that way small packets, like DNS and VoIP have headroom to get through [17:53] with fq_codel you don't have to worry about diff traffic [17:53] you just have to not go over your connection speed too much [17:53] preferably slightly under [17:54] i'm on 38.5 megabit sync rate i think [17:54] ptm reduced overhead from atm [17:54] i actually get max sync rates for once [17:55] 79.7 down and 19.9 up and it's an upto 80/20 [17:55] wire speed i get about 77 down and 18 up [17:56] shaper resevers 1mbit either way for "priority" trafffic [17:56] which is DNS and VoIP in my case [17:57] nice [17:57] yeah with fq_codel you don't even have to worry about that [17:58] you need to install a package on openwrt to do it i think [17:58] but it works better than sfq [17:58] which was my old standard [17:58] so if you shape to 77 megabit or something,y ou stick fq_codel on that 77 megabit shape [17:58] cant remember what schedular im using [17:58] PRIQ or something [18:00] the faster your connection speed is the less it matters. [18:00] well depending on how heavy your use is [18:01] indeed, but over the weekend i turn on my torrents (have just about every ISO for just about every FOSS OS you can think of) and it gets hit hard [18:01] yeah, torrents can bean issue [18:01] the fact i can still make a crisp voip call whilst hitting up a youtube video and the missus watching netflix is when you know it's shit hot [18:01] i graph my connection [18:02] i was getting loss when maxing it out before [18:02] now i can get 1 or 2 msec extra ping sometimes [18:02] i'm not even doing anything on the upstream :/ [18:02] fortunately it's only my pfsense that tracks connections, and it can handle a good 60'000 state entries befoe it will stat buckling [18:02] i have only managed to get it upto about 20k [18:03] state tracking sucks :) [18:03] it's the NAT networks that hammer it if they start torrenting [18:03] i'm not using nat so i don't have that issue [18:03] lol [18:03] only for wifi [18:03] the non NAT networks dont seem to hit the cpu at all [18:04] fortunately my torrenter sits on a public IP with no stateful inspection on the firewall [18:04] but the shaper will still snag it [18:04] i have 65536 conntrack max [18:04] i never adjusted it [18:06] 73000 states mine is set to [18:06] 3000 tables [18:06] 200000 table entries [18:06] thats default to my amount of RAM [18:07] only an 800MHz CPU, but 784MB is more than enough [18:07] weird i wonder why mine is lower [18:09] http://imgur.com/AvAVvF4 [18:10] maybei should graph my connection [18:11] been rrd graphing my pfsense since install [18:11] am still using an archaic Neoware CA10 [18:11] 35W of pure awesomeness [18:12] even had an ancient aesni capability so 128bit VPN takes no overhead on the CPU [18:12] can handle 6 users easily [18:13] sweet [18:14] i sure hope my VPS does use too much CPU.. got a lot of packages to recompile :> [18:15] ezjails is pure epic, way i ran my jails before was horrifying [18:16] considering i only have a single core, the performance is still impressive [18:16] don't think burst cpu sage is generally an issue with vps's. [18:16] err usage [18:16] well that's why it's generally not an issue :/ [18:16] i rarely use my resources much, once it's running as meant, it passes small amounts of traffic with little CPU/mem load [18:17] yeah that's what most people are like [18:17] just compiling in multiple jails at the same time starts stressing it [18:17] they're all dual cpu machines. [18:17] so lots of people would have to be stressing it to be an issue [18:18] it's more of an issue when people give out lots of cpu cores, and then people hose it. [18:18] ideally you'd have a single core VPS per core (one core say in an octa) [18:18] even if it's dividing up resources between users there are heaps of context switches, and it hammers cpu caches. [18:18] so yea, i doubt a few hours would make an impact [18:19] well it'll shift you between cores normally [18:19] not as bad as freebsd does :/ [18:19] have you ever looked at cpu core usage on freebsd? [18:19] i always wondered if that was efficient.. numerous threads running and them being passed between cores [18:19] it bounces processes around all the time. [18:19] nah it's not efficient [18:19] i'd have though one thread per thread ability per core [18:20] they tend to stay in one place when they're active though [18:20] yea [18:20] it's more if it's on/off [18:20] so it tends to put jitter up a little, and it's less efficient if you're doing less (so it matters less) [18:20] if you have 20 processes that wake up occasionally how would you distribute them? [18:21] like 2 threads per core on a 4 core CPU, and im only actively running 3 threads, say ffmpeg whilst running VLC whilst sftp is downloading something.. i'd imagine 1 and a half cores would become occupied [18:21] it's one of those "complicated problems". [18:21] mm [18:21] i dont code enough to make a technically correct opinion really [18:21] i've been wondering how well hyperthreading works in virtualisation. [18:21] becauuse that complicates matters even more. [18:22] from my understanding kvm handles this stuff better than xen [18:22] xen isn't very intelligent when you have dual cpus. [18:22] epsecially if the host starts doing stuff whilst a guest is hammering [18:22] there's some huge complications with intel dual cpu systems [18:22] if you are both high cpu and high network activity it gets even more complicated. [18:22] i had a dual AMD system with two 8 cores.. that was a hell [18:23] amd are just as bad probably, i just understand the intel problems more. [18:23] but basically intel just use really fast interconnects between the cpu, because people don't deal with it well. [18:23] i believe i had to run windows 2008 on that in order to utilize the motherboard features [18:23] so in the end inefficent for most things is only 20% slower or something [18:23] but some of the network stuff is higher overhead than that [18:24] "use whats available as it's available" [18:24] like if the network queue is on one cpu, and the vm is on the other cpu, then it has to wake up both cpus [18:24] and forward data between them [18:25] if you're just doing cpu, it can just leave it running on the cpu that the network isn't connected to [18:25] but with numa systems often memory is owned by a cpu too [18:25] i do sometimes wished i was good at code [18:25] and they shifted to having pci-e lanes owned by a cpu [18:25] oh don't worry, normal people aren't good with this stuff :/ [18:25] i curse the way some things work and would just ♥ to do it properly [18:25] i only understnad a little.. [18:26] i don't know if you're interested, but facebook had an interesting article on memcached scaling. [18:27] basically they really struggled with memcached scaling with "random" data. [18:27] memory bandwidths etc have gone up, but latency still sucks. [18:27] i doubt most people on facebook would understand it [18:27] so if you randomly go all over memory then you're pretty latency bound. [18:27] g+ has a generally more knowledgable audience, wich im finding out [18:28] this is facebook for their own systems. [18:28] they're using memcached. [18:28] wow [18:28] to cache peoples feeds etc. [18:28] i despise facebook - never like it [18:28] but it's relaly unpredictable. [18:28] i just find large systems fascinating. [18:28] one site i do like is waybackmachine [18:28] but yeah you can't just throw cpu/resources at the problem. [18:29]