[06:38] *** heavysixer has joined #arpnetworks [06:38] *** ChanServ sets mode: +o heavysixer [06:57] *** heavysixer has quit IRC (Quit: heavysixer) [07:00] *** anisfarhana has joined #arpnetworks [07:18] *** heavysixer has joined #arpnetworks [07:18] *** ChanServ sets mode: +o heavysixer [08:17] *** dj_goku has joined #arpnetworks [08:17] *** dj_goku has quit IRC (Changing host) [08:17] *** dj_goku has joined #arpnetworks [08:17] *** dj_goku has quit IRC (Client Quit) [08:37] up_the_irons do you rate limit icmp on your network? [08:37] outbound [08:49] only UDP is rate limited iirc [08:55] oh, hah [08:55] up_the_irons: ^ [08:56] i liked the SP campaigns in mw + mw2 - mw3 bummed me out [09:21] *** first2know has quit IRC (Ping timeout: 260 seconds) [09:30] *** first2know has joined #arpnetworks [09:59] *** anisfarhana has quit IRC (Quit: Leaving) [10:30] staticsafe: UDP over 5 Mbps on non-OpenVPN port is rate-limited [10:33] no icmp limiting? [10:33] ie, if i use smokeping on a bunch of hosts [10:34] it wont be affected? [10:36] stevenc: workin' fine for me [10:37] No deviation from the ping times I get on other smokeing hosts [10:37] ive got about 100 ips in mine and theres a bunch of jitter compared to other locations [11:21] *** heavysixer has quit IRC (Quit: heavysixer) [11:22] *** heavysixer has joined #arpnetworks [11:22] *** ChanServ sets mode: +o heavysixer [11:45] *** heavysixer has quit IRC (Quit: heavysixer) [11:48] *** heavysixer has joined #arpnetworks [11:48] *** ChanServ sets mode: +o heavysixer [11:53] *** forgotten has quit IRC (Remote host closed the connection) [12:54] *** rgouveia has quit IRC (Ping timeout: 264 seconds) [12:55] *** rgouveia has joined #arpnetworks [12:55] *** rgouveia has quit IRC (Changing host) [12:55] *** rgouveia has joined #arpnetworks [13:14] *** Bluerise has quit IRC (Ping timeout: 260 seconds) [13:16] *** Bluerise has joined #arpnetworks [13:41] *** m0unds has quit IRC (Quit: REBOOTING. O NOES.) [13:42] *** wak has joined #arpnetworks [13:45] *** CaZe` has joined #arpnetworks [13:45] *** CaZe has quit IRC (Read error: Connection reset by peer) [13:45] *** thestereobus has joined #arpnetworks [13:45] *** CaZe` is now known as CaZe [13:45] *** tabthorpe has quit IRC (Ping timeout: 260 seconds) [13:48] lovely, ARP's network just died [13:49] v4 anyways [13:49] looks alive to me... [13:50] *** heavysixer has quit IRC (Quit: heavysixer) [13:52] yeah, it's working here [13:52] ferrovax.asininetech.com [13:53] *** m0unds has joined #arpnetworks [13:53] that doesn't ping on its v4 addr. v6 works.. [13:54] indeed [13:56] *** avj has quit IRC (Ping timeout: 268 seconds) [13:59] makes it to arp, then stops [13:59] I get as far as 3: any2-ix.la.arpnetworks.com 57.561ms [14:00] Also my ipv4 is down vps3.cobryce.com [14:00] me too. can't get to ARP. it's been going up and down for about ten minutes. [14:00] damnit up_the_irons [14:00] fix your shit [14:00] ^ jokingly - I'm sure he's on it, and it's not shit [14:01] I blame the exchange <.< [14:01] >.> [14:03] I can get in from an oklahoma library [14:11] *** sinecurist has joined #arpnetworks [14:12] I blame the exchange too [14:13] toddf: got a traceroute from that .ok.us library? [14:14] I'm coming from chunkhost who is directly on Any2IX it looks like, because my traceroute is 64.34.218.1 64.34.218.1 64.34.218.66 any2-ix.la.arpnetworks.com [14:14] http://pastebin.ca/2454349 [14:16] Looks like high CPU util on s1.lax [14:16] I'm investigating [14:16] Yep, different transit [14:16] (toddf is coming in through mzim) [14:16] (and hitting s7.lax) [14:20] anything weird with overall net connectivity from 12:30 yesterday to 12:30 today? [14:20] not just arp... but a bigger thing [14:20] nothing I've noticed [14:21] our queries per day cut in half for exactly 24 hours. [14:21] brycec hit http://pastebin.ca/2454349 again, I have various points around the net all of which seem to come in the same way [14:25] RandalSchwartz: there were some regional issues w/level3 and limelight networks, as well as west coast comcast connectivity issues sun/mon [14:26] yeah, this just ended 2 hours ago [14:26] we're in a brainstorming meeting to try to analyze [14:26] huh, funky [14:27] randalschwartz: analyze the internet or analyze how to keep the click rate up with network outages? [14:27] analyze the outage/connectivity issues probably [14:27] see if it was something we did to ourselves too [14:27] and the answer of course is to make a nice donation to kickstart up_the_irons' fund for datacenter #2 [14:28] I was asking a leading question so I could give the above answer, m0unds ;-) [14:28] bahaha [14:29] RandalSchwartz: presumably that's an easy question to answer: "Did anybody touch anything? Nope? Good!" [14:29] just because I have an opinion doesn't mean I can't have fun with it [14:29] :) [14:32] is anyone else having network weirdness? [14:32] On my network? Nope, 100% here. :P [14:32] hazardous: but if you're asking about ARP... see above :p [14:32] lol [14:32] figures [14:32] that's uhh.... [14:32] like i have lost connetvity on some IPs [14:33] fully in and out [14:33] on ARP? Consider it a known issue. [14:33] but i have one ip online still, i can ssh into that, then ssh from that into my others [14:33] yeah it's all on arp [14:33] hazardous: seems to be an issue with a transit provider/its router [14:33] i'm more confused than anything that some are working and some aren't [14:33] lol [14:33] hazardous: check your traceroutes [14:34] Anything going through 206.223.143.166 (Arp's any2ix router) seems to be dieing [14:34] i idle in here right from my vps. no problem for me at the moment. [14:34] ah [14:34] 6 ge0-arpnet.cust.lax07.mzima.net (67.199.135.102) 53.246 ms 52.504 ms 52.744 ms [14:34] that's last hop on broken ips [14:34] orly? toddf had a working traceroute through mzima [14:35] the stuff i still have online ends with 8 ge-0-7-0-24.r04.lsanca03.us.ce.gin.ntt.net (129.250.198.186) 9.476 ms 10.302 ms 9.825 ms [14:35] 9 vl5.s1.lax.arpnetworks.com (208.79.88.129) 33.604 ms 33.869 ms 34.121 ms [14:35] 14:15:36 <@up_the_irons> Looks like high CPU util on s1.lax [14:35] 14:15:36 <@up_the_irons> I'm investigating [14:35] odd that you're seeing opposite symptoms... Or maybe things are just balancing out [14:36] yeah i dunno [14:36] just tried from several systems from sf, oakland, and sj [14:36] hitting mzima is going down, hitting that other one is connecting fine [14:36] I've only tried from LA [14:36] that being said, i'm unsure as to why .98 is mzima, .99 is mzima, .100 is ntt [14:36] considering they're the same physical system from both ends [14:36] (I didn't even notice, all my own traffic is IPv6 which is unaffected) [14:37] haha [14:37] mmkay I can confirm I'm not able to reach over timewarner/nlayer/mzima [14:38] (I love that Timewarner routes my traffic through San Jose, instead of going straight through Seattle) [14:38] he.net or whatever routed me once [14:38] from sf to la [14:38] via seattle > japan > hong kong > into la [14:39] lol HE does silly things sometimes [14:39] TW routes all my traffic through Denver. Makes no sense. (I'm way closer to Seattle, and it's not like there aren't links between Spokane and Seattle) [14:40] what amuses me is that vl5.s1.lax.arpnetworks.com is a standard 10-15ms, but the last hop being my server up spikes to a consistent 250-300 [14:40] (Then again, Washington/Spokane/Seattle are owned by Comcast) [14:40] i don't know routing :V [14:41] ahh, so I'm not the only one experiecing issues :) [14:42] welcome to the party [14:42] * hazardous hands you a clown [14:42] hah [14:42] *** heavysixer has joined #arpnetworks [14:42] *** ChanServ sets mode: +o heavysixer [14:43] 8 any2-ix.la.arpnetworks.com (206.223.143.166) 48.691 ms 12.586 ms 12.562 ms [14:43] My last hop before things die out [14:43] and I'm back... [14:44] me too [14:44] *** sinecurist has quit IRC (Quit: leaving) [14:44] *** tabthorpe has joined #arpnetworks [14:45] me too [14:47] *** thestereobus has quit IRC (Quit: thestereobus) [14:48] CPU util coming down [14:56] *** heavysixer_ has joined #arpnetworks [14:56] *** ChanServ sets mode: +o heavysixer_ [14:58] *** heavysixer has quit IRC (Ping timeout: 248 seconds) [14:58] *** heavysixer_ is now known as heavysixer [15:05] *** heavysixer_ has joined #arpnetworks [15:05] *** ChanServ sets mode: +o heavysixer_ [15:06] *** thestereobus has joined #arpnetworks [15:07] *** heavysixer has quit IRC (Ping timeout: 264 seconds) [15:07] *** heavysixer_ is now known as heavysixer [15:10] <_elf> up_the_irons: s1 is a switch? [15:10] _elf: yes, a Cisco 4500 [15:10] <_elf> ah; someone loop something, or spanning tree, multicast snooping, something else? [15:12] something else. might have been a "log" keyword on an ACL. However, that ACL has been in place since the beginning of time, and hasn't caused an issue.. but i removed it for good measure [15:12] CPU util came down [15:12] I then put it back and CPU util was still good [15:12] so perhaps coincidence [15:12] not really clear [15:12] <_elf> funny [15:12] I'm going to keep the "log" ACL off for now [15:12] <_elf> sh proc cpu | e 0\.00 but I am guessing you know about that [15:13] i generally do "sh proc cpu s 5m" [15:13] <_elf> ah [15:13] thanks though, i *didn't* know about that :) [15:13] <_elf> np [15:15] i hope this doesn't jinx anything, but i'm quite proud of this: [15:15] "s1.lax uptime is 4 years, 30 weeks, 3 days, 2 hours, 25 minutes" [15:15] <_elf> haha [15:15] Nice. [15:16] <_elf> you have this one beat by a bit: cs3.511 uptime is 4 years, 9 weeks, 6 days, 13 hours, 32 minutes [15:19] *** heavysixer_ has joined #arpnetworks [15:19] *** ChanServ sets mode: +o heavysixer_ [15:20] _elf: hah cool [15:20] _elf: what make/model switch is cs3.511 ? [15:21] <_elf> Cisco 3560G-24TS; sort of sits in the middle between some border routers and aggregation [15:21] <_elf> (along with another) [15:21] *** heavysixer has quit IRC (Ping timeout: 248 seconds) [15:21] *** heavysixer_ is now known as heavysixer [15:21] _elf: ah cool [15:36] *** heavysixer_ has joined #arpnetworks [15:36] *** ChanServ sets mode: +o heavysixer_ [15:39] *** heavysixer has quit IRC (Ping timeout: 264 seconds) [15:39] *** heavysixer_ is now known as heavysixer [15:48] *** heavysixer has quit IRC (Ping timeout: 240 seconds) [15:51] *** heavysixer has joined #arpnetworks [15:51] *** ChanServ sets mode: +o heavysixer [15:54] *** heavysixer has quit IRC (Client Quit) [15:56] *** thestereobus has quit IRC (Quit: thestereobus) [16:51] *** thestereobus has joined #arpnetworks [16:51] *** thestereobus has quit IRC (Client Quit) [17:09] *** mnathani has quit IRC (Quit: WeeChat 0.4.1) [17:11] *** mnathani has joined #arpnetworks [17:55] *** heavysixer has joined #arpnetworks [17:55] *** ChanServ sets mode: +o heavysixer [18:01] *** heavysixer has quit IRC (Quit: heavysixer) [18:03] *** heavysixer has joined #arpnetworks [18:03] *** ChanServ sets mode: +o heavysixer [18:20] *** mkb is now known as sdkmvx [19:43] *** heavysixer has quit IRC (Quit: heavysixer) [23:36] *** avj has joined #arpnetworks [23:36] *** avj has quit IRC (Changing host) [23:36] *** avj has joined #arpnetworks [23:51] Anyone here familiar with basic javascript document.write functionality? [23:51] I cant seem to find the right resource for including content from another site into my page