[00:30] *** EhtyarWRK has quit IRC (Quit: I was raided by the FBI and all I got to keep was this lousy quit message!) [01:32] *** LT has joined #arpnetworks [01:37] *** Ahmed has left [02:26] *** _id has quit IRC (Quit: changing servers) [02:38] *** _id has joined #arpnetworks [06:05] *** DDevine has joined #arpnetworks [07:11] *** hien has joined #arpnetworks [07:17] *** baklava has quit IRC (Disconnected by services) [07:17] *** baklava- has joined #arpnetworks [07:22] *** hien has quit IRC (Quit: leaving) [07:51] *** diamondshakes has joined #arpnetworks [07:52] is there an ongoing network outage again -.- [07:52] *** reardencode has quit IRC (Ping timeout: 260 seconds) [07:52] *** [FBI] has quit IRC (Ping timeout: 276 seconds) [07:53] *** [FBI] starts logging #arpnetworks at Thu Aug 04 07:53:07 2011 [07:53] *** [FBI] has joined #arpnetworks [07:53] *** pjs has quit IRC (Ping timeout: 240 seconds) [07:53] * diamondshakes pokes up_the_irons [07:53] same symptoms as yesterday at least [07:53] *** koan has joined #arpnetworks [07:53] *** mjp has quit IRC (Ping timeout: 246 seconds) [07:54] *** kennyz has joined #arpnetworks [07:54] *** milki has joined #arpnetworks [07:54] hm [07:54] they are back? [07:54] *** R496 has joined #arpnetworks [07:55] *** mjp has joined #arpnetworks [07:55] *** bitslip has joined #arpnetworks [07:56] *** Wraithan has joined #arpnetworks [07:56] *** pjs has joined #arpnetworks [07:57] *** pjs is now known as Guest45192 [07:57] What's up with the internet connection down there the last couple days [07:57] twice now I've lost my connection in the morning [07:57] *** reardencode has joined #arpnetworks [07:58] same here [07:58] *** mick_laptop has joined #arpnetworks [07:58] as you can see from all those ping timeouts as well [07:58] no updates on twitter [07:59] Not acceptable. [08:01] *** Guest45192 has quit IRC (Ping timeout: 264 seconds) [08:02] My monitoring software shows it was down yesterday at 6:55 until 7:10, then it was spotty for about 15 minutes. mind you the resolution on those times kinda sucks, I think it is 3 or so minutes, but still [08:32] my monitoring software doesn't show an outage yesterday. Are you sure it's not a routing issue between you and them? [08:34] mine did too... I'm sure quite a few other people here saw it as well [08:37] I too had a problem yesterday at that time [08:37] so whatever is being monitored isn't capturing the user experience fully [08:37] my nagios couldn't ping *itself*. that's insane [08:38] when maayan reports "maayan is unpingable"... [08:38] maayan also couldn't reach chris, another machine on a different kvm [08:44] hrm, are you two on the same host, by chance? [08:44] maybe it's something specific to that host? [08:45] if a host can't ping itself.. thats a bit extreme. are you monitoring the external IP? and internal IP? or loopback? and this a common thing? [08:48] i'd be really interested in knowing if internal pings fail, and if loopback pings fail.. if it's a re-occuring thing. [08:48] I think I'm on 13 [08:49] *** heavysixer has quit IRC (Remote host closed the connection) [08:50] RandalSchwartz: another question. does your 'uptime' change? I'm wondering if your seeing a hardware watchdog/restart thing? maybe watchdog reset is failing, and the machine is restarting even though the hosts are responsive [08:51] (I've actually seen that before. and as the host is restarting, the networking isn't available at the host level, even though the host is running (in the process of shutting down) [08:51] err, starting up rather. [09:02] hmm, actually I'm on kvr08 - my memory is not would it should be [09:19] *** LT has quit IRC (Quit: Leaving) [09:29] *** DDevine has quit IRC (Ping timeout: 264 seconds) [09:51] I'm on kvr02 also also saw some Internet "pausing" earlier. [09:51] I waited it out and it came back within 10 minutes. [09:51] But I don't really do anything but email and IRC from this server. [09:55] We run our ticket system on a VPS there, not sure what hardware that one is on, my personal VPS is on kvr09 and I lost connection [09:55] *** HighJinx has quit IRC (Ping timeout: 240 seconds) [09:56] jpalmer: server wasn't restarted, I would have had a notification for that, also I would have really felt it on my vps as I have some things just running in tmux not auto started (since I am deving on them) [10:04] no uptime restart [10:04] 10:08AM up 47 days, 16:09, 5 users, load averages: 0.00, 0.00, 0.00 [10:04] some irc connections dont ping out [10:04] screen sessions still alive [10:05] I saw the same thing yesterday from 7:50 to 8:15 [10:05] my internet went away, but the box was otherwise unaffected. [10:05] as in "went away", I mean I couldn't even ping my own address. [10:05] it was very weird. [10:06] yup [10:08] Hmm. I thought you could change the CD in the tray via the console already [10:08] kvr12 had some load issues for a bit (yesterday) [10:08] I guess that part isn't automated. [10:08] i just reset a bgp session now (packetexchange peer disappeared, wth?) [10:10] :o [10:10] up_the_irons++ [10:11] diamondshakes++ [10:11] RandalSchwartz: "own address" meaning external IP? internal IP? loopback ip? [10:11] my public IP [10:11] jpalmer: www.arpnetworks.com was unreachable during the time as well [10:12] that's what I'm testing in nagios [10:12] maayan couldn't ping its own public IP [10:12] nor could it reach chris [10:12] (.insightcruises.com) [10:12] RandalSchwartz: any chance you could set your monitoring to ping a local IP? as well as loopback? I wonder if it's a deeper issue. [10:12] *** pjs has joined #arpnetworks [10:13] jpalmer: well, maybe up_the_irons's reset might help things? [10:13] diamondshakes: if he can't ping from his box TO his box, I doubt it's a BGP peering issue. [10:13] diamondshakes: he's saying his nagios box can't ping itself. [10:14] o [10:14] yeah - I've seen this before too. [10:14] yeah that's not a bgp issue [10:14] about once every six weeks [10:15] lasts about 10 minutes [10:15] basically, net connections die, and the box can't ping itself [10:16] and as we see, it's not just my boxes. seems to be rather systemic [10:16] it's odd though. because if you're pinging the same IP (from that box) the traffic shouldn't even be hitting the switch. so it's not an ARP table corruption issue or something.. [10:16] but it still has to go through the VM's drivers [10:16] so maybe there's a VM problem [10:16] as in, I'm pinging the "emulated" em0 [10:16] thats kinda the direction I'm headed. [10:17] you're right in that a good test to add would be to ping the lo0 interface too [10:17] and see if they both die at the same time [10:17] I'll get on that [10:18] I think you should add 2 more "hosts" to your nagios. bring up an RFC1918 address (as an alias on em0) and also monitor the loopback address as a seperate host. this will tell you if it's specific to "em0" or something else. [10:18] ooh. yeah [10:18] more things to add for this weekend's "hours billable to Neil" weekend :) [10:20] I wonder if there is a newer version of the virtualization osftware than ARP is using. might be worth investigating the Changelog from that version on. [10:20] you don't have to do rfc1918 [10:20] ok - in transit... back soon. [10:20] just ping the link local v6 address [10:21] toddf: good call. also, it's probably worth finding out if it's isolated to ipv6, or ipv4. [10:21] *** heavysixer has joined #arpnetworks [10:21] *** ChanServ sets mode: +o heavysixer [10:29] given the description, I'd lend towards os specific handling of the emulated nic. would be interesting if any openbsd people had the same issue, or other os's .. I suspect as has been stated it is an 'emulated nic stops responding properly' maybe interrupts stop happening properly or something .. for 10mins .. then resumes again [10:31] *** diamondshakes has left [10:31] *** HighJinx has joined #arpnetworks [10:34] toddf: completely different versions of the linux kernel having this problem? Seems unlikely. [10:34] especially given that we use kvm/qemu at work and I've never seen that issue on our own hardware. [10:37] Wraithan: linux? I thought guests were generally *BSD [10:37] I'm talking guest os support for the emulated nics here [10:38] toddf: thats a good call. all my VM's are centos, and I haven't noticed my monitoring complaining. [10:38] I guess from context you use loonix inside as the guest os. interesting to know. more info is useful until it is determined what is exactly happening. (I suspect one vps going bezerk for a bit then settling down, but thats just a wild stab in the dark) [10:38] toddf: The two VPSs on ARP that I have are linux, one Ubuntu 10.10 the other Arch Linux [10:41] I have a total of 3 VPS's, all are centos. (2 5.x, 1 6.x) and I haven't seen this issue. [10:41] toddf: and it has been 2 different hardware machines, since LT said he is on 09 and I am on 08 [10:42] murphy [10:44] bezerk can be something that effects networking in general since vlans can (and are for some clients) available on multiple host systems [11:12] *** heavysixer has quit IRC (Remote host closed the connection) [11:27] got to promote arp on the freebsd mailing list a moment ago [11:28] awesomeness [11:30] oo, i see the email [11:30] 13 minutes ago [11:31] freebsd-questions@ [11:31] and its already archived in the web [11:32] http://tinyurl.com/3gdwzdm [11:32] :P [11:33] http://v.gd/arp_plug comeon, create one with some style? [11:33] ;-) [11:33] >.> [11:34] it doesnt go direct >.> [11:34] though I will take exception, last I checked, elastic hosts is not what I would consider a similar pricepoint. [11:34] oo, qr code too [11:35] its in the 'cheaper than ec2' bucket perhaps but .. I've yet to find anything similar to ARP until it gets to be a very large machine and then why not just host real hardware at that point? ;-) [11:35] (I look when people try to convince me otherwise, and always walk away knowing I've got the best deal around) [11:36] maybe i am like 3 years behind, but is "cloud" another word for "virtual server" ? [11:36] it wasn't "triple the price of ARP" like the other vendors I found [11:36] no [11:36] and people are just selling the same old thing under a new name? [11:36] so it made it closer to "similar" [11:37] but you can treat it as such [11:37] similar but more :) [11:37] tons more [11:37] and no ipv6 yet [11:37] haha indeed [11:38] cheapest one i can find there is $4/mo [11:38] *$44 [11:56] yeah - but once you start pricing the $75-ish and up VPS, it gets closer [12:08] randalschwartz: ah, I hadn't remembered that, yeah the size of vps's you are using makes them similar indeed. I'm looking at the low end ;-) [13:19] gamarco: "cloud" is a useless marketting term, that has no real meaning or value. ask 100 people what "cloud" is, and you'll get 85 different answers. [13:20] it's a vps, no it's a colocated server, no, it's a web app. no, it's virtualization on the local lan, no it's.. all bullshit. [13:22] the only 'concrete' definition for cloud is "it's a real, or virtual thing, where you can host your data or application, either on the internet, or locally" which essentially means.. every server, website, application, or database is a compnent of this nebulous 'cloud' idea. [13:23] * jpalmer keeps meaning to write a blog article about the uselessness of that term, and post it.. in the cloud. [13:24] whenever i heard cloud computing [13:25] i think of peoples heads up in the clouds being blinded by them cluelessly [13:25] saying the world clouds over and over [13:26] and only apple could take such a useless term, and make it not only useless, but redundant by calling it 'icloud!' [13:26] lol [13:27] what have those clouders been smoking [13:27] (AND successfully SELL it, no less.) [13:27] i want some [13:32] o/~ I've looked at clouds from both sides now... o/~ [13:32] Its no shocker that magazines such as PC world [13:33] are pushing the term cloud [13:33] just another buzzword [13:33] HAH! I just quick polled the office. asked 18 people. guess what? no TWO people answered "when I say something about using 'the cloud' what exactly am I referring to' no two people said the same answer. [13:33] lol [13:33] its because its so rediculously vague [13:33] yes - everyone has their own individual cloud! that's the beauty! [13:33] My Personal Cloud [13:34] Jean Cloud van Damme [13:34] I hope my cloud does not rain away [13:52] ok, finally.. 2 similar answers (I'm at 33 people polled now) and similar in a "white and balck are similar, in that they are both colors" kinda way. [13:54] (one said "internet" one said "anything on a network, or not on your local machine") [13:55] That's about as vague as possible; seems like the right answer. [13:55] *** wallshot has joined #arpnetworks [13:55] hehe [13:56] *** heavysixer has joined #arpnetworks [13:56] *** ChanServ sets mode: +o heavysixer [13:56] by the second definition.. if bobby and sue are in an office, and bobby saves a word doc to his own machine.. to bobby, it's not a cloud document. but to sue, it is! [14:05] love that cloud [14:05] it's got everything in it [14:19] *** heavysixer has quit IRC (Remote host closed the connection) [14:41] *** nomadlogic has joined #arpnetworks [14:54] i wanted to say thanks to the folks at arpnetworks! [14:54] just setup a new vps and was really happy to see ipv6 configured by default! [14:55] it was also trivial to reinstall my node to freebsd-8.2, so thanks guys! [14:55] yeah the ipv6 is nice [14:56] gonna spend tonight migrating my rootbsd server over to my new arpnetwoks node [14:56] :) [14:56] that's always good fun [14:57] tho i guess almost anything with FreeBSD is good fun [14:57] lol - good point [15:10] nomadlogic welcome to Arp [15:10] are you using zfs as / on your node? [15:10] if not, may I strongly suggest it? :) [15:12] thanks! [15:12] there are wiki entries about how to do it here [15:12] or I can give you the URL I have bookmarked [15:13] my plan was to purchase more space from you all (or get a second instance) and run zfs on there [15:13] i'm gonna need to buy more space probably at some point :( [15:13] zfs as root is easily extended though [15:13] echo "please increase my disk to 120GB" | mail support@arpnetworks.com [15:13] (time passes) [15:14] heh [15:14] boot from DVD, type "gpart recover" "gpart resize blah" [15:14] and reboot [15:14] can you do that live [15:14] or no [15:14] and now you have more ZFS disk [15:14] I don't think so [15:14] or not sure with ZFS [15:14] i have boring UFS / =/ [15:15] i was initially going to run 9.0-CURRENT on this guy for some testing, but there is a bug with the installer that ships with the 9.0-BETA1 iso :( [15:15] I tend to avoid bleeding edge [15:16] in fact I stay off a release for at least a few weeks [15:16] just call me conservative :) [15:16] heh - yea, for prod systems no doubt :) [15:17] i was hacking on the freebsd amazon port that colin percival is doing for a while and was hoping to do more testing on my own [15:17] guess i'm not a good tester though since i didn't report the installer bug i ran into :p [15:17] heh - I interviewed him about that yesterday [15:17] twit.tv/floss176 [15:17] nice! [15:17] "it's what I do" [15:52] jpalmer: i like the way you think [16:06] is there a place to check how much bandwidth I've used so far this month? [16:23] cricket graphs, from your dashboard [16:23] thanks! [16:29] *** EhtyarWRK has joined #arpnetworks [16:38] *** bob^^ has quit IRC (Read error: Connection reset by peer) [16:38] *** bob^^ has joined #arpnetworks [16:40] *** Yamazaki-kun has quit IRC (Ping timeout: 260 seconds) [16:40] *** Yamazaki-kun has joined #arpnetworks [16:40] *** plett has quit IRC (Ping timeout: 264 seconds) [16:40] *** plett has joined #arpnetworks [17:05] *** nomadlogic has quit IRC (Ping timeout: 246 seconds) [17:49] *** heavysixer has joined #arpnetworks [17:49] *** ChanServ sets mode: +o heavysixer [17:58] RandalSchwartz: you could have easily gotten $25 consult fee for that [18:00] for the interview, or for pointing me at bandwidth graph? [18:00] :) [18:00] graph. didn't see the interview. $50 easy for that =) [18:01] yeah i mighta gone another month or 3 before actually asking support@ [18:01] but irc helps me fight my own laziness [18:17] *** wallshot has quit IRC (Remote host closed the connection) [18:50] *** HighJinx has quit IRC (Ping timeout: 252 seconds) [19:45] A number of your customers complain about inability to access their machines, I have data, others have data, showing it happened yesterday as well. Please, please, tell me what to call that, other than down. [19:46] Don't be dicks and try to hide when bad shit happens to multiple customers, address it, even if you are still investigating just say so, don't call me out as a liar instead. [19:48] And if you aren't investigating it, say so, so I can take mine, my company, and several friends to some other service who cares. [19:48] *** tigerpaw- has left "rawr?" [19:48] *** tigerpaw has joined #arpnetworks [19:50] Wraithan: have you created a support ticket? [19:51] G: I had assumed that due to mutliple @'d people in this channel replying to the multiple people in this channel reporting problems that it was being taken care of [19:52] Wraithan: as the topic says, all bar one @ are long-time customers, and generally IRC isn't really an official communication method [19:52] As this is a channel with a single # that means it is official, unless there is a violation of the freenode rules. Under that assumption, reporting something here and having an @ say something in return, espcially one like up_the_irons who I know for a fact works for the company. [19:54] I only had one report of an issue and my monitoring did not alert me to any issues either; I reported this morning that I bounced a BGP session as well [19:54] up_the_irons: should I assume you don't read anythign when you are typing in here [19:54] as the problem this mornign was reported right before you said you rebooted that [19:55] So, apparently, the "outage" was enough to have people talk about it in here, but not enough to email support. That doesn't sound like much of an outage. [19:55] up_the_irons: well the day before the network really went nuts [19:55] up_the_irons: If I talk about it in here, when you are around, I'd assume you are paying attention. [19:55] Should I not assume that. [19:55] ? [19:56] yeah, this morning i saw some people talk about an issue, so I checked my bandwidth graphs, noticed a session down. So I bounced it. I do not monitor IRC 24/7, but I check the support desk quite often [19:56] Wraithan: do not assume that at all, I run IRC in a screen session. I'm *always* logged in, but don't assume I'm actually at the computer. [19:57] up_the_irons: I assumed because you spoke during the convoration [19:57] not because you were online [19:57] I am online all the time too. similar setup. [19:57] *** mike-burns has quit IRC (Read error: Operation timed out) [19:57] *** mhoran has quit IRC (Read error: Operation timed out) [19:57] if I speak, then I'm here [19:57] fuck it, not worth the energy [19:57] I'll file a ticket [19:57] i may not necessarily read all the scrollback, but i usually do [19:57] *** Wraithan has left "WeeChat 0.3.6-dev" [19:58] eek what the... cloudkick deleted my node from it's monitoring [19:58] ouch [19:58] *** mhoran has joined #arpnetworks [19:58] *** ChanServ sets mode: +o mhoran [19:58] *** mike-burns has joined #arpnetworks [19:58] *** ChanServ sets mode: +o mike-burns [19:58] I have no idea either [19:58] *** Wraithan has joined #arpnetworks [20:05] Wraithan: for the record, I don't hide when bad things happen. I was *not* investigating anything either, since I had no reports of problems and my monitoring did not page me (this is a weakness on my part for not having geographically disperse nagios setups). So if this makes you uncomfortable, I'm more then happy to admit it now so you can find a provider that suits you better. I value [20:05] honestly and transparency; I don't want to hold anyone here if they don't want to be here. I was moving into a new house all day yesterday, so Iwasn't online at all except for checking emails (e.g. support) from my android [20:06] and with that, I must leave the office [20:06] cd $home [20:18] *** HighJinx has joined #arpnetworks [21:31] * tuv just discovered lovevps.com. 512MB xen vm for $7 [22:09] *** DDevine has joined #arpnetworks [22:25] hey up_the_irons [22:25] at some point, we shall meet :) [22:25] I'm in pasadena this weekend [22:29] Wraithan - up_the_irons is pretty transparent, but ARP may not be the right place to host you. [22:29] lots of nines, but maybe not enough. :) [22:30] it's lots of cheap nines, but if you want more, you'll pay [22:32] tuv - remember that xen = paravirtualized [22:39] RandalSchwartz: hmm.. didn't know that. what's (or where can i learn about) the difference? [22:40] try http://lmgtfy.com/?q=xen [22:41] RandalSchwartz: xen != always paravirtualised. [22:41] Xen can do full virtualisation as well. [22:42] HVM. [22:42] not originally [22:42] Yeah [22:58] RandalSchwartz: IMO 9's are practically worthless. The simple fact is that you have to architect your own availability not your hoster. [22:58] Not even Amazon is bulletproof - whole regions *have* gone unavailable before. [22:58] recently. [22:59] They basically DDOSed themselves. [22:59] Not DDOS, just DOS. [23:04] too bad chunkhost is so persnickety with their setup, supposedly full HVM, but 'one partition filesystem' doesn't convince me. [23:26] in any case, $7 for a 512MB xen vm is pretty damn cheap, and googling lovevps didn't yield any complaints. though they are relativelly new [23:27] Interesting. [23:27] They could be overcommitting. [23:28] who knows. but they do have 'NO OVERSELLING' on their home page header [23:29] heh yeah I just noticed that. [23:30] they seem to be reselling from few other services: burst.net, hostdime, ... [23:30] (they have an openvz option) [23:51] RandalSchwartz: ah cool :) [23:52] heh, 'NO OVERSELLING'. 1) don't believe everything you read, 2) "overselling" is relative ;)