:) do you still have to specify ip unnumbered these days? now, if i assign the .0 to *another* router (but same vlan), for redundancy let's say, then have the host configured to have .0 as a lower priority default route, this could present some level of gateway redundancy on /30 links, something that is somewhat difficult (VRRP and HSRP don't support /30's "out of the box", need tricks) mercutio: as of IOS 12.0, it is the default up_the_irons: i have a diff idea :) mercutio: do tell :) can you get kvm to restrict ip's? and then just use /24s and vrrp? mercutio: so a shared vlan type setup (which I think you're getting at) is not feasable (counter to my ideals of customer segregation and network design) Who the hell needs a 16 TB file, or a 1EB file system? btrfs is even more insane, 16 EB for both. up_the_irons: ok.. so it's partially about being shared on one network/ arenlor: insane segment up_the_irons: Do you even have 16 TB total? well the other way to go is you take over the ip completely with a different router if the router goes down mercutio: well, two things: 1) majority of customers have only 1 IP (/30), 2) VRRP or HSRP aren't exactly noted for their success stories. It is better for the client / host to make the proper routing desicision then to move some "magic" IP between separate switches because i think it's a different story using 4 ip addresses on a /30 then using a /31 with 2 addresses. arenlor: 16TB total of what?:) up_the_irons: Disk space. well what i hate most about vrrp is when things half work arenlor: on all my hosts? because it's going out one router and coming back through the other. up_the_irons: Yeah, just wondering if ext4 could create a file larger than all your hosts could hold. but vrrp does work reasonably often.. it's just when it doesn't it's more complicated. mercutio: that's actually not too big a problem; the bigger problem is the IP moving around oh? i've never found that to be a problem? mercutio: routing is asynchronous anyway, so it doesn't matter if traffic comes in one way and leaves another up_the_irons: i just mean when things break it makes it harder to debug but all redundant solutions are harder to debug arenlor: in total, across all hosts, i have about 104TB of space; but that couldn't be used for a single file obviously... mercutio: yeah up_the_irons: you're using local storage? mercutio: yep mm up_the_irons: True, at least makes me a little more sane to know that you have more space than ext4 can take up in one file. Btrfs would still think that's a tiny file. how often do you have router failures? arenlor: just because things "can" scale doesn't mean they will arenlor: lol mercutio: I've only ever seen one I think, and it was for only a short bit. i would be very hestitant about having a 100tb file system with btrfs hell i'm hesistant about having a 1tb file system with btrfs mercutio: Try 16 EB cos i think in some ways automatic failover isn't really necssary but having backups of router config being able to get something up and going quickly etc is handy mercutio: i've never had a router failure, but i know one day it is inevitable. i am preparing for that possibility. i hate single points of failure up_the_irons: well so do i :) .. but i think host failure is more likely and while you're using local storage... up_the_irons: Have you looked at netflix's monkey to see if it's usable for you? mercutio: well, yes, and so i've already prepared for that :) there's a higher chance of a protracted outage from that then router. ok :) arenlor: i only just heard of it haven't really looked further not trying to tell you what to do. just giving my input :) chaos monkey or something? up_the_irons: Yep mercutio: we have failures in local storage frequently, but it is a RAID 10, so the drive gets replaced. No big deal, that problem is already solved. up_the_irons: Why 10? i watched a talk by a guy from Edgecast; they have 6000 servers. he says they have a HD failure every 12 hours :) well as long as you don't have two disks die at once :) arenlor: speed and more drives can fail mercutio: sure, of course up_the_irons: i wonder how many memory stick failures they get yeah i've only had 1 memory stick failures can be confusing though spinning things die a lot faster... ;) yeh are you using sata disks? Heh, this is how efficient RAID 2 is: 1 − 1/n ⋅ log2(n-1) huh? what's raid 2? and those don't show oh hamming codes :) mercutio: You don't know what RAID 2 is? well it's not exactly common but i found wikipedia entry ;) mercutio: True. I have to go lookup what exactly RAID 1+0 does, but I know RAID 2. I think something is wrong with me. arenlor: heh heard of raidz-2? mercutio: Nope. basically 4 disks plus 2 partiy or such but any two disks can die mercutio: yeah, sata up_the_irons: re4s? mercutio: Nice. mercutio: nah, too expensive and i've had just as many failures with so-called "enterprise" drives than regular drives, so i stay with the regular ones up_the_irons: ahh.. i've been using diff disks and i've noticed that some disks are a lot faster with raid 10 type configs than others err or some are slower but it's not the same as desktop benchmarks yeah mercutio: which are you finding are the fastest? umm slowest is ST32000641AS WD2002FAEX-0 is faster i'm not sure what fastest are :) zfs is kind of special in that it'll push more load to faster disks though err i mean "idle" disks like it's not strictly balanced ah but yeh the seagates seem wlower that's an older seagate though older wd too i haven't tried in raid yet but in desktop usage the new seagates are fast the new ones are 1tb/platter think is it's about seek time and stuff more when you do lots of small tranfers that happens with raid those seagates are from when the flooding happened and limited availability of hard-disks yeah time for a break up_the_irons: around? got your message from last night the VPS was indeed booted, and vnc never responded when it is shut down, i get a 'connection refused' which is a slightly different scenario rebooting did solve the problem, vnc is now back up, but I was hoping to not have to resort to that, as I lost my console session. anyone know roughly how much disk it would take to apt-mirror the ubuntu 12.04 updates locally? mikeputnam: No idea, but I think that up_the_irons has/was working on a local mirror. found it https://wiki.ubuntu.com/Mirrors/ ~600GB mikeputnam: yeah, 600GB is about the size of our Ubuntu mirror ryk: see the "Some things to note" section here: http://support.arpnetworks.com/kb/vps/out-of-band-management ryk: you might have had another session (stale) open somewhere else interesting. what if it was on my computer? as in, the original session was on my computer, and i rebooted; shouldn't that have reset it? ryk: oob connects to a tcp socket the kvm instance that is the equivalent of 'qemu' running on his system. ryk: the only time you get disconnected is if you poweroff your kvm instance *egg on face* sorry up_the_irons I forgot you have the older libvirt that can't do the 'system_reset' equivalent just yet so if you reboot your guest os, your oob will survive if you hard poweroff your vps, your oob should die and force you to reconnect .clear heh oops hum, My server ping time out 4 time tonight is there any reason i havent had any problems Seems like some network issues, but I'm not having any troubles anymore. Possibly routing, I only lost connection from home. it's ok now but 13h30 to 14h35 (GMT -5) got 4 connectivity drop i lost connection too must be sunflares or thor is sucking up our bandwidth heh i was wondering where you were connecting from redsteal but you're connecting frmo your vps i see :) redsteal: what is your vnc host? (so i know what host you are on) think I found it, kvr25 No issues on that host tonight < redsteal> hum, My server ping time out 4 time tonight is there any reason <-- yeah, up_the_irons got bored this evening, went into the datacenter, and was randomly unplugging cables and plugging them back in to see if anyone would notice seriously, though, the issues could have been at any hop in between you and your vps jlg: mhoran had issue too? but they both may be using hte same isp or path according to nanog bell canada had a big issue recently and peering between glbx and level 3 broke mzima seems to send a lot of traffic through glbx... and level 3 are huge.. so if your isp is using level 3 that could explain it (and mzima don't directly peer with level 3.. but peer through glbx) i thought level3 owned glbx though and i do mean gblx not glbx mm reading furhter http://www.bgpmon.net/bell-leak.txt bell canada did a route leak.. which included arp networks http://seclists.org/nanog/2012/Aug/231 my monitoring systems didn't report any issues reaching my vps' jlg: i didn't notice any issues too (my own instance of opsview -- through level3 in chicago -- and pingdom -- through multiple carriers throughout the world) arp has several different prefixes, though, so maybe they only hijacked a portion of them (and not the ones my vps' are on) it was all of them it seems to be because arp have transit with tata oh hangon arp doens't have transit with tata do they? i thought it was trit they had transit with but basically the gist is bell canada was advertising when they shouldn't have been and some people may have got a preferred route through them which is what route filters are foor limited by bell canada and yo'll find the thread *nod* flipped over to my nanog folder and that thread is right on top Other than massive ping times, can anyone tell me what could be causing this traceroute: http://pastebin.com/ip8ata5y because I'm wondering if my Internet is about to disappear on me. latency/congestion in the ATM network between your CPE and your ISP's BRAS ? or solar flares you could be uploading a file arenlor it's normal for adsl connections to ping upwards of 1 second when uploading due to "bufferbloat" mercutio: Nope, that's just what my Internet does to me. well there's extreme congestion is there any open wifi? bittorrent? mercutio: Nope and nope. is your sync rate ok? mercutio: Yep. It just is really shitty service. I've seen it drop to ~80kbps what's your sync rate? are there errors on the adsl statistics? did you watch the video of the nanog presentation i linked to a few days ago? jlgaddis: I downloaded, but have not had time to watch it yet. jlg: what's the presentation on? i've made all my colleagues watch it a while back and suggest it to others all the time jlg: is it worse during the evening? mercutio: traceroute i think the biggest problem with traceroute is you need to do it frmo both sides and with udp and tcp and icmp now days :) well sometimes Heh, seems there are tons of errors with my dsl. arenlor: what's your atteunation? attenuation? downstream and upstream if it's like > 60db you are likely to have shit connectivity mercutio: Not sure where to find it. what kind of modem is it? Shitty? what moedl? model a whole lot of them have mock interfaces on the net lol or docuemntaiton so i can tell you where to go :) you could also try telnet lots of broadcom chipsets have telnet open admin/admin mercutio: It says Sagecom SE567 on it, but no model number.\ adsl info that's a model number It's one of the ISP provided things. are you with bell canada? mercutio: Frontier. no idea who that is but i serached that modem and it came up with bell canada who had massive issues today :) mercutio: 4th largest telecom in the US oh ok :) i'm in new zealand... mercutio: I know ok just saying i'm not completely ignorant just not aware of stuff there so much If it was Verizon, AT&T, or CenturyLink you'd have heard of it. http://www.broadbandreports.com/comments/1677?1=1&p=7 i know comcast, verizon, at&t dunno centurylink verizon fios seems common? well with geeks i suppose :) mercutio: Third largest telecom. Those three are the surviving Baby Bells. i'm not having luck finding much is it the same as gigaset? http://www.wikidevi.com/wiki/Sagemcom_SE567_(Frontier_Communications) i think you had typo :) oh god it's TI chipset Got a clue what OAM end-to-end is? ok do you have atm statistics? Yep ok pastebin me them? http://pastebin.com/bSZf1a8K it really doesn't line up very well mercutio: How so? click "raw" oh, doesn't help 88520 rx errors? arenlor: the columns to the words 88520 errors isn't bad enough to explain his problems he's been doing a lot of packets also look how high his transmit is compared to receive i mean he's doing more transmitting than receiving aren: you might have a virus are you using windows? mercutio: HGC (Hutchinson Global Communications) also leaked routes, for months actually. I tried to contact them, as well as Internode, but to no avail. I actually had to shut off their peer. Was causing very bad routing from AU -> US through Internode mercutio: Yeah, it could be that Freenet isn't being fair. arenlor: freenet? mercutio: The Freenet Project up_the_irons: curious that internode are hard to contact arenlor: do you do soemthign with them? well you need to stop uploading as much arenlor is the gist. mercutio: Internode isn't hard, it is HGC that is hard mercutio: I have it limited to < 20k up up_the_irons: oh right i know nothing about HGC arenlor: just disable it and see if problem goes away mercutio: I've tried, totally shut down the computer it's on, didn't help. up_the_irons: you don't have transit through TATA do you? arenlor: how many computers do you have? http://i48.tinypic.com/34g9buo.jpg should line it up better. dude you have 384 kbit upload that sucks :) well i'm sure you know that sucks i wouldn't upload anything with 384kbit mercutio: Depends on how you count. 3 computers, 2 cells, plus a verizon wireless extender. mercutio: no Tata one of the problems with adsl is that if you have lots of small packets it can easily use up more than the kbit rate mercutio: You're wrong, I have 384kbit MAX. also when it says 384 that includes overhead The other day it was at a nice strong 8kbit. arenlor: you are likely to get about 38k/sec max arenlor: can you hook up just one computer? mercutio: I'm not sure what you mean by hook up one computer? you've done like 3.7 gig of upload in 10 days. 1 megabit can do 300 gig in a month (rough rule of thumb) it's actually a bit higher I wonder how much of that is that it's our cell tower too. so that's like doing around 10 gig of upload a month so 1/30th of 1 megabit which means your average utilisation is about 1/10th of you rbandwidth mercutio: MAX bandwidth. do you have a static ip? mercutio: Nope, and every time you've seen my disconnect in here (~1/4hrs) I get a new one. i'd suggest running smokeping oh bah just change isp's :) if you have a linux host at hoem you could run it from home generally speaking if it's your area/dslam being congested mercutio: Yeah, speaking of, if next Wednesday 16:30 ET passes and I'm not happily annoucing the comcast guy is here, turn on the news that evening to learn about the destruction my wrath brings on them. aand what someone said about atm ould be correct then it'll be worse during the evening mercutio: My tower is Linux, and I duel boot. yeh the thing is you want to not pay your bill :) based upon them not providing you a reasonable level of service mercutio: I'm in America. you can't do that there? If it's on most of the time, they have fulfilled their contract. don't you have any "reasonable" expectations? like 8kbit isn't broadband web sites not loading etc mercutio: *I* do. They just have no requirement to meet them. i meant in law :) i dunno mercutio: That's what I mean, they have no legal requirements. there have been court cases here over packet loss :/ but it's a bit diff for an individual well yeh comcast may fix it but yeah 384kbit upload will suck at the best of times well kind of dropbox will screw it badly web browsing will screw it uploading files etc mercutio: up_the_irons can provide shit service and all we get a choice to do is leave. Most ISPs in America require you to be in a contract, so you can't leave. hmm i hate contracts i had a year long contract for dsl once and it started sucking if service is shitty, even with a contract, you can always cancel for non-performance mercutio: only one year? although it was near the end of the year i bitched and it got faster well it didn't quite go like that up_the_irons: Yeah but I only get "up to" so much. basically there are two ways for isp's to get DSL here with different handover/uplink and one of them sucks and the other is good and i got them to move me to the good one by specificalyl asking for it and most people would probably not know about hta that I'm basically just happy right now that the phone can ring without taking down the Internet. but like most providers in this country resell dsl and all had the same issues arenlor: do you have a master filter? oh bah you're moving to cable anyway aren't you my dsl disconnected today :( mercutio: Yes mercutio: That sucks, what happened? and it wouldn't resync well it resynced but pppoa didn't come up i rebooted it and it came back up but generally dsl is ok here about 8 to 9 msec latency mercutio: I called in and after a few hours they did something on their end, including something outside the house, that fixed the issue. i have smokeping of it I'm not even sure what in the hell could cause your phone line ringing to take down your DSL. hmm it's around 9 msec atm arenlor: there's mre than one rason that can happen i forget though mercutio: I had good filters on. yeh there's another reason to too when i got my line hooked up my phone didn't work but my net did and it didn't ring lots of werid stuff can happen then found out it did ring but with no noise mercutio: That's really screwed up, but at least you had the Internet. oh it was dead copmletely at first yeh then the internet went away and i'm like wtf then someone turned up bah my attenuation is still higher than it should be though which blocks me getting vdsl atm probably so i'm stuck with shitty 16/1 internet 16/1? 16 megabit down 1 megabit up hahaha yeh i dunno comcast is pretty fast isn't it? I'll have 30 down, not sure what up, but the next lower was 4 up. They sadly don't have the 105/105 out here :'( tbh 10 an d20 megabit isn't that diff for just web browsing mercutio: I don't just browse the web. multiple users and simultaneous transfers etc more helps you've got 3 megabit atm :/ mercutio: Max. yeh i had 1 megabit temp at the beginning of the year I measured it a bit ago. 1.25 oh real on speedtest.net? i get over my line speed on speedtest.net it's unreliable it's better to just do an http download mercutio: Comcast and speedtest yeh don't trust those things they can overestimate i mean my adsl sync rate is like 17 or 18 megabit One entertaining thing with this is that it takes time to echo keystrokes when I'm ssh'd into my VPS. and there are overheads and i can get like 18.5 megabit mercutio: Comcast has a file you download. oh real i wonder what i get to it i found some flash one 11.79/0.84 to san jose comcast peak 17.75 megabit that peak is wrong oh and i have to snr tweak agian I found a file while googling. ahh ok cool resynced Path: 0, Upstream rate = 1144 Kbps, Downstream rate = 17603 Kbps Nice, but what K are they using? http://stage.results.speedtest.comcast.net/result/160281622.png that's kilobit the upload was way too short and ping is like twice what it is mercutio: DSL reports may be better. I haven't checked theirs for a long time. http://stage.results.speedtest.comcast.net/result/160281772.png see how the download speeds are higher than they should be Yeah so yeh that's why i don't trust them texas has lower ping than san jose mercutio: I'd like to see how that network happens. what do you mean? Still, if I'm supposed to have 2.5-3M and I'm getting 1.25 on the tests, I can assume that 1.25 is closer. Chris: Hi, I'm a live Comcast product specialist. What questions can I answer for you today? Chris: Just type your question below. You: stop spamming me Chris: I apologize for the inconvenience. Our chat window box will pop up once you are on the home page and is left idle. Chris: Please enjoy your time on our site. In the meantime, let's move this window. That way, you'll still see this web page and I'll be here to answer any questions you may encounter. Please click 'Move Chat Window' to move it now. Chris: Once you click on 'Move Chat Window' you can view our site and I can continue to assist you. Chris: How are things coming along? Chris: Are you still with me? am i the only one that hates those things? mercutio: I don't see those. Chris: I haven.t heard from you in a while. Would you like to continue to chat? You: Look! Leave me alone. when does it go to a human? mercutio: Try cursing or saying call 911 or something? http://www.dslreports.com/im/102361591/78188.png i'm asking if they have gigabit what did comcast speed test say? mercutio: Careful, it's an American company, we've all seen what an American company can do to Kiwis. huh? mercutio: Megaupload? http://stage.results.speedtest.comcast.net/result/160282572.png that pignis fine is traceroute still high? mercutio: It's lower. In 300s now, which is normal for half the time. heh, tweet from @bsdvps popped up in the bottom right of my screen and as i was reading it an email notification (alert for vps) popped up in the top right. good timing. mercutio: usually typing in (or saying, if you're on the phone) "cancel" gets you transferred to a real person pretty quick jlg: ahh up_the_irons: "Date: 03/01/2010" whoops jlgaddis: yeah i noticed that right after :( a bit flustered right now, i've been holding the hand of this raid since this morning understood. no worries. up_the_irons: Maybe you should stop the raid and pay attention to your servers :P wut?:) up_the_irons: MMO joke. LOL you cant stop the raid you must continue ah:) i'll leave you alone now so you can concentrate. good luck! jlgaddis: thanks! :) up_the_irons: I'm fully willing to participate in distracting you if you need it. man, of *course* this host couldn't have stayed up a couple more weeks for me to reach 1000 days of uptime. grr!!! :) up_the_irons: That sucks. srsly up_the_irons: You could just randomly reset them, then you wouldn't have to worry about uptime anymore. arenlor: lol up_the_irons: "uptime of 10 days, that's odd, why isn't it reseting itself?" man, i was afraid this would happen when i came here today. Both drives in a mirror had reallocated sectors, but one had > 900 and the other (newer one) only 27. Which one did I replace? The > 900 one of course, but now it looks like the one with only 27 rasect is barfing :( up_the_irons: if a disk shows any errors it's bad :) hmm cos it'a already used up it's ecc etc up_the_irons: in that case, data loss? jlgaddis: yup, data loss ouch time for raid 6? mercutio: I still think RAID 2 is better, but no one implements it. there's also raidz-2 mercutio: no, reallocated sectors are normal, all disks have a reserve of sectors that get remapped by the firmware. however, if you start getting LOTS of them, then that is probably a sign the disk is gonna go jlgaddis: actually, data loss is not necessarily the only option. it is possible the bad parts of the disk are unused. I had 2.18TB free in this volume, so if the bad parts are there, then it is possible the "good" stuff can still be copied off. that's what i'm investigating now.. ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 105 099 006 Pre-fail Always - 8276127 this is bad right? err actualyl i don't know if that's accurate not sure... well both disks are showing hugely high numbers they're not raided the newer disk has a count of lba read/written which seems insanely high too "Once again, with most SMART registers the RAW Values are only meaningful to the drive maker. They are of no interest or meaning to users." oh