G: fixed console.cust, your new vps appears god i hate budget speeches :| up_the_irons; how many engi's do you guys have? and whats your QEMU stuff running on ? eyecue: engi's? sorry, ops staff :) "engineers" sysadmins, noc guys, rah rah eyecue: you're lookin at him :) ahh :D whats your infrastructure footprint like atm ? eyecue: jpalmer helps me with stuff sometimes and wickedSA does data center work for me from time to time sorry for all the questions, just interested :) yeh, jp rocks, long time #freebsd guy here eyecue: that's a broad question, i will opt not to answer ;) hehe :) was kinda getting at the 'how much crap do you alone have to take care of' :) 20 servers, all the networking gear nice power, cage, etc... *nods* im taking care of a 25 server alloc atm really want to get into playing with vm in production up_the_irons; can i ask what DC? we've got 15 in servepath eyecue: coresite in LA cool :D up_the_irons: can't reach the arp networks website :/ portertech: give it time, i'm adjusting some routes up_the_irons: ack portertech: paste a traceroute, it should work by now i can get to it fine from here there it goes yeah it shouldn't have bounced for more than a minute i put in an aggregate route yesterday, more than enough time to propagate up_the_irons: working and i took the more specifics out just now (japan) portertech: cool redundant network and upstream links ? eyecue: you ask as if there is an alternative? ;) alternative/ideal/bestpractice and reality are often two different things :d up_the_irons: is the dashboard a sinatra app? portertech: no, the main website is sinatra, the portal is Rails eyecue: indeed, true up_the_irons: neat yeah :D up_the_irons: yay, I'm an arp networks customer once again! :P just sent my order portertech: nice; how is the latency from japan? currently surf'n on 4G, not really a good indication, I'll know more once I have an arp vps provisioned ah ok up_the_irons; is there an alternative to 95th percentile billing and/or volume bandwidth across multiple servers? eyecue: as in? those two are all i've ever heard of up_the_irons; two questions, can you do average usage billing, and second, can i purchase a block of bandwidth as a seperate line item to utilise across multiple servers, instead of a XXXgb block per server up_the_irons; rationale behind the question, we currently have 15 servers, 5 of which run high network utilisation workloads for 6 hours a day, and 0 for the rest of it, and the other 10 run more evently distributed network loads eyecue: bandwidth is already a separate line item (1 vps can use all of it, and the others can use none, etc...). i don't do average usage (other than 95th) eyecue: which 6 hours of the day? :) 95th percentile works out to almost twice as expensive for us for the servers that burst for that short period a day, and the bandwidth that comes along with the other 10, goes under utilised over the month up_the_irons; umm, nighttime at gmt+10 how high is the utilization? *looks at zabbix brb im going to drop :D vpn fail 15-20Mbps per server x 4 in/out sometimes 30-50% up on that up_the_irons: thakns, I'll check it out soon G: np eyecue: i c :] eyecue: i'm low on space for colo, but i think someone that would be able to accommodate that is Syminet: http://syminet.com. The owner is Mark Symonds and he's a real cool guy. He buys bandwidth from me too, so it is same quality that I offer. up_the_irons; even vps's ? or straight physical colo only ? eyecue: oh, we're talking vps' here? thought you meant physical servers hopefully recomming some physicals as vps's no point having a quad core around for 6 hours of usage not very IO heavy workloads, so its primarily a cpu/ram thing eyecue: i can handle the vps needs; but why do average billing for that, i could give you a bulk discount on total data transer (GB) usage eyecue: yeah true up_the_irons; sure, i was wondering about flexibility over the billing model more so than the actual solution :) i know how hard it is to change in contract :D eyecue: generally, i'm flexible, if it makes sense. but i'd rather not create a whole new billing model if i don't have to ;) absolutely, everyone needs to make moneys in our .au deployment, we pay per month for the physical hardware, and buy a few tb of bandwidth to utilise coming into our distribution switches up_the_irons; ahh heres a good one, do you charge 95th percentile on the highest of up/down utilisation, or over the sum ? eyecue: cacti says: HRULE: |95:bits:0:max:2| :] i keep forgetting what that means ;) hehe i think it is the higher of the two roger that eyecue: where are you located? sydney, au deployments in bne, au, syd, au, and sfo, us nice ahhh, good ol' BNE :) :] I love that city, walking alongside the river etc up_the_irons; currently got a large mail infrastructure in US, HA-WWW arch in sydney, and dns services in BNE want to get back there sometime G: not last year it wasnt (sure you saw the flood coverage) one of the biggest DC's in the country got literally shutdown eyecue: cool eyecue: yeah, that flood... wow up_the_irons; exclusively freebsd eyecue: sweet eyecue: which DC was that? G: AAPT iirc ahhhh are they still owned by Telecom NZ? or did Telecom finally offload it to TPG? http://goo.gl/UNDIP no idea who knows who owns who these days check out those google results though ;) true I guess :) eyecue: the bit of the CBD I used to work in was pretty much unaffected iirc *nods* i had a server in the Pipe DC on creek st it was up on a 45 degree street Creek Street, which one is that again? Pipe Networks eyecue: no, I meant which street :) I got the Pipe networks bit :) which street? creek street :D ohhhh next one over from Edward St yeh, google maps was playing up there o_O eyecue: Google Maps was giving me nowhere near the CBD yeh same Redcliff for instances i even tried to redo the query i got 4034 where are you at at the moment ? yep, I also got a bunch in the 4030's eyecue: New Zealand ah eyecue: I moved back last year hopefully im coming over to Mt Hutt soon for skiing :D never seen snow in my life, very excited I used to work on the otherside of the CBD... North Quay/Herschel Street way ive decided i dont like 95th percentile bandwidth models :| eyecue: do you get up to BNE much? useto live there moved back to syd a few years back lived in stones corner (near the gabba) have you seen the Kurilpa Bridge? worked at park road milton ahhh yep, know Park Road hehe, tank street bridge :D iirc thats where APNIC were based yah yup, they were downstairs from us @ APN :] 33 park road *win* eyecue: I generally refer to it as the Ugly Bridge :) i asked one of the apnic guys in the elevator one day for a /8 :] eyecue: iirc that's also the building QTAC is/were in right? he laughed yup eyecue: haha, thats a great one :) Red Hat used to also be down Park Road mm, news to me how long ago ? eyecue: ummm they shifted from there in 09 iirc ah k no wait, 08 memories :] I think it was Mid 08 ithat was right about the time i was there, ish. I interviewed at their new place in August/September-ish 08, and started working for them in Nov 08 one of the guys i worked with @ APn took this photo of my dd and his best mate in front of la dolce vita :D i can tell you exactly when we were there 18 august 08 :] waiting for linky ahh here we go http://keithrowell.com/365/2008-08-18.jpg oh thats right, it wasn't directly on Park Rd, it was off a sidestreet http://keithrowell.com/365/2008-09-17.jpg <-- this ones inside la dolce vita eyecue: nice :] But yeah, that Tank Street bridge (imo) is Ugly :) big time eyecue: what made it worse, was I had to see it litterally every single day :| where were you working? I'd say the only day I didn't see it, was the week I was couped up in bed sick as a dog :) eyecue: North Quay/Herschel Street kk and I lived on Tank Street wow I had a very fast commute :) 20 a month for a fbsd vps on arp, nice except I used to walk the long way around the block to get decent coffee :P :) i drove in after 9:30 to avoid traffic eyecue: I didn't have that luxury eyecue: how did you hear about arp, btw? up_the_irons; mentioned it earlier, @obfuscurity up_the_irons: btw, liked your tweet about Slicehost, didn't realise they were migrating everyone to RS Cloud uh, jason dixon eyecue: oh ok G: yeah i just heard about it a few days ago up_the_irons; https://twitter.com/#!/obfuscurity/status/67602845507125248 up_the_irons; i thought, if they do obsd, theyll do fbsd eyecue: yep :) freebsd is the most popular OS around here yay:) oo, 30loops using freebsd for their firewalls, cool. portertech: so did you move to slicehost and then come back here? ;) I have vps's all over the map (except rackcloud/slicehost atm :P) nice I've been meaning to drop a few and move back to arp anyone know if you can use CARP on a single /30 subnet? I know which HSRP and VRRP, you have to jump through hoops :) why do you think you couldnt ? 2 usable ? up_the_irons: for what it's worth, before I asked you about the 2nd VPS, I signed up to Rack Cloud and took a look (really, I only need bursts of a second VPS every now and then, so I thought, all save heaps only pay a few dollars), I was mistaken :) I could have done it, but it would have been ugly G: hah easier to just do it as a second VPS on ARP nice :) up_the_irons; each physical needs its own ip, plus the virtual ip up_the_irons; thus at least a /29 eyecue: it's the virtual IP that concerns me. so, you have device A and device B (2 IPs), a broadcast address (1 IP), and network number (1 IP). no room for virtual IP correct you could proxy arp's, so they could speak but meh. up_the_irons: they count powered down, time for costs, and the only way to keep the data w/o hourly charges is to move it to their files thing, and then delete the server (and get a new IP) etc so then you can't do CARP on a /30 ;) so meh correct. ive read at some point that future versions may support it G: that's dumb but no specifics i c but carp does rock, majorly. running lagg+carp for our HA inf in sydney +haproxy+nginx (ssl) HSRP on the upstreams and for the private lan ahhh I see CARP is kinda HA... G: its a patent free replacement for VRRP fail cisco. ah ha, yep, I recognise it now :) now about that beta freebsd vps ;D eyecue: Cisco seems to be the perfect example of "Do what I say, but not what I do" :) mm, indeed. the one thing I really like about Cisco though, is IOS really? :) eyecue: yeah, when they consistently apply the IOS style through devices, it's a godsend imo fair call i really wouldnt mind a small l3 cisco switch to play with eyecue: I had to jump in the deep end w/ two of their Fibre Switches the other year, hadn't touched Cisco stuff for ages, but the basics stuck with me, and it all worked easy as pie i did that with my first 2621XM with one of my old clients (keeping in mind that I'd only dealt w/ Cisco switches and routers in the past as well) Ethernet that is howd you find it ? eyecue: easy as pie eyecue: sure I had to look at the documentation to understand the specifics, but as they keep the basic syntax and concepts/style the same, it was pretty easy of course, but looking at docs isnt a bad thing in the same vein as being able to regurg doco isnt necessarily a good thing :) eyecue: sounds about right up_the_irons: off the top of my head, i think you'll need a 4000/4500/5500/6500 to get that many vlans unless you go non-cisco, of course oh dear, MS gets skype. eyecue: as I just said in another channel, hope Facebook is their next target :| that aint gonna happen :) sooner trust Microsoft w/ my personal data than Zuckerberg eyecue: they already own a small portion :) dont have a FB account :) so i dont have that problem yeah, I've been meaning to delete mine for ages time for another glass of red bbs thinking maybe i should use nginx as a frontend instead of as complete apache replacement, always such a pain to set sites up with things all ready to roll on apache not that anyone here will be testing ariane's vmmap_sys diff at ARP but me, but I recommend you not do so, a system behaving as if it is 1mhz is not my cup of tea. hello everyone who do i have to talk to for getting extra ips ? alainb: that would be up_the_irons if he's around, but generally those type of requests get shuffled to support@ anyway. prepare for $1/additional v4 IP. ok well i just sent a email thanks anyway jlgaddis: yeah i was aware that 4500/6500 do it, but i'm looking for like a 1U solution that isn't $7K+ (4900) ;) up_the_irons: you can use CARP on a single IP w/out an IP in the system itself, just expect the BACKUP system can't access the network .. but that should be qed toddf: i'd put a management IP on the backup system on a separate interface, so accessing it wouldn't be a problem or just do v6 for accessing, and v4 for legacy apps.. *grin* or if your apps speak v6, don't worry about that /30 ;-) (not quite clear what is being done, though, so some advice may be irrelevent) toddf: i want to set up some kind of HA (hrsp/vrrp/carp/etc) for customer gateways up_the_irons: ah, then you want something like this: np for h in /etc/hostname.vlan*; do id=${h##*vlan}; ( echo "vhid $id pass $(pwgen -s 64) carpdev vlan$id"; egrep "^description|^inet " $h ) > /etc/hostname.carp$id; grep -v "^inet " $h > $h.tmp && mv $h.tmp $h; done in english, move your 'inet ' stuff from /etc/hostname.vlanX to /etc/hostname.carpX specifying carpdev in /etc/hostname.carpX in reference to vlanX sweet note that if you want to do carp with IPv6 I'm currently trying to convince someone to do something about the fact that 'rtadvd carpX' will work, but 'ping6 -c 1 fe80::X%em0' is neccessary upon client's initial bootup or somehow packets don't get to the right place on the router. wish i could use it, however, my primary side router is a cisco 4500, so only hsrp and vrrp are my options (probably qed but 'rtadvd' takes a list of interfaces, don't run one instance per interface .. *grin*) i c which also makes me curious, tell me if this is asking too much info, but do you have anything non default on your openbsd v6 router? typical v6 routers respond to a ping6 of ff02::2%em0 but yours doesn't.. I will try to reproduce with qemu -serial tcp:... but if anybody can point out what is required to send a break sequence over conserver to an ARP vps.. I'd be much obliged ;-) the ^Ecl0 seems not to be working for me.. toddf: it's pretty default I'm sure there's a reason. even if it has to do with loonix filtering something on the vlans (which I doubt, but you never know..) toddf: nah, there should be no filtering at all on the vlans pf.conf icmp filtering on the routers? nope its just .. odd. because any other system I've seen with net.inet6.ip6.forwarding=1 responds to ff02::2, but it clearly is forwarding packets... ;-) random curious itch I'm scratching. no big deal, just .. odd. ;-) yeah odd, seems I have a local switch blocking something or something .. my laptop doesn't see traffic sent to ff02::2%fxp0 from a system on the same ethernet segment. hehe up_the_irons: as much as i shit on hp, i quite like their stuff for layer 2 jlgaddis: o'rly? then maybe i'll check it out, cuz all i need is layer 2 yeah, the layer 2 stuff is solid. just don't try to route anything. =) jlgaddis: ok :) i'm not sure what list is on the e4210-48g, but i think that'd do what you need and should be ~$1800 or less list awesome i also got a suggestion for extreme networks x350 and an juniper EX4200-48P so i got some homework to do bet the juniper will burn my fucking bank down easily 2x the hp bankburner! the ex4200 is nicer though they're stackable, too, if that matters virtual chassis shit jlgaddis: stackable may be nice when i exceed 48 vm hosts i'm at 22 now, so it's not unlikely that'll happen within the next year or two out of curiousity, how many physical... ahh ok =) :) gonna need another cage nah, i can go to at least 48 in the same space i have an empty cab that can do 15 servers and then i might have a colo customer giving up a cab within the next year they got sucked into "the cloud" oh, one of those srsly jlgaddis: what is the cli like on the hp's? is it "cisco-like" or... ? "VLAN support and tagging: support IEEE 802.1Q, with 4094 simultaneous VLAN IDs " yeah, so the HP can do what I want, cool can't wait to replace this Dell PowerConnect 5324 to be fair, it did its job for two years and was < $300 up_the_irons: not exactly like ios, but not majorly different. biggest thing you'll notice is how vlan's are configured. jlgaddis: ok i wonder if there is a rancid profile for HP instead of 'conf t ; int ; switchport vlan access ' its 'vlan ; untagged ' or 'vlan ; tagged ' instead of "switchport trunk ..." basically yes, there is. i use rancid with all my procurves. s/cisco/hp/ in the router.db file jlgaddis: oh sweet, cool jlgaddis: (re rancid profile) jlgaddis: that vlan config is like force10 i don't have any force10, but a buddy of mine let me work on one once yeah the old foundry's were the same way i c if you do end up getting an hp, ping me if you need anything. i, unfortunately, have way too much experience with them. jlgaddis: sure will do jlgaddis: why are some blue-ish and some grey? i take it the grey ones are the older model? grey/white that is procurve? the e4210's were 3com's originally oh, which means the cli is different and they probably don't have a lifetime warranty glad i just thought of that. not sure if that makes a difference to you or not. jlgaddis: it actually does say lifetime warranty why can't i find an f'in picture of one of these that is bigger than a thumbnail?! up_the_irons: which model(s)? pilgrimd: e4210-48g up_the_irons: Is this large enough: http://h18000.www1.hp.com/products/quickspecs/PH_001737/PH_001737.JPG pilgrimd: omg, yes, thank you! that's not it i have some of those in my office, i think they're 2510s yeah looks like the 2510 those guys are pretty weak it seems, only 64 vlans thinking about it, i don't think any of the procurves (made by hp) can do 4094 vlan's. well, not 1u fixed-port layer 2 only ones. just the stuff they got from 3com those 4210s are probably the lowest end model that can roger that i wonder how good the ex-3com stuff is haven't used any of it personally, but i've heard good things if it were me and i could afford it, i would go with an ex4200 though cisco > juniper > hp, imo =) yeah :) (vendor j)++ jesus the ex4200 is expensive (why am i surprised) like $7K talk to a sales guy and tell them it's between the ex4200 and a c4500 but you're "just not quite sure yet" and ask if he can do anything juniper salespeople right now have a lot of leeway in throwing discounts out (especially if you're a new customer) oh i c found one at buy.com for $5K $7k for a 48-port gig switch? sheesh. 3750X's are less than that. oh the ex4200 is L3 don't need L3 3750-x can't do 4094 vlans pilgrimd: don't think the 3750's can do 4K active vlans though you don't have to use l3 features, of course (but you do have to pay for them!) yeah but you have to pay for it :) 2200 *might* work, one sec ahhh, nope. only 1k vlans up_the_irons: you have 4k vlans in your environment? with 48 physical servers on a switch, it could easily be hit since each customer has a vlan jpalmer: i'll have more than 1k vlans in about a year, so i don't want to reach that limit too soon ex3200 can do it, but it's layer 3 also is it cheaper? :) gotcha. that's what I was asking. are you *at* 4k, or is that expansion sized. ;) $2870 says google shopping better jpalmer: yeah not at 4k yet, not even close ;) ex3200 48p, there ya go looking to replace the 4500 or whatever it is you're running now? jlgaddis: no, the 4500 is great and is here to stay, but i am building a redudant diverse path network for each VM host, that won't touch the 4500 (this is for redundancy obviously) s/redundancy/ha/ redundancy is so.. redundant. lol heading home, bbl moin! the demo at the bottom is pretty cool: http://www.juniper.net/us/en/products-services/switching/ex-series/ex4200/ fail server failure. i was out at juniper and some young guy was talking about mpls and this guy with us corrected him. the juniper guy started to argue with him until we told him that the guy was ivan pepelnjak. http://images.amazon.com/images/P/1587050021.01._SCLZZZZZZZ_.jpg =) LOL the juniper EX3200 and EX4200 are not too different in price they don't seem to be very different either, apart from virtual chassis and dual power supplies actually, let me say, the *ebay* pricing isn't too much different (even for new in box ) can't believe newegg sells it: http://www.newegg.com/Product/Product.aspx?Item=N82E16833415005&nm_mc=OTC-Froogle&cm_mmc=OTC-Froogle-_-Network+-+Switches-_-Juniper+Networks++Inc.-_-33415005 no f'in way would i pay that well holy fuck, cdw is actually *cheaper* for once! yeah like half comparison shopping on the internet is exhausting any admins around? niccos: what's wrong? up_the_irons: google shopping? fink: indeed fink: and going to manufacturer websites trying to find data sheets to compare features / req's I think I fixed it up_the_irons: newegg is good for brand new stuff but they might not carry the kind of things you need yeah yep, fixed it :p ok, so I thought I did, but maybe I didnt I bound my set of ip's to my interfaces, and I can ping them, but irssi wont bind to them I get connection timed out niccos: OS? ubuntu server 10.04 any ideas?