anyone tried upgrading to OpenBSD 4.8? Sunil: 4.8 and current works fine, just recall to 'disable mpbios' i was wondering, if i can nslookup -type=aaaa six.nullbnc.com from my localbox but ping6 six.nullbnc.com on my local box says "network is unreachable", however, it ping6 fine on my server and i can connect to irc with it... something must be configured incorrectly :( i'm running debian, any hints or help would be nice :D hey guys need some help with my freebsd how can I resize the root partition? ahh looks like my localhost doesn't support ipv6 at home, no wonder i got network is unreachable :> ipv6 is nice i hear it is the future of the internet!! ye can you see if you can "ping6 six.nullbnc.com" ? nvm up_the_irons: How do you share your libvirt config files? Storage, network, VM config, etc across your host servers? I was considering using git as we do with Xen but that seems to be frowned upon. mhoran: iscsi? Sure, we use iSCSI for block devices, but /etc/libvirt is what I want to share. And sharing that directory (or at least /etc/libvirt/qemu) is frowned upon -- https://www.redhat.com/archives/libvir-list/2009-October/msg00033.html Really, all I need shared is /etc/libvirt/storage, since that's where my storage pools live (which are iSCSI). But to migrate a domain that lives on an iSCSI block devices, the pool must first exist and be started on the remote end, which isn't always the case. So really, libvirt should just have better support for migrating iSCSI pools. And should start them up automatically when a domain is migrated (though I see why it doesn't do that). So in the meantime I have to find a way to share my storage pools, and to do that I've checked everything in /etc/libvirt save qemu dir into git. mhoran: i was being facetious... ... i've not actually looked into the details of qemu much So I think I'm just going to write a wrapper around iscsiadm that automatically creates the poolsfor the available targets ... that seems the best way to do this ... and then let libvirt migrate the domain configs when it wants to. Unless up_the_irons has a better option for me. :) mhoran: up_the_irons doesn't do auto distribution, from what I can see it is a manual migration process and he doesn't do iscsi, he has found physical disks on the servers to be more reliable toddf: do you know what OS is used to host the virtual machines? Ubuntu. nesta: Linux is all I need to know. ;-) hehe I've found iSCSI to be just as reliable as physical disks, it's just more expensive. Though our iSCSI vendor blows. It's nice for e.g. live migration. nesta: a good ole BSD doesn't do kvm just yet Replication and failover as well. toddf: I imagine not :) mhoran: check the old logs about iscsi I know, just stating my opinion. aka regarding up_the_irons experiences with it Live migration (or even migrating betwen hosts -- what about failed disks) just seems so impossible without iSCSI. Granted, we don't currently have live migration even with Xen, but we do have iSCSI backed storage and can easily move VMs between host servers with no data loss. Our iSCSI infrastructure is also fully redundant, we've got N+1 replication and then RAID-5 on each cluster node. We've lost entire iSCSI systems and remained fully available. Not to mention losing a single disk and being fine. as soon as live migration is a requirement, iSCSI makes sense when the environment is large enough such setups make sense Yup. Well, it is, here. We just can't use it. We just need to be able to migrate between hosts, at a minimum. And that seems too difficult with local storage. at arp, there is a bit of common sense, simplicity, and economy in the mix, iscsi tends to take the simplicity and economy out of the equation IMHO though I don't disagree with the awesomeness of your setup I wish there was a free software iscsi target that did anything close to what you describe I wish our product did everything it was supposed to do. :p as it is the one free iscsi target I can run on OpenBSD (netbsd-iscsi-target ironically) can't even reload the targets config file w/out restarting, severing all iscsi connections in the process .. whee! Damn. Yeah, considering what it does, it's pretty solid. But sometimes it's really flakey, and it requires a clunky Java GUI to manage it. Though now it's got some SSH console, but the CLI is undocumented. are you able to divulge your 'product' ? ;-) I've got an equalogix array at a customers colo, seems like a very fancy iscsi target... The clustering is awesome, and it supports LACP at the link layer automatically so the NICs are redundant. HP/LeftHand SANs. that sounds a lot like equalogix wannabe Indeed. They were around for a while as LeftHand and then HP bought them. Apparently they've rose to huge success, at least that's their excuse for their shitty tech support. The whole core was some guys PhD thesis or something. ;-) Wish it were open source, it would kick ass. yeah The real cool thing is remote replication -- we can replicate our five san cluster to our second datacenter all in a half hour over a dedicated backbone link. thats what equalogix touts also do you do more granularity than 15mb data chunks? I believe so, though I've not tuned any of that. it seems rather rediculous if one byte is modified every 15mb .. it transfers the entire 15mb chunk Yeah, looks like LH is 256k. much more efficient I'm wondering if the new version of the software fixes some of thes issues we've seen with the version we're running, but I don't think I'm going to get the go-ahead to upgrade ... we've just had major headaches with upgrades in the past, and, if it ain't broke it, don't fix it. But I know the second I log a support call, they're going to tell me to upgrade. heh Oh, of course listStoragePools() and listDefinedStoragePools() are mutually exclusive, why not! mhoran: the config is not shared up_the_irons: can you double check some small configs for me? this is ipv6 part of /etc/network/interfaces -> http://pastebin.com/BKN6usGX kitkatbar: shouldn't the netmask for link local be /48 i'm not sure, i had hell of a time getting it to the point where it is now :) :) kitkatbar: the mask should be /48 if you are routing a /48 block of ipv6 over link local also, do you have packet fowarding enabled yes any crazy firewall rules /etc/sysctl.conf i added net.ipv6.conf.all.forwarding = 1 default firewall ok i even added that protocol 41 i think to iptables kitkatbar: has he fowarded the block to you yet? yes i already have reverse delegation to my nameservers can you print out "ifconfig" and "route" sure one sec and pastebin them http://pastebin.com/gcFQAvXA inet6 addr: fe80::2/64 Scope:Link is incorrect should be fe80::2/48 your routes look correct bout to restart networking or might reboot to see if the netmask 48 will change things vcs: fe80::2/48 is incorrect. link local addresses are /64's :X kitkatbar: ^ fe80::2/64 is correct ::/0 2607:f2f8:34c0::1 UG 1 0 0 eth1 it looks like you have an extra default route for ipv6 that was probobly leftover from your /64 err wait im not used to non BSD route, lol 16 bytes from 2607:f2f8:34c0::3, icmp_seq=0 hlim=63 time=1.264 ms that looks good :) kitkatbar: you had an extra default ipv6 route that was the problem ;) ::/0 2607:f2f8:34c0::1 UG 1 0 0 eth1 was still loaded from your /64 so rebooting cleared it out, and now you are on ipv6 ;) congrats man sweet hehe can you check to see if you can reverse this ip 2607:f2f8:34c0::3 maybe it just hasn't propagated yet, i sure to hell hope i got bind9 configured properly Host 3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa not found: 2(SERVFAIL) hmm kitkatbar: are you running BIND? yea i get this from my server though nslookup -type=AAAA six.nullbnc.com six.nullbnc.com has AAAA address 2607:f2f8:34c0::3 don't use nslookup for debugging or troubleshooting. use dig. nslookup will make several assumptions about the information you are looking for, and will occasionally work even when things aren't configured properly. if you are troubleshooting, this is not a good thing. thx :D i'm not a network guru yet this one had me baffled dig says no error when i dig that domain, so i guess it's a propagation issue kitkatbar: I'm pretty decent with bind. let me scroll up a sec and read from the top. i was having ipv6 issues with my interfaces config with the netmask i think changing it and rebooting solved that problem kitkatbar: it looks like ns1 and ns2.nullbnc.com are delegated the ip6.arpa for that address space. are those your NS's? dig @your.dns.server.com 2607:f2f8:34c0::3 PTR try that yes http://pastebin.com/ZmuJwBtM kitkatbar: I'm getting a SERVFAIL from your NS for that record. can you paste your bind config and zone config? yes, one sec http://pastebin.com/VazMSimU kitkatbar: I'm not seeing anything obviously wrong with the configs/zone. do you see any errors in bind when loading the configs? no i added that ipv6 address to ns1.nullbnc.com thinking it might help not sure if thats correct, probally should work without it ahh, your SOA, and origin don't match in the ip6.arpa zonefile you have a zone definition of: 0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa, you have an SOA of: 0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa but you have an ORIGIN of: 0.0.0.0.0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa ahhh so should which one should i change lol sorry should i add 4 0's to the SOA or remove 4 0's from the origin? must of been the ipv6 reverse dns zone builders fault, i should've known better than to use it haha try removing the extra nibbles from the ORIGIN, and add them to the PTR record itself. says status: noerror on dig now instead of nxdomain but still says answer: 0 hmm i have no idea i've tried every combination possible with the 0 thingy keek-a-boo anyone use gitosis with gitweb? i'm now getting a response with 'dig @ns1.nullbnc.com six.nullbnc.com AAAA' but when i try 'host 2607:f2f8:34c0::3' it says servfail Anyone experiencing connectivity issues? nope not yet :X I can't even ping arpnetworks.com no pings [sundial@puffy ~]$ ping6 arpnetworks.com 16 bytes from 2607:f2f8:0:102::3, icmp_seq=0 hlim=63 time=1.054 ms ipv6 is up What about ipv4? working for me Some of us still use that, ya know. :P i have stuff hosted on ipv4 and its all working ok arp networks site is up for me maybe its a routing problem between you and them 11 ae-1-69.edge1.SanJose1.Level3.net (4.68.18.14) 75.912 ms ae-3-89.edge1.SanJose1.Level3.net (4.68.18.142) 76.104 ms ae-4-99.edge1.SanJose1.Level3.net (4.68.18.206) 76.735 ms 12 * * * 13 * * * dies at level3 Yeah. Having trouble here as well. Just a few minutes ago. man I've got 2 vps boxes at softlayer two times today no response argh The level3 to MZIMA link may be down. Yeah. I can get in over v4 from my house, but not from work. Home goes over Comcast to Mzima, work goes over L3. My monitoring service is also complaining. I'm on roadrunner and it dies at the Level3 MZIMA connection. What does your route look like vcs? The link from Trit Networks may still be up. vapor: Soflayer is going downhill? They used to have a very good reputation. alright, one sec ill get you a traceroute on http://pastebin.ca/1986763 my traffic is being routed over mzima it appears I love softlayer just don't like random weird issues I run a small datacenterand we never run into these kinds of problems :) level3 routing issues...those happen a lot level3 sucks Thanks vcs. mzima is good transit considering they default to level3 mostly im supprised i dont go through level3 since im a few miles from them lol guess thats a good thing level3 has awesome latency but they do run into routing issues frequently their network is AGED like a fine wine i have some servers collocated at their datacenter here no ipv6 :X guess thats aged equip for ya vcs, yeah they still dont run v6 you have to run tunnels I remember when level3 was considered modern jazz57: like 10 years ago? it's so cool to have 1.1 ms ping times to my VPS from my desk :) Here is my trace: http://pastebin.ca/1986770 RandalSchwartz: I've got a couple of hundred I usually get routing through Los Angeles. Now it's going up to SJC and getting lost. vcs: That sounds about right. :-) hurricane electric is cool :) i like them :) I remember someone recommending Level3 over ATT because the had a new network built from scratch. a couple of hundred milliseconds? I like hurricane too. ipv6 and they have a cool NTP servers too. still seeing lvlt issues it actually gets to mzima now so if its bgp then should only take another 60 seconds I'm still getting destination unreachable Ooh, my nagios just went nuts did we have some downtime recently? is anyone else experiencing network lag? kitkatbar: I can't get connect at all. me either i've been working all day on my server Me too. :-) I just through. I can't figure out why nagios can't even ping its own host I've got "host down" on localhost. :( And yes, something went down before, RandalSchwartz randalschwartz: fd depletion or other dos? firewalling localhost? no. nothing unusual. maybe the whole VM was frozen or at least the network stack outside my view bad stuff from 1405 to 1426 across multiple VPS nagios acts funny if resources take too long to allocate .. wheterh thats sockets or disk blocks etc since it forks a process to run the check and considers things bad if it doesn't check in so to speak within a certain timeframe well - that's pointing at VM issues, not guest OS issues Looked like a network issue to me. so it's Garrey's problem not mine any of you guys can help me figure out why my rdns isn't working for my ipv6 but it's his *because* I'm gonna get yelled at because of it randalschwartz: think thundering herd of disk io .. slowing nagios process creation if the sectors happen to not be cached in memory .. sure. Still sounds like a virtual-hardware fault though I'd like those to go away given my experience with nagaios, I'd expand the timeframe during which individual checks timeout and/or generate warnings by 20-40% ... just because nagios couldn't meet its own criteria of happy days doesn't mean external usage was effected as much as nagios might have you think .. *shrug* .. I've written a replacement to nagios and uninstalled it from my systems due to its inability to stop producing false positives on real hw, leta lone ... ... virtual hw Yes - I wait 10 checks before I send a hard notice but I still shouldn't be getting soft notices like this both mabel and red went nuts at the same time so I'm gonna blame somethign higher mind you, 99.96% availability isn't bad but another 9 wouldn't hurt especially if it's a fault that Garrey can fix and it sounds like all of us experienced something odd during that time that smells of VM oops. It's Garry, and Dolley I have a hard time remembering that. Blame jet lag. ) Here is my trace now:http://pastebin.ca/1986789 The route has changed back to Los Angeles. So somehow a partial network outage causes weirdness with the VMs. ok, so who here is setting their vm to go 'fritz' when upstream network outages occur? ;-) anyone run debian & bind w/ ipv6? maybe a bunch of console notifications (high uninterruptible priority) kitkatbar, dig -x ip +trace is your friend I did get some "runtime went backwards" notices the other day in my logs. Interrupts may be getting dropped when activity is too high. Someone suggested dropping kern.HZ to 10. (assuming freeBSD) I think it's that already maybe it was the mysterious missle launch earlier off the coast of la the caused it Oh. I have hz = 100 did the missle launch affect the cruise ship that lost power? :) maybe the missle took out our route :X bastards! that's what I get for putting that info record in! $ host -t loc stonehenge.com stonehenge.com location 34 2 53.000 N 118 15 21.000 W 50.00m 1m 10000m 10m vcs, can you post your reverse zone for ipv6 please? wait kitkatbar: sure i already have the link i used same generator you did but my reverses dont work :/ http://pastebin.com/r5mb6mwn did you use your own ip /48 range and not mine yes perhaps your named.conf file is bad? http://pastebin.com/J3VcyXaf or you did not run rndc reload yes i have lol ive tryed everything i know 201011090 ; Serial number (YYYYMMdd) errr you have only changed to today? i've been at this all day every time you make a change it requires an incriment then rndc reload Uh - that's not 00 else it will not realod 20101109xx even for 'host ipv6:address' to work? oh i see to check to see if the config has changed lemme test this I'm old school... /etc/rc.d/named reload :) well i have mostly been restarting it with /etc/init.d/bind9 restart reload is enough ah well i still have a problem then :/ and you should alwaysalwaysalways increment the serial yes i just did and tested it same result but atleast i know in the future when i make a modification, to increment the serial are there any error messages when bind is reloading? what does your local soa report? host -t soa $domain $localhost you've checked your /var/log/* logfiles, right? no errors when bind is reloading host -t soa six.nullbnc.com nullbnc.com Using domain server: Name: nullbnc.com Address: 174.136.101.162#53 Aliases: six.nullbnc.com has no SOA record no no - the rdns since that's what you're having trouble with or am I mixing up conversations here no your bind should report an SOA with the same serial as what you defined host -t soa 0.c.5.a.8.f.2.f.7.0.6.2.ip6.arpa. localhost something like that Wait a second that SOA is wrong @ IN SOA your.forward.host.name.here.com your.email.host.com. ( that's what you want not the reverse you're defining the authoritative *master* host for *this* domain for example, I have @ IN SOA red.stonehenge.com. hostmaster.stonehenge.com. ( so your saying '@ IN SOA 0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa. ns1.nullbnc.com. (' is wrong in my reverse-ip6.arpa file? Yes. Very Wrong. guess thats what i get for using damn generator for this reverse file @ IN SOA master.host.com youremail.youremailhost.com hahaha. wrong indeed. no - you just put the wrong data in it asked you things, you told it wrong :) GIGO ah lol lemme see if this will fix it and what does your delegation in your master file look like? in my nullbnc.com.db file it looks like @ IN SOA ns1.nullbnc.com. ns2.nullbnc.com. no... in your named.conf ... zone "0.8.0.3.8.f.2.f.7.0.6.2.ip6.arpa" { something like that? yea ok - so fix your soa, reload and then you should be able to check your soa locally once that's working, you can worry about the right upstream delegations host -t soa -i six.nullbnc.com six.nullbnc.com has no SOA record wrong question indeed host -t soa -i nullbnc.com localhost but I'm asking about the reverse-6 records he should be able to see his ip6.arpa soa host -t soa -i nullbnc.com localhost Using domain server: Name: localhost Address: 127.0.0.1#53 Aliases: nullbnc.com has SOA record ns1.nullbnc.com. ns2.nullbnc.com. 201011091 86400 1800 172800 259200 kitkatbar - that's still broken unless your contact email is ns2@nullbnc.com although, knowing you, it might be :) @ IN SOA MASTERHOSTGOESHERE YOURCONTACTEMAILGOESHERE ( there... clearer now? so now nullbnc.com has SOA record ns1.nullbnc.com. admin.nullbnc.com. 201011092 86400 1800 172800 259200 ok - that's good for the forward now check the reverse ... host -t soa 0.c.5.a.8.f.2.f.7.0.6.2.ip6.arpa. localhost or whatever your numbers are basically, you should get a SOA record for everything you list in named.conf as a master host -t soa 0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa. localhost Using domain server: Name: localhost Address: ::1#53 Aliases: Host 0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa not found: 2(SERVFAIL) well - there's your problem then are you *sure* your named.conf is correct? pretty sure kitkatbar: sorry I had to run earlier. work called. did you get ip6.arpa resolution working? jpalmer - not yet I smell something wrong with named.conf at this point but can't be certain about anything now :p please pastebin your "zone" entry for that reverse zone "0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa" { ... until the end of that block does it have "type master" and "file '...'" correct at least? http://pastebin.com/Hv2UBRpG yes, the reason i have ip commented under slave is cos i tryed it both ways earlier and the file is in the right place? called reverse-2607 blah blah? yea, /etc/bind/zones/master/reverse-2607blahblah.ip6.arpa reverse-2607-f2f8-34c0_48.IP6.ARPA to be exact :p wait - I have "master/" in front of mine is your filename like the others in that file (the ones that work)? probably some option to prefix /etc/bind/zones infront or a chroot I have file "master/stonehenge.com" chroots tend to be in /var/named and file "master/..." for all the rest are you sure you don't need "master/..." ? options { directory "..." } kitkatbar: what OS, and what version of BIND? it's relative to that, I think debian 5 bind 9 im adding it jus tto see im curious if i need to go shoot myself for not seeing that haha host -t soa 0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa. localhost Using domain server: Name: localhost Address: 127.0.0.1#53 Aliases: 0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa has SOA record ns1.nullbnc.com. nullbnc.com. 201011092 86400 1800 172800 259200 kitkatbar - was that it... "master/..." ? ha ha yea kitkatbar - still wrong SOA but you're getting closer it was thinking it was in the same dir as named.conf i know i just fixed it when i saw it :) unless your email is "nullbnc@com" :) <- slow learner but persistant lol <- slow teacher but easily distracted ok - next step is to see if the delegation is coming at you from above host -t ns 0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa better yet host -t ns 0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa 4.2.2.2 so you're not getting a local value Yeah - it's pointing at ns1 and ns2 so you have to get both ns1 and ns2 to agree on this record now but ns2 is just a mirror of ns1 for me :D on a different ip "mirror"? lol @ i dont even think ns2 is setup that's the next step then make all the errors on this go away, and you'll be done: http://www.intodns.com/0.c.4.3.8.f.2.f.7.0.6.2.ip6.arpa yea i was there earlier had quite a few errs thx for your assistance i'll try to finish up was my goal for today :D depending on your timezone, you're either just in time for a beer, or now can begin your workday. :) i'd be on the side of the beer lol It's always beer o clock beer:30 so do i really need two nameservers for ipv6 reverses to work? no - but your ns delegations need to agree shit so so if you don't want two nameservers, don't let your upstream say that if i said ns1 and ns2 on the delegation request it's all about the upstream yeah i need to email them to remove ns2? uh, whatever can you do that for me? :) but you really should have a secondary especailly a secondary that is far far away from you he.net provides those for free that's what I'm using ok i'll check into it tunnelbroker? yes oh i was looking for kitkabar. just went through the logs. his reverse zone is ok, but he's missing a few zeros in his ptr reconrds.