whitefang: I don't need it done any time soon, but yeah, I'm having regrets :P up_the_irons: whitefang: can't say right now. all these VMs are going onto kvr14, it will probably fill up the box
time to order more hardware.. yikes whitefang: well, it would be +256MB RAM and +20GB disk space.
I'm not thinking major upgrade, as much as I'd like to throw down the money on 4GB of RAM all to me.
you could just shave a few MB here and there off of everyone moving over to the machine and allocate it to me...I'd be OK with that.
time to bounce for a bit, good luck with the damage contol up_the_irons. up_the_irons: whitefang: thanks! DDevine: up_the_irons: Are the servers branded or built from parts? up_the_irons: DDevine: i build them, they are supermicro DDevine: Ah no next day parts :S up_the_irons: well, i *could* have got another raid controller fedex overnight, but i figure what is the use, the data is already corrupted DDevine: Whole new box? How long has that one been in service? ***: nakano_ is now known as nakano G: DDevine: he said earlier, first one DDevine: G: Sorry, I have been skipping around many channels and doing lots of little things so I miss a lot of the finer points. up_the_irons: DDevine: mercury has been in service since 01-17-2009 ***: schmir has joined #arpnetworks
nakano is now known as nakano_ whitefang: don't you mean "had been"? sorry, too soon? -: whitefang ducks. ***: wwrl_home has joined #arpnetworks
schmir has quit IRC (Remote host closed the connection)
schmir has joined #arpnetworks up_the_irons: whitefang: oh man
kick me while i'm down why don't ya ;) DDevine: Also, just got an interesting email from Amazon. They now do 5TB files and the Multipart upload can do some interesting stuff. http://fpaste.org/qM53/
You can stream the file into S3 as it is created. up_the_irons: yeah i got that too ***: Arendtsen has joined #arpnetworks
whitefang has quit IRC (Read error: Operation timed out)
Ehtyar has joined #arpnetworks
whitefang has joined #arpnetworks
_Ehtyar has joined #arpnetworks
wwrl_home has quit IRC (Quit: leaving)
Ehtyar has quit IRC (Remote host closed the connection)
ziyourenxiang has joined #arpnetworks
richardquisumbin has quit IRC (Ping timeout: 265 seconds) mike-burns: I expect tarsnap sales to go up. MuskyHusky: oh
RIP mercury ***: fink has joined #arpnetworks DDevine: After resolving to actually think more logically about my problems and to be a bit more attentive to detail -- I got my new mail setup working.
The whole experience has been fairly enlightening.
I hate mail systems, but I conquered this one. ***: DDevine has quit IRC (Remote host closed the connection)
nerdd_ has joined #arpnetworks MuskyHusky: im definitely setting up some sort of cronned rsync solution from this
i can't wait to setup my file server...wait whats stopping me? i have the free time....i dont know what distro i want to use fink: freebsd! ***: nerdd has quit IRC (Ping timeout: 276 seconds) mike-burns: FreeBSD. ziyourenxiang: 8.1!
zfs! MuskyHusky: i already have freebsd computers out the yingyang ;p
i want to use linux this time ok
maybe debian. fink: meh MuskyHusky: what im particularly interested in setting up is a lvm with 8 hard disks amdprophet: I personally use FreeBSD with ZFS but I have a friend with Linux with BTRFS MuskyHusky: zfs can do single partition across multiple disks? amdprophet: yea ***: ziyourenxiang has quit IRC (Quit: ziyourenxiang) amdprophet: and has some really neat features like filesystem compresssion MuskyHusky: oh cool
might have to check it out then. fink: jbod MuskyHusky: well they are mostly ide drives on different controlelrs ***: ziyourenxiang has joined #arpnetworks
ziyourenxiang has quit IRC (Quit: ziyourenxiang)
schmir has quit IRC (Ping timeout: 276 seconds) jpalmer: whitefang: RAID is for business continuity in the event of a failure. backup is for data recovery in the event of a catastrophic event.
a *lot* of people make the mistake of thinking RAID = backup, when it's actually not. (it seems like you fall into this category, hence my mentioning this after the fact) toddf: RAID is for some failures, oviously it is not foolproof ;-) ***: sentabi has joined #arpnetworks
lucky has joined #arpnetworks lucky: Silly question. I have the (old) 768 MB / 20 GB / 200 GB for $20. I see a 768 / 20 GB disk / 400 GB for $20 now. Can I get that extra 200 GB? :)
or new customers only? ***: nerdd_ is now known as nerdd
cubelogic has joined #arpnetworks jpalmer: lucky: you can upgrade to the new bandwidth tier for a 1 time fee of $20, if interested, submit a support ticket with your UUID, and permission to charge the $20
lucky: it's $20 per VPS, just to be sure I'm clear. nerdd: Does anybody know what to do, when freebsd-update tells me that it can't identify running kernel when I try upgrading from 7.2 to 8.1? fink: jpalmer: what about upgrading RAM?
nerdd: do you have a custom kernel? lucky: jpalmer, alright. thanks :) jpalmer: fink: not sure the cost of RAM upgrades. I should probably find out. nerdd: Nope, it's generic, and it's a clean setup (my vps was on mercury before the crash) fink: jpalmer: i think i want that in hte next few months ***: vapor has quit IRC (Read error: Operation timed out) up_the_irons: mike-burns: LOL, I should make a deal with tarsnap ***: robotarmy has quit IRC (Remote host closed the connection)
robotarmy has joined #arpnetworks
robotarmy has quit IRC (Remote host closed the connection) whitefang: we all want more ramz! ***: baklava has quit IRC (Ping timeout: 255 seconds)
baklava has joined #arpnetworks
baklava has quit IRC (Read error: Connection reset by peer)
baklava has joined #arpnetworks bitslip: ***: robotarmy has joined #arpnetworks mattx86: I was just looking back at tarsnap, and it looks pretty good
would you guys recommend them? mike-burns: Yup. up_the_irons: mike-burns: how easy is it to do a restore? mike-burns: It has the same syntax as tar, approximately, so it's about that easy.
So, kinda easy. mattx86: I've never tried with tar, but can you extract one particular file with tarsnap archives? ***: robotarmy has quit IRC (Remote host closed the connection) mike-burns: Yes. I think.
To be honest I've only played with it and hope to never need to do a restore. mattx86: nice, looks like its possible; the general usage page shows them restoring two users' home directories toddf: someone should make something more user friendly than venti from plan9port. there's a redundant block storage shrink-o-matic backup mechanism. it's achilles heel: no data ever gets deleted that is archived. *sigh*. epitome was to do that, but it's klunky, epitome2 was never released to the public nor finished, bleh.
backing up whole lvm partitions seems rather non-guaranteed in the consistency department.
if there is serious desire for backups at arpnetworks it might be best to follow the (very good and useful) model at other hosting companies: provide a server (with no bandwidth penalty) to dump backups on for the vps customers .. mattx86: that's an interesting concept toddf: maybe there should be some form of quota (lvm per vps for backups) .. or maybe even an iSCSI initiator ..
s/initiator/target/ up_the_irons: toddf: i had issues with that and access control. in order to save owner/group/perms, you need root access on the backup server. so if everyone has root, what is to keep someone from grabbing other customers' data? toddf: up_the_irons: chroot ftp/sftp access?
and why root on the backup server?
that sounds like a poorly designed backup app in the security department up_the_irons: toddf: if no root on backup server, everything is saved as the same user
UID's can't change
so a restore is then useless
might be ok for /home/foo, but not for like the root fs toddf: are you talking rsync or what? I'm talking 'vps people create archive of their own desired flavor, I'd do gzip'ed dumps, randalshwartz would do zfs send/dumps, etc ..' up_the_irons: toddf: rsync / tar
toddf: so each customer would get a 2nd vps for backup purposes? toddf: tar = no problem, rsync .. I'm not the type that prefers that, then again my parents systems are rsnapshot'ed to my brother's systems *shrug*
up_the_irons: disk storage is all I'm theorizing about
how to access it, is the question. do you setup 'backup.cust.arpnetworks.com' or do you setup 'backup-iscsitarget.cust.arpnetworks.com' or 'backup-ftp.cust.arpnetworks.com' or something similar? up_the_irons: toddf: ah, so rsnapshot keeps metadata about the owner/group/perms and then restores them? toddf: aka make the concept of backups a little more abstract than 'use rsync' up_the_irons: and obviously tar keeps that data toddf: I'm pretty sure rsnapshot is not going to do what you want I only mentioned it because I'm not a 'rsync for backups' fan, I'm a 'rsync for moving my pics of my kid around' fan .. yet my parents get their systems backed up by my brother via rsnapshot which is essentially rsync + subdirs + hardlinks
everybody has their idea of backups of course up_the_irons: roger toddf: mine centers around 'system needing backed up shovels data via some protocol in some format to some method of archive server for later retrieval' .. typically tar and dump + gzip/lzma/bz2 based .. mike-burns: I check everything into git and backup my git repos. toddf: now, being someone who had to re-install his vps this morning, I have to wonder if it'll effect your business model if you start offering an archive server for vps customers backups .. if I had to choose between $10/mo and no backups and $12/mo and backups, I'll do my own backups .. since my vps is primarily 'regurgitate info from other servers' (dns slave, mirrored http files, etc) .. the backup in my case is in my head. mike-burns: I'd pick a backup plan. toddf: obviously other users have other usage cases.
mike-burns: you'd consider `backup' a thing to add to your monthly bill? (like insurance, I guess?) up_the_irons: i check things into git and don't backup my repos ;) given that git is distributed, there's always some computer that has it. i have 2 laptops and a gitosis server, so 3 copies right there already mike-burns: I'd pay extra for someone else to handle backups in a completely transparent way.
Ah, I delete clones of repos constantly. up_the_irons: ah
mike-burns: the "completely transparent" way is where I have trouble. I haven't found a cost effective solution. there are expensive iscsi master/slave solutions, but they are.. expensive ;) toddf: mike-burns: with an operator account to your vps with accessto read your disks (aka do a dump) or backup the lvm on the host system directly? mike-burns: Yeah, the "completely transparent" thing is the real issue here, and probably why it won't happen.
I mean that I don't want to notice that my system is being backed up (until I need it).
I could tolerate an operator account, but anything more than that risks cutting into disk space. toddf: iscsi can be inexpensive if you're not searching for `hardware' and `warranties' etc ..
http://pastebin.ca/2015847 up_the_irons: toddf: backup the lvm on the host directly is pretty taxing on the host, and if the majority of customers select backups ,then performance will degrade fast toddf: up_the_irons: definately... up_the_irons: toddf: wow, what are all those volumes? :) toddf: about the only way to do lvm backups is if you had some snazzy iscsi target that did snapshots of the blocks and then permitted a 2nd host to backup the snapshots, even then filesystem consistency would be an issue. I'd expect things to be more efficient and more useful to people if the native os tools in their vps were used to create backups from consistent data states and use that to restore with in a recovery effort.
the server name gives a big hint
under my company I have 2 afs cells, one internal only (fries.net) and one publically accessable (freedaemon.com)
since most of the data is infrequently accessed, and the local disk io hits 6mbit/s on the iscsi volumes that's pleanty fast enough when I'm normally not accessing the fileserver locally (1.5mbit/s max outbound at the office) up_the_irons: i c toddf: (and yes I'm using claudio@'s iscsid under openbsd for the initiator and netbsd's iscsi-target for the server also running on OpenBSD)
he had a good laugh at me when I said 'yes, it works with native v6 over ipsec over wifi from home to the office' up_the_irons: lol toddf: it's not meant for production, but it works well enough to put my data on it knowing that if any number of things go bump I'm rebooting afs4 and fsck'ing for 360s at least .. openbsd source trees and other re-downloadable stuff seems ok to put there jpalmer: what about a SAN mounted to each VPS (for an additional cost) all backups are done by user in their preferred style (rsync, dump/restore, tar, etc) and just saved to the SAN mount. mike-burns: TBH, the issue with all of these is that they're on-site. jpalmer: mike-burns: IMO, if a client wants offsite, they need to appropriate those resources themselves. mike-burns: Sure, I can understand that. ***: baklava has quit IRC (Ping timeout: 255 seconds) jpalmer: up_the_irons: another vps I used a couple years ago, every vps had the option of installing a bacula agent, and the VPS provider ran a centralized backup bacula server.
it sounded like a lot of overhead on the providers part.. so it may not be suitable here. providing an iscsi target to be mounted in a VPS, and letting the client handle his own backups sounds a lot more.. maintainable. ***: baklava has joined #arpnetworks
baklava has quit IRC (Changing host)
baklava has joined #arpnetworks toddf: maybe I should do more than just .. add it to my todo list .. netbsd's iscsi-target doesn't have config reload capabilities, you restart it to reload the config, and disconnect all clients in the process .. brutal! ;-) ***: ww has quit IRC (Read error: Connection reset by peer)
richardquisumbin has joined #arpnetworks
MuskyHusky is now known as knot
knot is now known as sheath
G has quit IRC (Ping timeout: 264 seconds)
G has joined #arpnetworks up_the_irons: yay, I just thought, the only good thing to come about mercury dying is that i can finally get rid of the LSI MegaRAID controller; i HATE its CLI. Gonna pop in a 3ware 9650-4LPML and be back in action ***: sentabi has quit IRC (Quit: Leaving) jdoe: eh
3ware's cli sucks too.
and their current firmware for that controller requires a newer version of the driver than is in a lot of current distro versions.
... which sucks, since you then have to remember to update your initrd with their driver copy or leave your machine unbootable :P ***: nesta has joined #arpnetworks
robotarmy has joined #arpnetworks
robotarmy has quit IRC (Remote host closed the connection)
whir has joined #arpnetworks
nesta has quit IRC (Ping timeout: 272 seconds)
whir is now known as nesta up_the_irons: jdoe: i haven't found this to be the case, I always put on the latest firmware and it works no problem
jdoe: and the CLI is way better than LSI's. It's *simple* jdoe: up_the_irons: I'm talking reasonably old at this point, ie centos up_the_irons: i c jdoe: are we talking about tw_cli? up_the_irons: yeah jdoe: I dunno about "simple" exactly, I always get lost trying to figure out which sub-section I want to be in up_the_irons: ./tw_cli /c2 show
/c2/u0 show all jdoe: /c0/p0 /c0/u... etc
yeah up_the_irons: /c2/u0 show alarms
easy jdoe: the, uh, sadder thing
is that I haven't figured out how to make the auto-scrub/disk check work properly.
I give it a day to run on, but what happens it just runs *continuously* all day, looping.
rather than one-shot.
don't suppose you know what I'm doing wrong? :P up_the_irons: not sure, i don't run the auto-verify jdoe: doh. mattx86: hm.. I think I'm going to use duplicity + amazon S3 for backups
seems like a pretty solid choice ***: nakano_ is now known as nakano
wwrl has quit IRC (Quit: leaving)
fink has quit IRC (Ping timeout: 255 seconds)
baklava has quit IRC (Ping timeout: 255 seconds)
baklava has joined #arpnetworks
cubelogic has quit IRC (Remote host closed the connection)
fink has joined #arpnetworks richardquisumbin: isn't 3ware an LSI company? ***: nakano is now known as nakano_ jdoe: they are now, yeah.
he's right though, 3ware-branded gear sucks less to deal with than LSI-branded gear :P ***: lucky has quit IRC (Remote host closed the connection)
fink has quit IRC (Quit: fink)
baklava has quit IRC (Ping timeout: 255 seconds)
baklava has joined #arpnetworks
lucky has joined #arpnetworks
boogeyman has quit IRC (Ping timeout: 240 seconds)
lucky has quit IRC (Remote host closed the connection)
lucky has joined #arpnetworks
DDevine has joined #arpnetworks
alcy has joined #arpnetworks
Ehtyar has joined #arpnetworks
__Ehtyar has joined #arpnetworks
_Ehtyar has quit IRC (Remote host closed the connection)
Ehtyar has quit IRC (Remote host closed the connection)