#arpnetworks 2011-05-02,Mon

fink

RandalSchwartz: did you get an arp tshirt yet?

RandalSchwartz

something we clueful people can help with?
fink - I don't think so

fink

RandalSchwartz: me neither :(

RandalSchwartz

haven't seen any recent packages

unfortunately not.

RandalSchwartz

ww - then send email to support@
it'll get done as fast as it can get done then. :)

[11:14]

machine has become unstable, probably because of zfs and insufficient ram (and i recently added some things that eat an extra 400Nb of ram)
so need a ram upgrade... erm urgently...

[11:14]

RandalSchwartz

then you should probably back that out until you get more ram

[11:14]

i sent an email...

[11:15]

RandalSchwartz

"urgent add of ram" is not a service provided here
"adding ram in a day or two" is more likely
you could temporarily add more swap space

[11:15]

***

ivan-kanis has joined #arpnetworks

[11:16]

RandalSchwartz

that'd at least give you the option of pushing stuff out to disk that's less active.

[11:17]

backing out would be hard, in this case... spinning up another host elsewhere more likely if it'll take a while, but i'd rather not

[11:17]

RandalSchwartz

poor planning then. :(

swap... well... 4s-backend (see 4store.org) gets very unhappy if it gets swapped (which happens now from time to time and is painful)

fink

damn, 400Nb!

RandalSchwartz: planning? well...

RandalSchwartz

Nanobytes!

fink

nebibyte?

fink: 400Mb out of 1024 is... well... significant

fink

ww: out of curiosity, what did you add?
i switched from apache to cherokee, and saved a shload of ram

actually, added nothing substantial new in terms of software (some minor upgrades)
what i did add was a pile of data

[11:19]

RandalSchwartz

... Optimum price/performance can be got with clusters of machines with large amounts of RAM. 16+GB per node is typical currently.
you mean you didn't read that and go "heck, I need more ram">
that'd be my first clue

[11:19]

RandalSchwartz: depends on the quantity of data, actually

RandalSchwartz

yeah - but anytime people say "this will need a lot of ram" what they mean is "I don't code very well" :)

with 16Gb of Ram you're talking ~100-150 million triples (rows, approximately)
i've only about 10 million

RandalSchwartz

so why can't you just get rid of your data for now
roll it back, and wait for ram?
you didn't go live without testing did you?

because of a dog and pony show tomorrow

fink

here we go…

RandalSchwartz

again - poor planning. :(

what me plan?

fink

;)

RandalSchwartz

remind me not to subcontract to you. :)

fink

i DO have that tshirt

ivan-kanis

oh, not a paying customer dog and pony show
if it were a paying customer they would be paying for bigger hardware

[11:22]

RandalSchwartz

you missed the point, but that's fine
everyone else here got it. :)

[11:22]

anyways, re: planning... plenty of time to spin up a machine elsewhere if this can't be sorted quickly

[11:26]

RandalSchwartz

I don't know why you seem to think that answers or contradicts my claim

[11:27]

fink

ww: what OS?

[11:27]

because... there is no problem that can be attributed to inadequate planning?
fink freebsd 8.1
and i'm *assuming* that Fatal trap 9: general protection fault while in kernel mode
is caused by ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior.

[11:28]

fink

ww: i run fbsd 8.1 with zfs on root on a 768 vps, but i tweaked the zfs settings a bit, and i doubt i have your load
what processes / data are you running?

[11:30]

misc small things - it's the laboratory host
and, the metadata aggregator
which i would point you to, if the machine wasn't thrashing around again at the moment

[11:31]

fink

ww: ok, what's your largest process group, maybe we can think of ways to make it leaner?

[11:34]

4s-backend
not much to be done really. it alone occupies half the ram.
and there's an obligatory 4s-httpd whose size depends on the complexity of queries
then there's nginx, and a harvesting daemon
most everything else has been turned off
ww just spins up a new machine...

[11:35]

***

fink has quit IRC (Quit: fink)
fink has joined #arpnetworks

[11:42]

phreak has joined #arpnetworks

[11:57]

..... (idle for 20mn)

reardencode

fink: mind sharing some of the tweaking you did to run ZFS on a 768VPS? I considered it but figured that it'd freak out due to lack of ARC space if I used ~650mb of that 768 for actual running processes

[12:17]

ww is interested in that as well

[12:19]

..... (idle for 20mn)

RandalSchwartz

... http://wiki.freebsd.org/ZFSTuningGuide

[12:39]

............ (idle for 57mn)

for interest's sake, the service in question http://semantic.ckan.net/ (pretty pictures here, http://semantic.ckan.net/record/2ab28263-a1fe-4839-9409-fa7ef03e123e and here http://semantic.ckan.net/group/?group=http://ckan.net/group/lld) is noww back up and running smoothly
unfortunately, elsewhere (rackspace)

[13:36]

.... (idle for 19mn)

***

ivan-kanis has quit IRC (Remote host closed the connection)

[13:55]

RandalSchwartz

elastichosts is a lot more flexible than rackspace
at least in my observation
we're using elastichosts for BCP behind ARP
with elastichosts, if you want more ram, you slide a slider, and reboot. :)
it's *that* fast

[13:57]

I'm building a 8.2 box on elastichosts right now

[14:05]

yes, i'm familiar with them, but there's already an account, etc. set up with rs...

[14:06]

toddf

randalschwartz: elastichosts.com/could-hosting/pricing clearly shows that their smallest server is 1gb mem .. if you do a 'subscription' for 1gb mem 40gb disk 400gb xfer (same as arp's $30/mo system) you'd pay $107.92/mo there. huge difference in my world.
though as BCP plans go, it surely is good to make them

[14:19]

RandalSchwartz

that doesn't make sense

toddf

I also don't note IPv6 mentioned, perhaps you could answer if there is remote encrypted console access and/or serial console access

RandalSchwartz

I was dialing in some plans at $70/mo
so the minimum can't be 107.92

toddf

randalschwartz: minimum isn't 107.92
its just that if you compare apples to apples their minimum mem is 1gb
finding the 1gb mem option at arpnetworks, I chose the values from arpnetworks and plugged it into their little slider pricing guide

RandalSchwartz

... https://sat-p.elastichosts.com/docs/pricing
yeah - it's a bit more expensive overall
nothing is as cheap as ARP :)
and yeah, no ipv6 yet
they say "just use a tunnel" :(

[14:23]

toddf

"let he.net deal with IPv6 till we get our heads out of the sand"

[14:24]

RandalSchwartz

indeed. anyone not dealing with ipv6 today has their heads firmly hidden :)

[14:25]

toddf

I've seen some 4U plans for 100mbit links unmetered if I provided the server. so after it approaches $100 for finite resources i tend to gloss over and out
"plans" => for $100

[14:26]

up_the_irons

toddf: I'm ready for OpenBSD 4.9 BTW, so whenever you want to make those templates...

[14:37]

toddf

ack

[14:41]

RandalSchwartz

ahh - so you *can* use freebsd-update to go from 8.1 to 8.2 without recopying the boot blocks
unlike 8.0 to 8.1
for zfs-on-root
just verified this in a private vm

[14:42]

.... (idle for 15mn)

up_the_irons

nice

[14:57]

RandalSchwartz

so that means I can upgrade my 4 VPS with minimal steps
everything at the # prompt :)

[14:59]

............ (idle for 57mn)

dferris

dferris <3 the Dell iDrac 6

[15:56]

jpalmer

up_the_irons: tell me how to make templates. I'll make you one for CentOS 5.6 (and 6.0 if they ever release it) and RHEL 5.6/6.0

[16:02]

***

EhtyarWRK has joined #arpnetworks

[16:06]

up_the_irons

jpalmer: can you do FreeBSD 8.2 as well? I can write-up the docs tonight

[16:14]

jpalmer

up_the_irons: I'd love to.
I can likely do FreeBSD this week. are you interested in centos/rhel?

[16:16]

RandalSchwartz

yeay freebsd :)

[16:17]

jpalmer

and, can I get away from the single / partition? like, the sysinstall defaults? :)

[16:18]

reardencode

jpalmer: I think that single partition on these relatively small VPSs is probably preferable?

[16:22]

jpalmer

single partition is NEVER preferable.
(but, I'll do whatever garry wants)

[16:22]

jpalmer: ummm strictly speaking, Red Hat likely wouldn't like the idea of a RHEL VM image if there is now VM/Cloud licensing
*no

[16:23]

jpalmer

G good point. they'd want licensing. so CentOS only

that said....

reardencode

jpalmer: due to the different access patterns on the different POSIX dirs, or something else I'm not thinking of?

I guess you could label it as bring your own license, no updates otherwise

jpalmer

reardencode: filling up you / is a lot more dangerous than filling up most other mountpoints.
reardencode: but that aside, a small / is preferable, because all of the other paritions can be background fsck'd. / has to be done at boot. in the case of an unclean mount, a 20g / takes longer to fsck than a 512m /

reardencode

jpalmer: both valid points -- I guess you'd probably have to do at least two different templates, one for 5G vps and another for any other size, in order to leave even a reasonable amount of space for /usr on the 5G

[16:28]

RandalSchwartz

a single ZFS partition is fine
because I can rehack it at will

[16:30]

jpalmer

personally, I'm ok with whatever garry wants for a template. I can always resize on my own.

reardencode

really as ZFS matures, the many filesystem roots in a single pool of storage will win
RandalSchwartz: yep.

RandalSchwartz

so if you could also offer /-as-ZFS as a template, that'd be great

jpalmer

I doubt most people will want to run ZFS on a vps due to the limited RAM resources.

RandalSchwartz

and yet, I have four of them
and it's all working fine

jpalmer

also, to this day, I've never messed with ZFS (I don't have a need for it) so, I don't think I'd be the right guy to build a ZFS-as-root template

fink

RandalSchwartz: guess we're not most people

RandalSchwartz

apparently
we don't run in jpalmer's circles :)

jpalmer

heh
my circles are small, I chase my own tail a lot.

[16:33]

RandalSchwartz

I chase tail a lot too.
oh - "my own" no

[16:33]

reardencode

RandalSchwartz: how much RAM do you use for programs of your 768MiB with ZFS?

[16:45]

RandalSchwartz

I have to think which one of the boxes are like that.
probably mabel. lemme look
says 278MB free
that box is just email and DNS

[16:50]

pilgrimd

What do you have arc and kmem tuned to?

[16:51]

RandalSchwartz

the recommedations on ZFSTuningGuide
arc = 40M
kmem = 330M

search for 768 in http://wiki.freebsd.org/ZFSTuningGuide

I'm using those
haven't had a problem in 10 months or so
even with a fair amount of spam
running postfix with postgrey and amavisd

[16:52]

***

jazz57 has joined #arpnetworks

[17:00]

.... (idle for 17mn)

fink

amavisd is such a memory hog!
i have two vpses with the same setup as RandalSchwartzDoppleGanger

[17:17]

***

jazz57 has left

[17:28]

jaja

up_the_irons here?

[17:32]

RandalSchwartz

jaja - why?

[17:35]

jaja

I dont see the payment on my CC, I want to know if there's a problm.

[17:36]

fink

jaja: it can take a couple days

[17:38]

up_the_irons: second VPS requires the /29 subnet right?

[17:38]

fink

jaja: wait…payment or charge?
is this a soviet russia joke?
arp pays you?

[17:38]

jaja: I don't think the billing cycle has happened yet

[17:39]

jaja

charge sorry
I am french my english isn't perfect, and I am abit tired tonight

[17:39]

fink: payment is fine imo, because he's making a payment to arp

[17:39]

RandalSchwartz

you won't get your service disabled until he calls you
nothing to worry about
so if there's a problem with payment, it won't matter until he gets around to turning it off :)

[17:40]

just wait for the thugs in the black suits to appear :)

[17:40]

RandalSchwartz

and the helicopters!
don't forget the helicopters!

[17:40]

RandalSchwartz: I thought he replaced the helicopters w/ drones

fink

with the silent blades

jaja

I prefer fellow my billing, I am here since 6 month and generally charge is done 1th day of each month, sorry !

RandalSchwartz

whooop whooop whooop

all of the sudden they dart out of nowhere and pin you to the nearest brick wall :)