mnathani: I invited [FBI] back
LG should be back up today
Thank you!
up_the_irons: Just curious, what kind of drives is ARP buying these days? I'm trying to shop and having a helluva time (maybe I'm just out of practice) finding *new* 4-6TB drives. I don't need more than that (a modest initial roll-out, 4 servers @ 4 drives each), under 100TB (raw) is more than enough. But... gotdamn I'm having a time. It seems the brand new drives I can find these days are 10-14TB (wow)
and they're $300+/ea (breaking the bank); anything <8TB stopped being made at least 3 years ago it seems.
(I'm obviously looking at enterprice/datacenter "grade" drives, not WD Blacks or whatever)
(PS I got your sales response and I'll definitely be following-up as soon as I get this initial proposal done... which is currently hung-up on sourcing fucking drives.)
huge hard-disks are bad for VM work loads ok for archival
mercutio: precisely why I'm avoiding them too
well 4tb is huge
You think? It was my upper limit, but it just seemed "large" not "huge"
well most people are doing ssd these days
and not 4tb ssd :)
In any case, I'm about to start budgeting SSDs instead
lol
you really want as many drives as you can
we have hundreds
I also feel really out of touch with buying hard drives in general. There was a time I felt comfortable buying from Amazon or Newegg, provided the seller was Amazon/Newegg and not some rando third-party that's offloading "renewed" inventory.
4x4 is the wrong approach..
I get that, or at least I thought I did (quantity over capacity).
4*8?
well you also want journal disk usually
so you want servers that can take 12+ hard-disks really
Already accounted for that, dedicated SSD for that, dedicated SSD for OS
you want ssd per 3 or 4 disks
so like 12 disk server you may have 3 ssd and 9 data disks
it's changing a bit with blue store..
I'm limited by overall rackspace so I'm going with 2U servers, 12 slots (plus 2 for OS & journal)
i think with bluestore it can journal rather than full copy data so you may be able to have slightly fewer ssd
Yeah I've heard Bluestore is quite improved.
and if you're not logging shit loads then you should be able to use the same disks for journal and OS
although that does complicate things a little
Yeah I intend to do a bit of benchmarking there - RAID1 the 2 SSDs vs OS/journal split.
with mdadm you can do raid 1 with part of the disk and not all of it
you can also do things like 3 way mirror for OS
curious i wonder why that is
did you scope out what performance you need?
They probably don't trust me to not be starting my own reseller
mercutio: From my users, "better than it is now"
damn it costs the same for 5tb and 2tb?
what iops can you do now?
I honestly couldn't say :/
have you tried fio?
ioping can be handy too
like you can do ioping -R /dev/vda
and it'll tell yuo your access times
it's pretty latency dependant
Thanks I'll check those out
fio is a little more difficult to work
but you can do things like simultaneous 4k, or 64k, or whatever requests
and you can do read/write, mixed etc to a file
it's basically the best tool for in depth benchmarking
but you have to figure out what it is that you want to benchmark :)
also for ssd for journal you want an ssd that has battery backed write cache
or power loss protection
as it means you get a lot lower latency...
https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
brycec: I got 4TB HGST Ultrastar 7K6000 drives for our office bulk storage array last year - they've been fine (not using Ceph though)
damn they have 14tb disks now
i wonder if going for such a small cluster if it is better to just go all ssd from the get go
especially when you say density is important
you can get servesr that take a lot of 2.5" disks