***: [FBI] has joined #arpnetworks up_the_irons: mnathani: I invited [FBI] back
LG should be back up today mnathani: Thank you! ***: ziyourenxiang has quit IRC (Ping timeout: 268 seconds) brycec: up_the_irons: Just curious, what kind of drives is ARP buying these days? I'm trying to shop and having a helluva time (maybe I'm just out of practice) finding *new* 4-6TB drives. I don't need more than that (a modest initial roll-out, 4 servers @ 4 drives each), under 100TB (raw) is more than enough. But... gotdamn I'm having a time. It seems the brand new drives I can find these days are 10-14TB (wow)
and they're $300+/ea (breaking the bank); anything <8TB stopped being made at least 3 years ago it seems.
(I'm obviously looking at enterprice/datacenter "grade" drives, not WD Blacks or whatever)
(PS I got your sales response and I'll definitely be following-up as soon as I get this initial proposal done... which is currently hung-up on sourcing fucking drives.) mercutio: huge hard-disks are bad for VM work loads ok for archival brycec: mercutio: precisely why I'm avoiding them too mercutio: well 4tb is huge brycec: You think? It was my upper limit, but it just seemed "large" not "huge" mercutio: well most people are doing ssd these days
and not 4tb ssd :) brycec: In any case, I'm about to start budgeting SSDs instead
lol mercutio: you really want as many drives as you can
we have hundreds brycec: I also feel really out of touch with buying hard drives in general. There was a time I felt comfortable buying from Amazon or Newegg, provided the seller was Amazon/Newegg and not some rando third-party that's offloading "renewed" inventory. mercutio: 4x4 is the wrong approach.. brycec: I get that, or at least I thought I did (quantity over capacity).
4*8? mercutio: well you also want journal disk usually
so you want servers that can take 12+ hard-disks really brycec: Already accounted for that, dedicated SSD for that, dedicated SSD for OS mercutio: you want ssd per 3 or 4 disks
so like 12 disk server you may have 3 ssd and 9 data disks
it's changing a bit with blue store.. brycec: I'm limited by overall rackspace so I'm going with 2U servers, 12 slots (plus 2 for OS & journal) mercutio: i think with bluestore it can journal rather than full copy data so you may be able to have slightly fewer ssd brycec: Yeah I've heard Bluestore is quite improved. mercutio: and if you're not logging shit loads then you should be able to use the same disks for journal and OS
although that does complicate things a little brycec: Yeah I intend to do a bit of benchmarking there - RAID1 the 2 SSDs vs OS/journal split. mercutio: with mdadm you can do raid 1 with part of the disk and not all of it
you can also do things like 3 way mirror for OS -: brycec grumbles - Amazon has a limit of 20 drives/customer mercutio: curious i wonder why that is
did you scope out what performance you need? brycec: They probably don't trust me to not be starting my own reseller
mercutio: From my users, "better than it is now" mercutio: damn it costs the same for 5tb and 2tb?
what iops can you do now? brycec: I honestly couldn't say :/ mercutio: have you tried fio?
ioping can be handy too
like you can do ioping -R /dev/vda
and it'll tell yuo your access times
it's pretty latency dependant brycec: Thanks I'll check those out mercutio: fio is a little more difficult to work
but you can do things like simultaneous 4k, or 64k, or whatever requests
and you can do read/write, mixed etc to a file
it's basically the best tool for in depth benchmarking
but you have to figure out what it is that you want to benchmark :)
also for ssd for journal you want an ssd that has battery backed write cache
or power loss protection
as it means you get a lot lower latency...
https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ dne: brycec: I got 4TB HGST Ultrastar 7K6000 drives for our office bulk storage array last year - they've been fine (not using Ceph though) ***: acf_ has joined #arpnetworks mercutio: damn they have 14tb disks now
i wonder if going for such a small cluster if it is better to just go all ssd from the get go
especially when you say density is important
you can get servesr that take a lot of 2.5" disks ***: ziyourenxiang has joined #arpnetworks