[00:59] brycec: 4TB is still the best value right now; I did a calculation a few weeks ago [01:00] something like this, for example: https://www.newegg.com/Product/Product.aspx?Item=N82E16822236350 [01:00] as for us personally, we are still just re-using the myriad of drives we have left over from the old kvr machines [01:00] so, 1TB each [01:06] depends on the workload if it's enough iops going with 4tb though [01:07] archiving data is way less iops than lots of virtual servers reach writing to log files etc. [01:07] s/reach/each/ [01:07] archiving data is way less iops than lots of virtual servers each writing to log files etc. [04:45] *** mhoran has quit IRC (Quit: mhoran) [04:54] *** mhoran has joined #arpnetworks [04:54] *** ChanServ sets mode: +o mhoran [05:38] *** hive-mind has quit IRC (Ping timeout: 258 seconds) [05:39] *** hive-mind has joined #arpnetworks [08:39] brycec: did you have an ideal pricepoint per disk for enterprise drives? [09:34] *** ziyourenxiang has quit IRC (Ping timeout: 268 seconds) [16:52] *** ziyourenxiang has joined #arpnetworks [17:47] *** JC_Denton has quit IRC (Quit: The whole problem with the world is that fools and fanatics are always so certain of themselves, but wiser people so full of doubts.) [18:01] *** JC_Denton has joined #arpnetworks [21:20] Thank you mercutio and up_the_irons for the input. [21:21] Alas I'm but a mere dev who's managed to do sysadmin work (or vice-versa), but doing the hardware thing? Ooof I am out of my depth. [21:24] up_the_irons: 4TB certainly seems like a sweet spot, but even that WD drive (which I have reliability concerns with, thanks to Backblaze's data) suffers the same exact fault of the HGST drive I had my eye on. Finding actually new inventory is hit-and-miss at best. [21:24] And buying a hard drive with who-knows-how-many "miles" on it already feels like it's seriously asking for trouble. No? [21:24] (^ Referring to "refreshed"/"renewed" drives) [21:28] m0unds: Ideal price point? Ideally, $75 for 4TB I guess. But only because that's a price I saw when I first started searching, and because $1200-$1500 (16-20 drives) is a relatively easy pill to swallow. [21:29] But then, even, $150 for a 4TB feels "reasonable" for an "enterprise/DC" drive. [21:29] just assume disks will fail [21:29] you need to figure out how many iops you need though [21:30] But how the hell do I do that? :/ [21:30] 4x4tb hard-disks will only give iops slightly higher than 4/3 of one hard-disk [21:30] by looking at statistics from your current work load [21:30] (I know you described figuring out what I've got now, but that's not quite the same as "what do you need") [21:31] yeah well i reckon it's always best to overspec slightly in the beginning [21:31] as storage requirements can both increase over time suddenly, can start failing worse and worse past a certain point.. [21:31] "4x4tb hard-disks" reminder than I'm talking 16x4TB disks (spread across 4 servers) [21:31] and can perform much more poorly in partial failure situations [21:32] well i used 4 as en easier example [21:32] if you do 3 way replication it usually reads from a single drive, but it needs to write to all 3 [21:32] so you'll get slightly more than 16/3 iops [21:33] if each hard-disk does 100 iops, then that is ~533 iops [21:33] Ohhh that's what you meant, gotcha. (I misunderstood what you were saying) [21:33] it can read from different hard-disks as each block lives on one primary [21:33] but if you have significant write loads then you can still run into performance deficits [21:33] (Very RAID5-esque in that way) [21:34] yeah [21:34] like heaps of people used to do raid 5 for vm work loads [21:34] and it often failed pretty badly [21:34] even with raid10 like we had, neighbours could still disrupt each other. [21:34] mercutio: My predecessor did :( [21:34] (RAID5, that is) [21:34] yeh it's quite common [21:35] it used to not be so bad for things like mail work loads [21:35] We tried evaluating Logzilla on that array... it went *horrifically* [21:35] back when we had 9gb scsi disks etc [21:35] thing is hard-disk sizes went way up, and iops didn't... [21:35] like even u320 scsi etc had 15000 rpm hard-disks [21:36] ssds have helped a little [21:36] one of the cool things about ssds is that performance tends to be pretty good in mixed work loads, and with random accesses [21:36] <3 SSD (in general) [21:36] whereas with legacy hard-disk systems things like reading 10,000 files etc can bog down other things [21:37] but yeah, i went through all the vm hosts to look at iostat output btw [21:37] but it's still hard to estimate things like peak load [21:38] I've got munin doings its thing, but it's rubbish for precise values (naturally) [21:38] *doing its [21:38] anything within a vm is not going to be as good as the host [21:38] (Also, apparently I disabled munin on the VM hosts at some point so I don't have current data *facepalm* [21:38] what system are you running atm? [21:39] well you can still set it up again and get current data [21:39] better than nothing [21:39] Yeah I will be [21:39] Proxmox [21:39] one thing that i noticed about our work load is that we're higher on writes than reads btw [21:39] ah so you can get an estimate using iostat [21:39] both now [21:39] and since reboot [21:39] do iostat -x [21:39] for since reboot [21:40] you want to look at w/s and r/s [21:40] My ARP Metal host (also running Proxmox, and _does_ have working munin data) seems to peak at 400 IOPS reads on both of its spinning rust so that's probably the drives' limit there. And 300 on writes. So that gives me a number I can understand at least. [21:40] also you can get an idea how bogged down the current system is with r_await, and w_await [21:41] (Seems that Debian no longer packages iostat) [21:41] how many hard-disks is that with? [21:41] sysstat package [21:41] Oh nevermind, yeah sysstat [21:42] mercutio: I'm on an STL host, so 2. [21:42] iops on hard-disks can still go up with sequential data too [21:42] (A pair of WDC WD1003FBYX-01Y7B1 in my ARP Metal system, configured as a ZFS mirror vdev) [21:51] * brycec watches a RAID10 array do what RAID10 arrays do... via iostat. [21:52] (It amuses me to see it in action - reads from 2 disks, but writes to all 4, etc) [21:53] Thanks again mercutio for helping me through my minor meltdown [21:53] I'm still over my head, but at least I have some IOPS numbers in front of me I can understand [21:54] Now if only I could buy brand new drives of the capacity and spec that I want. [22:01] i suspect you can buy new hard-disks fine [22:01] you were saying amazon has buy limit? [22:02] there's probably someone else you can deal with too [22:05] gah, buffer was too small [22:06] mercutio: any idea why ssd storage on a "thunder" vm would be slower than spinning disk? more contention on the host or something? [22:06] it shouldn't be slower [22:07] it's like twice as slow, iops vs spinning using ioping -R . [22:07] oh latency higher [22:07] umm probably because it's going through network [22:07] well, latency is also higher [22:07] ioping is single access at once [22:07] ah, gotcha [22:07] spinning disks are physically attached? [22:07] so if you run two ioping at once it shouldn't degrade as much [22:08] yeh it's one of the negative things about ceph [22:08] not a lot that can be done about it [22:08] gotcha, not a big deal, was just curious [22:08] that said, if rdma support was stable it'd be a little better [22:08] figured that was the case though, attached vs network - better parallelization or whatever? [22:08] well there's multiple memory copys [22:08] and layers of overhead on the ceph side [22:09] whenever you receive network traffic and have to act on it and respond it's slower than staying within the cache of a local system [22:09] brycec: i missed it - did you say whether you had a working price range for disks? i was just having to do some shopping for a client, so i had some numbers for a bunch of vendors, haha [22:09] even if you had very very low network latency the unpredictableness of incoming traffic means that cpu caches etc aren't ready [22:09] mercutio: gotcha [22:09] so you're more likely to have cold cpu cache etc [22:09] plus the memory copies not doing rdma [22:10] if you look at rdma latency tests they're pretty good if both sides and ready/waiting just checking latency [22:10] this intel octane memory is lower latency than nvme ssd too [22:10] apparently it's noticably faster [22:11] interesting [22:11] m0unds: $75-$150 for a 4TB drive but for no better reason than that's where prices seemed to be when I started shopping [22:12] 21:27 m0unds: Ideal price point? Ideally, $75 for 4TB I guess. But only because that's a price I saw when I first started searching, and because $1200-$1500 (16-20 drives) is a relatively easy pill to swallow. [22:12] 21:28 But then, even, $150 for a 4TB feels "reasonable" for an "enterprise/DC" drive. [22:13] cool, thanks (i need to resize my znc buffer, it's too damned short) [22:13] m0unds: [FBI] should have a log too [22:13] rbi is back [22:13] oh, right, forgot about that [22:13] yeah [22:13] fbi even and bryce said [22:16] https://ceph.com/community/part-3-rhcs-bluestore-performance-scalability-3-vs-5-nodes/ [22:16] yeah, looks like that's about the range i saw, even with some qty discounts [22:18] tl;dr "More nodes == way higher IOPS achieved" [22:18] but similar performance for bulk data [22:19] well it also looked around the number of nodes you may be looking at [22:19] hahaha [22:19] the blog is pertty cool [22:19] mercutio: thanks (4) [22:19] https://ceph.com/community/ceph-block-storage-performance-on-all-flash-cluster-with-bluestore-backend/ [22:20] of course all flash is going to give high iops :) [22:20] I really appreciate their "Executive Summaries" [22:20] they're so well written [22:20] i'm used to things like anandtech [22:20] which are pretty average [22:23] I wish I could afford 5 nodes with 7 $1200 SSDs each :/ [22:24] (Their all-flash cluster uses 7 Intel 4TB SSDs) [22:25] fancy fancy fancy [22:25] yeah i think intel are sponsoring it or something [22:26] maybe overheads aren't as bad as i thought they were [22:26] it's improved a bit i think [22:27] i haven't even hawerd of this cpu.. xeon platinum 8180 [22:27] 38.5mb cache! [22:27] 28 core [22:27] oh there's two of them [22:28] ok that is weird, there is 196gb of ram [22:28] usualyl it'd be 192gb? [22:28] >>> 192*1024/1000 [22:28] 196.608 [22:28] hmm.. [22:29] also as well as those 4tb disks they've got those stupid fast intel optane memory ssd [22:29] some of intel's 3d nvm chips are mfg here [22:30] https://www.newegg.com/Product/Product.aspx?Item=9SIAH9B99N0963&Description=intel%20optane%20375gb&cm_re=intel_optane_375gb-_-9SIAH9B99N0963-_-Product [22:30] only $1500 each :) [22:30] hahahaha [22:30] it'll trickle down i'm hoping [22:31] haha, those are the chips they build here [22:31] where is here? [22:31] the ones in optane and micron quantx [22:31] well, 5 mins north of here in rio rancho, nm, usa [22:31] ah [22:31] Intel Optane SSDs have roughly 10X lower latency and 5-8X better throughput at low queue depths compared to the fastest PCIe NVMe NAND-based SSDs. [22:32] kind of crazy... [22:35] ah they talk about how they're benchmarking etc too, adn they are using fio like i suspected [22:35] but there's actually a librbd module [22:35] so it's easier than setting up virtual environments i suppose [22:35] cool [22:49] So, opinion poll: "Renewed" drives, yea/nay? [22:49] i'm guessing that you shoudl probably just get cheap 2tb seagate and shit loads of them [22:50] and then redo it with ssd later :) [22:50] or add another ssd storage pool [22:50] On the one hand, drives are gonna fail, plan/expect that. On the other, I just don't trust others' used stuff. [22:50] the only problem with cheap drives is that they don't "fail" quickly and they don't have vibration protection [22:50] yeah i'd buy new if not getting mass discount [22:50] i'll bbl [22:51] but it's really what you can do for bulk servers [22:51] like if you can take 20x hard-disks its' better than 12 [22:51] you might be able to use nvme and have internal journal [22:51] but you wnat to be careful about TBW [22:54] TBW = terabytes written [22:54] (for those like me and didn't know the term) [23:14] *** mnathani has quit IRC () [23:34] *** hive-mind has quit IRC (Ping timeout: 268 seconds) [23:41] *** hive-mind has joined #arpnetworks [23:56] ah sorry, i was on my way out the door basically [23:56] yeah some SSDs aren't so good with TBW compared to others. it's mostly significant for journals.. [23:56] because journals don't need a lot of storage, but reuse that storage a lot