[00:08] *** ryk has quit IRC (Ping timeout: 264 seconds) [00:11] *** ryk has joined #arpnetworks [00:11] *** ryk has quit IRC (Changing host) [00:11] *** ryk has joined #arpnetworks [00:38] *** ryk has quit IRC (Ping timeout: 268 seconds) [01:07] up_the_irons: any suggestions for server grade boards with many pcie lanes? [01:07] im looking for something at home that is capable of supporting 3xsas cards and an infiniband card [01:09] Webhostbudd: intel or amd? [01:09] most likely intel, but amd might be an option [01:10] i was thinking intel because i may end up using it as a hypervisor, and freebsd / illumos have better intel than amd support [01:10] Webhostbudd: the intel boards we use are a bit shy on pci-e (we don't need much), but one amd board we're using has a lot: http://www.supermicro.com/aplus/motherboard/opteron6000/sr56x0/h8dgi-f.cfm [01:10] i see [01:11] im guessing you are using x<>scm boards? [01:11] oh nice [01:11] you get 3 16x to start, then 1 8x, and 2 4x [01:11] it's pretty loaded [01:11] yea, i was browsing around on supermicro's site trying to find the best deal [01:11] it seems as though you need dual socket to get a decent amount of lanes [01:12] we only use dual socket, so i wouldn't know about the single sockets [01:12] oh, alright [01:12] what about the new dedi's? [01:12] oh, well, yeah those are single [01:12] i see [01:12] and the kvm hosts are all dual socket? [01:13] the dedi's don't have shit for pci-e ;) [01:13] web: impressive home setup :) [01:13] 20:08:20 Webhostbudd im looking for something at home that is capable of supporting 3xsas cards and an infiniband card ???gneous [01:13] thanks [01:13] i can't say i've heard of anyone else wanting such high spec for home :) [01:13] one of the dedi models are blades: http://www.supermicro.com/products/motherboard/xeon/c202_c204/x9scd-f.cfm [01:13] 1 pci-e :) [01:13] ahh [01:14] yea, i was looking at the x9scm-f [01:14] I'm just not sure how well it will do [01:14] the other is: http://www.supermicro.com/products/motherboard/xeon/c202_c204/x9scl-f.cfm [01:14] my current consumer board just isn't cutting it [01:14] so the x9scl-f is pretty shy as well [01:14] up_the_irons: yea, the x9scm has 1 more pcie 4x [01:15] 2x8 and 2x4 in 8 mechanical [01:15] which would work, but i was thinking that might limit bandwidth [01:15] so, looks like you may need to go dual socket [01:15] yea, that's how it seems [01:16] the problem is the price of a dual socket setup is crazy [01:16] why 3x sas with infiniband? i would imagine the infiniband bus would take your disk traffic elsewhere... [01:16] it does [01:16] unless that "elsewhere" is what you're building :) [01:16] the infiniband is connected to a switch [01:16] connected to my desktop and other machines [01:16] local ip routed [01:17] got it [01:17] so you're building a big storage machine [01:17] mhmm [01:17] ? [01:17] cool [01:17] my current consumer board has issues handling multiple sas cards though [01:17] well yeah, consumer [01:18] and that's where the problems start [01:18] i seriously wonder if i need the x8 bandwidth for all cards though [01:18] yeah [01:18] my 8x raid 10 cards are just 4x [01:18] yea [01:18] 8x being 8 drives, not 8x pci-e [01:18] i figure if a dedicate a x8 for the infiniband it should be fine [01:19] yea, ill probably end up taking the x9scm because i don't really need the computing power of two processors [01:20] an e3-1230v2 should be plenty [01:20] that's a nice proc [01:21] yea, it seems like a sweet spot of $/performance [01:21] only problem with those e3s is you can only take 32gb of ram [01:21] it's what we use in the dedi's [01:21] and even then you need 8gb sticks [01:21] yes [01:21] if that's not a problem for you you should be sweet [01:21] that is a huge problem [01:21] well, my biggest problem i should say [01:21] if that's a problem you end up needing to spend quite a bit more money [01:22] exactly [01:22] they actually had a 4x8GB kit for ~$240 [01:22] since most of our dedi's are blades with only 4x slots, isn't too big a deal. we'll figure out something else if we get requests for 64GB ram, etc... [01:22] yup [01:22] up_the_irons: well that will lack harddisk bandwidth anyway? [01:22] i wouldn't have brought it up if he hadn't brought up 3x sas cards and infiniband [01:23] which means a lot of disk badnwidth is likely :) [01:23] mercutio: indeed, only 2x drive bays in the blades, so those who want more get the near identical spec 1U instead [01:23] or i put in an H [01:23] B [01:23] A [01:23] whuops [01:23] ...HBA card into the blades [01:23] up_the_irons: tbh, if i wanted a machine with 64gb ram + i'd prob want to jump to 2u [01:24] mercutio: yeah [01:25] oh webhost.. one thing about those e3s is that you have to flash rom for lsi controllers using that new fangled ipmi bios command line thingy [01:25] if you want to use the other mode thingy [01:25] err use disks direclty [01:25] what? [01:26] like if you want to use zfs or md or such [01:26] ie software raid [01:26] i have to flash the bios of the mobo? [01:26] nah of the card [01:26] if you want direct disks [01:26] rather than "card raid" [01:26] my cards are all flashed into it mode [01:26] oh yip [01:26] they are actually ibm m1015's but they are flashed to 9211-8i it [01:26] oh you done it already [01:26] yeh [01:26] yea [01:27] well if you do it on the e3 boards you have to do it with ipmi [01:27] interesting [01:27] can't do it with freedos? [01:27] it's like DOS [01:27] nope [01:27] shit [01:27] well, i don't have to worry about that much [01:27] it's like DOS but it isn't [01:27] yeh [01:27] you can pre flash [01:27] my consumer board can handle one card at a time if nothing else [01:27] it's not too bad once you get the hang of it [01:27] but it's a huge wtf if you don't knopw ahead [01:28] yea, i actually had to do that on my uefi board before [01:28] oops [01:28] i mean uefi [01:28] not ipmi [01:28] mhmm [01:28] oh so you've done it already :) [01:28] that's what i thought [01:28] the board has ipmi too [01:28] yea, ipmi is pretty awesome [01:28] but i've been finding that the security is horribly subpar [01:29] what did you think of the eufi command line? [01:29] just use ipsec/l2tp? [01:29] it will be on an internel network [01:29] ahh ok [01:29] it's just sad how terrible it is [01:29] of course [01:29] it's at home :) [01:30] yeah [01:30] it will have a public ipv6 address though =p [01:30] *** HighJinx has quit IRC (Ping timeout: 244 seconds) [01:30] i was reading about this ethernet switch vulnberability [01:30] ipmi security is horrible, we don't even offer it for our dedi's outside of our new customer vpn: http://support.arpnetworks.com/kb/main/customer-vpn [01:30] granted, i won't allow traffic through the gateway pointed at it [01:30] and it had a bypass password [01:30] and it was an industrial one [01:30] and it's liek [01:30] people never want to update industrial things [01:31] exactly [01:31] that's why i prefer dumb switches [01:31] but like who is accountable? [01:31] dunno [01:32] are there any dumb level 3 switches?> [01:32] level 3? [01:32] *** ryk has joined #arpnetworks [01:32] layer 3 [01:32] yea [01:32] what i meant [01:32] isn't layer 3 not dumb? [01:32] i would say layer 3 implies some sort of management [01:33] *** HighJinx has joined #arpnetworks [01:33] well, you could still make it so there isn't a management interface [01:33] you mean serial port only? [01:33] something not tied to the ip network [01:33] you could just disable ssh/telnet? [01:33] but then what would be the point? [01:33] well if you stuck all the ports on vlans [01:33] then nothing can reach management [01:33] it at least prevents attacks at a layer 2 level [01:34] mercutio: that's true [01:34] i've never heard of a vlan escalation in a cisco switch [01:34] and in normal use people tend to use vlans on expensive switches [01:35] even if just to justify the cost of the switch [01:35] lol [01:35] it sucks that going dual socket is going to double the price of the parts i need [01:35] i used to say that hubs were faster than switches [01:36] but then i realised it was pointless [01:36] hahaha [01:36] well they do cut-through forwarding normally [01:36] so they have lower latency [01:36] which matters more at 100 megabit than faster speeds [01:37] but like switch latency has got a lot better on cheap swithces [01:37] and is often worse on more expensive ones [01:37] cos of all the fancy stuff [01:38] i waited a little while to see if anyone would come out with a home user cut-through switch for home users [01:38] but it seems that is never likely to happen [01:38] what do you mean cut-through/ [01:38] http://en.wikipedia.org/wiki/Cut-through_switching [01:39] apparently infiniband often uses it [01:39] basically you wait for the source/destination to come through [01:39] then you start forwarding the packet [01:39] rather than wait the whole packet to come then send to the destination [01:39] oh [01:39] i see [01:40] so hubs used to operate like that anyway [01:40] also intel ethernet cards used to cheat [01:40] which made them get higher speeds on network [01:41] by reducing interpacket gap i think it was [01:41] but now ethernet is pretty stagnant [01:41] yea [01:41] i've had the same ethernet stuff for years [01:41] i had lower latency 10 years ago [01:41] not surprising [01:42] like 0.060 msec? [01:42] now you're lucky to get under .1 msec [01:42] mhmm [01:42] most of the time im sitting at 0.2ms [01:42] what kind of switch? [01:42] netgear gigabit [01:43] layer 2 [01:43] the cheap managed one? [01:43] some cheap 5 port deal [01:43] nope [01:43] oh [01:43] my d-link is pretty good for latency [01:43] i actually have the cheap managed one at home [01:43] the cheap netgear managed are a bit worse [01:43] yeh [01:43] yes they are [01:43] linux or windows? [01:43] or something else? [01:43] *nix [01:44] generally [01:44] solaris? [01:44] who says *nix [01:44] got a freebsd storage server [01:44] heh :) [01:44] openindiana kvm box [01:44] linux desktops [01:44] ahh [01:44] so the whole spectrum [01:44] openindiana needs tuning to lower latency [01:44] i dunno about freebsd [01:44] oh yea [01:44] linux isn't too bad [01:44] the oi box is mostly testing [01:44] e1000g0 is easy to tune [01:45] i just wanted to play around on oi [01:45] oh yip [01:45] how are you finding it? [01:45] im not sure i really like it at this point [01:45] heh [01:45] but then again, i never really liked the solaris userland much [01:45] that's kind of how i feel about it [01:45] but then i consider alternatives [01:45] the userland is what sucks [01:45] and i'm like "ARGH!" [01:45] did you find /usr/gnu/bin/dd? [01:45] the userland drives me nuts [01:45] yeh but you have kvm [01:45] yea, i know about the gnu part [01:45] what do you think of smf? [01:46] ehhh [01:46] i think i like systemd more [01:46] honestly [01:46] i haven't tried systemd [01:46] i fucking hate xml [01:46] i hate xml too :) [01:46] so smf is always just meh [01:46] i'm using openindiana/xen [01:46] and xen isn't even supported any more [01:46] but it works [01:47] i really need to try out stormos [01:47] my core2duo can't run kvm on openindiana [01:47] and tbh i like xm console [01:47] it might not be as frustrating [01:47] i want to try uhh [01:47] yea, im not a xen guy [01:47] smartos? [01:47] is it called [01:47] well, it's really just stripped down illumos [01:47] i like kvm cos it run openbsd [01:47] really? [01:48] but i don't like havikng to use vnc [01:48] oh [01:48] use spice [01:48] spice? [01:48] http://spice-space.org/ [01:49] it's actually pretty nice even though it is only beta quality [01:49] is there a client for windopws? [01:49] yes [01:49] cool [01:49] does it work on openindiana? [01:49] i have no idea if you can compile this on the illumos qemu-kvm though [01:49] oh [01:50] because it compiles into the package [01:50] openindiana had another release recently [01:50] did it? [01:50] they're not moving very fast [01:50] yeh not much updated [01:50] 151a6 [01:50] that's a shame [01:50] what version are you on ? [01:50] 151a? [01:50] a5 [01:50] something? [01:50] oh wow [01:50] that's quite up to date [01:51] im always up to date with most of my machines [01:51] i think i jumped from a2 to a5 [01:51] Bump illumos to 13793:10c3656ccf76 [01:51] don't you love their release notes [01:51] like wtf does that mean [01:51] Bump KVM driver [01:51] Bump NVIDIA driver to 295.71 [01:51] mhmm [01:51] Bump qemu-kvm [01:51] well they'd doing stuff with kvm but they don't say what [01:52] woah, they have gcc 4.4.4 [01:52] i didn't even realize that [01:52] that's not that recent? [01:52] but yeh non gcc3 :) [01:52] well, better than gcc3 [01:52] yea [01:53] you'd think they'd go to 4.6.1 or soemthing [01:53] does the whole system build with 4.4.4 though? [01:53] do you remember when redhat got "told off" for using a cvs compiler? [01:53] i don't think so [01:53] i think it was egcs [01:53] but like now ubuntu is bundling a prerelease compiler [01:54] gcc version 4.7.1 20120814 (prerelease) (Ubuntu/Linaro 4.7.1-7ubuntu1) [01:54] well in quantal [01:54] im using arch [01:54] i have a arch host too [01:54] pretty much always have a prelease compiler [01:54] i have't shifted to systemd yet [01:54] gcc version 4.7.1 20120721 (prerelease) (GCC) [01:54] second is arch [01:54] yep [01:55] have you noticed how much faster pacman -Sy is than apt-get update [01:55] pacman blazes [01:55] especially in the 3.4 to 3.5 bump [01:55] it's like 5x faster than apt-get at installing things [01:55] when they added the ram caching [01:55] it's one of the things i notice most often about it [01:56] see, i don't run any debian or rpm systems [01:56] i have one problem with arch though [01:56] just arch [01:56] what's that? [01:56] oh god openindiana is slow package management [01:56] umm [01:56] i can't have seperate /usr easily [01:56] it's really complicated compared to ubuntu [01:56] separate? [01:56] oh [01:56] yeh [01:56] like another file system [01:56] they are trying to fix that [01:56] cos i've been experimenting with /usr on btrfs with compression=lzo [01:57] on ubuntu it seems to make the system feel faster [01:57] and i figure that it's lower risk than data on btrfs :) [01:57] i've pretty much ditched btrfs [01:57] how come? [01:57] kernel panics [01:57] real? [01:57] all sorts of weird shit [01:57] i've never had any panics from it? [01:57] i use recent kernels though [01:57] most people haven't [01:58] do you have bad ram? [01:58] i just updated to kernel 3.5.1 and boom [01:58] oh [01:58] panics every time i write a lot of data [01:58] i don't think i used 3.5 much [01:58] err kernel 3.5.3 [01:58] i'm using 3.6.0-rc3 [01:58] so i just ditched it an did ext4 [01:58] and rc4 [01:58] it's so much faster on my ssd than btrfs was [01:58] but before that i was using like 3.3 ? [01:59] really? [01:59] oh yea [01:59] i wonder why [01:59] me too [01:59] i even had all of the ssd optimizations set up [01:59] i want to figure out how to get compression on my cellphone easiest [01:59] linux seems fast even on my slowest ssd [01:59] that sounds like a challenge [02:00] my cellphone only has 512mb internal memory [02:00] well, i got a 256gb m4 for my windows machine and my linux machines on hdd's boot faster [02:00] i managed to compile a kernel for it [02:00] im not even starting much [02:00] what kind of phone? [02:00] my windows boots faster than linux [02:00] lg p500 [02:00] oh [02:01] but i think that's partially changing video moed [02:01] *** CaZe has quit IRC (Remote host closed the connection) [02:01] mode [02:01] mercutio: you also don't use systemd =p [02:01] *** CaZe` has joined #arpnetworks [02:01] linux has always been slow at changing video mode [02:01] oh i'm using ubuntu for desktop [02:01] arch on my router [02:01] *** CaZe` is now known as CaZe [02:01] i mostly use windows atm though [02:01] same =( [02:01] bootchart is screwed in ubuntu for ages [02:01] i don't know how fast it's booting [02:01] hmm [02:01] maybe i should install arch linux [02:02] i like ubuntu for desktop cos all the packages are there [02:02] well, what de or wm do you use? [02:02] what packages is arch missing? [02:02] but i hate it for having all this crap everywehre i don't want [02:02] notion [02:03] umm package it's missing umm [02:03] i can't remember [02:03] i just remember eeven on my router every now and then it doesn't have something [02:03] it doesn't have apache traffic server for instance [02:03] it has notion [02:03] actually i hardly use anything [02:04] i see [02:04]