up_the_irons: any suggestions for server grade boards with many pcie lanes?
im looking for something at home that is capable of supporting 3xsas cards and an infiniband card
Webhostbudd: intel or amd?
most likely intel, but amd might be an option
i was thinking intel because i may end up using it as a hypervisor, and freebsd / illumos have better intel than amd support
Webhostbudd: the intel boards we use are a bit shy on pci-e (we don't need much), but one amd board we're using has a lot: http://www.supermicro.com/aplus/motherboard/opteron6000/sr56x0/h8dgi-f.cfm
i see
im guessing you are using x<>scm boards?
oh nice
you get 3 16x to start, then 1 8x, and 2 4x
it's pretty loaded
yea, i was browsing around on supermicro's site trying to find the best deal
it seems as though you need dual socket to get a decent amount of lanes
we only use dual socket, so i wouldn't know about the single sockets
oh, alright
what about the new dedi's?
oh, well, yeah those are single
i see
and the kvm hosts are all dual socket?
the dedi's don't have shit for pci-e ;)
web: impressive home setup :)
20:08:20   Webhostbudd   im looking for something at home that is capable of supporting 3xsas cards and an infiniband card                                                     ???gneous
thanks
i can't say i've heard of anyone else wanting such high spec for home :)
one of the dedi models are blades: http://www.supermicro.com/products/motherboard/xeon/c202_c204/x9scd-f.cfm
1 pci-e :)
ahh
yea, i was looking at the x9scm-f
I'm just not sure how well it will do
the other is: http://www.supermicro.com/products/motherboard/xeon/c202_c204/x9scl-f.cfm
my current consumer board just isn't cutting it
so the x9scl-f is pretty shy as well
up_the_irons: yea, the x9scm has 1 more pcie 4x
2x8 and 2x4 in 8 mechanical
which would work, but i was thinking that might limit bandwidth
so, looks like you may need to go dual socket
yea, that's how it seems
the problem is the price of a dual socket setup is crazy
why 3x sas with infiniband?  i would imagine the infiniband bus would take your disk traffic elsewhere...
it does
unless that "elsewhere" is what you're building :)
the infiniband is connected to a switch
connected to my desktop and other machines
local ip routed
got it
so you're building a big storage machine
mhmm
?
cool
my current consumer board has issues handling multiple sas cards though
well yeah, consumer
and that's where the problems start
i seriously wonder if i need the x8 bandwidth for all cards though
yeah
my 8x raid 10 cards are just 4x
yea
8x being 8 drives, not 8x pci-e
i figure if a dedicate a x8 for the infiniband it should be fine
yea, ill probably end up taking the x9scm because i don't really need the computing power of two processors
an e3-1230v2 should be plenty
that's a nice proc
yea, it seems like a sweet spot of $/performance
only problem with those e3s is you can only take 32gb of ram
it's what we use in the dedi's
and even then you need 8gb sticks
yes
if that's not a problem for you you should be sweet
that is a huge problem
well, my biggest problem i should say
if that's a problem you end up needing to spend quite a bit more money
exactly
they actually had a 4x8GB kit for ~$240
since most of our dedi's are blades with only 4x slots, isn't too big a deal.  we'll figure out something else if we get requests for 64GB ram, etc...
yup
up_the_irons: well that will lack harddisk bandwidth anyway?
i wouldn't have brought it up if he hadn't brought up 3x sas cards and infiniband
which means a lot of disk badnwidth is likely :)
mercutio: indeed, only 2x drive bays in the blades, so those who want more get the near identical spec 1U instead
or i put in an H
B
A
whuops
...HBA card into the blades
up_the_irons: tbh, if i wanted a machine with 64gb ram + i'd prob want to jump to 2u
mercutio: yeah
oh webhost.. one thing about those e3s is that you have to flash rom for lsi controllers using that new fangled ipmi bios command line thingy
if you want to use the other mode thingy
err use disks direclty
what?
like if you want to use zfs or md or such
ie software raid
i have to flash the bios of the mobo?
nah of the card
if you want direct disks
rather than "card raid"
my cards are all flashed into it mode
oh yip
they are actually ibm m1015's but they are flashed to 9211-8i it
oh you done it already
yeh
yea
well if you do it on the e3 boards you have to do it with ipmi
interesting
can't do it with freedos?
it's like DOS
nope
shit
well, i don't have to worry about that much
it's like DOS but it isn't
yeh
you can pre flash
my consumer board can handle one card at a time if nothing else
it's not too bad once you get the hang of it
but it's a huge wtf if you don't knopw ahead
yea, i actually had to do that on my uefi board before
oops
i mean uefi
not ipmi
mhmm
oh so you've done it already :)
that's what i thought
the board has ipmi too
yea, ipmi is pretty awesome
but i've been finding that the security is horribly subpar
what did you think of the eufi command line?
just use ipsec/l2tp?
it will be on an internel network
ahh ok
it's just sad how terrible it is
of course
it's at home :)
yeah
it will have a public ipv6 address though =p
i was reading about this ethernet switch vulnberability
ipmi security is horrible, we don't even offer it for our dedi's outside of our new customer vpn: http://support.arpnetworks.com/kb/main/customer-vpn
granted, i won't allow traffic through the gateway pointed at it
and it had a bypass password
and it was an industrial one
and it's liek
people never want to update industrial things
exactly
that's why i prefer dumb switches
but like who is accountable?
dunno
are there any dumb level 3 switches?>
level 3?
layer 3
yea
what i meant
isn't layer 3 not dumb?
i would say layer 3 implies some sort of management
well, you could still make it so there isn't a management interface
you mean serial port only?
something not tied to the ip network
you could just disable ssh/telnet?
but then what would be the point?
well if you stuck all the ports on vlans
then nothing can reach management
it at least prevents attacks at a layer 2 level
mercutio: that's true
i've never heard of a vlan escalation in a cisco switch
and in normal use people tend to use vlans on expensive switches
even if just to justify the cost of the switch
lol
it sucks that going dual socket is going to double the price of the parts i need
i used to say that hubs were faster than switches
but then i realised it was pointless
hahaha
well they do cut-through forwarding normally
so they have lower latency
which matters more at 100 megabit than faster speeds
but like switch latency has got a lot better on cheap swithces
and is often worse on more expensive ones
cos of all the fancy stuff
i waited a little while to see if anyone would come out with a home user cut-through switch for home users
but it seems that is never likely to happen
what do you mean cut-through/
http://en.wikipedia.org/wiki/Cut-through_switching
apparently infiniband often uses it
basically you wait for the source/destination to come through
then you start forwarding the packet
rather than wait the whole packet to come then send to the destination
oh
i see
so hubs used to operate like that anyway
also intel ethernet cards used to cheat
which made them get higher speeds on network
by reducing interpacket gap i think it was
but now ethernet is pretty stagnant
yea
i've had the same ethernet stuff for years
i had lower latency 10 years ago
not surprising
like 0.060 msec?
now you're lucky to get under .1 msec
mhmm
most of the time im sitting at 0.2ms
what kind of switch?
netgear gigabit
layer 2
the cheap managed one?
some cheap 5 port deal
nope
oh
my d-link is pretty good for latency
i actually have the cheap managed one at home
the cheap netgear managed are a bit worse
yeh
yes they are
linux or windows?
or something else?
*nix
generally
solaris?
who says *nix
got a freebsd storage server
heh :)
openindiana kvm box
linux desktops
ahh
so the whole spectrum
openindiana needs tuning to lower latency
i dunno about freebsd
oh yea
linux isn't too bad
the oi box is mostly testing
e1000g0 is easy to tune
i just wanted to play around on oi
oh yip
how are you finding it?
im not sure i really like it at this point
heh
but then again, i never really liked the solaris userland much
that's kind of how i feel about it
but then i consider alternatives
the userland is what sucks
and i'm like "ARGH!"
did you find /usr/gnu/bin/dd?
the userland drives me nuts
yeh but you have kvm
yea, i know about the gnu part
what do you think of smf?
ehhh
i think i like systemd more
honestly
i haven't tried systemd
i fucking hate xml
i hate xml too :)
so smf is always just meh
i'm using openindiana/xen
and xen isn't even supported any more
but it works
i really need to try out stormos
my core2duo can't run kvm on openindiana
and tbh i like xm console
it might not be as frustrating
i want to try uhh
yea, im not a xen guy
smartos?
is it called
well, it's really just stripped down illumos
i like kvm cos it run openbsd
really?
but i don't like havikng to use vnc
oh
use spice
spice?
http://spice-space.org/
it's actually pretty nice even though it is only beta quality
is there a client for windopws?
yes
cool
does it work on openindiana?
i have no idea if you can compile this on the illumos qemu-kvm though
oh
because it compiles into the package
openindiana had another release recently
did it?
they're not moving very fast
yeh not much updated
151a6
that's a shame
what version are you on ?
151a?
a5
something?
oh wow
that's quite up to date
im always up to date with most of my machines
i think i jumped from a2 to a5
Bump illumos to 13793:10c3656ccf76
don't you love their release notes
like wtf does that mean
Bump KVM driver
Bump NVIDIA driver to 295.71
mhmm
Bump qemu-kvm
well they'd doing stuff with kvm but they don't say what
woah, they have gcc 4.4.4
i didn't even realize that
that's not that recent?
but yeh non gcc3 :)
well, better than gcc3
yea
you'd think they'd go to 4.6.1 or soemthing
does the whole system build with 4.4.4 though?
do you remember when redhat got "told off" for using a cvs compiler?
i don't think so
i think it was egcs
but like now ubuntu is bundling a prerelease compiler
gcc version 4.7.1 20120814 (prerelease) (Ubuntu/Linaro 4.7.1-7ubuntu1)
well in quantal
im using arch
i have a arch host too
pretty much always have a prelease compiler
i have't shifted to systemd yet
gcc version 4.7.1 20120721 (prerelease) (GCC)
second is arch
yep
have you noticed how much faster pacman -Sy is than apt-get update
pacman blazes
especially in the 3.4 to 3.5 bump
it's like 5x faster than apt-get at installing things
when they added the ram caching
it's one of the things i notice most often about it
see, i don't run any debian or rpm systems
i have one problem with arch though
just arch
what's that?
oh god openindiana is slow package management
umm
i can't have seperate /usr easily
it's really complicated compared to ubuntu
separate?
oh
yeh
like another file system
they are trying to fix that
cos i've been experimenting with /usr on btrfs with compression=lzo
on ubuntu it seems to make the system feel faster
and i figure that it's lower risk than data on btrfs :)
i've pretty much ditched btrfs
how come?
kernel panics
real?
all sorts of weird shit
i've never had any panics from it?
i use recent kernels though
most people haven't
do you have bad ram?
i just updated to kernel 3.5.1 and boom
oh
panics every time i write a lot of data
i don't think i used 3.5 much
err kernel 3.5.3
i'm using 3.6.0-rc3
so i just ditched it an did ext4
and rc4
it's so much faster on my ssd than btrfs was
but before that i was using like 3.3 ?
really?
oh yea
i wonder why
me too
i even had all of the ssd optimizations set up
i want to figure out how to get compression on my cellphone easiest
linux seems fast even on my slowest ssd
that sounds like a challenge
my cellphone only has 512mb internal memory
well, i got a 256gb m4 for my windows machine and my linux machines on hdd's boot faster
i managed to compile a kernel for it
im not even starting much
what kind of phone?
my windows boots faster than linux
lg p500
oh
but i think that's partially changing video moed
mode
mercutio: you also don't use systemd =p
linux has always been slow at changing video mode
oh i'm using ubuntu for desktop
arch on my router
i mostly use windows atm though
same =(
bootchart is screwed in ubuntu for ages
i don't know how fast it's booting
hmm
maybe i should install arch linux
i like ubuntu for desktop cos all the packages are there
well, what de or wm do you use?
what packages is arch missing?
but i hate it for having all this crap everywehre i don't want
notion
umm package it's missing umm
i can't remember
i just remember eeven on my router every now and then it doesn't have something
it doesn't have apache traffic server for instance
it has notion
actually i hardly use anything
i see
i mostly just use terminal/window manager/chrome
and /ssh
the aur has everything though
i tried using it in vmware under windows for a while
then i upgraded and X broke
lol
my router is fine on the current type system
oh maybe that's fixed now
i kind of want fast virtualisation under windows of linux base dektop
or unix
cos i like notion
and a rapid/fast environment
but i want to be able to play games without rebooting
doesn't virtualbox do that?
virtualbox has always been fast for me
mm
vmware kind of worked
i wonder what virtualbox is like now
virtualbox has been great for arch guests
vmware's got thie annoying bar up the top of the screen
i seem to have virtualbox installed and smartos and openbsd
lol
openbsd seems to only do 1024x768
oh are you using arch kernel?
the default one?
yes
wow real recent relase
for iso
downloading now
9-7
yeh it's 9/9 here
and it's dated 9/7
mhmm
they release very frequently
i'm +12 GMT
i wish ati had open source drivers that worked with current video cards
they used to do yearly images, but since AIF is dead they release often
all of my ati stuff is semi old
so i have no issues
have you done an arch install recently?
it doens't ilke my iso for some reason
nope
i can't even remember how i did arch in xen
the new installer or lack thereof is awesome
bah i need coffee
trying newer virtualbox
dunno why it'd not like my iso
but i'm using windows 8 for desktop...
weird
and im not really sure how im going to put up with windows 8
but i kinda want smb 3.0
and other goodies
i dont' like window s8 much
but i tend to not like things
i like openbsd for what it does
windows 8 i'm using a start replacement
and the start replacement isnt' that good
wtf
windows 8 won't start it cos it's not compatible
even if i try starting it with windows 7 compatibility mode
lol
oh it didn't donwload the whole way
no error or aynthing
same thing happened again
but even less data
it was downlopading slower too akamai prob maybe
cos the other version downloaded faster and whole way
oh wtf
arch download is small too
i can't believe chrome doesn't even detect incomplete downloads
chrome's downloader is terrible
i've had better results on firefox and even IE
well i downloaded with curl on my router with arch and it got the file fine
i'm stil curious how that can even happen
when i loked at external download programs they were pretty disgusting
getright seemed to be best
but uhh ick
hah ok that's better
no weird error message just a shell prompt
not too bad
nice that it has zsh
not so nice it doesn't have vim
i know....
also you have to google how to install it :)
nah
vi install.txt
done
and to be honest the only real utils you may not know are pacstrap, arch-chroot, genfstab
the rest you probably already know
anyway
night
it should tell you that
didn't know the grub config file generator
doesn't it say to open install.txt
which didn't work properly aynway
nope
oh
lol
i feel like it should print that at startup
i can't screen cap
yes!
it says automatic login
root@archiso
anyway, im out
ahh this is like web page i found
ok
woohoo, got near perfect ipv6 setup in my home
probably the only good thing about having comcast
U-Verse assigns me an ipv6 but it can't ping anything
its weird
sounds gay
yeah
I might set up a local ipv6 tunnel
that'd be kind of cool
it's just not worth it
any service works on ipv4 these days
and the latency of ipv4 will almost certainly be lower
i would just wait for real ipv6 support from the isp
yeah
even with real ipv6 pings can be higher
Wish my ISP provided IPv6.
heh
would like an ISP that provides real ipv6
not like... a /128 :|
I mean, you can do anything with a /128 ...
Just gotta get something to route to it.
Google seems to be waging war against HE.net tunnels.
I keep getting a "suspicious activity" CAPTCHA across all the devices on my network.
I think their reputation is... somewhat checkered, no?
For sure, but it's the only way I can get IPv6 at home.
lucky no one actually _needs_ IPv6 :p
I _need_ it to connect to my webserver over IPv6. :p
but your webserver has an ipv4 address?
mhoran: Tried SixXS?
whoa. creepy!  Just worked though my first OpenBSD IPv6 + IPv4 dual-stack setup. AND #arpnetworks IS TALKING ABOUT IPV6!!!1
incidentally, is my arpnetworks ipv6 being tunneled off somewhere via tunnel provider or is arpnets au natural ipv6? just wondering.
native
well, that's what garry says anyway
but seeing as i get ~1-10ms to google i would assume it is, unless the tunnel broker is located in the same building, at which point it really wouldn't matter anyway
cool!
as long as i stay inside my purchased bandwidth, i suppose i can tunnel my own ipv6 though arpnet
or just call and complain to my residential ISP
My DSL's been down for three days.
I've been tethering throuhg my phone.
My throughput is almost four times faster.
An extra 200 ms latency though.
heh
mikeputnam: it's native
it's actually been faster in my ping6 testing
i just checked the web sites of every company i've worked for since college and not one of them is ipv6. but our hacker/makerspace web site is by golly.
shocking.
a few (very) major sites on ipv6
and enthusiasts
and very little in between
yup
no one really gives a crap right now
ipv4 works for them
plus, it will be years before consumers even upgrade their hardware to be ipv6 compatible
"average consumers"