brycec: 1) Why eat up bandwidth? 2) Wanted to be sure it was GbE through-and-through, 3) I forgot there was IPv4 access. mnathani: Plus jumbo frames on the vlan mercutio: oh there's jumbo frames on the vlan?
jumbo frames don't actuually make much real world difference ime mnathani: If you get a second nic on your vps mercutio: it can use less cpu on some old systems
i can't believe how much i'm struggling to stick intel stock cpu cooler on brycec: (and new ones, depending on bw/pps usage) mercutio: it's not like i've never done it before brycec: Me either. They're pretty easy. mercutio: and i never remembered it being a big deal before brycec: Are you sure you've done everything else correctly? CPU latched properly? mercutio: found the issue
one of the bits that goes in is "to it's side
can't see how cpu couldn't be latched properly brycec: There was a time I would have said that too.
Never, EVER underestimate human ingenuity when it comes to screwing up the idiot-proof.
They forced it shut with the cpu not properly seated and actually bent the CPU
And after that day, management understood why I referred to the production workers as monkeys.
(well they understood before, but that really illustrated it) -: brycec does not feel like delving into SASL setup for LDAP binding auth brycec: But everything else on this Prosody server is looking good, woo
Including the most important https://code.google.com/p/prosody-modules/wiki/mod_swedishchef mercutio: finally got it
swapped to another cpu cooler which "just worked"
so then i plug in computer and nothing comes up :/
took me a while to realise that the monitor wasn't auto selecting dvi
i hate hardware :)
and monitors seem to hate me atm brycec: Go install, go! (reinstalling my dedi machine with raid-1 and zfs root)
(zpool mirror, that is) mercutio: hah
freebsd or linux? brycec: Linux, just for fun (and because I like Proxmox) mercutio: i bet there are systems out there already, but i want to see how fast i can make an install happen on bare hardware over the network
ie the right of compression file system extraction :)
err mix of compression and file system extraction
lz4 has just introduced new faster compression, again.
it's not altogether exciting - you can sacrifice some compression ratio for "even faster" performance.
on that note, on zfs i find lz4 works really well.
and i'd suggest using it :) brycec: I really, REALLY, REEEAALLY wish the java console viewer would stop stealing focus every dang time the video mode changes. Getting REALLY FUCKING PISSED-OFF. *deep breaths* mercutio: haha god
supermicro gear may be ok
but their out of band management really sucks. brycec: it needs to die, in a fire. mercutio: i dunno if it's changed, but in the past supermicro have had this remote iso functionality, and it doesn't even use decent sized window sizes
so it's painfully slow.
err painfully slow if you have like 20msec+ ping.
it's probabl fine for < 1 msec.
i think it was using 16k or something
hmm, i ahve to figuure out how to make arch linux stop setting graphics vga mode at some point
actually i wonder if there's a way to get arch to do serial console for rescue system brycec: https://wiki.archlinux.org/index.php/Working_with_the_serial_console mercutio: yeah that's not about the "rescue" cd thogh brycec: oh I thought you meant rescue.target mercutio: i hosed my file system by doing a mv /* or something :)
oh what's rescuue.target? brycec: A systemd target for doing rescue stuff
(not helpful I know) mercutio: hmm brycec: systemd.unit=rescue.target on the kernel line mercutio: yeah it's not common, and ie worked :/ brycec: aka run level 1 mercutio: hp's lights out pisses me off too :)0
like it's cool yo can type textcons and don't even need serial console setup to get remote text mode
err via ssh
and vsp to get a serial console
but if things are in graphics mode you have to use java or activex
and java seems to keep giving issues :/
i don't use it that much though brycec: I keep forgetting IPv6 traffic is limited to 100mbps and then wondering why I'm only getting ~7MB/s to mirrors.arpnetworks.com mercutio: yeah it bugs me too :)
damn i was curiouus how the Ubuntu LTS kernels work, and i try and read up about it and i'm even more confused.
it seems ubuntu now by default installs newer kernels for patch releases.
but i dunno how long support for these kernels lasts for brycec: Wheee 75MB/s off S3 mercutio: wow
how close is the s3 server? brycec: LAX - North California
(I'll run a traceroute in a bit)
but it's s3-us-west-1.amazonaws.com I believe mercutio: so it seems precise can be upgraded to trusty kernel and be supported until 2017 brycec: (Don't look at me - I avoid Ubuntu wherever possible :P) mercutio: heh
i prefer self-compiled kernels brycec: Woo, booting OpenBSD off a zdev
(via kvm) mercutio: zvol? brycec: zvol.
block storage carved out of a zpool mercutio: heh yeh pretty awesome
what volblocksize did you use? brycec: whatever Proxmox defaulted to. I didn't even think to look mercutio: erk :/
there can be a bit of overhead, so raising it above the 8k can help
but it can mean read/write
but now that everyone seems to have 4k disks, zfs overhead stuff is getting a little scary
the mirror case isn't as bad as raidz though afaik brycec: (It's using the ZFS default of 8k) mercutio: but yeah i've started using bigger block sizes. also using lz4 compression thouugh
yeah
if you're bored sometime you can try playing with it :)
does proxmox make it easy to do autosnapshotting? brycec: Don't yet know. (FreeNAS sure does though, very nice) mercutio: yeah it is nice
can use up a lot of space, but so handy :)
you can expose the snapshot directory on real flie systems
err when using native zfs as opposed to zvol i mean brycec: yeah
or just cd to the directory that was hidden there anyways :p mercutio: heh
tab completion is nice :/ brycec: aw dang 20GB written before I realized it's using lzjb rather than lz4. Oh well, not a big deal... mercutio: damn
yeh i used to use lzjb with opensolaris
lz4 is a little quicker
but both are way quicker than gzip brycec: Nothing wrong with lzjb, still got 1.26x from it even, but lz4 is better mercutio: hmm i wonder what my ratio is like
refcompressratio 1.10x
that's for /home which probably has some big tarballs on it somewhjere brycec: rpool/vm-201-disk-1 1.96x mercutio: the cool thing abouut lz4 is it's so cheap it doesn't matter if you have bulky stuff on it brycec: for the VM I just restored from backup :) (it's pretty empty though)
aye' mercutio: it's even cheaper for stuff tha tdoesn't compress
1.41x for /hoem on the box i'm irc'ing from brycec: My best ratio is 2.47x for a mysql database volume, and 2.26 for a fat postgresql volume mercutio: but yeah again with 4k disks, lz4 if it doesn't get under 4k will use 8k still with 8k volblocksize
it doesn't condense :(
now youu justu need a ssd cache :)
may have to be pci-e :) brycec: I have an SSD cache, and a nasty FreeBSD bug that causes my host machine to crash :P mercutio: i'm mostly kidding, you probably don't do many reads brycec: (that's my home box, that is) mercutio: heh
i've bewen using l2arc only for metadata brycec: Yes o/ Just realized my favourite feature with using ZFS on Proxmox -- I don't have to manually format+mount anything, "zfs create" does it all. mercutio: it gives me most of the boost i care about brycec: Nice
I have a few spare 60GB SSD's so I just threw them at it. mercutio: it means you ls in a directory with heaps of files and it doesn't delay
yeah
by default it won't do much for sequential anyway
and if you have "plenty" of ram then all the important stuff will be in memory anyway
and if you use zvol's etc youu're likely to get into double caching brycec: (oh many I forgot how nice zfs set quota= is too... It's been too long) mercutio: not that double caching is necessarily bad, but it doesn't alert zfs for most frequently used.
and the most recently used is "old" data.
i wonder if you could get linux to cache less
are you using zil bryce?
that i think can help more... brycec: Not on this proxmox box, no
But I do on my home system
(mirrored, no less) mercutio: cool
yeh i added another ssd to a hard-disk server recently
going to look at setting up both as l2arc and some zfs pool first
but considering trying zil brycec: hard to go wrong :P Besides, ZIL/L2ARC can always be added/removed any time in the life of the pool so there's no cost to trying it out.
(assuming your pool is "fixed" and you're just adding/removing disks or ssds) mercutio: yeah
and you only need like a gig
i've made my cache drive way too big hah
atm it's 80gb
using 60gb
wow it's had more reads than all of the other drives together though brycec: "WARNING: MD5 signatures do not match:" Dammit Amazon, stop running up my S3 bill. mercutio: why are using s3? :/ brycec: Because I needed somewhere to stash 150GB briefly.
(and cheaply, with a fast connection) mercutio: ahh
so cos temp
i could have stored 150gb for you :/
it's cool that you can even do that easily these days. brycec: Heh, thanks for the late offer mercutio: heh i didn't know you needed some temp space. ;) brycec: It's very cool. With a fast enough connection, storage is completely elastic. mercutio: even with vdsl i'm using offsite storage more nad more
lots of things aren't really performance sensitive brycec: I'm probably going to continue using it once I pare down what I sync to it as another offsite storage location. mercutio: i backed up all of my important home stuff remotely.
took like 24 hours or so :)
with 9.5 megabit upload. brycec: I've been backing up personal stuff for S3 for ages (thanks to duplicity). Cheap, fast, reliable, easily encrpyted. mercutio: but as long as you set up aqm, it doesn't impact other stuff too badly. brycec: But those backups don't even break 100GB
heh mercutio: my home directory volume is only using 98gb brycec: (and this dedi box is 9ms and 14 hops from S3) mercutio: heh
my dediacted box was 5 msec ping from me :)
damn interleaving beign on now
i have a personal dedicated server in nz
with zfs etc. brycec: wow, 5ms? that's practically beside you mercutio: well it's like 30km from me brycec: my first hop past my home router is 12-16ms mercutio: well it terminates on the same lan as my internet connection too
so it's like single hop away :)
mine's uhh 12 or 13 msec now, due to 8 msec downstream interleaving
but yeah it's not laggy at all :) -: brycec is always entertained watching RAID-1 divvy up reads between disks. Just something fun watching it in iostat. mercutio: yeah it doesn't work very well with hard-disks ime
but it works well with ssd's brycec: seems to work well enough
Frankly with the speed of SSD's the improvement is less than the improvement in read performance seen with hdd's mercutio: do you ever look at zpol iostat -v 1 brycec: watching it now :) mercutio: ame haha
my second ssd cache only has 57.4mb allocated
but it's still doing reads for some reason.
hmm so much for l2arc being a waste of time :)
the other ssd seems to actually do quite a lot of requests.
probably means i need more ram in there :)
it does more reads then all the hard-disks, but less writes.
but less writes than any of the hard-disks
i suppose 60gb of ram isn't cheawp
i'm semi tempted to try this zil thing brycec: 'night mercutio mercutio: it must be like 3:30 am for you
'night :) brycec: precisely right mercutio: 'night RandalSchwartz: awesome... got icinga for floss weekly mercutio: icinga?
ahh a monitoring system
graphite sounds nice RandalSchwartz: it's a hostile fork of nagios
I've decided those are called "pitchforks" :) mercutio: haha i liek the name
pitchfork that is
i've been wanting to do some kind of real time web ping thing
so i'm hoping graphite will make that easier. ***: tabthorpe has joined #arpnetworks RandalSchwartz: yeah
a fork that is hostile is a pitchfork
hopefully my show will set the meme ***: RandalSchwartz has quit IRC (Remote host closed the connection) brycec: I wonder what makes it a "hostile" fork mike-burns: Is that to differentiate against a GitHub fork? plett_: mike-burns: Normally that the old project is still in active development and doesn't want to split its userbase with the fork
And yes. The github workflow actively promotes forking to deploy your fix and request that the original version pull in your patch ***: RandalSchwartz has joined #arpnetworks plett_: With the implicit assumption that if you write new patches and the original developer(s) do nothing and don't pull them in, you become the defacto standard version ***: mkb has quit IRC (Remote host closed the connection)
mkb has joined #arpnetworks mkb: reinstall to 5.7 was easy enough brycec: Excellenty
*-y mkb: siteXX.tgz makes things so easy brycec: It does :) ***: mkb has quit IRC (Remote host closed the connection)
mkb has joined #arpnetworks -: brycec is pulling 99MB/s off AWS, wheeee brycec: I could get used to this GbE connection... mercutio: heh
bryce is hogging all the bandwidth :) brycec: lol, am not :P (Because ARP has a bigger pipe than just 1Gbps) mercutio: yeh i know
i like the idea of gigabit for home users. brycec: GbE on a LAN is old hat to me, but getting GbE over THE INTERNET is blowing my mind. mercutio: although more would be better :) -: brycec is afraid to see this month's Amazon bill mercutio: heh
you should check it out early then :) brycec: almost triple my usual monthly bill so far
(my usual monthly bill being <$5)
$10.75 on data transfer alone
$4.54 last month, and $18.86 projected this month.
wow so all this upload/download is costing me $10+ in transfer, but only $.30 for S3
Storage is stupid-cheap :D mercutio: sweet
that makes it not so good for short term. brycec: It makes for good backup storage - you only pay when you have to do a restore :p mercutio: heh brycec: (Presumably you backup and verify locally and S3 is just extra off-site storage that only needs to be verify periodically.)
And just remember - multipart uploads store an invalid MD5 on the object in S3.
(Looks like the sata drives in my dedi max out at 120MB/s sequential write, not too bad considering it's interspersed with other random reads+writes as I move files between volumes) mercutio: are they re4s? brycec: WDC WD1003FBYX-01Y7B1 mercutio: why is sda hotter than sdb/sdc hah
yeh
Model Family: Western Digital RE4
Device Model: WDC WD1003FBYX-01Y7B1 brycec: If my "sda" is hotter than sdb or sdc, I have real issues because sda is the IPMI virtual CD drive :P mercutio: heh
i was using smartctl
it's only 34c, that's not bad brycec: That's pretty reasonable, yeah mercutio: but the other two are 30/31 brycec: (thanks for reminding me to install smartmontools) mercutio: oh re4 might not be 4k
woot it's not brycec: That's a little surprising in 2015, but I guess that comes with "RE", perhaps for compatibility with controllers and other storage stuff. mercutio: the re4s are pretty good for random
4k is annoying for zfs overhead brycec: According to the spec sheet http://www.wdc.com/wdproducts/library/SpecSheet/ENG/2879-701338.pdf the 1TB also has the lowest power draw over 2TB, 500GB, and 250GB. Interesting. mercutio: it'll be single platter.
the 250/500 maybe older brycec: (I would guess the 500/250 are 4/2 platters, and 2/1 are 2/1 platters)
Yeah 4/2 because they're laying around, and it can improve performance mercutio: single platter drives tend to die less too. brycec: (actually 500/250 are the same weight) mercutio: yeh it may just be short stroked brycec: oh wow, I'm an idiot, it's printed in the specs mercutio: i used to be into short stroking brycec: 500/250 are single platter and 2 or 1 heads mercutio: well i still am i suppose brycec: 2TB is 8 head/4latter, and 1TB is half that. mercutio: but back when drive performance was one of the normal hinderances, using less than all of the disk made quite a difference RandalSchwartz: Hmm. changed 3 more things
oops wrong window brycec: *4 things -- you changed to the wrong window :p mercutio: heh brycec: That's odd... eth0 on my dedi box is flapping
looks like about once a minute it goes up and then back down 2 seconds later mercutio: weird
what chipset is it? brycec: intel igb
(is the driver, I know) mercutio: gb
igb
hmm mine is e1000e brycec: 01:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) mercutio: what kernel are you using? brycec: but eth1 is fine mercutio: are they both 82580? brycec: so either my bonding didn't take effect right, or ARP has an issue.
yeah mercutio: bonding is weird
i'm only using one interface
82580 is one of the flakier chipsets, i'd make sure you were using recent igb driver
bloody intel adn their errata
my home server had onboard intel and it was flakey too
and that was i217v or i218v
i can't remember which brycec: (And because you asked 2.6.32-37-pve, but I'm about to reboot into a newer kernel) mercutio: yeh
i'd definitely try a newer kernel first
before any real debugging
i expect it to magically get better. brycec: (I didn't notice anything wrong before my reinstalls, and they would've been running the newer kernel too) mercutio: if you read intel errata there are heaps of edge cases that don't work properly.
that are patched around etc.
you were uusing openbsd though?
openbsd doesn't enable some of the flakier features :)
it's generally things like segment offload etc that have issuues
you can disable with ethtool brycec: No this box has been Debian for months until I started messing around with it last night mercutio: ahh right
2.6.32 is ancient :/ brycec: And yeah I've backported intel drivers before, plenty of reading
only as ancient as Debian Wheezy :p mercutio: broadcom are bad too :)
wheezy is ancient haha
i can't remember what kernel jessie has brycec: 3.16.0-4-amd64
jessie ^ mercutio: cool i found something from last year suggesting that
3.16 should be fine brycec: Now running 2.6.32-39-pve on this dedi... let's see if it continues mercutio: i think i'm using 3.13
yeah 3.13.
i've found 3.13 to be a nice stable kernel version brycec: You're behind Jessie? mercutio: this is on ubuntu trusty
with custom kernel
i dunno what trusty uuses by default brycec: ah mercutio: trusty uses 3.13 too
it was aboutu the time trusty came out that it got installed.
i think slightly before
and i figured that nothing big would change :)
i keep meaning to upgrade it actually
but probably worth waiting a bit more brycec: Looking solid post-reboot. Was either a driver bug or just something hadn't initialized right last time around. mercutio: driver bug i suspect
you could have probably used newer igb without newer kernel
but newer kernel is better in general brycec: (For those wondering, that kernel upgrade also brought an upgrade from 5.2.15 to 5.2.18 of the igb driver) mercutio: why does geotrust still need an intermediate cert?
it seems the same intermediate cert is used everywhere
woot, apnic whois is /finally/ back.
pity there's monopolies on registrars :( mnathani: @weather -v yyz BryceBot: Toronto-Pearson International, Ontario: Partly Cloudy ☁ 43°F (6°C), Humidity: 61%, Wind: From the WNW at 22 MPH Gusting to 29 MPH, Pressure: 30.07inHg (1018mb) and holding, Dewpoint: 30°F (-1°C), Feels like 34°F (1°C), Visibility: 15Mi (24km), UV index: 0, Sunrise 05:48, Sunset: 20:42, Lunar phase: New moon
Wednesday: Partly Cloudy 62°F/44°F (17°C/7°C) | Thursday: Clear 68°F/43°F (20°C/6°C) | Friday: Clear 57°F/38°F (14°C/3°C) | Saturday: Clear 69°F/50°F (21°C/10°C)
The average high for this date is 62°F (16°C), and the record of 81°F (27°C) was set in 2012. The average low is 45°F (7°C), and the record of 32°F (0°C) was set in 2002 MrMorden: Looks like I'm getting bogons inbound to my VM... mercutio: faked ip's?
hardly anyone filters ip source addreses. MrMorden: 03:20:51.452048 IP 10.8.19.209 > 174.136.105.34: ICMP time exceeded in-transit, length 36
pretty sure my VM isn't pinging 10.8 mercutio: and not enough people filter outbond addreses to just have them. people using bogan filters can be a pita with these new weird ip addresses in use due to starvation.
i dnuno what arin is like, but apnic is using some previously bogan addresses MrMorden: fun
looks like 240.0.0.0/4 wasn't in the bogon list before but was added
yay multicast
and some CIDRs aren't in the list but should be - 7/8, for example mercutio: i wouldn't really worry too much
there's some lists of worm addresses etc that may be useful
but most malicious traffic is using real addresses
10.8.19.209 responding could be because you did a mtr somewhere, and a router has a private ip.
i'd say it doesn't hurt having it come in, and means it doesn't show missing hops. MrMorden: didn't think it's a big issue or anything, just slightly weird
router loopback actually makes sense
and Apple gave back half their /8. this shows that I'm getting tvtropes-ed by 'show ip bgp' and should go to bed now. mercutio: there's a huge long thread on nanog about 10ge routers atm
i thought it would mostly about people suggesting routeros hah brycec: But instead they're suggesting pfSense? :p mercutio: well someone was talking about dpdk and line rate.
and netmap etc.
i wish somebody put together something proper heh
hmm there aren't actually many options it seems if you want small pcaket forwarding performance pyvpx: pfSense is really playing up their version 3, now with dpdk for FASTAR packetz
oof