#arpnetworks/ 2015-05-19,Tue

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
mercutioyouu get heaps of data though [00:04]
brycec1) Why eat up bandwidth? 2) Wanted to be sure it was GbE through-and-through, 3) I forgot there was IPv4 access. [00:09]
mnathaniPlus jumbo frames on the vlan [00:09]
mercutiooh there's jumbo frames on the vlan?
jumbo frames don't actuually make much real world difference ime
[00:10]
mnathaniIf you get a second nic on your vps [00:10]
mercutioit can use less cpu on some old systems
i can't believe how much i'm struggling to stick intel stock cpu cooler on
[00:10]
brycec(and new ones, depending on bw/pps usage) [00:10]
mercutioit's not like i've never done it before [00:11]
brycecMe either. They're pretty easy. [00:11]
mercutioand i never remembered it being a big deal before [00:11]
brycecAre you sure you've done everything else correctly? CPU latched properly? [00:11]
mercutiofound the issue
one of the bits that goes in is "to it's side
can't see how cpu couldn't be latched properly
[00:12]
brycecThere was a time I would have said that too.
Never, EVER underestimate human ingenuity when it comes to screwing up the idiot-proof.
They forced it shut with the cpu not properly seated and actually bent the CPU
And after that day, management understood why I referred to the production workers as monkeys.
(well they understood before, but that really illustrated it)
brycec does not feel like delving into SASL setup for LDAP binding auth
But everything else on this Prosody server is looking good, woo
Including the most important https://code.google.com/p/prosody-modules/wiki/mod_swedishchef
[00:13]
...... (idle for 28mn)
mercutiofinally got it
swapped to another cpu cooler which "just worked"
so then i plug in computer and nothing comes up :/
took me a while to realise that the monitor wasn't auto selecting dvi
i hate hardware :)
and monitors seem to hate me atm
[00:46]
brycecGo install, go! (reinstalling my dedi machine with raid-1 and zfs root)
(zpool mirror, that is)
[00:53]
mercutiohah
freebsd or linux?
[00:57]
brycecLinux, just for fun (and because I like Proxmox) [00:57]
mercutioi bet there are systems out there already, but i want to see how fast i can make an install happen on bare hardware over the network
ie the right of compression file system extraction :)
err mix of compression and file system extraction
lz4 has just introduced new faster compression, again.
it's not altogether exciting - you can sacrifice some compression ratio for "even faster" performance.
on that note, on zfs i find lz4 works really well.
and i'd suggest using it :)
[00:57]
brycecI really, REALLY, REEEAALLY wish the java console viewer would stop stealing focus every dang time the video mode changes. Getting REALLY FUCKING PISSED-OFF. *deep breaths* [00:59]
mercutiohaha god
supermicro gear may be ok
but their out of band management really sucks.
[00:59]
brycecit needs to die, in a fire. [01:00]
mercutioi dunno if it's changed, but in the past supermicro have had this remote iso functionality, and it doesn't even use decent sized window sizes
so it's painfully slow.
err painfully slow if you have like 20msec+ ping.
it's probabl fine for < 1 msec.
i think it was using 16k or something
hmm, i ahve to figuure out how to make arch linux stop setting graphics vga mode at some point
actually i wonder if there's a way to get arch to do serial console for rescue system
[01:00]
brycechttps://wiki.archlinux.org/index.php/Working_with_the_serial_console [01:03]
mercutioyeah that's not about the "rescue" cd thogh [01:03]
brycecoh I thought you meant rescue.target [01:03]
mercutioi hosed my file system by doing a mv /* or something :)
oh what's rescuue.target?
[01:03]
brycecA systemd target for doing rescue stuff
(not helpful I know)
[01:04]
mercutiohmm [01:04]
brycecsystemd.unit=rescue.target on the kernel line [01:04]
mercutioyeah it's not common, and ie worked :/ [01:04]
brycecaka run level 1 [01:04]
mercutiohp's lights out pisses me off too :)0
like it's cool yo can type textcons and don't even need serial console setup to get remote text mode
err via ssh
and vsp to get a serial console
but if things are in graphics mode you have to use java or activex
and java seems to keep giving issues :/
i don't use it that much though
[01:04]
........ (idle for 35mn)
brycecI keep forgetting IPv6 traffic is limited to 100mbps and then wondering why I'm only getting ~7MB/s to mirrors.arpnetworks.com [01:41]
mercutioyeah it bugs me too :) [01:42]
..... (idle for 21mn)
damn i was curiouus how the Ubuntu LTS kernels work, and i try and read up about it and i'm even more confused.
it seems ubuntu now by default installs newer kernels for patch releases.
but i dunno how long support for these kernels lasts for
[02:03]
brycecWheee 75MB/s off S3 [02:05]
mercutiowow
how close is the s3 server?
[02:05]
brycecLAX - North California
(I'll run a traceroute in a bit)
but it's s3-us-west-1.amazonaws.com I believe
[02:05]
mercutioso it seems precise can be upgraded to trusty kernel and be supported until 2017 [02:06]
brycec(Don't look at me - I avoid Ubuntu wherever possible :P) [02:09]
mercutioheh
i prefer self-compiled kernels
[02:09]
brycecWoo, booting OpenBSD off a zdev
(via kvm)
[02:10]
mercutiozvol? [02:10]
bryceczvol.
block storage carved out of a zpool
[02:10]
mercutioheh yeh pretty awesome
what volblocksize did you use?
[02:10]
brycecwhatever Proxmox defaulted to. I didn't even think to look [02:11]
mercutioerk :/
there can be a bit of overhead, so raising it above the 8k can help
but it can mean read/write
but now that everyone seems to have 4k disks, zfs overhead stuff is getting a little scary
the mirror case isn't as bad as raidz though afaik
[02:11]
brycec(It's using the ZFS default of 8k) [02:12]
mercutiobut yeah i've started using bigger block sizes. also using lz4 compression thouugh
yeah
if you're bored sometime you can try playing with it :)
does proxmox make it easy to do autosnapshotting?
[02:12]
brycecDon't yet know. (FreeNAS sure does though, very nice) [02:13]
mercutioyeah it is nice
can use up a lot of space, but so handy :)
you can expose the snapshot directory on real flie systems
err when using native zfs as opposed to zvol i mean
[02:14]
brycecyeah
or just cd to the directory that was hidden there anyways :p
[02:14]
mercutioheh
tab completion is nice :/
[02:15]
brycecaw dang 20GB written before I realized it's using lzjb rather than lz4. Oh well, not a big deal... [02:16]
mercutiodamn
yeh i used to use lzjb with opensolaris
lz4 is a little quicker
but both are way quicker than gzip
[02:17]
brycecNothing wrong with lzjb, still got 1.26x from it even, but lz4 is better [02:18]
mercutiohmm i wonder what my ratio is like
refcompressratio 1.10x
that's for /home which probably has some big tarballs on it somewhjere
[02:18]
brycecrpool/vm-201-disk-1 1.96x [02:20]
mercutiothe cool thing abouut lz4 is it's so cheap it doesn't matter if you have bulky stuff on it [02:20]
brycecfor the VM I just restored from backup :) (it's pretty empty though)
aye'
[02:20]
mercutioit's even cheaper for stuff tha tdoesn't compress
1.41x for /hoem on the box i'm irc'ing from
[02:20]
brycecMy best ratio is 2.47x for a mysql database volume, and 2.26 for a fat postgresql volume [02:22]
mercutiobut yeah again with 4k disks, lz4 if it doesn't get under 4k will use 8k still with 8k volblocksize
it doesn't condense :(
now youu justu need a ssd cache :)
may have to be pci-e :)
[02:22]
brycecI have an SSD cache, and a nasty FreeBSD bug that causes my host machine to crash :P [02:23]
mercutioi'm mostly kidding, you probably don't do many reads [02:23]
brycec(that's my home box, that is) [02:23]
mercutioheh
i've bewen using l2arc only for metadata
[02:23]
brycecYes o/ Just realized my favourite feature with using ZFS on Proxmox -- I don't have to manually format+mount anything, "zfs create" does it all. [02:24]
mercutioit gives me most of the boost i care about [02:24]
brycecNice
I have a few spare 60GB SSD's so I just threw them at it.
[02:24]
mercutioit means you ls in a directory with heaps of files and it doesn't delay
yeah
by default it won't do much for sequential anyway
and if you have "plenty" of ram then all the important stuff will be in memory anyway
and if you use zvol's etc youu're likely to get into double caching
[02:24]
brycec(oh many I forgot how nice zfs set quota= is too... It's been too long) [02:26]
mercutionot that double caching is necessarily bad, but it doesn't alert zfs for most frequently used.
and the most recently used is "old" data.
i wonder if you could get linux to cache less
are you using zil bryce?
that i think can help more...
[02:27]
brycecNot on this proxmox box, no
But I do on my home system
(mirrored, no less)
[02:35]
mercutiocool
yeh i added another ssd to a hard-disk server recently
going to look at setting up both as l2arc and some zfs pool first
but considering trying zil
[02:38]
brycechard to go wrong :P Besides, ZIL/L2ARC can always be added/removed any time in the life of the pool so there's no cost to trying it out.
(assuming your pool is "fixed" and you're just adding/removing disks or ssds)
[02:44]
mercutioyeah
and you only need like a gig
i've made my cache drive way too big hah
atm it's 80gb
using 60gb
wow it's had more reads than all of the other drives together though
[02:45]
brycec"WARNING: MD5 signatures do not match:" Dammit Amazon, stop running up my S3 bill. [02:46]
mercutiowhy are using s3? :/ [02:47]
brycecBecause I needed somewhere to stash 150GB briefly.
(and cheaply, with a fast connection)
[02:47]
mercutioahh
so cos temp
i could have stored 150gb for you :/
it's cool that you can even do that easily these days.
[02:48]
brycecHeh, thanks for the late offer [02:49]
mercutioheh i didn't know you needed some temp space. ;) [02:49]
brycecIt's very cool. With a fast enough connection, storage is completely elastic. [02:49]
mercutioeven with vdsl i'm using offsite storage more nad more
lots of things aren't really performance sensitive
[02:50]
brycecI'm probably going to continue using it once I pare down what I sync to it as another offsite storage location. [02:50]
mercutioi backed up all of my important home stuff remotely.
took like 24 hours or so :)
with 9.5 megabit upload.
[02:51]
brycecI've been backing up personal stuff for S3 for ages (thanks to duplicity). Cheap, fast, reliable, easily encrpyted. [02:51]
mercutiobut as long as you set up aqm, it doesn't impact other stuff too badly. [02:51]
brycecBut those backups don't even break 100GB
heh
[02:52]
mercutiomy home directory volume is only using 98gb [02:52]
brycec(and this dedi box is 9ms and 14 hops from S3) [02:52]
mercutioheh
my dediacted box was 5 msec ping from me :)
damn interleaving beign on now
i have a personal dedicated server in nz
with zfs etc.
[02:52]
brycecwow, 5ms? that's practically beside you [02:53]
mercutiowell it's like 30km from me [02:53]
brycecmy first hop past my home router is 12-16ms [02:53]
mercutiowell it terminates on the same lan as my internet connection too
so it's like single hop away :)
mine's uhh 12 or 13 msec now, due to 8 msec downstream interleaving
but yeah it's not laggy at all :)
[02:54]
....... (idle for 31mn)
brycecbrycec is always entertained watching RAID-1 divvy up reads between disks. Just something fun watching it in iostat. [03:26]
mercutioyeah it doesn't work very well with hard-disks ime
but it works well with ssd's
[03:27]
brycecseems to work well enough
Frankly with the speed of SSD's the improvement is less than the improvement in read performance seen with hdd's
[03:27]
mercutiodo you ever look at zpol iostat -v 1 [03:28]
brycecwatching it now :) [03:28]
mercutioame haha
my second ssd cache only has 57.4mb allocated
but it's still doing reads for some reason.
hmm so much for l2arc being a waste of time :)
the other ssd seems to actually do quite a lot of requests.
probably means i need more ram in there :)
it does more reads then all the hard-disks, but less writes.
but less writes than any of the hard-disks
i suppose 60gb of ram isn't cheawp
i'm semi tempted to try this zil thing
[03:28]
brycec'night mercutio [03:35]
mercutioit must be like 3:30 am for you
'night :)
[03:35]
brycecprecisely right [03:35]
mercutio'night [03:35]
........................ (idle for 1h55mn)
RandalSchwartzawesome... got icinga for floss weekly [05:30]
....... (idle for 30mn)
mercutioicinga?
ahh a monitoring system
graphite sounds nice
[06:00]
RandalSchwartzit's a hostile fork of nagios
I've decided those are called "pitchforks" :)
[06:14]
mercutiohaha i liek the name
pitchfork that is
i've been wanting to do some kind of real time web ping thing
so i'm hoping graphite will make that easier.
[06:15]
***tabthorpe has joined #arpnetworks [06:21]
RandalSchwartzyeah
a fork that is hostile is a pitchfork
hopefully my show will set the meme
[06:25]
............................ (idle for 2h16mn)
***RandalSchwartz has quit IRC (Remote host closed the connection) [08:42]
........................ (idle for 1h58mn)
brycecI wonder what makes it a "hostile" fork [10:40]
mike-burnsIs that to differentiate against a GitHub fork? [10:42]
plett_mike-burns: Normally that the old project is still in active development and doesn't want to split its userbase with the fork
And yes. The github workflow actively promotes forking to deploy your fix and request that the original version pull in your patch
[10:43]
***RandalSchwartz has joined #arpnetworks [10:44]
plett_With the implicit assumption that if you write new patches and the original developer(s) do nothing and don't pull them in, you become the defacto standard version [10:45]
....... (idle for 33mn)
***mkb has quit IRC (Remote host closed the connection) [11:18]
..... (idle for 22mn)
mkb has joined #arpnetworks [11:40]
mkbreinstall to 5.7 was easy enough [11:41]
brycecExcellenty
*-y
[11:41]
mkbsiteXX.tgz makes things so easy [11:41]
brycecIt does :) [11:41]
...... (idle for 28mn)
***mkb has quit IRC (Remote host closed the connection)
mkb has joined #arpnetworks
[12:09]
............................. (idle for 2h23mn)
brycecbrycec is pulling 99MB/s off AWS, wheeee
I could get used to this GbE connection...
[14:34]
mercutioheh
bryce is hogging all the bandwidth :)
[14:36]
bryceclol, am not :P (Because ARP has a bigger pipe than just 1Gbps) [14:36]
mercutioyeh i know
i like the idea of gigabit for home users.
[14:36]
brycecGbE on a LAN is old hat to me, but getting GbE over THE INTERNET is blowing my mind. [14:37]
mercutioalthough more would be better :) [14:37]
brycecbrycec is afraid to see this month's Amazon bill [14:39]
mercutioheh
you should check it out early then :)
[14:40]
brycecalmost triple my usual monthly bill so far
(my usual monthly bill being <$5)
$10.75 on data transfer alone
$4.54 last month, and $18.86 projected this month.
wow so all this upload/download is costing me $10+ in transfer, but only $.30 for S3
Storage is stupid-cheap :D
[14:40]
mercutiosweet
that makes it not so good for short term.
[14:46]
brycecIt makes for good backup storage - you only pay when you have to do a restore :p [14:47]
mercutioheh [14:47]
brycec(Presumably you backup and verify locally and S3 is just extra off-site storage that only needs to be verify periodically.)
And just remember - multipart uploads store an invalid MD5 on the object in S3.
(Looks like the sata drives in my dedi max out at 120MB/s sequential write, not too bad considering it's interspersed with other random reads+writes as I move files between volumes)
[14:48]
mercutioare they re4s? [14:53]
brycecWDC WD1003FBYX-01Y7B1 [14:53]
mercutiowhy is sda hotter than sdb/sdc hah
yeh
Model Family: Western Digital RE4
Device Model: WDC WD1003FBYX-01Y7B1
[14:54]
brycecIf my "sda" is hotter than sdb or sdc, I have real issues because sda is the IPMI virtual CD drive :P [14:54]
mercutioheh
i was using smartctl
it's only 34c, that's not bad
[14:54]
brycecThat's pretty reasonable, yeah [14:55]
mercutiobut the other two are 30/31 [14:55]
brycec(thanks for reminding me to install smartmontools) [14:55]
mercutiooh re4 might not be 4k
woot it's not
[14:56]
brycecThat's a little surprising in 2015, but I guess that comes with "RE", perhaps for compatibility with controllers and other storage stuff. [14:57]
mercutiothe re4s are pretty good for random
4k is annoying for zfs overhead
[14:57]
brycecAccording to the spec sheet http://www.wdc.com/wdproducts/library/SpecSheet/ENG/2879-701338.pdf the 1TB also has the lowest power draw over 2TB, 500GB, and 250GB. Interesting. [14:58]
mercutioit'll be single platter.
the 250/500 maybe older
[14:58]
brycec(I would guess the 500/250 are 4/2 platters, and 2/1 are 2/1 platters)
Yeah 4/2 because they're laying around, and it can improve performance
[14:58]
mercutiosingle platter drives tend to die less too. [14:59]
brycec(actually 500/250 are the same weight) [14:59]
mercutioyeh it may just be short stroked [14:59]
brycecoh wow, I'm an idiot, it's printed in the specs [14:59]
mercutioi used to be into short stroking [15:00]
brycec500/250 are single platter and 2 or 1 heads [15:00]
mercutiowell i still am i suppose [15:00]
brycec2TB is 8 head/4latter, and 1TB is half that. [15:00]
mercutiobut back when drive performance was one of the normal hinderances, using less than all of the disk made quite a difference [15:01]
........ (idle for 36mn)
RandalSchwartzHmm. changed 3 more things
oops wrong window
[15:37]
brycec*4 things -- you changed to the wrong window :p [15:41]
mercutioheh [15:42]
brycecThat's odd... eth0 on my dedi box is flapping
looks like about once a minute it goes up and then back down 2 seconds later
[15:46]
mercutioweird
what chipset is it?
[15:54]
brycecintel igb
(is the driver, I know)
[15:54]
mercutiogb
igb
hmm mine is e1000e
[15:55]
brycec01:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) [15:55]
mercutiowhat kernel are you using? [15:55]
brycecbut eth1 is fine [15:55]
mercutioare they both 82580? [15:55]
brycecso either my bonding didn't take effect right, or ARP has an issue.
yeah
[15:55]
mercutiobonding is weird
i'm only using one interface
82580 is one of the flakier chipsets, i'd make sure you were using recent igb driver
bloody intel adn their errata
my home server had onboard intel and it was flakey too
and that was i217v or i218v
i can't remember which
[15:56]
brycec(And because you asked 2.6.32-37-pve, but I'm about to reboot into a newer kernel) [15:57]
mercutioyeh
i'd definitely try a newer kernel first
before any real debugging
i expect it to magically get better.
[15:58]
brycec(I didn't notice anything wrong before my reinstalls, and they would've been running the newer kernel too) [15:58]
mercutioif you read intel errata there are heaps of edge cases that don't work properly.
that are patched around etc.
you were uusing openbsd though?
openbsd doesn't enable some of the flakier features :)
it's generally things like segment offload etc that have issuues
you can disable with ethtool
[15:58]
brycecNo this box has been Debian for months until I started messing around with it last night [15:59]
mercutioahh right
2.6.32 is ancient :/
[15:59]
brycecAnd yeah I've backported intel drivers before, plenty of reading
only as ancient as Debian Wheezy :p
[15:59]
mercutiobroadcom are bad too :)
wheezy is ancient haha
i can't remember what kernel jessie has
[16:00]
brycec3.16.0-4-amd64
jessie ^
[16:01]
mercutiocool i found something from last year suggesting that
3.16 should be fine
[16:01]
brycecNow running 2.6.32-39-pve on this dedi... let's see if it continues [16:01]
mercutioi think i'm using 3.13
yeah 3.13.
i've found 3.13 to be a nice stable kernel version
[16:01]
brycecYou're behind Jessie? [16:05]
mercutiothis is on ubuntu trusty
with custom kernel
i dunno what trusty uuses by default
[16:05]
brycecah [16:05]
mercutiotrusty uses 3.13 too
it was aboutu the time trusty came out that it got installed.
i think slightly before
and i figured that nothing big would change :)
i keep meaning to upgrade it actually
but probably worth waiting a bit more
[16:05]
brycecLooking solid post-reboot. Was either a driver bug or just something hadn't initialized right last time around. [16:09]
mercutiodriver bug i suspect
you could have probably used newer igb without newer kernel
but newer kernel is better in general
[16:09]
............... (idle for 1h13mn)
brycec(For those wondering, that kernel upgrade also brought an upgrade from 5.2.15 to 5.2.18 of the igb driver) [17:23]
mercutiowhy does geotrust still need an intermediate cert?
it seems the same intermediate cert is used everywhere
[17:37]
................. (idle for 1h20mn)
woot, apnic whois is /finally/ back.
pity there's monopolies on registrars :(
[18:58]
.................................... (idle for 2h57mn)
mnathani@weather -v yyz [21:56]
BryceBotToronto-Pearson International, Ontario: Partly Cloudy ☁ 43°F (6°C), Humidity: 61%, Wind: From the WNW at 22 MPH Gusting to 29 MPH, Pressure: 30.07inHg (1018mb) and holding, Dewpoint: 30°F (-1°C), Feels like 34°F (1°C), Visibility: 15Mi (24km), UV index: 0, Sunrise 05:48, Sunset: 20:42, Lunar phase: New moon
Wednesday: Partly Cloudy 62°F/44°F (17°C/7°C) | Thursday: Clear 68°F/43°F (20°C/6°C) | Friday: Clear 57°F/38°F (14°C/3°C) | Saturday: Clear 69°F/50°F (21°C/10°C)
The average high for this date is 62°F (16°C), and the record of 81°F (27°C) was set in 2012. The average low is 45°F (7°C), and the record of 32°F (0°C) was set in 2002
[21:56]
.... (idle for 15mn)
MrMordenLooks like I'm getting bogons inbound to my VM... [22:11]
mercutiofaked ip's?
hardly anyone filters ip source addreses.
[22:11]
MrMorden03:20:51.452048 IP 10.8.19.209 > 174.136.105.34: ICMP time exceeded in-transit, length 36
pretty sure my VM isn't pinging 10.8
[22:12]
mercutioand not enough people filter outbond addreses to just have them. people using bogan filters can be a pita with these new weird ip addresses in use due to starvation.
i dnuno what arin is like, but apnic is using some previously bogan addresses
[22:12]
MrMordenfun
looks like 240.0.0.0/4 wasn't in the bogon list before but was added
yay multicast
and some CIDRs aren't in the list but should be - 7/8, for example
[22:13]
mercutioi wouldn't really worry too much
there's some lists of worm addresses etc that may be useful
but most malicious traffic is using real addresses
10.8.19.209 responding could be because you did a mtr somewhere, and a router has a private ip.
i'd say it doesn't hurt having it come in, and means it doesn't show missing hops.
[22:15]
MrMordendidn't think it's a big issue or anything, just slightly weird
router loopback actually makes sense
and Apple gave back half their /8. this shows that I'm getting tvtropes-ed by 'show ip bgp' and should go to bed now.
[22:19]
........... (idle for 51mn)
mercutiothere's a huge long thread on nanog about 10ge routers atm
i thought it would mostly about people suggesting routeros hah
[23:11]
brycecBut instead they're suggesting pfSense? :p [23:21]
mercutiowell someone was talking about dpdk and line rate.
and netmap etc.
i wish somebody put together something proper heh
[23:27]
hmm there aren't actually many options it seems if you want small pcaket forwarding performance [23:40]
.... (idle for 19mn)
pyvpxpfSense is really playing up their version 3, now with dpdk for FASTAR packetz
oof
[23:59]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)