#arpnetworks 2015-01-08,Thu

acf_

probably wouldn't be too difficult to reverse engineer

RandalSchwartz

not like VNC is a standard or anything

acf_

it's because they don't want to use password authentication

mercutio

hp has an emulated text console
you can just ssh in and type a command and it'll show you the screen in text
(as well as having seperate serial port for remote console)
the hp one is kind of laggy and annoying and not wonderful, but you use it so rarely that it doesn't really matter.
that is, it works.
i can't relaly see supermicro suddenly going open, but maybe people should pressure them to be "feature complete" with hp/dell

RandalSchwartz

well. the supermicro java has a serial console feature
but again, since I haven't enabled that in freebsd, I don't have it

[15:19]

mercutio

so does openipmi
yeah enable it in freebsd, and you can just use ipmi console, using ipmitool.
have you got a windows box you can try from?

[15:21]

plett

ipmitool sol works great with supermicro ipmi cards, I use it all the time

[15:25]

mercutio

plett: oonce you set it up :)

[15:26]

plett

mercutio: On Debian, that's just a matter of telling the grub bootloader to use a serial port as it's console

[15:26]

mercutio

plett: it doesn't help if it's not setup though
afaik randal can't get into his box at all atm

[15:27]

acf_

RandalSchwarts: use Debian?

mercutio

serial console in freebsd is easy too
console="comconsole"

plett

Set up how? I don't recall having to set anything up in the ipmi card via the bios before hand

mercutio

in /boot/loader.conf

acf_

GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,19200n8"
in /etc/default/grub

http://lists.freedesktop.org/archives/systemd-devel/2015-January/026811.html

mercutio

he said he was using freebsd iirc

[15:29]

acf_

I see

[15:29]

mercutio

that was off hp box but i'm sure it's the same

[15:30]

acf_

yea, I've used that before
unless it requires a certain baud rate or something

[15:30]

plett

On Debian with systemd (and probably other systemd machines too) you don't have to enable the getty on the serial port, it notices that grub is using serial as the console and does it for you
All hail systemd :)

[15:31]

acf_

yesterday my systemd on Debian got into a tight loop and wouldn't do anything

[15:32]

mercutio

oh hmm,

[15:32]

acf_

[15:33]

mercutio

ttyu0"/usr/libexec/getty std.57600"vt100on secure
think you need that in /etc/ttys
it doesn't like the tabs i think

[15:33]

dne

and on recent freebsd there's "onifconsole" which I guess is a bit like what plett described except w/o the dbus crap :)

[15:35]

plett

dne: I didn't know that. How recent? I haven't seen it in 10.0

[15:36]

dne

10.1 at least, and I think 9.3

[15:36]

plett

That probably means that I need to upgrade one of my 10.0 test machines to 10.1 to try it

[15:37]

dne

yup it's in the release notes for 10.1 & 9.3

[15:39]

RandalSchwartz

I know it's a simple edit to /boot/loader.conf, but if I screw that up, the box doesn't boot at all, I presum.
so I have to have someone who can get to the console to diagnose and fix how it's broken.

[15:43]

mercutio

i think you can set it at the bootloader too

[15:43]

RandalSchwartz

but the bootloader doesn't see the serial port unless it's in that

[15:43]

mercutio

yeah it does

RandalSchwartz

and I can't get to the bootloader without vnc

dne

just a getty on SOL may be better that nothing?

mercutio

the supermicro bios outputs to serial port as well as screen

RandalSchwartz

oh, interesting
so if I reboot the machine, but watch the serial port, I should be ok

mercutio

if the bios option is set at least.
hmm

RandalSchwartz

ok - how would I verify that? :)

mercutio

you're booted right now right?

RandalSchwartz

yes, running now

mercutio

do you know the motherboard model?

RandalSchwartz

mercutio

let's just check the manual
uhh

RandalSchwartz

something in dmesg would tell me?

mercutio

yeah if you still got that

RandalSchwartz

yes

mercutio

dmidecode should tell you too

RandalSchwartz

command not found: dmidecode

mercutio

there's a package
i checked it was on my freebsd box i checked the serial on :)
dmidecode-2.11 A tool for dumping DMI (SMBIOS) contents in human-readable

RandalSchwartz

where is it in dmesg?
oh... I'd have to add that to my poudriere to install it

mercutio

i don't have a boot dmesg

RandalSchwartz

I don't install from fbsd pkg

mercutio

it's in ports as sysutils/dmidecode if that's easier?

RandalSchwartz

ACPI APIC Table: <SUPERM SMCI--MB>

[15:48]

mercutio

yeah acpi not good enough

[15:48]

RandalSchwartz

so that won't mirror to console

[15:48]

up_the_irons

RandalSchwartz: i have the console open, so if you want to do your thing, i'll watch

[15:49]

RandalSchwartz

ok... here we go

[15:49]

up_the_irons

roger

[15:50]

mercutio

open serial console as it boots

[15:50]

RandalSchwartz

woo hoo. I have cache!
seems to be working fine

[15:50]

mercutio

you saw bios?

RandalSchwartz

no... just typed "zpool add zroot cache ad8"

mercutio

RandalSchwartz

and "zpool info -v zroot" shows the cache in place

up_the_irons

sweet

mercutio

you have ssd?

RandalSchwartz

yes
120G of ssd

mercutio

cool.
don't use 120gb for ssd cache on zfs
it takes up too much ram

RandalSchwartz

and 864G of real disk, mirrored

mercutio

just a word of warning :)

RandalSchwartz

we got fat mem on that box too

mercutio

well 120gb is too much for 32gb of ram really

RandalSchwartz

too late. already added. :)
actually, I can remove it

mercutio

also it isn't persistent

RandalSchwartz

but is there a way to just use a portion?

mercutio

yeah use a partition

RandalSchwartz

why woudln't more ssd be better?

mercutio

you probably only have 60gb or less of hot data anyway
you can also add another cache device in addition to the first if you find it too small

[15:54]

RandalSchwartz

heh.. given I have only 120GB of data on disk, I bet yes.

mercutio

heh
yeah unfortunately zfs's memory consumption could do work some optimisation.

up_the_irons

we could add a 256GB SSD :)
even 2x 256GB SSD ;)

RandalSchwartz

I wonder if I should try to remove it, just to see if I can
since up_the_irons is watching

mercutio

what freebsd version?

up_the_irons

RandalSchwartz: i still have the console open

RandalSchwartz

8.4

mercutio

i think removing caches wasn't implemented at the start

[15:56]

RandalSchwartz

is there a way to look at cache stats?

[15:56]

mercutio

yeah look at arcstats
or zpool iostat -v

[15:56]

RandalSchwartz

yeah, watching iostat.
oh... -v
heh ... cache 1.68G used

mercutio

you can do zpool iostat -v 1

RandalSchwartz

118G free :)

mercutio

to see it every second

RandalSchwartz

yeah - doing that

mercutio

cache - - - - - -
sdg1 60.8G 19.2G 6 7 44.8K 88.3K
that's off a real machine
which i learnt about not having too big cache on
mercutio wonders how you can have a not real machine
but yeah you may be fine

RandalSchwartz

virtual machine

[15:58]

mercutio

true

[15:58]

up_the_irons

ghost in the machine

[15:58]

mercutio

so even with a 80gb cache it hasn't got up to 80gb utilisation yet.
i actualyl experimented with using flash drive as cache too, it actually worked quite well for metadata.
using the sandisk usb3 ones.
normally ssd isn't used for bulk storage.
you can adjust how fast it writes etc.
but by default it basically will only try and capture random data access
also depending on the ssd you're meant to short stroke them less or more
but as a general rule of thumb you should be short stroking them 5 to 30%
more towards 30% for cheap/old/worn/crap ssd's.
and more towards 5% for high end

[15:59]

RandalSchwartz

I might be hallucinating, but it feels a bit faster already

[16:04]

mercutio

probably hallucinating
it's actually quite hard to notice the difference. most issues come from writes.
given sufficient ram etc.
i suppose i'm a bit of a non believer. it helps a bit if you're tight on resources, but there seems to be more improvement with metadata than real data.
but slog generally helps more
and because it uses memory that would otherwise be used for caching to store lookups it can reduce performance.

[16:04]

RandalSchwartz

slog?

[16:08]

mercutio

zil?

[16:09]

RandalSchwartz

ahh... yeah... I recall that.
should I be using this ssd for that instead?

[16:09]

mercutio

as well normally
grr i can't paste atm it seems
slog/zil is for writes

[16:09]

RandalSchwartz

so if I set up two partitions, used one for zil, the other for l2arc?

[16:10]

mercutio

l2arc/cache is for reads
yeah
but if ssd fails you can lose more data
so some people say you should have two
err if ssd fails at the time of power loss
so slog is like bbwc
but with ssd
it puts synchronous writes there
which means you can really improve performance of synchronous writes.
but it's not used for async writes.
there are tools to measure.
generally speaking i'd say if you only have 120gb of data it's better to use 2x256gb ssd's

[16:10]

RandalSchwartz

and the day we did that, we'd need 257gb :)

[16:13]

mercutio

enable lz4 compression
lz4 compression is also good
but yeah on e3/i7 type machines i'd generally say go for 32gb of memory before considering l2arc.
but if you're dual using it for other things it can still be good

[16:13]

RandalSchwartz

Oooh.. building ports... now using 8.17G of my cache!

[16:18]

mercutio

heh
the question is is it data you will use again?

[16:18]

RandalSchwartz

probably not
hopefully, I'm not building gcc... again.

[16:18]

mercutio

it's pretty hard to know
yeah but this stuff going into cache will be stuff that was in memory before that

RandalSchwartz

true

mercutio

so it may be helping you, but no more than having 8gb more ram so far :)

up_the_irons

RandalSchwartz: do you still need me on the console?

RandalSchwartz

No - thank you
but when I enable the serial console... I'll probably want you around in case that hoses.
... ARC: 17G Total, 3044M MFU, 12G MRU, 14M Anon, 615M Header, 1352M Other

mercutio

i thought that's what you were going to do while he was around?

RandalSchwartz

no - I was adding the cache

mercutio

the mfu part is cool.

up_the_irons

RandalSchwartz: i can remain on it if you want to enable serial

mercutio

ahh

RandalSchwartz

and didn't know if that would freeze the machine
No - I don't want to reboot maayan right now

up_the_irons

RandalSchwartz: or we can do it later, i'm working at a client's office until like midnight tonight, so i'll be around

[16:21]

RandalSchwartz

it'll have to be closer to off hours
we're still mostly west coast prime-time for neil's stuff

[16:21]

mercutio

do you have cacti randal?
you can measure load average pretty easily

[16:21]

RandalSchwartz

zabbix

mercutio

that's one of the best ways to see what performance impact there is
oh that can do that too right?

RandalSchwartz

yeah
graphs and alerts

mercutio

yeah
i would be curious to know what difference you find

RandalSchwartz

build finished
dmidecode now available

[16:23]

mercutio

cool

RandalSchwartz

what am I looking for?

mercutio

Product Name

RandalSchwartz

SMBIOS 2.7 present.
Product Name: X9SCL/X9SCM

mercutio

cool yeah that's better

RandalSchwartz

there's lots of product names
which one?

mercutio

they'll be the same
for our purposes at least

RandalSchwartz

oh, here's BIOS info

mercutio

grr is it tmux or urxvt not letting me paste

http://www.supermicro.com/manuals/motherboard/C202_C204/MNL-1270.pdf

RandalSchwartz

no product name there
serial services are supported

[16:26]

mercutio

weird it doesn't work in putty too
i got the manual in front of me

[16:26]

***

merc2 has joined #arpnetworks

[16:27]

merc2

[16:27]

mercutio

it's actually not loading well :(
like 15k/sec
look at 4-12

[16:28]

merc2

COM0/COM1/SOL Console Redirection
Use this feature to enable console redirection for COM0 and COM1 ports. The options
are Enabled and Disabled. The default for all ports are Disabled.
Serial Port for Out-of-Band Management / Windows Emergency
Management Services (EMS)
Use this feature to enable console redirection. The options are Enabled and Disabled.
The default is Disabled.

[16:31]

mercutio

so yeah it's not enabled by default, so probably has to be set

[16:32]

RandalSchwartz

I would set that using the java app?

[16:33]

mercutio

yeah
in bios
unless it was done when the system was provisioned.

[16:33]

RandalSchwartz

and where would that be in ipmiview

[16:35]

mercutio

does ipmiview let you access bios settings?

[16:35]

RandalSchwartz

I don't know
RandalSchwartz grins
actually - that jpmi thingy might have had something taht said bios
oops no... I mean the web-based thing
what is the "SOL" in "Launch SOL"

[16:36]

mercutio

i think that's serial console
serial over lan

[16:38]

RandalSchwartz

Nothing says "BIOS" in the supermicro web interface

[16:39]

up_the_irons

mercutio: yes, you can access bios over ipmiview. it's the actual console.

[16:45]

mercutio

up_the_irons: do you enable console redirection?

up_the_irons

mercutio: yes

mercutio

cool well there's the answer randal
up_the_irons fixes the broken supermicro defaults :)

up_the_irons

my job is to make life easier for you guys
so yes
;)