mercutio: with windows i'd rather use l2tp/ipsec, but with linux openvpn is easy up_the_irons: i'm glad it was easy to set up for you :)
as request some weeks ago: http://support.arpnetworks.com/kb/main/is-there-a-firewall-filter-rate-limit-or-similar-device-applied-to-my-traffic
*requested
let me know if anything isn't clear CaZe: I guess I can remove that rule from my rule set. ***: jpalmer has joined #arpnetworks
jpalmer has quit IRC (Changing host)
jpalmer has joined #arpnetworks
tburke has quit IRC (Read error: Connection reset by peer) mercutio: up_the_irons: your openbsd 5.4 iso on http://mirrors.arpnetworks.com/ISO_Library/ is timestamped before openbsd 5.4 came out?
by like 4 months
actually 5.3 is early too. i imagine something has a grossly wrong time somehwere :) toddf: if the iso timestamp is early, it is because there is a creation of the iso prior to the actual release; media gits burned, factories generate the cd's and stickers, then finally release day arrives.
the official timestamp of 5.4 is: -rw-r--r-- 1 root wheel 243050496 Jul 30 15:38 5.4/amd64/install54.iso
so it seems that arpnetworks has simply preserved the timestamp from the mirror site the image was retrieved from CaZe: Yeah, releases get tagged 2-3 months before actual release date.
A couple weeks for testing, and a few archs take months to build packages. mkb: VAX... mercutio: toddf: oh real
that's months prior though?
i suppose it can take a while to press
i suppose it's 3 motnhs not four
i was thinking 7=july, 11=november
but iot's 30 july to 1st november
i was also looking at how 5.2 said feb, ..
but 5.1 is actually more cent than 5.2
s/cent/recent/ BryceBot: <mercutio> but 5.1 is actually more recent than 5.2 mercutio: oh wow
s/^o/O/ BryceBot: <mercutio> Oh wow mercutio: i had to try that :)
s/mercutio/Mercutio/
so, it's just on the output, not the nick. up_the_irons: all i did was rsync them, i swear ;) mercutio: yeh
well toddf's explanation works
btw up_the_irons have you considered using unbound for recursive dns? up_the_irons: i have mercutio: i assume it'd be one more thing to maintain
cos you'd have to keep resolving working with the old dns with people using that currently. up_the_irons: in fact, unbound *is* running on 208.79.88.9 mercutio: oh?
actually you don't host dns for people do you?
so i suppose you could remap your authorative
hmm i was using .7
i'm not atm for some reason
oh, 89.9 is the secondary normally?
oh, nah it does say 88.9
i'd just looked at host -t ns arpnetworks.com not close enough
hmm, 88.9 resolves to ns2 which resolves to 89.9
and 88.7 appaers to be faster than 88.9, probably cos most people using primary staticsafe: just run your own locally ;) ***: ThalinVien has quit IRC (Quit: leaving) m0unds: i haven't run my own recursive or authoritative ns' in years gizmoguy: m0unds: :(
missing out m0unds: nah mercutio: i think authorative is more important to run yourself than recursive
recursive you can just cache locally if you do lots of dns gizmoguy: yeah, recursive if you're an ISP or have lots of machines in one place m0unds: i like to pretend fairies answer queries and never give it a second thought gizmoguy: and authoratative if you need control of your zone mercutio: recursive if you have more than one transit provider
is more where i'd say it starts
whether or not isp gizmoguy: well, multiple recursive :) mercutio: i don't reckon wireless isp's need their own recursive dns rather than just cache if they are single-homed.
well single-homed transit, with peering recursive still good. gizmoguy: I've seen ISPs go horribly wrong when they have split upstreams and their recursive server on a different upstream to their customers staticsafe: TBH I think everyone should run their own recursive mercutio: what do you mean a different upstream? gizmoguy: so say you have business customers and resi customers
and a cheap upstream and a more reliable upstream mercutio: do you mean having non PI space and having some IP on one transit provider, and the other on another transit provider? gizmoguy: shove business customers on the reliable one falling back to the cheap one
and visa versa mercutio: so PI space, right. gizmoguy: and you get to use the bandwidth on both transits mercutio: or at least dual advertising
yip gizmoguy: yeah
google does some really funky stuff when you mess that up mercutio: oh right i get yah
you mean like it'll push stuff over one provider or the other
but the DNS picks something for the other isp
and then the route sucks for the other isp? gizmoguy: yeah so, all the google CDN stuff is based on the IP it thinks the resolver is coming from mercutio: i think that's generally not too bad as long as you send outgoing via the best path
as google will choose an isp that's good for the connecting IP gizmoguy: so when you do a lookup to google.com the CDN goes, alright you have a GGC node at X IP nearby and returns that
and then you try connect to that IP from the client machine, and the GGC goes, hold on you're not using my transit mercutio: yeah you have to dual advertise routes at least gizmoguy: and falls you back mercutio: yaeh, that's why you need a dual advertiseement gizmoguy: yeah mercutio: even if you have like much longer as-path~ length gizmoguy: yup mercutio: it'll still let you use it
at least from what i've seen
you can actually use other isp's to transit to upstream ggc gizmoguy: mmm, but it would have been far more efficient to use the GGC node on the upstream you're connected to :) mercutio: on the alternate transit provider and it'll work
no, not really
cos generlaly speaking it'll still come down the transit provider that you're connecrted to
's link
because most tarnsit providers will prefer transit routes over learned routes
but
it'll suck from a load balancing point of view
the GGC cache stuff is nasty anyway :/ gizmoguy: also depends on how you're doing your forward routes mercutio: problem is it's so much traffic
so it matters. gizmoguy: yeah, I notice a fault on GGC about once a week :/ mercutio: yeah i'd advocate doing best path forward path regardless. gizmoguy: I've got some fantastic graphs if you give me one second m0unds: graphmaster gizmoguy mercutio: 1 .. 2 .. 3 gizmoguy: google have been recently sending us to pretty much the furthest location in their network mercutio: i've had that issue
it was amsterdam m0unds: second class citizens gizmoguy: oh look yay mercutio: and only on some videos gizmoguy: they're doing it again today
http://amp.wand.net.nz/graph/rrd-smokeping/23235/1384215383/1384388183 mercutio: i'm in new zealand gizmoguy: oh hai mercutio: oh so are you? gizmoguy: me too :) mercutio: was it amsterdam? gizmoguy: google wouldn't tell me
and I couldnt' work out from traceroutes mercutio: oh that doesn't really tell you where you're going
necessarily gizmoguy: but the biggest spike I see on the graph is 600ms mercutio: do your videos actually go to there?
oh
actually gizmoguy: I don't really care about that mercutio: that shows me local gizmoguy: (not an ISP :)) mercutio: if you look at tcpdupm or such when viewing videos
and trace to those ip's
it can be kind of random where it sends you gizmoguy: yeah for sure
when I send those graphs to google mercutio: but i've seen that go to amsterdam gizmoguy: and the IPs we were hitting mercutio: via australia gizmoguy: they said I couldn't have got much further down their network mercutio: with horrible route gizmoguy: :)
yeah GGC is an interesting beast
do you host a GGC box? mercutio: nope gizmoguy: ah yup mercutio: i work for small isp
don't have ggc gizmoguy: sweet, where 'bouts in NZ? mercutio: but have dual upstreams
i'm in auckland gizmoguy: ah cool
not too far from me
hamiltron here
(small world) mercutio: heh
i was from chch
but moved after the earthquakes gizmoguy: fair enough mercutio: i haven't actually had youtube performance issues recently m0unds: what was wrong with old zealand? mercutio: hey what
http://www.youtube.com/my_speed
check out global on that
m0unds: i think it's in the netherlands actually
http://en.wikipedia.org/wiki/Zeeland BryceBot: Zeeland :: Zeeland (Dutch pronunciation: [ˈzeːlɑnt] ( listen), Zeelandic: Zeêland), also called Zealand in English, is the westernmost province of the Netherlands. The province, located in the south-west of the country, consists of a number of islands (hence its name, meaning "sea-land") and a strip bordering Belgium. Its capital is Middelburg. With a population of about 380,000, its area is about 2,930 km², of which almost 1,140... gizmoguy: hamilton representing at 6.99mbit mercutio: gizmoguy: look at the global grpah though
the grey line gizmoguy: large spike? staticsafe: lol at google thinking im in Ottawa mercutio: yeah
upwards gizmoguy: interesting mercutio: i wonder what google did
then look at the new zelanad graph
so it's not new zeealand boosting the global speeds.
it seems to be geteting worse not better gizmoguy: why
are the averge speeds
showing a time related trend m0unds: weird dutch mercutio: google have said it's higher in weekends
err lower in weekdns
i can't remember which gizmoguy: lower speed? mercutio: something about people having faster or slower net at home/work gizmoguy: interesting mercutio: i can't rembmer now gizmoguy: ah, facinating mercutio: it probably varies by country if people have faster net at work or home gizmoguy: my work tubez a lot faster than my home tubez mercutio: in korea i imagine it's faster at home for instance
cos fibre so common gizmoguy: I'm still failing in my quest to get 10gig to my desk mercutio: but probably "shared" connectino at work and non shared at home
i have infiniband m0unds: gizmoguy: gotta work harder at it gizmoguy: hah!
nice mercutio: giving > 10 gigabit
it's cheap
do it gizmoguy: I got close to 10gig at my desk mercutio: it's not 10 gigabit internet though :/ gizmoguy: found some spare fibre pairs
found a 10gig internet provider m0unds: i got nowhere near 10gig anywhere near me mercutio: it does like 1.4 gigabytes/sec nfs gizmoguy: just needed to run some patch leads and buy an optic m0unds: well, i guess i could walk in the other room and look at 10gig interconnects but meh
hahaha gizmoguy: but I got lazy mercutio: gizmoguy: 10 gigabit international in new zealand? gizmoguy: only 1 gig international :(
going up soon mercutio: i've found it interesting looking at peoples expectations of international performance in new zelaand.
it wasn't that long ago people were happy with 2 megabit/sec international
which i thought was a bit off but hey
but as you start going up you end up getting less and less improvement
but like with UFB coming etc
theyl'l soon figure out that single threaded 100 megabit isn't really likely to work out that often
to anywhere furhter than australia gizmoguy: well 1 gig to states 1 gig to syd
though that's going up shortly mercutio: hell, i was only getting 100 megabit between chicago and arp
both on gigabit gizmoguy: UFB will be interesting to see rolled out mercutio: i don't know anyone with ufb yet
you're in the UFF area right? gizmoguy: hamilton/tauranga/christchurch are so far ahead of the rest of the country in UFB
yeah
I work closely with two UFF RSPs
one in hamilton one in tauranga mercutio: ahh ok.
what do you think of UFF? gizmoguy: in waikato we actually have 2 cities fully rolled out with fibre
UFF are the worst mercutio: my area of auckalnd isn't in the UFB thing gizmoguy: they contract out everything mercutio: but there's fibre on my street gizmoguy: huawei build the network mercutio: what latency you been seeing? gizmoguy: they contract out to another company to oversee the design of the network
latency is pretty good mercutio: is it getting under 1 msec yet? gizmoguy: no way
well mercutio: i was seeing 1.5 m0unds: huawei, oh boy gizmoguy: I haven't tested mercutio: with someone i know
but they don't have fibre to their house
actually they're not in a ufb area either
it's meant to be about 80% of new zealand isn't it? gizmoguy: the fun ones with UFF
are the funky ways they handle their vlans across the network mercutio: m0unds: huawei is big in nz
sort of
oh i read something about that gizmo
i got confused. gizmoguy: yeah
it's using lots of huawei technologies m0unds: big because of cost, or because of some other factor?
just curious - the guys i know who have had to work with huawei gear hated it gizmoguy: cost probably m0unds: does NZ have import taxes on stuff? gizmoguy: yup ***: ThalinVien has joined #arpnetworks m0unds: does it depend on hw cost or where it's imported from? mercutio: m0unds: new zealand is cheap too
m0unds: hw cost i think m0unds: ah, ok mercutio: internet is expensive in new zealand gizmoguy: here's another good graph mercutio
http://wand.net.nz/smokeping/?displaymode=n;start=2013-11-04%2013:44;end=now;target=Off-net.google
google.com likes to move around a lot
which is strange considering we have 2 GGC nodes upstream of us mercutio: gah i give up :) gizmoguy: ~30ms away is SYD, and ~50ms away is japan mercutio: on trying to find email gizmoguy: haha no worries
email is hard -: mercutio makes mental note to organise his email better mercutio: uhh
is that frmo christchurch? gizmoguy: nah, hamilton mercutio: 50 msec seems rather high to google gizmoguy: yes, yes it does
we get taken to japan sometimes mercutio: i wonder if it's going to melbourne
yeah i seen that before
i have seen apple.com go to malaysia too
do you work in hamilton uni? gizmoguy: I don't even bother tracking that one :)
yeah, I'm at the network research group at waikato uni mercutio: ahh cool.
i thought it was interesting that hamilton had network reserach stuff going on
i want to do a bit of research myself in a way :) gizmoguy: mmm we do a lot of random stuff
which is quite fun mercutio: cool. gizmoguy: doing a heap of openflow/SDN stuff right now mercutio: you know how there's a standard for specifying maximum link bandwidth in a path?
but every hop would have to be able to diminish the bandwidth if necessary?
so it needs router support so it went nowehere gizmoguy: as do most standards mercutio: err proposed standrad i think gizmoguy: well, IETF drafts I should say :) mercutio: i want to know how much benefit that has in the real world. gizmoguy: does PMTUD not do enough for you? mercutio: cos like on adsl connections, if you get the transmitting host to rate limit
single threaded throughput can often be significantly higher internationally.
it's not about mtu
it's about not overflowing buffers/queues. gizmoguy: Ohh right mercutio: in between
and losing heaps of packets. gizmoguy: sorry with you now mercutio: and then having to retransmit.
so like if you have a 20 megabit dsl connection
20 emgabit sync rate
and do single threadd download, you'll probably only get 10 megabit
but if you cap the speed at 16 megabit you'll probagbly get 12 megabit
if you increase the download size, then it matters less in a way
because tcp/ip adapts.
but lots of downloads are small
and cubic is pretty good at ramping up
and tbh i care more about the speed of downloading 10mb than 200mb normally
generally speaking for smaller downloads i'm more likely to be wanting it "now" and larger ones i can go make coffee or whatever gizmoguy: yeah, tcp is fun times mercutio: gah gizmoguy: a general protocol meant to support speeds of 1k up to 100s of gbit mercutio: so i try downloading my 10mb file from arp and it's going slow atm
well cubic is pretty good staticsafe: TCP slow start o/ mercutio: fwiw i downtune for infiniband
the same way i downtune for gigabit
like i set my maxium window sizes down
and for some reason something's been going around for a while advocating massive window sizes
and saying that tcp/ip will deal
but often it actually diminishes performance being set too high
static: well cubic takes over with slow start staticsafe: o mercutio: and actually works pretty well
basically it looks at time between packets
and uhh
something else
i don't think it's simple packet pairs though
but it's not ack 16 times 32 more packets.
oh, also, linux is sneaky, and acks every packet for short connections.
for the first 16 packets. gizmoguy: SACK helps out a lot mercutio: sack doesn't end up acking enough packets.
for my liking
wireless throughput is limited with tcp versus udp quite significantly.
because wireless is half duplex. gizmoguy: wireless is the worst :) mercutio: yeah but wireless also buffers randomly etc
not showing it to the applications
or giving feedback gizmoguy: speaking as someone who used to work for a company that build embedded wireless devices
s/build/built mercutio: so if it's going to do that anyway, you may as well just throw more packets at it, and not worry about having the acks in a timely fashion BryceBot: <gizmoguy> speaking as someone who used to work for a company that built embedded wireless devices mercutio: closely synced.
and you may as well ack less often gizmoguy: mercutio: got a masters student working on 802.11ad at the moment
that stuff is funky
"WiGig" mercutio: what's 802.11ad?
is it like ac but better? gizmoguy: 60ghz wireless mercutio: ahh ok
does it use more forward error correction? gizmoguy: it gets most of it's awesome from beam forming mercutio: ahh gizmoguy: it's only built for short lengths mercutio: i thought that was just a buzzword gizmoguy: like in a room for example mercutio: that wasn't really significantly implemented gizmoguy: so with wigig
you usually have an antenna array mercutio: like all around your device? gizmoguy: and then with phase shifting your array you can point/focus the 60ghz more closely to your device
you can make a flat square one I believe mercutio: wat happens if you have a wireless 802.11ad ball
and you throw it around the room gizmoguy: I suspect it doesn't work very well
it's more for stationary devices mercutio: ahh gizmoguy: as everytime you move it needs to recalculate the beam forming stuff
the example given in the 802.11ad docs
"Imagine your at an airport just about to board, but you want a bluray on your phone to watch on the plane"
just walk up to the movie transmitter pole, buy your movie and have it in a few seconds mercutio: hey i had a better idea
high bandwidth plug in ports for phones
like network ! gizmoguy: it's also "widely" used for wireless laptop dongles
dell has a wireless dongle you can buy
so you sort of just place your laptop near it
and you get vga/dvi/usb etc mercutio: i suppose it is kind of cool.
infiniband supports iommu
video cards support iommu
i was thinking it'd be cool to copy straight from video card to infiband before
for remote display
i think video card iommu support is pretty new though
and mostly focused on opencl
but i've always liked the idea that you can basically map memory over network gizmoguy: i've not played too much with infiniband
as in, i've never played with infiniband :( mercutio: my cards cost $75 USD
i think it was
for four
dual 20 gigabit
plus $45 USD shipping
or something gizmoguy: the next fun toy i'm hoping to have a play with is some infinera kit mercutio: cables are about US $15 or something i htink gizmoguy: 5TB/s on a single fibre pair mercutio: so it's way cheaper than getting 10 gigabit ethernet
and it can do about 15.8 gigabit/sec
with single threaded iperf gizmoguy: * Disclaimer 3 racks worth of infinera multiplexing gear required at each end of cable mercutio: with 600k window size gizmoguy: what do you use to swtich infiniband? mercutio: hahha
i don't
it's back to back gizmoguy: ah right mercutio: you can just use opensm as a subnet manager on linux
also the cables are thick.
i'm using copper.
and samba doesn't do rdma
on linux
and samba's normally code doesn't seem to work well
limiting throughput to 400mb/sec or so
with like 100% cpu on samba process
err 400 megabytes/sec
but nfs will go faster.
even without rdma
oh and that's using recent i7 cpus
at each end
and ssd on both sides
but yeah, i wonder if samba 4 will speed things up
but if 10 gigabit ethernet bcomes common people will be like "i can't get my full ssd speeds over ethernet" gizmoguy: lol
let's see what's the fastest I can do over 10gig HTTP mercutio: i really wish 2.5 gigabit ethernet came around
try cachefly?
http://cachefly.cachefly.net/100mb.test
maybe a little small
but it's australia gizmoguy: Fedora-19-x86_64-DVD.iso Length: 4444913664 (4.1G) [application/octet-stream]
4,444,913,664 779M/s in 6.9s mercutio: nice
where's that to? gizmoguy: AKL <--> WLG mercutio: nice gizmoguy: I'll try AKL <--> CHC mercutio: do you feel like running a random binary? gizmoguy: depends on the binary :P mercutio: i been working on a small curl like program
would be curious how it compared :) gizmoguy: I'll run it locally on my test 10gig network
rather than the prod 10gig NZ network :) mercutio: http://202.49.71.58:24/microcurl
you can su to nobody or such
it doesn't support output file name yet
only just started adding getopt support :/
but it'll output to stdout
just > /dev/null
oh and you need to include http:// atm
would be curious to see how it compares with the "time" command gizmoguy: so just microcurl http://file > /dev/null?
time microcurl http://file > /dev/null ? mercutio: yeah
err http://<host>/<file> > /dev/null
you can include port too
i usually use port 24 to bypass any transparent proxies
not because i'm against transparent proxies
but so i can test speds gizmoguy: these two machines are directly connected mercutio: yeah gizmoguy: well technically there's a switch in the middle mercutio: i found that it's about half as much cpu on haswell
as curl gizmoguy: hopefully not proxying :) mercutio: on infiniband
it doesn't listen to http_proxy atm
and doesn't even support -x yet
it does use some low granuality timer that is only supported in recent lijnux
i haven't checked how it handles that not working yet
(that's for showing the speed, which i don't care about granuality with) gizmoguy: grabbing an ISO from a mirror
only getting 111MB/s :(
to test with mercutio: heh
local gigabit gizmoguy: yeah :( mercutio: i been looking at cpu utilisation over network
i was first looking at speed over lcoalhost
but even with infiniband speed shoudl be the same.
well other than it starts up faster than curl gizmoguy: Hmm what have I cocked up here mercutio: which is like 6 msec difference or something
72 to 78 msec or such gizmoguy: 80MB/s over 10gig
that's a little slow mercutio: and curl does that too? gizmoguy: that's with wget mercutio: is the file too big for memory?
wget uses http_proxy
by default
you haven't got that set right? gizmoguy: locally wgetting on the same machine i get 900MB/s mercutio: that's slow
:/ gizmoguy: spindles
it'll be as fast as it can read from disk mercutio: 100 200M 100 200M 0 0 2317M 0 --:--:-- --:--:-- --:--:-- 2325M
you should output to null :)
oh irght
i keep assuming cached speed
yeah
100 200M 100 200M 0 0 889M 0 --:--:-- --:--:-- --:--:-- 892M gizmoguy: I have to read it from somewhere first :) mercutio: i got that the first time gizmoguy: ok what is going on here mercutio: gigabit needs no tweaking locally normally gizmoguy: it starts fast mercutio: what does iperf say? gizmoguy: and trails off to 80mbit mercutio: oh
do you have broadcom ethernet? gizmoguy: intel 10gig cards mercutio: hmm gizmoguy: screw off with broadcom mercutio: i hit something weird like that with broadcom
:/ gizmoguy: they're engineering samples though mercutio: still shouldn't matter
in normal adapative coalescing mode? gizmoguy: I was playing with coalescing last time I was logged into these machines mercutio: actually it's probably simpler than that.
ahh
try ethtool -c3 ?
err
ethtool -c rx-usecs 3
that's what the gigabit stuff defaults to
but i dunno if the 10 gigabit stuff is the same gizmoguy: I have to be careful with the coalecscing stuff mercutio: how come ? gizmoguy: last time I played I managed to generate a DIV_BY_ZERO in the intel driver mercutio: err it's -C gizmoguy: and the linux kernel gets unhappy when you do that mercutio: hahaha
i played around with coalescing and dropped cpu usage by heaps
on broadcom core2duo gizmoguy: I just don't even touch broadcom these days mercutio: heh gizmoguy: they don't let you do as much in hardware as the intel nics mercutio: it's my colo box
it's
i've got a colo box at sky tower
i could stick intel ct adapter in
but it seems to go ok
for some reason hp ilke to use broadcom gizmoguy: alright
7gbit in iperf TCP mercutio: interesting gizmoguy: 1mbit in iperf UDP mercutio: what gizmoguy: yeah...
what on earth was I doing on this machine last time I was logged in mercutio: history :)
fwiw infiniband sucks for udp
is it 1 megabit sending speed?
or 1 megabit throughput? -: up_the_irons likes all the network talk gizmoguy: oh wait mercutio: cos if it's only sending at 1 megabit gizmoguy: I think it's just iperf being retarded mercutio: did youadd -b 2g
or such? :) gizmoguy: if I stop the iperf server
the client can still connect and reports i'm getting 1mbit/s mercutio: oh gizmoguy: even though you tcpdump and there's 0 traffic mercutio: firewall on port 5001? gizmoguy: yeah will be
stupid iperf mercutio: yeah the udp testing is pretty broken
it's also bad if you overshoot
it can't receive the end result back
because it gets packet loss gizmoguy: I wonder if I have any rate limiting in here mercutio: i would rather it see it do a bit of return traffic to make sure it's still ok to test too gizmoguy: because it gets through the first 2GB at full speed
and then it drops to 100mbit mercutio: tc -s qdisc gizmoguy: won't be tc mercutio: going to /dev/null ?
oh with iperf gizmoguy: possibly iptables or apache
nah this is wget / apache now
yeah going to /dev/null mercutio: ahh m0unds: lulz u needz nginx lol omg gizmoguy: node.js man BryceBot: BAREMETAL!!! mercutio: just instill lighttpd on alternative port?
or nginx :/ gizmoguy: ok, so these machines run no firewalls mercutio: i use lighttpd gizmoguy: possibly I should fix that m0unds: lighttpd is faster with static files mercutio: heh gizmoguy: and no TC
wonder how apache on this box is configured m0unds: crapache lulz lulz mercutio: i never see lighttpd having bad performance :/ gizmoguy: tbh I didn't even know there was apache on here so I never set it up :) m0unds: i feel brain cells dying just typing that mercutio: for some reason it's way more computationally expensive in linux to reecive files than send gizmoguy: m0unds: :) mercutio: well actually it's cos linux can't do zerocopy receive
gizmo, maybe you should use a 2gb file? :)
i only use 200mb file for testing normally gizmoguy: alright
we solved the issue mercutio: what was it ? gizmoguy: I put a 2.7gb file on instead :) mercutio: haha gizmoguy: these are just dev/test boxes
not a clue where the rate limit is mercutio: yah gizmoguy: right
wget gives me: 828M/s in 3.8s mercutio: with how much cpu usage?
i wonder why you're not getting full speeds gizmoguy: wget maxes at 45% mercutio: with sandy bridge era cpu? gizmoguy: probably a mixture of slow spindles/not running 9k mtus/a crappy dell switch in the middle
nah these will be older
model name : Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz mercutio: 5600 series?
oh
wow
that old ok ;) gizmoguy: :)
why fix what's not broken? mercutio: considering i get 20% cpu usage on gigabit with e3110
is that with tcp timestamps enabled or disabled? gizmoguy: I think i broke your program mercutio: probably gizmoguy: written -1200776 kbytes (24.455); average: -49101.4584 mercutio: it's probably the kernel version
doesn't like the timer gizmoguy: Linux marvin 2.6.32-5-amd64 #1 SMP Fri May 10 08:43:19 UTC 2013 x86_64 GNU/Linux mercutio: CLOCK_MONOTONIC_COARSE gizmoguy: oh look another machine I haven't upgraded to wheezy mercutio: it should still be accurate using the "time" command gizmoguy: it starts off ok
and then it pauses and jumps into the negative mercutio: umm
it was introduced in linux 2.6.23
oops
linux 2.6.32
which is what you're using
oh that's interesting gizmoguy: written -1166052 kbytes (25.291); average: -46105.4135
real 0m26.686s
user 0m0.004s
sys 0m6.476s mercutio: umm
you know
i bet it's a signed 32 bit integer
issue gizmoguy: haha yeah
that's what it feels like mercutio: it looks like it
but it's 64 bit binary
but yeah it looks exactly like that
and i use 200mb test ifle
ok gizmoguy: :) mercutio: so 6.476s cpu
what was wget? gizmoguy: 3.8s mercutio: oh
3.8s system cpu? gizmoguy: real 0m3.565s
user 0m0.184s
sys 0m1.628s
that's for wget mercutio: oh hangon
so it's being WAYYYY slower gizmoguy: yeah mercutio: i wonder why gizmoguy: i'll try with a smaller file mercutio: % time ./microcurl/microcurl http://192.168.50.254:24/200m > /dev/null
written 204800 kbytes (0.153); average: 1338562.092
./microcurl/microcurl http://192.168.50.254:24/200m > /dev/null 0.00s user 0.08s system 50% cpu 0.152 total
hmm that's slower than before
oh i shouldn't load it over nfs prboably gizmoguy: alright, debian cd1 this time mercutio: hhmm same diff gizmoguy: ~700mb
wget: real 0m0.776s user 0m0.028s sys 0m0.364s
written 654432 kbytes (1.689); average: 387467.140
real 0m2.303s
user 0m0.000s
sys 0m2.040s mercutio: wow
# time ./microcurl http://202.49.71.58:24/testfile.zip
http protocol.
written 204800 kbytes (0.160); average: 1280000.000
./microcurl http://202.49.71.58:24/testfile.zip 0.00s user 0.16s system 98% cpu 0.163 total
hmm
oh oops from wrong computer
# time ./microcurl http://202.49.71.58:24/testfile.zip
http protocol.
written 204800 kbytes (1.799); average: 113841.023
./microcurl http://202.49.71.58:24/testfile.zip 0.01s user 0.55s system 31% cpu 1.799 total
wget http://202.49.71.58:24/testfile.zip 0.07s user 0.61s system 38% cpu 1.784 total
that's how it compares for core2duo with the old version for me
with gigabit
(the old version always writes to /tmp/testwrite which needs to be on tmpfs)
./newmicrocurl http://202.49.71.58:24/testfile.zip > /dev/null 0.00s user 0.57s system 31% cpu 1.798 total
and that's new version
so new version behaving the same
what cpu usage difference do you get to my url? gizmoguy: how big is testfile? mercutio: 200 megabytes
it doesn't have 10 gigabit ethernet
but it has gigabit to APE gizmoguy: cpu usage up to 10% or so mercutio: it's not actually a zip file it's /dev/urandom gizmoguy: but I occasionally see it spike to 98% for your process
so maybe that's why it's slow mercutio: weird
i wonder why
beacuse i'm testing on e3110
what kind of speed you getting?
e3110 is basically ecc version of core2duo gizmoguy: hold up, just heading out for some tea mercutio: orig i found a slow down with adsl cos i was checking time too often / outputting to the screen too mcuh
but i fixed that
fixed the 32bit bug :/
by not linking against musl
oh actually no
it is still there
it's bloody 64 bit
integer should be 64 bit
ok now i fixed that
it seems that int is 32 bytes on 64 bit architectures still.
if i strace wget and my microcurl, wget does read of 8k and two writes of 4k, and microcurl does reads of 7300 and writes of 7300 bytes, but sometimes 14600
where 7300 is 1460*5
so if wget is using less cpu i must guess that it's due to getting less packets in a burst
and doing read/write of single packet or such ***: r0ni has joined #arpnetworks gizmoguy: mercutio: uint64_t :) mercutio: oh
is "long" wrong? gizmoguy: long works ok too mercutio: i haevnt' done much coding recently gizmoguy: but uint64 is how to get a 64bit integer
don't use int mercutio: i am also using %uz in printf
i assumed int was 64 bit now gizmoguy: nah, backwards compat yo mercutio: well i see that now :)
oh interesting
most of my programming was 15 years ago
not a lot has changed since then though.- gizmoguy: an integer is probably defined as a 32bit number somewhere in the C standard no doubt mercutio: but 64 bit is one of things that changed :)
i updated the url
err the code at the url
but it's purely for showing accurately
i'm still curious why it using so much cpu :/
oh, and i stalked you on linkedin :/ gizmoguy: :)
add me if you want mercutio: ok gizmoguy: go to NZNOG? mercutio: nope
i was thinking of going next year gizmoguy: lame mercutio: cos of SDN etc :/ gizmoguy: come on down mercutio: but it's expensive for flights
and accomodation and so forth gizmoguy: true mercutio: and i doubt my work would pay gizmoguy: work pays for everything for me :) mercutio: yeah if work paid for everything i probably wouldn't think twice about it gizmoguy: but yeah, it's a great place to have an informal chat with your upstreams over a few beers mercutio: but really it's on the interesting side rather than useful side gizmoguy: few (read: a lot of beer) mercutio: heh gizmoguy: I find the networking more useful than the talks
the talks are all live streamed anyway mercutio: yeah the talks themselves looked mostly boring gizmoguy: by richard @ r2 mercutio: i see it more of a social thing gizmoguy: yeah it is mercutio: i actually watched some wand thing
and it wasn't nearly technical enoguh
but it did suggest you were using old cheap hardware
and it seems you still are :) gizmoguy: lol
remember what the talk was on? mercutio: umm
it had a few things gizmoguy: we can go a lot more technical offline mercutio: one of them was about monitoring i think ? gizmoguy: but talks are usually fairly informational mercutio: and debugging problems? gizmoguy: yeah it'll be AMP
our "active measurement platform" mercutio: ahh ok gizmoguy: basically we have nodes in most ISPs in NZ mercutio: i want to do my own network monitoring stuff gizmoguy: and do DNS/ICMP/Traceroute/HTTP testing mercutio: i've got many ideas :) gizmoguy: and we publish them all to http://erg.cms.waikato.ac.nz/amp mercutio: like that you should be able to esimate bandwidth without using up lots of bandwidth gizmoguy: though all that code is ancient and icky
so we have a government grant to rewrite it all mercutio: but lots of ideas can still take a long time to actually put into effect :) gizmoguy: and we're adding event/anomaly detection mercutio: well i got interested when i emailed truenet gizmoguy: truenet suck balls mercutio: and their responses were like OMG
exactly
like they're single threading wget gizmoguy: we actually went for that contract
but we were too expensive mercutio: you need multiple tcp connections to represent web page downloads gizmoguy: but at least our results would have meant something mercutio: but you also need to statr them at realistic times
rather than just 8 at once or such gizmoguy: they also were recently doing all their bandwidth tests on 200kb files loaded off trademe.co.nz mercutio: which means you want to download, do parallel requests, request new data when you see the actual urls
etc gizmoguy: they've now moved to 1MB files because of UFB mercutio: i do most of my bandwidth tests on 200kb files
i just confirm with 10mb files
200k often shows if a connection is good or bad gizmoguy: but they report on stuff like average connection speed :/ mercutio: but anyway, like the way i see it is there's thresholds
it doesn't really matter if it's 10% faster or 10% slower
but if it takes 2 secodns or 3 seconds to display a web page it's significant
yeah but average to where?
i also herad that there's some messy routing stuff involved
i've got a few vps's with test downloads gizmoguy: so we do full mesh testing mercutio: international gizmoguy: where we can mercutio: ahh interesting
so do you have many locations? gizmoguy: each node to every other node
we don't do bandwidth tests at the moment though mercutio: just ping? gizmoguy: because amp has been around 10 years or so mercutio: what i really want to test is tcp latency gizmoguy: and people didn't like us using "all" their bandwidth mercutio: which is kind of messy
like ping != tcp latency necessarily gizmoguy: I think we have around 30 sites or so mercutio: ahh ok gizmoguy: so you'd like our HTTP test mercutio: yeah i think i saw it once
you have soem kind of traecrouet from many hosts thing?
or something
some kind of network path thing gizmoguy: yup
http://erg.wand.net.nz/amp/graph.php?src=ampz-auckland&dst=ampz-fx-aknnr mercutio: i think http can behave differently
i think a lot of things though
i'd like more real world data :) gizmoguy: scroll to the 'path analysis' graph mercutio: taht's a graph
i saw some uhh
something that showed all the different upstreams
andi t ihnk something else gizmoguy: could be TR mercutio: http://tr.meta.net.nz/tr.php
maybe this is nunrelated gizmoguy: one of our researchers wrote it
for fun mercutio: ahh ok gizmoguy: perry mercutio: well it's cool gizmoguy: he works for the google now mercutio: ahh ok
adhh yeah it shows the image afterwards gizmoguy: my most active URL on wand.net.nz is still this: mercutio: of the different upstreams gizmoguy: http://wand.net.nz/~perry/max_download.php mercutio: oh god gizmoguy: it kinda became the industry standard TCP calculator heh mercutio: can that be updated?
i hate that tool gizmoguy: hahaha
he wrote it in a night mercutio: i hate it when people argue about tcp limitations
when it's all old data
yeah but i mean
cubic is VERY VERY common now
and bic is around a tiny bit
and windows has new stuff
that i haven't really played with
and linux has all these improvements
i think they even decreased the initial timeout down from 3 seconds
back when BUBA was common in new zealand i looked into some of those things a little
because there was so much packet loss that it was interesting to see how badly things dealt with it
and what could be done to improve things
and decreasing that 3 second timeout makes a very significant difference in ~5% loss links
that said, i put it down myself before linux did :/ gizmoguy: :) mercutio: cos i was looking at openbsd source code for tcp/ip
and i adjusted some stuff
then i thought that was easy
so i try doing the same on linux
and i'm like omg
i swear, if you want to look at how tcp/ip works in the kernel it's easier to understand with openbsd than linux by a LARGE margin
and it also compiles faster :/
well it helps that openbsd doesn't have cubic etc
it just has newreno
but yeah are you looking at doing your own truenet type testing?
the other big problem i saw with truenet is that they only want to test idle connections
and i think it's way better to just test all the time, and expect that there's noise
like if i'm playing a game, and someone downloads a 1mb file, ond my game gets laggy, i'd rather that be reported
and different aqm on different connections can influence positively/negatively gizmoguy: we are effectively replacing truenet yeah
we have a ISP partner who wants to drop all our software on embedded linux boards and put them on customers internet mercutio: cool
shouldn't it be isp agnostic though? gizmoguy: so we partner with local ISPs for testing the stuff we write mercutio: oh right
so the isp provides the hw
but you provide the software? gizmoguy: it's more the data they provide us
as writing network monitoring software with a network is hard :)
err
without a network* mercutio: with or without a network is hard :) gizmoguy: lol
yeah especially without customers mercutio: yeah the more i thought abuot it the more compilcated it got
but i was planning on trying to do something myself
i figure you should be able to stick something on something ilke openwrt pretty easily
but that you could get wider uptake by making it work on a windows desktop gizmoguy: we have a wandboard i'm looking to get our monitoring software running on
heh
we tried that mercutio: how'd it go? gizmoguy: http://nettest.wand.net.nz/
nobody really ran it
so we kinda abandoned it mercutio: is your stuff open source? gizmoguy: some of it is
the stuff NBIE pays for, we don't typically release the source code for mercutio: haha
why am i trying to download the windows version on linux
oh i have wine :) gizmoguy: :)
all stuff we develop inhouse is open source
Libtrace is our most popular tool we write mercutio: cool
is it bsd licensed? gizmoguy: http://research.wand.net.nz/
I think we standardise on GPL mercutio: ahh gizmoguy: have a look at scamper too mercutio: i dunno if lenny is the most recent or not gizmoguy: scamper is awesome
libprotoident is cool too mercutio: but it'll probably work i imagine gizmoguy: we can identify ~140 different applications purely on 4 bytes of the packet mercutio: oh it's running now what haha gizmoguy: http://research.wand.net.nz/software/libprotoident.php mercutio: nice
that is 4 initial bytes?
or within each pcaket? gizmoguy: "Unlike many techniques that require capturing the entire packet payload, only the first four bytes of payload sent in each direction, the size of the first payload-bearing packet in each direction and the TCP or UDP port numbers for the flow are used by libprotoident" mercutio: ahh so first four
hmm gizmoguy: one of the other things we do at wand is take a lot of network traffic dumps
cept most ISPs don't like giving you full payload
so we standardised on the first 4 bytes
to keep it anonymous mercutio: cool
i like that gizmoguy: and no chance of password leaks etc
also it's easier to store :) mercutio: yes
is it very compressable? gizmoguy: but libprotoident lets us look at what protocols people are using mercutio: umm
so it's full tcp header gizmoguy: http://wand.net.nz/wits/waikato/8/ mercutio: with 4 bytes data? gizmoguy: download some traces and try it out :)
Snapping Method Packets truncated four bytes after the end of the transport header, except for DNS mercutio: dude
try xz compression
i'm not downloading this direct to hme :/
home gizmoguy: come on mercutio: heh gizmoguy: we split them into small tarballs for ya mercutio: what
6 gb gz? gizmoguy: yeah mercutio: on adsl :) gizmoguy: lame
I got vdsl at home mercutio: oh what it's ftp gizmoguy: protocol of the future mercutio: hmmm
it's downloading slow too gizmoguy: do you not have v6 at home? :O
We rate limit ipv4 access
to 'encourage' other researchers who want access to our datasets to move to ipv6 mercutio: i had v6 at home
oh
i'm not downloading to hoem
hangon i'll go via ipv6
i find ipv4 works better :/
i suppose you get to be impractical :) gizmoguy: haha yup! mercutio: 30megabytes/sec now
40
50
60
55
it ramped up slow
80
i wonder if it's disk :)
0 6203M 0 27.1M 0 0 1184k 0 1:29:24 0:00:23 1:29:01 1216k^C
72 6203M 72 4476M 0 0 66.5M 0 0:01:33 0:01:07 0:00:26 84.0M
kind of a big difference between thee two :) gizmoguy: haha, one v4 one v6? mercutio: yeah
i thought soemthing was broken
is that gigabit international?
and hangon, that's faster speed than your core2duo got
curl -O ftp://wits.cs.waikato.ac.nz/waikato/8/20111104-000000-0.gz 5.20s user 26.06s system 33% cpu 1:32.05 total
oh hangon, you had 800 megabit, not 80mb/sec didn't you
y'know if i didn't have to pull that file back home i wouldn't really complain about you using gz :) BryceBot: <mercutio> hh halghl, yhu hal 800 megabut, lht 80mb/sec lull't yhu mercutio: what
tracestats: error while loading shared libraries: libtrace.so.3: wrong ELF class: ELFCLASS32
hmmmm
oh fixed
i hate how ubuntu doesn't include /usr/local/lib in search path by default
but that's a really bizzare error message for such
ohh
i haev a 32 bit version of that library installed by that test client thingy m0unds: boy, making serial cables is fun mercutio: old school :)
oh
i can just use tcpduump
rather than get a 6gb dump m0unds: yea, we use them for control surfaces
but we have no spares, so i used the cable kits provided with the controllers and made 4 9-25 null modem cables out of cat 6
hahaha gizmoguy: right home time I think
see you suckers later m0unds: later man gizmoguy: i'm going home to build a network for a lan this weekend
7 juniper switches :P m0unds: EX3200s? mercutio: heh
you just have 7 juniper switches lying around? -: m0unds wants to play with J gear m0unds: i wish i did, i just have a stack of 2960s gizmoguy: mercutio: heh yup mercutio: i hate 2960s gizmoguy: 6x EX2200s m0unds: nowhere near as much as i do gizmoguy: and 1x EX4200[20:15] <m0unds> figured i'd guess middle of the road :) m0unds: j switches are so nice gizmoguy: the 2200s are pretty basic, but i'm just using it as the access network m0unds: yeah gizmoguy: the EX4200 is a dream to work on mercutio: i wouldn't mind 7 ex4200s gizmoguy: and i have an M320 for the lulz in storage m0unds: i need to get a 2200 or 3200 for home gizmoguy: 2200c if you want to get the small one m0unds: yeah
i'd probably shoot for 24 ports mercutio: for home? m0unds: yep
prewired house, lots of devices mercutio: i'm using 4 gigabit ports at home
and 4 10/100 m0unds: i'm using 13 mercutio: err that's how amny ports i haev at least m0unds: and 3 10/100 on my SRX210HE mercutio: and one of them connecs the two together
i do have an unused mangaed switch
but i dunno how you'd end up with 24 devices at home :)
ahh ok, so 24 makes sense m0unds: yeah, more than 12
if it was less, i'd go for 12
for sure
because $$$
hahahaha mercutio: heh
you could always go dual 8 port switches :/ m0unds: nooooo mercutio: haha
one managed one unmanaged m0unds: nooooo mercutio: i thought you were trying to be chaep m0unds: not that cheap mercutio: heh m0unds: bang for the buck mercutio: yeah
just get a cheap 24 port switch :/ m0unds: i have a cheap 24 port switch mercutio: go cheaper than juniper?
oh m0unds: yeah mercutio: you haev a 29600? m0unds: not at home mercutio: err 2960s m0unds: at work mercutio: 2960g
but 2960s would work too
2960 takes up lots of power m0unds: mine are WS-C2960S-48FPS-L
so, ~900W with full POE load mercutio: it'll be like 150w or more without poe :/ m0unds: yeah mercutio: hence lots of power :) m0unds: they're also loud, and hot
and crappy
i hate cisco gear mercutio: gcet something with energy efficient ethernet for home maybe?
so do i
GET A CLOUD CORE ROUTER
(kidding) m0unds: hahahaha mercutio: i need to have hard-disk storage at home aagain
i've been living off just ssd staticsafe: mercutio: you mean like this - http://routerboard.com/CCR1036-12G-4S mercutio: but i'm like where am i going to put this packet capture heh
static: yes
you get to run a beta OS m0unds: yep staticsafe: i have a routerboard as my home router m0unds: hell, if you buy the middle of the road RB1100AH you end up with two or more ports that just don't work correctly
and they'll tell you as much mercutio: at least that includes a switch m0unds
cloud core router doesn't even have a swiitch m0unds: that's being generous mercutio: every port goes throguh the cpu m0unds: hahaha mercutio: yoeu get two switches i think
with different mtu limits etc staticsafe: the lower end routerboards are perfect for home use tbh m0unds: i had an rb450g and it died a hero's death (bad caps, bad thermal mgmt on the cpu) staticsafe: compared to the other garbage on the market m0unds: sure mercutio: xz is so damn slow to compress
--- % 523.0 MiB / 2693.7 MiB = 0.194 2.1 MiB/s 21:32
i think it's about 1/6th of the way through?
so like two hours or something
it was half an hour with -1
and still 50% better than gzip ratio
err 33% better ***: ese has joined #arpnetworks gizmoguy: m/ my machine from last year's lan still boots brycec: good grief! I have never had so much backlog to catch up upon in this channel milki: you dont type /clear enough brycec: I could just ignore it... and I am. It overran my buffer :p
(And I never type /clear when I'm afk :P) gizmoguy: brycec: sorry
me and mercutio worked out we live an hour from each other brycec: haha no worries mate :) mnathani: up_the_irons: Have you considered licensing your automation and various tools for Virtual Machine provisioning to your competition? mercutio: gizmoguy:
i'm still trying to get libtrace to work :/
it doesn't like my lack of encapuslation
i found i had an old dump of my own connection around
being the geek i am :/
reading from file ben, link-type NULL (BSD loopback)
but it's in that format
ben: tcpdump capture file (little-endian) - version 2.4 (No link-layer encapsulation, capture length 128)
i suppose i should at least be happy there's source so i can try and figure out how it works :)
mnathani: why should his compettion have them?
also it probably would mean he'd end up with heaps of questions
and even less spare time gizmoguy: mercutio: report it to me tomorrow and I'll raise a bug for you :) mercutio: ahh real gizmoguy: i'm busy drinking and configuring linux mercutio: those don't go tegether do they m0unds: they sure do mercutio: i thought i should be able to figure out how to do it :) gizmoguy: they go very well together
i even managed to partition a disk correctly!
well, ask me tomorrow if i configured it correctly m0unds: it's the only way to fly mercutio: cos it looks like openbsd loopbackup, but with host byte order rather than network byte order from what i gather mnathani: mercutio: Just a thought for alternate means of revenue generation, since the code has already been written and tested mercutio: mnathani: have you seen the solusvm forum? m0unds: gizmoguy: i forgot to ask, are you on untappd? mercutio: gizmoguy: you remembered to use gdt? gizmoguy: m0unds: oh man, I should join
everyone else I know does it
mercutio: gpt? mercutio: partition table
guid partition table it seems
use gdisk to do it
doesn't have the silly 4 parition limit
or extended partition hack gizmoguy: I use parted mercutio: dunno if that does it or not
ahh parted does support gizmoguy: yeah it does mercutio: if you have modern computer yo ucan use uefi too ***: ese has quit IRC (Ping timeout: 245 seconds) mercutio: oh it doesn't like infiniband either :)
unless i broke it
yeah doesn't like ib gizmoguy: I'm not sure we've ever tested with IB :) mercutio: yeah
it's nice how there's so many different formats :/
like it probably works on freebsd on ethernet gizmoguy: it should work on freebsd
we test on lots of different platforms
osx support too :P mercutio: it's ppp on freebsd though
yeah it works on ethernet
although dunny why EHLO is Invalid_SMTP
it actually seems to work with most stuf
i want to make graphs i think ;) ***: ThalinVien has quit IRC (*.net *.split)
tabthorpe has quit IRC (*.net *.split)
r0ni has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
ese has joined #arpnetworks
ese has quit IRC (Ping timeout: 272 seconds)
ese has joined #arpnetworks