#arpnetworks 2014-02-27,Thu

BryceBot

Why did the chicken cross the road?;To get to the other side., (ha, ha)

mnathani

@wa why is the sky blue?

BryceBot

Why is the sky blue?;The sky's blue color is a result of the effect of Rayleigh scattering. Shorter-wavelength blue light is more strongly scattered in the earth's atmosphere than longer-wavelength red light; the human eye perceives the color blue when looking at the sky as a result.

mercutio

@wa why do we exist?

BryceBot

Couldn't grab results from json stringified precioussss.

mercutio

mercutio wonders what issue happened an hnour ago

mnathani

@wa when will ipv6 reach widespread adoption

BryceBot

Couldn't grab results from json stringified precioussss.

mercutio

well it wasn't on arp
but from my local monitoring quite a few things experienced packet loss then without me noticing

mnathani

@wa will there come a time when ISPs will hand out IPv6 addresses and tunnel IPv4 traffic over it

mercutio

haha

BryceBot

Couldn't grab results from json stringified precioussss.

mercutio

@wa tldr

BryceBot

TLDR (acronym);too long didn't read

mercutio

mercutio goes back to tv

[00:23]

.... (idle for 17mn)

***

Guest40141 is now known as easymac
easymac is now known as Guest45464

[00:40]

alienresidents has quit IRC (Ping timeout: 240 seconds)
alienresidents has joined #arpnetworks

[00:50]

gizmoguy_ has quit IRC (Ping timeout: 245 seconds)
grepidemic has joined #arpnetworks

[01:06]

gizmoguy has joined #arpnetworks

[01:13]

gizmoguy

mmm whisky
mercutio: nah, we've not done anything with netamp
we're looking to have a play with netfpga in the near future though

[01:17]

..... (idle for 24mn)

***

Guest45464 is now known as easymac
easymac is now known as Guest55282

[01:41]

..... (idle for 21mn)

mercutio

dunno what gave me the idea that you did hmm

[02:02]

gizmoguy

we used to run a thing called nettest
and our main software we're developing at the moment is called amp (http://amp.wand.net.nz/)
nettest + amp = netamp? :P

[02:06]

mercutio

nice
i want some nice way to trace from multiple difference places and record what networks it traverses
and identify where things break
when arp had issues with ntt and any2ix i didn't have much luck finding a path that didn't use those two in a short space of time manually

[02:15]

gizmoguy

we can pretty much do that

[02:17]

mercutio

cool.

[02:17]

gizmoguy

assuming you build your test schedule properly

mercutio

well routes can change over time
i see path length interesting

gizmoguy

best way to visualise an outage like that would be our matrix - http://amp.wand.net.nz/matrix/absolute-latency/both/nzamp/nzamp/

mercutio

oh it still hard to tell if it changes by a little
the site is kind of slow from here
dunno if it backend or what

gizmoguy

sorry for speed. i have a new major database improvement to push out shortly

[02:19]

mercutio

ahh ok
lambda is broken?

[02:19]

gizmoguy

lambda is in the US :)

mercutio

how often does it update?

gizmoguy

everything else is in NZ

mercutio

oh red is < 300 msec and > 160
rather than > 300 msec

gizmoguy

if you look at relative latency rather than absolute, you'll get a better picture of changes
our postgres database is a bit broken at the moment
~200million rows of data in a very naive schema

mercutio

mada doesn't measure that well on that either

[02:21]

gizmoguy

I've got all that fixed, just need to convert all the data and push out

[02:21]

mercutio

ahh right
so does this need lots of resources to run?
actually the loss thing is prob fine
nah i see vocus is the borken one hah

[02:21]

gizmoguy

vocus is really broken
I'm not sure what they do to our AMP monitor machine
but we see packet loss to everywhere

mercutio

rate limit icmp?

gizmoguy

forwarded ICMP?

mercutio

maybe

gizmoguy

as we see packet loss end to end

mercutio

on icmp?

gizmoguy

yeah

mercutio

or tcp/udp/etc?

gizmoguy

this is my favourite graph type - http://amp.wand.net.nz/view/amp-traceroute-rainbow/9238/1393323647/1393496447

mercutio

i saw a route going via cogent drop icmp completely once
from a host doign monitoring

gizmoguy

shows up broken routing quite quickly

mercutio

it was very bizzare :/
trademe bounces between wgtn/akdl a lot

gizmoguy

note - trademe have the worst high availablity setup ever
a lot?

mercutio

contsantly

gizmoguy

try every 10 minutes

http://202.49.71.24:24/smokeping/smokeping.fcgi?target=Wider.trademe
yes

mercutio

gizmoguy

i.e their TTL on their A record

mercutio

1 msec to akld 10 msec to wgtn

http://amp.wand.net.nz/view/amp-icmp/2415/1393384891/1393476724

gizmoguy

[02:25]

mercutio

but it pretty consistent
3i think it similar behaviour really

[02:25]

gizmoguy

yeah, their DNS server just randomly selects from AKL or WLG whenever you query it
then it gets cached for the TTL

[02:25]

mercutio

yeh
i know it's mental
they should just do anycast
imo

gizmoguy

i'm amazed they can share state so well between their datacentres

mercutio

if they have a real outage on one of them, then people will reload page anyway
they don't afaik

gizmoguy

how do they keep you logged in?

mercutio

it's basically just a proxy type setup
because it goes to the same backend constantly
just with a cookie i think
not based on ip or anything

gizmoguy

yeah, but the cookie is a session id
that the backend needs to know about
hence they must do constant replication between each site to keep session IDs in sync

[02:27]

mercutio

yeh well it goes back to the db probably
what do they need to keep state on anyway?

[02:27]

gizmoguy

that's why it's generally advisable to direct the same group of people to the same set of frontend/backend servers
your session

mercutio

hmm

gizmoguy

because the cookie doesn't have your username/pass in it
it has some secondary auth (like a session ID) to verify you've logged in

mercutio

maybe they replicate

gizmoguy

yeah they will

mercutio

i dunno really

gizmoguy

very very often

mercutio

i had a slow image loading issue on one of them once
it was fine on one bad on the other
i tried contacting them to no avail :/
but i don't see why they don't just cdn their images
and direct all their normal web traffic to one location

[02:29]

gizmoguy

yeah
did you watch any of the NZNOG videos btw?

[02:29]

mercutio

they don't even really need to cdn their traffic probably

gizmoguy

from janurary

mercutio

a couple

gizmoguy

the GCSB one?

mercutio

i watched fincham's one on rpki

gizmoguy

a+++++ for the GCSB one
so funny

mercutio

and the apnic one
i dunno there was terrible video/sound quality
and most of the talks sounded pretty uninteresting :/

gizmoguy

yeah the richard naylor couldn't be there this year (the dude who normally does the streaming)

mercutio

i liked the idea of 1.2.3.4 as standard anycast dns

gizmoguy

the GCSB one was awesome

mercutio

and i think the idea of rpki is slightly interesting

gizmoguy

watch the Q&A
geoff houston's talk was awesome as usual too
was chatting to him at the pub the night before

mercutio

i dunno bad audio quality is common for most talks
oh god
i watched some of the one by that guy about not keeping state
i think that's hwen i stopped heh
i already know about state issues etc

gizmoguy

haha, Roland?

mercutio

yes

gizmoguy

I feel asleep during that one

mercutio

dibbins or something?
heh

gizmoguy

it was a good message
but it didn't need to go on for what felt like 60 minutes

mercutio

well before hand i ntoiced he'd give the same exact talk before
and then i noticed the slides were the same
it was kind of amusing

gizmoguy

he is mr stateless firewall

[02:33]

mercutio

i'm in between
you need state for NAT :/
i hate nat
but snyhcronising is a pita
and arp's route just changed
did ntt go down again
(i left mtr running)
any2ix and ntt are down it looks like
but it maybe up_the_irons doing ios things :/
cos it didn't generate alerts etc.

[02:33]

***

notion has quit IRC (Read error: Connection reset by peer)
twobithacker has quit IRC (Read error: Connection reset by peer)
Guest1774 has quit IRC (Read error: Connection reset by peer)

mercutio

so this monitoring thingy is it light on clients?

***

twobitha- has joined #arpnetworks
notion has joined #arpnetworks

mercutio

netamp
and does it require lots of ram on server?

gizmoguy

mercutio: yup

***

Guest1774 has joined #arpnetworks

gizmoguy

we have plans to run on embedded devices

mercutio

cool

gizmoguy

it's just a little c program

mercutio

sweet

gizmoguy

and you can tack on rabbitmq if you want proper persistance (but requires 50mb of memory)

mercutio

is it ready for testing now?
on the server or client?

gizmoguy

the server is closed source at the moment (because of funding requirements / commercialisation potential)
but the client app is pretty open

mercutio

oh but is it available for other people to run?

gizmoguy

we try and run it on as much as we can
but we have a few trials soon on a few NZ ISPs

mercutio

i suppose it prob not then

gizmoguy

to trial monitoring a large residential ISP core

mercutio

interesting

[02:38]

gizmoguy

mercutio: http://wand.net.nz/amp/
this is the link we gave out at NZNOG
for large ISPs we're giving them 1U dell servers to run as monitors
for smaller ISPs we're looking to give out embedded boxes to handle the monitoring
and we should have all the performance issues fixed up by the end of march

[02:38]

mercutio

so it'll stick all of the isp's in a list?

[02:39]

gizmoguy

we have a 4 year MBIE grant to develop the software to the stage of being able to monitor NZ's internet

[02:39]

mercutio

rather than individuals being able to monitor different sites?

[02:39]

gizmoguy

we're working towards ISPs being able to pick and choose what the monitor

mercutio

tbh, nz internet seems better than overseas internet for domestic traffic

gizmoguy

yeah, it's really not too bad

mercutio

but i mean one user can't have 5 different sites that all monitor each other

gizmoguy

though we find some pretty broken stuff now and then

mercutio

other than silly people like you rate limiting to 10 megabit :/
you mean like routing via australia?

gizmoguy

So, the most recent funny one

mercutio

i suppose there's less infrastructure to go wrong in nz

gizmoguy

REANNZ <--> Callplus didn't work

mercutio

that should be able to go over ape

gizmoguy

their routers both said they should talk to each other over APE

mercutio

in both directions

gizmoguy

but on the APE layer 2, they couldn't talk

mercutio

what
why not

gizmoguy

so it was a blackhole between them

***

Guest55282 is now known as easymac
Guest1774 has quit IRC (Read error: Connection reset by peer)
twobitha- has quit IRC (Read error: Connection reset by peer)

gizmoguy

we logged into reannz's APE router

***

twobithacker has joined #arpnetworks

mercutio

bloody route servers

gizmoguy

ran ping [callplus ape ip]

mercutio

should directly connect :/

gizmoguy

and it started working :)

mercutio

oh god

gizmoguy

yeah....

mercutio

ubergroup had an issue like that before
except it'd start working

***

easymac is now known as Guest95370

gizmoguy

it hasn't fallen over again yet

mercutio

they're running vpls or something from there?

gizmoguy

probably because we run probes once every 30 seconds
and make sure the ARP table never expires :P

***

Guest1774 has joined #arpnetworks

mercutio

ubergroup would drop the first few pings to them always
then start to work

gizmoguy

also, callplus had a few ipv6 related issues we got fixed up

mercutio

callplus have ipv6?

gizmoguy

yeah

mercutio

did you hear about the i217-lm issues?

gizmoguy

nah?

mercutio

apparently i217-lm have bugs
and disabling ipv6 makes them stop breaking switches

http://www.edugeek.net/forums/hardware/132287-optiplex-9020-systems-spewing-ipv6-multicast-traffic-while-asleep-causing-havok.html

gizmoguy

s/i217-lm/everything

[02:44]

BryceBot

<mercutio> apparently everything have bugs

[02:44]

mercutio

and i217-lm are on standard hp and dell hosts these days
apparently disabling power saving stuff might fix it

stuff like that

[02:44]

gizmoguy

:( sigh

mercutio

there's a few things floating around

gizmoguy

we use a lot of dell at work
haven't hit that one yet

mercutio

older ones probably
i217-lm is on haswell ones afaik

gizmoguy

all our dells come with broadcom chips
but I buy proper intel server NICs for all of them

mercutio

i217-v is integrated on consumer desktop boasrds with haswel onwards
broadcom isn't terible

gizmoguy

it really is

mercutio

it's a lot better than it used to be
for a desktop?

gizmoguy

depends on the broadcom chipset I guess

mercutio

my windows computer at home has onboard broadcom

gizmoguy

we have some benchmarks of consumer broadcom vs consumer intel NICs
for iperf

mercutio

i can do 972 megabit/sec
iirc

gizmoguy

intel requires 20% less CPU usage (less spurious soft interrupts) for a given iperf run

mercutio

it's something close to that if not that
hmm

up_the_irons

mercutio: you probably didn't see: https://twitter.com/arpnetworks/status/438949834670612481

BryceBot

TWITTER: We will be upgrading the IOS on our s7.lax router between 02:00-02:30 PST; expect NTT and Any2 IX to go down (Thu Feb 27 08:12:50 +0000 2014)

gizmoguy

up_the_irons: it's only 12am here! :)

mercutio

ahh
no i didn't up_the_irons
but i remember you said something about ios

up_the_irons

yeah

mercutio

and doing osmething when the first issue came up

up_the_irons

happy to say s7.lax is now running a shiny new IOS

mercutio

is it going to crash less now? :)

gizmoguy

crash less or more, hard to say :)
need more data points

mercutio

gizmoguy: 20% less cpu usage on iperf?

up_the_irons

i give it 50 / 50 chance. this will rule out IOS issues, but there is still a possibility of bad hardware

mercutio

20% cpu usage on iperf for gigabit is pretty high
that's what my core2duo does :/

gizmoguy

mercutio: I believe it is because intel do a lot better with coalescing (http://www.intel.com/support/network/adapter/pro100/sb/CS-032546.htm)

up_the_irons

i actually ordered a new sup for s7 today

mercutio

gizmoguy: i dunno they both suck with udp traffic :/

up_the_irons

if everything ends up being OK, then it'll just be a spare
otherwise, it'll replace the current one

mercutio

but yeah it could be
i went from 40% to 20% cpu usage from i7-3770 to i7-4770 for infiniband
well ip over infiniband
at over 10 gigabit

gizmoguy

yeah

mercutio

i was surprised at the difference

gizmoguy

it is quite amazing
how much poor interrupt handling can screw you

mercutio

but it's the same card in both machines

gizmoguy

just better interrupt sharing?

[02:50]

mercutio

i dunno, the intel coalescing is better though
yeh these infiniband careds are terrible for that
i can generate 80,000 interrupts a sec iirc
but at high interrupt loads different cpus/cache etc can mkae more diff

[02:50]

gizmoguy

we tend to tune coalescing / MSI-X / irqbalance on any machine we care about doing throughput tests on

mercutio

and tcp offload stuff is way better than udp offload stuff
i tend to disable irqbalance
what tuning do you do?

gizmoguy

yeah it's all about sending irqs to the right CPU cores
my notes are at work

mercutio

i was trying to figure out if there was anyway i could reduce cpu for infiniband

gizmoguy

rather - the student I make tune all my machines is at work :)

[02:52]

mercutio

yeah single cpu here
so it shouldn't matter afaik
so all the pci-e lanes on the same cpu
but yeah, i think i was going somewhere
but i can't remember where
gigabit isn't too complicated
it's 10 gigabit+ where things get complicated

[02:52]

gizmoguy

anyway. i should sleep. got a meeting in 9 hours from now and I need my beauty sleep :)

mercutio

heh

gizmoguy

people who schedule meetings for 9am friday should be shot

mercutio

what's that embedded monitoring device btw?
heh

gizmoguy

we're trialing a few

mercutio

i should stick app in :/

gizmoguy

I have one of these on my desk - http://www.freescale.com/webapp/sps/site/taxonomy.jsp?code=IMX6X_SERIES

mercutio

but it'll be small

gizmoguy

and we've also ordered one of those new intel embedded thingys
but it hasn't arrived yet

mercutio

ahh ok

gizmoguy

this is a better link for the freescale thing - http://www.element14.com/community/community/knode/single-board_computers/sabrelite

mercutio

i dunno why can't just do a virtual machine

gizmoguy

we're working on it
just have a few timing related issues to solve first

mercutio

ahh ok

gizmoguy

time drift is a big problem for us
and time drift occurs rather frequently in vm environments

mercutio

i see
well if it's small it fine anyway

gizmoguy

so, we're just building a list of common "correct time settings" for all the popular VM environments
to serve as a minimum set of requirements to host our VM image

[02:56]

mercutio

i've only seen clock drift on vmware

[02:56]

gizmoguy

and parallels

[02:56]

mercutio

oh that the windows one

gizmoguy

macosx :/
we had some people running it
it was soo bad

mercutio

haha
virtualbox!

gizmoguy

i'll let you in on a secret (lambda is already AMP running on a VM)

mercutio

gizmoguy

because I don't care about accurate timing myself

mercutio

haha i guessed it would be
but the funders do

gizmoguy

but unfortunately working for an academic instute, we get in trouble for not considering the 'science' aspect :P

mercutio

i see
smokeping has way higher latency with ping to localhost on xen

gizmoguy

so we just have to do some timing verification to prove it's not bad
then we can deploy it

mercutio

it jumps from like 10 usec to 50 usec

gizmoguy

yeah, generally if you use virtualised clock drives you're fine

[02:59]

mercutio

not that 50 usec is a long time
but it's about as much as most ethernet coalescing is for

[02:59]

gizmoguy

like KVMs pvclock - https://rwmj.wordpress.com/2010/10/15/kvm-pvclock/

[02:59]

mercutio

ahh ok
xen in pv mode is probably fine too?

[03:00]

gizmoguy

i'll admit i've not looked at xen

[03:00]

mercutio

i'd rather real hardware
with coalescing disabled
most embedded hardware doesn't even support coalescing though

[03:00]

gizmoguy

http://wiki.xen.org/wiki/Xen_FAQ_DomU#How_can_i_synchronize_a_dom0_clock.3F
maybe? ^
"I have problem with domU clock. It lose 30 minutes each day. How can i synchronize it with dom0 clock? "
yeah, this is why we want to run our tests...

mercutio

maye it does have issues heh

gizmoguy

some VM software is terrible at keeping time

mercutio

i use ntp anyway

gizmoguy

yeah NTP mostly insulates you

mercutio

but i only care for second accuracy
for logs

gizmoguy

mmm

mercutio

gizmoguy

we're technically only worried about drift
we don't care if we have an accurate time

mercutio

well i'll chat with you when you don't need to sleep i suppose :)

gizmoguy

just that 1 second == 1 second

[03:03]

mercutio

ahh
so you need way to measure it
some thernet cards can timestamp too

[03:03]

gizmoguy

yup
PTP
yeah, it's pretty cool
Precision Time Protocol

[03:03]

mercutio

hmm i think you should use udp :/

gizmoguy

lol

mercutio

with traffic volumes
slingshot prioritise icmp i heard :/
because otherwise people complain about high pings

gizmoguy

lol

mercutio

i dunno if it true

gizmoguy

our AMP monitor is in slingshot/callplus's core

mercutio

but if anyone is doing that it'd be slingshot

gizmoguy

so we don't have too many issues
also turns out when you tell people you're going to publish your results on a public webpage

mercutio

i think it's more that they rate limit some traffic and icmp and http aren't rate limited as much

gizmoguy

they give you the best possible connectiviy they can

mercutio

yeh

gizmoguy

sounds right

mercutio

but that isn't
err typical
10 gigabit to the core!

gizmoguy

our machines only do 1gig :(

mercutio

telstraclears performance is shocking

http://amp.wand.net.nz/view/amp-icmp/1779/1393326283/1393499083

gizmoguy

mercutio

but they actually advertise being fast at broadband
or at least they were
based on that silly testing thingy

gizmoguy

callplus probably have the best google connectivity from everyone we monitor
(because we're two P nodes away from their google cache :))

mercutio

what
is that gmail

gizmoguy

youtube.com
red line == ipv6
blue line == ipv4

mercutio

oh
umm
you mean to their local cache?

gizmoguy

yeah google are terrible at deploying ipv6
yeah their local cache

mercutio

www.gmail.com is better
cos it still in sydney
it's bloody difficult to measure these things

gizmoguy

so is www.google.com

mercutio

www.google.com isn't

gizmoguy

if you have questions about google
yes it is

mercutio

www.google.co.nz is a cname to www.google.com
and is in auckland

gizmoguy

you won't find www.google.com hosts in NZ

mercutio

usually
yes you do

gizmoguy

err what IP do you see it hosted on?

mercutio

err you did
it's the syedney one atm
but it's been in nz before

gizmoguy

exactly
nah, it won't
I've had lengthy chats to google frontend engineers about this before

mercutio

nah it did for a while

gizmoguy

they can't serve www.google.com from a cache at the moment
google.com and google.co.nz they can
but now the www. varients
s/now/not

BryceBot

<gizmoguy> but not the www. varients

mercutio

weird
well it was only temp i think

gizmoguy

the short answer is windows XP is the problem

mercutio

stuff shifts around a lot

gizmoguy

the longer answer is SSL is the problem

mercutio

at least vocus and orcon were hosting nz google mirrors last i knew
for a while
on their normal youtube caches

[03:10]

gizmoguy

the GGC (https://peering.google.com/about/ggc.html) nodes I know about in NZ:
callplus / reannz / FX
there are others as well, I just haven't seen them myself
but they can't serve all of google from those cache nodes

[03:10]

mercutio

there is vocus, fx, orcon, snap, with ip's at least

[03:11]

gizmoguy

oh yes, sorry i've seen vocus and snap too

[03:11]

mercutio

i thnk telstraclear had one too
and telecom didn't?
but maybe telecom do now

[03:11]

gizmoguy

telecom's an interesting beast

mercutio

i assume vodafone probably do

gizmoguy

I have no visibility inside it really

mercutio

well they still have terrible email
% host www.google.co.nz alien.xtra.co.nz
Using domain server:
Name: alien.xtra.co.nz
Address: 202.27.184.3#53
Aliases:
Host www.google.co.nz.meh.net.nz not found: 5(REFUSED)
woot
they fixed that at least :/

gizmoguy

lol
from a telecom residential connection

[03:13]

mercutio

telstraclears is still open
actually that's strange

[03:13]

gizmoguy

google takes me straight to SYD

mercutio

www.google.co.nz has address 74.125.237.191
Host www.google.co.nz not found: 3(NXDOMAIN)
with some stuff in between
google sometimes bounces it to somewhere distant too

gizmoguy

alright sleepy time, catcha later

mercutio

night

gizmoguy

if you want an amp monitor just fill out the form

mercutio

gizmoguy

and one of us can get back to you

mercutio

% host www.google.co.nz 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:
www.google.co.nz has address 60.234.81.176
for some reason 8.8.8.8 gives me orcon :/

gizmoguy

try www.google.com

mercutio

same diff
it's also a LOT of addresses

gizmoguy

not for google

mercutio

nah i did it
i mean it showed the same thing

gizmoguy

oh right

mercutio

% host www.google.com 8.8.8.8 | wc -l
22

gizmoguy

and try a curl --head http://www.google.com ?

mercutio

can't
transparently proxied

gizmoguy

cause it might just 302 you to a different server
ah, right

mercutio

i suppose i could
but another time
i'd have to bounce somewhere

gizmoguy

google can be pretty messy :)

mercutio

yes
and email comes from ages away

gizmoguy

yes

mercutio

so do 8.8.8.8 dns requests actually
it you dump on the server

gizmoguy

unfortunately NZ is too small for them to properly fix our connectivity

mercutio

yeh i understand

gizmoguy

which is suprising considering I have two friends who work for the google frontend/traffic team and both are from NZ

mercutio

if they just fixed sydney it'd be ok
searching is faster int he US than sydney

gizmoguy

they just sigh and grunt everytime I report issues

mercutio

if you really want to speed up browsing route to google via US
err searching

gizmoguy

lol

mercutio

it's about 50% faster
or more

gizmoguy

right
I assume SYD datacentre doesn't actually have much searching cached

mercutio

yeah

gizmoguy

so requests go to SYD, then SYD forwards to the US

mercutio

and it goes to somewhre in asia
that prob has less cached than in the US

gizmoguy

or yeah, asia

mercutio

yeh either or
it's faster to go to US from NZ

[03:19]

gizmoguy

the internet is too big :(

[03:19]

mercutio

than from AU
and i think it goes via asia not the US
but yeah
monitoring shit is good
i'd like to see perforamnce measuring of google too :/
of real queries
time curl -v 'http://www.google.co.nz/search?q=nzehrald&oq=nzehrald&aqs=chrome..69i57l2j69i59l2.2448j0j7&sourceid=chrome&espv=2&es_sm=122&ie=UTF-8#nfpr=1&q=nzherald' > /dev/null
like that or such

[03:19]

..... (idle for 22mn)

***

Guest95370 is now known as easymac
easymac is now known as Guest91023

[03:42]

hive-mind has quit IRC (Ping timeout: 240 seconds)

[03:51]

......... (idle for 40mn)

joepie91 has quit IRC (Changing host)
joepie91 has joined #arpnetworks

[04:31]

Guest91023 is now known as easymac
easymac is now known as Guest28839

[04:43]

..... (idle for 23mn)

Guest28839 has quit IRC (Quit: leaving)

[05:07]

................... (idle for 1h34mn)

RandalSchwartz has joined #arpnetworks
RandalSchwartz has quit IRC (Changing host)
RandalSchwartz has joined #arpnetworks

[06:41]

....... (idle for 33mn)

tabthorpe has joined #arpnetworks
m0unds_ has joined #arpnetworks
robonerd- has joined #arpnetworks
staticsafe has quit IRC (*.net *.split)
robonerd has quit IRC (*.net *.split)
abthorpet has quit IRC (*.net *.split)
m0unds__ has quit IRC (*.net *.split)
robonerd- is now known as robonerd
robonerd has quit IRC (Changing host)
robonerd has joined #arpnetworks

[07:14]

staticsafe has joined #arpnetworks

[07:30]

........ (idle for 36mn)

phlux has quit IRC (Ping timeout: 246 seconds)
phlux has joined #arpnetworks

[08:06]

....................... (idle for 1h51mn)

robonerd has quit IRC (Quit: ...)
hive-mind has joined #arpnetworks

[09:58]

......... (idle for 44mn)

easymac has joined #arpnetworks

[10:46]

Guest1774 is now known as pjs
thestereobus has joined #arpnetworks

[10:57]

thestereobus has quit IRC (Quit: thestereobus)

[11:07]

............................................ (idle for 3h38mn)

novae has quit IRC (Ping timeout: 240 seconds)

[14:45]

novae has joined #arpnetworks

[14:50]

novae has quit IRC (Ping timeout: 264 seconds)
novae has joined #arpnetworks

[15:02]

novae has quit IRC (Ping timeout: 240 seconds)
novae has joined #arpnetworks

[15:18]

novae has quit IRC (Ping timeout: 240 seconds)
novae has joined #arpnetworks

[15:32]

raptelan_ has quit IRC (Ping timeout: 272 seconds)
raptelan has joined #arpnetworks

[15:39]

RandalSchwartz has quit IRC (Ping timeout: 272 seconds)

[15:46]

...... (idle for 27mn)

m0unds

just evacuated two more wasps out of my server room
haha

[16:13]

brycec

"evacuated"?
That conjures a specific mental image

[16:14]

m0unds

yeah, threw them in a dvd-r spindle cover and threw them in the parking garage

[16:14]

brycec

ohlawd

m0unds

they're lethargic because it's cold in there
hahaha
so they move verrrrrrrry slow

brycec

Nice!

m0unds

so i just took 'em out one at a time
one had flown away after a few mins in the sun
i'm just happy the guy who pulled this gear from outside put it in the server room and not our closet
because it's 40-something in the server room, but it's 70 in the closet

brycec

sweet, good guy

[16:18]

My new 2560x1440 27" display is gorgeous, and just freaking enormous :D

[16:25]

mercutio

my 27" is kind of small

brycec

twss

BryceBot

Okay! twss! 'my 27" is kind of small'

mercutio

well text i stiny
is
what kind of monitor is it bryce?
like ips or pls or what

brycec

IPS iirc
http://www.monoprice.com/Product?c_id=114&cp_id=11401&cs_id=1130703&p_id=10509&seq=1&format=2

[16:28]

mercutio

yeh ips

brycec

The thing is exceptional. Great picture. The whole backside is solid metal. etc

mercutio

i have one ips one pls

m0unds

whoa, monoprice sells monitors? funny

brycec

lol m0unds yep
Bought this one through Massdrop for $345