I'm seconding the image list issue. Especially with the BSDS, you have to check against what is currently supported. If something isn't supported (9.1 I think is still supported) there's no point in sticking with 9.0 on the list.
Devil's Advocate: But what if a customer wants 9.0? Answer: Then they're the kind of customer that knows how to install it themselves.
Yeah, if you have very specific requirements for 9.0 you should be able to install it.
 7. et-3-1-0.lax22.ip4.gtt.net                                                                                       0.0%    18  174.8 175.4 174.7 182.0   1.8
 8. ge0-arpnet.cust.lax07.mzima.net                                                                                 58.8%    18  174.9 177.3 174.9 182.0   3.1
 9. anycast-arp.mainstreamhosting.net                                                                               70.6%    18  173.3 173.4 173.3 173.6   0.1
whats up?
pl like crazy? :)
goddamnit
yes there is
ddos?
yeh likely
ok better upgrade internet :)
or have good ddos communities aka nullroute :)
Stinking packet loss
yeh havent had pl in my network last couple years
but on the other hand I got 20gb transit with 0.5 % usage :)
again :(
60% loss is amazing
That's what she said!!
my mobile go crazy with sms :) but hey its fun I watch tv soo its okey only 00.30 here :)
not any monitoring knowing destination easy that is attacked and can do nullroute? and provider null it and it goes away?
it is null routed on some providers
this one give 65% pl ge0-arpnet.cust.lax07.mzima.net
still
yeh it's mzima that's broken
shutdown peer there then :) reroute over working route :)
i hate it when my ii bot disconnects from irc when I get to many alerts quickly :)
PROBLEM anycast-arp.mainstreamhosting.net/CPU Load CRITICAL CHECK_NRPE: Socket timeout after 10 seconds. kinda like that one :)
is arp back online now?
Not dead?
urghl i have issues too
But my packets do seem to be lagged
Ill shutdown my anycasting until I wake up as I have a huge campain going and would be not to good if it cant reach the web :)
I got 77% loss on the mzima.net route in still
im on twc rr in ohio - v4 path is rr->level3->giglinx (??)->arp
i freaked out and assumed hardware
im seeing tons of pl
Guess this explains why my vpn can't reconnect. I just figured it was an El Capitain bug as I just rebooted from that upgrade.
my level3 route. is like 60-70% loss too.. its huge.. seems to be major ddos
this dude 4.68.111.17 looks crushed
:(
smokeping has lost its mind, it's hilarious to see the sudden spike
i whipped up a ghetto perl monitoring script this past weekend - this is it's first real test (its actually working as desired...)
http://smokeping.mainstreamhosting.net/smokeping/smokeping.cgi?target=network.anycast-arp.normal this is from london. :)
nagios is a good monitoring tool :)
https://smokeping.cobryce.com/?target=ARP.ARPWebsite this is from 5 different networks (but served from ARP) :p
Thanks up_the_irons https://twitter.com/arpnetworks/status/654797205249757185
TWITTER: We're experiencing a large DDoS attack in our Los Angeles data center. Sorry for the lost packets, we're working on it. (Thu Oct 15 23:13:13 +0000 2015)
brycec: believe it or not, some people still order really old versions because they have legacy apps they claim won't run on newer stuff
any idea if arp might be interested in other locations. Like in europe?
up_the_irons: I still regard it as essentially irresponsible to include unmaintained releases. Besides it's clear that they can install whatever else they need if they're one of those special people.
(That is of course just an opinion, and you're free to go about your life as you wish)
mrsaint: I'm working on a location in  Germany
brycec: to each his own :)
up_the_irons: I have hosting germany netherlands and uk and good really good in .se I can offer good pricing.
up_the_irons: (In case you missed it in the scrollback, I'm not against the images existing, or their ISO's. Just trying to clean up the signup form because wow that list just kept on going and going and going)
I'm definitely happy to see that IPv6 traffic isn't affected nearly as bad as IPv4 in this ddos
tobad 0.00001% use ipv6 :(
>50% of my daily traffic is ipv6
damn
less then 1% of my 4gb used bw is ipv6 :)
brycec: I saw it. People order the old versions, so...
time to get some sleep. up in 6h to work :) Good luck with the ddos guys
Is the DDoS still going on?
there's some mitigation
most routes are fine now
at the same time freenode's been having ddos attacks it seems
so you are saying rackspace is behind this? ;)
Is the target ARP? or is it one of the hsoted clients?
lets blame rackspace for everything.
most notably suckage.
also I don't know if it was network side or server side, but I'm getting some insane server lag.
heh
can't really talk about targets.
sorressean: on what host?
mercutio:  arpnetworks dedicated
oh
probably network then
it's still happening?
I presume if anything was making the box lag that hard it would'/ve had higher load averages, top showed nothing and no swap usage.
it's quite recently it got better
mercutio:  seems to be slightly better. it's spiking now.
maybe a separate issue but if you're seeing loss and high ping spikeyness near the arp network then send in support ticket with traces in both directions
hrm. will do
i think the ddos has actually stopped now
Oh, was there an attack on the network?
Was Linode hit too? I'm trying to hit a site that has two VPS at Linode serving their authoritative DNS and connections are just timing out, confirmed from several other home ISPs. And a third site with DNS from name.com isn't resolving either.
I just recently joined IRC after a server reboot.
brycec:  I was just using my linode vps. seems okay.
could have been, i hvae no idea :)
sorressean: so strange
thanks
I know freenode was under for a while as well.
bryce along with these freenode dos's.
First the ddos and then a bunch of different DNS servers appear broken, I start getting suspicious.
I can't confirm on their DNS though. not using their dns for anything. may have been attack directly at their DNS systems.
(To be clear, these aren't "Linode DNS" or whatever, but actual VPS at Linode running BIND)
ah gotcha.
my system is in dallas if that helps.
I've poked those servers' owners, but wondered if there was something more widespread happening, or if this is just an unfortunate coincidence.
But in other exciting news, I received my OpenBSD 5.8 sets today :)
brycec:  that's only exciting if it's FreeBSD! ;)
Why would you be excited for FreeBSD 5.8??
Does FreeBSD sell CD/DVD sets any more?
openbsd 5.8
mercutio: yes.
oh right
was reading bottom to top :)
lolol
bloody ddos attacks.
for some reason they never really evolved though
Your current account balance is $8.796585458091763898.
Love tarsnap.
haha
http://www.nytimes.com/2015/10/16/technology/ultradns-server-problem-pulls-down-websites-including-netflix-for-90-minutes.html?_r=0
thanks sorressean
Friend's blog is still down though :( blog.anthrobsd.net
the nameservers simply don't seem to have any records besides an SOA for his domain
Also arpnetworks tweated about DDOS issues
brycec:  any idea what provider?
whois data says name.com
I have a domain hosted with them. seems to be fine from my end. (bscbstudentsofma.org)
thansk for checking
*thanks
friend says he hasn't touched anything but hasn't looked into further.
And I decided not to care about it more than he does.
heh, those are great facedesk moments.
And to move on to my next problem. Firewall can hit an address, but a system behind the firewall (not router, no nat, just a transparent bridging firewall) gets "no route to host" for no apparent reason :)
If it's not one thing, it's another.
sounds like my day.
I've never spent so much time in ssh.
sorressean: i've been trying to get office 2k13 to activate all day
i'd rather be network troubleshootin'
:P
also I never thought I'd curse so much about perl.
It wasn't as fun as network troubleshooting.
somewhere somehow spamassassin and perl and amavisd-new broke. but recompililing, even if you get all dependencies didn't matter, so it was a matter of grabbing random perl libraries that were missing, recompiling and then running again tos ee what was broken still.
to see*
Okay... so this host can ping host X, but curl bombs out immediately with "No route to host". lolwut
ping wouldn't update routing tables would it?
that can be firewalling
if it returns icmp-unreachable
thanks mercutio
I forgot this box has its own pf.conf that limits outgoing traffic
the modern trend seems to be to just drop traffic..
(This is why I'm trying to consolidate firewalls too)
OpenBSD 5.8 (GENERIC.MP) #723: Wed Aug  5 00:21:20 MDT 2015
w00t
your cookie did not arrive. It was blocked by pf.conf
I didn't send it to you, sorressean :P
well, mercutio 's cookie didn't arrive!
it was blocked by *his* pf.conf!
heh
it feels so weird to have such an early release
I'm just excited to have it before the release date
heh
(this is the first time I've actually bought a set. and a shirt.)
i'm waiting for the release date.
I'm 90% done with my first upgrade
are you upgrading to 5.8-stable?
And as an admin of the preeminent (if I do say so myself) OpenBSD shell host, I plan to have it upgraded before release date too
Si.
oh wow i thought you didn't do updates there.
?
well not very often i meant
damnit i've forgotten the name of it now :)
On devio.us? We're running 5.7 now :p
oh
it was a couple of versions behind before wasn't it
Sorta, we've had a checkered history
brycec:  I knew I recognized the name, didn't realize you worked on devious.
Ran 4.7 for the longest time (until around 5.5 or so) due to technical issues
er, 4.9 crept in there too
is it down atm?
I got an account, but I haven't found anything fun to do with it yet. just made sure a package compiled on BSD.
That's what she said!!
mercutio:  nope, it's up.
until we were running 5.4, and were "stuck" on that until we had a remote KVM to ensure the time_t upgrade went smooth
OpenBSD wolfman.devio.us 5.7 GENERIC.MP#3 i386
i'm not getting dns
mercutio: our DNS is hosed-ish, I mentioned it earlier in #arpnetworks (Linode)
oh
right
dns is down completely :)
what's the ip?
Yeah :( 2001:470:4:2a5::feed:dead
ipv4
98.142.108.75
Was going to say "I have no idea" but remembered I have open ssh connections to it
it's in motd too it seems
bitrig?
(Our DNS isn't setup in the most optimal way right now... mainly because I don't admin it, and what's setup right now was meant to be temporary.)
Congrats mercutio you're the 3rd person ever to see that gag
[and say something about it]
oh it's not really? :)
i see :)
It's OpenBSD, really
It was an April Fools joke
well bitrig is openbsd too
6 months later...
Not enough people laughed yet.
heh
And only in the sense that OpenBSD is NetBSD :P
invalid system call
gah
brycec:  I just connected via dns.
I think. one sec
sorressean: you must have it cached, because ns1.devio.us and ns2.devio.us don't respond
i wish openbsd would keep supporting older binaries :)
@tdr
Theo de Raadt says: "thisistheone8888" is someone who throws out uneducated thoughts without any practice.  Oh wait, this is misc.  I forgot.
I'm sure Theo's said something about "Fuck backwards compat"
maybe it's propigating now or something. for whatever reason I can ping it from my linode, never connected from linode. well, if I did it was months ago.
hm maybe Linode is filtering 53/udp to the Internet
major lag
it was all good earlier
Well it is a busy box...
well it looks like he.net being lame
I am curious, what is the general use case? I doubt really heavy development can go on there.
shells were all the rage in the 90s.
i had one :)
mercutio++
email, text web browsing, code dev, testing connectivity
Also IRC bouncers, but we frown on that
well, if you frown at me I can say I didn't see it!
heh i got ddos'ed when irc bouncing years back
but i suppose that's what you get for logging in to efnet
someone wanted my nick
sorressean: just to clarify, dig devio.us @173.230.139.107 works for you?
or @74.207.236.79 ?
(from your Linode VPS)
no, it doesn't. maybe it's cached at Linode's DNS servers.
i can reach .108 but not .107
Must be it I guess.
hangon
i can reach both now
yeah. sorry for the false hope there...
I know it's not cached on my specific instance, but it's probably cached on their DNS.
i still can't dig though
sorressean: no worries, just wanted to be sure before I told the guy whose dns servers they are something false
He must be rebooting .107 right now
woo, dns is back
same here
Do we have any stats on how large this DDoS was, traffic wise?
Meh. OVH spam/support is seriously shitty, but I have to give them props for doing cool stuff with energy usage.
..was it hard to find?
nope
at least once dns was working :)
i really want something like php-fpm that can just listen on heaps of sockets and start php if necessary for a user.
and then leave a pool running for more common sites, and just start as needed for less common
$OneOfTheseDays I'll write something just for us
oh i want a general solution
because php startup cost is really high
well, I'll open-source it and stuff
of course i'd rather php just had a cheaper startup cost.
but for shared web hosting in general it'd be great.
You're talking to someone that oversees a system that calls php-cgi hundreds of times a minute, trust me, I know :P
*php-fastcgi
heh
it's one of those glaring problems
that would require considerable effort to fix.
of course one way to look at it is "ram is cheap"
but you still get process table bloating
and php caches don't work when you run as cgi
also this talk has good audio
i'm so used to talks that you can hardly make out what people are saying
We see the load mostly on disk IO load (but that may have something to do with what our users are doing)
WOULD YOU LIKE ME TO TALK LOUDER? :p
you talk loud enough
all the disk i/o should be cached
that may just be from not having enough ram for reads.
Or their stuff is loading various random stuff
Fun fact: we don't have 100GB of RAM to cache all of /home :P
heh
i dunno
We have *drumroll* 2GB.
if i was specing hardware i would have gone with at least 24gb of ram :)
yeh i know
ram is cheap, but that'll be ddr1 :/
Nah we're ddr2
pretty sure
oh real
now I'm not so sure...
i doubt it's ddr2
Whatever, cheaper than ram is "free hardware"
heh
true, you could probably get something better donated now days
but shifting around and stuff
We're actually working on getting another box up
but it's having hardware issues apparently
ahh
do you get much rack space?
As I recall, it has 24GB or so
I have no idea
oh ok
I don't know much beyond what sudo grants me :p
ahh
but yeah i still get surprised by how well openbsd runs on old hardware
it never used to seem that amazing performance wise
it just hasn't got that much slower over time
lol
generating keys for ssh is slow on old hw
ill generate some for you if you like :)
hmm there's al ong video on network performance on freebsd there too
That was a good talk
maybe that'll tell me how linux compares to freebsd network wise
It won't.
oh ok
thanks :)
it should still be interesting :)
It will describe ways of bettering network metrics and improving network performance on FreeBSD
as memory serves
yeah i think i'll leave that for later :)
oh yeh
i just bought 3 dual port 10 gigabit ethernet cards
for US $50 :)
total
yay for ebay and make best offer :)
hot damn
Congrats
they're untested
but i bought one like that for $20 untested before and it was fine
GL
*Good luck
What sort?
there's no driver on openbsd
mellanox connectx
they generate a lot of interrupts, but are fine performance wise
Cool
by a lot i mean i was doing 150k+
before i played with coalescing
(there's been some recent list traffic about 10GbE cards)
yeah i want to try writing a driver for openbsd
but it's way down the list :)
s/writing/porting/
<mercutio> yeah i want to try porting a driver for openbsd
i got a box of sfp+ direct attach cables before
so now i have heaps of cables, so i thought i needed some more cards :)
lol
you can get 1x the single port card for like $16 or something
and the make offer was selling for $175
http://www.ebay.com/itm/121605173632?_trksid=p2057872.m2748.l2649&ssPageName=STRK%3AMEBIDX%3AIT
Well man good luck with those, hope they work out for ya. $50 isn't bad for a gamble.
yeah i suspect they just have no way to test
and dual port cards for back to back is just so much more convenient
ebay seems flooded with them
my only dive into kernel code was trying to patch up a screen reader for Linux that ran fully in the kernel.
It was... less than fun. Mostly because screen readers don't belong in the kernel.
Sounds like a thing Linux would do.
There was not even an attempt at decoupling. It's just fully there.
is it still in the kernel?
i remember people getting excited about tux
That's what she said!!
the in kernel web server
and i'm like uhh
i would like to see more kernel level stuff for medium level type stuff
like receive file and output straight to disk
you can send file, but not receive file
but normal coalescing could help there too.  downloading using dsl or such with low speed and no coalescing is a bit of an unnecessary burden
ddos gone now?
Supposed to be, yep
yeah it's the nastiest ddos i've seen on arp ever.
it was
yeah.. found the destination also I guess? some specific target ?
as it used to be..
yeah weirdly it always seems to be
sometimes you hear about people attacking dns servers and so forth of providers afterwards.
but for the vast majority of cases it's one user.
for residential users it's usually people playing call of duty and mouthing off or such
yeah its boring..
such is life :)
i think it was lithuania that once had huge coordinated attack on banking system
but most attacks are more to annoy than orderly.
its easy for a client to configure ntpd wrong and you get ddos due to those aswell.. Its not easy to keep everything secure either.
it's hard to configure openntpd wrong