Anything weird going on with the network? not that i can see Hmm. Getting some packet loss. CaZe: can you get a mtr? mtr? linux command I don't have Linux. should be som eqvivalent on bsd i guess 100 packets transmitted, 83 packets received, 17.0% packet loss CaZe: FreeBSD? OpenBSD Well I'll bbl. Maybe it'll have fixed itself by then. mtr is in OpenBSD ports mtr is in ports my openbsd works fine :D should upgrade :D also apparently there is a fiber cut in the SF Bay area CaZe: # pkg_add mtr Well maybe when I upgrade the snapshot. I think my VM crashed. I can ping it, but I see nothing on serial. caze: did you configure serial output? Yes. Heh. control-ecl0 will send a reset incase you had ddb.console enabled I saw a prompt on VNC. I typed in my login, but the password prompt doesn't come up. CaZe: where are you seeing packet loss from? (Location) Boston The packet loss actually just went away. Weird, my machine's still responding to pings, but I can't get it to respond on serial or VNC. I guess I'll just try a hard reboot. Well, I still have 2% packet loss. boston links migt be full due to yesterdays bombing Well I think it's fixed. I wonder if the packet loss I was experiencing was OpenBSD slowly dying. Though it had gotten a lot better right before I rebooted it. Down to 0% now, though there were a few pings in the 1000ms range. maybe it was related to that cable cut thing Maybe, how long ago was that? no idea hmm looks like 4 hours ago caze: I've misdiagnosed my vps having issues vs he.net having issues getting me to my vps weird my vm is going slow it never goes slow oh it's just mirrors.arpnetworks.com ipv4 doesn't seem to respond to ping ipv6 does :o well by being slow i meant pkg_add was taking forever can other people get to mirrors.arpnetworks.com on ipv4? I see it on v4 ok why can't i 208.79.88.133... do you get that ip? yes # telnet mirrors.arpnetworks.com 80 Trying 208.79.88.133... and you can connect on port 80? my browser works, if that's what you mean # telnet -6 mirrors.arpnetworks.com 80 Trying 2607:f2f8:0:101::2... Connected to mirrors.arpnetworks.com. from arp i meant oh oh, I'm doing it from outside i can trace to mirrors.arpnetworks.com from home mercutio: yep can't reach v4 from my arp node i wonder what changed up_the_irons: are you around? i wonder if there's a way to make pkg_add use ipv6 i think it's timing out after forever and falling back, but the ipv4 isn't doign connection resfused or anything to it move on quickly all: there was a fiber cut between LAX and SJC earlier, causing some Any2 peers to bounce "UPDATE: Field technicians have confirmed that the cable was cut in the existing slack coil in the manhole. Field Operations are prepping the cables at this time while awaiting the arrival of the splice crews." holy crap: "UPDATE: The fiber cut was located in San Jose, CA and the technicians have completed prepping the fibers. The two splice crews are onsite at this time and have begun splicing the first fibers. The Transport NOC +is monitoring alarms as systems should begin to clear. Field Services reports it was a full cut of a 132 count cable. They estimate 2 - 2.5 hours to complete all splicing and for all services to restore." 132 fibers! fun not as much fiber as SUPER colon blow this now appears fixed though: "UPDATE: Our Transport provider has completed splicing and we have verified services are back up. Outage is now complete this will conclude our updates for this issue." seperate paths are good up_the_irons: can you look into why mirrors.arpnetworks.com is down on ipv4 from arp vm's? but up from internet at large, and up on ipv6 tonight - laundry night. exciting mercutio: strange, i can hit it find from our VMs well it didn't work me and static at a minimum with ipv4 it doesn't make it past gateway still not working? dies at - 208.79.93.113 RandalSchwartz: did you try it from inside? static: is that your gateway? actually - not for me inside either ie is your vps ip one higher than that v6 no problem v4 hurting mercutio: yep its my gateway mtr shows it not making it off the first hop RandalSchwartz: yeh same as us I wonder how many other servers I can't talk to pkg_add -uvr 0.00s user 0.00s system 0% cpu 3:09:59.82 total dunno why user/system time showing as 0 but over 3 hours to run pkg_add -uvr NTT just went live (yet i will probably not announce all our prefixes over it until further testing) \o/ that was SNL wasn't it? yes this is the 5th time pingdom is reporting my site as down while its not.... who's watching the watcher over there are there any known peering/network issues going on? there was, and maybe is ssh: connect to host github.com port 22: Connection timed out well, to most places I try to talk to. http://hastebin.com/weherequnu.bash oh that's nice, tinet route from los angeles to san jose to connect to ntt mercutio: you also experience connectivity issues? not until changed route to go over ntt took it back out it was giving packet loss from tinet onwards /and/ tinet was sending lax sj, sj, lax connecting to ntt in sj steadfast seems ok but i can't download my test file for some reason Perhaps a stupid question, but how do i re-route outbound from my arpnetworks? I thought what was out of scope outbound from my isp I can't do shit from my host ah and short answer is you can't thought what -> thought that. haven't found outbound routes going over ntt yet looks like no outbound routes over ntt yet so yeah it's only partially up and only some subnets which is probably a good thing if giving issues that tinet issue maybe tinet issue though :/ hey i am having connectivity issues i think reverse path fiterling or the like icmp works :/ well stateful crap up_the_irons: yuo around? hopefully he knows about it yeah it can't see return traffic lol even traffic to www.ntt.com routes over HE silly up_the_irons all outbound is not going over ntt it's not completely silly it's a complex issue to have it going over both outbounds atm so i understand him doing incoming first mercutio: I figured it would be logical for traffic to NTT to go over NTT (v4 or v6) brycec: his primary router can only take 100k routes his ntt connection is on secondary router ah all the vm connectinos are terminating on the primary router he currently has 96k routes even all the close ntt routes is likely to be bigger than 4k so what i imagine is likely is that further down the line he'll start shifting people to the new router that can take full bgp table and start shifting the upstreams over to that but in the interim he's trying out the ntt with incoming where it doesn't need to store huge route list ah gotcha in the interim ntt giving isuses mercutio: are you bored or what? static: must be :) it's lunchtime, it's wet, it's hot and i can't even decide what order to eat my lunch in since this issue is actually making large parts of my srevices to not work; should I escalate it somehow? I created a support ticket an hour ago i kind of assume it's known at this point :o heh well up_the_irons hasn't been talking in channel he may be busy he may be busy doing related things or unrelated things but surely things like these must have monitoring abnormalities? depends icmp works but tcp connections don't jbergstroem: you sure its not your ISP? big problem I think is the upstream althought it's just weird jbergstroem: your trace shows ntt in the path right? staticsafe: tried from multiple locations, and more importantly - nothing going out from the host is working ouch nothing? can you get to http://lwn.net on the host? mercutio: icmp is fine. curl google.com times out, just as ssh oh i see curl google doesn't work for me either hrm what kvr are you guys on? kvr15 i think I'm on 05 04 im not seeing any issues on mine what is your ip? I can traceroute google er /24 is good enough ferrovax.asininetech.com red.stonehenge.com 174.136.101.154 you're not being advertised static 208.79.88.0/21 isn't being advetised oh derp now i understand 174.136.96.0/20 is which is what i'm on, and jberg is on how are you checking this? route-server? global bgp route table Uh - red.stonehenge.com is reachable from the real world or I wouldn't be talking to you randal: it's only some routes down the paths that come into ntt oh if you can't get to your vm and traceroute shows ntt you'll be screwed but same applies for other hosts and it's hard to see reverse path like how do you get google to run a traceroute i was about to try from my .au vps, but then I realized its Vocus for intl transit My traceroute to my VPS shows ntt :D so, 1h 30m in; how do we call on the gods? Though leaving my VPS takes mzima. caze can you ssh to your vps? static: vocus had issues last night it was routing via hong kong return path via he.net they're usually on any2ix direct Yes, I'm on it now. this was before i read about the fibre cuts which is prob related to why it was scrwed caze: oh intersting url to fibre cuts? can you reach other ports? jber: it was in my email jbergstroem: fibre cuts have been repaired umm yeah, just never read about it http://www.santacruzsentinel.com/capitola/ci_23040335/at-t-customers-cut-off-by-outage ports? This is over IPV4, btw. take that for example it was in san jose thanks oh it was talked about in this channel too someone said 132 fibres were lost http://permalink.gmane.org/gmane.org.operators.isotf.outages/5634 that's the mailing list i saw it on btw yea same it was on NANOG too that's a *lot* of fiber ... http://www.dailymotion.com/video/x7ms6e_saturday-night-live-colon-blow_fun static: otuages goes to my main mail box, nanog a seperate one so yeah didn't see it on nanog nanog has toom uch mail sometimes :/ mercutio: im on too many mailing lists - http://okazaki.tomoyo.ca/AIi.png somewhere along the road too many ml's somehow turns into a chase for that 0 unread for me :( nanog has lots of crap like that stuff about google tld or something heh yea did you see the drama on AUSNOG? that's way more than me you use gentoo-users, ubuntu-users, freebsd-users what's the point of those/ :) i'm not on ausnog linux-kernel orly? :) what happened? i'm on linux-kerenel too ubuntu-users is for entertainment, gentoo-users because i use gentoo, freebsd-users same thing only so i can scan for things if i have weird problem or something jbergstroem: its marked read by the filter i keep it in case i have to look up something static: just like me :) the only lists i try to stay on par with nowadays is llvm{dev,-commits} and cfe-{dev,commits} s/is/are so looking at ausnog pipenetworks had big outage gentoo-users gets fun every time udev and systemd gets brought up :P fedora-users was terrible when the last new release with the new installer came out staticsafe: USE=mdev emerge busybox :) *shrug* im not opposed to udev yet I got introduced by $client to Monit the other day as an alternative to nagios and/or zabbix and/or upstart staticsafe: guess it depends on use case. just saying that it's hard to motivate for headless stuff. yea good point my only gentoo install is my desktop once you've gotten used to portage, you get a bit picky in what you expect from a package manager. at least one of my reaons for using it [gentoo] for so long. I've been fanboying over FreeBSD lately jbergstroem: indeed portage is <3 yeah - longtime user of fbsd so. we just passed 2 hours. any suggestions on how to escalate? pretty sure up_the_irons is doing whatever he can escalation would be finding a way to get ec2 to host your site for a while, or something well dns TTL gives me an hour or two more to play with i guess i'd just like to be sure that people know there's a problem long DNS TTLs-- staticsafe: i don't optimise for exeptions. first time since i've moved to arp i've had any kind of issues exceptions, sorry jbergstroem: could be worse, 24hr TTL :P i guess :) after my exams are done I'm going to play with DNSSEC I need more time to play with Dart my clients both need it... but I haven't had enough time did up_the_irons make an appearance yet? mercutio earlier yes well still can't reach google etc yeah. like nothing can be done here push button. light doesn't light up. sad. you need to push it harder? sometimes, the internet breaks :) mercutio: we're test subjects :) [11:31am] :@up_the_irons: (yet i will probably not announce all our prefixes over it until further testing) mercutio: You can't reach google from your VPS? CaZe: yeh zsh 3008 # curl -v http://www.google.com/ * About to connect() to www.google.com port 80 (#0) * Trying 74.125.224.209... * Connection timed out * Trying 74.125.224.212... * Connection timed out * Trying 74.125.224.211... --- 74.125.224.209 ping statistics --- 8 packets transmitted, 8 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 0.687/0.801/1.012/0.093 ms ping works try curl lynx works for me. it seems that i actually can curl google now Don't have curl. through ntt, it seems * Timeout * Trying 74.125.224.210... * Timeout * Trying 2001:4860:4007:801::1013... * connected * Connected to www.google.com (2001:4860:4007:801::1013) port 80 (#0) yousure my curl fell back on ipv6 yeah, i'm sure im bouncing all over the place though hong kong, japan sg.bb? singapore ping google.com PING google.com (114.120.192.59) 56(84) bytes of data. 64 bytes from 114.120.192.59: icmp_seq=1 ttl=56 time=278 ms icmp is working hk > jp > sg less than optimal routing for sure Arpnetworks DNS is resolving google to : 114.120.192.35 currently try google public dns for better results 0.841ms for 74.125.224.209 couldn't reach googles public dns:es for a good while NTT transit related? mnathani: most likely i raelly wondre if up_the_irons doesn't know does anyone have his cellphone number and can sms him? 3 hours soon cos the simple solutoin would be to stop advertising to ntt temp mercutio: would seem starnge since he said he enabled ntt on irc. wouldn't you monitor such a change? jberg: maybe he went back to dc? i dunno I can't imagine he is in a state of both "not knowing this is happening" and "can do something about this" you'd think so yes jbergstroem but i'm just looking at what the situation is either, sure. but not both. he's pretty good about monitoring hi folks - anyone else having issues with outgoing connections on port 25? thestereobus - yes, some general network issues thestereobus: most stuff will be problematic if you're in 174.136.96.0/20 okay, thanks yup I am hey it's working for me now well to google yeah, same here. but not most other things oh right been like that for the past hour still broken to steadfast a looking glass on ARP networks would be nice right about now mnathani: it wouldn't actually help at all but i have bgp to arp and could setup lg do you see ntt as preferred for any destinations? his is getting weirder, my other ip works mnathani: nope ntt is in the path for only one subnet it's all incoming i misplaced their as but i checked before flags destination gateway lpref med aspath origin *> 103.14.194.0/24 174.136.111.233 100 0 25795 25795 25795 25795 9304 55821 9299 2914 2516 9658 58677 i that's the only route with ntt in path and that's not direct connect to ntt even Destination: 206.162.240.0/20 >> 4436 25973 25795 I from: http://www.us.ntt.net/support/looking-glass/ 4436 is intermediate mercutio: you have your own AS ? that's via mzima/nlayer? mnathani: yeh, why's that? so, since we passed 3 hours - is it possible to somehow get reach of him and at least verify that he/someone is working on it? have you emailed support@ 2h 30min ago he probably has access to that on a smartphone or pc of some kind i was hoping someone had his cell number so there's trouble with that announcement? i'll investigate wb :) up_the_irons: icmp works, acts like stateful filtering with only seeing one side quick fix: I stopped the announcement of 174.136.96.0/20 up_the_irons: cool yeah working now my dead connections just came back up confirmed. up_the_irons: with debugging, arp couldn't see return traffic from steadfast, even when arp created connection.. i think the issue is: while I redistributed connected routes into bgp so the new router sees all the customer vlan's, the routes learned via customer bgp sessions are not seen where steadfast was routing back via ntt hmm how come ping worked then? mercutio: yeah, that makes sense mercutio: no idea, that's weird, ping shouldn't have worked that's what made me thing stateful issue think dig @208.79.88.7 google.com is returning some strange addresses in 114.120.192.0/24 singapore mnath that's prob cos google detected an issue like some of fancy thing it'll prob fix itself after some time yeah, that happened ~2h in does anyone know what mobo the dedicated servers are running on? guessing its one of these? http://www.supermicro.com/products/nfo/xeon_3400.cfm?pg=MOBO jbergstroem: X9SCD-F up_the_irons: thanks. up_the_irons: you mention 2 nics on your website (and up to 4 ddrives), but it only seems to be 1 dedicated and 2x6G SATA's. Do you add extra nics/controller cards? jbergstroem: i have another MB for >2 drives, but it is functionaly identical to the X9SCD-F in fact, it is the X9SCL-F mercutio: can u check your routing now? i believe i have fixed the customer route propagation within my ibgp. i am announcing the /20 to NTT again ok still fucked take out the /20 again? (fucked here too) hmm.. wth What's the affected network? 174.136.96.0/20 a /20 is 16 wide removed announcement so 96 to 115 err i can't do maths :/ 96 to 111 i think I'm in there, and I don't have any problems. I just do IRC and my house, though. My connection to my house has been a lot more stable than it was this afternoon.