#arpnetworks 2013-03-04,Mon

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***nixbag has quit IRC (Quit: leaving)
nixbag has joined #arpnetworks
[00:18]
..... (idle for 23mn)
nixbag has quit IRC (Quit: leaving) [00:43]
nixbag has joined #arpnetworks [00:50]
........ (idle for 39mn)
amdprophet has joined #arpnetworks [01:29]
........ (idle for 39mn)
up_the_ironstoorop: it'll be run again automatically, but if you'd like to let someone know, just reply to the email
like brycec said
[02:08]
................................ (idle for 2h39mn)
***dj_goku has quit IRC (Ping timeout: 256 seconds) [04:47]
......... (idle for 42mn)
heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer
[05:29]
heavysixer has quit IRC (Quit: heavysixer)
dj_goku has joined #arpnetworks
[05:42]
.............. (idle for 1h8mn)
dj_goku has quit IRC (Ping timeout: 255 seconds) [06:54]
.... (idle for 16mn)
raptelan has joined #arpnetworks [07:10]
raptelan*grumble grumble* not having a server out on the net is really inconvenient sometimes.
time to sign up for arp again :)
[07:10]
***LEMONed has joined #arpnetworks [07:17]
LEMONed has quit IRC (Ping timeout: 260 seconds) [07:27]
..... (idle for 23mn)
raptelan has quit IRC (Quit: leaving) [07:50]
..... (idle for 22mn)
raptelan has joined #arpnetworks [08:12]
WaREX has quit IRC (Quit: leaving) [08:26]
brycecagreed, do it [08:30]
.................. (idle for 1h25mn)
***raptelan has quit IRC (Ping timeout: 245 seconds)
raptelan has joined #arpnetworks
[09:55]
raptelan has quit IRC (Ping timeout: 260 seconds)
raptelan has joined #arpnetworks
[10:07]
raptelan has quit IRC (Ping timeout: 245 seconds) [10:21]
................... (idle for 1h30mn)
phlux has quit IRC (Quit: WeeChat 0.4.1-dev)
phlux has joined #arpnetworks
[11:51]
.................... (idle for 1h35mn)
heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer
[13:27]
......... (idle for 43mn)
up_the_ironsup_the_irons nods [14:10]
brycecbrycec nods [14:11]
mercutiohas the arpnetworks had network issues in the last 3 hours?
i've noticed smokeping has lots of loss, but it's fine atm
[14:12]
brycecmercutio: my smokeping is quiet [14:13]
mercutioi have 4.11% average loss for last 3 hours, 44.42% max
it was worse from 3 hours to 2 hours ago
but it seems to alternate between rather broken and completely fine
[14:13]
brycecmercutio: Here are my last 8hrs, v4 and v6 http://imgur.com/jI587nI,PJg054g
think I'm on kvr07
[14:14]
mercutiothat's close
but yeah you have no issues :)
[14:14]
brycec(yep, kvro7) [14:15]
mercutioit's usually fine [14:16]
brycecbrycec wonders why he bothers smokepinging localhost [14:16]
mercutioheh
i do too
http://imgur.com/a/OvsG5#8qe0RNg
does that work?
[14:16]
brycecI think it was in the default config
loads yes
[14:18]
mercutiosee how it's fine some of the time completely
and screwed other times ?
and of course it's fine now when i noticed
[14:18]
brycecheh
Well I'm going to blame the other end of your smokeping
since... I can
[14:18]
mercutioi'm monitoring other points too
it looked like just arp effected
but it could be pccw
hardly anything goes via pccw except arp
[14:19]
brycec(My smokeping runs on a Chunkhost vps, looking inward at ARP) [14:20]
mercutioit's like pccw, trit, arp [14:20]
brycecmercutio: bummer
I go chunkhost - peer1 - arp
literally 1 host between networks
(never noticed that before, sweet)
[14:21]
mercutiothat's all peering traffic i imagine
well there's someone else before pccw too
http://i.imgur.com/PaJdJf0.png
that's another jost in san jose
but yeah not a global issue
oh hmm, i do have monitoring of arp from somewhere else too
[14:21]
brycechm yeah guess it's not a pccw issue, at least not in general. [14:23]
mercutioi should check [14:23]
bryceclol [14:23]
mercutioit could be the node or it could be transit provider etc
yeh it's fine from chicago
[14:23]
up_the_ironsi think i peer with peer1 [14:24]
brycecup_the_irons: yeah
you do
2 10ge-ten1-1.la-600w-cor-1.peer1.net (216.187.88.145) 0.371 ms 0.389 ms 0.418 ms
[14:25]
mercutiochicago is via mzima [14:25]
brycec3 any2-ix.la.arpnetworks.com (206.223.143.166) 2.031 ms 2.391 ms 2.559 ms [14:25]
up_the_ironsmust be through the route server cuz i don't see them in my official peering list [14:25]
mercutioin both directions [14:25]
up_the_ironssweet [14:25]
mercutioi'm going to assume it's some pccw/trit issue
i dunno what else goes via pccw/trit
and it's nlayer on the way back from arp
and if was nlayer issue more people would have issues i imagine
[14:26]
staticsafeyes [14:27]
mercutiooh, and because pccw/trit has had minor issues before [14:27]
staticsafehttp://smokeping.asininetech.com/smokeping.cgi?target=IRC.scruffy looking good [14:27]
mercutiooh hmm, i should be able to find some other host using pccw/trit
6. i-0-0-0-0.eqla01.bi.telstraglobal.net 0.0% 1979 174.0 125.8 125.5 204.1 5.2
7. 63-218-51-149.static.pccwglobal.net 0.2% 1979 132.6 149.8 130.2 503.5 41.1
my long running mtr found at least a little packet loss
but not to destination
it's really difficult to find good ways to check in between networks
when tehre are so many of them
err
static
did you not notice hte packet loss at the end? :)
oh
to multiple destinations
it'e the same to farnsworth/scruffy so local issue most likely
and bender and amy too
oh packet loss started again
6. i-0-0-0-0.eqla01.bi.telstraglobal.net 0.0% 2402 125.9 125.7 125.5 204.1 4.9
[14:28]
staticsafehmm [14:31]
mercutio7. 63-218-51-149.static.pccwglobal.net 0.2% 2402 130.5 150.0 130.2 503.5 41.4
8. 63-218-212-14.static.pccwglobal.net 2.1% 2401 221.2 159.4 157.4 354.4 11.0
9. cxc.r6.lax2.trit.net 1.6% 2401 195.8 134.5 133.5 197.5 7.4
10. arpnetworks-lax2-gw.cust.trit.net 2.2% 2401 195.5 133.0 126.2 472.0 26.0
11. arp 1.7% 2401 192.4 131.4 130.5 194.3 7.2
so yeh looks like a pccw issue to me
the tarceroute from arp has that annoying nlayer icmp deprioritisation
curiosuly the trace back from arp has this low/high alternating packet loss
and so does the forward but not as obviously
so i think ther'es some load balancing there
8. i-0-1-0.hptw-core01.bx.telstraglobal.net 0.8% 625 327.0 349.9 135.2 621.4 108.0
9. unknown.telstraglobal.net 1.3% 625 130.4 131.4 129.7 170.6 4.6
10. xe-0-3-0-1037.mdr.core.snap.net.nz 5.3% 625 131.2 134.5 130.6 222.4 12.9
11. 202-49-70-162.plain.net.nz 0.8% 625 131.9 131.3 130.4 288.2 6.9
12. 202.36.174.250 5.4% 625 131.3 132.5 130.5 195.3 10.4
but it heavily suggests the issue is sending to arp not receiving from arp
which was already my hunch anyway
[14:31]
jlgaddis< mercutio> the tarceroute from arp has that annoying nlayer icmp deprioritisation <--- so you finally watched that video, huh? ;) [14:43]
mercutiojlgaddis: i've always known about icmp deprioritisation
thing is, that it's actually rather an annoying thing
as there can be packet loss carried through too
like hop 11/12 aren't deprioritising icmp
but it makes it look like load balcnign issue
[14:45]
up_the_ironsicmp replies must be generated by the RP of a switch/router, so it is often rate limited, to save CPU resources
now if all u kids would stop tracerouting all the time, maybe we wouldn't have to do that... ;)
[14:52]
staticsafeheh [14:52]
jbergstroemscript kid if i may [14:53]
mercutiowow, in the whole bgp table there is only arpnetworks using pccw/trit combination
so much for finding anothe rhost to monitor
[14:53]
up_the_ironsthere's a Trit node in Amsterdam, but that's all i know... ;) [14:54]
mercutiothis is from new zealand
and pccw/trit interconnection is happening in los angeles
oh actually i did find another it's just not preferred
199.249.180.0/23
but that's cogent/pccw/trit
or xeex
[14:54]
up_the_ironsbtw, I signed with NTT last week, so that circuit will be up soon [14:58]
mercutiooh cool
what AS is ntt?
mercutio wonders if it'll route via it
it seems trit is mostly pccw
using the not-very-accurate bgp.he.net/AS40193
[14:58]
jlgaddisntt america is 2914 [15:01]
mercutiojust found that
searching for asn owrks beter than AS
as number
hmm wonder who AS703 is
[15:01]
staticsafeVerizon [15:02]
mercutioahh verizon
so yeh it shoudl be abl to go via as703 to ntt
i dunno if that is good or bad :)
actually i thought there was already NTT routes before
oh i think mzima had NTT too
to at&t for example
mzima/nlayer/at&t
err mzima/nlayer/ntt/at&t
so will make it more direct to at&t
oh well, so pccw issues should be abolished
up_the_irons: how are you finding trit in general?
[15:03]
jlgaddissh ip bgp regexp ? [15:05]
mercutioi';m using openbgpd
and transit-as
7. 63-218-51-149.static.pccwglobal.net 0.0% 254 132.7 151.0 130.2 399.0 43.6
8. 63-218-212-14.static.pccwglobal.net 10.6% 254 159.6 164.7 157.4 222.3 18.3
heh it's still happening
[15:05]
jlgaddisbtw, "wget ftp://ftp.arin.net/info/asn.txt", then you can just "grep -i whatever asn.txt" [15:06]
mercutioi normaly do whois AS<blah>
in openbsd it seems to lack a few though
but yeah i suppose that would make it easier to search name
[15:06]
jlgaddisthat works if you know the asn :P
jlgaddis nods
[15:06]
mercutiotrue good point
openbsd doesn't have AS701 or AS703
i think linux does
oh weird
AS701 and AS703 are both UUNET...
maybe they're not in some AS database that openbsd uses with whois cos they're legacy
and i think AS701 is alter.net ?
in traceroutes
[15:06]
jlgaddisVZ [15:08]
mercutioyeh but it's show up as alter.net? [15:08]
jlgaddissome places still refer to it as uunet [15:09]
mercutiooh that's a nice text file
mtr 161.159.0.1
like tracing to that shows alter.net
actually not from arp
arp shows ntt
[15:09]
jlgaddisdoes your traceroute support -a
?
[15:10]
mercutioi'll check
nope
[15:10]
jlgaddis-a == "show asn's", basically [15:10]
mercutionot even the linux one does [15:10]
jlgaddisdebian/ubuntu: apt-get install traceroute-nanog [15:10]
mercutiooh hangon
openbsd has -A
[15:11]
up_the_ironswhoa
you learn something new every day
[15:11]
mercutioyeh this is working
4 0.xo-1-0-0.XT4.AKL1.ALTER.NET (210.80.35.17) [AS703] 1.776 ms 1.776 ms 1.769 ms
5 0.so-0-0-0.IL2.LAX12.Alter.Net (210.80.50.89) [AS703] 193.172 ms 193.180 ms 193.397 ms
6 0.xe-3-3-0.IL4.LAX9.ALTER.NET (152.63.114.17) [AS701] 149.355 ms 152.660 ms 151.895 ms
[15:11]
up_the_ironsmercutio: trit is alright [15:11]
mercutioup_the_irons: does much go over it? [15:11]
up_the_ironsthe owner is my friend and we exchange traffic [15:12]
***heavysixer has quit IRC (Quit: heavysixer) [15:12]
jlgaddisahhh yeah, looks like traceroute-nanog is -A also. -a on os x. [15:12]
mercutioahh [15:12]
up_the_ironsmercutio: maybe 10% of my traffic; basically just the pccw routes [15:12]
mercutioheh
it's the pccw routes i'm having issues with :)
[15:12]
up_the_ironsyeah u need to bitch at pccw for that :) [15:13]
mercutioi'm not pccw customer [15:13]
up_the_ironsdoesn't matter, neither am i [15:13]
mercutioi wonder if they'd listen
arne't they like a china provider or something
[15:13]
up_the_ironsyeah
most NOCs are responsive to problems, whether you are a customer or not. cuz you just amplify what their customers are probably already telling them.
[15:14]
mercutioahh
usnoc@pccwglobal.com
seems to be addresse
-e
[15:15]
staticsafeim glad that ARP peers with Vocus, I have a Australian ircd node linked to it [15:16]
mercutiostatic: that should make things more stable [15:17]
staticsafehttp://bgp.he.net/AS9297#_asinfo my VPS provider in .au is in this AS [15:17]
mercutiook
i emailed them
wonder if they'll say anything
static: i know someone who has vocus transit in NZ
it seems very on/off performance wise
i think a lot of their problems come from using he.net
[15:18]
staticsafeI've only had it for a short time, but I've not noticed any significant issues yet [15:19]
mercutiobut even running two downloads one after the other of a file can be a 10x speed diff
i think it was like 4 or 5mb/sec to arp
and like 200 to 2000k/sec to europe
[15:19]
staticsafeI have it linked over v6 so if Vocus dies for whatever reason, its gone [15:21]
mercutiohmm it's only 13.5 megabit/sec to arp atm
vocus seem to be growing quick
will be interesting to see if they try and be bottom of the barrrel, or offer a higher level of service
i think AU market is like NZ market where people are cheap
oh up_the_irons are NTT doing you ipv6 too?
wow
ipv6 is way faster to arp with vocus
err from arp
it's not faster to arp
oh it is faster to too
bizzare, seems to be performance issue with vocus to arp atm even with peering
the diff in performance between ipv6 and ipv6 can be bizzare
like even within one provider
[ 3] 0.0-10.4 sec 9.25 MBytes 7.49 Mbits/sec
[ 3] 0.0-10.2 sec 22.9 MBytes 18.7 Mbits/sec
like that's ipv4 versus ipv6
vocus for both ipv4 and ipv6
and download was like 25 to 35 megabit
which don't make a lot of sense either
cos would be less change of congestion uploading to arp
[15:21]
staticsafefarnsworth.entropynet.net, how does it look from .nz? [15:33]
mercutio113.9 msec
over ipv6
54 msec over ipv4
via vocus
48 msec via other provider
which is still via vocus
the vocus one is 6 msec away from auckland though
[15:35]
staticsafethe v6 difference there is interesting [15:36]
mercutioso yeh it's good via ipv4
it's vocus all the way
6. ten-0-2-0-300.bdr02.akl01.akl.VOCUS.net.au 0.0% 20 7.7 8.3 7.6 12.0 1.0
7. 2402:7800:110:1::a 0.0% 20 31.7 31.9 30.7 38.1 1.6
8. ten-0-3-0-911.bdr02.per02.wa.VOCUS.net.au 0.0% 20 79.0 80.0 78.9 83.5 1.2
9. 2402:7800:40:1::d 0.0% 20 80.7 79.9 79.0 81.7 0.7
10. 2402:7800:30:1::15 0.0% 20 112.3 112.8 112.3 115.2 0.7
it has another vocus route at 11 with rdns
[15:36]
up_the_ironsmercutio: yeah, ipv6 too. will finally have a non-HE ipv6 provider. [15:37]
mercutioup_the_irons: yay :) [15:37]
up_the_ironsm/ [15:37]
mercutioi thought vocus dual homed all their routers on ipv4/ipv6
so it's weird when performance is diff
actually the ipv4 route goes over ape which is auckland peering exchange
and ipv6 is going over transit
static: you could do a reverse trace to 2403:b700:1::6
[15:37]
jlgaddisup_the_irons: provider or peer? [15:40]
mercutiobut i suspect the issue is routing via perth [15:40]
up_the_ironsjlgaddis: provider, full transit [15:40]
jlgaddisnice [15:40]
up_the_ironsyep:) [15:40]
mercutioand you're in adelaide? [15:41]
staticsafecorrect [15:41]
mercutiothey should be able to reduce latency from adelaide to sydney [15:41]
staticsafesoon [15:41]
mercutiolooking on a map, it seems that it's liekly it goes via melbourne etc [15:42]
staticsafeWAIA is in Adelaide now and Colocity is provisioning [15:42]
mercutioso your normal pings to sydney are 21 to 24 msec?
and auckland to sydney is about 24
hmm by road it's 1410km
and distance between sydney/auckland is 2153km by bird.
so i imagine they can get latency down to 15/16 msec
i suppose that doesn't make a lot of diff
[15:42]
staticsafeit doesn't [15:48]
mercutioi always underestimate how big australia is [15:48]
staticsafeinter-Australia should be much better once the WAIA provisioning is done
heh
[15:48]
mercutioNZ to australia can suck at times
well it's fine to sydney generally
but once you hit domestic routing you can get stuff like adelaide going via perth
i heard that some isp's have really broken routing internally too
so the same happens to dsl users
[15:48]
staticsafeoh yes [15:49]
mercutioand like perth will go via sydney for internet [15:49]
staticsafecase in point - AT&T/Bellsouth DSL
in USA
[15:49]
mercutioand brisbane
and darwin
i also heard that a whole lot of users in australia can still only get adsl1 with 1.5 megabit/384 kbit or something
which uhh, means no high def youtube
terrible congestion etc
then there are transit providers like pipe networks with real high latency going via guam
[15:49]
staticsafehttp://pastie.org/pastes/6385091/text?key=knxpf1n79zzlljnhufpqnq its not too bad for me from home [15:51]
mercutioi can't even find guam on the map atm
i see hawaii
isn't it near hawaii?
[15:51]
jlgaddisjust a bit west [15:52]
mercutiooh you have to zoom in
it's like between papa new guinea and japan
[15:52]
jlgaddisit's probably about as near to hawaii as LA is to NYC =) [15:52]
mercutiooh you're not in australia static?
jlg: it's closer to hawaii than it is to japan to us
[15:52]
staticsafemercutio: no, I'm in Canada [15:53]
mercutioerr it's closer to US going via guam/hawaii than guam/japan
268 is really high ping
that's what i get to UK
[15:53]
staticsafewell yes [15:53]
mercutioit's enough that it can be frustrating
you have 165 msec from san jose to australia
oh but it's mpls crap
[15:53]
staticsafeits only doing ircd, I don't even need to touch it except for any security updates [15:54]
mercutiowith keeping the ping high from start point to end point
suppose that's actually pretty good
yeh
not touching much is fine for that ping
but if you used tmux/irc itd' be annoying
[15:54]
staticsafeas long as it stays linked in a stable manner, I'm happy [15:55]
mercutioheh
what irc network is that?
back years ago undernet used to have so many splits
[15:55]
staticsafehttp://www.entropynet.net/ my small little thing [15:56]
mercutionow freenode is like undernet was [15:56]
...... (idle for 26mn)
***heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer
[16:22]
mercutioPlease provide the PCCW Global Circuit ID or Service ID.
i told them not direct customer.
[16:27]
staticsafeheh [16:28]
..... (idle for 24mn)
mercutioerr in my initial mail
it does seem to have calmed down now though
[16:52]
up_the_ironsup_the_irons goes and grabs In n' Out burgers [16:53]
***heavysixer has quit IRC (Quit: heavysixer) [16:54]
milkiburgers plural o.O [16:57]
staticsafemm burgers [16:58]
brycecmilki: well up_the_irons is married with kids... [16:58]
milkio, so its not all just for him [16:59]
brycecmaybe ;) [16:59]
....... (idle for 30mn)
jlgaddisi miss in 'n out [17:29]
brycecbrycec doesn't... it's just 2miaway [17:31]
mercutioup_the_irons: is that new router coming in with the NTT link? [17:33]
......... (idle for 41mn)
up_the_ironsmercutio: new router is up (just not doing anything). NTT link will go in it, yes. all VM hosts will then have cable to the new router. and finally, the ability to route in / out either one will be complete.
brycec: milki : ok, well, one burger (double double, no onions, protein style)
[18:14]
mercutioup_the_irons: ahh sweet, was wondering how that was going [18:17]
up_the_ironsyeah, slowly but surely.. :)
biggest part was getting the new upstream (research, negotiations, and finally signing)
[18:17]
mercutioyeh i thought was two seperate things though :)
but i suppose tying together isn't stupid
how's fallback going to work? vrrp?
[18:26]
..... (idle for 22mn)
***heavysixer has joined #arpnetworks
ChanServ sets mode: +o heavysixer
[18:48]
up_the_ironsmercutio: it will be a manual failover. shutting down vlan on one router and up'ing it on the other would effectively 'switch' a customer from one router to another. up'ing the vlan sends a gratuituous ARP, so the VMs would automatically pick up the new gateway MAC address.
mercutio: no automatic fallback method was viable in my testing. they all caused more problems than they cured. failover should be an extremely rare situation, if it even occurs at all (we've already gone 4+ years on the first router)
[18:50]
mercutiook
yeh i had a quick think through n my head and i couldn't think of any easy ways to do it
best answer in my head i came up with was two /31s instead of /30
but that wouldn't just work either
and if you have two /31s on private addresses going to internet /32
then that's a bit of a pita too, and there'd be shared link state
[18:53]
up_the_ironsmercutio: you can't do vrrp with /30's without hoops that make it not worth it; i wouldn't be surprised if those hoops would crash Cisco IOS
mercutio: those who have /29's however, perhaps i could offer vrrp as a service, who knows... like, elective auto failover
[18:56]
mercutioup_the_irons: ok
yeh, well i'd rather just two bgp sessions myself...
but i realise that you're not going to have bgp sessions to everyone.
[19:02]
up_the_ironsright [19:03]
mercutioand you can't just go using /24s or such, because of everyone having their own vlan
which means that they wouldn't be able to reach people on the same /24 easily
[19:04]
up_the_ironsyeah i can't stick entire /24's to one or the other
well, i could, but why..
the inbound traffic will come in through *both* routers
and ibgp (or perhaps ospf) will decide where to take it from there
[19:05]
mercutioi mean /24 with vrrp etc
3 ip addresses
you'd need bridging between vlans or such though
so then it gets more complicated
and it may remove some peoples assumptions about how the network works
ie would be change in design
cos i assume volume comes from vlan rather than from whether it's routed/bridged
[19:12]
up_the_ironsyeah vrrp was not designed for that [19:14]
mercutiowell that'd just be to enable access within the /24
you'd also need ip/mac enforcement
so yaeh more complicated
but could save ip's.
but yeah i understand completely reluctance to go down that path
[19:14]
up_the_ironsoh, that, no.. i decided against that common design long long ago
one customer == one vlan, solves so many practical problems
i would never do it another way :)
[19:15]
mercutioyeh you can still do the one customer to one vlan
but you could bridge between vlans for the same subnet
if wanting to use /24s rather than /30s
so like customer 1, vlan 4321, customer 2 vlan 4432, internal vlan 105
[19:15]
up_the_ironsi think i just barfed a little in my mouth [19:16]
mercutiooops customer 2 4322 [19:16]
up_the_irons;) [19:17]
mercutioso vlan4321 can bridge to vlan4322
ok
[19:17]
up_the_ironsbut what would bridge it? [19:17]
mercutiobut router is still on vlan105
the switch
[19:17]
up_the_ironscustomer vlan's terminate directly on the core (one day, perhaps an edge) [19:17]
mercutiohmm [19:18]
up_the_ironsi'm not sure where the "internal" vlan would live [19:18]
mercutiook the router then [19:18]
up_the_ironsdraw it out and show me if u wanna discuss more :) [19:18]
mercutiothere's also proxyarp
*cough*
i may be missing osmething, i'll try and have a further think about it in a few hours
[19:18]
up_the_ironsup_the_irons wanders off [19:22]
.......... (idle for 45mn)
***amdprophet has quit IRC (Ping timeout: 245 seconds) [20:07]
ant has quit IRC (Ping timeout: 264 seconds) [20:17]
....... (idle for 34mn)
ant has joined #arpnetworks [20:51]
..... (idle for 24mn)
heavysixer has quit IRC (Quit: heavysixer) [21:15]
vin has joined #arpnetworks
vin has quit IRC (Quit: Killed by BlackJac (Requested by panasync))
[21:28]
....... (idle for 32mn)
amdprophet has joined #arpnetworks [22:04]
........ (idle for 36mn)
mtve has quit IRC (Ping timeout: 255 seconds) [22:40]
.... (idle for 19mn)
Webhostbudd_ has joined #arpnetworks
Webhostbudd has quit IRC (Ping timeout: 255 seconds)
[22:59]
..... (idle for 23mn)
HighJinx has quit IRC (Ping timeout: 252 seconds) [23:26]
HighJinx has joined #arpnetworks [23:31]
..... (idle for 24mn)
LEMONerd has joined #arpnetworks
LEMONerd is now known as LEMONed
[23:55]
LEMONedGarry there? [23:57]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)