Suggestions for anyone who has to do IT for a hotel (or any non-corporate network setup)

Originally posted on Reddit an hour before this...

Having spent the past 2 weeks living in hotels...

• Don't run everything off the router if its more than 3 users: the CPU on a wireless router has to process DHCP, NAT, and wireless. A hotel using it, is asking for chaos.
• Use separate wireless access points when you can; and use Wireless-N! The hotel I'm at right now is still on Wireless-G; which has been obsolete for ~4 years now. You can convert wireless routers to access points by disabling their DHCP and SPI firewall options; and using a LAN port to plug it into your network.
• If you're going to make people put in a password to use your wifi: enable WPA2; vs using some agreement portal. You can still have agreement portals come up on first use (I've seen some Mikrotiks do that). By requiring a password on the wireless end, you're securing their connections, and still discouraging non-guests from logging in.
• If you are having users try to use some program for managing guests or clients (small biz), consider using a local server. I stayed at a place the other night that was asking my advice on trying to use TeamViewer to access their application computer from other workstations. A hotel I did get to do work for some years back, I moved their app to a 2008 server with RDP access. I also know there's web and Linux based apps for handing small biz and hotel stuff.
• Try to keep the hotel stuff separate from the guests via separate router and/or net connection.
• If you have crappy wiring options, don't forget that Powerline / HomeAV networking is a feasible option. As long as its on the same circuit panel, and there's not a crappy machine on it, you'll get connectivity. You could also look at wireless bridging, or delve into the mesh-networking stuff people are messing around with.

If you have to reboot your servers often, its probably port exhaustion

This issue can impact any networked system of note, but it seems particularly insidious on Windows servers. Two servers I recently instituted with 2012, have become unresponsive after varying amounts of uptime. Email and limited DNS service on one; that server can die within 4-5 days uptime, and used to be able to run for weeks as a 2008R2 VM. Brand new file-server, with 50-100 possible users: maybe two weeks.

Earlier today, I had found an article from a Microsoft blogger regarding this behavior. She used Process Explorer to diagnose instances occurring with software; but no immediate fixes. Another Microsoft post suggested using netstat to trace how many TCP/IP ports are open at a time. Both articles referred to behavior on Windows to limit the number of ephemeral ports on a system. Basically, when you make a request to a service on a port (say 143 for email); the server will reply back to the client using a different port number (say 50000). The "ephemeral port range" on modern systems is ports 49152 to 65535: and this is implemented separately for TCP, UDP, v4, and v6 (technically meaning you could have ~64000 open connects using all protocols). As more connections are open, and left waiting, the available pool of these randomly-assigned ports drops: eventually to the point you can't RDP to reboot the affected server.

I saw some other mentions online regarding expanding the port range down to as low as 1024 (common, standardized applications originate at the 0-1023 port range). This might be good for a large number of connections: but even that might not be enough for applications with sloppy use of networking (or iffy networks). A SQL buff tackled this issue: he recommended a few changes, which I have adopted some to resolve my server's issues...

* You can use netstat -n (or even pipe it to a text file) to list your current port usage, as well its state. If you have a lot of "wait", the next will apply.
* Make the following as a registry file to import. It will create values to change the retry time on timed-out connections to 30 seconds (from a default of ~4 minutes); and a ~15 minute wait to check on established connections (default is hours-days).

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"KeepAliveTime"=dword:000dbba0
"TCPTimedWaitDelay"=dword:0000001e

You should reboot as able to implement this; and also to clear out existing connections. You can check periodically with netstat -n  to see if the number of "waits" decrease.

Added: it might still help to boost the port range on these servers. Command sequence for expanding the range to half the usable ports...

netsh int ipv4 set dynamicport tcp start=32768 num=32767
netsh int ipv4 set dynamicport udp start=32768 num=32767
netsh int ipv6 set dynamicport tcp start=32768 num=32767
netsh int ipv6 set dynamicport udp start=32768 num=32767

Quick notes on loading an updated Debian kernel on a router / server box

I know Kernel 3.9 just came out, but Debian is fully caught up on 3.8 (beats the 3.2 it defaults to). If you're making an x64 router or server, the following will be handy...

1. apt-get update && apt-get install gcc-4.7

2. Use wget to grab the files from a listed mirror, and dpkg -i to load them. If you're not using x64, look for i386, arm, or other versions of "linux-headers-3.8.1" and "linux-image-3.8.1".

* initramfs-tools
* linux-kbuild-3.8
* linux-headers-3.8-1-common
* linux-image-3.8-1-amd64
* linux-headers-3.8-1-amd64

3a. Out-of-box Debian doesn't do anything with /etc/sysctl.conf : here's one I recommend trying.

net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.tcp_syncookies=0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_rfc1337=1
net.ipv4.route.flush=1
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_fack=1
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_low_latency=1
net.ipv4.tcp_fin_timeout=10
net.ipv4.ip_no_pmtu_disc=0
net.ipv4.tcp_frto=2
net.ipv4.tcp_frto_response=2
net.ipv4.tcp_workaround_signed_windows=1
net.ipv4.tcp_mtu_probing=1
net.ipv4.tcp_congestion_control=yeah
#Kernel 3.x network stuff
net.core.bpf_jit_enable=1
net.ipv4.tcp_limit_output_bytes=65536
  #Disable privacy addressing
net.ipv6.conf.all.use_tempaddr=0
net.ipv6.conf.default.use_tempaddr=0
#Swap file behavior: 0-100 aggressiveness
#If using zcache, you want more
vm.swappiness=10

3b. If using as a router, add net.ipv4.ip_forward=1 and net.ipv6.conf.all.forwarding=1 to it.
3c. To enable the "congestion control", add tcp_yeah to /etc/modules

Network interface bridging on FreeNAS

The information regarding bridging in FreeNAS is quite scattered. And frankly, I think they changed some stuff from version 8 to 8.3.1 (latest). Here's what a few hours produced...

* Enable SSH in FreeNAS. Don't bother using the web console: can't copy-paste (in Firefox at least). I assume you're logged in as root via SSH for the following steps...
* ifconfig | more : take note of what interfaces you need to bridge.
* mount -rw / : /etc gets replaced on boot, so we'll be editing /conf/base/etc instead.
* nano /conf/base/etc/rc.conf and add the following (change the interfaces as needed)...

gateway_enable="YES"
cloned_interfaces="bridge0"
autobridge_interfaces="bridge0"
autobridge_bridge0="em0 em1"
ifconfig_em0="up"
ifconfig_em1="up"

* nano /conf/base/etc/sysctl.conf and add the following (may not be needed for FreeNAS per se)...

net.link.bridge.pfil_member=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_local_phys=0
net.link.ether.bridge=1

 * reboot and then log back into FreeNAS. You should be able to assign an IP to the bridged interface.

References of note

* FreeBSD bridging manual: did not readily mention autobridge
* Post that I found autobridge in

Ads are fine, but video kills my network!

I understand ads are necessary for site revenue, as well as for actually informing people of more than the latest celebrity anti-acne aphrodisiac. But I live and work in the US: where any real business is stuck using T1 lines: streaming video kills those sites. Blocking Youtube is a broad-hammer: it has as much useful content, as it does junk. Video ads though: those hurt my network, period. Here's a way of resolving this situation...

* Review the list of DNS requests. If you're not using something like OpenDNS or Barracuda, you can have Wireshark or something monitor your DNS server for URL requests. On Linux routers, you may also be able to use netstat-nat or conntrack to do this.
* Found a decent listing of current video ad providers.
* Judicious use of domain tagging to see if certain domains were coming up in blocked categories or not.
* Visited popular websites + some of the ad provider sites (some didn't work). The properties from the Adobe Flash player, and the "Page Info" function in Firefox; help identify domains that stream video.
* Checked some domains against Google Safe Browsing to see if they were detected as having had malware. Google's own self-check is a rather amusing result: change the domain in the URL to test other ones.

Two things to consider regarding this kind of blocking. One, you don't want to block content inadvertently: many of the online media sources are used for streaming radio; which a number of my own users enjoy. Two, you won't get everything: the lists from 2009-2011 aren't even valid anymore; industry changes alter the list of active domains and providers on a regular basis.

A current block list I'm using at the moment of posting, minus category blocks...

Fixing clock time in Ubuntu

When you manage multiple systems across multiple time zones, it is inevitable the clocks will drift apart. And while I thought I was still loading NTP onto my work boxes (I've been loading ntpdate to use to force updates when needed), I was apparently mistaken. One other issue: BIOS hardware clocks; some may be set to UTC; some to a timezone other than where it was sent to. So I wrote a script to make sure the system clock was fixed correctly: this is best deployed as something like "clock.sh" you can copy with scp + run as sudo....

apt-get -y install ntp
service ntp stop
dpkg-reconfigure tzdata
ntpd -g -q
service ntp start
hwclock -w

Explanation of what it does...

* Installs NTP client: default Ubuntu settings use the NTP pool.
* Stop NTP client service: attempting forced NTP updates fail otherwise (open socket error).
* Make sure the correct time zone is set (a menu will let you set this).
* Force synchronize via NTP: otherwise, a slew system forces it to update in increments over long periods of time. Fine for operation, but not for inital correction.
* Restart NTP service.
* Force hardware clock to use the system time pulled from NTP.

References

* NTPD on Wikipedia
* hwclock man page

Checking monitor age in Linux

If you need to check on how old a monitor is, and you can't get to said monitor (say its 100 miles away): you can easily find this out via SSH and regular user account.

egrep -iw "Manufacturer|Year" /var/log/Xorg.0.log

The output should be like the following...

[    65.775] (II) NOUVEAU(0): Manufacturer: DEL  Model: a023  Serial#: 827732812
[    65.775] (II) NOUVEAU(0): Year: 2006  Week: 27
[    65.775] (II) NOUVEAU(0): Manufacturer's mask: 0

Some IDs...

DEL -> Dell
ACI -> Asus
HWP -> Hewlett-Packard
HSD -> Hanns-G