Nicolas314

All my geeky stuff ends up here. Mostly Unix-related

Archive for the ‘router’ Category

Long live NAT!

leave a comment »

ipv6-no-thanksHome networking can be a lot of fun: setting up a name service, a guest network, or traffic rules, leads to an endless joy of discovering new RFCs or creativity in the very active field of artistic configuration file syntax.

I thought I had seen everything until I tried to set up IPv6 connectivity for my home network. Little did I know that this would eat up so many of my precious free evenings. The following writeup is here to remind me never to try that kind of shit ever again, and as a warning to future generations who might want to dig into this kind of topic. Life is short, there are many better things to do than attempt to set up a new addressing scheme for your home network. Long live the NAT king!

The Start

It all began when I noticed that my ISP provided me with a unique (native!) IPv6 prefix to use on my home network. Something like:

2001:1234:5678:9abc::/56

Since I was not familiar with IPv6 addresses, it took me a while to find out that the first 64 bits of a 128-bit IPv6 address designate a network, and the last 64 are reserved to differentiate hosts on that network. My provider handing me a /56 means I have 64-56 = 8 bits to play with, i.e. I can instantiate 256 home networks, each having up to 2^64 = 18,446,744,073,709,551,616 hosts. Overshot a bit, maybe.

So where do I start? Do I have to install specific software? Where? Do I need to buy specific hardware? How many services are needed? And thereby started my long painful descent into the horrific world of IPv6. Toss and loose 1d20 sanity points immediately.

My ISP unfortunately did not provide any help as to what I am supposed to do with the IPv6 thingie they gave me. No single help page, very few discussions on their forums, and all exchanges I had with customer service were completely useless. Best I could find were discussions between customers of an ISP in the US that provides a similar setup. That is thin.

Say you received a /56 prefix from your ISP. If that prefix ever changes e.g. because you switched to a new ISP, you want things to work automagically because that is the way things currently work with IPv4: changing my public IPv4 address does not change anything to my home network.

In order to do that, IPv6 suggests that home networks use two sets of addresses: the public ones derived from the ISP-provided /56, and another private address space based on something else called a ULA (Unique Local Addresses). You get to choose your own ULA on your home network(s), preferrably based on a good random number generator, but nothing prevents you from taking something like fc00:caca:caca:caca:caca::/48. If anybody else on the Internet picks the same network prefix you will get into trouble when trying to get intimate with each other, e.g. by establishing a VPN between both worlds. We had exactly the same problem when trying to join two sites using IPv4 NAT’d 10.0.0.0/8 subnets, so this is not really a regression. Fun fact: if you have no ULA in France you can always say “Il manque ULA sur mon réseau”.

How do you get to choose this ULA? If you happen to have a single router on your home network it should just be a matter of digging through the router IPv6 setup until you find it. But most home networks are now running multiple routers that are all unaware of each other, and all convinced they are masters of the universe. You will most certainly end up with several ULAs. Some of your devices will get several addresses and you will have to understand your own network topology to know which address to use to access them. Prepare for glorious hours of debugging, which is particularly great when facing addresses that are mostly made of bloody random bits.

Why several routers on the same home network? Simply because you may be running several DSL connections, or maybe you have a VPN started somewhere away from your edge router, or maybe you connected your smartphone and it offers another potential exit to the Internet. You also get a virtual router when you start virtual machines on a desktop.

To make things simpler, every network interface on your machines will also generate a local address that is only valid for its closest neighbours, called a link-local address. Unfortunately you won’t go far with that one as it is not supposed to cross boundaries. Think of it as a 127.0.0.1 that extends to the other side of the cable but not further.

Ok so we have now several adresses for each machine on the network.  Figuring out which one should be used (incoming or outgoing) is just an unspecified, incredible mess. The link-local address can only be used on very specific physical links, the ULA address cannot be routed to the Internet, and the public addresses you have may change at any moment, e.g.  through your smartphone sharing a 4G access.

At that point we have just determined that your printer currently identified as ‘printer’ also known as 192.168.1.20 in IPv4 will now be accessible as:

– fe80:bffa:3d5f:5f8d:b4cf:1749:b01c:5b2f for machines directly connected to it through an Ethernet cable
– fc00:c465:3b76:b34d:38f7:da19:2586:1cbd for machines living on the same internal network.
– 2001:61af:ff44:b148:4fc3:0097:f35d:c806 for machines on the internet when reached through a first ISP, and another public address for each available ISP connection.

Oh joy.

Of course normal human beings are not meant to remember this kind of random shit. For this kind of thing you have DNS.

DNS you said? What DNS?

There are really two ways machines can obtain an IPv6 address: SLAAC and DHCPv6. SLAAC means Stateless Address Auto Configuration, whereby a machine obtains a prefix and derives its own IP address from it, e.g. based on its own MAC address. Cool, right? You do not have to assign individual addresses in static DHCP leases, every machine does it on its own. But then: how do you know which address was self-assigned by your very smart printer?

There are dedicated neighbour-discovery protocols for that, but they are mainly designed to make sure that addresses are locally unique and routers know where to find them. This is only taking care of establishing a link, there is nothing dedicated to associating a name to a self-assigned IP address. And if there was, how would you know who to believe? If two machines on the local network claimed to be ‘joe’, what should happen?

To be fair, there are solutions like Bonjour, also known as zeroconf, but they are unlikely to work on lightweight or old devices. Shoot again.

Back to square one: if you want to reach your own machines using human-usable names you need to run DHCPv6, a protocol that was designed to compensate for such things. And there you go: back to static leases, addresses assigned by a router, attached to a name, and you end up doing exactly the same kind of shit you used to do with IPv4 local networks, except this time the addresses are much easier to screw up.

Even worse: if the self-assigned IPv6 addresses are not related to MAC addresses, it means every single host on your local network will have generated its own random address, forcing you to manually harvest them from all devices. But you know how to do that on your connected toaster, right?

What’s in it for the average home network user? Pretty much nothing. The fact that every single one of your home devices has a potentially reachable address on the intertubes is downright scary. Internet service is for internet servers, not for sensors and other IoT bullshit. First thing you will want to do is bullet-proof your firewall to make sure nobody but you can access your printer from the Internet, and hope things are Ok with your IoT shit.

The story did not just end up with me reading thousands of pages on the Internet and a couple of paper books. I hacked every single computer in my house to run IPv6, starting with the routers under OpenWRT, LEDE, FreeBSD, OpenBSD, pfSense, OPNSense, and later moving on to all client OS machines: OSX, Linux, Android, *BSD, and even some Windows boxes, blimey.  I instantiated dedicated DHCP and DNS servers, configured static addresses, automatic ones, bridges and NATs and firewall rules and what-have-you, and I ended up with some machines working under IPv6 only, some under IPv4 only, some that could use both stacks, and some (a lot) that were just unreachable no matter what. Yeah, I also crashed my Internet access several times. Omelet and eggs.

Let me try to put it this way: some of my home machines are servers, e.g. a NAS or a printer. I want to be able to print on ‘printer’ or mount a share on ‘NAS’ without having to remember random 128-bit numbers. Silly me. Since I want to use names I have to assign addresses myself from a router running DHCPv6. Neither NAS nor printer need to be available to the public. So what did I gain compared to a local IPv4 network? Hmm… Address management is not fun with 32 bits, imagine with 128.

Or maybe I am just old-fashioned, trying to manually assign names to my home machines. This might be an idea for a new product: a router that would automatically identify hosts on the home network and show them on a single web interface, allowing you to assign names and forget about addressing altogether. Might get in trouble when you have several identical devices but I’m sure there would be a way. If such a product exists I have not seen it yet.

On the other hand, if I want to browse the Interwebs in v6, I found out that mounting a SOCKS proxy on a remote cloud box works perfectly well. No need to configure anything, just ssh -D and the IPv6 world is mine to browse.

Summing it all up

Address assignment is not easier than IPv4. Still requires a dedicated DHCP and DNS server, only more complicated to configure. You are facing the tedious task of gathering self-assigned IPv6 addresses from all hosts and copying them onto your DHCPv6 server, hoping the self-assignment method won’t change soon.

Routing is now different, but not easier. New constraints are imposed on knowing which interface to bind to when reaching out to the Internet.

Firewalling the whole thing with a mix of IPv4 and IPv6 might tear you a new one. I can already lock myself out of a router with human-readable firewall rules, I cannot imagine doing the same thing with batshit-crazy addresses and feel safe.

You know what? I will stick to glorious NAT’ing until this mess is sorted out. Good news is that there are many bright people currently working on the topic. All I hope is they eventually come up with something that you and me can use without having to read through a million pages of RFCs, compile obscure daemons, or purchase new boxes as if I did not have enough of them.

Talking about RFCs, this one is trying to gather very sensible requirements about home networks:

https://tools.ietf.org/html/rfc7368

If you have 20 minutes to spare, you should watch this talk:
https://www.youtube.com/watch?v=wQdfWUsG4uI

If you really insist on switching your home network to IPv6, I would recommend reading this rant first:

IPv6 at home (published 2012, still relevant):
http://www.kloepfer.org/ipv6-homenet.html

And to get an idea about how messy it is to get IPv6 configured on Linux:

IPv6 Set up an IPv6 LAN with Linux
https://www.jumpingbean.co.za/blogs/mark/set-up-ipv6-lan-with-linux

In its current state I can only dismiss the current IPv6 definition for home networks as very incomplete and unworkable for non-professionals.  Let’s hope RFC 7368 will be handled by qualified, creative, and pragmatic people.

Til then, there is no place like 127.0.0.1

Advertisements

Written by nicolas314

Tuesday 28 February 2017 at 11:41 pm

My own little farm

with 3 comments

zotac_ci323_03Virtualization is fun! Virtual Machines are nothing new, we have all been using VirtualBox, qemu, or VMWare at some point to try out new stuff, bring up the odd Windows instance to run annoying software, or whatever. At work we use thousands of VMs for millions of things. The hardware price tag is pretty hefty though: if you want to start a reasonable number of VMs on the same racked server you need very large amounts of RAM and disk space, placing it beyond reach in terms of price for home usage.

Not any more! Prices are dropping for heavy machinery faster than the time it takes to look up prices on Amazon. I found this little gem from Zotac and purchased one for a mere 180 euros from a French site:

Zotac CI323

The little beast sports a quad-core CPU, two Realtek NICs, and a whole bunch of USB ports (including two USB3). Add on top of that an extension card for WiFi and Bluetooth. Perfect choice to build a home router in a VM and leave space for other VM instances. You need to add RAM and disk, the box comes empty. I scavenged 8GB RAM and an SSD disk from a previous build and off we go.

It has been a while since I last had a look at virtualization solutions.  Took me several days to look them up individually and find out what they offer. All the solutions I tried are described below.

Option 1: run VirtualBox on a desktop

Install a convenient desktop like Mint or Ubuntu, run VirtualBox on top.  Unfortunately not a very good option as the VMs would not be as close to the metal as I would want. Dismissed.

Option 2: run Linux containers

Containers are neat but they are Linux only. I would like to run BSD and maybe Windows VMs too on the same hardware, so dismissed.

Option 3: Run a bare metal hypervisor

The main options I could find are:

  • VMWare: run VMWare OS as hypervisor, run any OS on top.
  • bhyve (pronounced like beehive), the FreeBSD hypervisor
  • Proxmox
  • KVM: use virtualization routines offered in the Linux kernel. This can be started from any Linux distro and conveniently run pretty much any OS.
  • Xen: use a Xen kernel as bare-metal hypervisor, run any OS on top.

VMWare ESXi was my first choice but had to be quickly dismissed: my box NICs are Realtek and VMWare dropped support for those a few versions back.  Annoying. There are convoluted HOWTOs explaining how to hack the install ISO to add missing drivers and stuff but I do not want to play that game. The whole setup would probably be broken in the following release so no thanks.

I installed FreeBSD 11 and tried out bhyve. Installing FreeBSD on this particular hardware was a real chore: for some reason the integrated SD card reader has driver issues and booting the machine took up to 10 minutes because of a nasty timeout spitting out kernel traces. I finally succeeded in disabling the driver on boot by adding stuff to device.hints after hours of googling and tests. To be fair, I have always faced issues with hardware support on FreeBSD, but to be completely fair: these are the only issues I ever faced. The OS is so polished and professional it is a real pleasure to use. Other parts of the box were immediately recognized and activated: Realtek NICs and the WiFi+Bluetooth (Intel) board.

Anyway: bhyve is relatively easy to learn, documentation is good enough, and it should run any BSD or Linux-based VM without any effort. Running Windows or OSX VMs would probably not be a good idea though. I have not tried but it seems a bit daring. If bhyve offered an easy-to-use GUI I might have stuck with it, but I finally dismissed it because it is still too young compared to other existing solutions.

KVM: the idea would be to install a very small Linux instance and use it to manage VMs on top with KVM. I tried several:

Ubuntu desktop is far too heavy for a “very small Linux instance”. I cannot believe a simple desktop is using so much RAM and CPU. I tried to manually remove stuff after a default installation and broke the machine most completely after having erased ‘evolution’. Forget it.

Ubuntu server is fine enough without GUI, but I would like to have a minimal X11 environment to run VM management software. Unfortunately, as soon as you start adding GUI stuff to an Ubuntu server you start piling up gigs of desktop software you do not want. I could probably figure it out but did not have the patience to do it.

Arch Linux is a royal pain to install. Manjaro (a fairly straight Arch derivative) gets you to a fully configured machine in a matter of minutes.  Problem is: I do want stability on my VM farm and a rolling release is probably not the best choice. Dismissed.

Minimal Debian install worked great. All hardware perfectly supported. And then I tried some KVM tutorials, messed up a bit further with Xen tutorials, and ended up with a completely borked machine. Don’t ask me what went wrong, I just got frustrated of randomly killing processes and rebooting the hardware. There are certainly good HOWTOs out there explaning how to transform a base Debian install into a Xen/KVM server but I did not find them. Dismissed.

Alpine Linux to run KVM: did not try, but seems like a possible option.

I tried Proxmox but the default ISO does not install, it crashes miserably after a few minutes of timeout. I have no idea what is going on, but I dismissed Proxmox at that point and came back to it later. Read on.

At that point I was left with Xen as bare metal hypervisor. I focused on Xen Server, a free Citrix project. The OS is based on CentOS 7 with a modified kernel and a GUI on top.

The XenServer install procedure is rather straightforward. Answer a few questions and let it roll. On the next reboot you get an ncurses-based interface on the console that allows you to achieve the bare minimum: configure the host, start/stop VMs, that kind of stuff. You can also do the same through ssh (ssh in then use xconsole).

Beyond that you need to find a Windows desktop because the only management solution they offer is a heavy Windows client. You get a very decent management interface that looks a lot like the VMWare Sphere client, from which you can control pretty much everything. The fact that it only runs on Windows is a major pain but to be honest: you only use it to configure new VMs. Once they are started you access them through ssh, vnc, or rdesktop, so no need to maintain a live Windows machine just for that.

In less than two hours I managed to install on XenServer:

  • A minimal Alpine Linux running nginx
  • An OPNSense instance
  • A pfSense instance
  • A Windows 8.1 desktop
  • A FreeBSD 11.0 VM, no X11

I still felt like something was missing though: XenServer would not recognize my WiFi/Bluetooth board. It would have been cool to dedicate a VM to make a stand-alone access point, so I kept trying more stuff.

Among all the options I tried, the only one that had all my hardware covered without hitch was Debian. Proxmox is based on Debian jessie, so if I succeed in installing it there should be a way to make things work. Let’s try again. I started from Debian and installed Proxmox on top. The guide I used is here:

https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie

This works and happens to be quite smooth.

NB: I managed to completely destroy my setup when I decided to change the host IP address without telling Proxmox first. Rebooting the machine does not help, it goes into an endless loop, fails to reconfigure the network, and dies in horrible pain. I took the shortest path and re-installed from scratch. Good advice: DO NOT CHANGE THE PROXMOX HOST IP ADDRESS.

Proxmox is now working beautifully well. The advantages over XenServer for me are multiple:

  • LXC + KVM support: Proxmox supports LXC containers and KVM Virtual Machines in approximately the same way. Of course, containers are much lighter to install, start up, shut down, or backup.
  • Proxmox is completely open-source. XenServer probably has proprietary parts somewhere, though I did not investigate more than that.
  • Proxmox offers a pure Web interface: no need for a heavy Windows client.  You can also open a VNC console on any virtual machine directly from your browser, which is incredibly convenient.
  • Based on Debian, Proxmox identified and supports all my hardware.

Just for fun, I created a local WiFi access point based on alpine Linux by instantiating an LXC container, assigning the wlan0 interface to it, and booting the right daemons.

The next VMs I created are:

  • An alpine Linux desktop under LXC
  • Various alpine Linux boxes under LXC to run simple services
  • An Ubuntu desktop (under KVM)
  • A Windows 8 desktop (under KVM)
  • A MacOS Sierra desktop 
  • pfSense and OPNSense as KVM appliances, to evaluate them
  • An OpenBSD box to play with pf in command-line mode
  • A FreeBSD11 box

All these virtual goodies run on the same hardware as I write these lines.

My next task will be to select a solution to use as a home virtual firewall appliance. Meanwhile I am just having fun popping up and down virtual machines as my mood goes.

Completely useless but tons of fun!

Written by nicolas314

Tuesday 8 November 2016 at 3:43 pm

OpenWRT on EdgeRouter Lite

with one comment

erlite-3-900x500Installing OpenWRT on EdgeRouter Lite

This installation procedure does not require any extra hardware beyond a Phillips screwdriver to open the router box. I believe it is completely reversible and (hopefully) does not void your warranty.

Objective: replace EdgeOS on your EdgeRouter Lite by a recent version of
OpenWRT picked from the LEDE project.

You need:

  • A powerful Linux box, preferrably a multi-proc 64-bit machine with tons of RAM. This will only be used just once to compile OpenWRT.
  • An extra USB thumb drive. Its dimensions should be short: less than a few centimeters, otherwise it won’t fit inside the box.

Ok now off to build an OpenWRT image:

. Log onto your Linux box

. Download the latest sources from LEDE project:

 git clone https://github.com/lede-project/source
 cd source

. Prepare the tree for compilation for ERLITE:
You will have to select a number of options to build an image tailored to
the EdgeRouter Lite. I could put here a ready-made config file but as these
things tend to evolve quickly, it would probably be obsolete in a matter of
days. So bear with me: start the configuration with

make menuconfig

. Target System: Cavium Networks Octeon
. Target Profile: Ubiquiti EdgeRouter Lite

. Target Images: make sure ‘ext4’ is selected, then Select that line to
open up a menu for ext4 configuration. Change the number of inodes to
60,000 instead of the default 6,000. Also select GZip images, and finally modify the root filesystem partition size to something more comfortable,
say 500 MB. This space will be taken off your USB stick so if you have
more space you can increase that to whatever you have. With 500 MB you
should have enough space to put all the packages you need.
. Global build settings: select Select all kernel module packages by
default.

Beyond that take your pick for packages you want included by default in
your image. My selection is:

  • Base system: base-files, block-mount, busybox, ca-bundle, ca-certificates, dnsmasq, dropbear, firewall, fstools, jsonfilter, lede-keyring, libc, libgcc,
    libpthread, librt, libstdcpp, mtd, netifd, opkg, procd, rpcd, sqm-scripts, sqm-scripts-extra, swconfig, ubox, ubus, ubusd, uci, usign
  • Administration: sudo
  • Development: ar, binutils, gcc, gdb, make, objdump
  • Kernel modules: everything should already be selected as module. You want
    to change some of these to be compiled into the kernel otherwise it will
    fail to find the ext4 root on USB:

    • Filesystems: select kmod-fs-ext4, kmod-fs-msdos
    • USB Support: kmod-usb-core, kmod-usb-storage, kmod-usb-storage-extras
  • Languages: select whatever programming languages you want to see in a
    default install. I usually make sure at least Lua and Python are selected.
  • LuCI: make sure LuCI is selected. Take your pick for applications you
    want to install. I usually select luci-app-openvpn, luci-app-commands,
    luci-app-firewall.
  • Network: if you want your router to act as an OpenVPN client or server,
    make sure it is selected under VPN. Pick either openvpn-openssl or openssl-polarssl.
  • Utilities: bash, bc, file, grep, gzip, less, lsof, openssl-util, strace, tar, tmux, usbutils

Feel free to select more packages but each additional one will take extra
compilation time.

. Type ‘make’ and let the magic go on.
. When finished, the result is stored as:

bin/targets/octeon/generic/lede-octen-erlite-ext4-sysupgrade.tar

This file contains everything we need to build a bootable USB drive for
the EdgeRouter Lite. This file should be 500 MB large since you
selected that size above for your root filesystem, but it is mostly made
of zeroes so if you use bzip2 you should be able to reduce its size to a
more manageable 50-60 MB, which is more convenient if you need to toss it
around the network.

. Put the sysupgrade.tar file onto a local Linux machine and extract it:

tar xvf lede-octen-erlite-ext4-sysupgrade.tar

. The directory contents are:

sysupgrade-erlite/
sysupgrade-erlite/kernel
sysupgrade-erlite/root
sysupgrade-erlite/CONTROL

. Now insert your USB thumb drive into a local Linux machine and prepare
the filesystem. We need a first (small) FAT32 partition to hold the kernel,
a second 500 MB partition to hold the root:

fdisk /dev/sdX # Where X is the letter assigned to your USB drive
New partition: 1, 32 MB in size, type c (WIN95 FAT32 LBA)
New partition: 2, 500 MB in size, type Linux (default)
Optional: New partition: 3, the rest of your drive, type Linux (default)
Make the first partition bootable (a).
Type 'w' to save your changes.

Create the FAT32 filesystem with:
mkfs.vfat /dev/sdX1

. The default uboot configuration on the EdgeRouter Lite wants a file
called ‘vmlinux.64’ in the first (DOS) partition, so let’s do just that:

mount /dev/sdX1 /mnt # Mount the DOS partition
cp sysupgrade-erlite/kernel /mnt/vmlinux.64
umount /mnt

. Dump the root filesystem contents onto the second partition:

dd if=sysupgrade-lite/root of=/dev/sdX2 bs=1M

. If you have a third partition, create a new filesystem on it with:

mkfs.ext4 /dev/sdX3

. You are done with the USB drive!
. Open the EdgeRouter Lite. There are three small screws to remove on the
back. The box slides open if you push gently.

. Remove the existing USB stick inserted in the reader on the motherboard.
Be gentle: you need to insist a bit to take it off but it is not stuck.

. Insert the USB drive you prepared. Close the box, put the screws back,
and boot the router.

. If you connect a PC to the central NIC (labeled eth1) you should receive
an address on 192.168.1.0/24 from which you can ssh to 192.168.1.1 or open
a browser to http://192.168.1.1

. Welcome to OpenWRT/LEDE! Set a root password and you should be done.

The first things you probably want to do:

  • Change interface names to associate eth0 to WAN, and bridge eth1 and eth2 to
    LAN.
  • Edit the configuration to mount the third partition on the USB drive on /home. This is cool to add non-root users and give them a real flash storage.
  • Run ‘opkg update’, install missing packages.
  • Install OpenVPN configurations and test them.
  • Add ssh keys for root login in /etc/dropbear/authorized_keys
  • Install dotfiles to feel at home

Problems I have seen and their solutions:

The LEDE build is not so robust, sometimes it fails in parallel mode
because some dependencies seem to be compiled too late. If you get
compilation errors, using ‘make -j1’ should solve all issues. On a powerful
server with tons of RAM you need 2-3 hours to compile the whole set,
depending on how many packages you selected.

The version of LEDE you just compiled will quickly be out of sync with the
official package repositories. As soon as the kernel is changed in the LEDE project HEAD, all kmod packages from the LEDE repositories will refuse to install with opkg. This is the reason why you had to select “Build all kernel modules” in menuconfig: all kernel modules are already part of the image you created. This problem should go away once LEDE has released its first stable version.

I had most trouble with the ext4 filesystem definition: my first attempt
generated an ext4 of 50 MB which is far too small. After increasing that
size to 1GB, I still ran into “not enough disk space” errors and figured
out the number of inodes was too low (6,000). If you install a lot of
packages you need more inodes. Both points are addressed in the above
procedure. I also tried with an insanely high number (600k inodes) and the
resulting filesystem cannot be mounted.

Filesystem size is indicated in bytes, but fdisk counts in MB, based on a
power of 2. This yields a small discrepancy between the 500MiB filesystem
you generated with the build and the 500MB you reserved in the partition
table.

Once up and running, my router quickly ran into starvation problems. One
machine on the network could use the whole bandwidth and cut off every
other machine. I installed QoS packages: sqm-scripts, sqm-scripts-extra,
and luci-app-sqm, configured the queue to a fair scheduler, and got rid of
starvation issues. For some reason I could not get the pre-compiled
versions of these packages to work, I had to re-install them from official
repositories.

I wanted to add a Samba server to be able to use the rest of the USB drive
as a shared space but it is not a good idea. Samba takes ages to compile
and the daemon uses too many resources for such a small piece of hardware.

If you want to add ssh keys for the root user, remember the default ssh
server is dropbear, not openssh. dropbear expects root ssh keys to be
stored in /etc/dropbear/authorized_keys. You can also add root ssh keys
through LuCI.

The default shell for root is /bin/sh. You can change it to /bin/bash after
installing it and modifying the root entry in /etc/passwd.

Enjoy your fancy new router!

 

Written by nicolas314

Sunday 16 October 2016 at 4:58 pm

EdgeRouter Lite

with 3 comments

erlite-3-900x500

My endless search for the ideal home router made me buy a piece of hardware called EdgeRouter Lite by Ubiquiti. The price point is sweet (around $100), making it a damn expensive home router or a damn cheap professional one. For that price you get:

  • A Cavium Octeon processor: 500MHz, two cores, rated 1000 bogomips, MIPS64 architecture, big-endian.
  • Half a gig of RAM
  • Three GBit NICs
  • No wireless
  • No fan, no noise
  • OS completely contained on an easily accessed USB stick on the motherboard, so essentially as much drive space as you want.

The last point is the most important: by just removing three small Phillips screws you can unplug the original USB thumb drive and replace it with your own, equipped with your favourite operating system. If everything fails you can always switch back to your previous state, put the screws back and call it a day. That should not void your warranty but I am no lawyer.

The provided operating system is called EdgeOS, based on Vyatta, itself based on Debian. It seems Vyatta development is now halted and Ubiquiti is now steering EdgeOS alone. I used EdgeOS on that router for about six months and have to admit being rather satisfied. The router is sold as the fastest switching home appliance on the market, claiming 1 million packets per second. In order to reach that kind of speed with a (dual-core) 500MHz processor on three GBit NICs you need additional specialized hardware that is only available through proprietary drivers provided with EdgeOS. So be it.

I have a beef with proprietary router firmware though: each vendor seems to feel obliged to invent their own management language. Cisco, Mikrotik, Ubiquiti, you name it. Everything is meant to be controlled from the command line, which is great, but instead of navigating through a familiar Unix environment you need to learn half a million new (proprietary) commands, their syntax, side effects, and how to commit, save, or restore configurations.  This is a royal pain in the butt and I have no desire to go get some training to configure a home appliance.

To be fair, open source versions have had the same issue for years, though some made a huge effort to provide good web-based GUIs for configuration and avoid having to invent a new configuration language altogether. Tomato and DD-WRT have really pushed things forward to reach a decent level of user-friendliness. You only need to know about networking and do not have to worry about learning yet another obscure syntax to handle those.

Too bad: both projects seem to be pretty much abandoned today. DD-WRT has not seen a stable release in almost a decade and Tomato still courageously lives on, maintained by a handful of dedicated devs working from home. The communities for Tomato and DD-WRT are dwindling fast in favour of OpenWRT.

OpenWRT is the most advanced open source router project today. It is well designed, based on a single syntax for configuration files, and supports pretty much every piece of router hardware under the sun. The project was recently forked by its own developers into the LEDE project, which is now the version I am following as closely as possible.

Back to the EdgeRouter Lite: what’s not to love?

Beyond the proprietary software and syntax, EdgeOS offers a web-based GUI that looks fancy and neat but covers only a very, very limited portion of what can be achieved through a command-line interface. This is very frustrating. I love command lines as your next geek but don’t force me to learn a syntax I will use nowhere else just to achieve mundane stuff.

After six months of customizing my home router to my own needs, I had gathered scripts lying around e.g. to extract a list of known MACs or some stats.  And when I updated EdgeOS to another minor version, everything fell apart.  That irked me to no end, pushing me once more into the arms of an open source alternative.

Support for alternative firmware for this router is not obvious to find.  OpenWRT has an incomplete wiki page about it. A couple of guys have succeeded in installing FreeBSD but I did not feel up to the task. Debian supports big-endian MIPS64 machines, and a project called DebWRT offers support for this router, merging both Debian and OpenWRT in a single solution. This is cool but I am a bit terrified about using a straight Linux distro to build a router. If all I have to handle iptables is a bash shell and miles of manual pages, this is not going to work, I hate the iptables syntax with a true passion. The unique config file format used by OpenWRT is a real blessing, there is no way I am going back to one config file format per daemon.

So I started from scratch, built my very own version of a LEDE instance, including all the software I want to run on this box. The process is error-prone and it took me several evenings to get straight. In order not to lose information, I will be detailing everything I did in a next post, hoping it could be useful for someone else.

The net result is a pure LEDE box that has been running without hiccups for a few days now. Configuring routes, VPN, DHCP, DNS is a walk in the park thanks to user-friendly OpenWRT. All my scripts are working again, I can handle backups myself, and I even installed dedicated web and Samba servers. Next step will be to install an ad-blocking name server.

I am certainly losing in terms of performance but I won’t see the difference. Without proprietary drivers, hardware acceleration is gone.  This should not be an issue considering my home GBit network is currently handled by a separate switch and my Internet connection is limited to a mere 20MBit/s, magnitudes below what the router needs to provide. The day I get a GBit Internet connection at home, I will always have a choice to switch back to EdgeOS with just one unplug/plug of a USB key. Or maybe someone will have reverse-engineered the proprietary drivers by then?

There is one alternative I have been looking deep into: using pfSense or OPNsense to build my own firewall. The approach sounds good. I believe the BSD family is technically a lot better than anything Linux-based. This is particularly true in terms of network security software.

Trouble is: pfSense/OPNsense is extremely greedy. You can build a 15 euro router with OpenWRT but you need PC-sized gear to run pfSense, including at least 1 GB of memory and a lot more than mere megabytes of storage (OpenWRT fits in just 4 megs). The cost of a pfSense appliance can easily run in 400-500 euros, which does not make any sense from a budget point of view.  Most people going down that road recommend re-purposing an old PC for the task, but I have absolutely no intention of storing a hungry 300W loud old PC box next to my 20Mbit/s DSL modem, this would be insane.

There lies the whole beauty of this exercise: find the cheapest, least power-hungry, and most efficient way to set up a home routing solution that is easy and fun to configure, flexible enough, and secure. I stopped building my own PCs years ago and cover that need now by building small appliances from scratch, compiling the whole OS myself.

Tinkering is fun!

Written by nicolas314

Wednesday 5 October 2016 at 10:03 pm

OpenWRT on Ubiquiti AC Lite

with 4 comments

unifi-ac-lite

Stealth AP with hidden logo

A year ago, I thought I had upgraded my home WiFi to AC with the purchase of a cheap TP-Link Archer C5, but it only gave me trouble. The 2.4GHz band is working perfectly, the 5GHz not so. The first version of OpenWRT I installed had no support at all for the 5GHz mode, I had to dig out from the openwrt mailing-lists that something was under way to add support for it, find the right rxxxx release, compile it myself and install it, only to be disappointed. In the end I got it almost working: network would drop every minute or so, and the range would not exceed a few meters away from the access point. Quite worthless.

Ars Technica reported last year about switching from consumer-grade WiFi access points to professional ones here:

Ars Technica review of Ubiquiti Access Points

Took me a while to finally give up on the Archer C5 and decided to get the cheapest Access Point from Ubiquiti: the Unifi AC Lite unit.

Several things play in favour of this access point: it is designed to be hanging on a wall or a ceiling for a better wave spread, offers much higher power than your usual consumer-grade WiFi device, and the firmware is in the hands of Ubiquiti so they are taking care of making sure the unit is performing as it should. Or so I thought.

First surprise: the device cannot be started out of the box. You need to download the Unifi Controller software, a 200MB piece of java software that is meant to control the unit. Install the Controller on any computer on your local network, start it up, let it discover your access point, and you are good to go. The Controller starts up a local web-based interface which is accessed via a local web browser.

Ubiquiti designed it this way because this product is not meant to be purchased as a single unit but in batches and installed in hotels or offices. The Controller software has obviously been designed to maintain a long list of such access points. Having just one looks a bit ridiculous but Ok, I’ll play along.
To be fair: if you do not want to install the Controller locally you can also run it from an internet box (look Ma, it’s the cloud!). For companies or hotels who do not have servers on site it is an excellent idea but for a single AP this is largely overkill. I am not keeping a java server up full time on the Internet to manage an Access Point in my living-room. And if you do not want to run the Controller at all, you can also just start it whenever you need to modify your AP configuration and shut it down afterwards, which is what I did.
I mounted my AP on a top shelf, ran the initial configuration, and was delighted to see a double-band WiFi emerge immediately. Strong signal everywhere in the house, no connection issues, great! A welcome enhancement to my home network.

The only notable modification I brought was to stick a white Apple sticker on the Ubiquiti logo to hide it. The point was not to rebrand it as an Apple product but to hide this ugly U and this was the only sticker I had available that day. The Ubiquiti guys are probably not aware that they have the very same logo as a cheap French supermarket brand and I got tired of seeing that prominently displayed in my living-room.

After about a month, a few things started bothering me though. The AP seemed to have trouble waking up after losing power. First boot would report everything fine but WiFi would drop every connection immediately.  Rebooting the AP solved it every time. Did not feel too good about this.

Nmap’ing the unit revealed an open ssh port, which accepted the admin credentials that were set on the controller software. Once logged in, I found myself in front of some kind of heavily modified OpenWRT. Interesting… So is there anything on the Ubiquiti web site about this modified OpenWRT? After all, OpenWRT is under GNU license (v2) so I expected to find sources, some kind of build system, or anything related to the modified OpenWRT version running on my Access Point, but I could not find anything, at least nothing obvious from the Ubiquiti web site. Bad point for Ubiquiti but I am not a lawyer.

Nosing around the logs for some explanation for the needed reboots, I found nothing obvious. What I found was that a process was spitting a few lines every five seconds about having no contact with the Ubiquiti Controller software. Several questions on the Ubiquiti forums on that topic were answered with: “yes we know, this is a low-priority fix, just ignore it for the moment”. I have to say that just pushed me over the edge. I understand that my case is not the general use case and I truly do not blame Ubiquiti, but this is not the way I want my AP to work. I do not care much about log files filling up with useless messages but the lack of interest for single-AP users like me is disturbing.

So off to installing a real OpenWRT firmware this time. I finally got it working but it took me a whole afternoon of research to do so, which is summarized here in case it can be useful to somebody else.

First: the current (3.7) firmware embeds an RSA signature check preventing any attempt to install non-Ubiquiti firmware. This is probably due to the recent FCC firmware lockup rules. While I do understand the FCC concerns and the important role they play in Homeland Security, I regretfully do not feel obliged to follow US rules on European ground. This is my hardware now, if I decide to mess it up with another firmware I should be able to do so.

Solution: downgrade the firmware to version 3.4, which is signed by Ubiquiti and does not check firmware update signatures. This firmware can be found by googling a bit around, I got a working URL and the complete procedure from this page:

LEDE/OpenWRT for Ubiquiti UniFi AP AC (LITE + LR + PRO)

While we’re at it, it may be a good idea to switch from OpenWRT in favour of the recent project fork called LEDE project. Seems they have added official support for this hardware and the documentation seems a lot cleaner, though very, very incomplete for the moment, which is perfectly Ok for a two-week old project.

If you did not follow what happened to the OpenWRT project recently, you may be interested in learning a bit more about why the team forked:
https://lwn.net/Articles/686767/

Some pre-built images are available from the LEDE project site, but I chose to go all the way and clone the github repository, configure the build to include the software I need, and recompile everything myself. On a beefy x64 server this took about 2 hours and 11GB of disk space, ending up with a 3.3MB image that was happily installed in a single command on the downgraded Access Point.

The default configuration is smart: the AP tries to obtain an address for itself through DHCP on its only wired interface and acts as a bridge for wireless clients, making it immediately operational when connected to a network equipped with a proper DHCP server. Sweet.

Net result: my Unifi AC AP is now completely stand-alone. I happily removed the Ubiquiti Controller software and customized my AP to death with various scheduling and logging scripts. Wireless range in the 5GHz band covers the whole house and the 2.4GHz allows me to walk outside during Skype calls without losing signal. No more spontaneous rebooting, my logs are clean, and most importantly: I feel empowered :-)

I have to say I do not see Open-source firmware disappearing any time soon.  For tinkerers like me who like to have complete control over their network, this is absolutely brilliant.

Edit: 2016-12-09 Adding some config files and diagnostics to this page, might be helpful if you are trying to replicate this

/etc/config/wireless defines two access points: ap24 for 2.4GHz and ap50 for 5GHz, with passwords SECRETPASSWORD.

config wifi-device 'radio0'
    option type 'mac80211'
    option hwmode '11g'
    option path 'platform/qca956x_wmac'
    option txpower '20'
    option country 'FR'
    option distance '50'
    option channel '3'

config wifi-iface
    option device 'radio0'
    option network 'lan'
    option mode 'ap'
    option ssid 'ap24'
    option encryption 'psk2+tkip+ccmp'
    option macfilter 'deny'
    option key 'SECRETPASSWORD'

config wifi-device 'radio1'
    option type 'mac80211'
    option hwmode '11a'
    option path 'pci0000:00/0000:00:00.0'
    option htmode 'VHT80'
    option txpower '20'
    option country 'FR'
    option distance '50'
    option channel '136'

config wifi-iface
    option device 'radio1'
    option network 'lan'
    option mode 'ap'
    option ssid 'ap50'
    option encryption 'psk2+tkip+ccmp'
    option key 'SECRETPASSWORD'

Here are the outputs of iw phy0 info:

#iw phy0 info
Wiphy phy0
 max # scan SSIDs: 16
 max scan IEs length: 199 bytes
 max # sched scan SSIDs: 0
 max # match sets: 0
 Retry short limit: 7
 Retry long limit: 4
 Coverage class: 0 (up to 0m)
 Device supports AP-side u-APSD.
 Available Antennas: TX 0x3 RX 0x3
 Configured Antennas: TX 0x3 RX 0x3
 Supported interface modes:
 * managed
 * AP
 * AP/VLAN
 * monitor
 * mesh point
 Band 2:
 Capabilities: 0x19ef
 RX LDPC
 HT20/HT40
 SM Power Save disabled
 RX HT20 SGI
 RX HT40 SGI
 TX STBC
 RX STBC 1-stream
 Max AMSDU length: 7935 bytes
 DSSS/CCK HT40
 Maximum RX AMPDU length 65535 bytes (exponent: 0x003)
 Minimum RX AMPDU time spacing: 8 usec (0x06)
 HT TX/RX MCS rate indexes supported: 0-15
 VHT Capabilities (0x338001b2):
 Max MPDU length: 11454
 Supported Channel Width: neither 160 nor 80+80
 RX LDPC
 short GI (80 MHz)
 TX STBC
 RX antenna pattern consistency
 TX antenna pattern consistency
 VHT RX MCS set:
 1 streams: MCS 0-9
 2 streams: MCS 0-9
 3 streams: not supported
 4 streams: not supported
 5 streams: not supported
 6 streams: not supported
 7 streams: not supported
 8 streams: not supported
 VHT RX highest supported: 0 Mbps
 VHT TX MCS set:
 1 streams: MCS 0-9
 2 streams: MCS 0-9
 3 streams: not supported
 4 streams: not supported
 5 streams: not supported
 6 streams: not supported
 7 streams: not supported
 8 streams: not supported
 VHT TX highest supported: 0 Mbps
 Frequencies:
 * 5180 MHz [36] (20.0 dBm)
 * 5200 MHz [40] (20.0 dBm)
 * 5220 MHz [44] (20.0 dBm)
 * 5240 MHz [48] (20.0 dBm)
 * 5260 MHz [52] (20.0 dBm) (radar detection)
   DFS state: usable (for 2032442 sec)
   DFS CAC time: 60000 ms
 * 5280 MHz [56] (20.0 dBm) (radar detection)
   DFS state: usable (for 2032442 sec)
   DFS CAC time: 60000 ms
 * 5300 MHz [60] (20.0 dBm) (radar detection)
   DFS state: usable (for 2032442 sec)
   DFS CAC time: 60000 ms
 * 5320 MHz [64] (20.0 dBm) (radar detection)
   DFS state: usable (for 2032442 sec)
   DFS CAC time: 60000 ms
 * 5500 MHz [100] (27.0 dBm) (radar detection)
   DFS state: available (for 2032379 sec)
   DFS CAC time: 60000 ms
 * 5520 MHz [104] (27.0 dBm) (radar detection)
   DFS state: available (for 2032379 sec)
   DFS CAC time: 60000 ms
 * 5540 MHz [108] (27.0 dBm) (radar detection)
   DFS state: available (for 2032379 sec)
   DFS CAC time: 60000 ms
 * 5560 MHz [112] (27.0 dBm) (radar detection)
   DFS state: available (for 2032379 sec)
   DFS CAC time: 60000 ms
 * 5580 MHz [116] (27.0 dBm) (radar detection)
   DFS state: usable (for 2032442 sec)
   DFS CAC time: 60000 ms
 * 5600 MHz [120] (27.0 dBm) (radar detection)
   DFS state: usable (for 2032442 sec)
   DFS CAC time: 60000 ms
 * 5620 MHz [124] (27.0 dBm) (radar detection)
   DFS state: usable (for 2032442 sec)
   DFS CAC time: 60000 ms
 * 5640 MHz [128] (27.0 dBm) (radar detection)
   DFS state: usable (for 2032442 sec)
   DFS CAC time: 60000 ms
 * 5660 MHz [132] (27.0 dBm) (radar detection)
   DFS state: usable (for 2032442 sec)
   DFS CAC time: 60000 ms
 * 5680 MHz [136] (27.0 dBm) (radar detection)
   DFS state: usable (for 2032442 sec)
   DFS CAC time: 60000 ms
 * 5700 MHz [140] (27.0 dBm) (radar detection)
   DFS state: usable (for 2032442 sec)
   DFS CAC time: 60000 ms
 * 5720 MHz [144] (disabled)
 * 5745 MHz [149] (disabled)
 * 5765 MHz [153] (disabled)
 * 5785 MHz [157] (disabled)
 * 5805 MHz [161] (disabled)
 * 5825 MHz [165] (disabled)
 valid interface combinations:
 * #{ AP, mesh point } <= 8, #{ managed } <= 1,
   total <= 8, #channels <= 1, STA/AP BI must match, radar detect widths: { 20 MHz (no HT), 20 MHz, 40 MHz, 80 MHz }
 HT Capability overrides
 * MCS: ff ff ff ff ff ff ff ff ff ff
 * maximum A-MSDU length
 * supported channel width
 * short GI for 40 MHz
 * max A-MPDU length exponent
 * min MPDU start spacing
 Device supports VHT-IBSS.

Here are the ouputs of iw phy1 info:

# iw phy1 info
Wiphy phy1
 max # scan SSIDs: 4
 max scan IEs length: 2257 bytes
 max # sched scan SSIDs: 0
 max # match sets: 0
 Retry short limit: 7
 Retry long limit: 4
 Coverage class: 1 (up to 450m)
 Device supports AP-side u-APS.
 Device supports T-DLS.
 Available Antennas: TX 0x3 RX 0x3
 Configured Antennas: TX 0x3 RX 0x3
 Supported interface modes:
 * IBSS
 * managed
 * AP
 * AP/VLAN
 * WDS
 * monitor
 * mesh point
 * P2P-client
 * P2P-GO
 * outside context of a BSS
 Band 1:
 Capabilities: 0x11ee
 HT20/HT40
 SM Power Save disabled
 RX HT20 SGI
 RX HT40 SGI
 TX STBC
 RX STBC 1-stream
 Max AMSDU length: 3839 bytes
 DSSS/CCK HT40
 Maximum RX AMPDU length 65535 bytes (exponent: 0x003)
 Minimum RX AMPDU time spacing: 8 usec (0x06)
 HT TX/RX MCS rate indexes supported: 0-15
 Frequencies:
 * 2412 MHz [1] (20.0 dBm)
 * 2417 MHz [2] (20.0 dBm)
 * 2422 MHz [3] (20.0 dBm)
 * 2427 MHz [4] (20.0 dBm)
 * 2432 MHz [5] (20.0 dBm)
 * 2437 MHz [6] (20.0 dBm)
 * 2442 MHz [7] (20.0 dBm)
 * 2447 MHz [8] (20.0 dBm)
 * 2452 MHz [9] (20.0 dBm)
 * 2457 MHz [10] (20.0 dBm)
 * 2462 MHz [11] (20.0 dBm)
 * 2467 MHz [12] (20.0 dBm)
 * 2472 MHz [13] (20.0 dBm)
 * 2484 MHz [14] (disabled)
 valid interface combinations:
 * #{ managed } <= 2048, #{ AP, mesh point } <= 8, #{ P2P-client, P2P-GO } <= 1, #{ IBSS } <= 1,
   total <= 2048, #channels <= 1, STA/AP BI must match, radar detect widths: { 20 MHz (no HT), 20 MHz, 40 MHz }
 * #{ WDS } <= 2048,
   total <= 2048, #channels <= 1, STA/AP BI must match
 HT Capability overrides:
 * MCS: ff ff ff ff ff ff ff ff ff ff
 * maximum A-MSDU length
 * supported channel width
 * short GI for 40 MHz
 * max A-MPDU length exponent
 * min MPDU start spacing

 

Written by nicolas314

Monday 30 May 2016 at 4:41 pm

OpenWRT on MR3020

with one comment

TL-MR3020_3_1600x1600

 

Situation: you live in a one-room appartment and receive Internet through a single RJ45 plug in the wall. How do you extend it to all devices in the room for the cheapest price?

Objective: provide WiFi connectivity on a dedicated LAN where users can see each other and share files easily. The system should be easy to repair or upgrade. Bonus points for extra functionalities like VPN provided to the whole WiFi LAN, ad filtering, or shared folders on Samba.

My first go-to for anything cheap related to computing is the RaspberryPi.  How can you beat it? For 35 euros (amazon.fr in December 2015) you get a full-powered Linux box with RJ45 and enough USB ports to power an external disk and a WiFi dongle. I gave it a try and came to the conclusion that the result would probably be a little too expensive and probably brittle. I have several WiFI USB adapters lying around and none of them was able to create a WiFi Access Point on a RPi, though they do work flawlessly as WiFi clients. They almost got there but not completely. You need to recompile your own WiFi access point software, and taking care of my own iptables rules is just beyond my patience.

Better option: Openwrt! This open-source Linux distribution is not meant for your average PC but to run on screen-less network appliances. You won’t find desktop apps or anything related to X11 but you have all possible network daemons and tools running there. Openwrt runs on low-power processors like ARM, MIPS, and x86 too of course. It comes bundled with its own package distribution tools (opkg) making administration relatively easy. It is really meant for tinkerers, people who like to open the box and modify what’s inside to do different stuff.

Obvious applications for Openwrt are of course network-related. Build your home router for 25 euros supporting IPv6, customized firewalling, guest WiFi, kid protection, quality of service, network monitoring, Virtual Private Networks, or file sharing, to name a few. A popular application is PirateBox: start the box, create a local WiFi network, all clients connected to it can easily share files through the local access point.  Other cool projects include running your own home telephony over the Internet (asterisk), making a sound box or an Internet radio.

There are also thousands of cool projects if you are so inclined and ready to take your soldering iron out of the dust. Most router hardware parts have hackable GPIO ports you can connect to a breadboard to pilot a 0-5V trigger or even read signals at a reasonable frequency. Check out the DIY section at the bottom of this page to get a few ideas:

https://wiki.openwrt.org/toh/tp-link/tl-wr703n

Note that the WR703N is a 10-20 euro piece of hardware so it is about half of the retail price of an Arduino with Ethernet shield, or a RaspberryPi.  Sure, you get less hardware, but if your project is purely network-oriented this is by far a better alternative.

Back to our task: find cheap hardware that gets Internet from an RJ45 plug and offers decent WiFi for a few devices.  I set my limit to 25 euros and chose the TP-Link MR3020.  It has the minimal 4MB of internal storage needed to install Openwrt. You read that right: 4 megabytes of persistent storage, in an era where your average smartphone sports several gigabytes of RAM.

There are several ways to obtain an Openwrt image for a given piece of hardware. First and simplest one is to download it directly off openwrt.org. That does not always fit the bill though, because the team had to make choices about the packages that are present in each image. If space is tight (4 megs is kind of narrow) you may not be able to install anything more than was pre-bundled already. That is actually the case for the MR3020. Solution: extend the local storage. So I plugged in an 8GB USB dongle (5 euros).

Plug the USB in, wait, and… nothing. ‘dmesg’ shows the kernel recognized something USB but did not offer to mount the filesystem. Of course: the kernel modules have been trimmed down to a minimum and USB storage was not in. Quick install with opkg? No way. There is not enough space left to install a mere kernel module for USB storage so the default image falls a bit flat. Removing packages also does not work. I first tried to trim down all un-needed packages until I realized that every time I uninstalled something, space was running even lower on the root filesystem. How comes?

To understand why, you need to know that the default firmware image is built on top of squashfs, a read-only compressed filesystem. On top of that, openwrt adds an overlay filesystem, an image that registers all changes from the underlying read-only partition. When you delete a file from such a setup you actually write on top of it in the overlay filesystem a mention that this file should not appear any more. There is another option using a read-write filesystem (jffs2) for the firmware but apparently it can get into trouble on many routers so I did not even try.

One solution is to build your own openwrt image and embed only the modules you need. That one is tough. You need to dedicate an x64 machine for that, download a million source files, prepare all the compilers and build tools you ever heard of, fine tune the package configuration, press make, and wait for a few hours. You end up with an image a bit smaller than the four dreaded megabytes with a build system taking about 3GB of space on the build host.

To be fair, the openwrt build system is a wonderful thing. It starts by creating its own toolchain for cross-compilation for your target hardware, then builds up a whole Unix from scratch and ends up packaging everything into this tiny image. Very impressive. I have seen very few professional build systems that are as clean as this one. Congratulations to the Openwrt for working out such a beautiful system!

If you are not into compiling your own stuff, you have another option:
Image Generator
https://wiki.openwrt.org/doc/howto/obtain.firmware.generate

This method uses pre-built components which you assemble into a minimal firmware image. That goes a lot faster (a matter of seconds) and does not eat up gigabytes of disk space. Select the few minimal modules you need to at least boot up the router on the network, and be able to mount USB storage. That is what I ended up doing.

I flashed an image that contained just enough software to run a kernel with network support (duh), USB storage support, package management, and some basic utilities. Plugging the USB storage worked immediately. Yay!

Let’s start again: I have 8GB of space on USB dongle, far more than I will ever need for the router itself. I created a first ext4 partition for 700 megs, which should be enough to hold every possible package, configuration files, and even log files if needed. I added a second 128 meg partition for swap space, and left the rest as a single ext4 partition.

The router was booted again with USB dongle plugged in. Let it boot, log onto it as root, and set up the root filesystem as an overlay from the USB first partition. Reboot the router and there you go: your router space has grown from 4 megs to 700.

This brilliant howto here explains exactly how to do it:
https://wiki.openwrt.org/doc/howto/extroot

Once you have enough space you can start adding back all the packages you need. Just to give you an idea, I have there:

  • A web interface to set everything up
  • Tools to monitor the device and display nice graphs on the web interface
  • DNS server, WiFi Access Point, Firewall
  • Development tools: gcc, git, python, lua, strace, screen, vim
  • A Samba server
  • sshfs to mount remote filesystems locally
  • OpenVPN either as client or server

After that it is a matter of taking time to configure each service specifically. The Openwrt wiki pages are a real treasure in this respect.  Everything you need is there, together with additional ideas for cool stuff to do. One hour later I was done adding VPN support, Samba service for the remaining of the USB dongle space, an ad blocker, a local web site, and a local fixed address for administration (an alias on eth0).

There is one point really worth mentioning about the Openwrt philosophy.  If you have some Unix experience, you know that every piece of server software you install comes with its own system of configuration directories and files with a specific new syntax to learn, a different place to find log files, and another way to handle PID files. Sure there is some standardization on the way with most config files under /etc and logs under /var/log, but it seems everybody needs to invent a new syntax for configuration and logging. See a previous blog post about that topic.

Openwrt places all configuration files in a single place and they all respect the same key/value syntax. It is straightforward to understand, edit, diff, store in version control, or pretty-print. That alone helps smooth out the learning curve every step of the way. Gives you a definite feeling of things done in a proper, consistent, and very clean manner. So there you go: for all your network-y things, you know where to look. The Internet of things starts right here with these boxes, guys.

My next endeavours with OpenWRT will probably need my soldering iron. Those GPIOs on the board look tempting…

Written by nicolas314

Wednesday 9 December 2015 at 12:31 am

Posted in hardware, openwrt, router

Tagged with , ,

Ad Blocking Appliance

with one comment

BH_LMC

Finally found some use for this Raspberry Pi B (version 1) I had lying around taking dust: an ad blocker for the whole household! Yay!
The whole thing took me less than an hour to install and no time at all to setup. You download and burn a DietPi image to an SD card, insert it into your RPi, and boot. Log through ssh as root and follow the instructions. Let it download black lists from the net and you are ready to go.
All machines on my network are now set to use the RPi address as DNS. This works furiously well it is nothing short of amazing! Spotify without the ads, Android apps without annoying popups and ads. Things just work out of the box. I wish I had found this earlier.
Oh, and if you do not want to dedicate a Raspberry Pi for this, you can always install the package separately on a Debian box, but you need to leave it running 24×7.

The author explains how it works in a dedicated blog post, but to put it in a nutshell here is how it works:

  • Pi-hole provides a script to aggregate lists of known ad sites: gravity.sh. Run this every once in a while.
  • Pi-hole uses dnsmasq, a lightweight DNS server. The pi-hole instance is set to use the list from gravity.sh as a black list.
  • Pi-hole sets up a local web server answering all queries with a blank page. You also get a local (PHP) web site giving you statistics about number of blocked domains.

Best use I have ever had for this tiny box so far!

Links:

 

Written by nicolas314

Sunday 29 November 2015 at 11:52 pm

Posted in hardware, raspberrypi, router, seo

Tagged with , ,