Nicolas314

All my geeky stuff ends up here. Mostly Unix-related

Put on your shoes

leave a comment »

shoes


– Mister engineer, we are about to leave the house. Could you please lace your shoes?

– I’m afraid I can’t do that before at least next year.

– What? No! We are leaving the house right now. Tie your shoes and let’s go!

– Well, it is obvious you have not been in the shoe-lacing business for quite a while mate. See: in order to tie my shoes I’d have to get my hands closer to my feet. I see three main possibilities:

1. I lower myself down to the level of my feet (and shoes), which is dangerously close to the ground. I could trip and fall, bringing me to ground level with sufficient speed to hurt my nose, probably causing bleeding in the process. Who would want to leave blood on the floor? You don’t want me to hurt myself, do you? This would take us to a large amount of blood cleaning and nose healing, which could take a lot of time and make us both look bad in case someone on the street asks why I have a bloody nose.

2. I could bring the shoes up to my level. Considering my feet would stop touching the ground, I would have very little time to complete the movement needed to effectively tie a knot to what could be considered decent shoe-lacing. Bad knots would make us look bad, and we do not want someone to notice that we are not even able to come out on the street with properly tied shoes.

3. The third and last possibility is to wait for my feet to grow up enough so that my shoes do not fit any more. This would probably trigger some shoe-buying and shoe-replacing, which could then be put to practical use to purchase a new pair of lace-free shoes, which would then solve all the above issues once and for all.

My conclusion is that we should wait until my feet have grown enough. See you in a couple of months.

– Man, you have reached the end of my patience. Let me tie those shoes for you.

– I’m afraid I can’t let you do that, Dave. Your role as a caretaker is not to take responsibilities and do things in my stead, but to teach me to be autonomous and let me do that myself. In addition, may I let you know that I have had these shoes for a few months now and you have never laced them before in your entire life, therefore I am the only suitable person to achieve that.

– C’mere, let me do it.

– Are you questioning my authority with respect to my own shoes? When you bought them you said they were mine!

– They are still yours, let me just lace them.

– You did not understand the above mentioned points. Apologies for my poor choice of words, I always forget that English is not your native language and you may not get the full power of the most subtle nuances.

– Don’t patronize me. Just don’t.

– Oh that was never my intention. In order to patronize someone…

– WILL YOU FUCKING TIE YOUR SHOES?

– Why the harsh language? Is that really needed? I have only given you the current status and all you can do is react strongly against me. I have not invented laces, nor did I decide to place my own hands at a different altitude than my own feet. I suggest you review our options and come to your senses before we do something we might regret.

– Do you see my hand? I swear it can fly and land on your face in no time.

– Let’s not be too hasty now. I would have to inform legal of your perceived intentions and will have to quote your language. Research indicates that people in your situation have very little chances of winning a legal fight that involves strong wording and physical violence.

– … You know what? You… You just stay here, Ok?

– That’s what I have been telling you all the time. Glad you finally came to your senses mate.

Advertisements

Written by nicolas314

Monday 9 July 2018 at 11:08 pm

What time is it?

leave a comment »

clock


– Hello Mr. Engineer, can you tell me what time it is?

– No I can’t.

– Why?

– Well then. You see, my watch is an electronic and mechanic device based on the oscillation of a quartz that imprints a periodic movement to a set of cogs, which are then de-multiplied to lower the base quartz frequency from 32,758 Hz to exactly 1 Hz, i.e. one beat per second.

– That’s very nice. And what time does your watch show now?

– I could tell you but it would not be useful. See, the quartz frequency is not exactly that power of two, it is itself oscillating with a larger period around that value, meaning that my watch can be ahead or behind by some amounts that are hard to measure, let alone predict.

– So it is inaccurate?

– Yes! You can never tell exactly the time with that kind of device.

– Ok… Seriously, what time is it?

– Not only are the watch mechanics imprecise, but they do not take relativistic effects into account.

– That so?

– Yep. Since Einstein we know time is nowhere absolute. When I put my arm up like this, time flows a little slower because of the Earth rotation, and if I put it down like this is goes a bit faster. Or is it the other way around? Anyway, my time reference is unlikely to be the same as yours since we are not moving around in sync.

– Listen, this is all very nice but that was not my question. Will you tell me the time it shows now and I will deal with the imprecision myself?

– No can’t do.

– Why is that?

– Even if you discard all relativistic effects and frequency drifts, the notion of time is not something universal on Earth.

– Care to explain?

– Time is only valid in a given time zone. Since the end of the 19th century we have split world regions according to time zones which keep changing at regular intervals based on political choices. In order to be able to tell you the time of day, I need to know a reference time in a given place and convert that depending on your position on the planet. We could use GMT, which stands for Greenwich Mean Time, but it is not even indicating the current time in Greenwich UK. I could then program a microservice that could give you the current date/time based on an estimated position from your IPv4 address, provided you are not too close to a time border. But then that assumes you have Internet access. Oh wait, do you have an iPhone or an Android?

– Er… Thanks mate. So let’s say we use the current time zone, Ok?

– Do you know if we apply Daylight Saving Time where you stand?

– How would I know? Yes, probably!

– Probably with what probability? Because we could weigh the answer depending on… Hey, where are you going?

– To lunch. I just remembered I wanted to ask you if it was time for lunch.

 

Written by nicolas314

Monday 9 July 2018 at 10:39 pm

Camels

leave a comment »

camels

I read somewhere in a math history book that numbers were actually invented to count camels. Someone wanted to send over a herd of camels to be sold on a market on the other side of the desert and they did not trust the camel escort. How would the receiving party know if some camels had not been stolen on the way? So they used a fairly simple principle: line up your camels, put one pebble in front of each. Gather the pebbles, put them in a small jar, burn the cork, hand it over to the escort.

On the receiving end, break the jar, put one pebble in front of each camel. You will know immediately if camels are missing.

This apparenly went on for a while, until someone figured out that instead of lining up pebbles and camels you could shorten the process by writing signs on the jar to indicate how many pebbles were inside. On the receiving end you just had to look at the signs and compare to what you saw. In case of doubt, break the jar and line up pebbles and camels. And then it was just a matter of time until somebody noticed you don’t need the pebbles and the jar. Just cook a clay tablet in an oven with a text indicating how many camels you are sending.

I have no idea if this story is true or not, but I like the way it stresses the breakthroughs that have happened. Going from a bijection pebbles/camels to a bijection in camels/signs was brilliant. I expect the first attempts were likely to just draw plain strokes on the jar, as many as there were camels in the herd. The next breakthrough was simplifying a whole bunch of strokes into a single sign, e.g. using a hand to signify the number 5. And the last one was to realize that the jar and pebbles were unneeded.

Another shift that amazes me to this day is how money actually works. The first currency tokens had actual value, they were made of metal you could melt and use if you so wanted. When the first bank notes were introduced, they switched from actual value to a potential: the note said that you could obtain real metal if you were to exchange that note in a bank.

We now live in a world where I can pay my lunch by waving a piece of plastic over a radio-equipped terminal connected to a bank. My plastic contains numbers that cannot be found on any other credit card, which are used to authenticate me. Now my bank makes a promise to pay my meal to the restaurant’s bank. No metal or paper changes hands.

Since a few years, things are shifting again. Instead of waving a credit card containing my unique account identification numbers, I can now use a mobile phone that contains a series of numbers that are only valid for myself, my account, for today, and for limited amounts. This is what they call tokenization and the reason it is booming is that it is a lot simpler to store temporary tokens with limited value than long-term banking credentials with unlimited powers. Security needs not be that high, though you still need to be able to authenticate account owners in a very secure way, but there are plenty of ways to achieve that.

Among the strongest methods we know today to authenticate someone, the most popular relies on the fact that you cannot split a big number into a multiplication of primes. If you tried with a gigantic computer, it would require more heat to power than is available in the universe.

We have come a long way since camel-counting.

Written by nicolas314

Wednesday 1 November 2017 at 11:37 pm

Posted in fun

Tagged with , ,

I am not your daughter

leave a comment »

sleepless

You called me quite late. Some time during the middle of the night, and for whatever reason I had left my phone on. I picked it up and was greeted by your anxious voice:
– Isabelle, is that you?
Part of my brain was still actively dreaming, but the part that was emerging found the idea preposterous. Do I sound like an Isabelle? I croaked:
– No Madam, this is not Isabelle.
– Oh come on Isabelle, it’s Mum. Stop playing fool with me, I recognize your voice, please..
Even half awoken, the tension in your voice was definitely noticeable. You wanted to talk to your daughter and nothing would stop you.
– Madam, I can assure you I am not your daughter. In fact I am a man and my name is Nicolas.
– Bullshit! Isabelle, talk to me!
Your old lady’s tone left me no choice but to obey, so I gave up and decided to play along.
– Alright Mum, you got me. What’s up?
You seemed surprised. Apparently Isabelle does not give up so easily when playing this game. But the sudden joy of being able to talk to your daughter was so great that you could not help it. You started talking about your neighbours at the retirement home, how the nurses were treating you, and had many complaints about the food and such. I listened very carefully at first and quickly dozed off, we were half way during the night after all.

You called again a bit later, and again, and again. We spent our night like this: you finally talking to your daughter, and me sleeping through 10-minute intervals. Finally one of your nurses must have found out you were secretly phoning at night and you stopped calling.

You never called again. I hope you found the right number for Isabelle and she takes good care of you.

Written by nicolas314

Monday 2 October 2017 at 9:42 am

Posted in Uncategorized

Long live NAT!

leave a comment »

ipv6-no-thanksHome networking can be a lot of fun: setting up a name service, a guest network, or traffic rules, leads to an endless joy of discovering new RFCs or creativity in the very active field of artistic configuration file syntax.

I thought I had seen everything until I tried to set up IPv6 connectivity for my home network. Little did I know that this would eat up so many of my precious free evenings. The following writeup is here to remind me never to try that kind of shit ever again, and as a warning to future generations who might want to dig into this kind of topic. Life is short, there are many better things to do than attempt to set up a new addressing scheme for your home network. Long live the NAT king!

The Start

It all began when I noticed that my ISP provided me with a unique (native!) IPv6 prefix to use on my home network. Something like:

2001:1234:5678:9abc::/56

Since I was not familiar with IPv6 addresses, it took me a while to find out that the first 64 bits of a 128-bit IPv6 address designate a network, and the last 64 are reserved to differentiate hosts on that network. My provider handing me a /56 means I have 64-56 = 8 bits to play with, i.e. I can instantiate 256 home networks, each having up to 2^64 = 18,446,744,073,709,551,616 hosts. Overshot a bit, maybe.

So where do I start? Do I have to install specific software? Where? Do I need to buy specific hardware? How many services are needed? And thereby started my long painful descent into the horrific world of IPv6. Toss and loose 1d20 sanity points immediately.

My ISP unfortunately did not provide any help as to what I am supposed to do with the IPv6 thingie they gave me. No single help page, very few discussions on their forums, and all exchanges I had with customer service were completely useless. Best I could find were discussions between customers of an ISP in the US that provides a similar setup. That is thin.

Say you received a /56 prefix from your ISP. If that prefix ever changes e.g. because you switched to a new ISP, you want things to work automagically because that is the way things currently work with IPv4: changing my public IPv4 address does not change anything to my home network.

In order to do that, IPv6 suggests that home networks use two sets of addresses: the public ones derived from the ISP-provided /56, and another private address space based on something else called a ULA (Unique Local Addresses). You get to choose your own ULA on your home network(s), preferrably based on a good random number generator, but nothing prevents you from taking something like fc00:caca:caca:caca:caca::/48. If anybody else on the Internet picks the same network prefix you will get into trouble when trying to get intimate with each other, e.g. by establishing a VPN between both worlds. We had exactly the same problem when trying to join two sites using IPv4 NAT’d 10.0.0.0/8 subnets, so this is not really a regression. Fun fact: if you have no ULA in France you can always say “Il manque ULA sur mon réseau”.

How do you get to choose this ULA? If you happen to have a single router on your home network it should just be a matter of digging through the router IPv6 setup until you find it. But most home networks are now running multiple routers that are all unaware of each other, and all convinced they are masters of the universe. You will most certainly end up with several ULAs. Some of your devices will get several addresses and you will have to understand your own network topology to know which address to use to access them. Prepare for glorious hours of debugging, which is particularly great when facing addresses that are mostly made of bloody random bits.

Why several routers on the same home network? Simply because you may be running several DSL connections, or maybe you have a VPN started somewhere away from your edge router, or maybe you connected your smartphone and it offers another potential exit to the Internet. You also get a virtual router when you start virtual machines on a desktop.

To make things simpler, every network interface on your machines will also generate a local address that is only valid for its closest neighbours, called a link-local address. Unfortunately you won’t go far with that one as it is not supposed to cross boundaries. Think of it as a 127.0.0.1 that extends to the other side of the cable but not further.

Ok so we have now several adresses for each machine on the network.  Figuring out which one should be used (incoming or outgoing) is just an unspecified, incredible mess. The link-local address can only be used on very specific physical links, the ULA address cannot be routed to the Internet, and the public addresses you have may change at any moment, e.g.  through your smartphone sharing a 4G access.

At that point we have just determined that your printer currently identified as ‘printer’ also known as 192.168.1.20 in IPv4 will now be accessible as:

– fe80:bffa:3d5f:5f8d:b4cf:1749:b01c:5b2f for machines directly connected to it through an Ethernet cable
– fc00:c465:3b76:b34d:38f7:da19:2586:1cbd for machines living on the same internal network.
– 2001:61af:ff44:b148:4fc3:0097:f35d:c806 for machines on the internet when reached through a first ISP, and another public address for each available ISP connection.

Oh joy.

Of course normal human beings are not meant to remember this kind of random shit. For this kind of thing you have DNS.

DNS you said? What DNS?

There are really two ways machines can obtain an IPv6 address: SLAAC and DHCPv6. SLAAC means Stateless Address Auto Configuration, whereby a machine obtains a prefix and derives its own IP address from it, e.g. based on its own MAC address. Cool, right? You do not have to assign individual addresses in static DHCP leases, every machine does it on its own. But then: how do you know which address was self-assigned by your very smart printer?

There are dedicated neighbour-discovery protocols for that, but they are mainly designed to make sure that addresses are locally unique and routers know where to find them. This is only taking care of establishing a link, there is nothing dedicated to associating a name to a self-assigned IP address. And if there was, how would you know who to believe? If two machines on the local network claimed to be ‘joe’, what should happen?

To be fair, there are solutions like Bonjour, also known as zeroconf, but they are unlikely to work on lightweight or old devices. Shoot again.

Back to square one: if you want to reach your own machines using human-usable names you need to run DHCPv6, a protocol that was designed to compensate for such things. And there you go: back to static leases, addresses assigned by a router, attached to a name, and you end up doing exactly the same kind of shit you used to do with IPv4 local networks, except this time the addresses are much easier to screw up.

Even worse: if the self-assigned IPv6 addresses are not related to MAC addresses, it means every single host on your local network will have generated its own random address, forcing you to manually harvest them from all devices. But you know how to do that on your connected toaster, right?

What’s in it for the average home network user? Pretty much nothing. The fact that every single one of your home devices has a potentially reachable address on the intertubes is downright scary. Internet service is for internet servers, not for sensors and other IoT bullshit. First thing you will want to do is bullet-proof your firewall to make sure nobody but you can access your printer from the Internet, and hope things are Ok with your IoT shit.

The story did not just end up with me reading thousands of pages on the Internet and a couple of paper books. I hacked every single computer in my house to run IPv6, starting with the routers under OpenWRT, LEDE, FreeBSD, OpenBSD, pfSense, OPNSense, and later moving on to all client OS machines: OSX, Linux, Android, *BSD, and even some Windows boxes, blimey.  I instantiated dedicated DHCP and DNS servers, configured static addresses, automatic ones, bridges and NATs and firewall rules and what-have-you, and I ended up with some machines working under IPv6 only, some under IPv4 only, some that could use both stacks, and some (a lot) that were just unreachable no matter what. Yeah, I also crashed my Internet access several times. Omelet and eggs.

Let me try to put it this way: some of my home machines are servers, e.g. a NAS or a printer. I want to be able to print on ‘printer’ or mount a share on ‘NAS’ without having to remember random 128-bit numbers. Silly me. Since I want to use names I have to assign addresses myself from a router running DHCPv6. Neither NAS nor printer need to be available to the public. So what did I gain compared to a local IPv4 network? Hmm… Address management is not fun with 32 bits, imagine with 128.

Or maybe I am just old-fashioned, trying to manually assign names to my home machines. This might be an idea for a new product: a router that would automatically identify hosts on the home network and show them on a single web interface, allowing you to assign names and forget about addressing altogether. Might get in trouble when you have several identical devices but I’m sure there would be a way. If such a product exists I have not seen it yet.

On the other hand, if I want to browse the Interwebs in v6, I found out that mounting a SOCKS proxy on a remote cloud box works perfectly well. No need to configure anything, just ssh -D and the IPv6 world is mine to browse.

Summing it all up

Address assignment is not easier than IPv4. Still requires a dedicated DHCP and DNS server, only more complicated to configure. You are facing the tedious task of gathering self-assigned IPv6 addresses from all hosts and copying them onto your DHCPv6 server, hoping the self-assignment method won’t change soon.

Routing is now different, but not easier. New constraints are imposed on knowing which interface to bind to when reaching out to the Internet.

Firewalling the whole thing with a mix of IPv4 and IPv6 might tear you a new one. I can already lock myself out of a router with human-readable firewall rules, I cannot imagine doing the same thing with batshit-crazy addresses and feel safe.

You know what? I will stick to glorious NAT’ing until this mess is sorted out. Good news is that there are many bright people currently working on the topic. All I hope is they eventually come up with something that you and me can use without having to read through a million pages of RFCs, compile obscure daemons, or purchase new boxes as if I did not have enough of them.

Talking about RFCs, this one is trying to gather very sensible requirements about home networks:

https://tools.ietf.org/html/rfc7368

If you have 20 minutes to spare, you should watch this talk:
https://www.youtube.com/watch?v=wQdfWUsG4uI

If you really insist on switching your home network to IPv6, I would recommend reading this rant first:

IPv6 at home (published 2012, still relevant):
http://www.kloepfer.org/ipv6-homenet.html

And to get an idea about how messy it is to get IPv6 configured on Linux:

IPv6 Set up an IPv6 LAN with Linux
https://www.jumpingbean.co.za/blogs/mark/set-up-ipv6-lan-with-linux

In its current state I can only dismiss the current IPv6 definition for home networks as very incomplete and unworkable for non-professionals.  Let’s hope RFC 7368 will be handled by qualified, creative, and pragmatic people.

Til then, there is no place like 127.0.0.1

Written by nicolas314

Tuesday 28 February 2017 at 11:41 pm

My own little farm

with 3 comments

zotac_ci323_03Virtualization is fun! Virtual Machines are nothing new, we have all been using VirtualBox, qemu, or VMWare at some point to try out new stuff, bring up the odd Windows instance to run annoying software, or whatever. At work we use thousands of VMs for millions of things. The hardware price tag is pretty hefty though: if you want to start a reasonable number of VMs on the same racked server you need very large amounts of RAM and disk space, placing it beyond reach in terms of price for home usage.

Not any more! Prices are dropping for heavy machinery faster than the time it takes to look up prices on Amazon. I found this little gem from Zotac and purchased one for a mere 180 euros from a French site:

Zotac CI323

The little beast sports a quad-core CPU, two Realtek NICs, and a whole bunch of USB ports (including two USB3). Add on top of that an extension card for WiFi and Bluetooth. Perfect choice to build a home router in a VM and leave space for other VM instances. You need to add RAM and disk, the box comes empty. I scavenged 8GB RAM and an SSD disk from a previous build and off we go.

It has been a while since I last had a look at virtualization solutions.  Took me several days to look them up individually and find out what they offer. All the solutions I tried are described below.

Option 1: run VirtualBox on a desktop

Install a convenient desktop like Mint or Ubuntu, run VirtualBox on top.  Unfortunately not a very good option as the VMs would not be as close to the metal as I would want. Dismissed.

Option 2: run Linux containers

Containers are neat but they are Linux only. I would like to run BSD and maybe Windows VMs too on the same hardware, so dismissed.

Option 3: Run a bare metal hypervisor

The main options I could find are:

  • VMWare: run VMWare OS as hypervisor, run any OS on top.
  • bhyve (pronounced like beehive), the FreeBSD hypervisor
  • Proxmox
  • KVM: use virtualization routines offered in the Linux kernel. This can be started from any Linux distro and conveniently run pretty much any OS.
  • Xen: use a Xen kernel as bare-metal hypervisor, run any OS on top.

VMWare ESXi was my first choice but had to be quickly dismissed: my box NICs are Realtek and VMWare dropped support for those a few versions back.  Annoying. There are convoluted HOWTOs explaining how to hack the install ISO to add missing drivers and stuff but I do not want to play that game. The whole setup would probably be broken in the following release so no thanks.

I installed FreeBSD 11 and tried out bhyve. Installing FreeBSD on this particular hardware was a real chore: for some reason the integrated SD card reader has driver issues and booting the machine took up to 10 minutes because of a nasty timeout spitting out kernel traces. I finally succeeded in disabling the driver on boot by adding stuff to device.hints after hours of googling and tests. To be fair, I have always faced issues with hardware support on FreeBSD, but to be completely fair: these are the only issues I ever faced. The OS is so polished and professional it is a real pleasure to use. Other parts of the box were immediately recognized and activated: Realtek NICs and the WiFi+Bluetooth (Intel) board.

Anyway: bhyve is relatively easy to learn, documentation is good enough, and it should run any BSD or Linux-based VM without any effort. Running Windows or OSX VMs would probably not be a good idea though. I have not tried but it seems a bit daring. If bhyve offered an easy-to-use GUI I might have stuck with it, but I finally dismissed it because it is still too young compared to other existing solutions.

KVM: the idea would be to install a very small Linux instance and use it to manage VMs on top with KVM. I tried several:

Ubuntu desktop is far too heavy for a “very small Linux instance”. I cannot believe a simple desktop is using so much RAM and CPU. I tried to manually remove stuff after a default installation and broke the machine most completely after having erased ‘evolution’. Forget it.

Ubuntu server is fine enough without GUI, but I would like to have a minimal X11 environment to run VM management software. Unfortunately, as soon as you start adding GUI stuff to an Ubuntu server you start piling up gigs of desktop software you do not want. I could probably figure it out but did not have the patience to do it.

Arch Linux is a royal pain to install. Manjaro (a fairly straight Arch derivative) gets you to a fully configured machine in a matter of minutes.  Problem is: I do want stability on my VM farm and a rolling release is probably not the best choice. Dismissed.

Minimal Debian install worked great. All hardware perfectly supported. And then I tried some KVM tutorials, messed up a bit further with Xen tutorials, and ended up with a completely borked machine. Don’t ask me what went wrong, I just got frustrated of randomly killing processes and rebooting the hardware. There are certainly good HOWTOs out there explaning how to transform a base Debian install into a Xen/KVM server but I did not find them. Dismissed.

Alpine Linux to run KVM: did not try, but seems like a possible option.

I tried Proxmox but the default ISO does not install, it crashes miserably after a few minutes of timeout. I have no idea what is going on, but I dismissed Proxmox at that point and came back to it later. Read on.

At that point I was left with Xen as bare metal hypervisor. I focused on Xen Server, a free Citrix project. The OS is based on CentOS 7 with a modified kernel and a GUI on top.

The XenServer install procedure is rather straightforward. Answer a few questions and let it roll. On the next reboot you get an ncurses-based interface on the console that allows you to achieve the bare minimum: configure the host, start/stop VMs, that kind of stuff. You can also do the same through ssh (ssh in then use xconsole).

Beyond that you need to find a Windows desktop because the only management solution they offer is a heavy Windows client. You get a very decent management interface that looks a lot like the VMWare Sphere client, from which you can control pretty much everything. The fact that it only runs on Windows is a major pain but to be honest: you only use it to configure new VMs. Once they are started you access them through ssh, vnc, or rdesktop, so no need to maintain a live Windows machine just for that.

In less than two hours I managed to install on XenServer:

  • A minimal Alpine Linux running nginx
  • An OPNSense instance
  • A pfSense instance
  • A Windows 8.1 desktop
  • A FreeBSD 11.0 VM, no X11

I still felt like something was missing though: XenServer would not recognize my WiFi/Bluetooth board. It would have been cool to dedicate a VM to make a stand-alone access point, so I kept trying more stuff.

Among all the options I tried, the only one that had all my hardware covered without hitch was Debian. Proxmox is based on Debian jessie, so if I succeed in installing it there should be a way to make things work. Let’s try again. I started from Debian and installed Proxmox on top. The guide I used is here:

https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie

This works and happens to be quite smooth.

NB: I managed to completely destroy my setup when I decided to change the host IP address without telling Proxmox first. Rebooting the machine does not help, it goes into an endless loop, fails to reconfigure the network, and dies in horrible pain. I took the shortest path and re-installed from scratch. Good advice: DO NOT CHANGE THE PROXMOX HOST IP ADDRESS.

Proxmox is now working beautifully well. The advantages over XenServer for me are multiple:

  • LXC + KVM support: Proxmox supports LXC containers and KVM Virtual Machines in approximately the same way. Of course, containers are much lighter to install, start up, shut down, or backup.
  • Proxmox is completely open-source. XenServer probably has proprietary parts somewhere, though I did not investigate more than that.
  • Proxmox offers a pure Web interface: no need for a heavy Windows client.  You can also open a VNC console on any virtual machine directly from your browser, which is incredibly convenient.
  • Based on Debian, Proxmox identified and supports all my hardware.

Just for fun, I created a local WiFi access point based on alpine Linux by instantiating an LXC container, assigning the wlan0 interface to it, and booting the right daemons.

The next VMs I created are:

  • An alpine Linux desktop under LXC
  • Various alpine Linux boxes under LXC to run simple services
  • An Ubuntu desktop (under KVM)
  • A Windows 8 desktop (under KVM)
  • A MacOS Sierra desktop 
  • pfSense and OPNSense as KVM appliances, to evaluate them
  • An OpenBSD box to play with pf in command-line mode
  • A FreeBSD11 box

All these virtual goodies run on the same hardware as I write these lines.

My next task will be to select a solution to use as a home virtual firewall appliance. Meanwhile I am just having fun popping up and down virtual machines as my mood goes.

Completely useless but tons of fun!

Written by nicolas314

Tuesday 8 November 2016 at 3:43 pm

OpenWRT on EdgeRouter Lite

leave a comment »

erlite-3-900x500Installing OpenWRT on EdgeRouter Lite

This installation procedure does not require any extra hardware beyond a Phillips screwdriver to open the router box. I believe it is completely reversible and (hopefully) does not void your warranty.

Objective: replace EdgeOS on your EdgeRouter Lite by a recent version of
OpenWRT picked from the LEDE project.

You need:

  • A powerful Linux box, preferrably a multi-proc 64-bit machine with tons of RAM. This will only be used just once to compile OpenWRT.
  • An extra USB thumb drive. Its dimensions should be short: less than a few centimeters, otherwise it won’t fit inside the box.

Ok now off to build an OpenWRT image:

. Log onto your Linux box

. Download the latest sources from LEDE project:

 git clone https://github.com/lede-project/source
 cd source

. Prepare the tree for compilation for ERLITE:
You will have to select a number of options to build an image tailored to
the EdgeRouter Lite. I could put here a ready-made config file but as these
things tend to evolve quickly, it would probably be obsolete in a matter of
days. So bear with me: start the configuration with

make menuconfig

. Target System: Cavium Networks Octeon
. Target Profile: Ubiquiti EdgeRouter Lite

. Target Images: make sure ‘ext4’ is selected, then Select that line to
open up a menu for ext4 configuration. Change the number of inodes to
60,000 instead of the default 6,000. Also select GZip images, and finally modify the root filesystem partition size to something more comfortable,
say 500 MB. This space will be taken off your USB stick so if you have
more space you can increase that to whatever you have. With 500 MB you
should have enough space to put all the packages you need.
. Global build settings: select Select all kernel module packages by
default.

Beyond that take your pick for packages you want included by default in
your image. My selection is:

  • Base system: base-files, block-mount, busybox, ca-bundle, ca-certificates, dnsmasq, dropbear, firewall, fstools, jsonfilter, lede-keyring, libc, libgcc,
    libpthread, librt, libstdcpp, mtd, netifd, opkg, procd, rpcd, sqm-scripts, sqm-scripts-extra, swconfig, ubox, ubus, ubusd, uci, usign
  • Administration: sudo
  • Development: ar, binutils, gcc, gdb, make, objdump
  • Kernel modules: everything should already be selected as module. You want
    to change some of these to be compiled into the kernel otherwise it will
    fail to find the ext4 root on USB:

    • Filesystems: select kmod-fs-ext4, kmod-fs-msdos
    • USB Support: kmod-usb-core, kmod-usb-storage, kmod-usb-storage-extras
  • Languages: select whatever programming languages you want to see in a
    default install. I usually make sure at least Lua and Python are selected.
  • LuCI: make sure LuCI is selected. Take your pick for applications you
    want to install. I usually select luci-app-openvpn, luci-app-commands,
    luci-app-firewall.
  • Network: if you want your router to act as an OpenVPN client or server,
    make sure it is selected under VPN. Pick either openvpn-openssl or openssl-polarssl.
  • Utilities: bash, bc, file, grep, gzip, less, lsof, openssl-util, strace, tar, tmux, usbutils

Feel free to select more packages but each additional one will take extra
compilation time.

. Type ‘make’ and let the magic go on.
. When finished, the result is stored as:

bin/targets/octeon/generic/lede-octen-erlite-ext4-sysupgrade.tar

This file contains everything we need to build a bootable USB drive for
the EdgeRouter Lite. This file should be 500 MB large since you
selected that size above for your root filesystem, but it is mostly made
of zeroes so if you use bzip2 you should be able to reduce its size to a
more manageable 50-60 MB, which is more convenient if you need to toss it
around the network.

. Put the sysupgrade.tar file onto a local Linux machine and extract it:

tar xvf lede-octen-erlite-ext4-sysupgrade.tar

. The directory contents are:

sysupgrade-erlite/
sysupgrade-erlite/kernel
sysupgrade-erlite/root
sysupgrade-erlite/CONTROL

. Now insert your USB thumb drive into a local Linux machine and prepare
the filesystem. We need a first (small) FAT32 partition to hold the kernel,
a second 500 MB partition to hold the root:

fdisk /dev/sdX # Where X is the letter assigned to your USB drive
New partition: 1, 32 MB in size, type c (WIN95 FAT32 LBA)
New partition: 2, 500 MB in size, type Linux (default)
Optional: New partition: 3, the rest of your drive, type Linux (default)
Make the first partition bootable (a).
Type 'w' to save your changes.

Create the FAT32 filesystem with:
mkfs.vfat /dev/sdX1

. The default uboot configuration on the EdgeRouter Lite wants a file
called ‘vmlinux.64’ in the first (DOS) partition, so let’s do just that:

mount /dev/sdX1 /mnt # Mount the DOS partition
cp sysupgrade-erlite/kernel /mnt/vmlinux.64
umount /mnt

. Dump the root filesystem contents onto the second partition:

dd if=sysupgrade-lite/root of=/dev/sdX2 bs=1M

. If you have a third partition, create a new filesystem on it with:

mkfs.ext4 /dev/sdX3

. You are done with the USB drive!
. Open the EdgeRouter Lite. There are three small screws to remove on the
back. The box slides open if you push gently.

. Remove the existing USB stick inserted in the reader on the motherboard.
Be gentle: you need to insist a bit to take it off but it is not stuck.

. Insert the USB drive you prepared. Close the box, put the screws back,
and boot the router.

. If you connect a PC to the central NIC (labeled eth1) you should receive
an address on 192.168.1.0/24 from which you can ssh to 192.168.1.1 or open
a browser to http://192.168.1.1

. Welcome to OpenWRT/LEDE! Set a root password and you should be done.

The first things you probably want to do:

  • Change interface names to associate eth0 to WAN, and bridge eth1 and eth2 to
    LAN.
  • Edit the configuration to mount the third partition on the USB drive on /home. This is cool to add non-root users and give them a real flash storage.
  • Run ‘opkg update’, install missing packages.
  • Install OpenVPN configurations and test them.
  • Add ssh keys for root login in /etc/dropbear/authorized_keys
  • Install dotfiles to feel at home

Problems I have seen and their solutions:

The LEDE build is not so robust, sometimes it fails in parallel mode
because some dependencies seem to be compiled too late. If you get
compilation errors, using ‘make -j1’ should solve all issues. On a powerful
server with tons of RAM you need 2-3 hours to compile the whole set,
depending on how many packages you selected.

The version of LEDE you just compiled will quickly be out of sync with the
official package repositories. As soon as the kernel is changed in the LEDE project HEAD, all kmod packages from the LEDE repositories will refuse to install with opkg. This is the reason why you had to select “Build all kernel modules” in menuconfig: all kernel modules are already part of the image you created. This problem should go away once LEDE has released its first stable version.

I had most trouble with the ext4 filesystem definition: my first attempt
generated an ext4 of 50 MB which is far too small. After increasing that
size to 1GB, I still ran into “not enough disk space” errors and figured
out the number of inodes was too low (6,000). If you install a lot of
packages you need more inodes. Both points are addressed in the above
procedure. I also tried with an insanely high number (600k inodes) and the
resulting filesystem cannot be mounted.

Filesystem size is indicated in bytes, but fdisk counts in MB, based on a
power of 2. This yields a small discrepancy between the 500MiB filesystem
you generated with the build and the 500MB you reserved in the partition
table.

Once up and running, my router quickly ran into starvation problems. One
machine on the network could use the whole bandwidth and cut off every
other machine. I installed QoS packages: sqm-scripts, sqm-scripts-extra,
and luci-app-sqm, configured the queue to a fair scheduler, and got rid of
starvation issues. For some reason I could not get the pre-compiled
versions of these packages to work, I had to re-install them from official
repositories.

I wanted to add a Samba server to be able to use the rest of the USB drive
as a shared space but it is not a good idea. Samba takes ages to compile
and the daemon uses too many resources for such a small piece of hardware.

If you want to add ssh keys for the root user, remember the default ssh
server is dropbear, not openssh. dropbear expects root ssh keys to be
stored in /etc/dropbear/authorized_keys. You can also add root ssh keys
through LuCI.

The default shell for root is /bin/sh. You can change it to /bin/bash after
installing it and modifying the root entry in /etc/passwd.

Enjoy your fancy new router!

 

Written by nicolas314

Sunday 16 October 2016 at 4:58 pm