Nicolas314

All my geeky stuff ends up here. Mostly Unix-related

Archive for the ‘Unix’ Category

My own little farm

with 3 comments

zotac_ci323_03Virtualization is fun! Virtual Machines are nothing new, we have all been using VirtualBox, qemu, or VMWare at some point to try out new stuff, bring up the odd Windows instance to run annoying software, or whatever. At work we use thousands of VMs for millions of things. The hardware price tag is pretty hefty though: if you want to start a reasonable number of VMs on the same racked server you need very large amounts of RAM and disk space, placing it beyond reach in terms of price for home usage.

Not any more! Prices are dropping for heavy machinery faster than the time it takes to look up prices on Amazon. I found this little gem from Zotac and purchased one for a mere 180 euros from a French site:

Zotac CI323

The little beast sports a quad-core CPU, two Realtek NICs, and a whole bunch of USB ports (including two USB3). Add on top of that an extension card for WiFi and Bluetooth. Perfect choice to build a home router in a VM and leave space for other VM instances. You need to add RAM and disk, the box comes empty. I scavenged 8GB RAM and an SSD disk from a previous build and off we go.

It has been a while since I last had a look at virtualization solutions.  Took me several days to look them up individually and find out what they offer. All the solutions I tried are described below.

Option 1: run VirtualBox on a desktop

Install a convenient desktop like Mint or Ubuntu, run VirtualBox on top.  Unfortunately not a very good option as the VMs would not be as close to the metal as I would want. Dismissed.

Option 2: run Linux containers

Containers are neat but they are Linux only. I would like to run BSD and maybe Windows VMs too on the same hardware, so dismissed.

Option 3: Run a bare metal hypervisor

The main options I could find are:

  • VMWare: run VMWare OS as hypervisor, run any OS on top.
  • bhyve (pronounced like beehive), the FreeBSD hypervisor
  • Proxmox
  • KVM: use virtualization routines offered in the Linux kernel. This can be started from any Linux distro and conveniently run pretty much any OS.
  • Xen: use a Xen kernel as bare-metal hypervisor, run any OS on top.

VMWare ESXi was my first choice but had to be quickly dismissed: my box NICs are Realtek and VMWare dropped support for those a few versions back.  Annoying. There are convoluted HOWTOs explaining how to hack the install ISO to add missing drivers and stuff but I do not want to play that game. The whole setup would probably be broken in the following release so no thanks.

I installed FreeBSD 11 and tried out bhyve. Installing FreeBSD on this particular hardware was a real chore: for some reason the integrated SD card reader has driver issues and booting the machine took up to 10 minutes because of a nasty timeout spitting out kernel traces. I finally succeeded in disabling the driver on boot by adding stuff to device.hints after hours of googling and tests. To be fair, I have always faced issues with hardware support on FreeBSD, but to be completely fair: these are the only issues I ever faced. The OS is so polished and professional it is a real pleasure to use. Other parts of the box were immediately recognized and activated: Realtek NICs and the WiFi+Bluetooth (Intel) board.

Anyway: bhyve is relatively easy to learn, documentation is good enough, and it should run any BSD or Linux-based VM without any effort. Running Windows or OSX VMs would probably not be a good idea though. I have not tried but it seems a bit daring. If bhyve offered an easy-to-use GUI I might have stuck with it, but I finally dismissed it because it is still too young compared to other existing solutions.

KVM: the idea would be to install a very small Linux instance and use it to manage VMs on top with KVM. I tried several:

Ubuntu desktop is far too heavy for a “very small Linux instance”. I cannot believe a simple desktop is using so much RAM and CPU. I tried to manually remove stuff after a default installation and broke the machine most completely after having erased ‘evolution’. Forget it.

Ubuntu server is fine enough without GUI, but I would like to have a minimal X11 environment to run VM management software. Unfortunately, as soon as you start adding GUI stuff to an Ubuntu server you start piling up gigs of desktop software you do not want. I could probably figure it out but did not have the patience to do it.

Arch Linux is a royal pain to install. Manjaro (a fairly straight Arch derivative) gets you to a fully configured machine in a matter of minutes.  Problem is: I do want stability on my VM farm and a rolling release is probably not the best choice. Dismissed.

Minimal Debian install worked great. All hardware perfectly supported. And then I tried some KVM tutorials, messed up a bit further with Xen tutorials, and ended up with a completely borked machine. Don’t ask me what went wrong, I just got frustrated of randomly killing processes and rebooting the hardware. There are certainly good HOWTOs out there explaning how to transform a base Debian install into a Xen/KVM server but I did not find them. Dismissed.

Alpine Linux to run KVM: did not try, but seems like a possible option.

I tried Proxmox but the default ISO does not install, it crashes miserably after a few minutes of timeout. I have no idea what is going on, but I dismissed Proxmox at that point and came back to it later. Read on.

At that point I was left with Xen as bare metal hypervisor. I focused on Xen Server, a free Citrix project. The OS is based on CentOS 7 with a modified kernel and a GUI on top.

The XenServer install procedure is rather straightforward. Answer a few questions and let it roll. On the next reboot you get an ncurses-based interface on the console that allows you to achieve the bare minimum: configure the host, start/stop VMs, that kind of stuff. You can also do the same through ssh (ssh in then use xconsole).

Beyond that you need to find a Windows desktop because the only management solution they offer is a heavy Windows client. You get a very decent management interface that looks a lot like the VMWare Sphere client, from which you can control pretty much everything. The fact that it only runs on Windows is a major pain but to be honest: you only use it to configure new VMs. Once they are started you access them through ssh, vnc, or rdesktop, so no need to maintain a live Windows machine just for that.

In less than two hours I managed to install on XenServer:

  • A minimal Alpine Linux running nginx
  • An OPNSense instance
  • A pfSense instance
  • A Windows 8.1 desktop
  • A FreeBSD 11.0 VM, no X11

I still felt like something was missing though: XenServer would not recognize my WiFi/Bluetooth board. It would have been cool to dedicate a VM to make a stand-alone access point, so I kept trying more stuff.

Among all the options I tried, the only one that had all my hardware covered without hitch was Debian. Proxmox is based on Debian jessie, so if I succeed in installing it there should be a way to make things work. Let’s try again. I started from Debian and installed Proxmox on top. The guide I used is here:

https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie

This works and happens to be quite smooth.

NB: I managed to completely destroy my setup when I decided to change the host IP address without telling Proxmox first. Rebooting the machine does not help, it goes into an endless loop, fails to reconfigure the network, and dies in horrible pain. I took the shortest path and re-installed from scratch. Good advice: DO NOT CHANGE THE PROXMOX HOST IP ADDRESS.

Proxmox is now working beautifully well. The advantages over XenServer for me are multiple:

  • LXC + KVM support: Proxmox supports LXC containers and KVM Virtual Machines in approximately the same way. Of course, containers are much lighter to install, start up, shut down, or backup.
  • Proxmox is completely open-source. XenServer probably has proprietary parts somewhere, though I did not investigate more than that.
  • Proxmox offers a pure Web interface: no need for a heavy Windows client.  You can also open a VNC console on any virtual machine directly from your browser, which is incredibly convenient.
  • Based on Debian, Proxmox identified and supports all my hardware.

Just for fun, I created a local WiFi access point based on alpine Linux by instantiating an LXC container, assigning the wlan0 interface to it, and booting the right daemons.

The next VMs I created are:

  • An alpine Linux desktop under LXC
  • Various alpine Linux boxes under LXC to run simple services
  • An Ubuntu desktop (under KVM)
  • A Windows 8 desktop (under KVM)
  • A MacOS Sierra desktop 
  • pfSense and OPNSense as KVM appliances, to evaluate them
  • An OpenBSD box to play with pf in command-line mode
  • A FreeBSD11 box

All these virtual goodies run on the same hardware as I write these lines.

My next task will be to select a solution to use as a home virtual firewall appliance. Meanwhile I am just having fun popping up and down virtual machines as my mood goes.

Completely useless but tons of fun!

Advertisements

Written by nicolas314

Tuesday 8 November 2016 at 3:43 pm

iPod on Ubuntu

with 3 comments

The iPod apparently just turned 10 today. Happy birthday iPod!

As it happens, both my kids have an iPod now, together with a docking station that enables them to listen to all of their music whenever they have a chance. Living in the future is fun! I have fond memories of piling up cassettes containing copies of the greatest albums ever, gathered here and there from friends. Blank tapes were quite expensive so we bought them in Germany where they did not levy the copy tax. I owned maybe 100 tapes in total, maybe 150 albums, the equivalent of a half-Gb today. My sons have each 32 times this on a device that fits their small pockets, without talking about the huge difference in terms of sound quality.

Anyway, my 14-year old has a desktop PC running Ubuntu and I wanted him to be completely independent with his iPod. Turns out he cannot.

Plugging the iPod into Ubuntu works fine: a popup indicates you have plugged a music device, the icon even looks like an iPod, and if you click away you end up starting Banshee or Rhythmbox or whatever you chose to handle music on that desktop.

Things start getting ugly when you add music to the device. Files are transferred and definitely stored there but the iPod does not recognize any of it as music, only “unknown data”. I tried re-formatting the device from scratch on Ubuntu but it failed. I had to re-format it twice again after that: first time on a Mac, which created a Mac filesystem on the iPod that was not recognized on Ubuntu (fffffffuuuuuuuuuuuu), and then on a Windows PC running iTunes to re-generate a Windows filesystem Ubuntu can work with.

Still no love. Whatever I put on the iPod is only visible to Ubuntu. I tried Banshee, Rhythmbox and a couple others to no avail. Back to square one.

I know there must be solutions out there: re-format the iPods with a Linux firmware or help the guys reverse-engineer the latest iPod filesystems for open-source support, but I am just tired with this. I just wanted my son to be independent with his music and ended up spending a whole evening just messing around, piling up experiments and wasting my time Googling my way around.

There is no technical reason why things should be this way. This completely artificial lock-in into iTunes is just ridiculous. I do not know yet which portable music player I will purchase next but something tells me it won’t be from Apple.

Written by nicolas314

Monday 24 October 2011 at 11:06 pm

Posted in ipod, Ubuntu

Tagged with , , ,

Config file hell

with one comment

Complexity

xkcd 963 nailed it once again. How much fun is it to have to open one of the zillion Unix config files on your Debian box and start tweaking until it finally works? The graph could just as well show “Time since I last opened wpa_supplicant.conf”, or “/etc/network/interfaces”, “fontconfig”, “httpd.conf”, or “crontab”.

Unix is famous for its configurability. Unfortunately it has never offered a single convenient base library to support configuration file parsing. Every little piece of software had to design its own format, generate its own syntax rules (which complexity mostly depend on the programmer’s talent for parser writing), and force you to learn yet another language that will take you hours to understand and minutes to forget. Examples:

crontab

Seems the syntax for crontab files is just a bit beyond what my brain can absorb. I have set millions of cron jobs in my life and still cannot write one without copy/paste from an existing file.

sudoers

The associated man page does not just describe a set of options, it goes as far as defining a full-fledged language by providing a formal BNF grammar, as if end-users were yacc compilers. It even features significant whitespace. Yummy.

procmailrc

Before we had spam filters, procmail was the only wall between a sane inbox and a wave of unsollicited messages. But there was a heavy price to pay: learn how to write a bug-free procmailrc with no way to test it except to send yourself half a million fake e-mails until you got it right. I still have a couple of procmailrc templates somewhere just in case I ever have to get into this again.

sendmail.cf

Sendmail.cf will probably earn the gold medal of the most obscure, un-debuggable, impossible to write and just frankly insane configuration file there ever was. It is now mentioned in the Geneva convention about the non-proliferation of mental-illness-inducing configuration formats. But let us not be too harsh. Without it we would not have had the case of the 500-mile email and a lot less horror stories to tell our children.

Hopefully things are getting better. Now that we have XML we can tear away the last shreds of hope of ever understanding how to configure a piece of software by editing a file. And if you are really vicious, you could go as far as creating an XML-like config file format that cannot be validated.

I once had a problem, then I discovered XML, and then I had two problems.

Written by nicolas314

Wednesday 12 October 2011 at 10:44 pm

Seagate dockstar rescue tips

leave a comment »

Seagate dockstar freeagent

Seagate dockstar freeagent

My Seagate dockstar freeagent (shortnamed: dock) has recently received a brand new hard disk to cope with my large data needs. dock has been serving me so well over the past months that I decided to entrust the new disk with the complete operating system (Debian) and let it run from there. Little did I know that the brand new hard disk would fail miserably one week later, taking away my only copy of Debian for dockstar. Now I do you unbrick these things again? I spent a couple of nights hunting for information and performing experiments so will document that and leave it here in case it may be useful to someone else.

The boot system

Dockstar is an ARM-based micro-computer with a very interesting feature: the whole boot system resides in flash memory. No grub, no LILO, no messing around with the boot sector every time you upgrade the kernel. Only problem is: out of the box the default dockstar only tries to boot its own PogoPlug system and nothing else. First thing you want to do is replace your stock flash boot by his.

Update uBoot on your Dockstar

In my case I happened to have fried the default Pogoplug installation on dock. I believe this happened when I mounted the flash partition from Debian as jffs2 filesystems. For some reason this corrupted everything on partitions /dev/mtd[123] and I could not recover them from old backups. Fortunately I did not try to mount /dev/mtd0, which is probably what saved Jeff’s boot sequence.

Another excellent point for the dockstar boot sequence is the Marvell bootloader installed in ROM on the device (this one cannot be fried). This is by far the most powerful and user-friendly boot system I have ever seen. You can get the OS from any connected device, from the network, over tftp, you name it. Commands are nicely documented and it is a pleasure to navigate. The only point is: if you want to have a chance to catch the boot sequence while the machine is loading, you need to connect a JTAG cable as there is no video card onboard. Quite unfriendly. The other solution is to ask the boot software to communicate with another machine on the same network. This is achieved by setting a few configuration variables and is completely described here:

Use netconsole to troubleshoot uBoot without a serial cable

You will see incoming console text and will be able to take over the boot sequence from a simple netcat. I cannot recommend enough that you do this before anything bad happens; finding a JTAG cable in the middle of the night is not always easy. Once you have configured dock for netconsole and made sure you use Jeff’s boot system, you should be on the safe side.

Rescue systems

Just in case you end up with no bootable drive as I did, you may want to keep a couple of files handy. Johannes explained on Jeff’s forum how to boot from tftp in this post:

Rescue system for use with USB stick and tftp

I would also recommend to go one step beyond and replace your PogoPlug installation by a real rescue system. Jeff again offers a complete downloadable system that will make sure your dockstar always boots on something even with no network (and netconsole) or USB drives attached.

Recovery system ready for use

Installing Debian

The easiest way to prepare a hard disk for Debian on the ARM-based Dockstar is to boot from it into a minimal system, hook up the destination hard drive and bootstrap Debian from there using debootstrap. This is exactly what Jeff put together on this page:

Run Linux on your Dockstar

Theoretically you could prepare the same hard drive from a very standard PC but debootstrap unfortunately does not support (yet) cross-platform installation. The only way to do that from an x86 or x64 PC would be to run an ARM emulator and run debootstrap from there, using your hard drive as a target. I tried using qemu in ARM mode but got some weird errors and gave up after many tries. If you want to avoid having to become an expert about cross-compilation, better make sure you can boot your dockstar and run the install from there.

As a beautiful side-effect: if you ever decide to move the dockstar Debian to another disk, no need for dd and gparted magic. A simple ‘cp -ar’ can do the trick since you do not have to deal with boot software on the boot sector or such things.

Executive summary

  • Do not mount /dev/mtd* under a running Debian. This fscked my flash partitions and restoring with dd proved useless
  • Flash your /dev/mtd0 with Jeff’s replacement. Do it now!
  • Activate netconsole so you can take over from another PC on the same network
  • Replace your default PogoPlug with a real rescue system

Putting the boot loader/manager in flash is a brilliant idea, I wish standard PCs had moved to such an option earlier. The Marvell bootloader is especially versatile with a surprisingly rich online help and excellent capabilities. The booting part is often neglectd by hardware vendors but it proves to make the difference between an expensive plastic brick and usable hardware.

A million thanks to Jeff Doozan for making his knowledge available and accessible!

Written by nicolas314

Monday 7 March 2011 at 12:32 am

Posted in Debian, dockstar, Unix

Tagged with , , , , ,

The upgrade disease

with one comment

It is getting harder and harder to simply use a computer due to a spreading disease that unfortunately affects all OS’s and major pieces of software: the upgrade disease.

Windows: at any moment there are countless processes living just for the sake of checking whether a piece of software is up-to-date or not. Last time I checked on a Windows box I saw:

  • Java update scheduler
  • Google update
  • Apple update
  • Blackberry software update

Cumulated, these processes use up to 100 Mbytes of memory and probably other resources like file descriptors and sockets, and of course CPU time. Most of them come with no option to turn off automatic updates and when they do, the update process may still live on but do nothing (looking at you, Java).

Windows update is also running in background at regular intervals, usually choosing the worst moment to interrupt my work. When it starts I can safely assume I will not be able to do anything on my PC for 15-60 minutes and will have to reboot one or more times. Oh joy!

Ubuntu runs UpdateManager at regular intervals too. Things would be fine if this bloody application did not use modal windows and steal my focus while I am typing something on a terminal. Half of the time I just happen to be hitting the Return key at the precise moment when the window appears, unwillingly triggering the upgrade process. CPU and network usage after that are just unbearable. I usually take a break at that point.

OSX is a bit friendlier: the upgrade manager pops up at some point asking for permission to install stuff and warning you when you need a reboot afterwards — at least the window is not modal. When you finally decide to run the upgrade you have to agree to endless pages of unlegible end-user license agreements for a frigging mp3 reader that you already bought with
the OS a few months back.

On Windows the situation is even worse if you take into account the inevitable anti-virus that continuously runs in background, keeping the CPU hot and eating away memory. I had at some point a corporate XP laptop running the IT-blessed anti-virus tools with such efficiency that it was just impossible to do anything else but watch the machine scan its disks
full-time. Remind me: why did I ask for a PC, again?

At some point it would be great to remind our friendly OS makers that some users care about actually doing something with their computers. Software developers: if you ever plan to add an auto-update feature on your code, try following these:

  • Do not impose a perpetually running process running in background just for updates. There are cron jobs for that kind of things. You could also check for newer versions when explicitly asked by the end-user.
  • Do not assume that because your software can find an Internet connection you are allowed to go download on your own several hundred megabytes of software upgrade. Sometimes I like to do something else with my bandwidth.

Eating away resources in background without notifying the user or even offering them some opt-out box is just plain rude. This is similar to a default Windows feature call pre-fetching where the OS thinks it is smart to start as many application as possible after booting, just in case the user chooses to run one. It took me easily 15 minutes after bootup of my XP laptop to be able to actually start doing stuff with it. The pre-loading of most commonly used applications had saturated memory (2Gb!!) and slowed everything to a crawl.

One last point about updates: I declare I have the right to refuse upgrading a piece of software. Upgrading vital pieces of software may just break something and I may not be interested in spending the rest of the day finding out how to repair it. The endless update and reboot reminders on Windows are driving me nuts.

Written by nicolas314

Sunday 17 October 2010 at 7:27 pm

Posted in osx, Ubuntu, windows

Tagged with

Unix world domination

leave a comment »

Jean-Louis Gassée is still alive and kicking! In this post he describes how Unix has taken over the world of operating systems today. Linux, Solaris, Mac OSX, but also iPhone OS, Android, MeeGo and a bazillion embedded computer systems are all based on Unix.

Fifteen years ago, Unix machines were workstations or mainframes, they cost a fortune and were quite slow (by today’s standards). As a student I would never have dreamed of having one at home some day, let alone carry one in my pocket to listen to music. The first time I ran ps aux on my iPod, something screamed inside me that Unix had finally achieved world domination.

Next step: realize Linux is a toy (a nice one, but a toy) and push something really professional to the top: QNX, BSD, Solaris, whatever. I would feel very insecure flying in a plane controlled by a Linux box, knowing that the OOM Killer could decide to shoot us down at any moment.

For reference: the infamous out-of-memory killer started its life on Linux with this discussion on the kernel mailing-list.

Written by nicolas314

Wednesday 6 October 2010 at 8:59 pm

Posted in Unix

Tagged with

Random programming

leave a comment »

Today’s random quote is simply beautiful.

“If you put a million monkeys at a million computers for a million years, one of them would eventually write a Java program. The rest would write Perl.” –Anonymous

Written by nicolas314

Wednesday 14 October 2009 at 10:19 am

Posted in Unix

Tagged with ,