All my geeky stuff ends up here. Mostly Unix-related

Archive for the ‘Debian’ Category

My own little farm

with 3 comments

zotac_ci323_03Virtualization is fun! Virtual Machines are nothing new, we have all been using VirtualBox, qemu, or VMWare at some point to try out new stuff, bring up the odd Windows instance to run annoying software, or whatever. At work we use thousands of VMs for millions of things. The hardware price tag is pretty hefty though: if you want to start a reasonable number of VMs on the same racked server you need very large amounts of RAM and disk space, placing it beyond reach in terms of price for home usage.

Not any more! Prices are dropping for heavy machinery faster than the time it takes to look up prices on Amazon. I found this little gem from Zotac and purchased one for a mere 180 euros from a French site:

Zotac CI323

The little beast sports a quad-core CPU, two Realtek NICs, and a whole bunch of USB ports (including two USB3). Add on top of that an extension card for WiFi and Bluetooth. Perfect choice to build a home router in a VM and leave space for other VM instances. You need to add RAM and disk, the box comes empty. I scavenged 8GB RAM and an SSD disk from a previous build and off we go.

It has been a while since I last had a look at virtualization solutions.  Took me several days to look them up individually and find out what they offer. All the solutions I tried are described below.

Option 1: run VirtualBox on a desktop

Install a convenient desktop like Mint or Ubuntu, run VirtualBox on top.  Unfortunately not a very good option as the VMs would not be as close to the metal as I would want. Dismissed.

Option 2: run Linux containers

Containers are neat but they are Linux only. I would like to run BSD and maybe Windows VMs too on the same hardware, so dismissed.

Option 3: Run a bare metal hypervisor

The main options I could find are:

  • VMWare: run VMWare OS as hypervisor, run any OS on top.
  • bhyve (pronounced like beehive), the FreeBSD hypervisor
  • Proxmox
  • KVM: use virtualization routines offered in the Linux kernel. This can be started from any Linux distro and conveniently run pretty much any OS.
  • Xen: use a Xen kernel as bare-metal hypervisor, run any OS on top.

VMWare ESXi was my first choice but had to be quickly dismissed: my box NICs are Realtek and VMWare dropped support for those a few versions back.  Annoying. There are convoluted HOWTOs explaining how to hack the install ISO to add missing drivers and stuff but I do not want to play that game. The whole setup would probably be broken in the following release so no thanks.

I installed FreeBSD 11 and tried out bhyve. Installing FreeBSD on this particular hardware was a real chore: for some reason the integrated SD card reader has driver issues and booting the machine took up to 10 minutes because of a nasty timeout spitting out kernel traces. I finally succeeded in disabling the driver on boot by adding stuff to device.hints after hours of googling and tests. To be fair, I have always faced issues with hardware support on FreeBSD, but to be completely fair: these are the only issues I ever faced. The OS is so polished and professional it is a real pleasure to use. Other parts of the box were immediately recognized and activated: Realtek NICs and the WiFi+Bluetooth (Intel) board.

Anyway: bhyve is relatively easy to learn, documentation is good enough, and it should run any BSD or Linux-based VM without any effort. Running Windows or OSX VMs would probably not be a good idea though. I have not tried but it seems a bit daring. If bhyve offered an easy-to-use GUI I might have stuck with it, but I finally dismissed it because it is still too young compared to other existing solutions.

KVM: the idea would be to install a very small Linux instance and use it to manage VMs on top with KVM. I tried several:

Ubuntu desktop is far too heavy for a “very small Linux instance”. I cannot believe a simple desktop is using so much RAM and CPU. I tried to manually remove stuff after a default installation and broke the machine most completely after having erased ‘evolution’. Forget it.

Ubuntu server is fine enough without GUI, but I would like to have a minimal X11 environment to run VM management software. Unfortunately, as soon as you start adding GUI stuff to an Ubuntu server you start piling up gigs of desktop software you do not want. I could probably figure it out but did not have the patience to do it.

Arch Linux is a royal pain to install. Manjaro (a fairly straight Arch derivative) gets you to a fully configured machine in a matter of minutes.  Problem is: I do want stability on my VM farm and a rolling release is probably not the best choice. Dismissed.

Minimal Debian install worked great. All hardware perfectly supported. And then I tried some KVM tutorials, messed up a bit further with Xen tutorials, and ended up with a completely borked machine. Don’t ask me what went wrong, I just got frustrated of randomly killing processes and rebooting the hardware. There are certainly good HOWTOs out there explaning how to transform a base Debian install into a Xen/KVM server but I did not find them. Dismissed.

Alpine Linux to run KVM: did not try, but seems like a possible option.

I tried Proxmox but the default ISO does not install, it crashes miserably after a few minutes of timeout. I have no idea what is going on, but I dismissed Proxmox at that point and came back to it later. Read on.

At that point I was left with Xen as bare metal hypervisor. I focused on Xen Server, a free Citrix project. The OS is based on CentOS 7 with a modified kernel and a GUI on top.

The XenServer install procedure is rather straightforward. Answer a few questions and let it roll. On the next reboot you get an ncurses-based interface on the console that allows you to achieve the bare minimum: configure the host, start/stop VMs, that kind of stuff. You can also do the same through ssh (ssh in then use xconsole).

Beyond that you need to find a Windows desktop because the only management solution they offer is a heavy Windows client. You get a very decent management interface that looks a lot like the VMWare Sphere client, from which you can control pretty much everything. The fact that it only runs on Windows is a major pain but to be honest: you only use it to configure new VMs. Once they are started you access them through ssh, vnc, or rdesktop, so no need to maintain a live Windows machine just for that.

In less than two hours I managed to install on XenServer:

  • A minimal Alpine Linux running nginx
  • An OPNSense instance
  • A pfSense instance
  • A Windows 8.1 desktop
  • A FreeBSD 11.0 VM, no X11

I still felt like something was missing though: XenServer would not recognize my WiFi/Bluetooth board. It would have been cool to dedicate a VM to make a stand-alone access point, so I kept trying more stuff.

Among all the options I tried, the only one that had all my hardware covered without hitch was Debian. Proxmox is based on Debian jessie, so if I succeed in installing it there should be a way to make things work. Let’s try again. I started from Debian and installed Proxmox on top. The guide I used is here:

This works and happens to be quite smooth.

NB: I managed to completely destroy my setup when I decided to change the host IP address without telling Proxmox first. Rebooting the machine does not help, it goes into an endless loop, fails to reconfigure the network, and dies in horrible pain. I took the shortest path and re-installed from scratch. Good advice: DO NOT CHANGE THE PROXMOX HOST IP ADDRESS.

Proxmox is now working beautifully well. The advantages over XenServer for me are multiple:

  • LXC + KVM support: Proxmox supports LXC containers and KVM Virtual Machines in approximately the same way. Of course, containers are much lighter to install, start up, shut down, or backup.
  • Proxmox is completely open-source. XenServer probably has proprietary parts somewhere, though I did not investigate more than that.
  • Proxmox offers a pure Web interface: no need for a heavy Windows client.  You can also open a VNC console on any virtual machine directly from your browser, which is incredibly convenient.
  • Based on Debian, Proxmox identified and supports all my hardware.

Just for fun, I created a local WiFi access point based on alpine Linux by instantiating an LXC container, assigning the wlan0 interface to it, and booting the right daemons.

The next VMs I created are:

  • An alpine Linux desktop under LXC
  • Various alpine Linux boxes under LXC to run simple services
  • An Ubuntu desktop (under KVM)
  • A Windows 8 desktop (under KVM)
  • A MacOS Sierra desktop 
  • pfSense and OPNSense as KVM appliances, to evaluate them
  • An OpenBSD box to play with pf in command-line mode
  • A FreeBSD11 box

All these virtual goodies run on the same hardware as I write these lines.

My next task will be to select a solution to use as a home virtual firewall appliance. Meanwhile I am just having fun popping up and down virtual machines as my mood goes.

Completely useless but tons of fun!

Written by nicolas314

Tuesday 8 November 2016 at 3:43 pm

Seagate dockstar rescue tips

leave a comment »

Seagate dockstar freeagent

Seagate dockstar freeagent

My Seagate dockstar freeagent (shortnamed: dock) has recently received a brand new hard disk to cope with my large data needs. dock has been serving me so well over the past months that I decided to entrust the new disk with the complete operating system (Debian) and let it run from there. Little did I know that the brand new hard disk would fail miserably one week later, taking away my only copy of Debian for dockstar. Now I do you unbrick these things again? I spent a couple of nights hunting for information and performing experiments so will document that and leave it here in case it may be useful to someone else.

The boot system

Dockstar is an ARM-based micro-computer with a very interesting feature: the whole boot system resides in flash memory. No grub, no LILO, no messing around with the boot sector every time you upgrade the kernel. Only problem is: out of the box the default dockstar only tries to boot its own PogoPlug system and nothing else. First thing you want to do is replace your stock flash boot by his.

Update uBoot on your Dockstar

In my case I happened to have fried the default Pogoplug installation on dock. I believe this happened when I mounted the flash partition from Debian as jffs2 filesystems. For some reason this corrupted everything on partitions /dev/mtd[123] and I could not recover them from old backups. Fortunately I did not try to mount /dev/mtd0, which is probably what saved Jeff’s boot sequence.

Another excellent point for the dockstar boot sequence is the Marvell bootloader installed in ROM on the device (this one cannot be fried). This is by far the most powerful and user-friendly boot system I have ever seen. You can get the OS from any connected device, from the network, over tftp, you name it. Commands are nicely documented and it is a pleasure to navigate. The only point is: if you want to have a chance to catch the boot sequence while the machine is loading, you need to connect a JTAG cable as there is no video card onboard. Quite unfriendly. The other solution is to ask the boot software to communicate with another machine on the same network. This is achieved by setting a few configuration variables and is completely described here:

Use netconsole to troubleshoot uBoot without a serial cable

You will see incoming console text and will be able to take over the boot sequence from a simple netcat. I cannot recommend enough that you do this before anything bad happens; finding a JTAG cable in the middle of the night is not always easy. Once you have configured dock for netconsole and made sure you use Jeff’s boot system, you should be on the safe side.

Rescue systems

Just in case you end up with no bootable drive as I did, you may want to keep a couple of files handy. Johannes explained on Jeff’s forum how to boot from tftp in this post:

Rescue system for use with USB stick and tftp

I would also recommend to go one step beyond and replace your PogoPlug installation by a real rescue system. Jeff again offers a complete downloadable system that will make sure your dockstar always boots on something even with no network (and netconsole) or USB drives attached.

Recovery system ready for use

Installing Debian

The easiest way to prepare a hard disk for Debian on the ARM-based Dockstar is to boot from it into a minimal system, hook up the destination hard drive and bootstrap Debian from there using debootstrap. This is exactly what Jeff put together on this page:

Run Linux on your Dockstar

Theoretically you could prepare the same hard drive from a very standard PC but debootstrap unfortunately does not support (yet) cross-platform installation. The only way to do that from an x86 or x64 PC would be to run an ARM emulator and run debootstrap from there, using your hard drive as a target. I tried using qemu in ARM mode but got some weird errors and gave up after many tries. If you want to avoid having to become an expert about cross-compilation, better make sure you can boot your dockstar and run the install from there.

As a beautiful side-effect: if you ever decide to move the dockstar Debian to another disk, no need for dd and gparted magic. A simple ‘cp -ar’ can do the trick since you do not have to deal with boot software on the boot sector or such things.

Executive summary

  • Do not mount /dev/mtd* under a running Debian. This fscked my flash partitions and restoring with dd proved useless
  • Flash your /dev/mtd0 with Jeff’s replacement. Do it now!
  • Activate netconsole so you can take over from another PC on the same network
  • Replace your default PogoPlug with a real rescue system

Putting the boot loader/manager in flash is a brilliant idea, I wish standard PCs had moved to such an option earlier. The Marvell bootloader is especially versatile with a surprisingly rich online help and excellent capabilities. The booting part is often neglectd by hardware vendors but it proves to make the difference between an expensive plastic brick and usable hardware.

A million thanks to Jeff Doozan for making his knowledge available and accessible!

Written by nicolas314

Monday 7 March 2011 at 12:32 am

Posted in Debian, dockstar, Unix

Tagged with , , , , ,

Goodbye Ubuntu, Hello Debian

leave a comment »

I bought a MicroClient Senior for my son to use as a desktop machine.  Nothing really fancy: just enough to read his e-mail, browse the web, play some games, edit files with OpenOffice, listen to music or watch movies.  The best part of it is that he can leave the machine on forever as it uses even less energy than the light on his desk.

In order to make things easy for him to use, I initially chose to go with Ubuntu (Intrepid) and go straight to the default Gnome desktop to avoid as many hassles as possible. Took me a while to put things together as it initially came without network, sound or video. Welcome back to 1995!

Wi-Fi: the dongle is claimed by a driver (rtl8187) that handles things great (with WPA encryption) for about 5 minutes and then crashes without warning. ndiswrapper does a fine job but you need to install the very latest version and locate the correct Windows drivers. Total: a couple of evenings spent trying to get the native driver to work, then another hour to locate a decent Windows driver.

Sound: Not correctly handled by ALSA, you need to download OSS directly from the 4-Front web site and then perform a global re-configuration of all sound-needing apps. Total: one evening spent tweaking ALSA in vain, followed by OSS download/install.

Video: the OpenChrome driver is not activated by default and the screen resolution is set to something weird. You need to hunt for a correct xorg.conf and tweak it to your needs. Total: one evening spent trying various openchrome drivers (including manually compiling stuff) and xorg.conf files.

Once I had everything stabilized and running, I still had to do some maintenance every now and then, in general whenever a newer kernel version is installed.

Last week I decided to upgrade the OS to Ubuntu Jaunty and it turned out to be a disaster. After the upgrade reported having performed successfully, the machine woke up again with no sound, no video and no network. No matter how I tweaked the options and configuration files, it remained in an unstable state, unable to even get to an ‘apt-get update’ because of the lack of network support. I finally gave up after spending two more evenings trying to get the situation back in control, and finished by installing a new Debian (Lenny) from scratch.

Now the machine is back on its feet. It showed the same symptoms as usual (no network, sound or video) but this time I applied the usual fixes and everything was back to normal in no time. Funnily enough, the rtl8187 driver worked perfectly fine until the first kernel update, after which it went back to an Ubuntu-like behaviour: 5 mins of network then crash.  ndiswrapper saves the day again. Not saying Ubuntu is bad but every time they break something and I have to find new ways of bringing it back to life. Debian is also buggy but at least it is easily corrected.

It is now the third machine from which I had to delete Ubuntu because it just did not work. The other ones were a Dell laptop (almost no hardware worked) and an AMD-64 desktop.

Written by nicolas314

Friday 19 June 2009 at 4:53 pm

Posted in Debian, junior, Ubuntu