All my geeky stuff ends up here. Mostly Unix-related

Posts Tagged ‘go

One-time file-sharing

leave a comment »

oneSay you rent a box somewhere on the Internet. You installed Debian stable on it because you want it to be nice and stable and run a few daemons that are useful to have online. Could be to hold your vast music collection, family pictures, or use it as remote storage for backup. Imagine you wanted to share some of the files hosted on this box with your relatives, who may or may not be computer-literate. Most of them would know how to use a webmail but asking them to install an ftp client is just beyond reach.  Obviously, you do not want to give these guys too many rights over your box (like an ssh access for scp). What are the solutions?

Setting up a dedicated HTTP server

Simple enough: set up an HTTP server to distribute static files. lighttpd is simple enough to setup in a couple of minutes and is very efficient for static stuff. But you do not want to distribute your files to the whole Internet. Sooner or later a web spider will crawl in and index your family pictures and all sorts of things you never meant to be public.  Next step: configure password-protection on the server

Fair enough. Now you have limited file downloads to people who know the password — provided they know how to enter a password. Do you create multiple accounts, one for each of your peers? It would be preferrable, otherwise you will never know who downloaded what. But then you have to communicate their passwords to your peers and make sure they have a procedure in case they forget it. You know you are headed straight to massive butt pains.

Second issue: passwords can be shared. You shared that 2GB movie with a couple of friends and a couple of weeks later you find out that there are currently 1,549 active downloads for this file. Sharing is in human nature and that is completely Ok, but you probably did not sign up to become a content distributor over the whole Internet, only with a couple of friends and relatives.

Next step: use one-time authentication

There are better solutions out there: since you only mean to share one single file (or set of files) each time, you do not need to create accounts for your friends. You give them a one-time download token and forget about it.

A one-time download token is a URL. It looks like the kind of URLs you get from URL shorteners with the funny string at the end. Something like http://shortener/12398741

One-time tokens can be shared but since they can only be used once, the person who shared it has lost it. The token is randomly generated and invalidated immediately after it is used to avoid having robots automatically scan all possible URLs in a row until they find a valid one.

There are many ways to achieve this on regular HTTP servers. Apache probably has a million configuration options for user authentication, including one-time passwords or something similar, but I have to admit I did not even try. I already wasted enough of my life in Apache config files. lighttpd can be configured to do that but the only solution I found required some Lua scripting and I did not feel up to the task.

Next-step: Do It Yourself

After reviewing countless pages of configuration options for various HTTP servers, I decided that it would be shorter for me to implement this in a tiny web app rather than try and understand complex configuration options.  My first iteration made use of a Python FCGI script in attached to a lighttpd process. Pointing out static files from a Python web app to the embedding lighttpd process is reasonably simple.

This implementation suffered from a number of pitfalls though. For one thing, performance was bad. For some reason, the Python process would eat insane amounts of CPU and RAM when sending big files, slowing down the server to a crawl. Second showstopper was the complexity involved for such a simple setup. I had to write a Python script to generate the lighttpd configuration file with a number of deployment options: where to put config files, log files, static files, port number, etc. And then came the inevitable issues with dependencies: Python version versus version versus lighttpd version.  Some combinations worked fine, some did not.  Nothing specific to Python or lighttpd, but the more you have gears, the more you have places for grains of sand to fit in.

I still survived with this setup for a year or so, when Go came in. I have already reviewed the language in the past and will not come back to that, but suffice it to say that developing HTTP servers in Go is the most natural thing. Adding the one-time token ingredient to the soup was implemented in just one evening.

Once rewritten in Go, I found out that the end-result was about just as big as the Python implementation, excluding the script that created the lighttpd config. The main difference was of course that I do not have to maintain cross-references between package versions for Python, lighttpd, and, since there is only one dependency to cover: Go itself.

It was straightforward to enhance the program to support more options, respond to favicon, and handle a JSON-readable database for active tokens.  Performance is astounding. The serving Go process never takes more than a few megs of RAM (about the size of the executable itself) and only uses tiny amounts of CPU since the process is mostly I/O based anyway.

There is one thing I should have foreseen and had to re-implement. I am sending the one-time links by email and more and more people are reading their emails from their smartphone or tablet. Many just clicked the link without thinking twice, triggering a 2-4GB download and killing both their mobile and data plan at the same time. Wrong move.

The next version features a two-time download page: the first link sends users to a page summarizing the download, offering a second link to actually start the real thing with a big warning about the size of what will actually be sent.

There are many other features I would like to add to the current version, and I am hoping other people have better ideas for new features, which is the reason why I shared it on github. Find it here:

Since we are talking about sharing private date between friends and relatives, protecting the download may be a good idea. A recently added feature was support for HTTPS. You only need to point your config to server certificate and key files and off you go. The HTTP/HTTPS thing is completely handled by Go.

The resulting program is far from top-quality but it fulfils the needs. Go give it a try if you want to. Careful though: it will only work on Linux boxes for now.

Written by nicolas314

Wednesday 24 July 2013 at 9:18 pm

Posted in fun, go, programming, webapp

Tagged with , , ,

Go recipe: 3DES

leave a comment »

Just posted a really basic example of 3DES encryption with Go. Check it out on github:

See godes.go

Can’t remember where I got the test vectors from, probably an RFC. Did not invent them myself.

Written by nicolas314

Thursday 10 May 2012 at 7:39 pm

Posted in go, programming

Tagged with , , , ,

Go language review

leave a comment »

Go is a relatively new language: designed in 2007, it was just released last month in version 1.0 under a BSD license. Go was invented at Google by Robert Griesemer, Rob Pike and Ken Thompson. Thompson is mostly known for his work on the B and C languages at Bell Labs when Unix was born and this is quite obvious in many of the language design decisions.

In a nutshell: Go is a compiled language meant for systems programming. It is part of the C family and is meant to be a 21st-century C with everything you would expect from a recent programming language. Think of it as what C++ should have been if it had not been taken over by an army of mad lobbyists, or if Java had been designed with efficiency in mind.

Go was written for Unix: Linux, BSD, OSX. A courageous team of volunteers apparently ported it to Windows.

Compared to its C ancestor, Go comes with a large number of features that draw from 40 years of experience with C. Every sore point in C has been addressed and an elegant solution proposed. Let me review the main points:

Fast compiler

Go’s compiler is fast. Small projects compile in terms of milliseconds, not seconds, which in effect gives the feel of an interpreted language. Launching a full compilation for a small-sized project and running the resulting (native) executable takes less time than to start a Python interpreter. Various benchmarks show that generated code is close to C in terms of speed.

Impressive standard library

The standard Go library comes with a large number of utilities for everything you may need at a system level: string processing, web services, crypto primitives, image processing, etc. Far from a CPAN but you already have quite a lot to get started. If all you are looking for is a solid base to start webapp programming, you have all you need.

Enhancements compared to C

  • Go has no pre-processor. Yay!
  • Functions can return multiple values, which enables returning an error status out of band. This simplifies enormously error handling in low-level libraries.
  • Type/variable is inverted: name the variable first then give its type. This feels weird at first but it avoids a number of common mistakes like:
    int* p1, p2 ;

    where p1 is a pointer to int but p2 is just an int.

  • Switch statements do not fall through: after a case you exit the switch. This is probably one of the most common errors encountered in C.
  • Mandatory braces after if, for. Yet another common C error.
  • Native lists and maps
  • Native (immutable) strings and arrays with bound-checking. Forget buffer overflows!
  • Pointers but no pointer arithmetic, i.e. get the full power of pointers without the dirty tricks.
  • Garbage collection. This does not prevent you from doing memory management but at least you get some help at the language level.

Enhancements compared to Java and C++

  • Go has exceptions but reserves them to… exceptional cases. No more flow-control through try/catch! Returning an error status from every function that needs to signal an error to its caller is much easier to handle and does not break programs in random places.
  • Go allows you to define methods on struct types but does not have the notion of classes or inheritance. This still allows the benefits of storing related data and methods into single objects without having to suffer from inheritance pitfalls.
  • Polymorphism: Go has interfaces but does not force the programmer to declare who implements what, this is automatically handled by the compiler (as it should). This promotes duck-typing while preserving strong typing.
  • Package names look like URLs instead of Java’s infamous reverse notation. You can import remote code with something you can understand at first glance:
    import ""

    instead of

    import com.example.project.user.module
  • No operator or function overloading. I have seen projects fail because developers abused these in C++. What you read is what you run!
  • Anonymous functions and closures: quite handy for callbacks and lots of other functional tricks

Concurrency handled through channels

This is a point I did not have enough time to study. Go programs are inherently multi-threaded but they handle concurrency by defining blocking interfaces (channels) between threads, which alleviates the need for mutexes. More on that topic as soon as I have time to play with it.

And more…

Go also includes a documentation tool, a code-formatting tool, integration with the most common Version Control tools (SVN, git, Mercurial), a couple of compiler candidates, it can be debugged using gdb and does not need Makefiles to build packages from source. All things you would expect from a modern programming language.

So what is Go good for?

Everything! It is easy to see a million opportunities for a language that has all the features of a high-level scripting language and the benefits of a strongly-typed compiled language.

The only missing point for now is a crucial lack of cookbook. The standard library comes with minimal documentation that could really use a more comprehensive tour. Or maybe it is just that searching Google for ‘Go’ is obviously bound to return a ton of irrelevant results?

Written by nicolas314

Monday 30 April 2012 at 6:07 pm

Posted in go, programming

Tagged with , , ,

Go recipes on github

leave a comment »

A blog is not an ideal place to post code. Whatever Go snippets I have written or found interesting will be placed on github:

See you there.

Written by nicolas314

Thursday 26 April 2012 at 11:10 pm

Posted in go, programming

Tagged with ,

Dance with the Yaks

leave a comment »

YakLet us do something useful tonight: Go 1.0 was just released, why not give it a try?

Finding something called go on Google is of course a tad difficult. The magic URL you want is Good news everyone: they have packages for OSX! Thinking that running it directly from your Mac Desktop would be more comfortable, you download the pkg file and double-click it to install.




  • Open terminal
  • vim hello.go
  • type in your hello world program

Something is wrong here: no vim syntax highlighting. No way you are going to use a modern editor without syntax highlighting. Ok, this is a new language and vim is probably outdated so update vim.

brew install vim won’t work. Brew tells you that Apple distributes vim with OS X, you can find it in /usr/bin. Ok, that is a very dumb answer. Maybe brew itself is outdated? Try a brew update/upgrade and see if things get better. Nope. Brew complains that your Xcode install is 4.2.1 and you need at least 4.3 to go any further. You already spent a whole evening installing Xcode less than a month ago but now you really want to get the latest and greatest, and you completely trust Apple. What could possibly go wrong?

Sigh. How do you upgrade Xcode? Launching it and browsing through menus does not bring anything. Quick Google search reveals you have to go through the App Store. Let me ponder that cause it bears repeating:


Cool down. Click Install, log in with your Apple ID. Now where in hell did you put that ID? Forgot password? Send e-mail, open e-mail, change password, login again — 10 minutes.

Select Xcode in the apps, click Yeah, accept a zillion-page EULA, sign with your blood, and… nothing. Just a grey button saying Installation. At first you do not dare click the only element of user interface you have on screen by fear of unwillingly requesting multiple installations, but after a few minutes nothing moves so you click… but the button stays put. So you click again, and again, and end up on the Apple forums to see if anybody had the same issue.

Oh yeah: Xcode is installing Ok, you just did not notice the snail-paced progress bar in the tiny icon on your (hidden) dock. Judging from the current speed you still have 2-3 hours before Xcode finishes upgrading.

Ok, forget OSX. The damn thing is built around iTunes anyway, trying to do any development on this thing is bound to deliver endless frustrations, no need to loose more time on that topic.

Start up a distant Linux desktop, connect through VNC. Find out that VNC server and your local client have gone out of sync. Update/Upgrade your Linux box (30 packages) and find out that some GPG package keys are obsolete. Spend another couple of minutes trying to fix this, then finally update/upgrade your box. Log in through ssh/VLC.

Start up a web browser to download Go again. Find out that the default web browser likes to start an archive manager once a tgz has been downloaded. That is cool except for one thing: you never used this package manager before so you end up garbling unknown directories and downloading the same file over and over again.

Don’t loose your nerves, you are almost there. Quite.

Time to su, install go to /usr/local. Finally! A quick modification of your PATH and you are ready to explore a new language. Again:

  • vim hello.go

… but still no syntax hilighting. This time hunt for vim syntax support, and find out that go distributes vim config files with the main tarfile. Should have started with that but the later it gets the less you can think clearly. Copy those files to your .vim/ directory and lo and behold: you have your very first syntax-hilighted go hello world in front of you. Cool!

2 hours so far. And Xcode is still merrily downloading away.

Time to run your first go program:

% go run hello.go
Hello, world

Impressive. Now try to compile it:

% go build hello.go
% ./hello
Hello, world

Beautiful! Just curious: how big would this be?

% ls -l hello
-rwxr-xr-x 1271748 Mar 29 22:52 hello*

1.3 megs.

Yeah, you know what? I think I’ll read a book tonight. Old-fashioned dead tree does not require much more than sit and read.

Written by nicolas314

Thursday 29 March 2012 at 11:13 pm

Posted in fun, google, osx, yak shaving

Tagged with , , , ,