Nicolas314

All my geeky stuff ends up here. Mostly Unix-related

Archive for the ‘go’ Category

Wunder Weather

leave a comment »

wunderJust released this small piece of code a few days back:

https://github.com/nicolas31/wunder

I wanted to be able to bring up the weather forecast for the place I am currently visiting without having to yield my address book to a shady app, or suffer from tons of annoying ads eating through my data plan and phone storage.

The Yahoo weather app is fantastic but has too many ads. Weather web sites are incredibly data heavy, making it nearly impossible to get right to the information I am looking for: is it going to rain today or tomorrow? Expected temperatures?  Android has some ad-less widgets but they usually request GPS positioning and I’d rather not activate location services when I don’t need them.

So I hacked something. Made a web app that identifies your position by geolocating the requester’s IP address, obtains the weather forecast from a reliable source, and displays the only weather information I need on a fast loading page.

First issue: geolocalize an IP address.

There are many free services on the net to achieve that. Alternatively, you can download a static list and refresh it at regular intervals, but I wanted to get something a bit more dynamic. I chose:

http://ip-api.com

Their API is dead simple and just works. Provide an IP address, get a country code, city name, latitude and longitude. You do not need to subscribe to their services, just make sure you are not choking them with too many requests.

Second issue: find a reliable weather source.

I fist tried openweathermap.org. This is a very cool site but has a few shortcomings:

You can get the weather for a given [city, country] or [lat, lon]. The list of supported [city, country] pairs is static and can be downloaded from their web site. While they do support a lot of cities in the world, the problem was figuring out how to match [city, country] between what is returned by ip-api.com and what is understood by openweathermap.org. The matching is not 100% accurate.

Getting the weather by coordinates would work but it is far from user-friendly.  You end up with Weather forecast for location Lat=XX Lon=YY. I’d rather look up the weather for San Francisco than for a pair of coordinates that are not obviously recognizable.

I ended up looking up [city, country] by computing the smallest distance on the openweathermap list, but that is just tedious and a lot of work for very little gain.

Other major issue: the weather forecast is only provided GMT, which is utterly useless. What I want is local time, always. What do I care if I am told that it will rain from 2 to 5am GMT if I cannot relate that to local time?

Figuring out a conversion between GMT and local time is a lot trickier than it looks. Thanks to Daylight Saving Time rules that are changed at random intervals in various countries, it is very hard to predict the time offset in some places more than a couple of weeks ahead. Relevant:

A bit of googling around revealed there is an actual API from Google Maps to convert a Unix time stamp + latitude and longitude to a local time. This API takes into account local DST rules at the considered date/time, which is exactly what we want. No need to register with Google, as usual the API is free to use and rate-limited.

Example code can be found here: https://github.com/nicolas314/tz

In summary: getting the weather from openweathermap would require:

  • One external API call to associate IP to [lat, lon]

  • A search to associate [lat, lon] to [city, country]

  • One external API call to obtain actual weather data

  • One external API call to convert GMT to local time

I have implemented that and the result is ugly. Ok let’s see if we can find something smarter.

Next try: wunderground.com

They also offer an API to obtain weather data for any place in the world and they take care of two things: converting [lat, lon] to [city, country], and converting weather forecast to local time. This is exactly what we want.

Their API can also take care of geolocating an IP address but I found their results to be a lot less reliable than what I get from ip-api.com, so will stick to that for geolocation.

Their terms and conditions are fair. You need to register with them to obtain an API key and that’s about it. Results are delivered in metric units and can be localized in several languages. You also get a pointer to icons symbolizing the weather, which is perfect to generate a nice web page effortlessly.

Some comments about my implementation:

Results from wunderground contain a whole bunch of information I am not interested in, like temperatures in Farenheit. Not an issue: the Go JSON API allows defining fewer fields than what is parsed, so you can keep your structs small with only relevant data.

When running behind a reverse proxy, the incoming requesting IP address you see is the one for the proxy. In order to get the real incoming IP address you need to configure the reverse proxy to pass it along, usually in an HTTP header. Since I am running this service behind nginx, I get the address from X-Real-IP. That is probably different for each reverse proxy out there.

Hardcoded handlers are provided to take care of requests for /favicon.ico and /robots.txt. I was tired of seeing 404 requests in my logs for these two.

Results are cached by IP address for one hour to avoid flooding upstream API services with requests. Results are displayed from a template that can easily be tweaked. The one I wrote fits nicely enough on both mobile and desktops, your mileage may vary.

I installed the end result on a tiny VPS instance, for my own use. Hoping that could be useful to somebody else.

Advertisements

Written by nicolas314

Tuesday 16 August 2016 at 1:46 pm

Posted in go, programming

Tagged with , ,

One-time file-sharing

leave a comment »

oneSay you rent a box somewhere on the Internet. You installed Debian stable on it because you want it to be nice and stable and run a few daemons that are useful to have online. Could be to hold your vast music collection, family pictures, or use it as remote storage for backup. Imagine you wanted to share some of the files hosted on this box with your relatives, who may or may not be computer-literate. Most of them would know how to use a webmail but asking them to install an ftp client is just beyond reach.  Obviously, you do not want to give these guys too many rights over your box (like an ssh access for scp). What are the solutions?

Setting up a dedicated HTTP server

Simple enough: set up an HTTP server to distribute static files. lighttpd is simple enough to setup in a couple of minutes and is very efficient for static stuff. But you do not want to distribute your files to the whole Internet. Sooner or later a web spider will crawl in and index your family pictures and all sorts of things you never meant to be public.  Next step: configure password-protection on the server

Fair enough. Now you have limited file downloads to people who know the password — provided they know how to enter a password. Do you create multiple accounts, one for each of your peers? It would be preferrable, otherwise you will never know who downloaded what. But then you have to communicate their passwords to your peers and make sure they have a procedure in case they forget it. You know you are headed straight to massive butt pains.

Second issue: passwords can be shared. You shared that 2GB movie with a couple of friends and a couple of weeks later you find out that there are currently 1,549 active downloads for this file. Sharing is in human nature and that is completely Ok, but you probably did not sign up to become a content distributor over the whole Internet, only with a couple of friends and relatives.

Next step: use one-time authentication

There are better solutions out there: since you only mean to share one single file (or set of files) each time, you do not need to create accounts for your friends. You give them a one-time download token and forget about it.

A one-time download token is a URL. It looks like the kind of URLs you get from URL shorteners with the funny string at the end. Something like http://shortener/12398741

One-time tokens can be shared but since they can only be used once, the person who shared it has lost it. The token is randomly generated and invalidated immediately after it is used to avoid having robots automatically scan all possible URLs in a row until they find a valid one.

There are many ways to achieve this on regular HTTP servers. Apache probably has a million configuration options for user authentication, including one-time passwords or something similar, but I have to admit I did not even try. I already wasted enough of my life in Apache config files. lighttpd can be configured to do that but the only solution I found required some Lua scripting and I did not feel up to the task.

Next-step: Do It Yourself

After reviewing countless pages of configuration options for various HTTP servers, I decided that it would be shorter for me to implement this in a tiny web app rather than try and understand complex configuration options.  My first iteration made use of a Python FCGI script in web.py attached to a lighttpd process. Pointing out static files from a Python web app to the embedding lighttpd process is reasonably simple.

This implementation suffered from a number of pitfalls though. For one thing, performance was bad. For some reason, the Python process would eat insane amounts of CPU and RAM when sending big files, slowing down the server to a crawl. Second showstopper was the complexity involved for such a simple setup. I had to write a Python script to generate the lighttpd configuration file with a number of deployment options: where to put config files, log files, static files, port number, etc. And then came the inevitable issues with dependencies: Python version versus web.py version versus lighttpd version.  Some combinations worked fine, some did not.  Nothing specific to Python or lighttpd, but the more you have gears, the more you have places for grains of sand to fit in.

I still survived with this setup for a year or so, when Go came in. I have already reviewed the language in the past and will not come back to that, but suffice it to say that developing HTTP servers in Go is the most natural thing. Adding the one-time token ingredient to the soup was implemented in just one evening.

Once rewritten in Go, I found out that the end-result was about just as big as the Python implementation, excluding the script that created the lighttpd config. The main difference was of course that I do not have to maintain cross-references between package versions for Python, lighttpd, and web.py, since there is only one dependency to cover: Go itself.

It was straightforward to enhance the program to support more options, respond to favicon, and handle a JSON-readable database for active tokens.  Performance is astounding. The serving Go process never takes more than a few megs of RAM (about the size of the executable itself) and only uses tiny amounts of CPU since the process is mostly I/O based anyway.

There is one thing I should have foreseen and had to re-implement. I am sending the one-time links by email and more and more people are reading their emails from their smartphone or tablet. Many just clicked the link without thinking twice, triggering a 2-4GB download and killing both their mobile and data plan at the same time. Wrong move.

The next version features a two-time download page: the first link sends users to a page summarizing the download, offering a second link to actually start the real thing with a big warning about the size of what will actually be sent.

There are many other features I would like to add to the current version, and I am hoping other people have better ideas for new features, which is the reason why I shared it on github. Find it here:

https://github.com/nicolas314/onetime

Since we are talking about sharing private date between friends and relatives, protecting the download may be a good idea. A recently added feature was support for HTTPS. You only need to point your config to server certificate and key files and off you go. The HTTP/HTTPS thing is completely handled by Go.

The resulting program is far from top-quality but it fulfils the needs. Go give it a try if you want to. Careful though: it will only work on Linux boxes for now.

Written by nicolas314

Wednesday 24 July 2013 at 9:18 pm

Posted in fun, go, programming, webapp

Tagged with , , ,

Go recipe: 3DES

leave a comment »

Just posted a really basic example of 3DES encryption with Go. Check it out on github: https://github.com/nicolas314/go-recipes

See godes.go

Can’t remember where I got the test vectors from, probably an RFC. Did not invent them myself.

Written by nicolas314

Thursday 10 May 2012 at 7:39 pm

Posted in go, programming

Tagged with , , , ,

Go language review

leave a comment »


Go is a relatively new language: designed in 2007, it was just released last month in version 1.0 under a BSD license. Go was invented at Google by Robert Griesemer, Rob Pike and Ken Thompson. Thompson is mostly known for his work on the B and C languages at Bell Labs when Unix was born and this is quite obvious in many of the language design decisions.

In a nutshell: Go is a compiled language meant for systems programming. It is part of the C family and is meant to be a 21st-century C with everything you would expect from a recent programming language. Think of it as what C++ should have been if it had not been taken over by an army of mad lobbyists, or if Java had been designed with efficiency in mind.

Go was written for Unix: Linux, BSD, OSX. A courageous team of volunteers apparently ported it to Windows.

Compared to its C ancestor, Go comes with a large number of features that draw from 40 years of experience with C. Every sore point in C has been addressed and an elegant solution proposed. Let me review the main points:

Fast compiler

Go’s compiler is fast. Small projects compile in terms of milliseconds, not seconds, which in effect gives the feel of an interpreted language. Launching a full compilation for a small-sized project and running the resulting (native) executable takes less time than to start a Python interpreter. Various benchmarks show that generated code is close to C in terms of speed.

Impressive standard library

The standard Go library comes with a large number of utilities for everything you may need at a system level: string processing, web services, crypto primitives, image processing, etc. Far from a CPAN but you already have quite a lot to get started. If all you are looking for is a solid base to start webapp programming, you have all you need.

Enhancements compared to C

  • Go has no pre-processor. Yay!
  • Functions can return multiple values, which enables returning an error status out of band. This simplifies enormously error handling in low-level libraries.
  • Type/variable is inverted: name the variable first then give its type. This feels weird at first but it avoids a number of common mistakes like:
    int* p1, p2 ;

    where p1 is a pointer to int but p2 is just an int.

  • Switch statements do not fall through: after a case you exit the switch. This is probably one of the most common errors encountered in C.
  • Mandatory braces after if, for. Yet another common C error.
  • Native lists and maps
  • Native (immutable) strings and arrays with bound-checking. Forget buffer overflows!
  • Pointers but no pointer arithmetic, i.e. get the full power of pointers without the dirty tricks.
  • Garbage collection. This does not prevent you from doing memory management but at least you get some help at the language level.

Enhancements compared to Java and C++

  • Go has exceptions but reserves them to… exceptional cases. No more flow-control through try/catch! Returning an error status from every function that needs to signal an error to its caller is much easier to handle and does not break programs in random places.
  • Go allows you to define methods on struct types but does not have the notion of classes or inheritance. This still allows the benefits of storing related data and methods into single objects without having to suffer from inheritance pitfalls.
  • Polymorphism: Go has interfaces but does not force the programmer to declare who implements what, this is automatically handled by the compiler (as it should). This promotes duck-typing while preserving strong typing.
  • Package names look like URLs instead of Java’s infamous reverse notation. You can import remote code with something you can understand at first glance:
    import "git.example.com/user/project"

    instead of

    import com.example.project.user.module
  • No operator or function overloading. I have seen projects fail because developers abused these in C++. What you read is what you run!
  • Anonymous functions and closures: quite handy for callbacks and lots of other functional tricks

Concurrency handled through channels

This is a point I did not have enough time to study. Go programs are inherently multi-threaded but they handle concurrency by defining blocking interfaces (channels) between threads, which alleviates the need for mutexes. More on that topic as soon as I have time to play with it.

And more…

Go also includes a documentation tool, a code-formatting tool, integration with the most common Version Control tools (SVN, git, Mercurial), a couple of compiler candidates, it can be debugged using gdb and does not need Makefiles to build packages from source. All things you would expect from a modern programming language.

So what is Go good for?

Everything! It is easy to see a million opportunities for a language that has all the features of a high-level scripting language and the benefits of a strongly-typed compiled language.

The only missing point for now is a crucial lack of cookbook. The standard library comes with minimal documentation that could really use a more comprehensive tour. Or maybe it is just that searching Google for ‘Go’ is obviously bound to return a ton of irrelevant results?

Written by nicolas314

Monday 30 April 2012 at 6:07 pm

Posted in go, programming

Tagged with , , ,

Go recipes on github

leave a comment »

A blog is not an ideal place to post code. Whatever Go snippets I have written or found interesting will be placed on github:

https://github.com/nicolas314/go-recipes

See you there.

Written by nicolas314

Thursday 26 April 2012 at 11:10 pm

Posted in go, programming

Tagged with ,

Go recipe: HTTPS server

leave a comment »

Go recipe: implement an HTTPS server that requires a client-side certificate for authentication but does not check certificate origin. Any client-side cert will be accepted, the Subject Common Name is printed upon visiting the page. Start the program and point your browser to https://localhost:4443

Generating a server key and cert is left as an exercise for the reader :-)

package main

import (
    "fmt"
    "net/http"
    "crypto/tls"
)

func Hello(w http.ResponseWriter, req * http.Request) {
    w.Header().Set("Content-type", "text/plain")
    fmt.Fprintf(w, "Hello\n")
    client_cert := req.TLS.PeerCertificates[0]
    fmt.Fprintf(w, "You are: %s\n", client_cert.Subject.CommonName)
}

func main() {
    http.HandleFunc("/", Hello)
    t := tls.Config {
            ClientAuth: tls.RequireAnyClientCert,
         }
    s := &http.Server {
            Addr:       ":4443",
            TLSConfig:  &t,
         }
    fmt.Println("Listening on 4443...")
    err := s.ListenAndServeTLS("server.crt", "server.key")
    if err!=nil {
        fmt.Printf("err: %s", err)
    }
}

Written by nicolas314

Tuesday 24 April 2012 at 11:02 pm

Posted in go, programming

Go recipe: hashes

leave a comment »

Playing with Go: here is a simple implementation that computes crypto hashes of a list of files provided on the command-line. Usage:

ghash md5|sha1|sha2|sha5 filenames...
package main

import (
    "bufio"
    "crypto/md5"
    "crypto/sha1"
    "crypto/sha256"
    "crypto/sha512"
    "errors"
    "fmt"
    "hash"
    "os"
)

func Hash(hashname string, filename string) (sum string, err error) {
    f, err := os.Open(filename)
    if err!=nil {
        return
    }

    var h hash.Hash

    switch hashname {
        case "md5":
        h = md5.New()
        case "sha1":
        h = sha1.New()
        case "sha2", "sha256":
        h = sha256.New()
        case "sha5", "sha512":
        h = sha512.New()
        default:
        err = errors.New("unknown hash: "+hashname)
        return
    }

    var nr int
    buf := make([]byte, h.BlockSize())
    bf  := bufio.NewReader(f)
    for {
        nr, _ = bf.Read(buf)
        h.Write(buf[0:nr])
        if nr<len(buf) {
            break
        }
    }
    f.Close()
    sum = fmt.Sprintf("%0x", h.Sum(nil))
    return
}

func main() {
    if len(os.Args)<3 {
        fmt.Println("use: md5|sha1|sha2|sha5 filenames...")
        return
    }
    for i:=2 ; i<len(os.Args) ; i++ {
        digest, err := Hash(os.Args[1], os.Args[i])
        if err!=nil {
            fmt.Println(err)
        } else {
            fmt.Printf("%s  %s\n", digest, os.Args[i])
        }
    }
}

Written by nicolas314

Tuesday 24 April 2012 at 10:56 pm

Posted in go, programming