Nicolas314

All my geeky stuff ends up here. Mostly Unix-related

Starbugs

leave a comment »

The night was clear, we would not be playing Bomberman in the VLT control room that night. Clear skies and a sub-arcsecond seeing meant we would have a full batch of data to process every hour or so until the next morning. Once the calibrations had finished, the telescope operator launched the first observation. I re-compiled the whole processing software once more, just to be sure we had not forgotten anything, ran a series of unit tests for good measure, and waited in front of my screen for the first incoming set of frames to appear on the local disk.

First batch of sixty frames was completed after exactly sixty minutes. As the machine started doing its number-crunching, everybody in the room turned to me, waiting for the first processed image to come out. It took a good fifteen minutes for all algorithms to run through the set: calibrate all frames, remove the infrared sky, take into account bad and crazy pixels that have been hit with cosmic rays during the observation, register all frames to a common position, and finally stack them to a single image. The final result appeared on the screen above me and I could see smiles all around. It seemed the results were up to what my customers were expecting.

Now we had a clear image of a set of bright object against a dark background. In order to assess how much infrared light is emitted by each object, it needs to be calibrated. Somewhere on the image is a standard: a star with precisely known photometry in the wavelength we had been observing. Compute how many photons were received in this image from this star and you can deduce the magnitudes for all other objects present on the same frame.

I checked once more the final frame position on the sky and then launched the photometry calibration routine. The standard star was found and identified by name, its photometry computed by integrating all received light in a small surrounding radius, and then all objects in the frame were suddenly known by magnitude rather than number of photons. Perfect score! With a sigh of relief, I finally pushed myself away from the desk and reached for some water. The memory routines had done their job, we did not crash in flight by lack of RAM this time. Eleven more hours to go and then we could all go to sleep.

Next incoming data batch was processed just fine. Another image emerged. And then another one. It seemed everything was working perfectly fine.

Around midnight, something weird happened: the result image was correctly processed but photometry calibration failed because it found no standard star in the frame.

– What? Emilio, did you include a standard in the last observation?
– Let me check… Yes I did. You should have it somewhere around the top-right corner.

The standard star was indeed there, so why did the photometry calibration routine fail to find it?
I immediately opened the database we had for infrared standards and started searching frantically for the star, finding it immediately. I reached for the debugger and re-ran the whole routine once more with breakpoints. Confirmed: the search for standard stars in this region returned nothing, and yet the database was correctly loaded and completely in memory. The debugger showed what seemed like correct values for star positions, but the search function failed for some reason.

The star database we had was pretty simple: a simple text file containing named columns: first the star name, then its position on the sky as Right Ascension and Declination (a couple of angles), and then its magnitude at various wavelengths. Something like:

# Name | Ra         |  Dec      | Sp |  J     |    H   |   K    
AS01-0 | 00 55 09.9 |  00 43 13 | -- | 10.716 | 10.507 | 10.470 
AS03-0 | 01 04 21.6 |  04 13 39 | -- | 12.606 | 12.729 | 12.827 
AS04-1 | 01 54 43.4 |  00 43 59 | -- | 12.371 | 12.033 | 11.962 
AS05-0 | 02 30 16.4 |  05 15 52 | -- | 13.232 | 13.314 | 13.381 
AS05-1 | 02 30 18.6 |  05 16 42 | -- | 14.350 | 13.663 | 13.507 
AS07-0 | 02 57 21.2 |  00 18 39 | -- | 11.105 | 10.977 | 10.946 
AS10-0 | 04 52 58.9 | -00 14 41 | -- | 11.349 | 11.281 | 11.259 
AS13-1 | 05 57 10.4 |  00 01 38 | -- | 12.201 | 11.781 | 11.648 
AS13-1 | 05 57 09.5 |  00 01 50 | -- | 12.521 | 12.101 | 11.970 
AS13-3 | 05 57 08.0 |  00 00 07 | -- | 13.345 | 12.964 | 12.812 
AS15-0 | 06 40 34.3 |  09 19 13 | -- | 10.874 | 10.669 | 12.628 
AS15-1 | 06 40 36.2 |  09 18 60 | -- | 12.656 | 11.980 | 11.792 
AS15-2 | 06 40 37.9 |  09 18 41 | -- | 13.711 | 12.927 | 12.719 
AS15-3 | 06 40 37.9 |  09 18 19 | -- | 14.320 | 13.667 | 13.415 
AS16-0 | 07 24 15.3 | -00 32 50 | -- | 14.159 | 14.111 | 13.305 
AS16-1 | 07 24 14.3 | -00 33 05 | -- | 13.761 | 13.638 | 13.606 
AS16-2 | 07 24 15.4 | -00 32 49 | -- | 11.411 | 11.428 | 11.445 
AS16-3 | 07 24 17.2 | -00 32 27 | -- | 13.891 | 13.855 | 13.818 
AS16-4 | 07 24 17.5 | -00 33 07 | -- | 11.402 | 11.106 | 11.043

J, H, K are infrared bands corresponding to a relatively narrow wavelength range.

Something went wrong in the star-loading routine, so I loaded the whole set into memory once more and dumped it back to a text file to plot it. The results were not particularly obvious:

catalog

Somebody in the room came up to the screen and asked what we were looking at. I said: “these are the positions of all known infrared standards we have. For some reasons we cannot find tonight’s star in here.”

Looking at it again, I found our star. It was not in the right position. It should have been below the x axis but had shifted symmetrically above it. Looking at the data set again, the Declination was indeed negative: something like -00 14 41, but it was plotted on the wrong side of the x axis.

And then it dawned on me: the star was plotted at +00 14 41 instead of -00 14 41.

How do you read numeric data in C? Using scanf(). When you scanf() for “-00”, what do you think ends up in memory? Zero. Positive zero, since it is technically the same as negative zero. Except the angle has now been flipped around the x axis.

Right: plotting a denser set of stars revealed a clear white patch for Declinations between zero and minus one. I had just forgotten to take into account the first character as a sign since scanf() does not make any difference between “00” and “-00”. Once I corrected the database-loading line, everything fell into place and photometry computations could take place as expected.

Interestingly enough, it seems the same bug hit a large number of GPS devices over the past years. The German C’t magazine told the story a few years back about somebody who planned a bike tour around Bordeaux and ended up with intermediate points in the middle of the ocean. Bordeaux is located around longitude zero (Greenwhich), so you do have data points located at an angle that starts with -00. In effect, you could see all points correctly plotted on the map except for the ones located between zero and minus one degree, which flipped over the other side of the meridian. As soon as I saw the map I knew exactly what had happened.

At least the guy was clever enough not to bike into high waters. It could have been worse: though probably related to time manipulation errors rather than angles, you may want to read how F22 Raptors spontaneously rebooted upon crossing the international date line:

F22 Raptor gets zapped by international date line

There are some assumptions you should not make about handling time in software. Some of them are presented in this blog article:

Falsehoods programmers believe about time

Time and angles can be tricky scalars.

Advertisements

Written by nicolas314

Wednesday 26 June 2013 at 12:39 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: