int foobar[100];

file2.c:

extern int *foobar;

stating that it won’t work as expected. Back then I referred people to Peter van then Linden’s book “Expert C Programming” for a detailed exposition. However, since a few people have since asked for an explanation I’ll attempt to give one below. Please refer to van then Linden’s book for a much more detailed and more well written exposition of the whole subject.

First off, it is very important to realize that arrays and pointers in C are not the same. Unfortunately there are many places in the C universe where this equivalence holds which has led many people to assume it is a general truth – it is not! So, once more, arrays and pointers in C are not the same! In other words, the two declarations

extern int *x;
extern int y[];

mean something different to the compiler.

That said, let’s first think about how elements of

int foo[] = "01234"

are accessed. In order to get at foo[2], say, we first retrieve the address of foo from the symbol table. Importantly, stored at this address is an int, namely, the first element of foo, foo[0]. Therefore, to get at foo[3] we simply start at foo and then grab the third int starting from that address. Hence, retrieving foo[3] involves two steps, get the address of foo itself, then march forward in memory to grab the 3rd int.

Now that that’s out of the way let’s consider

int *bar = "01234"

and again think about what is happening under the hood when retrieving bar[3]. Here, we have told the compiler that bar is a pointer to an int. So when the compiler retrieves the address of bar from the symbol table, bar does not store an int like in the case of foo above. Rather, bar holds a pointer to int which in turn tells us where in memory this int actually is, quite literally an “int*”, really. This is the important distinction! Hence, accessing bar[3] involves one additional redirection step compared to a direct array access like in the case of foo[3]. First, the compiler retrieves the pointer to int stored at bar. Then we follow this pointer to get at the actual int, bar[0]. Finally, we march ahead three ints worth of memory to get at bar[3].

With this in mind let’s again look at the example given at the beginning:
file1.c:

int foobar[100];

file2.c:

extern int *foobar;

In file1.c we declare the symbol foobar to be an array of ints. Since foobar is an array, its address stored in the symbol table will directly point to the first int in the array, foobar[0] in this case.

Unfortunately, in file2.c we tell the compiler that foobar is a pointer to an int. So what will happen if we try to access foobar[2] somewhere inside file2.c? According to the above the following:

1. Get the address of foobar from the symbol table
2. Retrieve the pointer to int stored at foobar

Ups! That’s bad, foobar does not store a pointer to int, it stores ints directly. Hence, the additional redirection step caused by declaring foobar as “int*” causes the compiler to falsely interpret the first bunch of ints stored at foobar as a pointer to int. Obviously, when following this bogus pointer nothing really good is going to happen and we may end up with a potentially very hard to track down bug.

That said, the correct way to declare foobar in file2.c is of course
file2.c

extern int foobar[];

Here, we’re telling the compiler that when retrieving the address of foobar from the symbol table it must directly interpret it as a memory location where ints a stored and all is well.

Hence, despite the fact that occasions abound where C will treat arrays as pointers (e.g., array names in expressions are treated as pointers), fundamentally they are different and it is important to keep this in mind.

So far so good but why not implement these sorting routines in Haskell as well, just for fun. Quicksort in Haskell is straightforward and strikingly beautiful. Here it is

> quicksort :: [Double] -> [Double]
> quicksort [] = []
> quicksort (x:xs) = (quicksort (filter (<= x) xs)) ++ [x]
>                              ++ (quicksort (filter (> x) xs))

Interestingly, even though this implementation uses filter twice, it is a tad faster on my system (~ 3%) than the version that uses a single call to partition to partition the list.

Of course, this version is no match for the C implementation in terms of speed due to the appends and the fact that each recursive call to quicksort uses its own private copy of the sub-array. Nevertheless, it is quite speedy and code-wise it can’t get much more concise.

My Haskell implementation of insertion sort is also not too much longer

> insertion_sort :: [Double] -> [Double]
> insertion_sort []       = []
> insertion_sort (x:xs) = my_insert x (insertion_sort xs)
>
>   where
>     my_insert a []               = [a]
>     my_insert a tot@(b:bs) = case compare a b of
>                                                  GT -> b:(my_insert a bs)
>                                                     _ -> a:tot

Yes, I know, I could have used prelude’s insert function instead, but
for some reason, using my_insert is about 30% faster for a reason I don’t understand since the functions are almost identical. The only reason I can think of is the additional redirection via insertBy used by insert. Of course, due to its O(n²) behaviour, insertion sort is quite a bit slower than quicksort.

Finally, there is my version of bubble sort and, frankly, it took me some time to find an implementation that ran in a somewhat decent amount of time without blowing my machine’s stack. My code threads a status variable through the bubble function which signals the absence of swaps indicating a sorted list. Here it is

> bubble_sort :: [Double] -> [Double]
> bubble_sort xs   = let
>                                 (state,bubbled) = bubble True xs
>                               in
>                                 if not state
>                                   then bubble_sort bubbled
>                                   else bubbled
>
>   where
>     bubble s []           = (s,[])
>     bubble s [a]         = (s,[a])
>     bubble s (a:b:as) = case compare a b of
>                                      GT -> let
>                                                  (s_n,new) = bubble False (a:as)
>                                                in
>                                                  (s_n,b:new)
>
>                                       _ -> let
>                                                 (s_n,new) = bubble (s && True) (b:as)
>                                               in
>                                                 (s_n,a:new)

This version of bubble sort is about 6 times slower than insertion sort. So not too bad but I have the feeling it can still be improved quite a bit. E.g., currently bubble walks through the whole list during each iteration even the already sorted part which is a waste.

So what is TABA? In a nutshell, the TABA pattern utilizes a recursive function to traverse a data structure such as a list at return time. This allows one to, e.g.,  convolve two list [x₁, x₂, ..., x_n] and [y₁, y₂, ..., y_n] to yield

[(x₁,y_n), (x₂,y_n-1), ..., (x_n,y₁)]

in only n recursive calls.

So how does this work? The main idea is that during the recursive traversal of the first list we build up a stack of functions each of which is responsible for generating one of the final list elements. During stack unwinding these function are being replayed and generate the final result. The authors of the TABA paper liken it to winding a spring during the recursive traversal. If this sounds a bit too abstract here’s a Haskell implementation of a list convolution function using the TABA pattern

>convolve :: [a] -> [b] -> [(a,b)]
>convolve xs ys = fst . walk $ xs
>  where
>    walk []        = ([],ys)
>    walk (a:as) = let
>                            (r, (b:bs)) = walk as
>                          in
>                            ((a,b):r, bs)

Isn’t that great!? Even better, the above can nicely be written as a fold like so

>convolve1 :: [a] -> [b] -> [(a,b)]
>convolve1 xs ys = fst . foldr (\x (r,(a:as)) -> ((x,a):r,as)) ([],ys) $ xs

If this has whet your appetite please consult the TABA paper for a much better and more in depth exposition. Enjoy!

In the first installation I will only provide a high level introduction to awk; next time we’ll dive in a bit deeper. Before we get started it is probably prudent to point out that there are several implementations of awk floating around these days. If you use Linux, chances are that you will have GNU awk, or gawk for short, available. I will use gawk in my examples.

After these preliminaries, let’s talk about what awk actually does. As it turns out, awk is not simply a tool but an actual programming language designed to efficiently deal with the content of (plain text) files in a line by line fashion. Whereas other languages (python, ruby, C, ..) require one to explicitly open files and read their content line by line, awk provides this infrastructure for free and furthermore splits each line into fields according to a given separator — space by default. Hence, whenever you find yourself reading files and then doing something to each line, awk may help make your life considerably easier. Let’s look at an example data file

# cat my_data.txt
01/01/2009 234.23 100
01/02/2009 221.11  50
01/03/2009 123.11  34
...

Here, each line contains three columns separated by a space. The first one is a date, the second one the dollar amount for a certain sales item, and the third the number of items sold. How would we go about converting this file into one that lists the total sales for each date in the first column and the corresponding date in the second one?

Sure, you could write a little python script but lets use awk instead. I mentioned above that awk splits each line into fields. These can be accessed via $0, $1, $2, …, where $0 holds the complete line, $1 the first field, $2 the second one, and so on. Hence,

# gawk '{ print $0 }' my_data.txt

will print each line of my_data.txt to stdout. awk automagically walks through all lines of the input file and applies the commands inside the curly braces to each line. That said, it’s now dead simple to write down the code to transform our sales example. Here it is

# gawk '{print $2*$3,$1}' my_data.txt

For each line we simply print token $2 (per item price) multiplied by token $3 (number of items) followed by $1 which is the date. Our output therefore consists of two columns; the comma makes sure that they are separated by a space.

As you can see, if you need to process column centric data files awk really can make your life a lot easier. If this has wet your appetite for awk you might be interested in the next installation on this blog. If you’re too impatient google is your friend or have a look at the excellent book by Dougherty and Robbins [1].

[1] sed & awk, 2nd Ed., Dale Dougherty, Arnold Robbins

Van der Linden is a former developer for Sun Microsystems and has combined his real world experience with the C programming language into Expert C Programming. The book is packed with helpful insights of the kind missing from most other C books I’ve read – any serious C programmer should definitely have it on her bookshelf. Despite being published in 1994 most of the book’s advice still applies to this very day [2]. Be aware that Expert C Programming is not a C tutorial and the reader should already have a good grasp of the language to get the most out of it. So, if you’ve ever wondered why

file1.c:

int foobar[100];

file2.c:

extern int *foobar;

/* piece of code that does something with foobar */

won’t work go get yourself a copy of Expert C Programming.

References

[1] http://www.afu.com
[2] Interestingly, C99 does address some of the C shortcomings discussed in the book.

Finally, it dawned on me what had happened. When octave executes commands either in the interactive shell or by running a script, it will look for the corresponding .m files in a number of directories (customizable by the addpath command) including the current directory. Importantly, it will (at least on my systems) try ‘.’ first. Hence, if your current directory contains a script foo.m it will overload octave’s foo command and instead execute whatever is in foo.m, which probably isn’t what you want at all. To make things more subtle, in our case the problem wasn’t even a std.m file that overloaded the real std, but rather a mean.m file that would overload the real mean which was called from somewhere inside std. After renaming my co worker’s version of mean.m (which, by the way did not really calculate the mean) everything was fine again and the original script worked as expected.

The moral of the story is that you should never choose names for your .m files that match actual octave commands unless you know what you are doing – or like surprises. Doing so will silently overload the real commands with your versions. Make it a habbit to always prefix your commands in an obvious manner, e.g. my_mean.m instead of mean.m or something similar.

-- imports
import Text.ParserCombinators.Parsec
import qualified Text.ParserCombinators.Parsec.Token as PT
import Text.ParserCombinators.Parsec.Language (emptyDef)               

-- | main
main :: IO ()
main = do                                                              

  -- get a line from stdin
  input <- getLine                                                     

  -- parse it
  case parse parse_calc "" input of
    Left er  -> putStrLn $ "Error: " ++ (show er)
    Right cl -> putStrLn (show cl)                                     

  main                                                                 

-- | function generating a token parser based on a
-- lexical parsers combined with a language record definition
lexer :: PT.TokenParser st
lexer  = PT.makeTokenParser emptyDef                                   

-- | token parser for parenthesis
parens :: CharParser st a -> CharParser st a
parens = PT.parens lexer                                               

-- | token parser for Integer
integer :: CharParser st Integer
integer = PT.integer lexer                                             

-- | token parser for Double
float :: CharParser st Double
float = PT.float lexer

-- | token parser for Either Integer Double
naturalOrFloat :: CharParser st (Either Integer Double)
naturalOrFloat = PT.naturalOrFloat lexer                               

-- | token parser for keywords
reservedOp :: String -> CharParser st ()
reservedOp = PT.reservedOp lexer                                       

-- | helper function for defining real powers
real_exp :: Double -> Double -> Double
real_exp a x = exp $ x * log a                                         

-- | grammar description for parser
parse_calc :: CharParser () Double
parse_calc = mul_term `chainl1` add_action                             

mul_term :: CharParser () Double
mul_term = exp_term `chainl1` multiply_action                          

exp_term :: CharParser () Double
exp_term = factor `chainl1` exp_action                                 

factor :: CharParser () Double
factor = parens parse_calc
             <|> parse_sqr
             <|> parse_number                                              

parse_sqr :: CharParser () Double
parse_sqr = reservedOp "sqrt" >> parens parse_calc >>=
                    \x -> return $ sqrt x                                      

multiply_action :: CharParser () (Double -> Double -> Double)
multiply_action = do {reservedOp "*"; return (*) }
                            <|> do {reservedOp "/"; return (/) }                   

add_action :: CharParser () (Double -> Double -> Double)
add_action = do { reservedOp "+"; return (+) }
                      <|> do { reservedOp "-"; return (-) }                       

exp_action :: CharParser () (Double -> Double -> Double)
exp_action = reservedOp "^" >> return real_exp

parse_number :: CharParser () Double
parse_number = naturalOrFloat >>=
                            \num -> case num of
                           Left i  -> return $ fromInteger i
                           Right x -> return x

The next step will be to allow for the definition of local variables that can be used during calculations. I’ll keep you posted on any progress

UPDATE (02/08/09): I’ve added quite a few more features to the little calculator app and named it husky. You can have a look at the current state of the code over at github (see sidebar).

UPDATE (04/04/09): husky has grown into a pretty decent calculator app and is pretty usable now. Check it out – the code should be pretty accessible and readable.

In the following few installations I’ll give a few examples of how I leverage the command line to make my life easier. If this wets your appetite, you owe it to yourself to check out “Unix Power Tools” [1] and “The Bash Cookbook” [2].

Let’s get started. Recently, I faced the following task: I had a bunch of subdirectories that each contained some simulation data files. The directories were – quite inconveniently – named iteration_50001, iteration_50002, etc. all the way up to iteration_75000.

Unfortunately, in order to feed them into the next step of some modeling pipeline they needed to be named iteration_1, iteration_2, ..

Bummer! I guess most people (including myself) wouldn’t have hesitated to throw a few manual “mv iteration_50001 iteration_1″, “mv iteration_50002 iteration_2″, etc. at their keyboard if there would have been no more than 10 files or so. But 25,000 of them! Even assuming an average of 1 s for producing each of the mv commands, this would still take about 6 hours to complete. What else could be done? Well, how about this:

#!/bin/bash
for ((new_count = 1; new_count <= 25000; new_count++)); do
   ((current_count = new_count + 50000))
   mv iteration_${current_count} iteration_${new_count}
done

This small bash script does it all and completes in just a few seconds. Let’s briefly go over it. The complete program is one big loop, incrementing the new_count variable starting at 1 all the way up to 25,000. In the second line we specify the loop condition using bash’s arithmetic expansion operator (the ((…)) expression) just like we’d do it in C or C++. The new_count variable will count our (new) target directories, i.e., using bash’s parameter substitution they will be called iteration_${new_count}. Next, we need to compute the counter for the directory to be moved. Luckily, this simply amounts to adding 50000 to new_count which is what line 3 is all about, again using bash’s arithmetic expansion operator. The name of the directory to be moved then is iteration_${current_count}. In line 4 we simply move the present directory to the new one via the mv command. That’s it!

Clearly, you can use a loop of this kind for many other purposes potentially saving you a lot of work and grief.

[1] http://oreilly.com/catalog/9780596003302/
[2] http://www.bashcookbook.com/

For Linux systems there exist a plethora of good backup solutions. Here, I will explain the use of DAR [1], which makes backing up data a breeze.

Before we get started, we need to decide on a certain backup strategy. One reasonable choice might be to perform a full backup initially, followed by incremental backups at certain time intervals (daily, weekly, whatever seems appropriate). Periodically, e.g., every 6 months, another full backup could be pulled, followed by another set of incrementals until the next full backup is due. If you are backing up mission critical things, daily snapshots are certainly appropriate. Otherwise, weeklies might be sufficient.

Next, we have to decide what files to back up and where to store them. Let’s say we want to back up all of user Susan’s home directory, /home/susan, and we will keep the backup files on a separate hard-drive (USB or whatever) mounted at /mnt/backup. We start the backup process via

/usr/bin/dar -R /home/susan -c /mnt/backup/susan-home-full -z -s 1000M -P /home/susan/random-stuff

It is pretty easy to understand what is going on. “-R” specifies the root of the directory tree to be backed up; “-c” is the path to the backup file(s); “-z” specifies that we want to compress the backup using gzip; “-s” gives the maximum size of individual backup files (we certainly don’t want one huge 200G backup file); “-P” excludes certain sub-directories/files from the backup. Please consult [2] for many more options. That’s it, you just backed up your data!

In addition, it is very useful to use dar_manager supplied with dar to generate a database of all files contained in our backup. This will greatly simplify, aka, automate, retrieval of data, especially when parts of it are spread over multiple full and/or incremental backup files:

# create database
/usr/bin/dar_manager -C /home/susan/dar_backup_database.dmd 

# append full backup to database
/usr/bin/dar_manager -B /home/susan/dar_backup_database.dmd -A /mnt/backup/susan-home-backup

Here, “-B” specifies the path to the database file and “-A” gives the path to our initial full backup.

Now we can simply run differential backups at whatever time intervals necessary. To do so, the following simple bash script backup_home.sh might be helpful:

#!/bin/bash

# backup
if [ -z "$1" ]; then
  echo "usage: backup-home-disk <reference archive>"
  exit
fi

date=$(date +%F)

# create incremental backup
/usr/bin/dar -R /home/susan -c /mnt/backup/susan-home-diff-${date} -A "$1" -z -s 1000M -P  /home/susan/random-stuff

# update database
/usr/bin/dar_manager -B /home/susan/dar_backup_database.dmd -A /mnt/backup/susan-home-diff-${date}

Using this script, doing an incremental backup amounts to typing

# ./backup_home.sh <path to previous backup file>

Voila!

Well, not so fast. Before calling it a day and feeling all “backed up”, we need to make sure that our backups work as intended. Most importantly, we need to make sure that we can actually recover our data from the backup. Here, I’ll explain how to recover a specific set of files from the backup. If you need to recover the whole archive, e.g, after a disk failure please consult [2] for a very nice step by step walk-through. Hence, let’s pretend we lost /home/susan/pics and would like to recover it from our backups. We will recover the files into /tmp/backup via

# set up dar manager for recovery
/usr/bin/dar_manager -B /home/susan/dar_backup_database.dmd -o -R /tmp/restore/ -O -w

# recover data
/usr/bin/dar_manager -B /home/susan/dar_backup_database.dmd -r /home/susan/pics

In the initial setup step, we use the “-R” switch to specify where to recover the data to (without it we will simply replace the original files) and “-w -O” to signal that it is ok to overwrite earlier versions of files that dar_manager might pull from the backup. The second step then gets our files back via specifying all files to be recovered via the -r switch. If everything went well, you should now have a pristine copy of /home/susan/pics at the time of your last backup inside /tmp/restore. If not, please make sure to investigate what went wrong until you are absolutely sure your backups work. It certainly doesn’t hurt to occasionally check if data recovery works as it should.

We have only touched the surface of what DAR can do. If the above is all you need, great. If not please consult the docs for more info. As usual, man dar and man dar_manager are your friends.

[1] http://dar.linux.free.fr/
[2] http://dar.linux.free.fr/doc/Tutorial.html

I am of course referring to the eternal vi vs. emacs vs. <insert your favorite text editor here> flame war. After reading yet another one just the other day I suddenly got really fed up with it.

What editor I am using during my daily typing, you ask? Well, over the past many years I have spent a considerable amount of time trying out a fair amount of different editors. At the end of the day it turned out that vim was, has been, and still is exactly the right tool for me to get stuff done. In fact, vim has become part of me to a degree that I don’t even realize its modes/keystrokes/tabs any more until I am forced to edit a document in, e.g., openoffice and find myself hitting the escape key for no apparent reason and, more annoyingly, plastering the text with i’s and j’s before reverting back to slow edit mode via the arrow keys or even (gasp) the mouse.

What is my point, you ask? Well, I simply think that just because of the fact that one likes editor A, editor B does not automatically suck (pardon my language). Rather, I would like to propose that anybody in need of a text editor should strive to find the one that allows them to work most productively (*). Period! If this turns out to be emacs, great; or vim, great; or kate, great. Most importantly, if all your friends think editor A is great, but after using it for some time (a few months, say) you still find yourself looking at a cheat sheet a lot, or the hot-keys just don’t seem right, maybe it is time to look again, at editor B, maybe.

(*) Of course, it may well be that a set of different text editors be used for certain types of tasks.