Configuration files in Python

There is a very nice wrapper for libconfig for python, right here https://github.com/keimoon/python-libconfig
(This is connected to the previous post on libconfig here)

Why class balancing happens 'automatically' in AdaBoost

This is a very interesting characteristic of AdaBoost, that explains why, in general, it is not necessary to balance the initial weights during training for very skewed datasets.

I will use the notation from the AdaBoost Wikipedia article.

First, assume that we have $N_p$ positive samples and $N_n$ negative samples, with $N_n >> N_p$, representing a skewed dataset. Let's name the ratio between the number of negative and positive samples as $K = \frac{N_n}{N_p}$.

Now, assume that we are at the first iteration, and we have uniform weights $D_{t=1}(i) = \frac{1}{m}$ for all training samples. At the first iteration $t=1$ the goal is to find a weak learner

$h_{t} = \underset{h_{t} \in \mathcal{H}}{\operatorname{argmax}} \; \left\vert 0.5 - \epsilon_{t}\right\vert$, with $\epsilon_{t} = \sum_{i=1}^{m} D_{t}(i)I(y_i \ne h_{t}(x_{i}))$

since all weights are equal, $\epsilon_t$ can be expressed as $\epsilon_{t} = \sum_{i=1}^{m} I(y_i \ne h_{t}(x_{i}))$, so it is just the unweighted misclassification error.

Now it comes the interesting part: assume that the weak learners in the weak learner pool are very weak, so that the lowest misclassification error $\epsilon_t$ is achieved with a weak learner that classifies all samples as negatives. (This seems unreasonable, but I have seen it happen frequently on skewed datasets).

With this in mind, $\epsilon_1 = \frac{N_p}{N_p + N_n}$. With simple math we can compute the value of $\alpha_1=\frac{1}{2} \log \frac{1 - \epsilon_1}{\epsilon_1}$ and update the distribution $D$ for iteration $t=2$, obtaining

$$\frac{D_{t=2}(+)}{D_{t=2}(-)} = \frac{N_n}{N_p} = K$$

where $D_{t=2}(+)$ and $D_{t=2}(-)$ are the weight of the positive and negative samples respectively.

This means that the effect of the first weak learner was to balance the dataset, so that the sum of the positive weights equal the sum of the negative ones. Another equivalent interpretation is that the first weak learner just adds a constant to the predictive function, namely $-\alpha_1$ in this particular case.

However, there are some differences w.r.t. balancing the weights initially:

• The first weak learner is 'wasted' and could be avoided.
• If shrinkage is introduced, it is likely that many weak learners will be 'wasted'. Though 'wasted' is not very meaningful, because they would still be useful and help minimize training loss.
• The fact that the first weak learner returns -1 for all training samples does not imply that it will also for any unseen new sample. The implications of this are hard to tell though.
• I am assuming that a weak learner can return the same value for all training samples, which is the case for a decision stump, but it may not happen with other types of weak learners (e.g. svm)
• .. I am pretty sure there are other differences

Cheers!

C++11: compile-time lookup-table/array with constexpr

I've been trying a few of the new features of C++11, wondering how difficult it would be to create a compile-time self-populated array.

Here there is a piece of code that creates an array that contains the values of  sin(0.2 x) + cos(0.5 x), completely at compile-time, and then prints it in ascii form to stdout. The most tricky part is the one containing a variadic template, which computes a set of indices from 0 to N-1.

You can see the output it produces here

#include <iostream> #include <vector> #include <string> #include <cmath> #include <iomanip> /** Range generation, * from http://stackoverflow.com/questions/13313980/populate-an-array-using-constexpr-at-compile-time **/ template<unsigned... Is> struct seq{}; template<unsigned N, unsigned... Is> struct gen_seq : gen_seq<N-1, N-1, Is...>{}; template<unsigned... Is> struct gen_seq<0, Is...> : seq<Is...>{}; /** A table consisting of indexes and values, * which will all be computed at compile-time **/ template<unsigned N> struct Table { unsigned indexes[N]; double values[N]; static constexpr unsigned length = N; }; template< typename LambdaType, unsigned... Is> constexpr Table< sizeof...(Is) > TableGenerator(seq<Is...>, LambdaType evalFunc) { return {{ Is... }, { evalFunc(Is)... }}; } template<unsigned N, typename LambdaType> constexpr Table<n> TableGenerator( LambdaType evalFunc ) { return TableGenerator(gen_seq<n>(), evalFunc); } /** Function that computes a value for each index **/ constexpr double myFunc( unsigned idx ) { return sin(0.2 * idx) + cos(0.5*idx); } int main() { constexpr unsigned length = 100; // create compile-time table constexpr Table<length> table = TableGenerator<length>( myFunc ); // print values in vertical form, pretty-looking ;) const double lineMult = 12; const double lineOffset = 30; for(auto v : table.values) { const unsigned numSpaces = (unsigned) ( lineOffset + v * lineMult + 0.5 ); std::cout << std::setfill(' ') << std::setw( numSpaces ) << "o" << std::endl; } std::cout << std::endl; return 0; }

Wacom Pen & Touch CTH670 and Linux

I've just got a Wacom Pen & Touch tablet CTH670. I could not make it work straight away with Archlinux, but then found out that I had to follow the steps in this forum post , which was originally meant to be for the CTL-470/k.


In case the attachment in the linked forum post doesn't work, I am duplicating it here.


UPDATE: there seems to be a small issue with that version posted above. The version that works like a charm with my tablet is http://sourceforge.net/projects/linuxwacom/files/xf86-input-wacom/input-wacom/

Trying to read/write configuration files?

Loading, parsing and writing configuration files is generally not a trivial task. I found this amazing library recently, which does the job with a few C++ lines.

For example, if you have a configuration file such as:

#---------------------------- # Example Configuration File #--------------------------- # application: { /* This section defines some settings for our * main application window, such as size and * position. */ window: { title = "My Application"; size = { /* width */ w = 640; /* height */ h = 480; }; pos = { x = 350; y = 250; }; }; a = 5; b = 6; } Fields can be easily read with something as simple as:

Speeding up Matlab over X11 Forwarding

Running Matlab over ssh is handy. If one wants to use the GUI, X11 forwarding is possible, but it is painfully slow if both machines are not inside a local network.

I found information about how to speed it up after googling for a while. In the end I think it is worth it to summarize the steps here:

• In the current folder where you are going to run matlab from, create a file name java.opts with the content 

                   "-Dsun.java2d.pmoffscreen=false" (without the double commas).

• Use compression with ssh, by connecting to the server with

                      ssh -c arcfour,blowfish-cbc -Y -C username@server

The speed up obtained is particularly noticeable when changing focus between different windows in the Matlab GUI.

GNOME3 Fallback mode 'fix'

I switched to archlinux recently. I was used to Gnome2, and Gnome3 is quite of a disappointment in the sense that it involves re-learning how to do things, when software is supposed to adapt to us. Anyway, I don't plan to start a discussion on that, googling already reveals many 'gnome3 sucks' posts which provide lot of detail on the 'why'.

In summary, I tried KDE4, but didn't feel comfortable either. Finally I ended up in GNOME3 fallback mode, whose default theme is disappointing as well. I found this theme which makes it look friendlier http://gnome-look.org/content/show.php?content=145210 , but had some problems in nautilus and background font color. So I modified it and uploaded it here.

Now my desktop reminds me of Gnome2 and sometimes I believe it is even friendlier and better looking ;)

ITK and C++11

Dacap made me aware of the new C++11 standard and the supported functionality in GCC 4.6.1. Right after that, I downloaded GCC 4.6.1 and compiled it to try it with the Insight Tool-kit (ITK), particularly because of all those super-templated types in ITK that make coding longer than what it should take, mostly because of the amount of typedefs and re-typing of class names. One of the 'wonders' of C++11 is the auto keyword, which happens to be close to a salvation angel for ITK users.

My main concern was that C++11's implementation in GCC is still quite new, such that incompatibilities may appear almost instantly; and that was the case with a particular GCC extension that -std=c++0x was considering invalid. Therefore, when compiling ITK-based code you may find 'constexpr' errors if you enable c++0x support, particulary in the file vnl.h. Fortunately, a patch has already been created and published here and can be applied directly to the 4.6.1 release source code.

To compile it with Ubuntu you can follow the instructions given here. In my case I didn't want to overwrite the previous gcc compiler in my system, nor I hadn't administrative rights, so I used make DESTDIR=/wherever/you/want install . This way I just had to modify the PATH environment variable to point to the gcc 4.6.1 binaries and tell CMake to use g++-4.6 as the compiler rather than the standard gcc one.

The patch mentioned earlier doesn't disable the constexpr error by itself; for that to be effective one must use the -fpermissive compiler flag. Some other compatibility problems are described here.

Moreover, it is very likely to run on another error regarding an include file missing. To obtain a successful build of ITK my CXX flags were set to: -std=c++0x -fpermissive -include cstddef

UPDATE: a better practice is to compile GMP, MPFR and other libs as static before compiling GCC, so that they are statically linked and the generated compiler binaries can be executed in other machines that may not have those specific libraries installed. Instructions for compiling these libraries and GCC can be found here. My particular configure line for GCC 4.6.1 is the following:

configure --prefix=/usr \

--enable-languages=c,c++,fortran \

--enable-threads=posix \

--enable-tls \

--enable-libgomp \

--enable-lto \

--disable-nls \

--disable-checking \

--disable-multilib \

--with-gmp=/tmp/gcc \

--with-mpfr=/tmp/gcc \

--with-mpc=/tmp/gcc \

--with-libelf=/tmp/gcc \

--with-fpmath=sse

Linux: Fixing resolution problem on external monitor

I recently got a Dell L502x laptop. In order to connect it to an external VGA monitor I am using the mini-Displayport (a la Apple).

The problem is that Linux (Ubuntu Natty) does not always figure out the right display modes for this particular monitor. Unfortunately, cvt nor gtf generate the correct modelines and I am stuck with a lower resolution.

The trick was to dual-boot Windows :S and use PowerStrip to extract the proper modelines, as explained here http://www.x.org/wiki/FAQVideoModes

A bunch of very useful information can also be found here: https://wiki.ubuntu.com/X/Config/Resolution

Excellent good-looking plots

When doing research one usually doesn't care about the aesthetics of plots or the visualized results, as long as they are clear enough to be interpreted. However, when making presentations or in publications, nice plots have an interesting impact and, even though they do not change how good results are, they still make things look more professional.

I have been usin Matlab for a while now. It is one of those softwares that you usually hate, especially if you come from a better-structured programming background. Leaving execution speed out of the equation, Matlab sucks in many ways but there a few pros that build up its popularity, such as extremely easy and straightforward debugging and the available toolboxes and functions on Matlab Central. However, it is very easy to find blogs like Abandon Matlab, where some posts really make the point about leaving Matlab forever and finding a better and more appropriate alternative.

Anyways, enough of Matlab hate. The point is that I was looking for nice plots and I remembered about that amazing piece of software called Mathematica. I will not discuss the differences between Mathematica and Matlab, but just say that they were made for different purposes. However, take a look at the following plots generated with Matlab and Mathematica respectively, from the same data (click to see the real image since blogger is automatically introducing some JPEG artifacts):

Matlab	Mathematica
load data.mat; stem(x,y); hold on; xlabel('x','Interpreter','LaTex'); ylabel('f(x)','Interpreter','LaTex');	data = Import["test.mat", "LabeledData"]; ListPlot[ Transpose[Flatten[{"x" /. data, "y" /. data}, 1]], Filling -> Axis, AxesLabel -> {x, f[x]} ]

The difference is easy to grasp by just looking at the plots: Mathematica does a great job, while Matlab looks just ok. Something I usually do to make Matlab plots is to apply a grid with grid on, but the plot still looks not as professional as with Mathematica. Obviously there is space for cheating here; maybe if you look at how much Mathematica code is needed you may say that I haven't been fair enough. However, most of the code in the Mathematica snippet is needed because the data is read from a Matlab MAT file.

In my opinion, the most amazing and simple detail that Mathematica uses and Matlab does not is antialiasing. It is a very subtle detail but it makes plots look softer and, somehow, more human and easier for our eyes to watch. There have been some attempts such as this script. However, that script is an smart user attempt to generate a plot with antialiasing but the antialiasing is simulated by resizing the plot, which still doesn't look as good as Mathematica's output (images not shown here but you can try it on your own).

Beyond that, there are some problems when exporting plots from Matlab. First, exporting to PDF generates a PDF file that contains a whole page such as A4 and the plot in the middle with huge white spaces around. This is very annoying when working with pdflatex and the PDF generated by Matlab must be cropped. Moreover, even if exporting to PNG, the final image file does not look exactly as what is seen on the computer screen. This can lead to an endless 'fight' with Matlab's exporting options, sometimes without success. All of this doesn't seem to happen to Mathematica, at least considering what I have tried so far. PDF files look great and have the exact size of the plot and can be inserted straight away into latex documents.

I don't want to go into much detail in this post since it could take a long time to discuss plot customization options in Matlab and Mathematica. My experience shows that Matlab works well to show the data properly and it is still interpretable, but it lacks the quality of softwares such as Mathematica or Matplotlib (this last one is another very nice plotting tool, free of charge). For more examples on the plotting power of Mathematica see here, and for Matplotlib here.

Nice Linux PDF manipulation utilities

I found myself writing many reports with LaTex lately. Using pdflatex has it advantages, but things can get quite annoying when one wants to insert a figure which was generated in Matlab or any other program.

Specially, MATLab does not do a nice job when exporting PDFs and leaves a whole blank area (the page itself actually) which is not desirable if we want to put a figure in a latex document. Fortunately, there is a linux program called pdfcrop that does the job correctly (not 100% trustable, but 90% of the times I get good results).

Another useful program is pdfimages, which extracts images from a pdf file. The pdfimages output is generally in huge pgm files, so it's better to convert it to something like png which results in smaller files, still usable with pdflatex.

Finally, pdfjoin and pdf90 allows one to join several files into a single one and rotate pages respectively. The Ubuntu package for these two is called pdfjam.

Ultra simple incremental backups with rsync

Recently I bought an external hard drive for backups. While searching for the best way to do the backups I found rsync, which looks well suited for these tasks.
Then I found this webpage that provides some scripts to achieve circular snapshots.

However, most scripts on the web are extremely large and complex compared to what I need. So finally I made my own script that makes automated incremental backups, with no cycling but keeping a history file to identify each backup.

Here is the code:
<br />#!/bin/bash<br />##############################################<br />## This script should be called from the dir<br />## where the backup will be made to<br />###############################################<br /><br /># directory to backup<br />sourceDir=/media/something<br />rsyncFlags='--modify-window=1 -a --verbose --delete '<br /><br />#find if we have a backup already<br />find backup.0 -maxdepth 0 > /dev/null;<br /><br />if (( $? != 0 )); then {<br /> echo 'First Backup';<br /> nextBackup='0';<br /> firstBackup=1;<br />} else<br />{<br /> # find last backup<br /> lastBackup=`find backup.* -maxdepth 0 | sed 's/backup.//' | sort -n -r | head -1`;<br /><br /> echo 'Resuming backup' $lastBackup '...';<br /><br /> # +1 for next one<br /> nextBackup=$(( $lastBackup + 1 ));<br /> echo $nextBackup;<br /> firstBackup=0;<br />}<br />fi<br /><br />lastBackup=backup.$lastBackup;<br />nextBackup=backup.$nextBackup;<br /><br />if (( $firstBackup == 1 )); then {<br /> rsync $rsyncFlags $sourceDir/ $nextBackup/;<br /> R=$?;<br />} else<br />{<br /> rsync $rsyncFlags --link-dest=../$lastBackup $sourceDir/ $nextBackup/;<br /> R=$?;<br />} fi<br /><br />if (( $R != 0 )); then {<br /> echo '=======> ERROR DURING BACKUP!!';<br /> exit $R;<br />} fi<br /><br /># make link to last backup<br />rm -f HEAD;<br />ln -s -f $nextBackup HEAD<br />if (( $? != 0 )); then {<br /> echo '=======> ERROR Creating link!!';<br /> exit $?;<br />} fi<br /><br /># save history<br />echo -e $nextBackup '\t\t' `date -R` >> backup-history;<br />if (( $? != 0 )); then {<br /> echo '=======> ERROR Saving History!!';<br /> exit $?;<br />} fi<br /><br />echo ' ';<br />echo '------------- BACKUP DONE ----------------';<br />echo ' ';<br />

This script will create backup directories called backup.xxxx, where xxxx is the backup number. This number is zero for the first backup done. Rsync hard-links the unchanged files to the previous backup which is a key method to save space.

With this script I am backuping a NTFS partition and that's why the option --modify-window=1 is needed in rsyncFlags. It is important to write sourceDir without a trailing forward slash!

In order to provide easy access to the last backup, a symbolic link called HEAD will point to the latest backup. Additionally, a file called backup-history holds the exact time and date where each backup was performed.

NOTE: I am not responsible for this script and it's correctness. There might be problems such as if there are backup.something folders or files in the backup directory where the script is called, and other bugs that may arise. This is a very simple and minimalistic script!

Merging Qt and Eigen

Again in ViBOT, image segmentation assignment, Matlab is really slow, wait minutes for results...

So I decided to try to use Qt for the GUI and OS abstraction layer together with Eigen which is another amazing template-based library for matrix manipulation. The important code to write was to link both libraries, taking advantage of Qt's amazing QImage class which is able to open several file formats and perform low-level pixel access. In a few words, I had to put all the image information contained in QImage into a Eigen's matrix.

Luckily, this task is very simple. Here there is some code:

#ifndef MIMG_H
#define MIMG_H

USING_PART_OF_NAMESPACE_EIGEN

#include <QImage>

#include <Eigen/Core>
#include <Eigen/Array>

//general type, maybe float or double needed
typedef MatrixXf    MImgType;

class MImg
{
public:
 //creates an all-black image
 MImg(unsigned int h, unsigned int w);

 //creates image from QImage
 MImg( const QImage &img );

 MImgType    R,G,B;  //each component
         //made public for faster access

 unsigned int    getHeight();
 unsigned int    getWidth();

 QImage *    toQImage();  //convert to QImage

 /**
   Maximizes dynamic range of three channels
   independently!
  **/
 void    maximizeIndependentDynamicRange();

private:
 unsigned int mH,mW; //height, width

};

#endif // MIMG_H

#include "mimg.h"

MImg::MImg(unsigned int h, unsigned int w)
{
  R = MImgType::Zero(h,w);
  G = MImgType::Zero(h,w);
  B = MImgType::Zero(h,w);

  mH = h;
  mW = w;
}

MImg::MImg( const QImage &img )
{
  int w = img.width();
  int h = img.height();

  R = MImgType::Zero(h,w);
  G = MImgType::Zero(h,w);
  B = MImgType::Zero(h,w);

   //now copy values..
    for (int y=0; y < h; y++)
        for (int x=0; x < w; x++)
        {
            QRgb color = img.pixel(x,y);
            R(y,x) = qRed(color)/255.0;
            G(y,x) = qGreen(color)/255.0;
            B(y,x) = qBlue(color)/255.0;
        }

  return img;
}

void    MImg::maximizeIndependentDynamicRange()
{
  double  min, max;

  min = R.minCoeff(); max = R.maxCoeff();
  R = (R.cwise() - min) / (max - min);

  min = G.minCoeff(); max = G.maxCoeff();
  G = (G.cwise() - min) / (max - min);

  min = B.minCoeff(); max = B.maxCoeff();
  B = (B.cwise() - min) / (max - min);
}

unsigned int    MImg::getHeight() {
  return mH;
}

unsigned int    MImg::getWidth() {
  return mW;
}

It is important to mention that this code only handles RGB and won't care about grayscale images or any other type of colour models. The advantage of having the image in this matrix form is that Eigen provides an easy syntax for matrix manipulation, along with many modules performing least squares, Cholesky, diagonalization, etc.

Matlab and n-dimensional array sorting

Wow.. it's been such a long time since the last post!! I've been quite busy with ViBOT, specially during the last two weeks. Anyway, I thought it would be nice to write a post about some nice Matlab functions I found quite useful:

reshape:this is a nice function that lets you reshape any array or matrix into any other array or matrix with different dimensions, as long as the number of elements is kept the same. It is very useful when one wants to loop through every element of a two or three dimensional array. If A=[1 2; 3 4] then reshape(A,[1 4]) will return [1 2 3 4]. To get back to the original array we can then use reshape([1 2 3 4],[2 2]).

ind2sub: this is a useful function when manipulating arrays that were linearised with reshape. From Matlab help: "The ind2sub command determines the equivalent subscript values corresponding to a single index into an array". A simple example is the following:

• A=[1 2; 6 5]; B = reshape(A,[1 4]);
• [sV, sI] = sort(B, 'descend'); % sort linearised array
• disp(sV(1)); %show max value: 6
• disp(sI(1)); %show max index: 2
• [x,y] = ind2sub( size(A), sI(1) );
• disp(x); disp(y); % (x,y) == (2,1), we got the coordinates in the original matrix

There is also a sub2ind function which does the reverse transformation. ind2sub was very useful to ease the sorting task when applying the Hough transform for circle detection, where there is an accumulator matrix which is 3-dimensional.

FPGAs are taking over!

From the moment I knew and learned about FPGAs (Xilinx) I looked forward to use them to replace large logic circuits. This way the system would be not only scallable and the logic programmable but costs should be reduced too.

Unfortunately, most of the times the FPGA alternative was much more expensive than the equivalent logic circuit implemented with separate logic ICs.

Finally the day came and I got the chance to build a board with an ARM7 core + Spartan3A FPGA. Total price was reduced and the system became fully programmable. The ARM7 chip (LPC23xx) configures the FPGA on startup which happens to be really fast (52kib for XC3S50A). The microcontroller and FPGA are connected through a parallel bus with many control lines.

The XC3S50A is optimal in the sense that it only requires 3.3V and 1.2V supplies so it can be directly connected to the microcontroller pins.

Here there are some pictures:

Qt-Embedded: Capturing screen with QPixmap

I've been working on a product manual lately. I needed to include several LCD screenshots into it so I tried to come up with an easy way to capture snapshots from our Qt/Embedded app.

Qt/Embedded provides a nice method to save window/framebuffer contents directly to an image file. However, I wanted to send the 'take-snapshot' command from a tty console (telnet/serial/etc) since there weren't any other buttons on the system to trigger that.

A QTimer is set up. Periodically it checks the file /tmp/doCapture. If it exists a snapshot is taken and an image file is saved. Its filename is taken from the contents of /tmp/doCapture. After saving the image /tmp/doCapture is deleted.

Here is the code, which should be placed in the main window, whose width and height cover the whole screen:

mainWindow::mainWindow()
{
 // captureTimer should be declared in mainWindow's class definition
 captureTimer = new QTimer(this);
 connect( captureTimer, SIGNAL(timeout()), this, SLOT(captureTimerEvent()) );

 captureTimer->start(1000); //check interval
}

void mainWindow::captureTimerEvent()
{
 QString tmpFile = QString("/tmp/doCapture");

 if ( !QFile::exists(tmpFile) )
     return;

 QFile f(tmpFile);
 if ( !f.open( QIODevice::ReadWrite ) )
     return;

 char buf[200];
 if ( f.readLine( buf, sizeof(buf) - 4 ) == -1 )
     return;

 buf[strlen(buf)-1] = '\0'; //remove \n created by 'echo'-- not safe!

 strcat( buf, ".png" );

 //capture
 QPixmap p = QPixmap::grabWindow( this->winId() );

 if ( p.save( buf ) )
     printf("------- GRAB OK\n");
 else
     printf("------- ERR GRAB!\n");

 /* delete file */
 f.remove();
}

This way, all I have to do to take a snapshot is to write:

echo pngfilename > /tmp/doCapture

Compiling and using GDB for arm-linux

Some days ago I had to compile gdb manually in order to debug an arm-linux app. It's quite trivial but it's so useful that I thought it would be a nice idea to post the instructions here. Below there are some explanations about debugging remotely with KDevelop.

This was done with gdb-6.8, you can grab it here. It is assumed that arm-linux tools are available (PATH correctly set).

Compiling the GDB client

Decompress gdb-6.8 and compile it by issuing:

tar xvf gdb-6.8.tar.gz
cd gdb-6.8
./configure --build=x86 --host=x86 --target=arm-linux
make

After compiling just copy gdb/gdb to arm-linux-gdb where the arm-linux toolchain binaries reside (this is purely for organization and proper naming). You can now remove the gdb-6.8 directory.

NOTE: if your host computer architecture isn't intel-based just replace x86 by the correct value for your platform.

Compiling the GDB server (ARM)

Decompress gdb-6.8 and compile it by issuing:

tar xvf gdb-6.8.tar.gz
cd gdb-6.8
./configure --host=arm-linux
make

After compiling just copy gdb/gdbserver/gdbserver to your arm-linux filesystem so that you can execute it from a remote shell (gdbserver will be an arm-elf binary). You can now remove the gdb-6.8 directory.

Testing connections

First the server should be started in the  arm processor by issuing something like:

gdbserver host:1234 _executable_

Where _executable_ is the application that is going to be debugged and host:1234 tells gdbserver to listen for connections to port 1234.

To run gdb just type this in a PC shell:

arm-linux-gdb --se=_executable_

After that you'll get the gdb prompt. To connect to the target type:

target remote target_ip:1234

To resume execution type 'continue'. You can get an overview on gdb usage here.

Debugging with KDevelop

KDevelop can be used to watch variables and debug the remote target (place breakpoints, stop execution, etc). Just go to Project -> Project Options and make sure you have something like this:

monitor-debug.gdb is a file that contains the following

target remote target_ip:1234

After this you will be ready to remotely debug any arm-linux application.

_

Catching uncaught exceptions in Java

This month looks quite javaized!

So here is another thing I came up with, regarding uncaught exceptions which usually end the application totally unexpectedly. I wouldn't mind about that, but I felt that the user should be notified with a minimum-sense message and some technical info so that he/she can send it over to the developers.

So here it is a 'BodyGuard' class that does the job by using a callback present in the Thread class.

package bodyguard;

public class BodyGuard implements Thread.UncaughtExceptionHandler
{ 
 static private BodyGuard    bGuard;

 static public void  registerGuard() {
     bGuard  = new BodyGuard();
     Thread.setDefaultUncaughtExceptionHandler(bGuard);
 }

 public void    uncaughtException( Thread thread, Throwable e )
 {
     java.io.StringWriter    sW = new java.io.StringWriter();
     e.printStackTrace( new java.io.PrintWriter(sW));
  
     String  s = "A fatal error was detected during execution: \n" +
             "Thread: " + thread.getName() + "\n" +
             "Exception: " + sW.toString() + "\n" +
             "The application will be closed now.";
  
     showFatalErr(s);
 }

 private void    showFatalErr( String err )
 {
     //JOptionPane.showMessageDialog(null, err,"Error", JOptionPane.ERROR_MESSAGE);
     BodyGuardDialog dlg = new BodyGuardDialog(null, true, err);
     dlg.setLocationRelativeTo(null);
     dlg.setVisible(true);
  
     System.exit(-1);    //exit n ow
 }
}

BodyGuardDialog is another class derived from JDialog that shows the corresponding error message and lets the user copy the error to the clipboard.

In order to register this handler one just has to call bodyguard.BodyGuard.registerGuard() when starting up (ie: static void main()).

This class will catch uncaught exceptions from all threads, displaying the thread's name where the exception was thrown.

_

Java Application and self-restart

Java's networking capabilities and prebuilt classes make self-updating applications an easy task for developers. A simple method is to store a file containing current version number on a web host, together with a zipped/tared file containing the whole or partial application files to update.

Also, java applications are truly self-updatable since the application itself can overwrite its class files (or jar ones), since the java VM loads all its contents into memory before start up (except for dynamic 'late' class loading).

After coding the necessary classes to perform the update I realised I needed a way to tell java to restart the application, loading the updated code from the new class files. There wasn't an automated way to do this, so I came up with the next piece of code which invokes the java VM to execute the JAR file where certain class belongs to.

public boolean  restartApplication( Object classInJarFile )
{
    String javaBin = System.getProperty("java.home") + "/bin/java";
    File jarFile;
    try{
        jarFile = new File
        (classInJarFile.getClass().getProtectionDomain()
        .getCodeSource().getLocation().toURI());
    } catch(Exception e) {
        return false;
    }

    /* is it a jar file? */
    if ( !jarFile.getName().endsWith(".jar") )
    return false;   //no, it's a .class probably

    String  toExec[] = new String[] { javaBin, "-jar", jarFile.getPath() };
    try{
        Process p = Runtime.getRuntime().exec( toExec );
    } catch(Exception e) {
        e.printStackTrace();
        return false;
    }

    System.exit(0);

    return true;
}

There are some important aspects to have in mind for this code:

• The application's main class must be in a jar file. classInJarFile must be an instance of any class inside the same jar file (could be the main class too).
• The called java VM will be the same that the application is currently running on.
• There is no special error checking: the java VM may return an error like class not found or jar not found, and it will not be caught by the code posted above.
• The function will never return if it doesn't catch an error. It would be a good practice to close all the handlers that could conflict with the 'duplicate' new application before calling restartApplication(). There will be a small time (which depends on many factors) where both applications will be running at the same time.

The code can be easily modified for a class file approach rather than jar ones.

A bash script or .bat (Windows) would work, calling the application indefinitely in the case the process' return value matches a specified number (which would be set after a successful upgrade). However, this wouldn't be platform independent.

Loop unrolling and speed-up techniques

Today I started some investigation on DTMF detection. After some research I decided to try the Goertzel algorithm, wrote some code from scratch and tried it on a real equipment based on an ARM7 core. It did fine actually, but after some benchmarking I thought it was somewhat slow.

I wrote the code thinking on future optimization but keeping simplicity in mind, specially because I didn't know how it was going to behave. However, as said above, calculations took much longer than what I expected, even though they were quite fast, exceeding the requirements for real-time processing (telephone channel, 8khz, mono). I still wanted to keep the code in C for portability so I tried some tricks.

I wrote the code with fixed-point math in mind, since floating point arithmetic would kill performance.

First, here are some basic definitions:

/* Struct for a single Goertzel calculator */
typedef struct
{
 dsp_t sp1,sp2; //past values (IIR)
 dsp_t coeff;   //calculated only once

} GOERTZ_s;

/* Calculate Goertzel 'step' */
#define GOERTZ_calc( _s, _val ) \
 { \
     dsp_t _gs = (_val) + FP_MULT2( _s.coeff, _s.sp1 ) - _s.sp2; \
     _s.sp2 = _s.sp1; \
     _s.sp1 = _gs; \
 }

The first approach was the simplest one, I needed sixteen Goertzel calculations (8 frequencies + 8 second harmonic for the first frequencies). I had a buffer where the incoming PCM data was copied to so I had to process that array for each of the 16 Goertzel 'calculators'.

static inline void GOERTZ_processAll( dsp_t sample )
{
 int i;
 for (i=0; i < 16; i++)
     GOERTZ_calc( goertzs[i], sample );
}

/* this is inside another function */
int i;
for (i=0; i < BUFFER_SIZE; i++)
 GOERTZ_processAll( bufferData[i] );

Where BUFFER_SIZE is the size of bufferData[] and goertzs[] is a 16-element array of GOERTZ_s structs, already initialized. Notice the inline modifier in GOERTZ_processAll(). It's really important since function inlining will help in execution time, specially when the function is called so frequently.

That code worked alright, but was too slow, at least for my intuition. The progress into a more efficient scheme had many steps, but basically there were three important changes:

• dsp_t was defined as int16_t, but the processor's 'native' word length is 32-bit, so changing dsp_t to be int32_t speeded up the calculations by removing cast and special assignment instructions.
• Loop unrolling was done in GOERTZ_processAll(). That means more flash is used but it's a good price to pay for a good speed up. Besides that, here we are talking about repeating the same thing 16 times so it's not a big problem if we have, to say, a 512k flash microcontroller or dsp.
• Instead of passing a single value to GOERTZ_processAll() a 8-byte array is used. This improves time considerably but also increases code size. As said in the previous point that wasn't an issue this case. Note that a 16-byte array won't necessarily increase performance. It depends on the core architecture, and in this case it made things worse (but better than single value parameter passing).

With these modifications I got a 100% performance increase, which means that there can be twice as much DTMF decoders compared to the non-optimized version.

Here is the code with the modifications:

static inline void GOERTZ_processAll( dsp_t samples[8] )
{
 #define DO(j) \
     GOERTZ_calc( goertzs[j], samples[0] ); \
     GOERTZ_calc( goertzs[j], samples[1] ); \
     GOERTZ_calc( goertzs[j], samples[2] ); \
     GOERTZ_calc( goertzs[j], samples[3] ); \
     GOERTZ_calc( goertzs[j], samples[4] ); \
     GOERTZ_calc( goertzs[j], samples[5] ); \
     GOERTZ_calc( goertzs[j], samples[6] ); \
     GOERTZ_calc( goertzs[j], samples[7] );

 DO(0);
 DO(1);
 DO(2);
 DO(3);
 DO(4);
 DO(5);
 DO(6);
 DO(7);
 DO(8);
 DO(9);
 DO(10);
 DO(11);
 DO(12);
 DO(13);
 DO(14);
 DO(15);

#undef DO
}

/* somewhere in another function */
int i;
for (i=0; i < BUFFER_SIZE; i +=8 )
 GOERTZ_processAll( bufferData + i );

But it can do better...

After observing how good performance went after this modifications I tried to do more loop unrolling to see if I could improve it. If the data buffer is large enough then the for loop that process 8 samples at a time wastes useful CPU instructions, so I included it inside a function called GOERTZ_processBuffer() which takes a buffer pointer and its length. There was another 100% performance increase from the previous optimization. This means a 4x speed up from the original code! Note that, once again, more flash is being used for loop unrolling.

static inline void GOERTZ_processBuffer( dsp_t *samples, int num )
{
 #define DO(j) \
   for ( i=0; i < num; i+=32 ) { \
       GOERTZ_calc( gss[j], samples[0+i] ); \
       GOERTZ_calc( gss[j], samples[1+i] ); \
       GOERTZ_calc( gss[j], samples[2+i] ); \
       GOERTZ_calc( gss[j], samples[3+i] ); \
       ... up to 32 .. ; }


 DO(0);
 DO(1);
 DO(2);
 DO(3);
 DO(4);
 DO(5);
 DO(6);
 DO(7);
 DO(8);
 DO(9);
 DO(10);
 DO(11);
 DO(12);
 DO(13);
 DO(14);
 DO(15);

#undef DO
}