Ultra simple incremental backups with rsync

Recently I bought an external hard drive for backups. While searching for the best way to do the backups I found rsync, which looks well suited for these tasks.
Then I found this webpage that provides some scripts to achieve circular snapshots.

However, most scripts on the web are extremely large and complex compared to what I need. So finally I made my own script that makes automated incremental backups, with no cycling but keeping a history file to identify each backup.

Here is the code:
<br />#!/bin/bash<br />##############################################<br />## This script should be called from the dir<br />## where the backup will be made to<br />###############################################<br /><br /># directory to backup<br />sourceDir=/media/something<br />rsyncFlags='--modify-window=1 -a --verbose --delete '<br /><br />#find if we have a backup already<br />find backup.0 -maxdepth 0 > /dev/null;<br /><br />if (( $? != 0 )); then {<br /> echo 'First Backup';<br /> nextBackup='0';<br /> firstBackup=1;<br />} else<br />{<br /> # find last backup<br /> lastBackup=`find backup.* -maxdepth 0 | sed 's/backup.//' | sort -n -r | head -1`;<br /><br /> echo 'Resuming backup' $lastBackup '...';<br /><br /> # +1 for next one<br /> nextBackup=$(( $lastBackup + 1 ));<br /> echo $nextBackup;<br /> firstBackup=0;<br />}<br />fi<br /><br />lastBackup=backup.$lastBackup;<br />nextBackup=backup.$nextBackup;<br /><br />if (( $firstBackup == 1 )); then {<br /> rsync $rsyncFlags $sourceDir/ $nextBackup/;<br /> R=$?;<br />} else<br />{<br /> rsync $rsyncFlags --link-dest=../$lastBackup $sourceDir/ $nextBackup/;<br /> R=$?;<br />} fi<br /><br />if (( $R != 0 )); then {<br /> echo '=======> ERROR DURING BACKUP!!';<br /> exit $R;<br />} fi<br /><br /># make link to last backup<br />rm -f HEAD;<br />ln -s -f $nextBackup HEAD<br />if (( $? != 0 )); then {<br /> echo '=======> ERROR Creating link!!';<br /> exit $?;<br />} fi<br /><br /># save history<br />echo -e $nextBackup '\t\t' `date -R` >> backup-history;<br />if (( $? != 0 )); then {<br /> echo '=======> ERROR Saving History!!';<br /> exit $?;<br />} fi<br /><br />echo ' ';<br />echo '------------- BACKUP DONE ----------------';<br />echo ' ';<br />

This script will create backup directories called backup.xxxx, where xxxx is the backup number. This number is zero for the first backup done. Rsync hard-links the unchanged files to the previous backup which is a key method to save space.

With this script I am backuping a NTFS partition and that's why the option --modify-window=1 is needed in rsyncFlags. It is important to write sourceDir without a trailing forward slash!

In order to provide easy access to the last backup, a symbolic link called HEAD will point to the latest backup. Additionally, a file called backup-history holds the exact time and date where each backup was performed.

NOTE: I am not responsible for this script and it's correctness. There might be problems such as if there are backup.something folders or files in the backup directory where the script is called, and other bugs that may arise. This is a very simple and minimalistic script!

Merging Qt and Eigen

Again in ViBOT, image segmentation assignment, Matlab is really slow, wait minutes for results...

So I decided to try to use Qt for the GUI and OS abstraction layer together with Eigen which is another amazing template-based library for matrix manipulation. The important code to write was to link both libraries, taking advantage of Qt's amazing QImage class which is able to open several file formats and perform low-level pixel access. In a few words, I had to put all the image information contained in QImage into a Eigen's matrix.

Luckily, this task is very simple. Here there is some code:

#ifndef MIMG_H
#define MIMG_H

USING_PART_OF_NAMESPACE_EIGEN

#include <QImage>

#include <Eigen/Core>
#include <Eigen/Array>

//general type, maybe float or double needed
typedef MatrixXf    MImgType;

class MImg
{
public:
 //creates an all-black image
 MImg(unsigned int h, unsigned int w);

 //creates image from QImage
 MImg( const QImage &img );

 MImgType    R,G,B;  //each component
         //made public for faster access

 unsigned int    getHeight();
 unsigned int    getWidth();

 QImage *    toQImage();  //convert to QImage

 /**
   Maximizes dynamic range of three channels
   independently!
  **/
 void    maximizeIndependentDynamicRange();

private:
 unsigned int mH,mW; //height, width

};

#endif // MIMG_H

#include "mimg.h"

MImg::MImg(unsigned int h, unsigned int w)
{
  R = MImgType::Zero(h,w);
  G = MImgType::Zero(h,w);
  B = MImgType::Zero(h,w);

  mH = h;
  mW = w;
}

MImg::MImg( const QImage &img )
{
  int w = img.width();
  int h = img.height();

  R = MImgType::Zero(h,w);
  G = MImgType::Zero(h,w);
  B = MImgType::Zero(h,w);

   //now copy values..
    for (int y=0; y < h; y++)
        for (int x=0; x < w; x++)
        {
            QRgb color = img.pixel(x,y);
            R(y,x) = qRed(color)/255.0;
            G(y,x) = qGreen(color)/255.0;
            B(y,x) = qBlue(color)/255.0;
        }

  return img;
}

void    MImg::maximizeIndependentDynamicRange()
{
  double  min, max;

  min = R.minCoeff(); max = R.maxCoeff();
  R = (R.cwise() - min) / (max - min);

  min = G.minCoeff(); max = G.maxCoeff();
  G = (G.cwise() - min) / (max - min);

  min = B.minCoeff(); max = B.maxCoeff();
  B = (B.cwise() - min) / (max - min);
}

unsigned int    MImg::getHeight() {
  return mH;
}

unsigned int    MImg::getWidth() {
  return mW;
}

It is important to mention that this code only handles RGB and won't care about grayscale images or any other type of colour models. The advantage of having the image in this matrix form is that Eigen provides an easy syntax for matrix manipulation, along with many modules performing least squares, Cholesky, diagonalization, etc.

Matlab and n-dimensional array sorting

Wow.. it's been such a long time since the last post!! I've been quite busy with ViBOT, specially during the last two weeks. Anyway, I thought it would be nice to write a post about some nice Matlab functions I found quite useful:

reshape:this is a nice function that lets you reshape any array or matrix into any other array or matrix with different dimensions, as long as the number of elements is kept the same. It is very useful when one wants to loop through every element of a two or three dimensional array. If A=[1 2; 3 4] then reshape(A,[1 4]) will return [1 2 3 4]. To get back to the original array we can then use reshape([1 2 3 4],[2 2]).

ind2sub: this is a useful function when manipulating arrays that were linearised with reshape. From Matlab help: "The ind2sub command determines the equivalent subscript values corresponding to a single index into an array". A simple example is the following:

• A=[1 2; 6 5]; B = reshape(A,[1 4]);
• [sV, sI] = sort(B, 'descend'); % sort linearised array
• disp(sV(1)); %show max value: 6
• disp(sI(1)); %show max index: 2
• [x,y] = ind2sub( size(A), sI(1) );
• disp(x); disp(y); % (x,y) == (2,1), we got the coordinates in the original matrix

There is also a sub2ind function which does the reverse transformation. ind2sub was very useful to ease the sorting task when applying the Hough transform for circle detection, where there is an accumulator matrix which is 3-dimensional.

FPGAs are taking over!

From the moment I knew and learned about FPGAs (Xilinx) I looked forward to use them to replace large logic circuits. This way the system would be not only scallable and the logic programmable but costs should be reduced too.

Unfortunately, most of the times the FPGA alternative was much more expensive than the equivalent logic circuit implemented with separate logic ICs.

Finally the day came and I got the chance to build a board with an ARM7 core + Spartan3A FPGA. Total price was reduced and the system became fully programmable. The ARM7 chip (LPC23xx) configures the FPGA on startup which happens to be really fast (52kib for XC3S50A). The microcontroller and FPGA are connected through a parallel bus with many control lines.

The XC3S50A is optimal in the sense that it only requires 3.3V and 1.2V supplies so it can be directly connected to the microcontroller pins.

Here there are some pictures:

Qt-Embedded: Capturing screen with QPixmap

I've been working on a product manual lately. I needed to include several LCD screenshots into it so I tried to come up with an easy way to capture snapshots from our Qt/Embedded app.

Qt/Embedded provides a nice method to save window/framebuffer contents directly to an image file. However, I wanted to send the 'take-snapshot' command from a tty console (telnet/serial/etc) since there weren't any other buttons on the system to trigger that.

A QTimer is set up. Periodically it checks the file /tmp/doCapture. If it exists a snapshot is taken and an image file is saved. Its filename is taken from the contents of /tmp/doCapture. After saving the image /tmp/doCapture is deleted.

Here is the code, which should be placed in the main window, whose width and height cover the whole screen:

mainWindow::mainWindow()
{
 // captureTimer should be declared in mainWindow's class definition
 captureTimer = new QTimer(this);
 connect( captureTimer, SIGNAL(timeout()), this, SLOT(captureTimerEvent()) );

 captureTimer->start(1000); //check interval
}

void mainWindow::captureTimerEvent()
{
 QString tmpFile = QString("/tmp/doCapture");

 if ( !QFile::exists(tmpFile) )
     return;

 QFile f(tmpFile);
 if ( !f.open( QIODevice::ReadWrite ) )
     return;

 char buf[200];
 if ( f.readLine( buf, sizeof(buf) - 4 ) == -1 )
     return;

 buf[strlen(buf)-1] = '\0'; //remove \n created by 'echo'-- not safe!

 strcat( buf, ".png" );

 //capture
 QPixmap p = QPixmap::grabWindow( this->winId() );

 if ( p.save( buf ) )
     printf("------- GRAB OK\n");
 else
     printf("------- ERR GRAB!\n");

 /* delete file */
 f.remove();
}

This way, all I have to do to take a snapshot is to write:

echo pngfilename > /tmp/doCapture

Compiling and using GDB for arm-linux

Some days ago I had to compile gdb manually in order to debug an arm-linux app. It's quite trivial but it's so useful that I thought it would be a nice idea to post the instructions here. Below there are some explanations about debugging remotely with KDevelop.

This was done with gdb-6.8, you can grab it here. It is assumed that arm-linux tools are available (PATH correctly set).

Compiling the GDB client

Decompress gdb-6.8 and compile it by issuing:

tar xvf gdb-6.8.tar.gz
cd gdb-6.8
./configure --build=x86 --host=x86 --target=arm-linux
make

After compiling just copy gdb/gdb to arm-linux-gdb where the arm-linux toolchain binaries reside (this is purely for organization and proper naming). You can now remove the gdb-6.8 directory.

NOTE: if your host computer architecture isn't intel-based just replace x86 by the correct value for your platform.

Compiling the GDB server (ARM)

Decompress gdb-6.8 and compile it by issuing:

tar xvf gdb-6.8.tar.gz
cd gdb-6.8
./configure --host=arm-linux
make

After compiling just copy gdb/gdbserver/gdbserver to your arm-linux filesystem so that you can execute it from a remote shell (gdbserver will be an arm-elf binary). You can now remove the gdb-6.8 directory.

Testing connections

First the server should be started in the  arm processor by issuing something like:

gdbserver host:1234 _executable_

Where _executable_ is the application that is going to be debugged and host:1234 tells gdbserver to listen for connections to port 1234.

To run gdb just type this in a PC shell:

arm-linux-gdb --se=_executable_

After that you'll get the gdb prompt. To connect to the target type:

target remote target_ip:1234

To resume execution type 'continue'. You can get an overview on gdb usage here.

Debugging with KDevelop

KDevelop can be used to watch variables and debug the remote target (place breakpoints, stop execution, etc). Just go to Project -> Project Options and make sure you have something like this:

monitor-debug.gdb is a file that contains the following

target remote target_ip:1234

After this you will be ready to remotely debug any arm-linux application.

_

Catching uncaught exceptions in Java

This month looks quite javaized!

So here is another thing I came up with, regarding uncaught exceptions which usually end the application totally unexpectedly. I wouldn't mind about that, but I felt that the user should be notified with a minimum-sense message and some technical info so that he/she can send it over to the developers.

So here it is a 'BodyGuard' class that does the job by using a callback present in the Thread class.

package bodyguard;

public class BodyGuard implements Thread.UncaughtExceptionHandler
{ 
 static private BodyGuard    bGuard;

 static public void  registerGuard() {
     bGuard  = new BodyGuard();
     Thread.setDefaultUncaughtExceptionHandler(bGuard);
 }

 public void    uncaughtException( Thread thread, Throwable e )
 {
     java.io.StringWriter    sW = new java.io.StringWriter();
     e.printStackTrace( new java.io.PrintWriter(sW));
  
     String  s = "A fatal error was detected during execution: \n" +
             "Thread: " + thread.getName() + "\n" +
             "Exception: " + sW.toString() + "\n" +
             "The application will be closed now.";
  
     showFatalErr(s);
 }

 private void    showFatalErr( String err )
 {
     //JOptionPane.showMessageDialog(null, err,"Error", JOptionPane.ERROR_MESSAGE);
     BodyGuardDialog dlg = new BodyGuardDialog(null, true, err);
     dlg.setLocationRelativeTo(null);
     dlg.setVisible(true);
  
     System.exit(-1);    //exit n ow
 }
}

BodyGuardDialog is another class derived from JDialog that shows the corresponding error message and lets the user copy the error to the clipboard.

In order to register this handler one just has to call bodyguard.BodyGuard.registerGuard() when starting up (ie: static void main()).

This class will catch uncaught exceptions from all threads, displaying the thread's name where the exception was thrown.

_