Resumable SFTP/SCP Transfers

Transmission errors inevitably occur during any online activity. This is specially true when tranferring large files. Murphy always prefers to poke his nose in at the worst possible time to disrupt an important file transfer.

Although HTTP and FTP transfers can be resumed without starting all over again, SFTP and SCP do not provide this capability out of the box. There are essentially two methods to obtain resumable SFTP and SCP transfers, but they are not 100% reliable. However, if you are stuck with an unreliable communication channel and a server supporting only SCP/SFTP, then it's the only ray of hope.

Method 1:
rsync supports resumable transfers across secure connections, and is the most easy to use solution. The downside is that both the server and the client must have rsync installed. If you're fortunate enough to have a server that supports rsync, the following command is all you will need.
rsync --partial --progress --rsh=ssh file_name user@host:server_path

If you use SCP on a daily basis, then it's worthwhile to add an alias to quickly start resumable transfers by editing ~/.bashrc and adding a line similar to:
alias scpr='rsync --partial --progress --rsh=ssh'
The invocation syntax will be similar to a normal scp command. For example:
scpr myfile.tar me@myremotehost.com:~/mydir/

Method 2:
cURL the multi-protocol URL grokker also provides resumable transfers with the -C flag.
curl -C -Tfile_name -u username sftp://myhost/mypath/

libcurl provided with my Fedora 9 setup is not built with libssh2 to enable SFTP and SCP. You can check whether your distribution has a libcurl that supports SCP/SFTP by running:
curl-config --protocols

I downloaded and compiled cURL from source to enable SCP and SFTP. However, I did not dare to replace the libcurl shipped with Fedora because there are over 85 programs depending on it. The downside is that my binary gets built by linking to the old libcurl which does not support SFTP and SCP ! However, I can run the curl static executable from the build directory with full SCP and SFTP support. Building a separate dynamic executable shouldn't be that difficult, but it requires wading through the Makefiles. Since I am not in a dire need for curl with SFTP and SCP support, I will defer this adventure to another time.

Fedora Sytem Restore

One of the good features of Microsoft Windows is the system restore tool which allows quick rollbacks to a previous state. When I was a Windows user, I had to make use of it extensively because the auto-updates always seemed to screw up something and make my system unusable. With Linux, that sort of problems are rare because the repositories are well managed. But sometimes things do tend to get messy after an update. I have had my fair share of updates gone awry, but they were minor infractions that didn't leave my system completely unusable. However, the time I spent on untangling the mess would have been better spent somewhere else.

Today by chance I came across a Linux Journal article about a little known feature of yum and rpm which allows rollbacks to previous states, just like Windows system restore does ! I have no idea why this feature is not enabled by default or why is it not more widely publicized, but who am I to argue with the collective inteligence of the wizards at RedHat and Fedora ?

Basically, to manually enable rollbacks of a certain install, the user has to repackage the old rpm so that it's state is saved. However, doing this manually for each and every package is cumbersome and error prone. Follow these steps to make this process automatic.

1. Edit /etc/yum.conf and add the line tsflags=repackage
2. Edit /etc/rpm/macros (You might need to create this file) and add
%_repackage_all_erasures 1
%_repackage_dir /repackaged
%_repackage_dir is not neccessary for the funcationality. But it's advisable to set it to a partition with lots of space, so that it doesn't eat out the disk space from the important data partitions.

To rollback to a previous state, simply run rpm with the -rollback option and the desired time.
rpm -Uvh -rollback '2 hour ago'

ONLamp blog lists several other variants to invoke rollback such as:
rpm -Uvh -rollback '8.00 am'
rpm -Uvh -rollback 'november 1'


Edit: After enabling repackaging on my Fedora 9 system, packagekit reported repackaging errors during an update. I am still trying to find a solution to get rid of the error messages while keeping repackaging instact. Proceed at your own risk.

Wine 1.0 and Firefox 3.0

Well.. today seems to be a big day for FOSS. Major releases of two of the best and greatest open source software were released a few hours apart on a single day !. Version 3 of Firefox, undoubtedly the best web browser ever, was released a few hours ago. (Don't forget to help the Mozilla team in their world record attempt by downloading a copy of Firefox.) Then to add icing to the cake, the Wine team have released version 1.0 of wine, which has been 15 years in the works ! This is indeed a great day for FOSS supporters everywhere.

I compiled wine 1.0 on my Fedora 9 system with minimum hassle. In fact, all I had to do was download the sources, untar them and then run the command:
./tools/wineinstall

At one point the compilation process failed throwing an error on dlls/user32/tests/menu.c. I simply deleted menu.c from the sources list of dlls/user32/tests/Makefile and everything worked perfectly !

Congratulations are in order for all Mozilla and Wine developers. Great job guys. Keep it up !

Installing Google Gadgets in Fedora 9

Google recently open-sourced the Google Gadgets framework for Linux (http://code.google.com/p/google-gadgets-for-linux/). As is most Google products, this is still in beta, so the installation experience on my Fedora 9 machine wasn't exactly smooth. But the quirks were easy to iron out.

Dependencies
The only dependency not available in the official repos is the SpiderMonkey library. Follow these steps to compile SpiderMonkey 1.7 on your machine.

Step 1:
Grab the source tarball and extract the contents.
wget http://ftp.mozilla.org/pub/mozilla.org/js/js-1.7.0.tar.gz
tar xvf js-1.7.0.tar.gz

Step 2:
By default the build options for SpiderMonkey is set to create a debug version. Since I am not planning to use SpiderMonkey for development purposes, I compiled an optimized version of the library.
cd js/src
make -f Makefile.ref BUILD_OPT=1


Step 3:
If the compilation was successful, a folder titled Linux_ALL_OPT.OBJ will be created in the src directory. To test whether everything is working fine type:
Linux_ALL_OPT.OBJ/js perfect.js
You should see an output similar to the following:


A number is 'perfect' if it is equal to the sum of its
divisors (excluding itself).

The perfect numbers up to 500 are:
6 = 1 + 2 + 3
28 = 1 + 2 + 4 + 7 + 14
496 = 1 + 2 + 4 + 8 + 16 + 31 + 62 + 124 + 248
That's all.

Step 4:
Create the following symbolic links for ease.
sudo ln -s Linux_ALL_OPT.OBJ/libjs.so /usr/lib/libjs.so
ln -s Linux_All_OPT.OBJ/jsautocfg.h
ln -s Linux_All_OPT.OBJ/jsautokw.h

SpiderMonkey is now ready for Google Gadgets.

Installing Google Gadgets
The public release of Google Gadgets threw up a lot of compilation errors at me. Most of them were trivial and I could fix them with single edits to the source files, but soon things got really ugly. Since I didn't want to spend time reading and understanding the WHOLE code base, I decided to grab the SVN source in the hopes that most of the bugs were fixed. It turns out I was right !

Step 1:
Grab the sources.
svn checkout http://google-gadgets-for-linux.googlecode.com/svn/trunk/ google-gadgets-for-linux-read-only
cd google-gadgets-for-linux-read-only/


Step 2:
Setup the build environment.
sh autotools/bootstrap.sh
cp /usr/share/automake-1.10/mkinstalldirs ./libltdl/


Step 3:
Run configure with the path to SpiderMonkey sources. I also disabled QT elements because I primarily use Gnome.
./configure --with-smjs-incdir=~/bin/spidermonkey/js/src --disable-libggadget-qt --disable-qt-system-framework --disable-qtwebkit-browser-element --disable-qt-host

Step 4:
Make and install.
make && sudo make install

Step 5:
Run GoogleGadgets !
ggl-gtk