IT Fuzzy Blog

2010-08-09

Check Out Which Files Are Double

I wrote a little script to figure out which files are double inside a directory tree. You can see how many bytes are wasted by storing data twice.


#!/usr/bin/perl -w

# Permission is hereby granted, free of charge,
# to any person obtaining a copy of this software
# and associated documentation files (the
# "Software"), to deal in the Software without
# restriction, including without limitation the
# rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of
# the Software, and to permit persons to whom
# the Software is furnished to do so, subject
# to the following conditions:
#
# This permission notice shall be included in
# all copies or substantial portions of
# the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT
# WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 
# INCLUDING BUT NOT LIMITED TO THE WARRANTIES 
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
# PURPOSE AND NONINFRINGEMENT. IN NO EVENT
# SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
# TORT OR OTHERWISE, ARISING FROM, OUT OF OR
# IN CONNECTION WITH THE SOFTWARE OR THE USE
# OR OTHER DEALINGS IN THE SOFTWARE.

use strict;
use File::Find;
use Digest::MD5;
use Digest::SHA;


# Debug
my $debug = 0;

# ARGV contains the basis directories
@ARGV = qw(.) unless @ARGV;
if(@ARGV > 1) {
  print "Base directories: @ARGV\n\n";
} else {
  print "Base directory: @ARGV\n\n";
}

# Save data in this structure
my %md5_sha_data;

# Process files
find(\&process_file, @ARGV);

# Create report
report();


sub process_file {
  my $file = $_;
  my $fullfile = $File::Find::name;
  print "Processing: $file\n" if $debug;

  my $skip = 0;

  if(-d $file) {
    $skip = 1;
  } elsif(-l $file) {
    $skip = 1;
  }

  if($skip) {
    print "skipped\n\n" if $debug;
  } else {
    open(FILE,'<',$file) or die "Error: $!\n\n";
    binmode(FILE);
    my  $md5 = Digest::MD5->new->addfile(*FILE)->hexdigest;
    my  $sha = Digest::SHA->new->addfile(*FILE)->hexdigest;
    close(FILE);
    my $size = -s $file;
    print "$md5 $sha $size\n\n" if $debug;

    save_data($fullfile, $md5, $sha, $size);
  }
}


sub save_data {
  my $file = shift;
  my $md5 = shift;
  my $sha = shift;
  my $size = shift;
  my $hash = "$md5$sha";

  if(!exists($md5_sha_data{$hash})) {
    $md5_sha_data{$hash}{"count"} = 1;
    $md5_sha_data{$hash}{"size"} = $size;
    push @{$md5_sha_data{$hash}{"filenames"}}, $file;
  } else {
    $md5_sha_data{$hash}{"count"} += 1;
    push @{$md5_sha_data{$hash}{"filenames"}}, $file;
  }
}


sub report {
  my $total = 0;
  for my $hash ( sort keys %md5_sha_data ){
    if($md5_sha_data{$hash}{"count"} > 1) {
      print "Multiple copies of equal file\n";
      foreach (@{$md5_sha_data{$hash}{"filenames"}}){
        print "  $_\n";
      }
      my $sum = $md5_sha_data{$hash}{"count"} *
                $md5_sha_data{$hash}{"size"} -
                $md5_sha_data{$hash}{"size"};
      $total += $sum;
      print "$sum Bytes\n\n";
    }
  }

  my $KBytes = $total / 1024;
  $KBytes = sprintf("%.0f", $KBytes);
  my $MBytes = $KBytes / 1024;
  $MBytes = sprintf("%.0f", $MBytes);

  print "$total Bytes ($KBytes KiB, $MBytes MiB) wasted space\n\n";
}

2010-08-03

Automatic FlightGear Scenario Downloader

FlightGear is a very nice open source flight simulator software. A major component of any flight simulator is the world scenery. The FlightGear project provides scenery files for the complete world on their FTP server. Unfortunately, the world scenery consists of more than 500 files. Each file approx 50 to 70 mega bytes in size.

A manual download of these files would require much endurance, as the server only permits four parallel downloads at the same time.

Therefore, I developed a script for automatic scenario downloading. Now, my computer can download the whole world scenery without any user interaction.

The script provides two interesting features:

It extracts a download list from the MD5SUM file that is available on the webserver. Thus, the downloader automatically detects which files are available. Later the MD5 sums are used to verify if the download was successful or not.

Furthermore, the scipt checks if files have been downloaded in the past. It skips these files to prevent downloading files twice. This way the user can easily interrupt and continue the downloading process.


#!/bin/bash

# Permission is hereby granted, free of charge,
# to any person obtaining a copy of this software
# and associated documentation files (the
# "Software"), to deal in the Software without
# restriction, including without limitation the
# rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of
# the Software, and to permit persons to whom
# the Software is furnished to do so, subject
# to the following conditions:
#
# This permission notice shall be included in
# all copies or substantial portions of
# the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT
# WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 
# INCLUDING BUT NOT LIMITED TO THE WARRANTIES 
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
# PURPOSE AND NONINFRINGEMENT. IN NO EVENT
# SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
# TORT OR OTHERWISE, ARISING FROM, OUT OF OR
# IN CONNECTION WITH THE SOFTWARE OR THE USE
# OR OTHER DEALINGS IN THE SOFTWARE.


URL="ftp://ftp.ibiblio.org/pub/mirrors/flightgear/ftp/Scenery-1.0.1/"
MD5SUM="MD5SUM"

WGET="/usr/bin/wget"
MD5="/usr/bin/md5sum"

if [ ! -f $MD5SUM ]
then
  echo "### Downloading MD5SUM file"
  wget "$URL$MD5SUM"
fi

if [ -f $MD5SUM ]
then
  FILELIST=$(cat $MD5SUM | awk '{print $2}')
  MD5LIST=$(cat $MD5SUM | awk '{print $1}')
  CNT=1;

  for FILE in $FILELIST
  do
    if [ ! -f $FILE ]
    then
      echo "### Downloading $FILE"
      wget "$URL$FILE"

      MD5ORIG=$(echo $MD5LIST | cut -d ' ' -f $CNT)
      MD5FILE=$($MD5 $FILE | awk '{print $1}')

      if [ $MD5ORIG == $MD5FILE ]
      then
        echo "###  Success"
      else
        echo "###  Error"
      fi
    else
      echo "### File $FILE exists!"
    fi

    let CNT=$CNT+1
  done


else
  echo "### ERROR: Can not find file $MD5SUM"
fi

If you are not interested in FlightGear, you maybe use this scipt to download other files based on a MD5SUM file provided on a Web or FTP site.

2009-12-31

Slow Eye of Gnome

Recently, Eye of Gnome (eog) was extremly slow during startup and when switching between images. Starting eog took around 4-8 seconds which is to much for an acceptable working performance.

Gnome Bug 500772 brought me to the following workaround: Simply remove ~/.recently-used.xbel

It seems that this XML file becomes very large and parsing will take much time. After removing this file eog starts in less than a second and also switching between images is really fast now.

2009-12-28

X-Man Brought Microsoft Flight Simulator X

Hurray, I survived X-MAS (christmas for the noobs)!!!

The X-Man (Santa Clause for the noobs) brought Microsoft Flight Simulator X. He knew that I like to play flight simulators but he forgot that I avoid to use the Windows OS. Regardless, I tryed to play it.

Installing MFSX on a Xeon 2,4 GHz system with 1 GB RAM took more than 1 hour and wasted 15 GB of my harddisk.

No Protection of Data Privacy with Microsoft Products

After the installation it is necessary to activate the game. Therefore the serial number must be entered and the game will connect to a Microsoft server and send all kind of data about your system.

Naturally, I denied sending information about MY SYSTEM! But, without activation I cannot use the game without limitations.

Finally, I decided to remove MFSX from my hard disk and to send the game back to Microsoft. Now, I will spend the money on X-Plane which works on my Linux OS and does not require activation.

ATI Radeon DRI and AGP Problem

The last weeks I have been fighting with my ATI Radeon 9700 Pro graphic card. 3D support won't work and glxgears -info listed at least 320 FPS. Too less for playing 3D games.
I was using the open source radeon driver because the closed source fglrx driver worked not at all.

I was trying all kind of options in my /etc/X11/xorg.conf but they did not change anything.
Yesterday I found the following line in my /var/log/Xorg.0.log:

(EE) RADEON(0): [agp] AGP failed to initialize. Disabling the DRI.

This information lead me to a bug listing of the Ubuntu distribution. They figured out that EDAC causes a problem with AGP.

EDAC (Error Detection and Correction) is a set of Linux kernel modules for handling hardware-related errors. Its major focus has been ECC memory error handling.

A workaround is to blacklist the EDAC module in the /etc/modprobe.d/blacklist file.

/etc/modprobe.d/blacklist:
blacklist e7xxx_edac

The exact name of the EDAC module depends on the chipset of your mainboard. So it is necessary to figure out which module relates to your board before blacklisting.

Now, 3D support works fine. And I get up to 4550 FPS with the open source radeon driver of my Lenny distribution.

I hope this bug will be fixed in future kernel releases because error detection is also a important feature and it would be fine to use it together with a working AGP bus.

2009-07-27

Debian Mirror Howto

Today I will show you how to setup your own Debian repository.

Step 1: Install the apt-mirror package

# aptitude install apt-mirror

Step 2: Edit the configuration file /etc/apt/mirror.list


set base_path    /var/spool/apt-mirror
set mirror_path  $base_path/mirror
set skel_path    $base_path/skel
set var_path     $base_path/var

set defaultarch i386
set _tilde 0

# lenny's section
deb       http://ftp.debian.org/debian lenny main contrib non-free
deb-amd64 http://ftp.debian.org/debian lenny main contrib non-free
deb-src   http://ftp.debian.org/debian lenny main contrib non-free

# Cleaning section
clean http://ftp.de.debian.org/

skip-clean http://ftp.debian.org/doc/
skip-clean http://ftp.debian.org/tools/
skip-clean http://ftp.debian.org/debian-cd/
skip-clean http://ftp.debian.org/debian-minicd/
skip-clean http://ftp.debian.org/debian/dists/lenny/main/installer-i386/
skip-clean http://ftp.debian.org/debian/doc/
skip-clean http://ftp.debian.org/debian/tools/
skip-clean http://ftp.debian.org/debian/project/

Step 3: Install a cron job for regular mirror updates

For this create the file /etc/cron.d/apt-mirror:


#
# Regular cron jobs for the apt-mirror package
#
0 5     * * *   apt-mirror      /usr/bin/apt-mirror \
  > /var/spool/apt-mirror/var/cron.log

Step 4: Add some lines to your Apache configuration


Alias /mirror-debian/ /var/www/mirror-debian/
<directory>
  AllowOverride None
  Options -MultiViews FollowSymLinks Indexes
  Order allow,deny
  Allow from all
</directory>

Step 5: Start initial download


# su - apt-mirror -c apt-mirror

Step 6: Configure your clients to use the new repository


deb     http://your.hostname/mirror-debian/debian/ lenny main contrib non-free
deb-src http://your.hostname/mirror-debian/debian/ lenny main contrib non-free

deb     http://security.debian.org/ lenny/updates main contrib non-free
deb-src http://security.debian.org/ lenny/updates main contrib non-free

Thats all!

Automatic VLC Start at System Boot Time

Last week we installed a machine with VLC for centralized music playback in our room. Now, everybody can start or stop music streams via the build-in HTTP interface of VLC.

To start VLC automatically at boot time we wrote the following start script:

#!/bin/sh
# Start/stop the vlc daemon.
#

PLAYLIST="/usr/share/vlc/playlist.pls"

case "$1" in
start)
       PROC=$(ps -ef|grep "vlc -I http"|grep -v grep|wc -l)
       if [ $PROC -eq 0 ]
       then
         sudo -u vlc /usr/bin/vlc -I http --http-host 0.0.0.0:8000 $PLAYLIST 1>/dev/null 2>&1 &
         echo "VLC started"
       else
         echo "There is still a vlc process running..."
         exit 1
       fi
       ;;

stop)
       PROC=$(ps -ef|grep "vlc -I http"|grep -v grep|awk '{print $2}')
       for TMP in $PROC
       do
         logger "Killed VLC - PID $TMP"
         kill -9 $TMP 1>/dev/null 2>&1
       done
       echo VLC killed
       ;;

restart)
       $0 stop
       $0 start
       ;;

reload|force-reload)
       $0 stop
       $0 start
       ;;
*)      log_action_msg "Usage: /etc/init.d/vlc {start|stop|restart|reload|force-reload}"
       exit 2
       ;;
esac
exit 0

For security reasons we run VLC as a non-root user. To change the user id we are using sudo inside the start script.