• 2010-08-21 13:38:00

    Create an inexpensive hourly remote backup

    There are two kinds of people, those who backup regularly, and those that never had a hard drive fail

    As you can tell the above is my favorite quote. It is so true and I believe everyone should evaluate how much their data (emails, documents, files) is worth to them and, based on that value, create a backup strategy that suits them. I know for sure that if I ever lost the pictures and videos of my family I would be devastated since those are irreplaceable.

    So the question is how can I have an inexpensive backup solution? All my documents and emails are stored in Google, since my domain is on Google Apps. What happens to the live/development servers though that host all my work? I program on a daily basis and the code has to be backed up regularly so as to avoid any hard drive failures and thus result in loss of time and money.

    So here is my solution. I have an old computer (IBM Thincentre) which I decided to beef up a bit. I bought 4Gb of RAM from eBay for less than $100 for it. Although this is was not necessary since my solution would be based on Linux (Gentoo in particular), I wanted to have faster compilation times for packages.

    I bought two external drives (750Gb and 500Gb respectively) and one 750Gb internal drive. I already have a 120Gb hard drive in the computer. The two external ones are connected to the computer using USB while the internal ones are connected using SATA.

    The external drives are formatted using NTFS while the whole computer is built using ReiserFS.

    Here is the approach:

    • I have installed and have a working Gentoo installation on the machine
    • I have an active Internet connection
    • I have installed LVM on the machine and set up the core system on the 120Mb drive while the 500Mb is on LVM
    • I have 300Mb active on the LVM (from the available 500Mb)
    • I have generated a public SSH key (I will need this to exchange it with the target servers)
    • I have mounted the internal 500Mb drive to the /storage folder
    • I have mounted the external USB 750Mb drive to the /backup_hourly folder
    • I have mounted the external USB 500Mb drive to the /backup_daily folder

    Here is how my backup works:

    Every hour a script runs. The script uses rsync to syncrhonize files and folders from a remote server locally. Those files and folders are kept in relevant server named subfolders in the /storage folder (remember this is my LVM). So for instance my subfolders will be /storage/beryllium.niden.net, /storage/nitrogen.niden.net, /storage/argon.niden.net etc.

    Once the rsync completes, the script continues by compressing the relevant 'server' folder and creates the compressed file with a date-time stamp on its name.

    When all compressions are completed, if the time that the script has executed is midnight, the backups are moved from the /storage folder to the /backup_daily folder (which has the external USB 500Gb mounted). If it is any other time, the files are moved in the /backup_hourly folder (which has the external USB 750Gb mounted).

    This way I ensure that I keep a lot of backups (daily and hourly ones). The backups are being recycled, so older ones get deleted. The amount of data that you need to archive as well as the storage space you have available dictate how far back you can go in your hourly and daily cycles.

    So let's get down to business. The script itself:

    #!/bin/bash
    DATE=`date +%Y-%m-%d-%H-%M`
    DATE2=`date +%Y-%m-%d`
    DATEBACK_HOUR=`date --date='6 days ago' +%Y-%m-%d`
    DATEBACK_DAY=`date --date='60 days ago' +%Y-%m-%d`
    FLAGS="--archive --verbose --numeric-ids --delete --rsh='ssh'"
    BACKUP_DRIVE="/storage"
    DAY_USB_DRIVE="/backup_daily"
    HOUR_USB_DRIVE="/backup_hourly"
    

    These are some variables that I need for the script to work. The DATE and DATE2 are used to date/time stamp the backups, while the DATEBACK_* are used to clear previous backups. In this case it is set to 6 days ago (for my system). It can be set to whatever you want provided that you do not run out of space.

    The FLAGS variable keeps the rsync command options while the BACKUP_DRIVE, DAY_USB_DRIVE and HOUR_USB_DRIVE hold the locations of the rsync folders, daily backup and hourly backup sorage areas.

    The script works with arrays. I have 4 arrays to do the work and the 3 of them must have exactly the same elements.

    # RSync Information
    rsync_info[1]="beryllium.niden.net html rsync"
    rsync_info[2]="beryllium.niden.net db rsync"
    rsync_info[3]="nitrogen.niden.net html rsync"
    rsync_info[4]="nitrogen.niden.net html db"
    rsync_info[5]="nitrogen.niden.net html svn"
    rsync_info[6]="argon.niden.net html rsync"
    

    This is the first array which holds descriptions to what needs to be done as far as source is concerned. These descriptions get appended to the log and helps me identify what step I am in.

    # RSync Source Folders
    rsync_source[1]="beryllium.niden.net:/var/www/localhost/htdocs/"
    rsync_source[2]="beryllium.niden.net:/niden_backup/db/"
    rsync_source[3]="nitrogen.niden.net:/var/www/localhost/htdocs/"
    rsync_source[4]="nitrogen.niden.net:/niden_backup/db"
    rsync_source[5]="nitrogen.niden.net:/niden_backup/svn"
    rsync_source[6]="argon.niden.net:/var/www/localhost/htdocs/"
    

    This array holds the source host and folder. Remember that I have already exchanged SSH keys with each server, therefore when the script runs there is a direct connection to the source server. If you need to keep things a bit more secure for you, then you will need to alter the contents of the rsync_source array so that it reflects the user that you log in with as well as the password.

    # RSync Target Folders
    rsync_target[1]="beryllium.niden.net/html/"
    rsync_target[2]="beryllium.niden.net/db/"
    rsync_target[3]="nitrogen.niden.net/html/"
    rsync_target[4]="nitrogen.niden.net/db/"
    rsync_target[5]="nitrogen.niden.net/svn/"
    rsync_target[6]="argon.niden.net/html/"
    

    This array holds the target locations for the rsync. These folders exist in my case under the /storage subfolder.

    # GZip target files
    servers[1]="beryllium.niden.net"
    servers[2]="nitrogen.niden.net"
    servers[3]="argon.niden.net"
    

    This array holds the names of the folders to be archived. These are the folders directly under the /storage folder and I am also using this array for the prefix of the compressed files. The suffix of the compressed files is a date/time stamp.

    Here is how the script evolves:

    echo "BACKUP START" >> $BACKUP_DRIVE/logs/$DATE.log
    date >> $BACKUP_DRIVE/logs/$DATE.log
    
    echo "BACKUP START" >> $BACKUP_DRIVE/logs/$DATE.log
    date >> $BACKUP_DRIVE/logs/$DATE.log
    
    # Loop through the RSync process
    element_count=${#rsync_info[@]}
    let "element_count = $element_count + 1"
    index=1
    while [ "$index" -lt "$element_count" ]
    do
        echo ${rsync_info[$index]} > $BACKUP_DRIVE/logs/$DATE.log
        rsync $FLAGS ${rsync_source[$index]} $BACKUP_DRIVE/${rsync_target[$index]} > $BACKUP_DRIVE/logs/$DATE.log
        let "index = $index + 1"
    done
    

    The snippet above loops through the rsync_info array and prints out the information in the log file. Right after that it uses the rsync_source and rsync_target arrays (as well as the FLAGS variable) to rsync the contents of the source server with the local folder. Remember that all three arrays have to be identical in size (rsync_info, rsync_source, rsync_target).

    The next thing to do is zip the data (I loop through the servers array)

    # Looping to GZip data
    element_count=${#servers[@]}
    let "element_count = $element_count + 1"
    index=1
    while [ "$index" -lt "$element_count" ]
    do
        echo "GZip ${servers[$index]}" > $BACKUP_DRIVE/logs/$DATE.log
        tar cvfz $BACKUP_DRIVE/${servers[$index]}-$DATE.tgz $BACKUP_DRIVE/${servers[$index]} > $BACKUP_DRIVE/logs/$DATE.log
        let "index = $index + 1"
    done
    

    The compression method I use is tar/gzip. I found it to be fast with a good compression ratio. You can choose anything you like.

    Now I need to delete old files from the drives and copy the files on those drives. I use the servers array again.

    # Looping to copy the produced files (if applicable) to the daily drive
    element_count=${#servers[@]}
    let "element_count = $element_count + 1"
    index=1
    
    while [ "$index" -lt "$element_count" ]
    do
        # Copy the midnight files
        echo "Removing old daily midnight files" > $BACKUP_DRIVE/logs/$DATE.log
        rm -f $DAY_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK_DAY*.* > $BACKUP_DRIVE/logs/$DATE.log
        echo "Copying daily midnight files" > $BACKUP_DRIVE/logs/$DATE.log
        cp -v $BACKUP_DRIVE/${servers[$index]}-$DATE2-00-*.tgz $DAY_USB_DRIVE/${servers[$index]} &nbsp>>; $BACKUP_DRIVE/logs/$DATE.log
        rm -f $BACKUP_DRIVE/${servers[$index]}-$DATE2-00-*.tgz > $BACKUP_DRIVE/logs/$DATE.log
    
        # Now copy the files in the hourly
        echo "Removing old hourly files" > $BACKUP_DRIVE/logs/$DATE.log
        rm -f $HOUR_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK_HOUR*.* > $BACKUP_DRIVE/logs/$DATE.log
        echo "Copying daily midnight files" > $BACKUP_DRIVE/logs/$DATE.log
        cp -v $BACKUP_DRIVE/${servers[$index]}-$DATE.tgz $HOUR_USB_DRIVE/${servers[$index]} > $BACKUP_DRIVE/logs/$DATE.log
        rm -f $HOUR_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK*.* > $BACKUP_DRIVE/logs/$DATE.log
        let "index = $index + 1"
    done
    
    echo "BACKUP END" >> $BACKUP_DRIVE/logs/$DATE.log
    

    The last part of the script loops through the servers array and:

    • Deletes the old files (recycling of space) from the daily backup drive (/storage/backup_daily) according to the DATEBACK_DAY variable. If the files are not found a warning will appear in the log.
    • Copies the daily midnight file to the daily drive (if the file does not exist it will simply echo a warning in the log - I do not worry about warnings of this kind in the log file and was too lazy to use an IF EXISTS condition)
    • Removes the daily midnight file from the /storage drive.

    The reason I am using copy and then remove instead of the move (mv) command is that I have found this method to be faster.

    Finally the same thing happens with the hourly files

    • Old files are removed (DATEBACK_HOUR variable)
    • Hourly file gets copied to the /backup_hourly drive
    • Hourly file gets deleted from the /storage drive

    All I need now is to add the script in my crontab and let it run every hour.

    NOTE: The first time you will run the script you will need to do it manually (not in a cron job). The reason behind it is that the first time rsync will need to download all the contents of the source servers/folders in the /storage drive so as to create an exact mirror. Once that lengthy step is done, the script can be added in the crontab. Subsequent runs of the script will download only the changed/deleted files.

    This method can be very effective while not using a ton of bandwidth every hour. I have used this method for the best part of a year now and it has saved me a couple of times.

    The last thing I need to present you is the backup script that I have for my databases. As you notice above the source folder of beryllium.niden.net as far as databases are concerned is beryllium.niden.net/db/. What I do is I dump and zip the databases every hour on my servers. Although this is not a very efficient way of doing things and it adds to the bandwidth consumption every hour (since the dump will create a new file every hour) I have the following script running on my database servers every hour at the 45th minute:

    #!/bin/bash
    
    DBUSER=mydbuser
    DBPASS='dbpassword'
    DBHOST=localhost
    BACKUPFOLDER="/niden_backup"
    DBNAMES="`mysql --user=$DBUSER --password=$DBPASS --host=$DBHOST --batch --skip-column-names -e "show databases"| sed 's/ /%/g'`"
    OPTIONS="--quote-names --opt --compress "
    
    # Clear the backu folder
    rm -fR $BACKUPFOLDER/db/*.*
    
    for i in $DBNAMES; do
        echo Dumping Database: $i
        mysqldump --user=$DBUSER --password=$DBPASS --host=$DBHOST $OPTIONS $i > $BACKUPFOLDER/db/$i.sql
        tar cvfz $BACKUPFOLDER/db/$i.tqz $BACKUPFOLDER/db/$i.sql
        rm -f $BACKUPFOLDER/db/$i.sql
    done
    

    That's it.

    The backup script can be found in my GitHub here.

    Update: The metric units for the drives were GB not MB. Thanks to Jani Hartikainen for pointing it out.

  • 2009-12-10 12:00:00

    Faster rsync and emege in Gentoo

    Scenario

    Recently I have started setting up a cluster of 7 Gentoo boxes for a project I am working on. The problem with boxes coming right out of the setup process of a hosting company is that they do not contain the packages that you need. Therefore you need to setup your USE flags and emerge the packages you require as per the role of every box.

    I have implemented the following procedure many times in my local networks (since I have more than one Gentoo boxes) and have also implemented the same process at work (we run 3 Gentoo boxes).

    The way to speed up rsync and emerge is to run a local rsync mirror and to use http-replicator. This will not make the packages compile faster but what it will do is reduce the resource usage (downloads in particular) of your network since each package will be downloaded only one time and reduce the time you have to wait for each package to be downloaded. The same applies with the rsync.

    My network has as I said 7 boxes. 5 of them are going to be used as web servers so effectively they have the same USE flags and 2 as database servers. For the purposes of this tutorial I will name the web servers ws1, ws2, ws3, ws4, ws5 and the database servers db1, db2. The ws1 box will be used as the local rsync mirror and will run http-replicator.

    I am going to set up the /etc/hosts file on each machine so that the local network is resolved in each box and no hits to the DNS are required. So for my network I have:

    10.13.18.101  ws1
    10.13.18.102  ws2
    10.13.18.103  ws3
    10.13.18.104  ws4
    10.13.18.105  ws5
    10.13.18.201  db1
    10.13.18.202  db2
    

    Modify the above to your specific setup needs.

    Setting up a local rsync

    Server setup (ws1)

    There is a really good tutorial can be found in the Gentoo Documentation but here is the short version:

    The ws1 box already has the rsync package in there. All I need to do is start the daemon. Some configuration is necessary before I start the service:

    nano -w /etc/rsyncd.conf
    

    and what I should have in there is:

    # Restrict the number of connections
    max connections = 5
    # Important!! Always use chroot
    use chroot = yes
    # Just in case you are allowed only read only access
    read only = yes
    # The user has no privileges
    uid = nobody
    gid = nobody
    # Recommended: Restrict via IP (subnets or just IP addresses)
    hosts allow = 10.13.18.0/24
    # Everyone else denied
    hosts deny  = *
    
    # The local portage
    [niden-gentoo-portage]
    path = /usr/portage
    comment = niden.net Gentoo Portage tree
    exclude = /distfiles /packages
    

    That's it. Now I add the service to the default runlevel and start the service

    rc-update add rsyncd default
    /etc/init.d/rsyncd start
    

    NOTE: If you have a firewall using iptables, you will need to add the following rule:

    # RSYNC
    -A INPUT --protocol tcp --source 10.13.18.0/24 --match state --state NEW --destination-port 873 --jump ACCEPT
    
    Client setup

    In my clients I need to edit the /etc/make.conf file and change the SYNC directive to:

    SYNC="rsync://ws1/niden-gentoo-portage"
    

    or I can use the IP address:

    SYNC="rsync://10.13.18.101/niden-gentoo-portage"
    

    Note that the path used in the SYNC command is what I have specified as a section in the rsyncd.conf file (niden-gentoo-portage in my setup). This path can be anything you like.

    Testing

    I have already run

    emerge --sync
    

    in the ws1 box, so all I need to do now is run it on my clients. Once I run it I can see the following (at the top of the listing):

    emerge --sync
    >>> Starting rsync with rsync://10.13.18.101/niden-gentoo-portage...
    receiving incremental file list
    ......
    

    So everything works as I expect it.

    Setting up http-replicator

    http-replicator is a proxy server. When a machine (the local or a remote) requests a package, http-replicator checks its cache and if the file is there, it passes it to the requesting machine. If the file doesn't exist though, http-replicator downloads it from a mirror and then passes it to the requesting machine. The file is then kept in http-replicator's cache for future requests. This way I save on resources by downloading once and serving many times locally.

    Although this might not seem as a 'pure speedup' it will make your installations and updates faster since the download factor will be reduced to a bare minimum. Waiting for packages like mysql, Gnome or others to be downloaded does take a long time. Multiply that time with the number of machines you have on your network and you can see the benefits of having a setup like this.

    Server setup (ws1)

    First of all I need to emerge the package

    emerge http-replicator
    

    Once everything is done I need to change the configuration file to suit my needs:

    nano -w /etc/conf.d/http-replicator
    

    and the file should have:

    GENERAL_OPTS="--dir /var/cache/http-replicator"
    GENERAL_OPTS="$GENERAL_OPTS --user portage"
    DAEMON_OPTS="$GENERAL_OPTS"
    DAEMON_OPTS="$DAEMON_OPTS --alias /usr/portage/packages/All:All"
    DAEMON_OPTS="$DAEMON_OPTS --log /var/log/http-replicator.log"
    DAEMON_OPTS="$DAEMON_OPTS --ip 10.13.18.*"
    ## The proxy port on which the server listens for http requests:
    DAEMON_OPTS="$DAEMON_OPTS --port 8080"
    

    The last line with the --port parameter specifies the port that the http-replicator will listen to. You can change it to whatever you want. Also the --ip parameter restricts who is allowed to connect to this proxy server. I have allowed my whole internal network; change it to suit your needs. Lastly the --dir option is where the cached data is stored. You can change it to whatever you like. I have left it to what it is. Therefore I need to create that folder:

    mkdir /var/cache/http-replicator
    

    Since I have specified that the user that this proxy will run as is portage (see --user directive above) I need to change the owner of my cache folder:

    chown portage:portage /var/cache/http-replicator
    

    I add the service to the default runlevel and start the service

    rc-update add http-replicator default
    /etc/init.d/http-replicator start
    

    NOTE: If you have a firewall using iptables, you will need to add the following rule:

    # HTTP-REPLICATOR
    -A INPUT --protocol tcp --source 10.13.18.0/24 --match state --state NEW --destination-port 8080 --jump ACCEPT
    

    You will need also to regularly run

    repcacheman
    

    and

    rm -rf /usr/portage/distfiles/*
    

    to clear the distfiles folder. I have added those in a bash script and I run it every night using my cron.

    Client setup

    In my clients I need to edit the /etc/make.conf and change the SYNC directive to:

    http_proxy="http://ws1:8080"
    RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp  \${URI} -O \${DISTDIR}/\${FILE}"</pre>
    

    I have commented any previous RESUMECOMMAND statements.

    Testing

    The testing begins in one of the clients (you can choose any package):

    emerge logrotate
    

    and see in the output that everything works fine

    ws2 ~ # emerge logrotate
    Calculating dependencies... done!
    
    >>> Verifying ebuild manifests
    
    >>> Emerging (1 of 1) app-admin/logrotate-3.7.8
    >>> Downloading 'http://distfiles.gentoo.org/distfiles/logrotate-3.7.8.tar.gz'
    --2009-12-10 06:46:47--  http://distfiles.gentoo.org/distfiles/logrotate-3.7.8.tar.gz
    Resolving ws1... 10.13.18.101
    Connecting to ws1|10.13.18.101|:8080... connected.
    Proxy request sent, awaiting response... 200 OK
    Length: 43246 (42K)
    Saving to: `/usr/portage/distfiles/logrotate-3.7.8.tar.gz'
    
    100%[=============================>] 43,246      --.-K/s   in 0s
    
    2009-12-10 06:46:47 (89.6 MB/s) - `/usr/portage/distfiles/logrotate-3.7.8.tar.gz' saved [43246/43246]
    .....
    

    Final thoughts

    Setting up local proxies allows your network to be as efficient as possible. It does not only reduce the download time for your updates but it is also courteous to the Gentoo community. Since mirrors are run by volunteers or non-profit organizations, it is only fair to not abuse the resources by downloading an update more than once for your network.

    I hope this quick guide will help you and your network :)

  • 2009-11-24 14:00:00

    Chromium OS - Part III

    Continued from Part II

    I had to shut the computer down because I had a train to catch. However booting up again the computer on the train and trying to enter the chroot environment produced some errors. It might very well be the fact that I do not have an Internet connection (at least a reliable one).

    So this post will have to wait until I get back home so that I can continue the installation. The key point here will be how to add the Chromium OS image over the second partition of my hard drive so that I can keep my dual boot system. I will run Image for DOS and clone my hard drive, in an effort to keep a mirror backup of my notebook at this particular point in time - you never know when you will use it.

    So I booted again into Ubuntu and started the process. I run

    ./create_chroot.sh
    

    but it complained that a chroot already exists. I then run

    ./create_chroot.sh --delete
    

    which effectively removed the existing chroot and replaced it with a new one. Little did I know that I only had to run

    ./enter_chroot.sh
    

    to re-enter my previous chroot and continue where I left off. Oh well, you live and learn :)

    Just to be on the safe side I rerun the ./build_platform_packages.sh, ./build_kernel.sh and ./build_image.sh scripts. I am now exactly where I was prior to shutting the computer off. I have built the platform packages, the kernel and the image.

    Using the image

    Check the contents of the image

    I will mount the image locally to ensure that everything is OK. Please note that the folder below is the one created on my machine and might very well be different than yours. At the end of the build_image.sh script you will see a message that will reveal where your image folder is.

    cd ~/trunk/src/build/images/999.999.32909.021312-a1/
    sudo mount -o loop rootfs.image rootfs
    sudo chroot rootfs
    

    Inside the image basic commands will reveal success or failure:

    df
    dpkg -l
    

    Once everything is OK (or at least seems OK) I exit and unmount the image.

    exit
    sudo umount rootfs
    

    Although I got a cannot read table of mounted file systems: No such file or directory when I run df, dpkg had a long list of packages installed. I will for the moment ignore the df output and proceed to the next steps.

    Copy the image to a USB key

    Somehow I have misplaced my 16GB USB drive so I had to borrow a 4GB one from a friend of mine. This step copies the actual image from the hard drive to the USB drive. The drive itself is wiped clean so make sure that you have backed up the data that you have on it prior to running this step.

    You need to find out the device that your USB drive corresponds to. Running:

    sudo fdisk -l
    

    will reveal which device is the USB drive. For my system it is /dev/sdc1. Outside the chroot you run the script image_to_usb.sh. The command is:

    ./image_to_usb.sh --from=~/chromiumos/src/build/images/SUBDIR --to=/dev/USBKEYDEV
    

    and for my system the command was:

    ./image_to_usb.sh --from=~/chromiumos/src/build/images/999.999.32909.021312-a1/ --to=/dev/sdc1
    

    The output on the screen running the above command is:

    [email protected]:~/chromiumos/src/scripts$ ./image_to_usb.sh --from=~/chromiumos/src/build/images/999.999.32909.021312-a1/ --to=/dev/sdc1
    Copying USB image /usr/local/chromiumos/chromiumos.git/src/build/images/999.999.32909.021312-a1 to device /dev/sdc1...
    This will erase all data on this device:
    Disk /dev/sdc1: 4013 MB, 4013917184 bytes
    Are you sure (y/N)? y
    attempting to unmount any mounts on the USB device
    Copying root fs...
    opening for read /usr/local/chromiumos/chromiumos.git/src/build/images/999.999.32909.021312-a1/rootfs.image
    opening for write /dev/sdc1
    seeking to 1992294912 bytes in output file
    copy 996147200 bytes took 102.384 s
    speed: 9.7 MB/s
    Creating stateful partition...
    mke2fs 1.41.9 (22-Aug-2009)
    Filesystem label=C-STATE
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    60800 inodes, 243200 blocks
    12160 blocks (5.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=251658240
    8 block groups
    32768 blocks per group, 32768 fragments per group
    7600 inodes per group
    Superblock backups stored on blocks:
     32768, 98304, 163840, 229376
    
    Writing inode tables: done
    Creating journal (4096 blocks): done
    Writing superblocks and filesystem accounting information: done
    
    This filesystem will be automatically checked every 30 mounts or
    180 days, whichever comes first.  Use tune2fs -c or -i to override.
    Copying MBR...
    opening for read /usr/local/chromiumos/chromiumos.git/src/build/images/999.999.32909.021312-a1/mbr.image
    opening for write /dev/sdc1
    copy 512 bytes took 0.001 s
    speed: 0.0 MB/s
    Done.
    

    I have booted the system using the USB drive and I logged in the system using my google account. Using Ctrl+Alt+T I open a terminal and enter:

    /usr/sbin/chromeos-install
    

    This asks now for the password that I have set up earlier (the one stored in the text file) and then it nukes the hard drive replacing everything (make sure you backup your hard drive).

    A bit later I unpluged the USB drive and rebooted. Unfortunately things did not work very well but that is probably due to my particular hardware. I will retry this on my other notebook and update this blog post.

  • 2009-11-24 13:00:00

    Chromium OS - Part II

    Continued from Part I

    The download took quite a while, so I thought it might be a good idea to split this post in parts, so as to ensure good readability.

    I need to create some symlinks. Also a good place to add my repository is /usr/local hence the commands for Chromium OS and Chromium respectively.

    sudo mv chromiumos/ /usr/local/
    sudo mv chromium/ /usr/local/
    

    and now adding the symbolic links

    ln -s /usr/local/chromiumos/chromiumos.git ~/chromiumos
    ln -s /usr/local/chromium ~/chromium
    

    Creating the local repository

    All the scripts are in the src/script folder. So let's go to that folder (the symbolic link set earlier helps :))

    cd ~/chromiumos/src/scripts
    

    and running the command to create the local repository:

    ./make_local_repo.sh
    

    This command will ask you for your password - and bare in mind you must run all this as a normal user with sudo access - and then it will create a debootstrap. It will fetch the necessary packages from the Chromium OS Package Management.

    NOTE

    If something fails you will need to do

    rm -rf ~/chromiumos/repo
    

    and then rerun the ./make_local_repo.sh script again.

    Creating the build environment

    All we need to do is run the following command:

    ./make_chroot.sh
    

    The script will check if all the dependencies are satisfied, and if something is missing it will pull the necessary files and compile them as necessary. Although I did not encounter any problems, the documentation states that the /etc/apt/sources.list is used for retrieving the packages. If your setup is pulling packages from somewhere else then you may need to get the most recent packages from the repository. You can do that by running:

    ./make_chroot.sh --mirror=http://build.chromium.org/buildbot/packages --suite=chromeos_dev
    

    Building Chromium OS

    I need to build Chromium first (since I chose to download it too). This is necessary since your build will fail if you try it the other way around :)

    ./build_chrome.sh --chrome_dir ~/chromium
    
    Enter the chroot build environment

    Run the following command gets us back in the chroot environment (you will be asked for your password)

    ./enter_chroot.sh
    
    Set the shared user password

    This is a one-off step for those of us that want to be able to sudo from a terminal. I am setting the shared user password running the following script:

    ./set_shared_user_password.sh
    

    This will prompt for the password and the output will be stored in the ./shared_user_password.txt file. Don't worry the password is encrypted so if you do not have anyone watching over your shoulder while typing your password you are OK. Just to be on the safe side, clear the screen.

    clear
    
    Build the platform packages

    In the chroot environment run

    ./build_platform_packages.sh
    

    Unfortunately I hit a snag :( The build_platform_packages script produced an error:

    Checking for latest build of Chrome
    Downloading http://chrome-web/buildbot/snapshots/chromium-rel-linux-chromiumos/LATEST
    --2009-11-24 19:44:49--  http://chrome-web/buildbot/snapshots/chromium-rel-linux-chromiumos/LATEST
    Resolving chrome-web... failed: Name or service not known.
    wget: unable to resolve host address `chrome-web'
    make: *** [build-stamp] Error 1
    

    I quickly found what I need to do (Google is your friend :)). It appears that this is known bug and it is easily fixable. All I had to do is edit the copy_chrome_zip.sh file. I tried using nano in the chroot environment but it was not there. For that I exited the chroot and edited the file.

    exit
    nano -w ~/chromiumos/src/platform/chrome/copy_chrome_zip.sh
    

    Locate the line with the BASE_URL variable and change chrome-web to build.chromium.org and save the file. After that enter again the chroot and rerun the build_platform_packages.sh script.

    ./enter_chroot.sh
    ./build_platform_packages.sh</pre>
    

    Quite a bit later the script execution ended with All packages built :)

    Build the kernel

    In the chroot environment run

    ./build_kernel.sh
    

    A bit later I am looking at this message and grinning :)

    Kernel build successful, check /home/ndimopoulos/trunk/src/build/x86/local_packages/linux-image-2.6.30-chromeos-intel-menlow_002_i386.deb
    
    Build the image

    In the chroot environment run

    ./build_image.sh
    

    The script starts with validations, configurations, unpacking and compilations - all too fast for my eye to capture.

    The script finished compiling and there are warnings and errors :(. They all have to do with the disk geometry and partition 1 extends past the end of the disk /shrug again....

    In the end I get this on the screen which hopefully is OK...

    Re-reading the partition table ...
    BLKRRPART: Inappropriate ioctl for device
    
    If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
    to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
    (See fdisk(8).)
    Done.  Image created in /home/ndimopoulos/trunk/src/build/images/999.999.32809.203441-a1
    To copy to USB keyfob, outside the chroot, do something like:
      ./image_to_usb.sh --from=/usr/local/chromiumos/chromiumos.git/src/build/images/999.999.32809.203441-a1 --to=/dev/sdb
    To convert to VMWare image, outside the chroot, do something like:
      ./image_to_vmware.sh --from=/usr/local/chromiumos/chromiumos.git/src/build/images/999.999.32809.203441-a1
    

    Continued in Part III

  • 2009-11-24 12:00:00

    Chromium OS Part I

    A lot of hype has been generated on the Internet a few months back regarding Google's announcement that they are building a new operating system. The announcement was met with skepticism but also enthusiasm/anticipation by a lot of people who are puzzled as to what direction Google is taking and where are they looking themselves positioned in the computer industry.

    Google has already established themselves in the mobile operating system with Android which for many is better than Apple's iPhone. What managed to get Google very high up in user satisfaction with Android was the fact that it is open source. It is backed by Google and they are putting a lot of effort into this but the fact that anyone can download the source code and build the operating system themselves is amazing. Giving people the freedom to choose should be something that every company should do (personal opinion).

    On Friday I watched the Chromium OS Webcast. A lot of people have been waiting for a presentation from Google of the new operating system. Google provided just that but with a small twist. The presenters outlined the features of the new operating system: Fast, Fast, Fast.

    Although the presenters clearly stated that they will not provide any links to hardware, what is supported, where Chromium OS runs etc. they made sure to address one of the core features of Chromium OS. It is Open Sourced! That alone gives people again freedom. Freedom to choose what they want to have as their operating system in their computer.

    The website for the new operating system is located here and there are instructions on how you can build this new OS even in its current state - which is not production ready.

    Curious (as usual) I tried installing the Chromium OS on a virtual machine. My experience installing the new OS and other comments is outlined below:

    Prerequisites

    My DELL Studio 17 has two hard drives and because I really really do not like Windows Vista, I have installed Ubuntu 9.10 32bit on the second partition. The notebook enjoys a 2.4GHz Intel processor and 6GB RAM.

    I applied all the relevant updates (just to be on the safe side) and a couple of reboots later I am ready to venture in the unknown. The documentation outlines the minimum required packages: 

    Building on Linux requires the following software.

    • Python >= 2.4
    • Perl >= 5.x
    • gcc/g++ >= 4.2
    • g++-multilib >=4.2
    • bison >= 2.3
    • flex >= 2.5.34
    • gperf >= 3.0.4
    • pkg-config >= 0.20
    • libnss3-dev >= 3.12
    • libasound2-dev
    • libgconf2-dev
    • libglib2.0-dev
    • libgtk2.0-dev
    • libnspr4-0d >= 4.7.1+1.9-0ubuntu0.8.04.5 (ubuntu0.8.04.1 causes duplicate dtoa references)
    • libnspr4-dev >= 4.7.1+1.9-0ubuntu0.8.04.5
    • msttcorefonts (Microsoft fonts)
    • freetype-dev
    • libcairo2-dev
    • libdbus-1-dev

    Optional (currently, all of these are only used by layout tests):

    • wdiff
    • lighttpd
    • php5-cgi
    • sun-java6-fonts (needed for Lucida)

    Because I didn't want to go and check the version of every package mentioned above I run the following command (mentioned also in the documentation)

    sudo apt-get install subversion pkg-config python perl g++ g++-multilib bison flex gperf libnss3-dev libgtk2.0-dev libnspr4-0d libasound2-dev libnspr4-dev msttcorefonts libgconf2-dev libcairo2-dev libdbus-1-dev
    

    And then the optional extras:

    sudo apt-get install wdiff lighttpd php5-cgi sun-java6-fonts
    

    A few minutes later the necessary packages had been downloaded and installed. One step closer to my goal :)

    Getting the code

    Now that all the prerequisites have been satisfied. I need to get the code!

    Navigating to this link in the chromium.org wiki, I get all the instructions on how to get the code. There are some prerequisites there (i.e. your system needs to be able to uncompress tar files) but nothing out of the ordinary.

    I installed the depot_tools which was really a two step process - svn checkout the tools and add the tools path in the current path. After that I installed the git-core:

    sudo apt-get install git-core
    

    I will pull the source code from the SVN repository. I can just as easy download the tarball and unzip it. The instructions in the chromium.org wiki explain both options.

    I am ready to get the source code for the Chromium OS package. You can get the code with the Chromium browser or without it. I am opting to get it with the Chromium browser. The following commands retrieve the necessary files for the OS as well as dependencies for the browser:

    mkdir ~/chroomiumos
    cd ~/chromiumos
    gclient config http://src.chromium.org/git/chromiumos.git
    gclient sync --deps="unix,chromeos" --force
    

    The download takes a bit of time since there are a lot of files that we need to retrieve. In situations like these, browsing, reading a book, going to get something to eat or working on the other computer are some of the activities you can engage yourself in so as to kill time. That is of course if you do not have a T1 or FIOS at which point this step will be finished by the time you read this sentence :) (/sigh I miss FIOS).

    I open another terminal window in order to retrieve the Chromium source code (the browser now).

    mkdir ~/chromium
    cd ~/chromium
    gclient config http://src.chromium.org/svn/trunk/src
    gclient sync
    

    and the wait continues....

    Continued in Part II

  • 2009-11-04 12:00:00

    Gentoo Stage 1 Installation

    This is my effort to install Gentoo Linux on my Acer Ferrari LMi 3000.

    The first thing I did was to start ssh. The live CD of Gentoo had already identified my network card which is based on the via-rhine module.

    # /etc/init.d/sshd start
    

    The key is generated and of course I needed to change the password to something known (it is scrambled for security reasons by the Live CD)

    # passwd
    

    Following that I left the notebook where it is and invoked PuTTY from my Windows box to access it. I printed the Gentoo Handbook guide just in case and had it near by as a reference.

    I decided to check the hard drive performance. I used

    # hdparm -tT /dev/hda
    

    and it reported:

    /dev/hda:
    Timing cached reads: 1068 MB in 2.00 seconds = 533.81 MB/sec 
    Timing buffered disk reads: 80 MB in 3.05 seconds = 26.22 MB/sec
    

    Just in case I activated DMA:

    # hdparm -d 1 /dev/hda
    

    and it reported:

    /dev/hda:
    setting
    using_dma to 1 (on)
    using_dma = 1 (on)
    

    So now with the hard drive tweaked for max performance the network works just fine (I am using ssh so it must be working ) I skip to chapter 4 to prepare my disks. I read thoroughly through the installation guide and decided to proceed with the following structure:

    Partition Filesystem  Size             Description
    /dev/hda1 ReiserFS    110 Mb           Boot partition
    /dev/hda2 swap       1024 Mb           Swap partition
    /dev/hda3 ReiserFS   Rest of the Disk  Root partition
    

    This partition scheme is nearly identical to the one used by the guide only that my choice of filesystem is ReiserFS and I have increased the swap to 1024 Mb.

    I used the cfdisk tool that comes with the CD.

    # cfdisk
    and in that program I defined
    <pre><code>Name Flags    Part   Type  FS Type [Label] Size (MB)
    ----------------------------------------------------
    hda1 Boot   Primary  Linux                   106.93
    hda2        Primary  Linux swap             1019.94
    hda3        Primary  Linux                 58884.78
    

    I toggled the Boot flag from the interface after having selected hda1. Once I finished with the partitioning I chose Write and confirmed it so that the partition table is written on the disk. I chose Quit and then rebooted the system just in case.

    # reboot
    

    I restarted the system and it booted again from the Live CD. Again I started sshd after setting a password for the root account. Now it is the time to format my partitions. The first one is the boot partition and I chose to label it boot

    # mkreiserfs -l boot /dev/hda1
    

    following that the root partition which was labeled root

    # mkreiserfs -l root /dev/hda3
    

    Finally time to format the swap partition

    # mkswap /dev/hda2
    

    and activate it

    # swapon /dev/hda2
    

    The partitions are now ready so all I have to do is mount them and start the installation.

    # mount /dev/hda3 /mnt/gentoo
    

    I will need to create a boot folder in the newly mounted partition

    # mkdir /mnt/gentoo/boot
    

    and now mount the boot partition in that folder

    # mount /dev/hda1 /mnt/gentoo/boot
    

    Moving on I need to check the date/time issuing the following command:

    # date
    

    The time was a bit off so I had to set it using the following command:

    # date 120123042004
    

    (where 12 is the month 01 is the day 23 is the hour, 04 the minute and 2004 the year)

    Now it is time to fetch the tarball. First I change the directory to /mnt/gentoo

    # cd /mnt/gentoo
    

    and then I use the links2 program (I like it better) to navigate through the mirrors and pick one which is closer to me (Austria)

    # links2 http://www.gentoo.org/main/en/mirrors.xml
    

    I chose the Inode network and then navigated to /releases/x86/2004.2/stages/x86 and downloaded the stage1-x86-2004.2.tar.bz2. Following that I unpacked the stage:

    # tar -xvjpf stage1-x86-2004.3.tar.bz2
    

    Then I had to tweak the make.conf file

    # nano /mnt/gentoo/etc/make.conf
    

    My make.conf is as follows:

    USE="-* X aalib acl acpi aim alsa apache2 apm audiofile 
    avi berkdb bidi bindist bitmap-fonts bzlib caps cdr 
    cpdflib crypt cscope ctype cups curl curlwrappers 
    dba dbx dga dio directfb divx4linux dvd dvdr encode 
    ethereal exif fam fastcgi fbcon fdftk flac flash 
    flatfile foomaticdb ftp gd gdbm ggi gif gmp gnome 
    gnutls gphoto2 gpm gtk gtk2 gtkhtml iconv icq imagemagick 
    imap imlib inifile innodb ipv6 jabber jack jikes jpeg 
    kerberos krb4 ladcca lcms ldap libwww mad maildir 
    mailwrapper mbox mcal memlimit mhash mikmod ming mmap mmx 
    motif moznocompose moznoirc moznomail mpeg mpi msn mssql 
    mysql -mysqli nas ncurses netcdf nhc98 nis nls offensive 
    oggvorbis opengl oscar pam pcmcia pcntl pcre pda pdflib 
    perl php pic pie plotutils png pnp posix ppds prelude 
    python quicktime readline samba sasl scanner sdl session 
    shared sharedmem simplexml slang slp snmp soap sockets 
    socks5 speex spell spl ssl svga sysvipc szip tcltk tcpd 
    tetex theora tidy tiff tokenizer truetype trusted uclibc 
    unicode usb vhosts videos wavelan wddx wmf xface xine 
    xml xml2 xmlrpc xmms xosd xprint xsl xv xvid yahoo yaz 
    zeo zlib x86"
    CHOST="i686-pc-linux-gnu"
    
    CFLAGS="-march=athlon-xp -O3 -pipe -fomit-frame-pointer"
    
    CXXFLAGS="${CFLAGS}"
    ACCEPT_KEYWORDS="~x86"
    PORTAGE_TMPDIR=/var/tmp
    PORTDIR=/usr/portage
    DISTDIR=${PORTDIR}/distfiles
    PKGDIR=${PORTDIR}/packages
    PORT_LOGDIR=/var/log/portage
    PORTDIR_OVERLAY=/usr/local/portage
    
    http_proxy="http://taurus.niden.net:8080"
        RESUMECOMMAND="
            /usr/bin/wget 
            -t 5 
            –passive-ftp \${URI} 
            -O \${DISTDIR}/\${FILE}"
    
    GENTOO_MIRRORS="
        http://gentoo.inode.at/ 
        http://gentoo.osuosl.org 
        http://gentoo.oregonstate.edu"
    SYNC="rsync://taurus.niden.net/portage"
    
    MAKEOPTS="-j2"
    
    AUTOCLEAN="yes"
    FEATURES="sandbox"
    

    You will notice that I use

    http_proxy="http://taurus.niden.net:8080" \
        RESUMECOMMAND="
            /usr/bin/wget 
                -t 5 
                –passive-ftp \${URI} 
                -O \${DISTDIR}/\${FILE}"
    

    because I have set up the httpd-replicator on my server and keep a local rsync mirror so that I don’t abuse the internet bandwidth. You will not need these lines on your installation. Additionally I set up my sync mirror to be my local server

    SYNC="rsync://taurus.niden.net/portage"
    

    whereas you will need to use one of the below (the closer to your location the better)

    Default: "rsync://rsync.gentoo.org/gentoo-portage" 
    North America: "rsync://rsync.namerica.gentoo.org/gentoo-portage" 
    South America: "rsync://rsync.samerica.gentoo.org/gentoo-portage" 
    Europe: "rsync://rsync.europe.gentoo.org/gentoo-portage" 
    Asia: "rsync://rsync.asia.gentoo.org/gentoo-portage" 
    Australia: "rsync://rsync.au.gentoo.org/gentoo-portage"
    

    Also I set up some Portage paths which have to be created (PORTDIR_OVERLAY and PORT_LOGDIR):

    # mkdir /mnt/gentoo/usr/local/portage 
    # mkdir /mnt/gentoo/var/log/portage
    

    Before chrooting I need to copy the resolv.conf file in our mounted partition

    # cp -L /etc/resolv.conf /mnt/gentoo/etc/resolv.conf
    

    mount the proc partition

    # mount -t proc none /mnt/gentoo/proc
    

    and chroot to the new environment

    # chroot /mnt/gentoo /bin/bash 
    # env-update 
    # source /etc/profile
    

    Now let us update the portage for the first time

    # emerge sync
    

    and here comes the wait - bootstraping

    # cd /usr/portage 
    # scripts/bootstrap.sh
    

    The compilation started at 13:30 and finished at 16:14, 3 hours later error free so I moved on to emerge my whole system.

    # emerge system
    

    73 packages were to be merged and for that I started at 06:00 and finished at 8.27. Not bad for my baby notebook!
    There was one config file that needed updating so I went on and updated it:

    # etc-update
    

    It appeared that there were trivial changes, nothing to report.

    So now off to set our timezone. For me it is Vienna, Austria. A little look at my system with:

    # ls /usr/share/zoneinfo
    

    reveals a Europe folder which in turn has the Vienna zone. Hence the command to set the link to my timezone:

    # ln -sf /usr/share/zoneinfo/Europe/Vienna /etc/localtime
    

    Okay now to the easy stuff. We need to grab a kernel. From the choices (and the handbook plus the Gentoo Kernel Guide have a wealth of information helping you choose) I opted for gentoo-dev-sources.

    # emerge gentoo-dev-sources
    

    Before I choose what I need for my kernel (modules or built in) I went and read the Gentoo udev guide. I choose to with udev since this is the way things are moving and I might as well get a head start with it.

    First I need to emerge the udev, which will emerge baselayout and hotplug

    # emerge udev
    

    and also coldplug for boot support on plugged devices

    # emerge coldplug
    

    Now I have to compile the kernel. This requires a bit more attention so I tried to get it first time right.

    # cd /usr/src/linux 
    # make menuconfig
    

    After making my choices I compile the kernel

    # make && make modules_install
    

    I installed the kernel by copying the relevant file in my boot partition:

    # cp arch/i386/boot/bzImage /boot/kernel-2.6.10-gentoo-r4
    

    I also copy the System.map and the .config file just in case:

    # cp System.map /boot/System.map-2.6.10-gentoo-r4 
    # cp .config /boot/config-2.6.10-gentoo-r4
    

    At this point I need to sort out the fstab file for my system to load properly.
    ```sh

    nano -w /etc/fstab

    ```

    My fstab is as follows:

    /dev/hda1 /boot      reiserfs noauto,noatime,notail 1 2
    /dev/hda2 none       swap     defaults              0 0
    /dev/hda3 /          reiserfs noatime               0 1
    none      /proc      proc     defaults              0 0
    none      /dev/pts   devpts   defaults              0 0
    none      /dev/shm   tmpfs    defaults              0 0
    none      /sys       sysfs    defaults              0 0
    /dev/hdc  /mnt/cdrom auto     noauto,ro             0 0
    

    What follows is the host name, domain name and network configuration.

    Hostname

    # nano -w /etc/conf.d/hostname
    

    Domain name

    # nano -w /etc/conf.d/domainname
    

    Adding the domain name to the default runlevel

    # rc-update add domainname default
    

    There is no need for me to touch the /etc/conf.d/net file since I will be using DHCP for my LAN. I won’t add it to the default runlevel either (the network) since I don’t usually connect to the network by the LAN interface rather than the wireless one - for that a bit later.

    Finally I need to set up the hosts file:

    # nano -w /etc/hosts
    

    with the available hosts in my network.

    What follows is the PCMCIA. This is handled by emerging the pcmcia-cs package (note that I am using the -X flag since I don’t want xorg-x11 to be installed now - the handbook is king!)

    # USE="-X" emerge pcmcia-cs
    

    Critical dependency is dhcpd. I need to merge it so that I can obtain an IP address from my router

    # emerge dhcpd
    

    Also critical is to set the root password

    # passwd
    

    I am also emerging pciutils. These will give me lsmod and lspci later on

    # emerge pciutils
    

    Now is the time for the system tools. I will install a system logger, a cron daemon, file system tools and bootloader.

    System Logger - I chose syslog-ng.

    # emerge syslog-ng
    

    and added it to the default runlevel

    # rc-update add syslog-ng default
    

    Cron daemon - I chose vixie-cron

    # emerge vixie-cron
    

    and added it to the default runlevel

    # rc-update add vixie-cron default
    

    File System tools - Naturally I need reiserprogs due to my file system

    # emerge reiserfsprogs
    

    Bootloader - I chose grub.

    # emerge grub
    

    once grub was compiled I setup my grub.conf

    # nano -w /boot/grub/grub.conf
    

    Now let us set grub properly by updating the /etc/mtab

    # cp /proc/mounts /etc/mtab
    

    and grub-install will finish the job

    # grub-install –root-directory=/boot /dev/hda
    

    Finally we are ready to reboot the system.

    Exit the chrooted environment

    # exit
    

    change directory to the root of the Live CD

    # cd /
    

    unmount the mounted partitions

    # umount /mnt/gentoo/boot/ /mnt/gentoo/proc/ /mnt/gentoo/
    

    and reboot

    # reboot
    

    Make sure you eject the CD when the system reboots because you don’t want to boot from it.

    Well it appears to be OK so far since the grub menu showed up and after the whole boot sequence I had my first Linux login.