• 2012-01-15 12:00:00

    Downgrading PHPUnit from 3.6 to 3.5

    Recently I had to rebuild my computer, and decided to install Linux Mint 12 (Lisa), which is a very lean installation - for my taste that is.

    Going through the whole process of reinstalling all the packages that I need or had, PHPUnit was one of them. Easy enough a couple commands did the trick

    sudo apt-get install php-pear
    sudo pear upgrade PEAR
    sudo pear config-set auto_discover 1
    sudo pear install pear.phpunit.de/PHPUnit
    

    I wanted to run my tests after that, only to find an error in the execution:

    PHP Fatal error:  Call to undefined method PHPUnit_Util_Filter::addFileToFilter()
     in /home/www/project/library/PHPUnit/Framework.php on line 48
    

    At first I thought that it was a path error, so I included the /usr/share/php/PHPUnit and others in the php.ini file but with no luck. With a little bit of Googling I found out that there have been some changes in the 3.6 version of PHPUnit and things don't work as they did before.

    Effectively, 3.6 had some refactoring done and thus the line:

    PHPUnit_Util_Filter::addDirectoryToFilter("$dir/tests");
    

    changed to

    PHP_CodeCoverage_Filter::getInstance()
            ->addDirectoryToBlacklist("$dir/tests");
    

    Since I didn't want to change my whole test suite, I had to find a solution i.e. downgrade PHPUnit to 3.5.

    Unfortunately specifying the version directly did not wok

    sudo pear install phpunit/PHPUnit-3.5.15
    

    since it would pull the latest version again and I would end up with the 3.6 files.

    So I went one step further and installed specific versions of the relevant dependencies to satisfy the 3.5.15 version.

    Uninstallation of 3.6

    pear uninstall phpunit/PHPUnit_Selenium
    pear uninstall phpunit/DbUnit
    pear uninstall phpunit/PHPUnit
    pear uninstall phpunit/PHP_CodeCoverage
    pear uninstall phpunit/PHP_Iterator
    pear uninstall phpunit/PHPUnit_MockObject
    pear uninstall phpunit/Text_Template
    pear uninstall phpunit/PHP_Invoker
    pear uninstall phpunit/PHP_Timer
    pear uninstall phpunit/File_Iterator
    pear uninstall pear.symfony-project.com/YAML
    

    Installation of 3.5.15

    pear install pear.symfony-project.com/YAML-1.0.2
    pear install phpunit/PHPUnit_Selenium-1.0.1
    pear install phpunit/PHP_Timer-1.0.0
    pear install phpunit/Text_Template-1.0.0
    pear install phpunit/PHPUnit_MockObject-1.0.3
    pear install phpunit/File_Iterator-1.2.3
    pear install phpunit/PHP_CodeCoverage-1.0.2
    pear install phpunit/DbUnit-1.0.0
    pear install phpunit/PHPUnit-3.5.15
    

    I hope you find the above useful :)

  • 2011-05-18 14:49:00

    Using autounmask to Unmask Packages in Gentoo

    Gentoo is one of my favorite Linux distributions. Although I am comfortable with other distributions, Gentoo has a special place in my heart and whenever I can use it I do :)

    There are however some times that I would like to install a package - mostly to test something - and the package is masked. Masked packages are not "production ready" so they are not included in the portage tree i.e. available to be installed.

    To allow a masked package to be installed, you will need to unmask that package by adding a corresponding entry in the etc/portage/package.keywords file.

    The problem happens when the masked package (that you just unmasked) depends on other packages that are also masked. You will then need to rinse and repeat the process to ensure that everything is in place so that you can install that unmasked package.

    NOTE: Playing with masked packages is like playing with fire. If you don't know what you are doing or you are not ready to potentially have an unusable system, don't follow the instructions below or unmask any packages.

    app-portage/autounmask

    The Gentoo developers have created a little utility that will unmask each package that needs to be unmasked. The utility is app-portage/autounmask.

    I wanted to unmask www-misc/fcgiwrap so my manual method would be:

    echo "=www-misc/fcgiwrap ~amd64" >> /etc/portage/packages.keywords
    

    and would then emerge the package

    emerge =www-misc/fcgiwrap
    

    Instead I used autounmask:

    emerge app-portage/autounmask
    

    and

    autounmask www-misc/fcgiwrap-1.0.3
    
    autounmask version 0.27 (using PortageXS-0.02.09 and portage-2.1.9.42)
    * Using repository: /usr/portage
    * Using package.keywords file: /etc/portage/package.keywords
    * Using package.unmask file: /etc/portage/package.unmask
    * Using package.use file: /etc/portage/package.use
    * Unmasking www-misc/fcgiwrap-1.0.3 and its dependencies.. this might take a while..
    * Added '=www-misc/fcgiwrap-1.0.3 ~amd64' to /etc/portage/package.keywords
    * done!
    

    Once that is done I can issue the emerge command and voila!

    emerge www-misc/fcgiwrap
    

    Although in my case there was only one dependency to unmask, when trying to unmask packages that have multiple dependencies such as gnome, kde etc., autounmask can be a very helpful utility.

  • 2011-03-21 14:10:00

    Slicehost vs. Linode

    Through the years I have hosted my sites on various hosting companies. I had the really good experiences like Vertexhost and really terrible ones - I don't remember the name of the host, but that kid, as it turned out to be later on, managed to lose 1.6GB of my data. You can safely say that I have been burned by bad hosting companies but also have enjoyed the services of good ones. In the case of Vertexhost, I part owned that company a few years back and I know that the current owner is a straight up guy and really cares for his customers.

    Since I moved my emails to Google Apps I only need the hosting for my personal sites such as my blog, my wife's sites (burntoutmom.com, greekmommy.net) and a few other small sites.

    I used to host those sites on one of my company's clusters. The bandwidth consumed was nothing to write home about (I think in total it was a couple of GB per month ~ 1.00 USD) so it didn't matter that I had them there. However, recent events forced me to move them out of that cluster. I was on the market for good and relatively cheap hosting. I did not want to purchase my own server or co-locate with someone else. My solution was going to be a VPS since I would be in control of what I install and what I need.

    Slicehost

    Without much thought, I signed up for Slicehost, which is a subsidiary of Rackspace, a very well known and reputable company.

    I got their 4GB package (250.00 USD per month) and installed Gentoo on it. Apart from the price which was a bit steep, everything else was fine. I was happy to be able to host my sites in a configuration that I was comfortable with, under the understanding that if the VPS failed, then all my sites would go down. That however is the risk that everyone takes while hosting their sites on a single machine. The higher the availability and redundancy the higher the cost.

    I must admit that signing up was not a very happy experience. I went and paid with my credit card, as they pro-rate your month based on your package. Almost immediately after signing up, came the email informing me that my credit card has been charged for the relevant amount. I got into the box through ssh, updated the /etc/make.conf file with the USE flags that I needed, run emerge --sync and then emerge --update --deep --newuse --verbose world so as to update the system.

    It must have been around 5-10 minutes into the process that I received an email from Slicehost saying that they are checking my account information and that I need to confirm my credit card details. I immediately replied to their email (gotta love the desktop notifications on GMail), with the information they needed.

    After I sent the email, I noticed that the box was not responding. I tried to log back in and could not. I was also logged out (and could not log back in) to their management console on Slicehost site. I was fuming! They severed the connection to the VPS in the middle of compilation to check my credit card information. I understand that they need to perform checks for fraud but two questions came to mind:

    • Why did they have to sever the connection and not just send an email, and if I did not reply, just block access to the box? That would have been a heck of a lot of an inconvenience to myself i.e. the end user.
    • Why did the initial email say that my credit card has been charged and it had not?

    No more than 10 minutes later the whole thing had been resolved. I received an email saying that "everything is OK and your account has been restored", at which point I logged back in to redo the compilations. I also received emails from their support/billing team apologizing but stating that although the initial email states that they charge the credit card, they don't. It is something they need to correct because it pisses people (like me) off.

    There was nothing wrong with my setup - everything was working perfectly but the price was really what was bothering me. I would be able to support the sites for a few months, but since literally none of them is making money (maybe a dollar here or there from my blog but that is about it), I would have to pay out of pocket for the hosting every month. I had to find a different solution that would be:

    • cheaper than Slicehost
    • flexible in terms of setup
    • easy to use in terms of controlling your VPS

    After a lot of research I ended up with two winners: Linode and Prgrm. I opted for Linode, because although it was quite a bit more expensive than Prgmr, it had the better console in handling your VPS. I will, however, try out Prgmr's services in the near future so as to assess how good they are. They definitely cannot be beat in price.

    Linode

    Setting up an account with Linode was very easy. I didn't have any of the mini-saga I had with Slicehost. The account was created right there and then, my credit card charged and I was up and running in no time. Immediately I could see a difference in price. Linode's package for 4GB or RAM is 90.00 USD cheaper (159.00 USD vs. 250.00 USD for Slicehost). For the same package, the price difference is huge.

    I started testing the network, creating my VPS in the Atlanta, GA datacenter (Linode offers a number of data centers for you to create your own). The functionality that was available to me was identical and in some cases superior to that of Slicehost. There are a lot more distributions to choose from, and you can partition your VPS the way you want it to name a couple.

    Shifting through the documentation, I saw a few topics regarding high availability websites. The articles described using DRDB, nginX, heartbeat and pacemaker etc. to keep your sites highly available. I was intrigued by the information and set off to create a load balancer using two VPSs and nginX. I have documented the process and this is another blog post that will come later on this week.

    While experimenting with the load balancer (and it was Saturday evening) I had to add a new IP address to one of the VPS instances. At the time my account would not allow such a change and I had to contact support. I did and got a reply in less than 5 minutes. I was really impressed by this. Subsequent tickets were answered within the 5 minute time frame. Kudos to Linode support for their speed and professionalism.

    Conclusion

    For a lot cheaper, Linode offered the same thing that Slicehost did. Moving my sites from one VPS to another was a matter of changing my DNS records to point to the new IP address.

    I have been using Linode for a week and so far so good. The support is superb and the documentation is full of how-to's that allows me to experiment with anything I want to - and the prices are not going to break me.

    Resources

  • 2011-02-20 13:56:00

    Keeping your Linux users in sync in a cluster

    As websites become more and more popular, your application might not be able to cope with the demand that your users put on your server. To accommodate that, you will need to move out of the one server solution and create a cluster. The easiest cluster to create would be with two servers, one to handle your web requests (HTTP/PHP etc.) and one to handle your database load (MySQL). Again this setup can only get you so far. If your site is growing, you will need a cluster of servers.

    Database

    The scope of this How-To will not cover database replication; I will need to dedicate a separate blog post for that. However, clustering your database is relatively easy with MySQL replication. You can set a virtual hostname (something like mysql.mydomain.com) which is visible only within your network. You then set up the configuration of your application to use that as the host. The virtual hostname will map to the current master server, while the slave(s) will only replicate.

    For instance, if you have two servers A and B, you can configure both of them to become master or slave in MySQL. You then set one of them as master (A) and the other as slave (B). If something happens to A, B gets promoted to master instantly. Once A comes back up, it gets demoted to a slave and the process is repeated if/when B has a problem. This can be a very good solution but you will need to have pretty evenly matched servers to keep with the demand. Alternatively B can be less powered than A and when A comes back up you keep it momentarily as a slave (until everything is replicated) and then promote it back to master.

    One thing to note about replication (that I found through trial and error). MySQL keeps binary logs to handle replication. If you are not cautious in your deployment, MySQL will never recycle those logs and therefore you will soon run out of space when having a busy site. By default those logs will be under /var/lib/mysql.

    By changing directives in my.cnf you can store the binary logs in a different folder and even set up 'garbage collection' or recycling. You can for instance set the logs to rotate every X days with the following directive in my.cnf:

    expire_logs_days = 5
    

    I set mine to 5 days which is extremely generous. If your replication is broken you must have the means to know about it within minutes (see nagios for a good monitoring service). In most cases 2 days is more than enough.

    Files

    There are numerous ways of keeping your cluster in sync. A really good tool that I have used when playing around with a cluster is csync2. Installation is really easy and and all you will need is to run a cron task every X minutes (up to you) to synchronize the new files. Imagine it as a two way rsync. Another tool that can do this is unison but I found it to be slow and difficult to implement - that's just me though.

    Assume an implementation of a website being served by two (or more) servers behind a load balancer. If your users upload files, you don't know where those files are uploaded, which server that is. As a result if user A uploads the file abc.txt to server A, user B might be served the content from server B and would not be able to access the file. csync2 would synchronize the file across the number of servers, thus providing access to the content and keeping multiple copies of the content (additional backup if you like).

    NFS

    An alternative to keeping everything synchronized is to use a NFS. This approach has many advantages and some disadvantages. It is up to you on whether the disadvantages are something you can live with.

    Disadvantages
    • NFS is slow - slower than the direct access to a local hard drive.
    • Most likely you will use a symlink to the NFS folder, which can slow things down even more.
    Advantages
    • The NFS does not rely on the individual web servers for content.
    • The web servers can be low to medium spec boxes without the need to have really fast and large hard drives
    • A well designed NFS with DRDB provides a raid-1 over a network. Using gigabit Network Interface Cards you can keep performance at really high levels.

    I know that my friend Floren does not agree with my approach on the NFS and would definitely have gone with the csync2 approach. Your implementation depends on your needs.

    Users and Groups

    Using the NFS approach, we need to keep the files and permissions properly set up for our application. Assume that we have two servers and we need to create one user to access our application and upload files.

    The user has been created on both servers and the files are stored on the NFS. Connecting to server A and looking at the files we can see something like this:

    drwxr-xr-x 17 niden  niden  4096 Feb 18 13:41 www.niden.net
    drwxr-xr-x  5 niden  niden  4096 Nov 15 22:10 www.niden.net.files
    drwxr-xr-x  7 beauty beauty 4096 Nov 21 17:42 www.beautyandthegeek.it
    

    However when connecting to server B, the same listing tells another story:

    drwxr-xr-x 17 508    510    4096 Feb 18 13:41 www.niden.net
    drwxr-xr-x  5 508    510    4096 Nov 15 22:10 www.niden.net.files
    drwxr-xr-x  7 510    511    4096 Nov 21 17:42 www.beautyandthegeek.it
    

    The problem here is the uid and gid of the users and groups of each user respectively. Somehow (and this is really easy to happen) server A had one or more users added to it, thus the internal counter of the user IDs has been increased by one or more and is not identical to that one of server B. So adding a new user in server A will get the uid 510 while on server B the same process will produce a user with a uid of 508.

    To have all users setup on all servers the same way, we need to use two commands: groupadd and useradd (in some Linux distributions you might find them as addgroup and adduser).

    groupadd

    First of all you will need to add groups. You can of course keep all users in one group but my implementation was to keep one user and one group per access. To cater for that I had to first create a group for every user and then the user account itself. Like users, groups have unique ids (gid). The purpose of gid is:

    The numerical value of the groups ID. This value must be unique, unless the -o option is used. The value must be non-negative. The default is to use the smallest ID value greater than 999 and greater than every other group. Values between 0 and 999 are typically reserved for system accounts.

    I chose to assign each group a unique id (you can override this behavior by using the -o switch in the command below, thus allowing a gid to be used in more than one group). The arbitrary number that I chose was 2000.

    As an example, I will set niden as the user/group for accessing this site and beauty as the user/group that accesses BeautyAndTheGeek.IT. Note that this is only an example.

    groupadd --gid 2000 niden
    groupadd --gid 2001 beauty
    

    Repeat the process as many times as needed for your setup. Connect to the second server and repeat this process. Of course if you have more than two servers, repeat the process on each of the servers that you have (and each accesses your NFS)

    useradd

    The next step is to add the users. Like groups, we will need to set the uid up. The purpose of the uid is:

    The numerical value of the users ID. This value must be unique, unless the -o option is used. The value must be non-negative. The default is to use the smallest ID value greater than 999 and greater than every other user. Values between 0 and 999 are typically reserved for system accounts.

    Like with the groups, I chose to assign each user a unique id starting from 2000.

    So to in the example above, the commands that I used were:

    useradd --uid 2000 -g niden --create-home niden
    useradd --uid 2000 -g beauty --create-home beauty
    

    You can also use a different syntax, utilizing the numeric gids:

    useradd --uid 2000 --gid 2000 --create-home niden
    useradd --uid 2000 --gid 2001 --create-home beauty
    

    Again, repeat the process as many times as needed for your setup and to as many servers as needed.

    In the example above I issued the --create-home switch (or -m) so as a home folder to be created under /home for each user. Your setup might not need this step. Check the references at the bottom of this blog post for the manual pages for groupadd and useradd.

    I would suggest that you keep a log of which user/group has which uid/gid. It helps in the long run, plus it is a good habit to keep proper documentation on projects :)

    Passwords?

    So how about the passwords on all servers? My approach is crude but effective. I connected to the first server, and set the password for each user, writing down what the password was:

    passwd niden
    

    Once I had all the passwords set, I opened the /etc/shadow file.

    nano /etc/shadow
    

    and that revealed a long list of users and their scrambled passwords:

    niden:$$$$long_string_of_characters_goes_here$$$$:13864:0:99999:7:::
    beauty:$$$$again_long_string_of_characters_goes_here$$$$:15009:0:99999:7:::
    

    Since I know that I added niden and beauty as users, I copied these two lines. I then connected to the second server, opened /etc/shadow and located the two lines where the niden and beauty users are referenced. I deleted the existing lines, and pasted the ones that I had copied from server A. Saved the file and now my passwords are synchronized in both servers.

    Conclusion

    The above might not be the best way of keeping users in sync in a cluster but it gives you an idea on where to start. There are different implementations available (Google is your friend) and your mileage might vary. The above has worked for me for a number of years since I never needed to add more than a handful of users on the servers each year.

    References

  • 2010-08-21 13:38:00

    Create an inexpensive hourly remote backup

    There are two kinds of people, those who backup regularly, and those that never had a hard drive fail

    As you can tell the above is my favorite quote. It is so true and I believe everyone should evaluate how much their data (emails, documents, files) is worth to them and, based on that value, create a backup strategy that suits them. I know for sure that if I ever lost the pictures and videos of my family I would be devastated since those are irreplaceable.

    So the question is how can I have an inexpensive backup solution? All my documents and emails are stored in Google, since my domain is on Google Apps. What happens to the live/development servers though that host all my work? I program on a daily basis and the code has to be backed up regularly so as to avoid any hard drive failures and thus result in loss of time and money.

    So here is my solution. I have an old computer (IBM Thincentre) which I decided to beef up a bit. I bought 4Gb of RAM from eBay for less than $100 for it. Although this is was not necessary since my solution would be based on Linux (Gentoo in particular), I wanted to have faster compilation times for packages.

    I bought two external drives (750Gb and 500Gb respectively) and one 750Gb internal drive. I already have a 120Gb hard drive in the computer. The two external ones are connected to the computer using USB while the internal ones are connected using SATA.

    The external drives are formatted using NTFS while the whole computer is built using ReiserFS.

    Here is the approach:

    • I have installed and have a working Gentoo installation on the machine
    • I have an active Internet connection
    • I have installed LVM on the machine and set up the core system on the 120Mb drive while the 500Mb is on LVM
    • I have 300Mb active on the LVM (from the available 500Mb)
    • I have generated a public SSH key (I will need this to exchange it with the target servers)
    • I have mounted the internal 500Mb drive to the /storage folder
    • I have mounted the external USB 750Mb drive to the /backup_hourly folder
    • I have mounted the external USB 500Mb drive to the /backup_daily folder

    Here is how my backup works:

    Every hour a script runs. The script uses rsync to syncrhonize files and folders from a remote server locally. Those files and folders are kept in relevant server named subfolders in the /storage folder (remember this is my LVM). So for instance my subfolders will be /storage/beryllium.niden.net, /storage/nitrogen.niden.net, /storage/argon.niden.net etc.

    Once the rsync completes, the script continues by compressing the relevant 'server' folder and creates the compressed file with a date-time stamp on its name.

    When all compressions are completed, if the time that the script has executed is midnight, the backups are moved from the /storage folder to the /backup_daily folder (which has the external USB 500Gb mounted). If it is any other time, the files are moved in the /backup_hourly folder (which has the external USB 750Gb mounted).

    This way I ensure that I keep a lot of backups (daily and hourly ones). The backups are being recycled, so older ones get deleted. The amount of data that you need to archive as well as the storage space you have available dictate how far back you can go in your hourly and daily cycles.

    So let's get down to business. The script itself:

    #!/bin/bash
    DATE=`date +%Y-%m-%d-%H-%M`
    DATE2=`date +%Y-%m-%d`
    DATEBACK_HOUR=`date --date='6 days ago' +%Y-%m-%d`
    DATEBACK_DAY=`date --date='60 days ago' +%Y-%m-%d`
    FLAGS="--archive --verbose --numeric-ids --delete --rsh='ssh'"
    BACKUP_DRIVE="/storage"
    DAY_USB_DRIVE="/backup_daily"
    HOUR_USB_DRIVE="/backup_hourly"
    

    These are some variables that I need for the script to work. The DATE and DATE2 are used to date/time stamp the backups, while the DATEBACK_* are used to clear previous backups. In this case it is set to 6 days ago (for my system). It can be set to whatever you want provided that you do not run out of space.

    The FLAGS variable keeps the rsync command options while the BACKUP_DRIVE, DAY_USB_DRIVE and HOUR_USB_DRIVE hold the locations of the rsync folders, daily backup and hourly backup sorage areas.

    The script works with arrays. I have 4 arrays to do the work and the 3 of them must have exactly the same elements.

    # RSync Information
    rsync_info[1]="beryllium.niden.net html rsync"
    rsync_info[2]="beryllium.niden.net db rsync"
    rsync_info[3]="nitrogen.niden.net html rsync"
    rsync_info[4]="nitrogen.niden.net html db"
    rsync_info[5]="nitrogen.niden.net html svn"
    rsync_info[6]="argon.niden.net html rsync"
    

    This is the first array which holds descriptions to what needs to be done as far as source is concerned. These descriptions get appended to the log and helps me identify what step I am in.

    # RSync Source Folders
    rsync_source[1]="beryllium.niden.net:/var/www/localhost/htdocs/"
    rsync_source[2]="beryllium.niden.net:/niden_backup/db/"
    rsync_source[3]="nitrogen.niden.net:/var/www/localhost/htdocs/"
    rsync_source[4]="nitrogen.niden.net:/niden_backup/db"
    rsync_source[5]="nitrogen.niden.net:/niden_backup/svn"
    rsync_source[6]="argon.niden.net:/var/www/localhost/htdocs/"
    

    This array holds the source host and folder. Remember that I have already exchanged SSH keys with each server, therefore when the script runs there is a direct connection to the source server. If you need to keep things a bit more secure for you, then you will need to alter the contents of the rsync_source array so that it reflects the user that you log in with as well as the password.

    # RSync Target Folders
    rsync_target[1]="beryllium.niden.net/html/"
    rsync_target[2]="beryllium.niden.net/db/"
    rsync_target[3]="nitrogen.niden.net/html/"
    rsync_target[4]="nitrogen.niden.net/db/"
    rsync_target[5]="nitrogen.niden.net/svn/"
    rsync_target[6]="argon.niden.net/html/"
    

    This array holds the target locations for the rsync. These folders exist in my case under the /storage subfolder.

    # GZip target files
    servers[1]="beryllium.niden.net"
    servers[2]="nitrogen.niden.net"
    servers[3]="argon.niden.net"
    

    This array holds the names of the folders to be archived. These are the folders directly under the /storage folder and I am also using this array for the prefix of the compressed files. The suffix of the compressed files is a date/time stamp.

    Here is how the script evolves:

    echo "BACKUP START" >> $BACKUP_DRIVE/logs/$DATE.log
    date >> $BACKUP_DRIVE/logs/$DATE.log
    
    echo "BACKUP START" >> $BACKUP_DRIVE/logs/$DATE.log
    date >> $BACKUP_DRIVE/logs/$DATE.log
    
    # Loop through the RSync process
    element_count=${#rsync_info[@]}
    let "element_count = $element_count + 1"
    index=1
    while [ "$index" -lt "$element_count" ]
    do
        echo ${rsync_info[$index]} > $BACKUP_DRIVE/logs/$DATE.log
        rsync $FLAGS ${rsync_source[$index]} $BACKUP_DRIVE/${rsync_target[$index]} > $BACKUP_DRIVE/logs/$DATE.log
        let "index = $index + 1"
    done
    

    The snippet above loops through the rsync_info array and prints out the information in the log file. Right after that it uses the rsync_source and rsync_target arrays (as well as the FLAGS variable) to rsync the contents of the source server with the local folder. Remember that all three arrays have to be identical in size (rsync_info, rsync_source, rsync_target).

    The next thing to do is zip the data (I loop through the servers array)

    # Looping to GZip data
    element_count=${#servers[@]}
    let "element_count = $element_count + 1"
    index=1
    while [ "$index" -lt "$element_count" ]
    do
        echo "GZip ${servers[$index]}" > $BACKUP_DRIVE/logs/$DATE.log
        tar cvfz $BACKUP_DRIVE/${servers[$index]}-$DATE.tgz $BACKUP_DRIVE/${servers[$index]} > $BACKUP_DRIVE/logs/$DATE.log
        let "index = $index + 1"
    done
    

    The compression method I use is tar/gzip. I found it to be fast with a good compression ratio. You can choose anything you like.

    Now I need to delete old files from the drives and copy the files on those drives. I use the servers array again.

    # Looping to copy the produced files (if applicable) to the daily drive
    element_count=${#servers[@]}
    let "element_count = $element_count + 1"
    index=1
    
    while [ "$index" -lt "$element_count" ]
    do
        # Copy the midnight files
        echo "Removing old daily midnight files" > $BACKUP_DRIVE/logs/$DATE.log
        rm -f $DAY_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK_DAY*.* > $BACKUP_DRIVE/logs/$DATE.log
        echo "Copying daily midnight files" > $BACKUP_DRIVE/logs/$DATE.log
        cp -v $BACKUP_DRIVE/${servers[$index]}-$DATE2-00-*.tgz $DAY_USB_DRIVE/${servers[$index]} &nbsp>>; $BACKUP_DRIVE/logs/$DATE.log
        rm -f $BACKUP_DRIVE/${servers[$index]}-$DATE2-00-*.tgz > $BACKUP_DRIVE/logs/$DATE.log
    
        # Now copy the files in the hourly
        echo "Removing old hourly files" > $BACKUP_DRIVE/logs/$DATE.log
        rm -f $HOUR_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK_HOUR*.* > $BACKUP_DRIVE/logs/$DATE.log
        echo "Copying daily midnight files" > $BACKUP_DRIVE/logs/$DATE.log
        cp -v $BACKUP_DRIVE/${servers[$index]}-$DATE.tgz $HOUR_USB_DRIVE/${servers[$index]} > $BACKUP_DRIVE/logs/$DATE.log
        rm -f $HOUR_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK*.* > $BACKUP_DRIVE/logs/$DATE.log
        let "index = $index + 1"
    done
    
    echo "BACKUP END" >> $BACKUP_DRIVE/logs/$DATE.log
    

    The last part of the script loops through the servers array and:

    • Deletes the old files (recycling of space) from the daily backup drive (/storage/backup_daily) according to the DATEBACK_DAY variable. If the files are not found a warning will appear in the log.
    • Copies the daily midnight file to the daily drive (if the file does not exist it will simply echo a warning in the log - I do not worry about warnings of this kind in the log file and was too lazy to use an IF EXISTS condition)
    • Removes the daily midnight file from the /storage drive.

    The reason I am using copy and then remove instead of the move (mv) command is that I have found this method to be faster.

    Finally the same thing happens with the hourly files

    • Old files are removed (DATEBACK_HOUR variable)
    • Hourly file gets copied to the /backup_hourly drive
    • Hourly file gets deleted from the /storage drive

    All I need now is to add the script in my crontab and let it run every hour.

    NOTE: The first time you will run the script you will need to do it manually (not in a cron job). The reason behind it is that the first time rsync will need to download all the contents of the source servers/folders in the /storage drive so as to create an exact mirror. Once that lengthy step is done, the script can be added in the crontab. Subsequent runs of the script will download only the changed/deleted files.

    This method can be very effective while not using a ton of bandwidth every hour. I have used this method for the best part of a year now and it has saved me a couple of times.

    The last thing I need to present you is the backup script that I have for my databases. As you notice above the source folder of beryllium.niden.net as far as databases are concerned is beryllium.niden.net/db/. What I do is I dump and zip the databases every hour on my servers. Although this is not a very efficient way of doing things and it adds to the bandwidth consumption every hour (since the dump will create a new file every hour) I have the following script running on my database servers every hour at the 45th minute:

    #!/bin/bash
    
    DBUSER=mydbuser
    DBPASS='dbpassword'
    DBHOST=localhost
    BACKUPFOLDER="/niden_backup"
    DBNAMES="`mysql --user=$DBUSER --password=$DBPASS --host=$DBHOST --batch --skip-column-names -e "show databases"| sed 's/ /%/g'`"
    OPTIONS="--quote-names --opt --compress "
    
    # Clear the backu folder
    rm -fR $BACKUPFOLDER/db/*.*
    
    for i in $DBNAMES; do
        echo Dumping Database: $i
        mysqldump --user=$DBUSER --password=$DBPASS --host=$DBHOST $OPTIONS $i > $BACKUPFOLDER/db/$i.sql
        tar cvfz $BACKUPFOLDER/db/$i.tqz $BACKUPFOLDER/db/$i.sql
        rm -f $BACKUPFOLDER/db/$i.sql
    done
    

    That's it.

    The backup script can be found in my GitHub here.

    Update: The metric units for the drives were GB not MB. Thanks to Jani Hartikainen for pointing it out.

  • 2010-08-01 13:11:00

    Subversion Backup How-To

    I will start this post once again with the words of a wise man:

    There are two kinds of people, those who backup regularly, and those that never had a hard drive fail

    So the moral of the story here is backup often. If something is to happen, the impact on your operations will be minimal if your backup strategy is in place and operational.

    There are a lot of backup scenarios and strategies. Most of them suggest a backup once a day, usually at the early hours of the day. This however might not work very well with a fast paced environment where data changes several times per hour. This kind of environment is usually a software development one.

    If you have chosen Subversion to be your software version control software then you will need a backup strategy for your repositories. Since the code changes very often, this strategy cannot rely on the daily backup schedule. The reason being is that, in software, a day's worth of work usually costs a lot more than the actual daily rate of the programmers.

    Below are some of the scripts I have used over the years for my incremental backups, that I hope will help you too. You are more than welcome to copy and paste the scripts and use them  or modify them to suit your needs. Please note though that the scripts are provided as is and that you must check your backup strategy with a full backup/restore cycle. I cannot assume responsibility of something that might happen in your system.

    Now that the 'legal' stuff are out of the way, here are the different strategies that you can adopt. :)

    svn-hot-backup

    This is a script that is provided with Subversion. It copies (and compresses if requested) the whole repository to a specified location. This technique allows for a full copy of the repository to be moved to a different location. The target location can be a resource on the local machine or a network resource. You can also backup on the local drive and then as a next step transfer the target files to an offsite location with FTP, SCP, RSync or any other mechanism you prefer.

    #!/bin/bash
    
    # Grab listing of repositories and copy each
    # repository accordingly
    
    SVNFLD="/var/svn"
    BACKUPFLD="/backup"
    
    # First clean up the backup folder
    rm -f $BACKUPFLD/*.*
    
    for i in $(ls -1v $SVNFLD); do
        if [ $i != 'conf' ]; then
            /usr/bin/svn-hot-backup --archive-type=bz2 $SVNFLD/$i $BACKUPFLD
        fi
    done
    

    This script will create a copy of each of your repositories and compress it as a bz2 file in the target location. Note that I am filtering for 'conf'. The reason being is that I have a conf file with some configuration scripts in the same SVN folder. You can adapt the script to your needs to include/exclude repositories/folders as needed.

    This technique gives the ability to immediately restore a repository (or more than one) by changing the configuration file of SVN to point to the backup location. If you run the script every hour or so then your downtime and loss will be minimal, should something happens.

    There are some configuration options that you can tweak by editing the actual svn-hot-backup script. In Gentoo it is located under /usr/bin/. The default number of backups (num_backups) that the script will keep is 64. You can choose 0 to keep them all but you can adjust it according to your storage or your backup strategy.

    One last thing to note is that you can change the compression mechanism by changing the parameter of the --archive-type option. The compression types supported are gz (.tar.gz), bz2 (.tar.bz2) and zip (.zip)

    Full backup using dump

    This method is similar to the svn-hot-backup. It works by 'dumping' the repository in a portable file format and compressing it.

    #!/bin/bash
    
    # Grab listing of folders and dump each
    # repository accordingly
    
    SVNFLD="/var/svn"
    BACKUPFLD="/backup"
    
    # First clean up the backup folder
    rm -f $BACKUPFLD/svn/*.*
    
    for i in $(ls -1v $SVNFLD); do
        if [ $i != 'conf' ]; then
            svnadmin dump $SVNFLD/$i/ > $BACKUPFLD/$i.svn.dump
            tar cvfz $BACKUPFLD/svn/$i.tgz $BACKUPFLD/$i.svn.dump
            rm -f $BACKUPFLD/$i.svn.dump
        fi
    done
    

    As you can see, this version does the same thing as the svn-hot-backup. It does however give you a bit more control over the whole backup process and allows for a different compression mechanism - since the compression happens on a separate line in the script.

    NOTE: If you use the hotcopy parameter in svnadmin (svnadmin hotcopy ....) you will be duplicating the behavior of svn-hot-backup.

    Incremental backup using dump based on revision

    This last method is what I use at work. We have our repositories backed up externally and we rely on the backup script to have everything backed up and transferred to the external location within an hour, since our backup strategy is an hourly backup. We have discovered that sometimes the size of a repository can cause problems with the transfer, since the Internet line will not be able to transfer the files across in the allocated time. This happened once in the past with a repository that ended up being 500Mb (don't ask :)).

    So in order to minimize the upload time, I have altered the script to dump each repository's revision in a separate file. Here is how it works:

    We backup using rsync. This way the 'old' files are not being transferred.

    Every hour the script loops through each repository name and does the following:

    • Checks if the .latest file exists in the svn-latest folder. If not, then it sets the LASTDUMP variable to 0.
    • If the file exists, it reads it and obtains the number stored in that file. It then stores that number incremented by 1 in the LASTDUMP variable.
    • Checks the number of the latest revision and stores it in the LASTVERSION variable
    • It loops through the repository, dumps each revision (LASTDUMP to LASTVERSION) and compresses it

    This method creates new files every hour so long as new code has been added in each repository via the checkin process. The rsync command will then pick only the new files and nothing else, therefore the data transferred is reduced to a bare minimum allowing easily for hourly external backups. With this method we can also restore a single revision in a repository if we need to.

    The script that achieves that is as follows:

    #!/bin/bash
    
    # Grab listing of folders and dump each
    # repository accordingly
    
    SVNFLD="/var/svn"
    BACKUPFLD="/backup"
    CHECKFLD=$BACKUPFLD/svn-latest
    
    for i in $(ls -1v $SVNFLD); do
        if [ $i != 'conf' ]; then
            # Find out what our 'start' will be
            if [ -f $CHECKFLD/$i.latest ]
            then
                LATEST=$(cat $CHECKFLD/$i.latest)
                LASTDUMP=$LATEST+1
            else
                LASTDUMP=0
            fi
    
            # This is the 'end' for the loop
            LASTREVISION=$(svnlook youngest $SVNFLD/$i/)
    
            for ((r=$LASTDUMP; r< =$LASTREVISION; r++ )); do
                svnadmin dump $SVNFLD/$i/ --revision $r > $BACKUPFLD/$i-$r.svn.dump
                tar cvfz $BACKUPFLD/svn/$i-$r.tgz $BACKUPFLD/$i-$r.svn.dump
                rm -f $BACKUPFLD/$i-$r.svn.dump
                echo $r > $CHECKFLD/$i.latest
            done
        fi
    done
    

    Conclusion

    You must always backup your data. The frequency is dictated by the rate that your data updates and how critical your data is. I hope that the methods presented in this blog post will complement your programming and source control should you choose to adopt them.

  • 2010-01-10 12:00:00

    Create a SSL Certificate in Linux

    There are times that I want to set up a secure communication with the server I am working on. This might be because I want to run phpMyAdmin over SSL (I do not like unencrypted communications over the Internet), install a certificate for an eShop for a client or just for my personal use.

    The first time I did this, I had to research on the Internet and after a bit of a trial and error I managed to get everything working. However if you do not do something on a regular basis you will forget. I am no exception to this rule hence this post to allow me to remember what I did and hopefully help you too.

    Prerequisites:

    This how-to assumes that you are running Gentoo, however these instructions can easily be applied to any other Linux distribution.

    I need to check if openssl is installed:

    vanadium ~ # emerge --pretend dev-libs/openssl
    
    These are the packages that would be merged, in order:
    
    Calculating dependencies... done!
    [ebuild  R  ] dev-libs/openssl-0.9.8l-r2
    

    If you do not see the [R] next to the package (and you see a N for instance) that means that you need to install the package. Issuing:

    vanadium  ~ # emerge --verbose dev-libs/openssl
    

    will do the trick.

    Generate the Private Key

    I like to generate keys with a lot of bits. All of my certificates have 4096 bits. This is a personal preference and it does not hurt to keep that value. Your host or Signing Authority (like GoDaddy, VeriSign, Thawte etc.) might ask you in their instructions to generate one with 2048 bits so don't be alarmed there.

    Creating the RSA private key with 4096 bits using Triple-DES:

    vanadium ~ # openssl genrsa -des3 -out /root/vanadium.niden.net.locked.key 4096
    Generating RSA private key, 4096 bit long modulus
    .............................................................++
    ...........++
    e is 65537 (0x10001)
    Enter pass phrase for /root/vanadium.niden.net.locked.key:
    Verifying - Enter pass phrase for /root/vanadium.niden.net.locked.key:
    

    Remove the passphrase from the Private Key

    The key that was created earlier has a passphrase. Although this is good, it does have a side effect that any web server administrator does not like - the passphrase itself. Once the certificate is installed using the key (with the passphrase), every time that Apache is restarted, it will prompt the operator for the passphrase. This can be very inconvenient if your web server reboots in the middle of the night. Since Apache will be waiting for the passphrase, your site will be inaccessible.

    To avoid this inconvenience, I am removing the passphrase from the key. If you noticed the key that I have created above has the 'locked' phrase in its name. The reason is that I know that that particular key has the passphrase on it. I first need to copy the key and then remove the passphrase:

    vanadium ~ # cp -v vanadium.niden.net.locked.key vanadium.niden.net.key
    `vanadium.niden.net.locked.key' -> `vanadium.niden.net.key'
    vanadium ~ # openssl rsa -in vanadium.niden.net.locked.key -out vanadium.niden.net.key
    Enter pass phrase for vanadium.niden.net.locked.key:
    writing RSA key
    

    Generate the Certificate Signing Request (CSR)

    The purpose of the CSR is to be sent to one of the Certificate Authorities (GoDaddy, VeriSign, Thawte etc.) for verification. Alternatively I can self-sign the CSR (see below).

    Upon generation of this CSR I am asked about particular pieces of information to be incorporated in the CSR. The most important piece of information that I need to ensure that is correct is the Common Name. The answer to that question has to be the name of my web server - vanadium.niden.net in my case.

    NOTE: I am using the key without the passphrase.

    The command to generate the CSR is as follows:

    vanadium ~ # openssl req -new -key vanadium.niden.net.key -out vanadium.niden.net.csr
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Country Name (2 letter code) [AU]:US
    State or Province Name (full name) [Some-State]:Virginia
    Locality Name (eg, city) []:Arlington
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:niden.net
    Organizational Unit Name (eg, section) []:IT
    Common Name (eg, YOUR name) []:vanadium.niden.net
    Email Address []:[email protected]
    
    Please enter the following 'extra' attributes
    to be sent with your certificate request
    A challenge password []:
    An optional company name []:
    

    Once this step is completed, I open the CSR file with a text editor, copy the contents and paste them in the relevant field of the Certification Authority (in my case GoDaddy), so that they can verify the CSR and issue the certificate.

    If however this is a development box or you do not want your certificate signed by a Certification Authority, you can check the section below on how to generate a self-signed certificate.

    Generating a Self-Signed Certificate

    At this point you will need to generate a self-signed certificate because you either don't plan on having your certificate signed by a CA, or you wish to test your new SSL implementation while the CA is signing your certificate. This temporary certificate will generate an error in the client browser to the effect that the signing certificate authority is unknown and not trusted.

    To generate a temporary certificate which is good for 365 days, issue the following command:

    vanadium ~ # openssl x509 -req -days 365 -in vanadium.niden.net.csr -signkey vanadium.niden.net.key -out vanadium.niden.net.crt
    Signature ok
    subject=/C=US/ST=Virginia/L=Arlington/O=niden.net/OU=IT/CN=vanadi[email protected]
    Getting Private key
    

    Installation

    For my system, the certificates are kept under /etc/apache2/ssl/ so I am going to copy them there (your need to adjust the instructions below to suit your system/installation):

    vanadium ~ # cp -v vanadium.niden.net.key /etc/apache2/ssl/
    vanadium ~ # cp -v vanadium.niden.net.crt /etc/apache2/ssl/
    

    I also need to open the relevant file to enable the certificate

    vanadium ~ # nano -w /etc/apache2/vhosts.d/00_default_ssl_vhost.conf
    

    In that file I need to change the following directives:

    SSLCertificateFile /etc/ssl/apache2/vanadium.niden.net.crt
    SSLCertificateKeyFile /etc/ssl/apache2/vanadium.niden.net.key

    If my certificate was issued by a Certificate Authority, the files that I have received have the CA certificate file. I can enable it in the following line:

    SSLCACertificateFile /etc/ssl/apache2/vanadium.niden.net.ca-bundle.crt
    

    Restarting Apache

    vanadium ~ # /etc/init.d/apache2 restart
     * Stopping apache2 ...                              [ ok ]
     * Starting apache2 ...                              [ ok ]
    

    Navigating to https://vanadium.niden.net should tell me if what I did was successful or not. If your browser (Google Chrome in my case) gives you a bright red screen with all sorts of warnings, that means that

    • either you self signed the certificate - in which case it complains about the certificate not being signed by a Certificate Authority or
    • you made a mistake and the Common Name in the certificate is not the same as the host name.

    Both these errors are easy to fix.

  • 2009-11-24 14:00:00

    Chromium OS - Part III

    Continued from Part II

    I had to shut the computer down because I had a train to catch. However booting up again the computer on the train and trying to enter the chroot environment produced some errors. It might very well be the fact that I do not have an Internet connection (at least a reliable one).

    So this post will have to wait until I get back home so that I can continue the installation. The key point here will be how to add the Chromium OS image over the second partition of my hard drive so that I can keep my dual boot system. I will run Image for DOS and clone my hard drive, in an effort to keep a mirror backup of my notebook at this particular point in time - you never know when you will use it.

    So I booted again into Ubuntu and started the process. I run

    ./create_chroot.sh
    

    but it complained that a chroot already exists. I then run

    ./create_chroot.sh --delete
    

    which effectively removed the existing chroot and replaced it with a new one. Little did I know that I only had to run

    ./enter_chroot.sh
    

    to re-enter my previous chroot and continue where I left off. Oh well, you live and learn :)

    Just to be on the safe side I rerun the ./build_platform_packages.sh, ./build_kernel.sh and ./build_image.sh scripts. I am now exactly where I was prior to shutting the computer off. I have built the platform packages, the kernel and the image.

    Using the image

    Check the contents of the image

    I will mount the image locally to ensure that everything is OK. Please note that the folder below is the one created on my machine and might very well be different than yours. At the end of the build_image.sh script you will see a message that will reveal where your image folder is.

    cd ~/trunk/src/build/images/999.999.32909.021312-a1/
    sudo mount -o loop rootfs.image rootfs
    sudo chroot rootfs
    

    Inside the image basic commands will reveal success or failure:

    df
    dpkg -l
    

    Once everything is OK (or at least seems OK) I exit and unmount the image.

    exit
    sudo umount rootfs
    

    Although I got a cannot read table of mounted file systems: No such file or directory when I run df, dpkg had a long list of packages installed. I will for the moment ignore the df output and proceed to the next steps.

    Copy the image to a USB key

    Somehow I have misplaced my 16GB USB drive so I had to borrow a 4GB one from a friend of mine. This step copies the actual image from the hard drive to the USB drive. The drive itself is wiped clean so make sure that you have backed up the data that you have on it prior to running this step.

    You need to find out the device that your USB drive corresponds to. Running:

    sudo fdisk -l
    

    will reveal which device is the USB drive. For my system it is /dev/sdc1. Outside the chroot you run the script image_to_usb.sh. The command is:

    ./image_to_usb.sh --from=~/chromiumos/src/build/images/SUBDIR --to=/dev/USBKEYDEV
    

    and for my system the command was:

    ./image_to_usb.sh --from=~/chromiumos/src/build/images/999.999.32909.021312-a1/ --to=/dev/sdc1
    

    The output on the screen running the above command is:

    [email protected]:~/chromiumos/src/scripts$ ./image_to_usb.sh --from=~/chromiumos/src/build/images/999.999.32909.021312-a1/ --to=/dev/sdc1
    Copying USB image /usr/local/chromiumos/chromiumos.git/src/build/images/999.999.32909.021312-a1 to device /dev/sdc1...
    This will erase all data on this device:
    Disk /dev/sdc1: 4013 MB, 4013917184 bytes
    Are you sure (y/N)? y
    attempting to unmount any mounts on the USB device
    Copying root fs...
    opening for read /usr/local/chromiumos/chromiumos.git/src/build/images/999.999.32909.021312-a1/rootfs.image
    opening for write /dev/sdc1
    seeking to 1992294912 bytes in output file
    copy 996147200 bytes took 102.384 s
    speed: 9.7 MB/s
    Creating stateful partition...
    mke2fs 1.41.9 (22-Aug-2009)
    Filesystem label=C-STATE
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    60800 inodes, 243200 blocks
    12160 blocks (5.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=251658240
    8 block groups
    32768 blocks per group, 32768 fragments per group
    7600 inodes per group
    Superblock backups stored on blocks:
     32768, 98304, 163840, 229376
    
    Writing inode tables: done
    Creating journal (4096 blocks): done
    Writing superblocks and filesystem accounting information: done
    
    This filesystem will be automatically checked every 30 mounts or
    180 days, whichever comes first.  Use tune2fs -c or -i to override.
    Copying MBR...
    opening for read /usr/local/chromiumos/chromiumos.git/src/build/images/999.999.32909.021312-a1/mbr.image
    opening for write /dev/sdc1
    copy 512 bytes took 0.001 s
    speed: 0.0 MB/s
    Done.
    

    I have booted the system using the USB drive and I logged in the system using my google account. Using Ctrl+Alt+T I open a terminal and enter:

    /usr/sbin/chromeos-install
    

    This asks now for the password that I have set up earlier (the one stored in the text file) and then it nukes the hard drive replacing everything (make sure you backup your hard drive).

    A bit later I unpluged the USB drive and rebooted. Unfortunately things did not work very well but that is probably due to my particular hardware. I will retry this on my other notebook and update this blog post.

  • 2009-11-24 13:00:00

    Chromium OS - Part II

    Continued from Part I

    The download took quite a while, so I thought it might be a good idea to split this post in parts, so as to ensure good readability.

    I need to create some symlinks. Also a good place to add my repository is /usr/local hence the commands for Chromium OS and Chromium respectively.

    sudo mv chromiumos/ /usr/local/
    sudo mv chromium/ /usr/local/
    

    and now adding the symbolic links

    ln -s /usr/local/chromiumos/chromiumos.git ~/chromiumos
    ln -s /usr/local/chromium ~/chromium
    

    Creating the local repository

    All the scripts are in the src/script folder. So let's go to that folder (the symbolic link set earlier helps :))

    cd ~/chromiumos/src/scripts
    

    and running the command to create the local repository:

    ./make_local_repo.sh
    

    This command will ask you for your password - and bare in mind you must run all this as a normal user with sudo access - and then it will create a debootstrap. It will fetch the necessary packages from the Chromium OS Package Management.

    NOTE

    If something fails you will need to do

    rm -rf ~/chromiumos/repo
    

    and then rerun the ./make_local_repo.sh script again.

    Creating the build environment

    All we need to do is run the following command:

    ./make_chroot.sh
    

    The script will check if all the dependencies are satisfied, and if something is missing it will pull the necessary files and compile them as necessary. Although I did not encounter any problems, the documentation states that the /etc/apt/sources.list is used for retrieving the packages. If your setup is pulling packages from somewhere else then you may need to get the most recent packages from the repository. You can do that by running:

    ./make_chroot.sh --mirror=http://build.chromium.org/buildbot/packages --suite=chromeos_dev
    

    Building Chromium OS

    I need to build Chromium first (since I chose to download it too). This is necessary since your build will fail if you try it the other way around :)

    ./build_chrome.sh --chrome_dir ~/chromium
    
    Enter the chroot build environment

    Run the following command gets us back in the chroot environment (you will be asked for your password)

    ./enter_chroot.sh
    
    Set the shared user password

    This is a one-off step for those of us that want to be able to sudo from a terminal. I am setting the shared user password running the following script:

    ./set_shared_user_password.sh
    

    This will prompt for the password and the output will be stored in the ./shared_user_password.txt file. Don't worry the password is encrypted so if you do not have anyone watching over your shoulder while typing your password you are OK. Just to be on the safe side, clear the screen.

    clear
    
    Build the platform packages

    In the chroot environment run

    ./build_platform_packages.sh
    

    Unfortunately I hit a snag :( The build_platform_packages script produced an error:

    Checking for latest build of Chrome
    Downloading http://chrome-web/buildbot/snapshots/chromium-rel-linux-chromiumos/LATEST
    --2009-11-24 19:44:49--  http://chrome-web/buildbot/snapshots/chromium-rel-linux-chromiumos/LATEST
    Resolving chrome-web... failed: Name or service not known.
    wget: unable to resolve host address `chrome-web'
    make: *** [build-stamp] Error 1
    

    I quickly found what I need to do (Google is your friend :)). It appears that this is known bug and it is easily fixable. All I had to do is edit the copy_chrome_zip.sh file. I tried using nano in the chroot environment but it was not there. For that I exited the chroot and edited the file.

    exit
    nano -w ~/chromiumos/src/platform/chrome/copy_chrome_zip.sh
    

    Locate the line with the BASE_URL variable and change chrome-web to build.chromium.org and save the file. After that enter again the chroot and rerun the build_platform_packages.sh script.

    ./enter_chroot.sh
    ./build_platform_packages.sh</pre>
    

    Quite a bit later the script execution ended with All packages built :)

    Build the kernel

    In the chroot environment run

    ./build_kernel.sh
    

    A bit later I am looking at this message and grinning :)

    Kernel build successful, check /home/ndimopoulos/trunk/src/build/x86/local_packages/linux-image-2.6.30-chromeos-intel-menlow_002_i386.deb
    
    Build the image

    In the chroot environment run

    ./build_image.sh
    

    The script starts with validations, configurations, unpacking and compilations - all too fast for my eye to capture.

    The script finished compiling and there are warnings and errors :(. They all have to do with the disk geometry and partition 1 extends past the end of the disk /shrug again....

    In the end I get this on the screen which hopefully is OK...

    Re-reading the partition table ...
    BLKRRPART: Inappropriate ioctl for device
    
    If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
    to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
    (See fdisk(8).)
    Done.  Image created in /home/ndimopoulos/trunk/src/build/images/999.999.32809.203441-a1
    To copy to USB keyfob, outside the chroot, do something like:
      ./image_to_usb.sh --from=/usr/local/chromiumos/chromiumos.git/src/build/images/999.999.32809.203441-a1 --to=/dev/sdb
    To convert to VMWare image, outside the chroot, do something like:
      ./image_to_vmware.sh --from=/usr/local/chromiumos/chromiumos.git/src/build/images/999.999.32809.203441-a1
    

    Continued in Part III

  • 2009-11-24 12:00:00

    Chromium OS Part I

    A lot of hype has been generated on the Internet a few months back regarding Google's announcement that they are building a new operating system. The announcement was met with skepticism but also enthusiasm/anticipation by a lot of people who are puzzled as to what direction Google is taking and where are they looking themselves positioned in the computer industry.

    Google has already established themselves in the mobile operating system with Android which for many is better than Apple's iPhone. What managed to get Google very high up in user satisfaction with Android was the fact that it is open source. It is backed by Google and they are putting a lot of effort into this but the fact that anyone can download the source code and build the operating system themselves is amazing. Giving people the freedom to choose should be something that every company should do (personal opinion).

    On Friday I watched the Chromium OS Webcast. A lot of people have been waiting for a presentation from Google of the new operating system. Google provided just that but with a small twist. The presenters outlined the features of the new operating system: Fast, Fast, Fast.

    Although the presenters clearly stated that they will not provide any links to hardware, what is supported, where Chromium OS runs etc. they made sure to address one of the core features of Chromium OS. It is Open Sourced! That alone gives people again freedom. Freedom to choose what they want to have as their operating system in their computer.

    The website for the new operating system is located here and there are instructions on how you can build this new OS even in its current state - which is not production ready.

    Curious (as usual) I tried installing the Chromium OS on a virtual machine. My experience installing the new OS and other comments is outlined below:

    Prerequisites

    My DELL Studio 17 has two hard drives and because I really really do not like Windows Vista, I have installed Ubuntu 9.10 32bit on the second partition. The notebook enjoys a 2.4GHz Intel processor and 6GB RAM.

    I applied all the relevant updates (just to be on the safe side) and a couple of reboots later I am ready to venture in the unknown. The documentation outlines the minimum required packages: 

    Building on Linux requires the following software.

    • Python >= 2.4
    • Perl >= 5.x
    • gcc/g++ >= 4.2
    • g++-multilib >=4.2
    • bison >= 2.3
    • flex >= 2.5.34
    • gperf >= 3.0.4
    • pkg-config >= 0.20
    • libnss3-dev >= 3.12
    • libasound2-dev
    • libgconf2-dev
    • libglib2.0-dev
    • libgtk2.0-dev
    • libnspr4-0d >= 4.7.1+1.9-0ubuntu0.8.04.5 (ubuntu0.8.04.1 causes duplicate dtoa references)
    • libnspr4-dev >= 4.7.1+1.9-0ubuntu0.8.04.5
    • msttcorefonts (Microsoft fonts)
    • freetype-dev
    • libcairo2-dev
    • libdbus-1-dev

    Optional (currently, all of these are only used by layout tests):

    • wdiff
    • lighttpd
    • php5-cgi
    • sun-java6-fonts (needed for Lucida)

    Because I didn't want to go and check the version of every package mentioned above I run the following command (mentioned also in the documentation)

    sudo apt-get install subversion pkg-config python perl g++ g++-multilib bison flex gperf libnss3-dev libgtk2.0-dev libnspr4-0d libasound2-dev libnspr4-dev msttcorefonts libgconf2-dev libcairo2-dev libdbus-1-dev
    

    And then the optional extras:

    sudo apt-get install wdiff lighttpd php5-cgi sun-java6-fonts
    

    A few minutes later the necessary packages had been downloaded and installed. One step closer to my goal :)

    Getting the code

    Now that all the prerequisites have been satisfied. I need to get the code!

    Navigating to this link in the chromium.org wiki, I get all the instructions on how to get the code. There are some prerequisites there (i.e. your system needs to be able to uncompress tar files) but nothing out of the ordinary.

    I installed the depot_tools which was really a two step process - svn checkout the tools and add the tools path in the current path. After that I installed the git-core:

    sudo apt-get install git-core
    

    I will pull the source code from the SVN repository. I can just as easy download the tarball and unzip it. The instructions in the chromium.org wiki explain both options.

    I am ready to get the source code for the Chromium OS package. You can get the code with the Chromium browser or without it. I am opting to get it with the Chromium browser. The following commands retrieve the necessary files for the OS as well as dependencies for the browser:

    mkdir ~/chroomiumos
    cd ~/chromiumos
    gclient config http://src.chromium.org/git/chromiumos.git
    gclient sync --deps="unix,chromeos" --force
    

    The download takes a bit of time since there are a lot of files that we need to retrieve. In situations like these, browsing, reading a book, going to get something to eat or working on the other computer are some of the activities you can engage yourself in so as to kill time. That is of course if you do not have a T1 or FIOS at which point this step will be finished by the time you read this sentence :) (/sigh I miss FIOS).

    I open another terminal window in order to retrieve the Chromium source code (the browser now).

    mkdir ~/chromium
    cd ~/chromium
    gclient config http://src.chromium.org/svn/trunk/src
    gclient sync
    

    and the wait continues....

    Continued in Part II