• 2012-11-25 12:00:00

    Building a web app with PhalconPHP and AngularJS Update

    It's been a while since I last wrote a blog post, so I wanted to touch on the effort to upgrade the application that I wrote for Harry Hog Fottball using PhalconPHP and AngularJS

    If you haven't read it, the first two blog posts were here and here.

    The application was written using the 0.4.5 version of PhalconPHP. Since then there have been significant changes to the framework, such as the introduction of a DI container, injectable objects and lately interfaces (in 0.7.0, to be released in a couple of days), I had to make some changes.

    There are a couple of things that I as a developer would like to see in PhalconPHP, which I am pretty sure will appear later on, since let's face it the framework is still very young (not even 1.0 version yet). Despite its "youth" it is a robust framework with excellent support, features and a growing community. One of these features is behaviors which I had to implement myself, and this was something new that came with this upgrade.

    Recently a new repo has been created on Github called the incubator, where developers can share implementations of common tasks, that act as drop ins to the framework and extend it. These implementations are all written in PHP so everyone can just download them and use them. The more submissions come in, the more the framework will grow and eventually these submissions will become part of the framework itself.

    Converting the 0.4.x application to 0.5.x

    The task of converting everything from 0.4 to 0.5 was a bit challenging. The reason behind it was the DI container and how best to use it to suit the needs of the current application. Now these challenges would not even be an issue if one started writing their application from scratch, but since I had everything in place, I ventured into upgrading vs. rewriting. Note that this kind of upgrade will most likely never happen again, since the framework has been changed accordingly so that future upgrades will not require developers to rewrite their code (like I did now). From 0.5.x onward the framework design has been kind of "frozen".

    I decided to create a new library that will help me with my tasks. I therefore created a custom bootstrap class, that would instantiate everything I wanted in my code. A short snippet of the class is below (the full code of course is in my Github repo which you are more than welcome to download and modify to suit your needs)

    namespace NDN;
    
    use \Phalcon\Config\Adapter\Ini as PhConfig;
    use \Phalcon\Loader as PhLoader;
    ....
    use \Phalcon\Exception as PhException;
    
    class Bootstrap
    {
        private $_di;
    
        /**
         * Constructor
         * 
         * @param $di
         */
        public function __construct($di)
        {
            $this->_di = $di;
        }
    
        /**
         * Runs the application performing all initializations
         * 
         * @param $options
         *
         * @return mixed
         */
        public function run($options)
        {
            $loaders = array(
                'config',
                'loader',
                'environment',
                'timezone',
                'debug',
                'flash',
                'url',
                'dispatcher',
                'view',
                'logger',
                'database',
                'session',
                'cache',
                'behaviors',
            );
    
    
            try {
                foreach ($loaders as $service)
                {
                    $function = 'init' . ucfirst($service);
    
                    $this->$function($options);
                }
    
                $application = new PhApplication();
                $application->setDI($this->_di);
    
                return $application->handle()->getContent();
    
            } catch (PhException $e) {
                echo $e->getMessage();
            } catch (\PDOException $e) {
                echo $e->getMessage();
            }
        }
    
        // Protected functions
    
        /**
         * Initializes the config. Reads it from its location and
         * stores it in the Di container for easier access
         *
         * @param array $options
         */
        protected function initConfig($options = array())
        {
            $configFile = ROOT_PATH . '/app/var/config/config.ini';
    
            // Create the new object
            $config = new PhConfig($configFile);
    
            // Store it in the Di container
            $this->_di->set('config', $config);
        }
    
        /**
         * Initializes the loader
         *
         * @param array $options
         */
        protected function initLoader($options = array())
        {
            $config = $this->_di->get('config');
    
            // Creates the autoloader
            $loader = new PhLoader();
    
            $loader->registerDirs(
                array(
                    ROOT_PATH . $config->app->path->controllers,
                    ROOT_PATH . $config->app->path->models,
                    ROOT_PATH . $config->app->path->library,
                )
            );
    
            // Register the namespace
            $loader->registerNamespaces(
                array("NDN" => $config->app->path->library)
            );
    
            $loader->register();
        }
    
        ....
    
        /**
         * Initializes the view and Volt
         *
         * @param array $options
         */
        protected function initView($options = array())
        {
            $config = $this->_di->get('config');
            $di     = $this->_di;
    
            $this->_di->set(
                'volt',
                function($view, $di) use($config)
                {
                    $volt = new PhVolt($view, $di);
                    $volt->setOptions(
                        array(
                            'compiledPath'      => ROOT_PATH . $config->app->volt->path,
                            'compiledExtension' => $config->app->volt->extension,
                            'compiledSeparator' => $config->app->volt->separator,
                            'stat'              => (bool) $config->app->volt->stat,
                        )
                    );
                    return $volt;
                }
            );
        }
        ....
    
        /**
         * Initializes the model behaviors
         *
         * @param array $options
         */
        protected function initBehaviors($options = array())
        {
            $session = $this->_di->getShared('session');
    
            // Timestamp
            $this->_di->set(
                'Timestamp',
                function() use ($session)
                {
                    $timestamp = new Models\Behaviors\Timestamp($session);
                    return $timestamp;
                }
            );
        }
    }
    

    I chose to show a few sections of this bootstrap which I will explain shortly. What this bootstrap class does is it initializes my whole environment and keeps my index.php file small.

    error_reporting(E_ALL);
    
    try {
    
        if (!defined('ROOT_PATH')) {
            define('ROOT_PATH', dirname(dirname(__FILE__)));
        }
    
        // Using require once because I want to get the specific
        // bootloader class here. The loader will be initialized
        // in my bootstrap class
        require_once ROOT_PATH . '/app/library/NDN/Bootstrap.php';
        require_once ROOT_PATH . '/app/library/NDN/Error.php';
    
        // Instantiate the DI container
        $di  = new \Phalcon\DI\FactoryDefault();
    
        // Instantiate the boostrap class and inject the DI container 
        // in it so that services can be registered
        $app = new \NDN\Bootstrap($di);
    
        // Here we go!
        echo $app->run(array());
    
    } catch (\Phalcon\Exception $e) {
        echo $e->getMessage();
    }
    

    As you can see the index.php is very small in terms of code.

    Let's have a look at a couple of the functions that are in the bootstrap.

        /**
         * Initializes the config. Reads it from its location and
         * stores it in the Di container for easier access
         *
         * @param array $options
         */
        protected function initConfig($options = array())
        {
            $configFile = ROOT_PATH . '/app/var/config/config.ini';
    
            // Create the new object
            $config = new PhConfig($configFile);
    
            // Store it in the Di container
            $this->_di->set('config', $config);
        }
    

    Pretty straight forward. The config INI file is read from its location and stored in the DI container. I need to do this first, since a lot of the parameters of the application are controlled from that file.

        /**
         * Initializes the loader
         *
         * @param array $options
         */
        protected function initLoader($options = array())
        {
            $config = $this->_di->get('config');
    
            // Creates the autoloader
            $loader = new PhLoader();
    
            $loader->registerDirs(
                array(
                    ROOT_PATH . $config->app->path->controllers,
                    ROOT_PATH . $config->app->path->models,
                    ROOT_PATH . $config->app->path->library,
                )
            );
    
            // Register the namespace
            $loader->registerNamespaces(
                array("NDN" => $config->app->path->library)
            );
    
            $loader->register();
        }
    

    The loader is what does all the discovery of classes for me. As you can see I store a lot of the paths in the config INI file, and I register my custom namespace NDN.

        /**
         * Initializes the view and Volt
         *
         * @param array $options
         */
        protected function initView($options = array())
        {
            $config = $this->di->get('config');
            $di     = $this->_di;
    
            $this->_di->set(
                'volt',
                function($view, $di) use($config)
                {
                    $volt = new PhVolt($view, $di);
                    $volt->setOptions(
                        array(
                            'compiledPath'      => ROOT_PATH . $config->app->volt->path,
                            'compiledExtension' => $config->app->volt->extension,
                            'compiledSeparator' => $config->app->volt->separator,
                            'stat'              => (bool) $config->app->volt->stat,
                        )
                    );
                    return $volt;
                }
            );
        }
    

    This is an interesting one. Registering the view and Volt. Volt is the template engine that comes with Phalcon. It is inspired by Twig and written in C, thus offering maximum performance. I set the compiled path, extension and separator for the template files and also I have a variable (set in the config of course) to allow the application to always create template files or not. In a production environment that variable (stat) will be set to false since templates do not change.

        /**
         * Initializes the model behaviors
         *
         * @param array $options
         */
        protected function initBehaviors($options = array())
        {
            $session = $this->_di->getShared('session');
    
            // Timestamp
            $this->_di->set(
                'Timestamp',
                function() use ($session)
                {
                    $timestamp = new Models\Behaviors\Timestamp($session);
                    return $timestamp;
                }
            );
        }
    

    The above is my implementation of behaviors. Of course it is far from perfect but it works the way I want to. A better implementation of this has been written by Wojtek Gancarczyk and is available in the incubator. All I do here is go through the behaviors I have (Timestamp only for now) and register them in the DI container so that I can reuse them later on with any model that needs them.

    Models

    Every model I have that interacts with my database tables extends the NDN\Model.

    class Model extends \Phalcon\Mvc\Model
    {
        protected $behaviors = array();
    
        /**
         * Adds a behavior in the model
         *
         * @param $behavior
         */
        public function addBehavior($behavior)
        {
            $this->behaviors[$behavior] = true;
        }
    
        public function beforeSave()
        {
            $di   = Di::getDefault();
    
            foreach ($this->behaviors as $behavior => $active)
            {
                if ($active && $di->has($behavior))
                {
                    $di->get($behavior)->beforeSave($this);
                }
            }
        }
    
        /**
         * @param array $parameters
         *
         * @static
         * @return Phalcon_Model_Resultset Model[]
         */
        static public function find($parameters = array())
        {
            return parent::find($parameters);
        }
    
        /**
         * @param array $parameters
         *
         * @static
         * @return  Phalcon_Model_Base   Models
         */
        static public function findFirst($parameters = array())
        {
            return parent::findFirst($parameters);
        }
    }
    

    The class itself is pretty simple, offering find and findFirst to the class that extends this. The interesting thing is that it also registers behaviors and calls the relevant validator function. So for instance the beforeSave validator checks the registered behaviors ($behaviors array), checks if they are active, checks if they exist in the DI container and gets them from there and then calls the beforeSave in the behavior class.

    The behavior class is equally simple:

    class Timestamp
    {
        protected $session;
    
        public function __construct($session)
        {
            $this->session= $session;
        }
    
        /**
         * beforeSave hook - called prior to any Save (insert/update)
         */
        public function beforeSave($record)
        {
            $auth     = $this->session->get('auth');
            $userId   = (isset($auth['id'])) ? (int) $auth['id'] : 0;
            $datetime = date('Y-m-d H:i:s');
            if (empty($record->created_at_user_id)) {
                $record->created_at         = $datetime;
                $record->created_at_user_id = $userId;
            }
            $record->last_update         = $datetime;
            $record->last_update_user_id = $userId;
        }
    }
    

    So effectively every time I call the save() function on a model, this piece of code will be executed, populating my fields with the date time and the user that created the record and/or updated it.

    In order to get this functionality to work, all I have to do in my model is to register the behavior like so:

    class Episodes extends \NDN\Model
    {
        /**
         * Initializes the class and sets any relationships with other models
         */
        public function initialize()
        {
            $this->addBehavior('Timestamp');
            $this->hasMany('id', 'Awards', 'episode_id');
        }
    }
    

    Controllers

    Very little has changed in the controller logic, so that was the easiest part of the upgrade. Of course I tweaked a few things but the code works as is. I still extended my custom NDN\Controller class which takes care of my breadcrumbs (NDN\Breadcrumbs) as well as the construction of the top menu. The biggest difference with the previous version is that I stopped using AngularJS to populate the menu (so I am no longer sending a JSON array in the view) and used Volt instead. It was a matter of preference and nothing more.

    Views

    Quite a bit of work had to be done in the views to switch everything to use Volt. Of course every view extension had to be changed to .volt but that was not the only change. I split the layout to use partials so that the header, navigation and footer are different sections (organizing things a bit better) and kept the master layout index.volt.

    I started using the built in Volt functions to generate content as well as tags and it was a nice surprise to see that everything was easy to use and it worked!

    <!DOCTYPE html>
    <html ng-app='HHF'>
        {{ partial('partials/header') }} 
        <body>
            <div id="spinner" style="display: none;">
                {{ image('img/ajax-loader.gif') }} Loading ...
            </div>
    
            {{ partial('partials/navbar') }}
    
            <div class='container-fluid'>
                <div class='row-fluid'>
                    <ul class='breadcrumb'>
                        <li>
                            {% for bc in breadcrumbs %}
                            {% if (bc['active']) %}
                            {{ bc['text'] }}
                            {% else %}
                            <a href='{{ bc['link'] }}'>{{ bc['text'] }}</a> 
                            <span class='divider'>/</span>
                            {% endif %}
                            {% endfor %}
                        </li>
                    </ul>
                </div>
    
                <?php echo $this->flash->output() ?>
    
                <div class="row-fluid">
                    <?php echo $this->getContent() ?>
                </div> <!-- row -->
    
                {{ partial('partials/footer') }}
            </div>
    
            {{ javascript_include(config.app.js.jquery, config.app.js.local) }}
            {{ javascript_include(config.app.js.jquery_ui, config.app.js.local) }}
            {{ javascript_include(config.app.js.bootstrap, config.app.js.local) }}
            {{ javascript_include(config.app.js.angular, config.app.js.local) }}
            {{ javascript_include(config.app.js.angular_resource, config.app.js.local) }}
            {{ javascript_include(config.app.js.angular_ui, config.app.js.local) }}
            {{ javascript_include('js/utils.js') }}
    
        </body>
    </html>
    

    The above is the index.volt. As you can see I call on the partials/header.volt, then the partials/navbar.volt (where the menu is generated) and then I construct the breadcrumbs (note the {% for bc in breadcrumbs %} block). After that the flash messenger comes into play, the main content displayed, the footer and finally the javascript includes that I need.

    I am still using AngularJS to make the necessary AJAX calls so that the relevant controller to retrieve the data but also to display this data on screen (which is cached to avoid unnecessary database hits).

    The Episodes view became

    {{ content() }}
    
    <div>
        <ul class='nav nav-tabs'>
            <li class='pull-right'>
                {{ addButton }}
            </li>
        </ul>
    </div>
    
    <div ng-controller='MainCtrl'>
        <table class='table table-bordered table-striped ng-cloak' ng-cloak>
            <thead>
            <tr>
                <th><a href='' ng-click="predicate='number'; reverse=!reverse">#</a></th>
                <th><a href='' ng-click="predicate='air_date'; reverse=!reverse">Date</a></th>
                <th><a href='' ng-click="predicate='outcome'; reverse=!reverse">W/L</a></th>
                <th><a href='' ng-click="predicate='summary'; reverse=!reverse">Summary</a></th>
            </tr>
            </thead>
            <tbody>
                <tr ng-repeat="episode in data.results | orderBy:predicate:reverse">
                    <td>[[episode.number]]</td>
                    <td width='7%'>[[episode.air_date]]</td>
                    <td>[[episode.outcome]]</td>
                    <td>[[episode.summary]]</td>
                    {% if (addButton) %}
                    <td width='1%'><a href='/episodes/edit/[[episode.id]]'><i class='icon-pencil'></i></a></td>
                    <td width='1%'><a href='/episodes/delete/[[episode.id]]'><i class='icon-remove'></i></a></td>
                    {% endif %}
                </tr>
            </tbody>
        </table>
    </div>
    

    The beauty of AngularJS! I only have to pass a JSON array with my results. ng-repeat with the orderBy filter allows me to present the data to the user and offer sorting capabilities per column. This is all done at the browser level without any database hits! Pretty awesome feature!

    For those that have used AngularJS in the past, you will note that I had to change the interpolate provider (i.e. the characters that wrap a string or a piece of code that AngularJS understands). Usually these characters are the curly brackets {{ }} but I changed them to [[ ]] to avoid collisions with Volt.

    This was done with a couple of lines of code in my definition of my AngularJS model:

    var ngModule = angular.module(
            'HHF', 
            ['ngResource', 'ui']
        )
        .config(
            function ($interpolateProvider) {
                $interpolateProvider.startSymbol('[[');
                $interpolateProvider.endSymbol(']]');
            }
        )
    

    Conclusion

    I spent at most a day working on this mostly because I wanted to try various things and see how it works. The actual time to convert the application (because let's face it, it is a small application) was a couple of hours inclusive of the time it took me to rename certain fields, restructure the folder structure, compile the new extension on my server and upload the data upstream.

    I am very satisfied with both AngularJS, which helps tremendously in my presentation layer, as well as with Phalcon. Phalcon's new design makes implementation a breeze, while AngularJS offers a lot of flexibility on the view layer.

    As written before, you are more than welcome to download the source code of this application here and use it for your own needs. Some resources are:

    References

  • 2012-09-02 12:00:00

    AngularJS - Simplicity in your browser

    Recently I was contacted by an acquaintance through my Google+ circles, who needed some help with a project of hers.

    Her task was to redesign a church website. Pretty simple stuff, CSS, HTML and content.

    Scope

    The particular church videotapes all the sermons and posts them on their channel in LiveStream for their followers to watch. One of the requirements was to redo the video archives page and to offer a link where followers can download the audio of each sermon for listening.

    Design (kinda)

    After the initial contact, I decided to get rid of all the bloated jQuery code that was there to control the video player and use AngularJS to control the generation of content. There were two key facts that influenced my decision:

    • the use of ng-repeat to generate the table that will list all the available sermons and
    • the variable binding that AngularJS offers to play the video in the available player.

    I also decided to switch the player to a new updated one that LiveStream offered, which features a slider to jump through the video, volume control and more.

    Previous code

    The previous code for that page was around 300 lines. The file had some CSS in it, quite a few lines of HTML but was heavy on javascript. There were a lot of jQuery functions which controlled the retrieval of the available videos per playlist. Each playlist would be effectively a collection of videos for a particular year. jQuery was observing clicks on specific links and make an AJAX call to the LiveStream API to retrieve the list of available data in JSON format, and output the formatted results (as a table) on screen. It was something like this:

    head
    title
    link - css
    style (inline)
    ....
    end inline style
    script jQuery
    script jQueryUI
    script jquery.timer
    link jquery-ui CSS
    link slider CSS
    style (inline)
    ....
    end inline style
    script jQuery
    $(document).ready ... // Document ready
    ....
    $("#div_col dd").click // Bind click to a year
    .....
    getPlaylists(year) // Not sure why this was ever here, not used
    ....
    getPlaylistClips(playlistID) // Gets the clips of the playlist
    .....
    playClip(clipID) // Plays the clip in the player
    .....
    end jQuery script
    script Video Player
    ....
    end head
    body
    navigation
    main container
    list of years
    instructions
    container to show video player
    container to show video list
    end body
    end html
    

    Enter AngularJS

    I checked the latest video player from LiveStream. The code was much cleaner and all I had to do is bind one variable, the GUID of the video, in the relevant call so that the video can be played. I also bound another variable (the video title) above the video so as to offer more information to the user.

    With a bit of Bootstrap CSS, I created two tabs and listed the two years 2012, 2011. A function was created in my AngularJS module to accept the year and make the relevant call to the LiveStream API to receive the data a a JSON object.

    ng-repeat (with >ng-cloak) was used to "print" the data on screen and the application was ready.

    I removed all the cruft and created one small file that is loaded and offers the functionality that we need. It is 50 lines of code (just the javascript part. The code is below with added comments for the reader to follow:

    // Create the module and inject the Resource object
    var ngModule = angular.module("CHF", ['ngResource']);
    
    // The main controller that needs the scope and resource
    ngModule.controller("MainCtrl", function ($scope, $resource) {
    
        // Calculates the current year
        //  ensures we always get the last year on first load
        $scope.currentYear = function () {
            var currentDate = new Date();
            return currentDate.getFullYear();
        };
    
        // This is the playlist array. This is obtained by 
        //  LiveStream and it changes once every year. 
        //  Hardly an effort by the administrator
        $scope.playlists = [
            {"year":"2012", "guid":"63426-xxx-xxx-xxx"},
            {"year":"2011", "guid":"84f84-xxx-xxx-xxx"}
        ];
    
        // This couldn't be simpler. It merely sets some variables 
        //  in the scope. By doing so, the binding in the relevant
        //  variables will allow the video to play and the title 
        //  to update.
        $scope.playVideo = function (element) {
            $scope.currentVideo  = element.guid;
            $scope.currentTitle  = element.title;
        };
    
        // This is the core. It makes the AJAX request to the 
        // LiveStream API so that it can get the JSON data back
        $scope.makeRequest = function (year) {
    
            // Calculating the current year and the year selected.
            // Their difference offers an offset which effectively 
            // is the offset of the array stored in $scope.playlists
            var thisYear    = $scope.currentYear();
            var diff        = thisYear - year;
    
            var objData = $scope.playlists[diff];
    
            // Just in case something was passed that is not valid
            if (objData.guid)
            {
                var reqData = $resource(
                    "http://livestream_url/2.0/:action",
                    {
                        action:'listclips.json', 
                        id:objData.guid,
                        query: {isArray: true},
                        maxresults:"500",
                        callback:'JSON_CALLBACK'
                    },
                    {get:{method:'JSONP'}}
                );
    
                // Set the year and get the data
                $scope.year    = year;
                $scope.listData = reqData.get();
           }
        };
    
        // This is the first load - load the current year
        $scope.makeRequest($scope.currentYear());
    
    });
    

    Now moving into the HTML side of things:

    <div id="playerContainer" style='text-align:center;'>
        <p ng-cloak>
            {{currentTitle}}
        </p>
        <iframe 
            width="560" 
            height="340" 
            src="http://ls_url?clip={{currentVideo}}&params" 
            style="border:0;outline:0" 
            frameborder="0" 
            scrolling="no">
        </iframe>
    </div>
    
    <br />
    <div>
        <ul class="nav nav-tabs">
            <li ng-repeat="playlist in playlists" 
                   ng-class="{true:'active',false:''}[year == playlist.year]">
                <a ng-click="makeRequest(playlist.year)">{{playlist.year}}</a>
            </li>
        </ul>
        <table class="table table-bordered" style="width: 100%;">
            <thead>
                <tr>
                    <th>Date/Title</th>
                    <th>Audio</th>
                </tr>
            </thead>
            <tbody>
                <tr ng-cloak ng-repeat="video in listData.channel.item">
                    <td ng-click="playVideo(video)">{{video.title}}</td>
                    <td>
                        <span ng-show="video.description">
                            <a href="{{video.description}}" title="Download Audio">
                                <i class="icon-download-alt"></i>
                            </a>
                        </span>
                    </td>
                </tr>
            </tbody>
        </table>
    </div>
    

    That is all the HTML I had to change. The full HTML file is 100 lines and 50 for the AngularJS related javascript, I can safely say that I had a 50% reduction in code offering the same functionality - and if I might say so, it is much much cleaner.

    The final page looks something like this:

    Pointers

    <div id="playerContainer" style='text-align:center;'>
        <p ng-cloak>
        {{currentTitle}}
        </p>
        <iframe 
            width="560" 
            height="340" 
            src="http://ls_url?clip={{currentVideo}}&params" 
            style="border:0;outline:0" 
            frameborder="0" 
            scrolling="no">
        </iframe>
    </div>
    

    This block displays the video player and due to the variable binding that AngularJS offers, the minute those variables change, the video is ready to be played.

    <ul class="nav nav-tabs">
        <li ng-repeat="playlist in playlists" 
               ng-class="{true:'active',false:''}[year == playlist.year]">
            <a ng-click="makeRequest(playlist.year)">{{playlist.year}}</a>
        </li>
    </ul>
    

    This block shows the tabs depicting each playlist. In our case these are years. ng-repeat does all the hard work, printing the data that is defined in our JS file. The ng-class is there to change the class of the tab to "active" when the tab is clicked/selected. The ng-click initiates a request through makeRequest, a function defined in our javascript file (see above).

    <tbody>
        <tr ng-cloak ng-repeat="video in listData.channel.item">
            <td ng-click="playVideo(video)">{{video.title}}</td>
            <td>
                <span ng-show="video.description">
                    <a href="{{video.description}}" 
                       title="Download Audio">
                        <i class="icon-download-alt"></i>
                    </a>
                </span>
            </td>
        </tr>
    </tbody>
    

    Finally the data is displayed on screen. ng-cloak makes sure that the content is displayed only when the data is there (otherwise browsers might show something like {{video.description}} which is not nice from a UI perspective). ng-repeat loops through the data and "prints" the table.

    The description of the video is used as a storage for the URL that will point to the MP3 audio file so as the users can download it. Therefore I use ng-show to show the link, if it exists.

    Conclusion

    This whole project was no more than 30 minutes, which included the time I had to research and experiment a bit with the LiveStream API. This is a flexible design, with much much cleaner code (and a lot less of it). When the admin needs to add a new playlist (aka year), all they have to do is open the JS file and type a new element in the $scope.playlists array. The application will take care of the rest automatically.

    I cannot think of doing this with less lines of code than this.

    If you haven't heard of AngularJS or used it, I would highly encourage you to give it a shot. Great project, awesome support and a very very responsive, helpful and polite community.

  • 2012-07-12 12:00:00

    Building a web app with PhalconPHP and AngularJS Part II

    This is Part II of a series of posts on building an application using Phalcon and AngularJS. Part I is located here.

    Preface

    I have recently discovered Phalcon and I was impressed with its speed and ease of use. At the time of this writing, PhalconPHP is at version 0.4.2, with some serious redesign coming down the line on 0.5.x.

    Phalcon takes a different approach than any other PHP framework (see Zend, Symfony, CakePHP etc.). It is written in C and compiled as a module which is then loaded on your web server. Effectively the whole framework is in memory for you to use, without needing to access the file system so that you can include a file here or a file there.

    Advantages

    The core advantage of this approach is speed. The framework is in memory, ready to deliver its functionality, so your application is now only concerned about its files and not the framework itself. Once a framework is mature enough for usage, its files don't change that much. Yet for any of the traditional frameworks, PHP needs to scan the files, load them and the interpret them. This in effect has a serious impact on performance, especially for large projects.

    Another advantage is that since the framework is a module on your web server, you don't need to upload library files to each and every application you install on your host.

    Finally, you can mix and match whatever you need, using any of the components as 'glue' components rather than the whole framework. Most of the major frameworks also use this methodology for most of their components, however performance still is an issue. Additionally, in the case of any other framework, one might need to upload a very complicated and deep file structure on their web server so as to take advantage of one component to be used in an application.

    Disadvantages

    Support and bug tracing are the two weaknesses of Phalcon. By support I do not mean support from the developers. On the contrary, the developers are doing a great job listening to the relatively young community, and issuing fixes. However, as with any framework, if you find a bug, you will try to trace the code back to each component in an effort to find a solution to your problem. When developing an application and have access to the source files (the library PHP files like Zend Framework has), not only you can learn from those implementations, but you can quickly fix something that might be broken and continue working. With Phalcon you will need to wait until the next version is released, unless you are fluent in C and play around with the source code. For most PHP programmers (like myself), the process will be report the bug and wait for the fix.

    Since the framework is a module on your web server, you will need to be careful on upgrades. If your applications do not take advantage of the latest functionality the framework offers, you might fix something in one application, while breaking something in another. You cannot mix and match versions of Phalcon per application.

    Consideration

    Phalcon is very young as a framework. It does have a lot of power, but there are a lot of things still missing (for instance relationships between models and a query builder). In time these pieces will be implemented and the framework will grow stronger :)

    Implementation

    I downloaded the INVO sample application and set it up on my web browser. Using that as a starting point, I started modifying it to fit my needs. I also set up the PhalconPHP developer tools and PHPStorm support.

    For this application, I needed a table to store information about every podcast episode, a table to store all players and a table to store the users (namely Aaron, Josh and John). The Awards table would be the one that would store all the information regarding the game balls and kick in the balls awards.

    Models

    Once those were in place I started building my models and relevant controllers/views. Setting a model up was really easy. I would create the table in my database and then run

    phalcon create-model --table-name episodes
    

    and my model would be ready for me to use (example below for Episodes).

    class Episodes extends Phalcon_Model_Base 
    {
        public $id;
        public $number;
        public $summary;
        public $airDate;
        public $outcome;
        public $createdAt;
        public $createdAtUserId;
        public $lastUpdate;
        public $lastUpdateUserId;
    }
    

    After a while I decided I wanted to keep a track on who created a record and when, and who last updated a record and when for certain tables. After some refactoring I created my own model class that would give me the functionality I needed, and extended that class in relevant models.

    My custom class (that would take care of the createdAt, createdAtUserId, lastUpdated, lastUpdatedUserId fields) also took advantage of the beforeSave hook to ensure that these fields were transparently updated. The find and findFirst static functions are used throughout the models and there is no reason to repeat them in each model, so they end up in this custom class. (Comments removed to preserve space)

    use NDN_Session as Session;
    
    class NDN_Model extends Phalcon_Model_Base
    {
        public $createdAt;
        public $createdAtUserId;
        public $lastUpdate;
        public $lastUpdateUserId;
    
        public function beforeSave()
        {
            if (empty($this->createdAtUserId)) {
                $auth     = Session::get('auth');
                $datetime = date('Y-m-d H:i:s');
    
                $this->createdAt        = $datetime;
                $this->createdAtUserId  = (int) $auth['id'];
            }
        }
    
        static public function find($parameters = array())
        {
            return parent::find($parameters);
        }
    
        static public function findFirst($parameters = array())
        {
            return parent::findFirst($parameters);
        }
    }
    
    Session

    Although Phalcon provides a flash messenger utility, I had an issue with using the _forward function on a controller, after an action (say Add or Edit) was completed. Effectively the data would not refresh on screen. To combat that I used _redirect. However, all the messages that I had in the flash messenger (Phalcon_Flash) would disappear. An easy solution was to extend the Phalcon_Session and create two new functions setFlash and getFlash. The setFlash is called whenever I want to set a message for the user to see. The function stores the message in a session variable. Before the controller is dispatched, the getFlash is called to return any messages waiting to be displayed, and after that the messages are cleared from the session and displayed on screen.

    class NDN_Session extends Phalcon_Session
    {
        public static function setFlash($class, $message, $css)
        {
            $data = array(
                'class'   => $class,
                'message' => $message,
                'css'     => $css,
            );
            self::set('flash', $data);
        }
    
        public static function getFlash()
        {
            $data = self::get('flash');
            if (is_array($data)) {
                self::remove('flash');
                return $data;
            } else {
                return null;
            }
        }
    }
    
    Breadcrumbs

    I wanted to show breadcrumbs to the user, as a way to easily navigate throughout the application. To do so, I created my own Breadcrumbs class which holds an array of areas that the user is in. The class has a generate function, which returns back a JSON string. This is to be parsed by AngularJS so as to display the breadcrumbs.

    Controllers

    I created my controllers using the Phalcon Developer Tools. Whether you use the webtools or the command line makes no difference. The skeleton of the controller is generated for you to use.

    Based on the flash messenger and _redirect that I mentioned in the previous section, I had to extend the base controller, so as to add functionality that would allow me to show messages on screen after a redirect. Other reasons for this new class were to allow for a prefix on each page title, generate breadcrumbs and menus.

    use Phalcon_Tag as Tag;
    
    use Phalcon_Flash as Flash;
    use NDN_Session as Session;
    
    class NDN_Controller extends Phalcon_Controller 
    {
        protected $_bc = null;
        public function initialize()
        {
            Tag::prependTitle('HHF G&KB Awards | ');
            $this->_bc = new NDN_Breadcrumbs();
        }
    
        public function beforeDispatch()
        {
            $message = Session::getFlash();
            if (is_array($message)) {
                Flash::$message['class'](
                    $message['message'], $message['css']
                );
            }
            $this->view->setVar('breadcrumbs', $this->_bc->generate());
        }
    
        protected function _constructMenu($controller)
        {
            $commonMenu = array(
                'index'      => 'Home', 
                'awards'     => 'Awards', 
                'players'    => 'Players', 
                'episodes'   => 'Episodes', 
                'about'      => 'About', 
                'contact'    => 'Contact Us', 
            ); 
            $auth = Session::get('auth'); 
    
            $class  = get_class($controller); 
            $class  = str_replace('Controller', '', $class); 
            $active         = strtolower($class); 
            $sessionCaption = ($auth) ? 'Log Out'         : 'Log In'; 
            $sessionAction  = ($auth) ? '/session/logout' : '/session/index'; 
    
            $leftMenu = array(); 
            foreach ($commonMenu as $link => $text) { 
                $isActive   = (bool) ($active == $link); 
                $newLink  = ('index' == $link) ? '/' : '/' . $link; 
                $leftMenu[] = array( 
                    'active' => $isActive, 
                    'link'   => $newLink, 
                    'text'   => $text, 
                ); 
            } 
    
            $menu = new StdClass(); 
            $menu->current = $active; 
            $menu->left    = $leftMenu; 
    
            if ($auth != false) { 
                $sessionCaption .= ' ' . $auth['name']; 
            } 
    
            $menu->rightLink = $sessionAction; 
            $menu->rightText = $sessionCaption; 
    
            return json_encode($menu); 
        } 
    }
    

    Each controller would extend my base controller. In the initialize function:

    • the page title is set,
    • the breadcrumbs are added (and generated later on in the beforeDispatch of the base controller),
    • the menu is generated and passed to the view for AngularJS to process,
    • additional variables would be generated for displaying elements based on whether a user is logged in or not.
    Views

    Creating the views was really easy. I already had the structure ready from the sample application (INVO) and with the help of Bootstrap CSS, I was done in no time. The views inherit from a base view (index.phtml) located at the root of the views folder. That view holds the skeleton of the web page and content is injected accordingly based on each controller (and its view).

    In that file I added the relevant variables that will be used by AngularJS as well as variables that hold conditional elements (i.e. elements that appear when a user is logged in).

    More on the views in the next installment of these series.

    Conclusion

    With all that the application was ready as far as the main structure was concerned. Tying everything with AngularJS was the next step, which will be covered in part III of this How-To.

    The whole application, from start to finish, took less than 4 hours to develop. This included breaks, reading the manual and making design decisions based on my ever changing requirements.

    References

  • 2012-07-10 12:00:00

    Building a web app with PhalconPHP and AngularJS Part I

    There are ample frameworks on the Internet, most free, that a programmer can use to build a web application. Two of these frameworks are PhalconPHP and AngularJS.

    I decided to use those two frameworks and build a simple application which will keep track of the Game Balls and Kick in the Balls awards of Harry Hog Football.

    Harry Hog Football is a podcast that has been going strong since 2005, created by Redskins fans for Redskins Fans. (for those that do not know, Washington Redskins is a team on the National Football League in the USA).

    Every week during the regular season, Aaron, Josh and John create a podcast, where they discuss the recent game, the injuries, the cuts, the new signings and they offer their Game Balls to the best players of the week as well as the Kick in the Balls awards for the ones that (according to the podcasters) 'suck'.

    I therefore created an application to record all those game balls and kick in the balls awards, so that we can all see, who is the most valuable player and who is the least valuable player for the Redskins throughout the years (the term valuable is used loosely here).

    As a starting point I used the INVO application that PhalconPHP showcases as an easy application to get you started. I modified it significantly to address my needs, refactoring classes as much as possible to get the least amount of code with maximum usability.

    After building the application, I listened to all the episodes I could find, and entered the game balls and kick in the balls in the database. The models use the PhalconModelBase class to handle data, while the rest of the application is handled by the Phalcon_Controller (and view of course).

    The data transfer between the application and the relevant sections is primarily handled by AngularJS, which is dominant in the view layer. AngularJS controllers handle menu creation, breadcrumbx as well as displaying results on screen.

    Twitter's Bootstrap CSS> is used to put the final touches for the application.

    In subsequent posts I will explain each layer in turn, starting with PhalconPHP and continuing with AngularJS.

    This of course is by no means the perfect implementation. It has been a fun project for me, working on it on my own free time. You are more than welcome to fork the project and make any modifications you need. For those that are interested in getting straight to the code, it is available on Github here.

    NOTE: The Github repository contains code that works with nginx. If you are having problems with Apache, check the public/index.php - there is a note there for nginx (probably will need to remove it)

    References

  • 2012-07-04 12:00:00

    2012 4th of July - Independence Day

    Today we celebrate the Declaration of Independence in the United States, also known widely as the 4th of July.

    During this day in 1776, delegates of 13 states signed a document which serves as the foundation of the Constitutional Republic, the United States of America.

    Happy 4th of July!

    The transcript is below

    The Declaration of Independence

    IN CONGRESS, July 4, 1776.

    The unanimous Declaration of the thirteen united States of America,

    When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.

    We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.--That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed, --That whenever any Form of Government becomes destructive of these ends, it is the Right of the People to alter or to abolish it, and to institute new Government, laying its foundation on such principles and organizing its powers in such form, as to them shall seem most likely to effect their Safety and Happiness. Prudence, indeed, will dictate that Governments long established should not be changed for light and transient causes; and accordingly all experience hath shewn, that mankind are more disposed to suffer, while evils are sufferable, than to right themselves by abolishing the forms to which they are accustomed. But when a long train of abuses and usurpations, pursuing invariably the same Object evinces a design to reduce them under absolute Despotism, it is their right, it is their duty, to throw off such Government, and to provide new Guards for their future security.--Such has been the patient sufferance of these Colonies; and such is now the necessity which constrains them to alter their former Systems of Government. The history of the present King of Great Britain is a history of repeated injuries and usurpations, all having in direct object the establishment of an absolute Tyranny over these States. To prove this, let Facts be submitted to a candid world.

    • He has refused his Assent to Laws, the most wholesome and necessary for the public good.
    • He has forbidden his Governors to pass Laws of immediate and pressing importance, unless suspended in their operation till his Assent should be obtained; and when so suspended, he has utterly neglected to attend to them.
    • He has refused to pass other Laws for the accommodation of large districts of people, unless those people would relinquish the right of Representation in the Legislature, a right inestimable to them and formidable to tyrants only. 
    • He has called together legislative bodies at places unusual, uncomfortable, and distant from the depository of their public Records, for the sole purpose of fatiguing them into compliance with his measures. 
    • He has dissolved Representative Houses repeatedly, for opposing with manly firmness his invasions on the rights of the people.
    • He has refused for a long time, after such dissolutions, to cause others to be elected; whereby the Legislative powers, incapable of Annihilation, have returned to the People at large for their exercise; the State remaining in the mean time exposed to all the dangers of invasion from without, and convulsions within.
    • He has endeavoured to prevent the population of these States; for that purpose obstructing the Laws for Naturalization of Foreigners; refusing to pass others to encourage their migrations hither, and raising the conditions of new Appropriations of Lands.
    • He has obstructed the Administration of Justice, by refusing his Assent to Laws for establishing Judiciary powers.
    • He has made Judges dependent on his Will alone, for the tenure of their offices, and the amount and payment of their salaries.
    • He has erected a multitude of New Offices, and sent hither swarms of Officers to harrass our people, and eat out their substance.
    • He has kept among us, in times of peace, Standing Armies without the Consent of our legislatures.
    • He has affected to render the Military independent of and superior to the Civil power.
    • He has combined with others to subject us to a jurisdiction foreign to our constitution, and unacknowledged by our laws; giving his Assent to their Acts of pretended Legislation:
    • For Quartering large bodies of armed troops among us:
    • For protecting them, by a mock Trial, from punishment for any Murders which they should commit on the Inhabitants of these States:
    • For cutting off our Trade with all parts of the world:
    • For imposing Taxes on us without our Consent: 
    • For depriving us in many cases, of the benefits of Trial by Jury:
    • For transporting us beyond Seas to be tried for pretended offences
    • For abolishing the free System of English Laws in a neighbouring Province, establishing therein an Arbitrary government, and enlarging its Boundaries so as to render it at once an example and fit instrument for introducing the same absolute rule into these Colonies:
    • For taking away our Charters, abolishing our most valuable Laws, and altering fundamentally the Forms of our Governments:
    • For suspending our own Legislatures, and declaring themselves invested with power to legislate for us in all cases whatsoever.
    • He has abdicated Government here, by declaring us out of his Protection and waging War against us.
    • He has plundered our seas, ravaged our Coasts, burnt our towns, and destroyed the lives of our people.
    • He is at this time transporting large Armies of foreign Mercenaries to compleat the works of death, desolation and tyranny, already begun with circumstances of Cruelty & perfidy scarcely paralleled in the most barbarous ages, and totally unworthy the Head of a civilized nation.
    • He has constrained our fellow Citizens taken Captive on the high Seas to bear Arms against their Country, to become the executioners of their friends and Brethren, or to fall themselves by their Hands. 
    • He has excited domestic insurrections amongst us, and has endeavoured to bring on the inhabitants of our frontiers, the merciless Indian Savages, whose known rule of warfare, is an undistinguished destruction of all ages, sexes and conditions.

    In every stage of these Oppressions We have Petitioned for Redress in the most humble terms: Our repeated Petitions have been answered only by repeated injury. A Prince whose character is thus marked by every act which may define a Tyrant, is unfit to be the ruler of a free people.

    Nor have We been wanting in attentions to our British brethren. We have warned them from time to time of attempts by their legislature to extend an unwarrantable jurisdiction over us. We have reminded them of the circumstances of our emigration and settlement here. We have appealed to their native justice and magnanimity, and we have conjured them by the ties of our common kindred to disavow these usurpations, which, would inevitably interrupt our connections and correspondence. They too have been deaf to the voice of justice and of consanguinity. We must, therefore, acquiesce in the necessity, which denounces our Separation, and hold them, as we hold the rest of mankind, Enemies in War, in Peace Friends.

    We, therefore, the Representatives of the united States of America, in General Congress, Assembled, appealing to the Supreme Judge of the world for the rectitude of our intentions, do, in the Name, and by Authority of the good People of these Colonies, solemnly publish and declare, That these United Colonies are, and of Right ought to be Free and Independent States; that they are Absolved from all Allegiance to the British Crown, and that all political connection between them and the State of Great Britain, is and ought to be totally dissolved; and that as Free and Independent States, they have full Power to levy War, conclude Peace, contract Alliances, establish Commerce, and to do all other Acts and Things which Independent States may of right do. And for the support of this Declaration, with a firm reliance on the protection of divine Providence, we mutually pledge to each other our Lives, our Fortunes and our sacred Honor.

    The 56 signatures on the Declaration appear in the positions indicated:

    Georgia
    Button Gwinnett
    Lyman Hall
    George Walton

    North Carolina
    William Hooper
    Joseph Hewes
    John Penn

    South Carolina
    Edward Rutledge
    Thomas Heyward, Jr.
    Thomas Lynch, Jr.
    Arthur Middleton

    Massachusetts
    John Hancock

    Maryland
    Samuel Chase
    William Paca
    Thomas Stone
    Charles Carroll of Carrollton

    Virginia
    George Wythe
    Richard Henry Lee
    Thomas Jefferson
    Benjamin Harrison
    Thomas Nelson, Jr.
    Francis Lightfoot Lee
    Carter Braxton

    Pennsylvania
    Robert Morris
    Benjamin Rush
    Benjamin Franklin
    John Morton
    George Clymer
    James Smith
    George Taylor
    James Wilson
    George Ross

    Delaware
    Caesar Rodney
    George Read
    Thomas McKean

    New York
    William Floyd
    Philip Livingston
    Francis Lewis
    Lewis Morris

    New Jersey
    Richard Stockton
    John Witherspoon
    Francis Hopkinson
    John Hart
    Abraham Clark

    New Hampshire
    Josiah Bartlett
    William Whipple
    Matthew Thornton

    Massachusetts
    Samuel Adams
    John Adams
    Robert Treat Paine
    Elbridge Gerry

    Rhode Island
    Stephen Hopkins
    William Ellery

    Connecticut
    Roger Sherman
    Samuel Huntington
    William Williams
    Oliver Wolcott

  • 2012-06-29 12:00:00

    How to build an inexpensive RAID for storage and streaming

    Overview

    I have written about this before and it is still my favorite mantra

    There are people that take backups and there are people that never had a hard drive fail

    This mantra is basically what should drive everyone to take regular backups of their electronic files, so as to ensure that nothing is lost in case of hardware failures.

    Let's face it. Hard drives or any magnetic media are man made and will fail at some point. When is what we don't know, so we have to be prepared for it.

    Services like Google Drive, iCloud, Dropbox, BoundlessCloud and others offer good backup services, ensuring that there is at least one safe copy of your data. But that is not enough. You should ensure that whatever happens, the memories stored in your pictures or videos, the important email communications, the important documents are all kept in a safe place and there are multiple backups of it. Once they are gone, they are gone for good, so backups are the only way to ensure that this does not happen.

    Background

    My current setup at home consists of a few notebooks, a mac-mini and a Shuttle computer with a 1TB hard drive, where I store all my pictures, some movies and my songs. I use Google Music Manager for my songs so that they are available at any time on my android phone, Picasa> to be able to share my photos with my family and friends and Google Drive so as to keep every folder I have in sync. I also use RocksBox to stream some of that content (especially the movies) upstairs on either of our TVs through the Roku boxes we have.

    Recently I went downstairs and noticed that the Shuttle computer (which run Windows XP at the time) was stuck in the POST screen. I rebooted the machine but it refused to load Windows, getting stuck either in the computer's POST screen or in the Starting Windows.

    Something was wrong and my best guess was the hard drive was failing. I booted up then in Safe mode (and that worked), set the hard drive to be checked for defects and rebooted again to let the whole process finish. After an hour or so, the system was still checking the free space and was stuck at 1%. The hard drive was making some weird noises so I shut the system down.

    Although I had backups of all my files on the cloud through Picasa, Google Music Manager and Google Drive, I still wanted to salvage whatever I had in there just in case. I therefore booted up the system with a Linux live CD, mounted the hard drive and used FileZilla to transfer all the data from the Shuttle's hard drive to another computer. There was of course a bit of juggling going on since I had to transfer data in multiple hard drives due to space restrictions.

    Replacing the storage

    I had to find something very cheap and practical. I therefore went to Staples and found a very nice (for the money) computer by Lenovo. It was only $300, offering 1TB 7200 SATA hard drive, i5 processor and 6GB of RAM.

    As soon as I got the system I booted it up, started setting everything up and before the end of the day everything was back in place, synched to the cloud.

    However the question remained: what happens if the hard drive fails again? Let's face it, I did lose a lot of time trying to set everything up again so I wasn't prepared to go through that again.

    My solution was simple:

    Purchase a cheap RAID (hardware) controller, an identical 1TB hard drive to the one I have, and a second hard drive to have the OS on. This way, the two 1TB hard drives can be connected to the RAID controller on a mirror setup (or RAID 1), while the additional hard drive can keep the operating system.

    I opted for a solid state drive from Crucial for the OS. Although it was not necessary to have that kind of speed, I thought it wouldn't hurt. It did hurt my pocket a bit but c'est la vie. For your own setup you can choose whatever you like.

    Hardware

    NOTE : For those that are not interested in having a solid state drive for the OS, one can always go with other, much cheaper drives such as this one.

    Setup

    After all the components arrived, I opened the computer and had a look at what I am facing with. One thing I did not realize was the lack of space for the third hard drive (the one that will hold the OS). I was under the impression that it would fit under the DVD ROM drive, but I did not account for the SD card reader that was installed in that space, so I had to be a bit creative (Picture 1).

    A couple of good measurements and two holes with the power drill created a perfect mounting point for the solid state hard drive. It is sitting now secure in front of the card reader connections, without interfering in any way.

    The second hard drive and the raid card were really easy to install, just a couple of screws and everything was set in place.

    The second hard drive ended up in the only expansion 'bay' available for this system. This is below the existing drive, mounted on the left side of the case. The actual housing has guides that allow you to slide the drive until the screw holes are aligned and from there it is a two minute job to secure the drive in place.

    I also had a generic nVidia 460 1GRAM card, which I also installed in the system. This was not included in the purchases for building this system, but it is around $45 if not less now. I have had it for over a year now and it was installed in the old Shuttle computer, so I wasn't going to let it go to waste.

    With everything in place, all I had to do is boot the system up and enter the BIOS screen so as to ensure that the SSD drive had a higher priority than any other drive.

    Once that was done, I put the installation disks in the DVD-ROM and restored the system on the SSD drive. 4 DVDs later (around 30 minutes) the system was installed and booted up. It took however another couple of hours until I had everything set up. The myriad of Windows Updates, (plus my slow Internet connection) contributed to this delay. However I have to admit, that the SSD drive was a very good purchase, since I have never seen Windows boot in less than 10 seconds (from power up to the login screen).

    The Windows updates included the nVidia driver so everything was set up (well almost that is). The only driver not installed was for the HighPoint RaidRocket RAID controller.

    The installation disk provided that driver, alongside with a web based configuration tool. After installing the driver and a quick reboot, the RAID configuration tool was not easy to understand but I figured it out, even without reading the manual.

    Entering the Disk Manager, I initialized and formatted the drive and from then on, I started copying all my files in there.

    As a last minute change, I decided not to install RocksBox and instead go with Plex Media Server. After playing around with Plex, I found out that it was a lot easier to setup than RocksBox (RocksBox requires a web server to be installed on the server machine, whereas Plex automatically discovers servers). Installing the relevant channel on my Roku boxes was really easy and everything was ready to work right out of the box so to speak.

    Problems

    The only problem that I encountered had to do with Lenovo itself. I wanted basically to install the system on the SSD drive. Since the main drive is 1TB and the SSD drive 128GB I could not use CloneZilla or Image for Windows to move the system from one drive to another. I tried almost everything. I shrank the 1TB system partition so as to make it fit in the 128GB drive. I had to shut hibernation off, reboot a couple of times in Safe Mode to remove unmovable files, in short it was a huge waste of time.

    Since Lenovo did not include the installation disks (only an applications DVD), I called their support line and inquired about those disks. I was sent from the hardware department to the software department, where a gentleman informed me that I have to pay $65 to purchase the OS disks. You can imagine my frustration to the fact that I had already paid for the OS by purchasing the computer. We went back and forth with the technician and in the end got routed to a manager who told me I can create the disks myself using Lenovo's proprietary software.

    The create rescue process required 10 DVDs, so I started the process. On DVD 7 the process halted. I repeated the process, only to see the same error on DVD 4. The following day I called Lenovo hardware support and managed to talk to a lady who was more than willing to send me the installation disks free of charge. Unfortunately right after I gave her my address, the line dropped, so I had to call again.

    The second phone call did not go very well. I was transferred to the software department again, where I was told that I have to pay $65 for the disks. The odd thing is that the technician tried to convince me that Lenovo actually doesn't pay money to Microsoft since they get an OEM license. Knowing that this is not correct, and after the fact that the technician was getting really rude, I asked to speak to a supervisor. The supervisor was even worse and having already spent 45 minutes on the phone, I asked to be transferred to the hardware department again. Once there, I spoke to another lady, explained the situation and how long I have been trying to get this resolved (we are at 55 minutes now) and she happily took my information and sent me the installation disks free of charge.

    Conclusion

    The setup discussed in this post is an inexpensive and relatively secure way of storing data in your own home/home network. The RAID 1 configuration offers redundancy, while the price of the system does not break the bank.

    I am very disappointed with Lenovo, trying to charge me for something I already paid for (the Operating System that is). Luckily the ladies at the hardware department were a lot more accommodating and I got what I wanted in the end.

    I hope this guide helped you.

  • 2012-05-28 12:00:00

    2012 Memorial Day

    Once again we celebrate Memorial Day in the USA.

    Today is Memorial Day in the USA.

    It is a special day, dedicated to celebrate, remember and honor the veterans that have gone and fought for all the liberties that we all enjoy today (and in most cases take for granted).

    I would like to personally say a thank you - which is not enough for sure - to all those that ensured and continue to do so,  that my family, friends and I are safe and free.

    You might also enjoy material from the following resources:

    My posts of This post I wrote in 2010, the site http://www.usamemorialday.org or the following video which is one of my favorites:

  • 2012-03-05 12:00:00

    Localization and Country Regions

    I was recently assigned a task in my current job, to try and standardize address related data.

    My approach was to use the ISO codes for countries and ISO codes for regions. A region is defined as the geographical split of areas in a country. For instance in the US regions are called states, in Canada provinces etc.

    I have used several sources on the internet but my main source was Wikipedia.

    In order to complete this task I used Zend Locale to retrieve the available countries and using the ISO codes (keys of the returned data).

    My method is:

    /**
     * Gets the countries based on a passed (or default) locale
     * 
     * @param string  $locale
     * @param boolean  $sort
     */
    public static function getCountries($locale = NULL, $sort = TRUE)
    {
        $results = parent::getTranslationList('territory', $locale, 2);
    
        if ($sort) 
        {
            asort($results);
        }
    
        return $results;
    }
    

    Having the list of countries and their ISO codes, allowed me to have a proper storage of country related data in my addresses table. The use of a select box in the view layer ensures that I keep this standardization in place.

    Extending that, I wanted to have an easy way to access the region related data, without having to hit the database. I therefore used JSON encoded files that contain an array with the type of the region as the key and Region ISO Code => Region Name as the array elements. Again I used a select box for that in the view layer, which is automatically refreshed when a new country is selected.

    So for instance for the US the JSON file is as follows:

    {
        "state":{
            "US-AL":"Alabama",
            "US-AK":"Alaska",
            ...
            "US-WV":"West Virginia",
            "US-WI":"Wisconsin",
            "US-WY":"Wyoming"
        }
    }
    

    where the key defines the region - in this case a State, and the array elements are the relevant data. In some instances I wanted to give a bit more selection and visual aids to the end user. I therefore have several sub arrays denoted by optiongroup tags for the relevant select box showing the regions.

    In the case of Great Britain:

    {
        "county":{
            "England":{
                "GB-BKM":"Buckinghamshire",
                "GB-CAM":"Cambridgeshire",
                ...
                "GB-WOK":"Wokingham",
                "GB-YOR":"York"
            },
            "Northern Ireland":{
                "GB-ANT":"Antrim",
                "GB-ARD":"Ards",
                ...
                "GB-OMH":"Omagh",
                "GB-STB":"Strabane"
            },
            "Scotland":{
                "GB-ABE":"Aberdeen City",
                "GB-ABD":"Aberdeenshire",
                ...
                "GB-WDU":"West Dunbartonshire",
                "GB-WLN":"West Lothian"
            },
            "Wales":{
                "GB-BGW":"Blaenau Gwent",
                "GB-BGE":"Bridgend (Pen-y-bont ar Ogwr)",
                ...
                "GB-VGL":"Vale of Glamorgan (Bro Morgannwg)",
                "GB-WRX":"Wrexham (Wrecsam)"
            }
        }
    }
    

    So with simple commands as file_get_contents and json_decode (or using Zend_Json::decode) I was able to have a flexible system that would load standardized address related data (regions and countries).

    You can download the list of JSON files (one per country) from my Github Repo.

    The list I have there is by no means complete, but it does have a lot of data that one can use. Additions are always welcome :)

    Enjoy!

  • 2012-01-15 12:00:00

    Downgrading PHPUnit from 3.6 to 3.5

    Recently I had to rebuild my computer, and decided to install Linux Mint 12 (Lisa), which is a very lean installation - for my taste that is.

    Going through the whole process of reinstalling all the packages that I need or had, PHPUnit was one of them. Easy enough a couple commands did the trick

    sudo apt-get install php-pear
    sudo pear upgrade PEAR
    sudo pear config-set auto_discover 1
    sudo pear install pear.phpunit.de/PHPUnit
    

    I wanted to run my tests after that, only to find an error in the execution:

    PHP Fatal error:  Call to undefined method PHPUnit_Util_Filter::addFileToFilter()
     in /home/www/project/library/PHPUnit/Framework.php on line 48
    

    At first I thought that it was a path error, so I included the /usr/share/php/PHPUnit and others in the php.ini file but with no luck. With a little bit of Googling I found out that there have been some changes in the 3.6 version of PHPUnit and things don't work as they did before.

    Effectively, 3.6 had some refactoring done and thus the line:

    PHPUnit_Util_Filter::addDirectoryToFilter("$dir/tests");
    

    changed to

    PHP_CodeCoverage_Filter::getInstance()
            ->addDirectoryToBlacklist("$dir/tests");
    

    Since I didn't want to change my whole test suite, I had to find a solution i.e. downgrade PHPUnit to 3.5.

    Unfortunately specifying the version directly did not wok

    sudo pear install phpunit/PHPUnit-3.5.15
    

    since it would pull the latest version again and I would end up with the 3.6 files.

    So I went one step further and installed specific versions of the relevant dependencies to satisfy the 3.5.15 version.

    Uninstallation of 3.6

    pear uninstall phpunit/PHPUnit_Selenium
    pear uninstall phpunit/DbUnit
    pear uninstall phpunit/PHPUnit
    pear uninstall phpunit/PHP_CodeCoverage
    pear uninstall phpunit/PHP_Iterator
    pear uninstall phpunit/PHPUnit_MockObject
    pear uninstall phpunit/Text_Template
    pear uninstall phpunit/PHP_Invoker
    pear uninstall phpunit/PHP_Timer
    pear uninstall phpunit/File_Iterator
    pear uninstall pear.symfony-project.com/YAML
    

    Installation of 3.5.15

    pear install pear.symfony-project.com/YAML-1.0.2
    pear install phpunit/PHPUnit_Selenium-1.0.1
    pear install phpunit/PHP_Timer-1.0.0
    pear install phpunit/Text_Template-1.0.0
    pear install phpunit/PHPUnit_MockObject-1.0.3
    pear install phpunit/File_Iterator-1.2.3
    pear install phpunit/PHP_CodeCoverage-1.0.2
    pear install phpunit/DbUnit-1.0.0
    pear install phpunit/PHPUnit-3.5.15
    

    I hope you find the above useful :)

  • 2011-12-09 12:00:00

    Change the encoding of a MySQL database to UTF8

    Overview

    As applications grow, so do their audiences. In this day and age, one cannot assume that all the consumers of a web based application will live in a particular region and use only one language. Even if the developer assumes that one country will be served by the particular web application, there are instances that the latin1 character set will not suffice in storing data.

    Therefore, developers and database designers need to implement an encoding on their database that will safely store and retrieve any kind of data, not only latin1 based (i.e. the English alphabet).

    For MySQL this encoding is utf8_general_ci.

    The problem

    MySQL usually comes with the latin1_swedish_ci encoding as a default. This encoding will allow the developer to store data of course but when non latin1 characters need to be stored, there will be a problem. Effectively latin1 encoding will store data in 8 bits but some languages like Japanese, Thai, Arabic, even French or German have special characters that need more space in the storage engine. Trying to store a 16 bit character in a 8 bit space will fail all the time.

    Latin1 based database:
    Input: abcdef...ABCD...#$
    Output: abcdef...ABCD...#$

    Input: 日本語 ภาษาไทย Ελληνικά
    Output: ??? ??????? ????????

    To combat this, all you have to do is change the encoding of your database to utf8_general_ci and the character set to utf8.

    The solution

    I wrote a script in PHP to analyze a database server and produce ALTER statements to be executed against your database(s). The script needs to run from a web server that supports PHP.

    First of all, the encoding of the database will change with the relevant SQL statement. Following that each table's encoding will change, again with the relevant SQL statement. Finally, each TEXT/VARCHAR/CHAR etc. field's encoding will change towards the target encoding you specify in the configuration section (see below).

    The safest way to transform data this way is to first change the field to a BINARY field and then change the field to the target encoding and collation.

    Configuration

    There are a few configuration variables that need to be set prior to running the script.

    $db_user       = 'username';
    $db_password   = 'password';
    $db_host       = 'hostname';
    $output_folder = '/home/ndimopoulos'; // Do not include trailing slash
    $db_name       = 'mysql';             // Leave this one as is
    
    set_time_limit(0);
    
    /**
     * The old collation (what needs to be changed)
     */
    $from_encoding = 'latin1_swedish_ci';
    
    /**
     * The new collation (what we will change it to)
     */
    $to_encoding = 'utf8_general_ci';
    
    /**
     * The new character set
     */
    $new_collation = 'utf8';
    
    /**
     * Add USE <database>; before each statement?
     */
    $use_database = TRUE;
    

    The $output_folder is a folder that is writeable from your web server and it is where the .sql files will be created filled with the ALTER statements. The script will output one file <hostname>.sql which will contain all the ALTER statements for all databases. It will also create files for individual databases <hostname>.<database>.sql. You can use either the big file or the individual database files. The choice is yours.

    The $from_encoding is what the script will check. In my script I was checking latin1_swedish_ci.
    The $to_encoding is what we need the encoding to be while the $new_collation is the new character set.

    The $use_database is a flag that will allow you to generate statements such as:

    USE <database>; 
    ALTER TABLE <table>.....
    

    if it is on, and if off, the statement will be:

    ALTER TABLE <table>.....
    

    Databases loop

    The script opens a connection to the server and runs the SHOW DATABASES command. Based on the result returned, it populates an array with the database names.

    The script ignores two databases information_schema and mysql, but editing the $exclude_databases array will allow you to ignore more databases if you need to.

    mysql_connect($db_host, $db_user, $db_password);
    mysql_select_db($db_name);
    
    $dbs = array();
    
    exclude_databases = array('mysql', 'information_schema',);
    
    /**
     * Get the databases available (ignore information_schema and mysql)
     */
    $result = mysql_query("SHOW DATABASES");
    
    while ($row = mysql_fetch_row($result))
    {
        if (!in_array($row[0], $exclude_databases))
        {
            $dbs[] = $row[0];
        }
    }
    

    The database names are stored in an array, so as not to keep the database resource active all the time. Had I not done that, I would have had to use three different resources (one for the database, one for the table and one for the field being checked - three nested loops).

    Tables loop

    The script then loops through the $dbs array and selects each database in turn. Once the database is selected, the SHOW TABLES query is run and a $tables array is populated with the names of the tables in that database. At the same time the ALTER DATABASE statements are being generated.

    mysql_select_db($db);
    
    $db_output = '';
    
    $statement  = "\r\n#-------------------------------------------------\r\n\r\n";
    $statement .= "USE $db;\r\n";
    $statement .= "\r\n#-------------------------------------------------\r\n\r\n";
    $statement .= "ALTER DATABASE $db "
               . "CHARACTER SET $new_collation COLLATE $to_encoding;\r\n";
    $statement .= "\r\n#-------------------------------------------------\r\n\r\n";
    
    $db_output .= $statement;
    $output    .= $statement;
    
    $tables     = array();
    
    $result = mysql_query("SHOW TABLES");
    
    while ($row = mysql_fetch_row($result))
    {
        if (!in_array($row[0], $exclude_tables))
        {
            $tables[] = mysql_real_escape_string($row[0]);
        }
    }
    

    Fields loop

    The script then loops through the $tables array and runs the SHOW FIELDS query so as to analyze each field.

    $fields_modify = array();
    $fields_change = array();
    
    $result = mysql_query("SHOW FULL FIELDS FROM `$table`");
    while ($row = mysql_fetch_assoc($result)) 
    {
        if ($row['Collation'] != $from_encoding)
        {
            continue;
        }
    
        // Is the field allowed to be null?
        $nullable = ($row['Null'] == 'YES') ? ' NULL ' : ' NOT NULL';
    
        if ($row['Default'] == 'NULL') 
        {
           $default = " DEFAULT NULL";
        } 
        else if ($row['Default']!='') 
        {
           $default = " DEFAULT '" . mysql_real_escape_string($row['Default']) . "'";
        } 
        else 
        {
           $default = '';
        }
    
        // Alter field collation:
        $field_name = mysql_real_escape_string($row['Field']);
    
        $fields_modify[] = "MODIFY `$field_name` $row[Type] CHARACTER SET BINARY";
        $fields_change[] = "CHANGE `$field_name` `$field_name` $row[Type] "
                         . "CHARACTER SET $new_collation "
                         . "COLLATE $to_encoding $nullable $default";
    }
    

    The two arrays generated ($fields_modify and $fields_change contain the MODIFY and CHANGE statements of each field. Using implode, we can easily construct the ALTER statement.

    $statement .= "ALTER TABLE `$table` " 
                . implode(' , ', $fields_modify) . "; \r\n";
    $statement .= "ALTER TABLE `$table` " 
                . implode(' , ', $fields_change) . "; \r\n";
    

    Notes

    You can use as mentioned earlier the $exclude_databases array to not allow certain databases to be processed. You can also use the $exclude_tables array to not allow certain tables to be processed.

    The $exclude_tables_fields array allows you to exclude a field from being processed. However this is not tied to a database so any database/table that has a field with that particular name will not be processed. With a bit of refactoring you can make the script best work for your needs.

    If you set the $use_database variable to TRUE then each line in your .sql statements will be prefixed with a 'USE ;' statement. This is to help the accompanying bash script to execute each statement in the respective database. If you intend on not running this process one statement at a time, you can set this to FALSE. You can then run each database .sql file (or the one that contains all of the statements from all databases) as one single command.

    Server processing

    Now that the relevant .sql files have been created, all you have to do is upload them on your web server. There are three ways of actually running the statements against the database.

    Please make sure you backup your data first!

    Single file processing

    mysql -h<host> -u<username> -p<password> < /path/to/scripts/<host>.sql
    

    The command above will run all the commands generated in the <host>.sql file for all databases. It is going to be taxing for your database server and there is no error handling or reporting. You can always pipe the results to an output file (just append "> /path/to/output/output.txt" at the end of the command). If this method fails for some reason (MySQL has gone away), it will be difficult to resume; you will need to edit the .sql file to remove the statements that have already been processed.

    Per database

    mysql -h<host> -u<username> -p<password> <database_name> \
           </path/to/scripts/<host>.<database>.sql
    

    The command above will run all the commands generated in the ..sql file for that particular database. This method is similar to the one above. Again you can always pipe the results to an output file (just append "> /path/to/output/output.txt" at the end of the command).

    Single file processing (per statement)

    /path/to/scripts/process.sh
    

    All you need to do is edit the process.sh script and change the relevant parameters to match your environment and upload it to your server. The source file that the process.sh script will read has to be generated with $use_database set to TRUE.

    The process.sh script is:

    #!/bin/bash
    
    DBUSER=root
    DBPASS=1234
    DBHOST=localhost
    SOURCE="/home/ndimopoulos/host.sql"
    LOG="/home/ndimopoulos/conversion.log"
    
    while read line
    do
        TIMENOW=`date +%Y-%m-%d-%H-%M`
        echo START $TIMENOW $line
        echo START $TIMENOW $line >> $LOG
        /usr/bin/time -f "%E real,%U user,%S sys" -v -o $LOG -a \
            mysql -h$DBHOST -u$DBUSER -p$DBPASS -e "$line"
    
        TIMENOW=`date +%Y-%m-%d-%H-%M`
        echo END $TIMENOW 
        echo END $TIMENOW >> $LOG
    
    done < $SOURCE
    
    exit 0
    

    The script above will start reading the source file (.sql) and execute each statement in turn, using time) to measure the time taken to execute that command. The output ends up in a log file which can easily be tailed to view the progress and used later on for analysis. The results of the processing are also sent to the screen. You can change the parameters for the time command to match your needs.

    The output will look something like the block below:

    START 2011-12-08-23-46 USE mydatabase; \
         ALTER TABLE `tablename` DEFAULT CHARACTER SET utf8;
     Command being timed: "mysql -uroot -p1234 -e USE mydatabase; \
         ALTER TABLE `tablename` DEFAULT CHARACTER SET utf8;"
     User time (seconds): 0.01
     System time (seconds): 0.00
     Percent of CPU this job got: 0%
     Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.16
     Average shared text size (kbytes): 0
     Average unshared data size (kbytes): 0
     Average stack size (kbytes): 0
     Average total size (kbytes): 0
     Maximum resident set size (kbytes): 8192
     Average resident set size (kbytes): 0
     Major (requiring I/O) page faults: 0
     Minor (reclaiming a frame) page faults: 610
     Voluntary context switches: 11
     Involuntary context switches: 5
     Swaps: 0
     File system inputs: 0
     File system outputs: 0
     Socket messages sent: 0
     Socket messages received: 0
     Signals delivered: 0
     Page size (bytes): 4096
     Exit status: 0
    END 2011-12-08-23-46
    

    Conclusion

    In order for a database to be best prepared to support localization, you need to make sure that the storage will accept any possible character. You can start by creating all your tables and fields with utf8_general_ci encoding, but for existing databases and data, you will need to run expensive processing queries on your RDBMS. Ensuring that the data does not get corrupted when performing the transformation process is essential so make sure you backup your databases before trying or running the output statements produced by the db_alter.php script.

    PHP script (db_alter.php)

    $db_user       = 'username';
    $db_password   = 'password';
    $db_host       = 'hostname';
    $output_folder = '/home/ndimopoulos'; // Do not include trailing slash
    $db_name       = 'mysql'; // Leave this one as is
    
    set_time_limit(0);
    
    /**
     * The old collation (what needs to be changed)
     */
    $from_encoding = 'latin1_swedish_ci';
    
    /**
     * The new collation (what we will change it to)
     */
    $to_encoding = 'utf8_general_ci';
    
    /**
     * The new character set
     */
    $new_collation = 'utf8';
    
    /**
     * Add USE <database> before each statement?
     */
    $use_database = TRUE;
    
    mysql_connect($db_host, $db_user, $db_password);
    mysql_select_db($db_name);
    
    $dbs = array();
    
    $exclude_databases     = array('mysql', 'information_schema',);
    $exclude_tables        = array('logs', 'logs_archived',);
    $exclude_tables_fields = array('activities');
    
    /**
     * Get the databases available (ignore information_schema and mysql)
     */
    $result = mysql_query("SHOW DATABASES");
    
    while ($row = mysql_fetch_row($result)) 
    {
        if (!in_array($row[0], $exclude_databases))
        {
            $dbs[] = $row[0];
        }
    }
    
    $output = '';
    /**
     * Now select each db and start parsing the tables
     */
    foreach ($dbs as $db)
    {
        mysql_select_db($db);
        $db_output = '';
    
        $statement  = "\r\n#----------------------------------------\r\n\r\n";
        $statement .= "USE $db;\r\n";
        $statement .= "\r\n#----------------------------------------\r\n\r\n";
        $statement .= "ALTER DATABASE $db "
                   . "CHARACTER SET $new_collation COLLATE $to_encoding;\r\n";
        $statement .= "\r\n#----------------------------------------\r\n\r\n";
    
        $db_output .= $statement;
        $output    .= $statement;
        $tables     = array();
    
        $result = mysql_query("SHOW TABLES");
    
        while ($row = mysql_fetch_row($result))
        {
            if (!in_array($row[0], $exclude_tables))
            {
                $tables[] = mysql_real_escape_string($row[0]);
            }
        }
    
        /**
         * Alter statements for the tables
         */
        foreach ($tables as $table)
        {
            $statement = '';
            if ($use_database)
            {
                $statement  = "USE $db; ";
            }
            $statement .= "ALTER TABLE `$table` "
                       . "DEFAULT CHARACTER SET $new_collation;\r\n";
            $db_output .= $statement;
            $output    .= $db_output;
        }
        $statement .= "\r\n#----------------------------------------\r\n\r\n";
    
        $db_output .= $statement;
        $output    .= $statement;
    
        /**
         * Get the fields for each table
         */
        foreach ($tables as $table)
        {
            if (in_array($table, $exclude_tables_fields))
            {
                continue;
            } 
    
            $fields_modify = array();
            $fields_change = array();
    
            $result = mysql_query("SHOW FULL FIELDS FROM `$table`");
            while ($row = mysql_fetch_assoc($result)) 
            {
                if ($row['Collation'] != $from_encoding)
                {
                    continue;
                }
    
                // Is the field allowed to be null?
                $nullable = ($row['Null'] == 'YES') ? ' NULL ' : ' NOT NULL';
    
                if ($row['Default'] == 'NULL') 
                {
                    $default = " DEFAULT NULL";
                } 
                else if ($row['Default']!='') 
                {
                    $default = " DEFAULT '"
                             . mysql_real_escape_string($row['Default']) . "'";
                }
                else 
                {
                    $default = '';
                }
    
                // Alter field collation:
                $field_name = mysql_real_escape_string($row['Field']);
    
                $fields_modify[] = "MODIFY `$field_name` $row['Type'] "
                                 . "CHARACTER SET BINARY";
                $fields_change[] = "CHANGE `$field_name` `$field_name` $row['Type'] "
                                 . "CHARACTER SET $new_collation "
                                 . "COLLATE $to_encoding $nullable $default";
            }
    
            if (count($fields_modify) > 0)
            {
                $statement = '';
                if ($use_database)
                {
                    $statement = "USE $db; ";
                }
                $statement .= "ALTER TABLE `$table` "
                            . implode(' , ', $fields_modify) . "; \r\n";
                if ($use_database)
                {
                    $statement = "USE $db; ";
                }
                $statement .= "ALTER TABLE `$table` "
                            . implode(' , ', $fields_change) . "; \r\n";
    
                $db_output .= $statement;
                $output    .= $statement;
            }
        }
    
        $bytes = file_put_contents(
            $output_folder . '/' . $db_host . '.' . $db . '.sql', $db_output
        );
    }
    
    $bytes = file_put_contents($output_folder . '/' . $db_host . '.sql', $output);
    
    echo "<pre>$db_host $bytes \r\n$output</pre>";
    

    Downloads

    You can use these scripts at your own risk. Also feel free to distribute them freely - a mention would be nice. Both scripts can be found in my GitHub.