• Advertisement

  • 2015-10-29 21:15:00

    Setting up AWS RDS MySQL replication offsite

    Preface

    Recently I worked on setting up replication between our AWS RDS instance and a server running as a MySQL slave in our location. Although the task was not difficult, there are quite a few areas that one needs to pay attention to.

    In this blog post, I am going to outline the process step by step, so that others can benefit and not lose time trying to discover what went wrong.

    Disclaimer: Google, DuckDuckGo, the AWS forums and this blog post have been invaluable guides to help me do what I needed to do.

    Setup

    • One RDS MySQL or Aurora instance (Master)
    • One server running MySQL in your premises (or wherever you want to put it) (Slave)
    • Appropriate access to IP of the Slave on the master.

    Master Setup

    There is little to do on our master (RDS). Depending on the database size and update frequency, we will need to set up the maximum retention time for the bin logs. For a very large database we need to set a high number, so that we are able to export the database from the master, import it in the slave and start replication.

    Connect to your database and run the following command:

    MySQL [(none)]> call mysql.rds_set_configuration('binlog retention hours', 24);
    

    You can use a different number of hours; I am using 24 for this example.

    Slave Setup

    I am assuming that MySQL is installed on the machine that has been designated as the slave, and also that that machine has ample space for the actual data as well as the binary logs that will be created for the replication.

    Edit my.cnf

    The location of this file is usually under /etc or /etc/mysql. Depending on your distribution it might be located elsewhere.

    [mysqld]
    ...
    
    #bind-address = 0.0.0.0
    
    # Logging and Replication
    general_log_file  = /logs/mysql.log
    general_log       = 1
    log_error         = /logs/mysql_safe.log
    log_slow_queries  = /logs/mysql-slow.log
    long_query_time   = 2
    slave-skip-errors = 1062
    log-queries-not-using-indexes
    
    server-id         = 1234567
    log_bin           = /logs/mysql-bin.log
    expire_logs_days  = 2
    max_binlog_size   = 100M
    

    Note: The configuration file will contain a lot more entries but the ones above are the ones you need to pay attention to.

    • bind-address: We need to comment this line so that we can connect to the instance from somewhere else in the network. Keep this line if you are going to work only on the slave machine and allow no connections from elsewhere.
    • general_log_file: The location of your query log file. You can disable this (see next entry) but it is always good to keep it on at least at the start, to ensure that replication is moving smoothly. Tailing that log will give you a nice indicator of the activity in your database.
    • general_log: Enable or disable the general log
    • log_error: Where to store the errors log
    • log_slow_queries: Where to store the slow queries log. Especially helpful in identifying bottlenecks in your application
    • long_query_time: Time to specify what a slow query is
    • slave-skip-errors: 1062 is the "1062 | Error 'Duplicate entry 'xyz' for key 1' on query. Default database: 'db'. Query: 'INSERT INTO ...'" error. Helpful especially when the replication starts.
    • log-queries-not-using-indexes: We want this because it can help identifying potential bottlenecks in the application
    • server-id: A unique ID for your slave instance.
    • log_bin: Where the binary replication logs are kept
    • expire_logs_days: How long to keep the replication logs for
    • max_binlog_size: Maximum replication log size (per file)

    Once you set these up, restart your MySQL instance

    /etc/init.d/mysql restart
    
    Download the SSH Public Key for RDS

    In your slave server, navigate to /etc/mysql and download the rds-combined-ca-bundle.pem file. This file will be used by the slave to ensure that all the replication traffic is done using SSL and nobody can eavesdrop on your data in transit.

    cd /etc/mysql
    wget http://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem
    

    NOTE You can put the rds-combined-ca-bundle.pem anywhere on your slave. If you change the path, you will have to modify the command to connect the slave to the master (shown further below) to specify the exact location of the key.

    Import timezone data

    This step might not be necessary depending on your MySQL installation. However since RDS works with UTC, you might find your replication breaking because your slave MySQL instance cannot understand the UTC timezone. The shell command you need to run on your slave machine to fix this is:

    mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql
    
    Creating the RDS related tables

    RDS uses its own tables to keep track of the replication status and other related data such as the replication heartbeat, configuration etc. Those tables need to be present in the mysql database of your slave in order for the replication to work.

    DROP TABLE IF EXISTS `rds_configuration`;
    CREATE TABLE `rds_configuration` (
      `name` varchar(100) NOT NULL,
      `value` varchar(100) DEFAULT NULL,
      `description` varchar(300) NOT NULL,
      PRIMARY KEY (`name`)
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
    
    DROP TABLE IF EXISTS `rds_global_status_history`;
    CREATE TABLE `rds_global_status_history` (
      `collection_end` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
      `collection_start` timestamp NULL DEFAULT NULL,
      `variable_name` varchar(64) NOT NULL,
      `variable_value` varchar(1024) NOT NULL,
      `variable_delta` int(20) NOT NULL,
      PRIMARY KEY (`collection_end`,`variable_name`)
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
    
    DROP TABLE IF EXISTS `rds_global_status_history_old`;
    CREATE TABLE `rds_global_status_history_old` (
      `collection_end` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
      `collection_start` timestamp NULL DEFAULT NULL,
      `variable_name` varchar(64) NOT NULL,
      `variable_value` varchar(1024) NOT NULL,
      `variable_delta` int(20) NOT NULL,
      PRIMARY KEY (`collection_end`,`variable_name`)
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
    
    DROP TABLE IF EXISTS `rds_heartbeat2`;
    CREATE TABLE `rds_heartbeat2` (
      `id` int(11) NOT NULL,
      `value` bigint(20) DEFAULT NULL,
      PRIMARY KEY (`id`)
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
    
    DROP TABLE IF EXISTS `rds_history`;
    CREATE TABLE `rds_history` (
      `action_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
      `called_by_user` varchar(50) NOT NULL,
      `action` varchar(20) NOT NULL,
      `mysql_version` varchar(50) NOT NULL,
      `master_host` varchar(255) DEFAULT NULL,
      `master_port` int(11) DEFAULT NULL,
      `master_user` varchar(16) DEFAULT NULL,
      `master_log_file` varchar(50) DEFAULT NULL,
      `master_log_pos` mediumtext,
      `master_ssl` tinyint(1) DEFAULT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
    
    DROP TABLE IF EXISTS `rds_replication_status`;
    CREATE TABLE `rds_replication_status` (
      `action_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
      `called_by_user` varchar(50) NOT NULL,
      `action` varchar(20) NOT NULL,
      `mysql_version` varchar(50) NOT NULL,
      `master_host` varchar(255) DEFAULT NULL,
      `master_port` int(11) DEFAULT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
    
    DROP TABLE IF EXISTS `rds_sysinfo`;
    CREATE TABLE `rds_sysinfo` (
      `name` varchar(25) DEFAULT NULL,
      `value` varchar(50) DEFAULT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
    

    NOTE I am not 100% that all of these tables are needed. I have seen only the rds_heartbeat2 and rds_replication_status used. You can experiment with these when you enable replication and add each table in turn if needed. You can confirm whether the above are correct for your instance by connecting to the master and taking a mysqldump of the mysql database.

    Replication

    Replication user

    We need to create a user in our master database that will have the appropriate rights to perform all the replication related actions. We need these commands to be run on the master. For this example I am creating a user called rpluser with the password 424242:

    MySQL [(none)]> CREATE USER 'rpluser'@'%' IDENTIFIED BY '424242';
    MySQL [(none)]> GRANT REPLICATION SLAVE ON *.* TO 'rpluser'@'%';
    
    Master Status

    Connect to your master and issue this command:

    MySQL [(none)]> show master status;
    

    The output will be something like this:

    +----------------------------+----------+--------------+------------------+-------------------+
    | File                       | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
    +----------------------------+----------+--------------+------------------+-------------------+
    | mysql-bin-changelog.000123 |   171819 |              |                  |                   |
    +----------------------------+----------+--------------+------------------+-------------------+
    

    Keep those values handy (File and Position) since we will use them to instruct the slave where to start requesting data from the master (binlog file and position).

    mysqldump

    Take a database dump of all the databases in RDS (exclude information_schema and mysql). If your database can afford a bit of downtime you can use the --opt flag in mysqldump, which will lock all tables until the backup completes. If not, you can use the --skip-add-locks flag. More information about mysqldump options can be found here

    mysqldump --host=192.168.1.2 --user='root' --password my_db > /backups/my_db.sql
    

    Adjust the above command to fit your needs. Once all databases have been dumped, we need to import them in the slave.

    Importing data in the slave

    Navigate to the folder you have all the *.sql dump files, connect to the slave database and start sourcing them.

    cd /backups
    mysql --host=192.168.1.2 --user='root' --password
    MySQL [(none)]> create database my_db;
    MySQL [(none)]> use my_db;
    MySQL [my_db]> source my_db.sql;
    

    Repeat the process of creating the database, using it and sourcing the dump file until all your databases have been imported.

    NOTE There are other ways of doing the above, piping the results directly to the database or even using RDS to get the data straight from it without a mysqldump. Whichever way you choose is up to you. In my experience, the direct import worked for a bit until our database grew to a point that it was timing out or breaking while importing, so I opted for the multi step approach. Have a look at this section in the AWS RDS documentation for more options.

    Connecting to the master

    Once your restore has been completed it is time to connect our slave to the master. In order to connect to the master from the slave we need to verify the following:

    • The name of the RDS instance (for the command below I will use myinstance.rds.amazonaws.com)
    • The name of the replication user (we chose rpluser)
    • The password of the replication user (we chose 424242)
    • The master log file (see above, we got mysql-bin-changelog.000123 from show master status;)
    • The master log file position (see above, we got 171819)
    • The location of the SSL certificate (we used /etc/mysql/rds-combined-ca-bundle.pem)

    The command we need to run on the slave MySQL server is (newlines added for readability):

    MySQL [(none)]> CHANGE MASTER TO 
        -> MASTER_HOST='myinstance.rds.amazonaws.com', 
        -> MASTER_USER='rpluser', 
        -> MASTER_PASSWORD='424242', 
        -> MASTER_LOG_FILE='mysql-bin-changelog.000123', 
        -> MASTER_LOG_POS=171819, 
        -> MASTER_SSL=1, 
        -> MASTER_SSL_CERT='', 
        -> MASTER_SSL_CA='/etc/mysql/rds-combined-ca-bundle.pem', 
        -> MASTER_SSL_KEY='';
    
    Starting the replication

    All we have to do now is to start the slave:

    MySQL [(none)]> START SLAVE;
    

    We can check if everything is OK either by using the general log (see my.cnf section) by tailing it from the shell:

    tail -f /logs/mysql.log
    

    or by issuing this command on the mysql prompt:

    MySQL [(none)]> SHOW SLAVE STATUS \G;
    *************************** 1. row ***************************
                   Slave_IO_State: Waiting for master to send event
                      Master_Host: myinstance.rds.amazonaws.com
                      Master_User: rpluser
                      Master_Port: 3306
                    Connect_Retry: 60
                  Master_Log_File: mysql-bin-changelog.000123
              Read_Master_Log_Pos: 171819
                   Relay_Log_File: mysqld-relay-bin.000002
                    Relay_Log_Pos: 123
    ...
                       Last_Errno: 0
                       Last_Error: 
    ...
               Master_SSL_Allowed: Yes
               Master_SSL_CA_File: /etc/mysql/rds-combined-ca-bundle.pem
               Master_SSL_CA_Path: 
                  Master_SSL_Cert: 
                Master_SSL_Cipher: 
                   Master_SSL_Key: 
    ...
          Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
    ...
    

    Congratulations on your working RDS slave (offsite) machine :)

    Conclusion

    This blog post is by no means exhausting all the topics that replication can cover. For additional information please see the references below.

    I hope you find this post helpful :)

    References

  • 2015-10-18 13:09:00

    Using Traits to add more functionality to your classes in PHP

    Traits are a mechanism for code reuse in single inheritance languages such as PHP.

    A Trait is similar to a class, but only intended to group functionality in a fine-grained and consistent way. It is not possible to instantiate a Trait on its own. It is an addition to traditional inheritance and enables horizontal composition of behavior; that is, the application of class members without requiring inheritance. Source

    Traits have been introduced in PHP 5.4.0. However, a lot of developers have not yet embraced them and taken advantage of the power that they offer.

    As mentioned above in the snippet of the PHP manual, Traits are a mechanism to reuse code, making your code more DRY.

    Let's have a look at a real life example of how Traits can help you with your Phalcon project, or any project you might have.

    Models

    With Phalcon, we have model classes which represent pretty much a table in our database, and allows us to interact with a record or a resultset for the needs of our application.

    Scenario

    We have an application where we need to store information about Companies. Each Company can have one or more Customers as well as one or more Employees. We chose to store that information in three different tables.

    For each Employee or Customer, we need to store their first name, middle name and last name. However we also need to be able to show the full name in this format:

    <Last name>, <First Name> <Middle Name>
    

    Using custom getter

    In each model we can use a custom getter method in the Phalcon model to calculate the full name of the record.

    Employee
    namespace NDN\Models;
    
    class Employee
    {
        ...
        public function getFullName()
        {
            return trim(
                sprintf(
                    '%s, %s %s',
                    $this->getLastName(),
                    $this->getFirstName(),
                    $this->getMiddleName()
                )
            );
        }
    }
    
    Customer
    namespace NDN\Models;
    
    class Customer
    {
        ...
        public function getFullName()
        {
            return trim(
                sprintf(
                    '%s, %s %s',
                    $this->getLastName(),
                    $this->getFirstName(),
                    $this->getMiddleName()
                )
            );
        }
    }
    

    The above introduces a problem. If we want to change the behavior of the getFullName we will have to visit both models and make changes to the relevant methods in each model. In addition, we are using the same code in two different files i.e. duplicating code and effort.

    We could create a base model class that our Customer and Employee models extend and put the getFullName function in there. However that increases the class extensions and could lead to maintenance nightmares.

    For instance we will have to create the base model class that only Customer and Employee models extend but what would happen if we need common functionality for other models? We will need to then create another base model class and so on and so forth. If we end up piling all the common functionality into one base model class then we will end up with functions that would not apply to all of our models and thus a maintenance nightmare.

    NOTE: We can also use the afterFetch method to create a calculated field which will be available for us to use. We can use either the getter or the afterFetch like so:

    namespace NDN\Models;
    
    class Customer
    {
        ...
        public function afterFetch()
        {
            $this->full_name = trim(
                sprintf(
                    '%s, %s %s',
                    $this->getLastName(),
                    $this->getFirstName(),
                    $this->getMiddleName()
                )
            );
        }
    }
    

    Traits

    We can use a trait to offer the same functionality, keeping our code DRY. Since a Trait is not a class that can be instantiated by itself, we attach it to wherever we need to, in this case the Employee and Customer models.

    namespace NDN\Traits;
    
    trait FullNameTrait
    {
        /**
         * Gets the user first/last/med name and formats it in a readable format
         *
         * @return  string
         */
        public function getFullName()
        {
            return trim(
                sprintf(
                    '%s, %s %s',
                    $this->getLastName(),
                    $this->getFirstName(),
                    $this->getMiddleName()
                )
            );
        }
    }
    

    We can attach now this trait to the relevant models

    Employee
    namespace NDN\Models;
    
    use NDN\Traits\FullNameTrait;
    
    class Employee
    {
        use FullNameTrait;
    }
    
    Customer
    namespace NDN\Models;
    
    use NDN\Traits\FullNameTrait;
    
    class Customer
    {
        use FullNameTrait;
    }
    

    Now we can use the getFullName() function in our two models to get the full name of the Employee or Customer calculated by the relevant model fields.

    // Customer:
    // first_name:  John
    // middle_name: Mark
    // last_name:   Doe
    
    // Prints: Doe, John Mark
    echo $customer->getFullName();
    
    // Employee:
    // first_name:  Stanley
    // middle_name: Martin
    // last_name:   Lieber
    
    // Prints: Lieber, Stanley Martin
    echo $employee->getFullName();
    

    Conclusion

    Traits can be very powerful and helpful allies, keeping our code very flexible and reusable.

    Give it a try!

  • 2015-10-04 13:26:00

    Building the Phalcon Blog (i)

    This is the first of a series of posts, describing how we built the Phalcon Blog (and this one of course). The intention is to showcase some of the features of Phalcon and discuss the reasons behind implementing the code in such a way. I will amend this post with the links of the future posts once I post them.

    These series will focus initially on the Phalcon blog (Github) and will then expand on this blog (Github). In the very near future all the features available in this blog will be available in the Phalcon one :)

    As I mentioned in a previous post, Andres and I were not 100% satisfied with Tumblr, the blogging platform that we have used for a few years for the purposes of the Phalcon blog. So we decided that it would not only be beneficial for us to build something of our own, but also for the community, since the software is open sourced and available for everyone to use.

    Bootstrapping process

    In this post I am going to concentrate on bootstrapping the application. By bootstrapping I do not mean using the Bootstrap open source library, despite the probably misleading image on the right.

    Bootstrapping is the class (in our case) that handles pretty much everything that our application needs to run prior to executing actions. This entails

    • Conditional execution between the normal app and the CLI one
    • Application Paths
    • Configuration Files
    • Loader (and composer autoloader) setup
    • Error handling
    • Routes
    • Dispatcher
    • Url
    • Views
      • Main View
      • Simple View (for emails, RSS/Sitemap etc.)
    • Cache
    • Utils
    • Post Finder class

    Some of the above components are also registered in the DI container for further use in the application.

    Implementation

    In other applications we have open sourced such as Vokuro, we have pretty much always included a couple of files in our index.php; one for the loader and one for the services as demonstrated here.

    <?php
    
    error_reporting(E_ALL);
    
    try {
        /**
         * Define some useful constants
         */
        define('BASE_DIR', dirname(__DIR__));
        define('APP_DIR', BASE_DIR . '/app');
        /**
         * Read the configuration
         */
        $config = include APP_DIR . '/config/config.php';
        /**
         * Read auto-loader
         */
        include APP_DIR . '/config/loader.php';
        /**
         * Read services
         */
        include APP_DIR . '/config/services.php';
        /**
         * Handle the request
         */
        $application = new \Phalcon\Mvc\Application($di);
        echo $application->handle()->getContent();
    } catch (Exception $e) {
        echo $e->getMessage(), '<br>';
        echo nl2br(htmlentities($e->getTraceAsString()));
    }
    

    There is nothing wrong with the above approach. We did however consider the fact that the particular index.php file has 3 different file inclusions and if we wanted to tinker with the setup of the application we would have to open all three.

    We opted for one file containing all of our services and application bootstrap. In addition to that, we altered the design so that later on we can add a CLI application without much effort and heavy refactoring.

    NOTE: The CLI application has been implemented on this blog and will very soon be merged to the Phalcon repository. We will cover that functionality in a future post.

    index.php

    Having one file that performs all the necessary initialization tasks a.k.a. bootstrapping our application allows us to have a much smaller index.php file. (comments removed to preserve space)

    <?php
    
    use \Phalcon\Di\FactoryDefault as PhDI;
    use \Kitsune\Bootstrap;
    
    error_reporting(E_ALL);
    
    try {
        require_once '../library/Kitsune/Bootstrap.php';
    
        $di = new PhDI();
        $bootstrap = new Bootstrap();
    
        echo $bootstrap->run($di, []);
    } catch (\Exception $e) {
        if ($di->has('logger')) {
            $logger = $di->getShared('logger');
            $logger->error($e->getMessage());
            $logger->error('<pre>' . $e->getTraceAsString() . '</pre>');
        }
    }
    

    We create a new bootstrap application and pass in it a DI container. For this part of the application the FactoryDefault DI container is used. However we will be able to inject a Phalcon CLI DI container for the CLI application we will discuss later on.

    Bootstrap.php

    Our bootstrap class contains all the code we need to run the application. It is a bit shy of 400 lines which according to PHP Mess Detector is not something we want to be doing because it increases complexity and if we are not careful it will create a mess :). We opted to ignore that rule and left the file as is because once we had everything working as we wanted, we were not going to be messing with that file again.

    Constants

    We use several constants throughout the application.

    • K_PATH - the top folder path of our installation
    • K_CLI - whether this is a CLI application or not
    • K_DEBUG - whether we are in debug/development mode. In this mode all volt templates are being created at every request and cache is not used.
    • K_TESTS - whether we are running the test suite or not (test suite is not implemented yet)
            /**
             * The app path
             */
            if (!defined('K_PATH')) {
                define('K_PATH', dirname(dirname(dirname(__FILE__))));
            }
    
            ....
    
            /**
             * Check if this is a CLI app or not
             */
            $cli   = $utils->fetch($options, 'cli', false);
            if (!defined('K_CLI')) {
                define('K_CLI', $cli);
            }
    
            $tests = $utils->fetch($options, 'tests', false);
            if (!defined('K_TESTS')) {
                define('K_TESTS', $tests);
            }
    
            ....
    
            /**
             * Check if we are in debug/dev mode
             */
            if (!defined('K_DEBUG')) {
                $debugMode = boolval($utils->fetch($config, 'debugMode', false));
                define('K_DEBUG', $debugMode);
            }
    
    Configuration

    The configuration is split into two files. The git tracked base.php (under /var/config/) contains an array of elements that are needed throughout the application, such as cache settings, routes etc. The config.php located in the same folder is installation dependent and is not tracked in git. You can override every element that exists in base.php.

            /**
             * The configuration is split into two different files. The first one
             * is the base configuration. The second one is machine/installation
             * specific.
             */
            if (!file_exists(K_PATH . '/var/config/base.php')) {
                throw new \Exception('Base configuration files are missing');
            }
    
            if (!file_exists(K_PATH . '/var/config/config.php')) {
                throw new \Exception('Configuration files are missing');
            }
    
            /**
             * Get the config files and merge them
             */
            $base     = require(K_PATH . '/var/config/base.php');
            $specific = require(K_PATH . '/var/config/config.php');
            $combined = array_replace_recursive($base, $specific);
    
            $config = new Config($combined);
            $di->set('config', $config, true);
    
    Loader

    The loader uses the namespaces defined in the base.php and config.php. Additionally the composer autoloader is included to offer functionality needed from the composer components we have.

            /**
             * We're a registering a set of directories taken from the
             * configuration file
             */
            $loader = new Loader();
            $loader->registerNamespaces($config->namespaces->toArray());
            $loader->register();
    
            require K_PATH . '/vendor/autoload.php';
    
    Logger

    The logger is set to create a log file every day (with the date as the prefix).

            /**
             * LOGGER
             *
             * The essential logging service
             */
            $format    = '[%date%][%type%] %message%';
            $name      = K_PATH . '/var/log/' . date('Y-m-d') . '-kitsune.log';
            $logger    = new LoggerFile($name);
            $formatter = new LoggerFormatter($format);
            $logger->setFormatter($formatter);
            $di->set('logger', $logger, true);
    
    Error handler

    We decided to have no errors thrown in the application even if those are E_NOTICE. A simple isset() in most cases is more than enough to ensure that there are no E_NOTICE errors thrown in our log. Any errors thrown in the log files slow our application down, even if the errors are suppressed using the php.ini directives. We also set the timezone to US/Eastern in that file. That particular piece could become configurable and stored in the config.php. Finally we specify a custom error handler, to offer verbosity in errors thrown as well as log metrics when in debug mode.

            /**
             * ERROR HANDLING
             */
            ini_set('display_errors', boolval(K_DEBUG));
    
            error_reporting(E_ALL);
    
            set_error_handler(
                function ($exception) use ($logger) {
                    if ($exception instanceof \Exception) {
                        $logger->error($exception->__toString());
                    } else {
                        $logger->error(json_encode(debug_backtrace()));
                    }
                }
            );
    
            set_exception_handler(
                function (\Exception $exception) use ($logger) {
                    $logger->error($exception->getMessage());
                }
            );
    
            register_shutdown_function(
                function () use ($logger, $memoryUsage, $currentTime) {
                    $memoryUsed = number_format(
                        (memory_get_usage() - $memoryUsage) / 1024,
                        3
                    );
                    $executionTime = number_format(
                        (microtime(true) - $currentTime),
                        4
                    );
                    if (K_DEBUG) {
                        $logger->info(
                            'Shutdown completed [Memory: ' . $memoryUsed . 'Kb] ' .
                            '[Execution: ' . $executionTime .']'
                        );
                    }
                }
            );
    
            $timezone = $config->get('app_timezone', 'US/Eastern');
            date_default_timezone_set($timezone);
    
    Routes

    Our routes are stored in the base.php. Additional routes can be set in the config.php. The router is not initialized if this is a CLI application.

            /**
             * Routes
             */
            if (!K_CLI) {
                $di->set(
                    'router',
                    function () use ($config) {
                        $router = new Router(false);
                        $router->removeExtraSlashes(true);
                        $routes = $config->routes->toArray();
                        foreach ($routes as $pattern => $options) {
                            $router->add($pattern, $options);
                        }
    
                        return $router;
                    },
                    true
                );
            }
    
    Dispatcher

    The dispatcher is instantiated with a listener, attaching to the beforeException event of the dispatcher. A custom plugin NotFoundPlugin is used to send output to the 404 page. Using the plugin allows us to reuse it anywhere in the application. This implementation is very beneficial when developing multi module applications.

    NOTE: For the CLI application later on, we will need the CLI dispatcher.

            /**
             * We register the events manager
             */
            $di->set(
                'dispatcher',
                function () use ($di) {
                    $eventsManager = new EventsManager;
    
                    /**
                     * Handle exceptions and not-found exceptions using NotFoundPlugin
                     */
                    $eventsManager->attach('dispatch:beforeException', new NotFoundPlugin);
    
                    $dispatcher = new Dispatcher;
                    $dispatcher->setEventsManager($eventsManager);
    
                    $dispatcher->setDefaultNamespace('Kitsune\Controllers');
    
                    return $dispatcher;
                }
            );
    
    Views

    The views are being initialized using Volt as the template engine. The main view is set up with the expected options.

            $di->set(
                'view',
                function () use ($config) {
                    $view = new View();
                    $view->setViewsDir(K_PATH . '/app/views/');
                    $view->registerEngines([".volt" => 'volt']);
                    return $view;
                }
            );
    
            /**
             * Setting up volt
             */
            $di->set(
                'volt',
                function ($view, $di) {
                    $volt = new VoltEngine($view, $di);
                    $volt->setOptions(
                        [
                            "compiledPath"  => K_PATH . '/var/cache/volt/',
                            'stat'          => K_DEBUG,
                            'compileAlways' => K_DEBUG,
                        ]
                    );
                    return $volt;
                },
                true
            );
    
    Cache

    The cache component is configured using the config.php. We can define the parameters in that file and thus use say the File cache for our local/development machine and a more advanced cache (Memcached for instance) for the production system.

            /**
             * Cache
             */
            $frontConfig = $config->cache_data->front->toArray();
            $backConfig  = $config->cache_data->back->toArray();
            $class       = '\Phalcon\Cache\Frontend\\' . $frontConfig['adapter'];
            $frontCache  = new $class($frontConfig['params']);
            $class       = '\Phalcon\Cache\Backend\\' . $backConfig['adapter'];
            $cache       = new $class($frontCache, $backConfig['params']);
            $di->set('cache', $cache, true);
    
            /**
             * viewCache
             */
            $frontConfig = $config->cache_view->front->toArray();
            $backConfig  = $config->cache_view->back->toArray();
            $class       = '\Phalcon\Cache\Frontend\\' . $frontConfig['adapter'];
            $frontCache  = new $class($frontConfig['params']);
            $class       = '\Phalcon\Cache\Backend\\' . $backConfig['adapter'];
            $cacheView   = new $class($frontCache, $backConfig['params']);
            $di->set('viewCache', $cacheView, true);
    
    Markdown Renderer

    We use Ciconia for the rendering of markdown with several plugins, existing and user defined. The registration is pretty straight forward.

            /**
             * Markdown renderer
             */
            $di->set(
                'markdown',
                function () {
                    $ciconia = new Ciconia();
                    $ciconia->addExtension(new FencedCodeBlockExtension());
                    $ciconia->addExtension(new TaskListExtension());
                    $ciconia->addExtension(new InlineStyleExtension());
                    $ciconia->addExtension(new WhiteSpaceExtension());
                    $ciconia->addExtension(new TableExtension());
                    $ciconia->addExtension(new UrlAutoLinkExtension());
                    $ciconia->addExtension(new MentionExtension());
    
                    $extension = new IssueExtension();
                    $extension->setIssueUrl(
                        '[#%s](https://github.com/phalcon/cphalcon/issues/%s)'
                    );
                    $ciconia->addExtension($extension);
    
                    $extension = new PullRequestExtension();
                    $extension->setIssueUrl(
                        '[#%s](https://github.com/phalcon/cphalcon/pull/%s)'
                    );
                    $ciconia->addExtension($extension);
                    return $ciconia;
                },
                true
            );
    
    Posts Finder

    This is a class we came up with, which is used to give us an easy way to get information about a specific post, the tag cloud, the index page etc. It is utilizing cache a lot!

            /**
             * Posts Finder
             */
            $di->set(
                'finder',
                function () use ($utils, $cache) {
                    $key        = 'post.finder.cache';
                    $postFinder = $utils->cacheGet($key);
                    if (null === $postFinder) {
                        $postFinder = new PostFinder();
                        $cache->save($key, $postFinder);
                    }
                    return $postFinder;
                },
                true
            );
    

    Conclusion

    In the next post of these series we will take a look at the router and discuss what each route means to our application.

    Comments are more than welcome. If you have any questions on the implementation, feel free to ask in the comments below.

    References

  • 2015-10-02 22:42:39

    Every microsecond counts!

    Preface

    One of the primary factors that always needs to be taken into consideration when designing and implementing an application is performance. In this day and age of information overload, everything is about speed. If your website is slow (more than a second or two) to appear on the browser, most likely your visitors will leave and go elsewhere. If the application you designed/implemented is slow, it will use more resources (memory/cpu) and thus cost more money to run. Time is money.

    The Nanosecond

    I have the highest admiration for Grace Hopper, a pioneer computer scientist who invented the first compiler for a computer language and paved the way for the evolution of programming languages in general. In a video she describes the value of a nanosecond in computer terms. I encourage you to click the link and watch it, it might give you a better perspective on why your application must be as fast as possible.

    This Blog

    As I wrote in a previous post, a new version of this blog has been launched, based on work done for for the Phalcon blog. While in development mode, metrics that are printed in the logger. The output looks something like this:

    [Fri, 02 Oct 15 20:10:27 -0400][INFO] Shutdown completed [2.798s] - [4,134.16 KB] 
    [Fri, 02 Oct 15 20:11:01 -0400][INFO] Shutdown completed [2.979s] - [4,134.00 KB] 
    [Fri, 02 Oct 15 20:14:43 -0400][INFO] Shutdown completed [2.891s] - [4,142.60 KB] 
    [Fri, 02 Oct 15 20:26:10 -0400][INFO] Shutdown completed [2.721s] - [1,075.02 KB] 
    [Fri, 02 Oct 15 20:30:16 -0400][INFO] Shutdown completed [2.735s] - [1,002.25 KB] 
    [Fri, 02 Oct 15 20:30:29 -0400][INFO] Shutdown completed [2.708s] - [1,002.29 KB] 
    [Fri, 02 Oct 15 20:54:04 -0400][INFO] Shutdown completed [2.674s] - [1,003.43 KB] 
    [Fri, 02 Oct 15 20:55:28 -0400][INFO] Shutdown completed [2.677s] - [1,003.31 KB] 
    [Fri, 02 Oct 15 21:12:33 -0400][INFO] Shutdown completed [2.013s] - [913.81 KB] 
    [Fri, 02 Oct 15 21:14:11 -0400][INFO] Shutdown completed [2.002s] - [895.35 KB] 
    [Fri, 02 Oct 15 21:32:48 -0400][INFO] Shutdown completed [2.054s] - [894.71 KB] 
    [Fri, 02 Oct 15 21:39:02 -0400][INFO] Shutdown completed [2.028s] - [894.04 KB]
    [Fri, 02 Oct 15 21:44:19 -0400][INFO] Shutdown completed [2.046s] - [895.59 KB]
    [Fri, 02 Oct 15 21:45:55 -0400][INFO] Shutdown completed [2.023s] - [893.75 KB]
    

    As you can see there is room for improvement. Granted these results come from my local installation, where the debugMode is set to 1, which means that there is no caching and everything gets recalculated on every request. Still, if I can make this local installation perform as fast as possible, then on the production server it will be even faster.

    The first few lines show a relatively OK response (2.7-3.0 seconds) but a high usage in memory. This had to be rectified and looking at the code, I managed to refactor the PostFinder class and reduce the memory consumption significantly. Removing objects and referenced objects in them made a huge difference. Arrays work just fine for my purposes.

    Additional optimizations led to dropping the execution time to just above 2.0 seconds and the memory consumption below 1Mb.

    There are still a lot of things I can try (and will) both on the application level as well as the server level. I am aiming to reduce the execution time to below 1 second and I will for sure share the results and the tools/techniques used.

    I hope you enjoy the performance increase (however noticeable). More to come in the near future.

    References

  • 2015-09-28 20:06:28

    New Blog (again), new look, hopefully more posts

    The excuse

    It has definitely been a very long time since I last blogged. Just like my previous hiatus, a lot of personal issues arose that would not allow me the luxury to devote time for this blog.

    Things managed to settle down for a bit, so hopefully I can start posting more frequently.

    New look

    For those that have been following this blog (all 3 of you), you will notice that the look and feel has changed significantly. I decided that I needed a fresh look so after evaluating many platforms and blog related software, I opted for the in house version.

    Many of you know that I have been heavily involved in the Phalcon Framework, where I serve as a member of the core team. For the blogging needs of the project, we were using Tumblr but were not 100% satisfied. So a couple months back we decided to write our own blogging software based on Phalcon!

    The results can be seen on our blog, which is also open sourced on Github.

    For my own purposes I modified the codebase, adding new functionality and modifying it somewhat for my needs. The changes are also on Github and will soon be propagated to the Phalcon repository for everyone to use. This software requires some libraries loaded by composer and of course Phalcon loaded on your web server. The content is served using Markdown.

    As far as the look is concerned, I tried (really did) but nothing came close to what I wanted. Since I am not a CSS developer by any means, I purchased this template by the guys that created Bootstrap. The result IMHO is really nice, responsive and clean. I hope you like it too!

    Future

    I intend on posting more content in the next few weeks. The goal is to post one article (minimum) per week. Other than technology, I am going to focus on Phalcon related posts, sharing experiences, tips and tricks.

    Missing...

    This blog software is still under construction. I have a few things that I need to address such as:

    • Pagination
    • RSS feeds
    • Posts by tag
    • Posts by archive
    • Comments - disqus integration
    • Tests!!!!

    Some of those are already in place but I need to make sure that they work as expected

    I hope you like the new look and stay tuned for more content.

  • 2013-09-15 12:00:00

    Let the RDBMS do more than just store data

    One of the common "mistakes" that programmers (and have been guilty as charged many a times in the past) is not to use the tools that are available to them to the maximum extent possible.

    A common example is using the RDBMS of your choice to only store and retrieve data, without taking advantage of its power and its features to the full extent.

    A RDBMS can do much, much more. One can use triggers that can auto update fields (as I will demonstrate in this blog post), log data into tables, trigger cascade deletes etc.; stored procedures can compute complex data sets, joining tables and transforming data; views can offer easier representations of data, hiding complex queries from the actual application. In addition, such features, like stored procedures/views, can offer security enhancements as well as maintainability to an application. Execution for instance can be restricted to particular groups/logins, while changing the stored procedure/view only requires a change on the database layer and not the application itself.

    In this blog post I will show you a simple example on how one can transfer some of the processing of an application to the RDBMS. I am using MariaDB as the RDBMS and PhalconPHP as the PHP framework.

    The RDBMS

    Each table of my database has several common fields that are used for logging and reporting as well as recording status.

    An example table is as follows

    CREATE TABLE IF NOT EXISTS co_address (
      id             int(11)      unsigned NOT NULL AUTO_INCREMENT,
      address_line_1 varchar(150) COLLATE utf8_unicode_ci DEFAULT NULL,
      address_line_2 varchar(150) COLLATE utf8_unicode_ci DEFAULT NULL,
      address_line_3 varchar(150) COLLATE utf8_unicode_ci DEFAULT NULL,
      region         varchar(6)   COLLATE utf8_unicode_ci DEFAULT NULL,
      post_code      varchar(24)  COLLATE utf8_unicode_ci DEFAULT NULL,
      country        varchar(2)   COLLATE utf8_unicode_ci DEFAULT NULL,
      created_id     int(11)      unsigned NOT NULL DEFAULT '0',
      created_date   datetime              NOT NULL,
      updated_id     int(11)      unsigned NOT NULL DEFAULT '0',
      updated_date   datetime              NOT NULL,
      deleted        tinyint(1)   unsigned NOT NULL DEFAULT '0',
      PRIMARY KEY (id),
      KEY created_id (created_id),
      KEY created_date (created_date),
      KEY updated_id (updated_id),
      KEY updated_date (updated_date),
      KEY deleted (deleted)
    ) ENGINE=InnoDB  
    DEFAULT CHARSET=utf8
    COLLATE=utf8_unicode_ci
    COMMENT='Holds addresses for various entities' AUTO_INCREMENT=1 ;
    

    The fields are:

    Field Name Description
    created_id The id of the user that created the record
    created_date The date/time that the record was created
    updated_id The id of the user that last updated the record
    updated_date The date/time that the record was last updated
    deleted A soft delete flag

    There is not much I can do with the user ids (created/updated) or the deleted column (see also notes below regarding this). However as far as the dates are concerned I can definitely let MariaDB handle those updates.

    Triggers

    The work is delegated to triggers, attached to each table.

    --
    -- Triggers address
    --
    DROP TRIGGER IF EXISTS trg_created_date;
    DELIMITER //
    CREATE TRIGGER trg_created_date BEFORE INSERT ON address
     FOR EACH ROW SET NEW.created_date = NOW(), NEW.updated_date = NOW()
    //
    DELIMITER ;
    DROP TRIGGER IF EXISTS trg_updated_date;
    DELIMITER //
    CREATE TRIGGER trg_updated_date BEFORE UPDATE ON address
     FOR EACH ROW SET NEW.updated_date = NOW()
    //
    DELIMITER ;
    

    The triggers above update the created_date and updated_date fields automatically upon insert/update.

    Phalcon Model

    I needed to make some changes to my model Address, in order to allow the triggers to work without interference from the model.

    class Model extends PhModel
    {
        public function initialize()
        {
            // Disable literals
            $this->setup(['phqlLiterals' => false]);
    
            // We skip these since they are handled by the RDBMS
            $this->skipAttributes(
                [
                    'created_date',
                    'updated_date',
                ]
            );
    
        }
    
        public function getSource()
        {
            return 'address';
        }
    
        public function getCreatedDate()
        {
            return $this->created_date;
        }
    
        public function getUpdatedDate()
        {
            return $this->updated_date;
        }
    }
    

    By using skipAttributes, I am instructing the Phalcon model not to update those fields. By doing so, I am letting my triggers worry about that data.

    Conclusion

    It might seem a very trivial task that I am delegating but in the grand scheme of things, the models of an application can be very complex and have a lot of logic in them (and so might controllers). Delegating some of that logic in the RDBMS simplifies things and also increases performance of the application, which now requires just a bit less computational power.

    NOTES

    For a soft delete feature i.e. automatically updating the deleted field when a DELETE is called, a trigger will not work. Instead one can use a stored procedure for it. See this Stack Overflow answer.

  • 2013-06-25 12:00:00

    Voting for Phalcon as a cPanel feature woes

    Those that have been following my blog and posts on Google+ know that for the last year or so I have been involved heavily in PhalconPHP, a C based PHP framework, which delivers its functionality as an extension loaded on the web server.

    I was honored a few months ago when I became a member of the Phalcon Team and have since tried my best to evangelize Phalcon and to help as much as possible with scheduling NFRs for development, helping in the forum, unit tests, blog posts etc.

    One of the most difficult issues that Phalcon is facing is helping developers installing it on their machines. This of course does not mean that it is really difficult to install. The installation is basically three commands (or if you are on a Windows box you just download the DLL, add the relevant directive in the php.ini and restart the web server).

    However since a lot of people are using shared hosting, they do not have access to the command line where sudo or su are available so that that Phalcon can be installed. This is left to the hosting company and some are very reluctant to install anything at all. I used to own a hosting company and I can assure you that it is indeed a hassle if a handful of clients ask for a library or an installation that is not part of the "norm". You have to maintain it, you have to ensure that it will not interfere with other packages on the server or hinder other clients that reside on the same hosting box.

    A few months ago I approached cPanel through their ticketing system, in an effort to make Phalcon an option for the extensions that can be installed and loaded through their EasyApache application.

    At the time I was granted a development account for the software so that I can try and create an installation script and also was prompted to go to http://features.cpanel.net/ and open a new feature request regarding this (the request is here). The purpose of this exercise is for cPanel to get a feel of what features the community needs and address them. In my communications with them I received the following (emphasis mine):

    Our EasyApache has multiple locations to add includes, or post hooks to compile third party libraries/software during a build, or after a build has completed. That being said, the extensions that we ship to be configured are maintained by the PHP & PECL groups. To answer the question as far as integration on all servers running cPanel, I would recommend creating a feature request for this, and allowing the community to vote for this:

    http://features.cpanel.net/

    And so I did. I created the feature request and we also advertised this in our community via our forum and our blog post.

    The feature received well over 180 votes, making it the third most requested feature in cPanel. Also if you follow the link, one of the engineers of cPanel requested additional information which was provided by myself and others.

    As time went by, I visited the feature and requested an update but never got a reply back. More and more votes kept on coming in so it was really a waiting game at that time.

    All of a sudden though a week ago, one of our users in the forum asked in our Forum why the feature request in cPanel has only 7 votes. This came to us as a surprise so I went and checked it out. Lo and behold the vote counter was at 7 and not at 180+. Assuming that this was some sort of a glitch, I opened a ticket with cPanel and inquired about this.

    A short time later I received a reply from a Vice President of Operations stating (emphasis mine):

    Our feature request system was designed for features requests from our customers and in reviewing this feature it was determined that most (if not all of the votes) came from an outside source. In an effort to validate this we polled a number of shared hosting providers that we work closely with and this was not a feature they wanted.

    We traced most of the votes down to this single source:

    https://blog.phalconphp.com/post/help-the-community-to-make-phalcon-available-on-cpanel

    While cPanel wants feedback from the general community, our focus is to deliver new features that our Partners and customers are asking for.

    We always appreciate community support and will keep an eye on this feature. Decisions for new features are made by both reviewing this feature request system and talking with Partners and customers. To date out of about 15 conversations, not a single Partner or customer told us this was something that was important to them and thus we adjusted the votes to what we felt was more in line with the community using our system.

    I have a couple of issues with the above reply.

    • For starters there was no communication and no warning that would have indicated what was going to happen with our votes. cPanel decided on this on their own. They do have the right to do so, it's their software after all, but a little courtesy would have gone a long way.
    • The definition of "customers" is different to me than it is to cPanel. As customers cPanel defines their partners and hosting companies, the ones that purchase their software. Myself on the other hand considers customers also the end user, me and you who visit a hosting company and purchase services from them. If we do not exist, then a hosting company and subsequently cPanel does not exist either.
    • cPanel asked a number of their partners who have not heard of Phalcon before, and as such they acted based on that premise. What would have happened if their sample was a different one? Say they asked partner X instead of partner Y? The bias of statistics is based on the sample one chooses and at times (such as this one) it could lead to false results.

    In my reply I pointed out the above, stating that a customer should effectively be the end user and if not, at least their voice should be heard. The reply that I got was as follows:

    Thanks for the reply and understanding. In reviewing the votes that came in, most appeared to be brand advocates of Phalcon and it was very difficult to discern the legitimacy of the votes. Had the votes come from active users of forums.cpanel.net (where most of the users originate from) or we were able to relate them back to some sort of hosting entity, we would have left the votes.

    From the outside looking in the votes just appeared to come from Phalcon users without ties to cPanel & WHM. What I would encourage you to do is the following:

    • Have your users ask their hosting provider about it.
    • Remove the blog links and tweets and allow the feature to grow it's own set of wheels.

    If this feature is truly in high demand from customers of cPanel, Inc. they will naturally vote for it.

    We do appreciate your support and ongoing efforts to get this in front of us. The massive amounts of votes and comments it received, put it on our developers radars and we will continue to monitor the situation.

    So in essence, if we want to achieve our objective, i.e. get Phalcon as an available extension for cPanel, we need to advise the community (but be careful to Remove the blog links and tweets and allow the feature to grow it's own set of wheels.) to contact their hosting companies (that use cPanel) to in turn contact cPanel and request Phalcon to be included as an extension.

    The fallacy of the above is that developers of Phalcon will not choose hosts that offer cPanel because they cannot install the application. If they cannot install the application they will not use hosts that offer cPanel, thus they cannot ask their hosts to include Phalcon as a cPanel extension, and so goes the chicken and egg situation.

    I totally respect cPanel's decisions - I don't agree with them but I do respect them. It's their house their rules as they say. I am however saddened by the fact that we never got any communication or warning that our votes were removed. A bit of communication there would have definitely saved a lot of frustration at least for our community.

    Concluding, if anyone has a cPanel hosting account and wants to see Phalcon available as an extension, feel free to contact your host and request it to be included as an available extension.

    2013-07-25 Update: A great analogy and reply has been posted by Andres in the Phalcon Forum.

  • 2013-05-27 12:00:00

    2013 Memorial Day

    Once again we celebrate Memorial Day in the USA.

    A very special day, dedicated to remembering and honoring all those men and women in uniform that fought for freedom. Freedom that nowadays - in most cases - is taken for granted.

    A personal thank you to all of those that paid the ultimate price, with their life, so that my family, myself, my neighbors, my co-patriots can enjoy what we have today: our freedom.

    You might also enjoy material from the following resources:

    My posts of 2010 and 2012 as well as http://www.usamemorialday.org.

  • 2012-11-25 12:00:00

    Building a web app with PhalconPHP and AngularJS Update

    It's been a while since I last wrote a blog post, so I wanted to touch on the effort to upgrade the application that I wrote for Harry Hog Fottball using PhalconPHP and AngularJS

    If you haven't read it, the first two blog posts were here and here.

    The application was written using the 0.4.5 version of PhalconPHP. Since then there have been significant changes to the framework, such as the introduction of a DI container, injectable objects and lately interfaces (in 0.7.0, to be released in a couple of days), I had to make some changes.

    There are a couple of things that I as a developer would like to see in PhalconPHP, which I am pretty sure will appear later on, since let's face it the framework is still very young (not even 1.0 version yet). Despite its "youth" it is a robust framework with excellent support, features and a growing community. One of these features is behaviors which I had to implement myself, and this was something new that came with this upgrade.

    Recently a new repo has been created on Github called the incubator, where developers can share implementations of common tasks, that act as drop ins to the framework and extend it. These implementations are all written in PHP so everyone can just download them and use them. The more submissions come in, the more the framework will grow and eventually these submissions will become part of the framework itself.

    Converting the 0.4.x application to 0.5.x

    The task of converting everything from 0.4 to 0.5 was a bit challenging. The reason behind it was the DI container and how best to use it to suit the needs of the current application. Now these challenges would not even be an issue if one started writing their application from scratch, but since I had everything in place, I ventured into upgrading vs. rewriting. Note that this kind of upgrade will most likely never happen again, since the framework has been changed accordingly so that future upgrades will not require developers to rewrite their code (like I did now). From 0.5.x onward the framework design has been kind of "frozen".

    I decided to create a new library that will help me with my tasks. I therefore created a custom bootstrap class, that would instantiate everything I wanted in my code. A short snippet of the class is below (the full code of course is in my Github repo which you are more than welcome to download and modify to suit your needs)

    namespace NDN;
    
    use \Phalcon\Config\Adapter\Ini as PhConfig;
    use \Phalcon\Loader as PhLoader;
    ....
    use \Phalcon\Exception as PhException;
    
    class Bootstrap
    {
        private $_di;
    
        /**
         * Constructor
         * 
         * @param $di
         */
        public function __construct($di)
        {
            $this->_di = $di;
        }
    
        /**
         * Runs the application performing all initializations
         * 
         * @param $options
         *
         * @return mixed
         */
        public function run($options)
        {
            $loaders = array(
                'config',
                'loader',
                'environment',
                'timezone',
                'debug',
                'flash',
                'url',
                'dispatcher',
                'view',
                'logger',
                'database',
                'session',
                'cache',
                'behaviors',
            );
    
    
            try {
                foreach ($loaders as $service)
                {
                    $function = 'init' . ucfirst($service);
    
                    $this->$function($options);
                }
    
                $application = new PhApplication();
                $application->setDI($this->_di);
    
                return $application->handle()->getContent();
    
            } catch (PhException $e) {
                echo $e->getMessage();
            } catch (\PDOException $e) {
                echo $e->getMessage();
            }
        }
    
        // Protected functions
    
        /**
         * Initializes the config. Reads it from its location and
         * stores it in the Di container for easier access
         *
         * @param array $options
         */
        protected function initConfig($options = array())
        {
            $configFile = ROOT_PATH . '/app/var/config/config.ini';
    
            // Create the new object
            $config = new PhConfig($configFile);
    
            // Store it in the Di container
            $this->_di->set('config', $config);
        }
    
        /**
         * Initializes the loader
         *
         * @param array $options
         */
        protected function initLoader($options = array())
        {
            $config = $this->_di->get('config');
    
            // Creates the autoloader
            $loader = new PhLoader();
    
            $loader->registerDirs(
                array(
                    ROOT_PATH . $config->app->path->controllers,
                    ROOT_PATH . $config->app->path->models,
                    ROOT_PATH . $config->app->path->library,
                )
            );
    
            // Register the namespace
            $loader->registerNamespaces(
                array("NDN" => $config->app->path->library)
            );
    
            $loader->register();
        }
    
        ....
    
        /**
         * Initializes the view and Volt
         *
         * @param array $options
         */
        protected function initView($options = array())
        {
            $config = $this->_di->get('config');
            $di     = $this->_di;
    
            $this->_di->set(
                'volt',
                function($view, $di) use($config)
                {
                    $volt = new PhVolt($view, $di);
                    $volt->setOptions(
                        array(
                            'compiledPath'      => ROOT_PATH . $config->app->volt->path,
                            'compiledExtension' => $config->app->volt->extension,
                            'compiledSeparator' => $config->app->volt->separator,
                            'stat'              => (bool) $config->app->volt->stat,
                        )
                    );
                    return $volt;
                }
            );
        }
        ....
    
        /**
         * Initializes the model behaviors
         *
         * @param array $options
         */
        protected function initBehaviors($options = array())
        {
            $session = $this->_di->getShared('session');
    
            // Timestamp
            $this->_di->set(
                'Timestamp',
                function() use ($session)
                {
                    $timestamp = new Models\Behaviors\Timestamp($session);
                    return $timestamp;
                }
            );
        }
    }
    

    I chose to show a few sections of this bootstrap which I will explain shortly. What this bootstrap class does is it initializes my whole environment and keeps my index.php file small.

    error_reporting(E_ALL);
    
    try {
    
        if (!defined('ROOT_PATH')) {
            define('ROOT_PATH', dirname(dirname(__FILE__)));
        }
    
        // Using require once because I want to get the specific
        // bootloader class here. The loader will be initialized
        // in my bootstrap class
        require_once ROOT_PATH . '/app/library/NDN/Bootstrap.php';
        require_once ROOT_PATH . '/app/library/NDN/Error.php';
    
        // Instantiate the DI container
        $di  = new \Phalcon\DI\FactoryDefault();
    
        // Instantiate the boostrap class and inject the DI container 
        // in it so that services can be registered
        $app = new \NDN\Bootstrap($di);
    
        // Here we go!
        echo $app->run(array());
    
    } catch (\Phalcon\Exception $e) {
        echo $e->getMessage();
    }
    

    As you can see the index.php is very small in terms of code.

    Let's have a look at a couple of the functions that are in the bootstrap.

        /**
         * Initializes the config. Reads it from its location and
         * stores it in the Di container for easier access
         *
         * @param array $options
         */
        protected function initConfig($options = array())
        {
            $configFile = ROOT_PATH . '/app/var/config/config.ini';
    
            // Create the new object
            $config = new PhConfig($configFile);
    
            // Store it in the Di container
            $this->_di->set('config', $config);
        }
    

    Pretty straight forward. The config INI file is read from its location and stored in the DI container. I need to do this first, since a lot of the parameters of the application are controlled from that file.

        /**
         * Initializes the loader
         *
         * @param array $options
         */
        protected function initLoader($options = array())
        {
            $config = $this->_di->get('config');
    
            // Creates the autoloader
            $loader = new PhLoader();
    
            $loader->registerDirs(
                array(
                    ROOT_PATH . $config->app->path->controllers,
                    ROOT_PATH . $config->app->path->models,
                    ROOT_PATH . $config->app->path->library,
                )
            );
    
            // Register the namespace
            $loader->registerNamespaces(
                array("NDN" => $config->app->path->library)
            );
    
            $loader->register();
        }
    

    The loader is what does all the discovery of classes for me. As you can see I store a lot of the paths in the config INI file, and I register my custom namespace NDN.

        /**
         * Initializes the view and Volt
         *
         * @param array $options
         */
        protected function initView($options = array())
        {
            $config = $this->di->get('config');
            $di     = $this->_di;
    
            $this->_di->set(
                'volt',
                function($view, $di) use($config)
                {
                    $volt = new PhVolt($view, $di);
                    $volt->setOptions(
                        array(
                            'compiledPath'      => ROOT_PATH . $config->app->volt->path,
                            'compiledExtension' => $config->app->volt->extension,
                            'compiledSeparator' => $config->app->volt->separator,
                            'stat'              => (bool) $config->app->volt->stat,
                        )
                    );
                    return $volt;
                }
            );
        }
    

    This is an interesting one. Registering the view and Volt. Volt is the template engine that comes with Phalcon. It is inspired by Twig and written in C, thus offering maximum performance. I set the compiled path, extension and separator for the template files and also I have a variable (set in the config of course) to allow the application to always create template files or not. In a production environment that variable (stat) will be set to false since templates do not change.

        /**
         * Initializes the model behaviors
         *
         * @param array $options
         */
        protected function initBehaviors($options = array())
        {
            $session = $this->_di->getShared('session');
    
            // Timestamp
            $this->_di->set(
                'Timestamp',
                function() use ($session)
                {
                    $timestamp = new Models\Behaviors\Timestamp($session);
                    return $timestamp;
                }
            );
        }
    

    The above is my implementation of behaviors. Of course it is far from perfect but it works the way I want to. A better implementation of this has been written by Wojtek Gancarczyk and is available in the incubator. All I do here is go through the behaviors I have (Timestamp only for now) and register them in the DI container so that I can reuse them later on with any model that needs them.

    Models

    Every model I have that interacts with my database tables extends the NDN\Model.

    class Model extends \Phalcon\Mvc\Model
    {
        protected $behaviors = array();
    
        /**
         * Adds a behavior in the model
         *
         * @param $behavior
         */
        public function addBehavior($behavior)
        {
            $this->behaviors[$behavior] = true;
        }
    
        public function beforeSave()
        {
            $di   = Di::getDefault();
    
            foreach ($this->behaviors as $behavior => $active)
            {
                if ($active && $di->has($behavior))
                {
                    $di->get($behavior)->beforeSave($this);
                }
            }
        }
    
        /**
         * @param array $parameters
         *
         * @static
         * @return Phalcon_Model_Resultset Model[]
         */
        static public function find($parameters = array())
        {
            return parent::find($parameters);
        }
    
        /**
         * @param array $parameters
         *
         * @static
         * @return  Phalcon_Model_Base   Models
         */
        static public function findFirst($parameters = array())
        {
            return parent::findFirst($parameters);
        }
    }
    

    The class itself is pretty simple, offering find and findFirst to the class that extends this. The interesting thing is that it also registers behaviors and calls the relevant validator function. So for instance the beforeSave validator checks the registered behaviors ($behaviors array), checks if they are active, checks if they exist in the DI container and gets them from there and then calls the beforeSave in the behavior class.

    The behavior class is equally simple:

    class Timestamp
    {
        protected $session;
    
        public function __construct($session)
        {
            $this->session= $session;
        }
    
        /**
         * beforeSave hook - called prior to any Save (insert/update)
         */
        public function beforeSave($record)
        {
            $auth     = $this->session->get('auth');
            $userId   = (isset($auth['id'])) ? (int) $auth['id'] : 0;
            $datetime = date('Y-m-d H:i:s');
            if (empty($record->created_at_user_id)) {
                $record->created_at         = $datetime;
                $record->created_at_user_id = $userId;
            }
            $record->last_update         = $datetime;
            $record->last_update_user_id = $userId;
        }
    }
    

    So effectively every time I call the save() function on a model, this piece of code will be executed, populating my fields with the date time and the user that created the record and/or updated it.

    In order to get this functionality to work, all I have to do in my model is to register the behavior like so:

    class Episodes extends \NDN\Model
    {
        /**
         * Initializes the class and sets any relationships with other models
         */
        public function initialize()
        {
            $this->addBehavior('Timestamp');
            $this->hasMany('id', 'Awards', 'episode_id');
        }
    }
    

    Controllers

    Very little has changed in the controller logic, so that was the easiest part of the upgrade. Of course I tweaked a few things but the code works as is. I still extended my custom NDN\Controller class which takes care of my breadcrumbs (NDN\Breadcrumbs) as well as the construction of the top menu. The biggest difference with the previous version is that I stopped using AngularJS to populate the menu (so I am no longer sending a JSON array in the view) and used Volt instead. It was a matter of preference and nothing more.

    Views

    Quite a bit of work had to be done in the views to switch everything to use Volt. Of course every view extension had to be changed to .volt but that was not the only change. I split the layout to use partials so that the header, navigation and footer are different sections (organizing things a bit better) and kept the master layout index.volt.

    I started using the built in Volt functions to generate content as well as tags and it was a nice surprise to see that everything was easy to use and it worked!

    <!DOCTYPE html>
    <html ng-app='HHF'>
        {{ partial('partials/header') }} 
        <body>
            <div id="spinner" style="display: none;">
                {{ image('img/ajax-loader.gif') }} Loading ...
            </div>
    
            {{ partial('partials/navbar') }}
    
            <div class='container-fluid'>
                <div class='row-fluid'>
                    <ul class='breadcrumb'>
                        <li>
                            {% for bc in breadcrumbs %}
                            {% if (bc['active']) %}
                            {{ bc['text'] }}
                            {% else %}
                            <a href='{{ bc['link'] }}'>{{ bc['text'] }}</a> 
                            <span class='divider'>/</span>
                            {% endif %}
                            {% endfor %}
                        </li>
                    </ul>
                </div>
    
                <?php echo $this->flash->output() ?>
    
                <div class="row-fluid">
                    <?php echo $this->getContent() ?>
                </div> <!-- row -->
    
                {{ partial('partials/footer') }}
            </div>
    
            {{ javascript_include(config.app.js.jquery, config.app.js.local) }}
            {{ javascript_include(config.app.js.jquery_ui, config.app.js.local) }}
            {{ javascript_include(config.app.js.bootstrap, config.app.js.local) }}
            {{ javascript_include(config.app.js.angular, config.app.js.local) }}
            {{ javascript_include(config.app.js.angular_resource, config.app.js.local) }}
            {{ javascript_include(config.app.js.angular_ui, config.app.js.local) }}
            {{ javascript_include('js/utils.js') }}
    
        </body>
    </html>
    

    The above is the index.volt. As you can see I call on the partials/header.volt, then the partials/navbar.volt (where the menu is generated) and then I construct the breadcrumbs (note the {% for bc in breadcrumbs %} block). After that the flash messenger comes into play, the main content displayed, the footer and finally the javascript includes that I need.

    I am still using AngularJS to make the necessary AJAX calls so that the relevant controller to retrieve the data but also to display this data on screen (which is cached to avoid unnecessary database hits).

    The Episodes view became

    {{ content() }}
    
    <div>
        <ul class='nav nav-tabs'>
            <li class='pull-right'>
                {{ addButton }}
            </li>
        </ul>
    </div>
    
    <div ng-controller='MainCtrl'>
        <table class='table table-bordered table-striped ng-cloak' ng-cloak>
            <thead>
            <tr>
                <th><a href='' ng-click="predicate='number'; reverse=!reverse">#</a></th>
                <th><a href='' ng-click="predicate='air_date'; reverse=!reverse">Date</a></th>
                <th><a href='' ng-click="predicate='outcome'; reverse=!reverse">W/L</a></th>
                <th><a href='' ng-click="predicate='summary'; reverse=!reverse">Summary</a></th>
            </tr>
            </thead>
            <tbody>
                <tr ng-repeat="episode in data.results | orderBy:predicate:reverse">
                    <td>[[episode.number]]</td>
                    <td width='7%'>[[episode.air_date]]</td>
                    <td>[[episode.outcome]]</td>
                    <td>[[episode.summary]]</td>
                    {% if (addButton) %}
                    <td width='1%'><a href='/episodes/edit/[[episode.id]]'><i class='icon-pencil'></i></a></td>
                    <td width='1%'><a href='/episodes/delete/[[episode.id]]'><i class='icon-remove'></i></a></td>
                    {% endif %}
                </tr>
            </tbody>
        </table>
    </div>
    

    The beauty of AngularJS! I only have to pass a JSON array with my results. ng-repeat with the orderBy filter allows me to present the data to the user and offer sorting capabilities per column. This is all done at the browser level without any database hits! Pretty awesome feature!

    For those that have used AngularJS in the past, you will note that I had to change the interpolate provider (i.e. the characters that wrap a string or a piece of code that AngularJS understands). Usually these characters are the curly brackets {{ }} but I changed them to [[ ]] to avoid collisions with Volt.

    This was done with a couple of lines of code in my definition of my AngularJS model:

    var ngModule = angular.module(
            'HHF', 
            ['ngResource', 'ui']
        )
        .config(
            function ($interpolateProvider) {
                $interpolateProvider.startSymbol('[[');
                $interpolateProvider.endSymbol(']]');
            }
        )
    

    Conclusion

    I spent at most a day working on this mostly because I wanted to try various things and see how it works. The actual time to convert the application (because let's face it, it is a small application) was a couple of hours inclusive of the time it took me to rename certain fields, restructure the folder structure, compile the new extension on my server and upload the data upstream.

    I am very satisfied with both AngularJS, which helps tremendously in my presentation layer, as well as with Phalcon. Phalcon's new design makes implementation a breeze, while AngularJS offers a lot of flexibility on the view layer.

    As written before, you are more than welcome to download the source code of this application here and use it for your own needs. Some resources are:

    References

  • 2012-09-02 12:00:00

    AngularJS - Simplicity in your browser

    Recently I was contacted by an acquaintance through my Google+ circles, who needed some help with a project of hers.

    Her task was to redesign a church website. Pretty simple stuff, CSS, HTML and content.

    Scope

    The particular church videotapes all the sermons and posts them on their channel in LiveStream for their followers to watch. One of the requirements was to redo the video archives page and to offer a link where followers can download the audio of each sermon for listening.

    Design (kinda)

    After the initial contact, I decided to get rid of all the bloated jQuery code that was there to control the video player and use AngularJS to control the generation of content. There were two key facts that influenced my decision:

    • the use of ng-repeat to generate the table that will list all the available sermons and
    • the variable binding that AngularJS offers to play the video in the available player.

    I also decided to switch the player to a new updated one that LiveStream offered, which features a slider to jump through the video, volume control and more.

    Previous code

    The previous code for that page was around 300 lines. The file had some CSS in it, quite a few lines of HTML but was heavy on javascript. There were a lot of jQuery functions which controlled the retrieval of the available videos per playlist. Each playlist would be effectively a collection of videos for a particular year. jQuery was observing clicks on specific links and make an AJAX call to the LiveStream API to retrieve the list of available data in JSON format, and output the formatted results (as a table) on screen. It was something like this:

    head
    title
    link - css
    style (inline)
    ....
    end inline style
    script jQuery
    script jQueryUI
    script jquery.timer
    link jquery-ui CSS
    link slider CSS
    style (inline)
    ....
    end inline style
    script jQuery
    $(document).ready ... // Document ready
    ....
    $("#div_col dd").click // Bind click to a year
    .....
    getPlaylists(year) // Not sure why this was ever here, not used
    ....
    getPlaylistClips(playlistID) // Gets the clips of the playlist
    .....
    playClip(clipID) // Plays the clip in the player
    .....
    end jQuery script
    script Video Player
    ....
    end head
    body
    navigation
    main container
    list of years
    instructions
    container to show video player
    container to show video list
    end body
    end html
    

    Enter AngularJS

    I checked the latest video player from LiveStream. The code was much cleaner and all I had to do is bind one variable, the GUID of the video, in the relevant call so that the video can be played. I also bound another variable (the video title) above the video so as to offer more information to the user.

    With a bit of Bootstrap CSS, I created two tabs and listed the two years 2012, 2011. A function was created in my AngularJS module to accept the year and make the relevant call to the LiveStream API to receive the data a a JSON object.

    ng-repeat (with >ng-cloak) was used to "print" the data on screen and the application was ready.

    I removed all the cruft and created one small file that is loaded and offers the functionality that we need. It is 50 lines of code (just the javascript part. The code is below with added comments for the reader to follow:

    // Create the module and inject the Resource object
    var ngModule = angular.module("CHF", ['ngResource']);
    
    // The main controller that needs the scope and resource
    ngModule.controller("MainCtrl", function ($scope, $resource) {
    
        // Calculates the current year
        //  ensures we always get the last year on first load
        $scope.currentYear = function () {
            var currentDate = new Date();
            return currentDate.getFullYear();
        };
    
        // This is the playlist array. This is obtained by 
        //  LiveStream and it changes once every year. 
        //  Hardly an effort by the administrator
        $scope.playlists = [
            {"year":"2012", "guid":"63426-xxx-xxx-xxx"},
            {"year":"2011", "guid":"84f84-xxx-xxx-xxx"}
        ];
    
        // This couldn't be simpler. It merely sets some variables 
        //  in the scope. By doing so, the binding in the relevant
        //  variables will allow the video to play and the title 
        //  to update.
        $scope.playVideo = function (element) {
            $scope.currentVideo  = element.guid;
            $scope.currentTitle  = element.title;
        };
    
        // This is the core. It makes the AJAX request to the 
        // LiveStream API so that it can get the JSON data back
        $scope.makeRequest = function (year) {
    
            // Calculating the current year and the year selected.
            // Their difference offers an offset which effectively 
            // is the offset of the array stored in $scope.playlists
            var thisYear    = $scope.currentYear();
            var diff        = thisYear - year;
    
            var objData = $scope.playlists[diff];
    
            // Just in case something was passed that is not valid
            if (objData.guid)
            {
                var reqData = $resource(
                    "http://livestream_url/2.0/:action",
                    {
                        action:'listclips.json', 
                        id:objData.guid,
                        query: {isArray: true},
                        maxresults:"500",
                        callback:'JSON_CALLBACK'
                    },
                    {get:{method:'JSONP'}}
                );
    
                // Set the year and get the data
                $scope.year    = year;
                $scope.listData = reqData.get();
           }
        };
    
        // This is the first load - load the current year
        $scope.makeRequest($scope.currentYear());
    
    });
    

    Now moving into the HTML side of things:

    <div id="playerContainer" style='text-align:center;'>
        <p ng-cloak>
            {{currentTitle}}
        </p>
        <iframe 
            width="560" 
            height="340" 
            src="http://ls_url?clip={{currentVideo}}&params" 
            style="border:0;outline:0" 
            frameborder="0" 
            scrolling="no">
        </iframe>
    </div>
    
    <br />
    <div>
        <ul class="nav nav-tabs">
            <li ng-repeat="playlist in playlists" 
                   ng-class="{true:'active',false:''}[year == playlist.year]">
                <a ng-click="makeRequest(playlist.year)">{{playlist.year}}</a>
            </li>
        </ul>
        <table class="table table-bordered" style="width: 100%;">
            <thead>
                <tr>
                    <th>Date/Title</th>
                    <th>Audio</th>
                </tr>
            </thead>
            <tbody>
                <tr ng-cloak ng-repeat="video in listData.channel.item">
                    <td ng-click="playVideo(video)">{{video.title}}</td>
                    <td>
                        <span ng-show="video.description">
                            <a href="{{video.description}}" title="Download Audio">
                                <i class="icon-download-alt"></i>
                            </a>
                        </span>
                    </td>
                </tr>
            </tbody>
        </table>
    </div>
    

    That is all the HTML I had to change. The full HTML file is 100 lines and 50 for the AngularJS related javascript, I can safely say that I had a 50% reduction in code offering the same functionality - and if I might say so, it is much much cleaner.

    The final page looks something like this:

    Pointers

    <div id="playerContainer" style='text-align:center;'>
        <p ng-cloak>
        {{currentTitle}}
        </p>
        <iframe 
            width="560" 
            height="340" 
            src="http://ls_url?clip={{currentVideo}}&params" 
            style="border:0;outline:0" 
            frameborder="0" 
            scrolling="no">
        </iframe>
    </div>
    

    This block displays the video player and due to the variable binding that AngularJS offers, the minute those variables change, the video is ready to be played.

    <ul class="nav nav-tabs">
        <li ng-repeat="playlist in playlists" 
               ng-class="{true:'active',false:''}[year == playlist.year]">
            <a ng-click="makeRequest(playlist.year)">{{playlist.year}}</a>
        </li>
    </ul>
    

    This block shows the tabs depicting each playlist. In our case these are years. ng-repeat does all the hard work, printing the data that is defined in our JS file. The ng-class is there to change the class of the tab to "active" when the tab is clicked/selected. The ng-click initiates a request through makeRequest, a function defined in our javascript file (see above).

    <tbody>
        <tr ng-cloak ng-repeat="video in listData.channel.item">
            <td ng-click="playVideo(video)">{{video.title}}</td>
            <td>
                <span ng-show="video.description">
                    <a href="{{video.description}}" 
                       title="Download Audio">
                        <i class="icon-download-alt"></i>
                    </a>
                </span>
            </td>
        </tr>
    </tbody>
    

    Finally the data is displayed on screen. ng-cloak makes sure that the content is displayed only when the data is there (otherwise browsers might show something like {{video.description}} which is not nice from a UI perspective). ng-repeat loops through the data and "prints" the table.

    The description of the video is used as a storage for the URL that will point to the MP3 audio file so as the users can download it. Therefore I use ng-show to show the link, if it exists.

    Conclusion

    This whole project was no more than 30 minutes, which included the time I had to research and experiment a bit with the LiveStream API. This is a flexible design, with much much cleaner code (and a lot less of it). When the admin needs to add a new playlist (aka year), all they have to do is open the JS file and type a new element in the $scope.playlists array. The application will take care of the rest automatically.

    I cannot think of doing this with less lines of code than this.

    If you haven't heard of AngularJS or used it, I would highly encourage you to give it a shot. Great project, awesome support and a very very responsive, helpful and polite community.