Continuous Deployment with Symfony2, Jenkins and Capifony

It seems that many people talk about setting up a continuous deployment system but few actually take the plunge and make it a reality. I’ve recently set up continuous deployment for an API project at work and thought I would blog about how I got it all to work.

The Plan

This project is relatively new and we’d setup the following process for it:

  1. We’re using Git and GitHub for the project. Every developer working on it uses a fork of the main repository.
  2. When some work is ready to be merged we send a pull request. Another developer reviews the code before it is merged.
  3. Once code is merged into the main repository GitHub notifies our Jenkins server, which then runs a build on the project. This performs many tasks including lint checking changed files, running our unit and functional test suite, checking that our database is in sync with our entities and performing various metrics against our code base.

What we needed is for code to be automatically deployed at the end of a build only if the build succeeds, i.e. all tests are passing, there are no syntax errors and the entities are in sync with the database. Deployments should also be able to be quickly rolled back should anything go wrong with code in production.

Enter Capifony

To automate the deployment we chose to work with Capifony. This is a deployment solution for Symfony 1 and 2 apps that is a series of deployment recipes that is built on top of Capistrano. This does entail installing Ruby and Ruby Gems on the system from which you want to run the deployment but the advantages it offers outweigh any hassle this may entail. Some of the advantages of using Capifony are:

  1. Once your deployment script is defined running a deployment is as simple as changing into the root directory of your project and typing the command ‘cap deploy’.
  2. Capifony stores multiple releases (you get to choose how many, the default is 3) of your project on your server. A symlink called current is created that links to the most recent release. This makes rolling back the project in the event of problems a cinch: to go back to the previous deployment simply type ‘cap deploy:rollback’ from the root directory of your project.
  3. Capifony performs deployments by logging into the production server over SSH. It uses SCM such as Git to pull code down to the server.
  4. Since Capifony connects to the server using SSH it can run any bash commands or symfony console commands that you wish during the deployment.
  5. The deployment is run as a ‘transaction’: if any part of it fails the entire deployment is rolled back and aborted.

Capifony adds a number of default tasks that it performs with every deployment. These can easily be extended or overridden in your own deploy.rb script. Capifony can also run any symfony console command on the live server as part of the deployment by using the keyword ‘symfony’. For example, to update the Symfony2 translation file you would simply need to add ‘run “symfony:translation:update”‘ to your deploy.rb file.

Once you have installed capifony you need to cd to the root directory of your project and to run ‘capifony .’ in the terminal. This will automatically detect if you’re working wth a Symfony 1 or 2 application and will create a Capfile in in the root directory as well as creating a  deploy.rb file in the config directory. These need to be added to source control. Once you’ve setup a basic deployment script run ‘cap deploy:setup’ to setup the deployment on the remote server.

A sample deployment script

Below is a sample deploy.rb script that I’m using to deploy a Symfony2 app:

A few notes on this:

  • Since Capifony performs actions on the live server it needs a user to log in as to perform actions. I created a user on the server called deploy with the primary group of www-pub and the secondary group of www-data. I then added www-pub as a secondary group to the user www-data that apache runs as. This solved any permission issues that I had.
  • Setting the option “set :deploy_via, :remote_cache” tells capifony to keep a local, cached copy of the repository on the server. This speeds up deployments since only changes to the code base need to be fetched. We needed to install Git on the server to allow this to work and to add a deployment key to our GitHub repo for the deploy user to use when fetching code from Git.
  • Capifony allows you to share files between deployments. This deployment script shares the parameters.yml file, the log, vendors and uploads directories.
  • The line “ssh_options[:keys] = [File.join(ENV["HOME"], “.ssh”, “KEY FILE NAME”)]” tells Capifony to use an SSH keyfile with the supplied name. This allows us to register a key on the server for the deploy user.
  • Since the deploy user needs to access the GitHub repository we needed to add a deployment key for the deploy user to use on GitHub.
  • The line “before “symfony:cache:warmup”, “symfony:doctrine:migrations:migrate”” automatically runs any new doctrine migrations with every deployment. The line “set :interactive_mode, false” ensures that Symfony won’t ask for confirmation when running this command.
  • The final part of the script overrides the restart method. This is left blank in Capifony (although I think it’s a standard task in Capistrano Ruby deployments). I needed to restart Apache with every deployment to make sure that the APC cache was cleared since a lot of cached data is stored in APC. The deploy user was given sudo permission only to restart the apache process.

Getting it working with Jenkins

We elected to have the Capifony deployment set up as a separate job on our Jenkins server rather than having it run as a post build task in our main Jenkins job for the project. The deployment job simply pulls in the latest version of the code from our GitHub repository and performs ‘cap deploy’ as a shell command for its single build action. This job is triggered as a downstream job once the main build for our project successfully completes. We chose this configuration for a couple of reasons:

  • This setup gives us a little more freedom to run the deployment job on it’s own if we ever need to.
  • If a build or a deployment fails it’s a little easier for is to see instantly where things have gone wrong.

We needed to add an ssh key on the Jenkins server for Jenkins to use when connecting to the live server and to add this to authorized_keys for the deploy user on the server. This setup did give us one other problem though. By default Capifony will deploy the latest commit to a Git repository. If a new commit is made while Jenkins is running a build then Capifony will deploy that, meaning that we could end up with code that has not been through our build process being deployed. Our solution was to have our main Jenkins build tag the repository and to push this tag back to GitHub using  the Git Publisher post build action. This tag is given the same number as the build number in Jenkins for easy identification. As you can see in the deployment script above our Capifony deployment looks for the latest tag in GitHub and deploys this to the live environment. To make sure there are no problems with this process we made sure that Jenkins will only build this project sequentially, not in parallel. This should ensure that the the latest tag in Git is the most recent tested version of the code. Naturally, for the deployment to work Capifony needs to be installed on the Jenkins server.

After setting all of this up I found that it wasn’t working. I could run ‘cap deploy’ from my development environment with it working perfectly, deploying the latest tag from GitHub. When the Jenkins server ran this it failed with a cryptic error about not being able to find the specified tag. I could see the tags being created in GitHub though and spent a couple of frustrating hours trying to work this out. Eventually, I found the problem. It seems that the Jenkins Git plugin creates an internal tag every time a build is run. This is only created on the Jenkins server and not pushed to GitHub. What was happening is that when the Capifony deployment was run on the Jenkins server it connected to the live server and then tried to checkout the tag that was only created on the Jenkins server, resulting in the deployment failing. The solution was to go into the advanced Git config and to make sure that the skip internal tag option is checked. The image below shows the option to check for this.

Conclusion

Setting all of this up did take me quite a lot of time, the majority of which was simply due to my needing to learn how to configure and use setup Capifony. As a result of it we have a great continuous deployment setup that works seamlessly. I’d definitely use Capifony again and am already looking at how we can use it to deploy a couple of legacy Symfony1 apps we have at work. If you have a Symfony app I’d strongly recommend using Capifony for deployments. If you have a Jenkins server, take it one step further and setup a continuous deployment.

22 thoughts on “Continuous Deployment with Symfony2, Jenkins and Capifony

  1. I am using capifony to deploy to an AWS EC2 Ubuntu instance. I am using a .pem file to SSH into Ubuntu which works fine, but the deployment server cannot access my private GitHub repo. I’m wondering what steps you did for the following:

    “Since the deploy user needs to access the GitHub repository we needed to add a deployment key for the deploy user to use on GitHub.”

    I presume I need to add my SSH key to GitHub, but all I have is a .pem file downloaded from EC2

    I have included my deploy.rb below

    set :application, “App Name”
    set :deploy_to, “/var/www”
    set :domain, “ec2-********.eu-west-1.compute.amazonaws.com”
    set :user, “ubuntu”

    default_run_options[:pty] = true
    set :ssh_options, {:forward_agent => true}
    ssh_options[:keys] = ["/Users/myname/plockkey.pem"]
    ssh_options[:auth_methods] = “publickey”

    set :scm, :git
    set :repository, “git@github.com:Username/reponame.git”
    set :deploy_via, :remote_cache

    role :web, domain
    role :app, domain
    role :db, domain, :primary => true

    set :use_sudo, true
    set :keep_releases, 3

    set :shared_files, ["app/config/parameters.yml"]
    set :shared_children, [app_path + "/logs", web_path + "/uploads", "vendor"]
    set :use_composer, true

    # Be more verbose by uncommenting the following line
    logger.level = Logger::MAX_LEVEL

    Thanks
    Patrick

    1. Hey Patrick,

      I think I need to make that section a little clearer in the article. Here are the steps needed to make this work (AFAIR):

      1. Log into your server and change to the user that the deployment is running as. For example, if capifony is logging in as ‘deploy’ enter ‘su deploy’ on the console.
      2. Generate an ssh keypair for that user using ssh-keygen.
      3. Copy the public key for that user and go to the GitHub page for your repo. Under the settings for that repo add the public key to the deploy keys.
      4. On your server type ‘ssh git@github.com‘ while still logged in as your deployment user. This should add github to the list of allowed hosts for the deployment user.

      Those steps should be enough to get that working for you. Let me know if you have any further questions or comments about this.

      1. Thanks, I’ve got as far as generating the key pair using the instructions here – https://help.github.com/articles/generating-ssh-keys

        When I try and copy the public key however Ubuntu does not have pbcopy. I have tried xclip but I get the following error

        Error: Can’t open display: (null)

        I also tried

        nano ~/.ssh/id_rsa.pub

        and just copying the contents but GitHub says it’s invalid.

        any ideas please?

        1. Hmm, not sure. I tend to use cat to display the contents of a file quickly, so ‘cat ~/.ssh/id_rsa.pub’ should display the public key in the console. Can you copy the key from that? What terminal and OS are you using to connect to the EC2 instance? Make sure that you’re only copying the characters in the key file and don’t have any extra characters or control characters in the text that you copy. You could also try copying the key file to you host computer using scp or sftp. Needless to say don’t use an insecure protocol when copying the key file over the internet (but I’m sure you knew that already) :).

          Let me know how you get on and I’ll see if I have any other ideas.

  2. Hi,
    thanks that worked using ‘cat ~/.ssh/id_rsa.pub’. I have now copied the key to GitHub and authorised the key on the server using ssh git@github.com
    Now when I run cap deploy from my local computer I get the following. I think it fails about halfway down where it says ‘git: not found’. Do you know why this might be please?

    Thanks for all your help so far.

    cap deploy
    * 2013-02-16 08:08:11 executing `deploy’
    * 2013-02-16 08:08:11 executing `deploy:update’
    ** transaction: start
    * 2013-02-16 08:08:11 executing `deploy:update_code’
    triggering before callbacks for `deploy:update_code’
    –> Updating code base with checkout strategy
    executing locally: “git ls-remote git@github.com:****/****.git HEAD”
    command finished in 3148ms
    * executing “git clone -q git@github.com:****/****.git /var/www/releases/20130216080814 && cd /var/www/releases/20130216080814 && git checkout -q -b deploy 86c0dc643a41d5a7b006a52a91b58294a76f6c62 && (echo 86c0dc643a41d5a7b006a52a91b58294a76f6c62 > /var/www/releases/20130216080814/REVISION)”
    servers: ["ec2-***-**-**-***.eu-west-1.compute.amazonaws.com"]
    [ec2-***-**-**-***.eu-west-1.compute.amazonaws.com] executing command
    ** [ec2-***-**-**-***.eu-west-1.compute.amazonaws.com :: out] sh: 1:
    ** [ec2-***-**-**-***.eu-west-1.compute.amazonaws.com :: out] git: not found
    ** [ec2-***-**-**-***.eu-west-1.compute.amazonaws.com :: out]
    command finished in 451ms
    *** [deploy:update_code] rolling back
    * executing “rm -rf /var/www/releases/20130216080814; true”
    servers: ["ec2-***-**-**-***.eu-west-1.compute.amazonaws.com"]
    [ec2-***-**-**-***.eu-west-1.compute.amazonaws.com] executing command
    command finished in 119ms
    failed: “sh -c ‘git clone -q git@github.com:****/****.git /var/www/releases/20130216080814 && cd /var/www/releases/20130216080814 && git checkout -q -b deploy 86c0dc643a41d5a7b006a52a91b58294a76f6c62 && (echo 86c0dc643a41d5a7b006a52a91b58294a76f6c62 > /var/www/releases/20130216080814/REVISION)’” on ec2-***-**-**-***.eu-west-1.compute.amazonaws.com

  3. Hi Jeremy, thanks, I have now installed Git on the server. I have also updated my deploy.rb as per instructions here – http://capifony.org/cookbook/upload-parameters-file.html with the following

    task :upload_parameters do
    origin_file = “app/config/parameters.yml”
    destination_file = shared_path + “/app/config/parameters.yml” # Notice the shared_path

    try_sudo “mkdir -p #{File.dirname(destination_file)}”
    top.upload(origin_file, destination_file)
    end

    after “deploy:setup”, “upload_parameters”

    If I run cap deploy it runs through the entire script and then gives me the following error

    [Symfony\Component\DependencyInjection\Exception\ParameterNotFoundException]
    You have requested a non-existent parameter “secret”.

    If I check on the server I can see it has created /var/www/shared/app/config/parameters.yml but it is empty

    The capifony docs say I need to create this file. But do I actually need to paste in the contents of my local parameters.yml into this file?

    Thanks
    Patrick

      1. Thanks, I have it deploying now with no errors, however, everything gets deployed into /var/www/current . Shouldn’t it not go into www ?

        1. No, that’s how Capifony is designed. Current is a symlink that always points to the latest release under the releases directory. You need to update your vhost config to make /var/www/current/web the document root and you should then be good to go. You may also need to make sure that Apache is configured to follow symlinks.

  4. Hi, I have changed my vhost to point to /var/www/current/web and I can now view my site. The only thing I had to do manually was set permissions on /var/www/releases/20130225164109/app/cache and /var/www/releases/20130225164109/app/logs manually. However, this seems to be a recurring problem. do you know how to set the permissions on this and the logs folder automatically Please?

    I have added cache to my shared children but this doesn’t seem to help.

    set :shared_children, [app_path + "/cache", app_path + "/logs", web_path + "/uploads", "vendor"]

    Thanks

      1. Thanks, I have tried both

        ps aux | grep httpd
        ps aux | grep apache

        But I still get the same permissions issues.

        If I do ls -l on the shared/app/cache I get ‘root’, but I am logging in as ‘ubuntu’ over SSH. If I try and login as root I get an error Please login as the user “ubuntu” rather than the user “root”.

        So it seems that the folders are created with permissions for root which I don’t have permissions for as ubuntu.

        Also parameters.yml doesn’t get uploaded on setup

        1. Take another look at the link I sent you in the last comment – there’s a section about permissions and using acl to set them. Also parameters.yml is not uploaded and had to be created on the server. It makes senses as the parameters file is probably different on every environment. I’d create it and set it as one of the shared files.

          1. Hi, I think I’m getting somewhere now, I needed to change my permissions on /var/www

            sudo chown -R ubuntu /var/www

            and also

            sudo chown -R ubuntu /home/ubuntu/.composer/cache

            This means I can now have

            set :use_sudo, false

            in my deploy.rb

            Everything deploys

            I have also followed the instructions on http://symfony.com/doc/current/book/installation.html
            regarding permissions

            I ran

            ps aux | grep httpd

            and my user was ubuntu

            my system doesn’t support chmod +a

            I installed acl using

            sudo apt-get install acl

            I then cd to /var/www/recent/ and ran

            sudo setfacl -R -m u:ubuntu:rwX -m u:`whoami`:rwX app/cache app/logs
            sudo setfacl -dR -m u:ubuntu:rwx -m u:`whoami`:rwx app/cache app/logs

            I now get the following exception when viewing my site in a browser

            : Unable to create the cache directory (/var/www/releases/20130226184018/app/cache/prod)
            My cache folder is a shared folder so I also ran the same in shared but I get the same error

            If I do

            ls -l releases/

            I get

            drwxrwxr-x 7 ubuntu ubuntu 4096 Feb 26 18:40

            in my deploy.rb I have

            set :shared_children, [app_path + "/cache", app_path + "/logs", web_path + "/uploads", "vendor"]
            set :writable_dirs, ["app/cache", "app/logs"]
            set :permission_method, :acl

            Do you know anything else I could try please? It feels like I’m at the final hurdle.

  5. Hi, I realised I had to use ‘www-data’ not ‘ubuntu’ like so:

    sudo setfacl -R -m u:www-data:rwX -m u:`whoami`:rwX app/cache app/logs
    sudo setfacl -dR -m u:www-data:rwx -m u:`whoami`:rwx app/cache app/logs

    And I had to do this within /var/www/shared

    The only thing remaining now is installing and dumping assets, I have run

    cap symfony:assets:install
    cap symfony:assetic:dump

    I don’t get an error, but the assets are not created within current or releases. I’m not sure where they are going if they are being created at all. They are missing when I try and preview in a browser. Do you know how to install and dump assets in the correct location

    Thanks

    1. Hi Patrick. I would use composer post install and post update commands to do this. I can’t remember the exact syntax off the top of my head but you’ll need to add a couple of lines to your composer.json file. You should be able to find more information on the Symfony or composer websites. This way whenever capifony runs ‘composer install’ composer will deploy your assets after installing dependencies.

  6. BINGO!

    I needed the following for assets

    set :dump_assetic_assets, true

    The final solution involved removing the cache folder from my shared_children

    set :shared_children, [app_path + "/logs", web_path + "/uploads", "vendor"]

    and changing my permissions in /var/www/current

    sudo setfacl -R -m u:www-data:rwX -m u:`whoami`:rwX app/cache app/logs
    sudo setfacl -dR -m u:www-data:rwx -m u:`whoami`:rwx app/cache app/logs

    Thanks so much for all your help! I think I’ll leave the Jenkins part of the puzzle for a later date…

  7. Hello, I’m deploying another app with capifony to AWS EC2 Ubuntu, and have come accross a new issue that I can’t resolve, I wonder if you could help please? Everything works fine up until the following:

    –> Updating Composer dependencies
    * executing “sh -c ‘cd /var/www/releases/20130515073822 && php composer.phar update –no-scripts –verbose –prefer-dist’”
    servers: ["ec2-[ip].eu-west-1.compute.amazonaws.com”]
    [ec2-[ip].eu-west-1.compute.amazonaws.com] executing command
    ** [out :: ec2-[ip].eu-west-1.compute.amazonaws.com] Loading composer repositories with package information
    ** [out :: ec2-[ip].eu-west-1.compute.amazonaws.com] Updating dependencies (including require-dev)
    ** [out :: ec2-[ip].eu-west-1.compute.amazonaws.com] Killed
    command finished in 128784ms
    *** [deploy:update_code] rolling back
    * executing “rm -rf /var/www/releases/20130515073822; true”
    servers: ["ec2-[ip].eu-west-1.compute.amazonaws.com”]
    [ec2-[ip].eu-west-1.compute.amazonaws.com] executing command
    command finished in 1031ms
    failed: “sh -c ‘sh -c ‘\\”cd /var/www/releases/20130515073822 && php composer.phar update –no-scripts –verbose –prefer-dist’\\”’” on ec2-[ip].eu-west-1.compute.amazonaws.com

    My deploy.rb is below

    # Sylius default deployment configuration.

    # Capifony documentation: http://capifony.org
    # Capistrano documentation: https://github.com/capistrano/capistrano/wiki

    # Be more verbose by uncommenting the following line
    # logger.level = Logger::MAX_LEVEL

    set :application, “My App Name”
    set :domain, “ec2-[ip].eu-west-1.compute.amazonaws.com”
    set :deploy_to, “/var/www”
    set :user, “ubuntu”

    role :web, domain
    role :app, domain
    role :db, domain, :primary => true

    set :scm, :git
    set :repository, “git://github.com/My-Git-repo.git”
    set :branch, “master”
    set :deploy_via, :remote_cache

    default_run_options[:pty] = true
    set :ssh_options, {:forward_agent => true}
    ssh_options[:keys] = ["key.pem"]
    ssh_options[:auth_methods] = “publickey”

    set :use_composer, true
    set :update_vendors, true

    set :dump_assetic_assets, true

    set :writable_dirs, ["app/cache", "app/logs"]
    set :webserver_user, “www-data”
    set :permission_method, :acl

    set :shared_files, ["app/config/parameters.yml", "web/.htaccess", "web/robots.txt"]
    set :shared_children, ["app/logs"]

    set :model_manager, “doctrine”

    set :use_sudo, false

    set :keep_releases, 3

    # Be more verbose by uncommenting the following line
    logger.level = Logger::MAX_LEVEL

    before ‘symfony:composer:update’, ‘symfony:copy_vendors’

    namespace :symfony do
    desc “Copy vendors from previous release”
    task :copy_vendors, :except => { :no_release => true } do
    if Capistrano::CLI.ui.agree(“Do you want to copy last release vendor dir then do composer install ?: (y/N)”)
    capifony_pretty_print “–> Copying vendors from previous release”

    run “cp -a #{previous_release}/vendor #{latest_release}/”
    capifony_puts_ok
    end
    end
    end

    after “deploy:update”, “deploy:cleanup”
    after “deploy”, “deploy:set_permissions”

    Thanks
    Patrick

Leave a Reply