JavaScript Test Automation using TeamCity

This article explains all steps you need to perform to setup your Continuous Integration System with TeamCity + PhantomJS + CoffeeScript (optionally) + Jasmine and/or CasperJS :)

There's an increasing demand for interactive web applications which dynamically load data using AJAX and handle user interaction with lots of fancy JavaScript stuff. That's why we have significant more Javascript code in our web apps than it was 10 years ago (you still remember how "hip" it was to use JavaScript back in 2000 as a web developer?).

Javascript code is essential for modern web applications. It needs to be tested. It wants to be tested. We needed a setup which allows to run our CasperJS and Jasmine test suites both locally and on our build server (TeamCity). This article covers the setup of TeamCity for local testing and especially automated headless testing of Javascript code in detail.

CoffeeScript

Most of our Javascript tests are written in CoffeeScript. CoffeeScript provides syntactic sugar to write cleaner, more readable code which transcompiles to JavaScript. However using CoffeeScript makes the setup of the buildserver slightly more sophisticated. Not too hard though. Give it a chance! ;)

Jasmine

A behaviour-driven JS testframework we use for writing JavaScript unit tests. It easily integrates with Ruby projects. Syntactically it looks quite similar to RSpec:

describe("A suite", function() {
	it("should work flawlessly", function() {
		expect(true).toBe(true);
	});
});
The same with CoffeeScript :)
describe 'A suite', ->
	it 'should work flawlessly', ->
		expect(true).toBe true

Links:

Local execution of jasmine tests

After installing the gem jasminerice in your rails app, it's really simple to execute jasmine tests locally: Simply start up your rails server and fire up http://localhost:3000/jasmine in the browser.

CasperJS

CasperJS is a navigation scripting and testing utility used by us to navigate through our web apps, create screenshots, fill and submit forms and much more. To get started simply add the gem casperjs to your rails project.

Local execution of CasperJS tests
Add the following rake tasks to your Rakefile:
  test_server_pid_file = "tmp/pids/test_server.pid"
  app_port             = 8787
  desc "Start Testing Server"
  task :start_test_server do
    counter = 0

    Thread.new {
      command_line = "bundle exec rails server -p #{app_port} -e test -P #{test_server_pid_file}"
      puts "Starting: #{command_line}"
      system command_line
    }

    while (not File.exist?(test_server_pid_file)) && counter < 90 do
      counter += 1
      sleep 2
    end
    if counter >= 30
      STDERR.puts "Start took too long!"
    else
      puts "Test server running ..."
    end
  end

  desc "Stop Testing Server"
  task :stop_test_server do

    puts "Stopping test server ..."
    pid = IO.read(test_server_pid_file).to_i rescue nil

    if pid.present?
      system("kill -9 #{pid}")
      FileUtils.rm(test_server_pid_file)
      puts "... Test server stopped"
    else
      STDERR.puts "Cannot stop server - no pid file found!"
    end
  end


  desc "run Casper JS Tests, starts rails server,run the tests and then stop the server "
  task :run_casper_js_tests => [:init, "db:fixtures:load", "projectname:start_test_server" ] do
    begin
      Rake::Task["projectname:start_test_server"].invoke

      spec_path = Rails.root.join("spec/javascripts/casperjs/")
      log_path  = Rails.root.join("log")
      puts "Running CasperJS Specs in #{spec_path}"
      # 1. first runs pre.js
      # 2. then runs all JS and CoffeeScript Specs in directory spec_path
      # 3. then runs post.js
      # 4. finally stores result to log/casper_spec_output.xml
      system("casperjs test \
              --testhost=http://localhost:#{app_port} \
              --save_to_xml=#{File.join(log_path, 'casper_spec_output.xml')} \
              --pre=#{File.join(spec_path, 'pre.js.coffee')} \
              --post=#{File.join(spec_path, 'post.js.coffee')} \
              --log-level=info \
              #{File.join(spec_path)}")
    ensure
      Rake::Task["projectname:stop_test_server"].invoke
    end
  end

You can then run your CasperJS tests (let it be JS- or coffeescript-tests) by running the new Raketask.

TeamCity Setup

TeamCity is a popular Continuous Integration Server developed by JetBrains. We use it to automatically build and test our Java, Rails and iOS projects. This chapter describes how to setup TeamCity for automatic JavaScript testing of Rails projects.

Setup buildagents for Jasmine Testing

We use PhantomJS - the scriptable headless WebKit engine - to execute our JS tests on the build server. Therefore initially we need to install a recent version of phantomjs on all buildagents which shall run js code. Here's an example for PhantomJS v.1.8.1:

ssh buildagent_1
cd /usr/local/share
wget http://phantomjs.googlecode.com/files/phantomjs-1.8.1-linux-i686.tar.bz2
tar xjf phantomjs-1.8.1-linux-i686.tar.bz2
sudo ln -s /usr/local/share/phantomjs-1.8.1-linux-i686/bin/phantomjs /usr/bin/phantomjs 
rm phantomjs-1.8.1-linux-i686.tar.bz2
Did it work? To check type: phantomjs -v That's it for the buildagents! :)

Prepare project for TeamCity integration

For the content of the following files we got heavy inspiration from Dan's great article http://blog.danmerino.com/continuos-integration-ci-for-javascript-jasmine-and-teamcity/.
Add the following files to your project:

  • /script/teamcity/test_runner.js: PhantomJS script which triggers the headless execution of the JS tests
    console.log('PhantomJS started TestRunner which loads a web page with a TeamCity Reporter...');
    var page = new WebPage();
    var system = require('system');
    var checkout_dir = system.args[1]; // pass checkout_dir parameter
    
    //Open local teamcity_reporter.html
    var url = "file://localhost" + checkout_dir + "/script/teamcity/teamcity_reporter.html";
    phantom.viewportSize = {width: 800, height: 600};
    //Required because PhantomJS sandboxes the website and doesn't show up the console messages form that page by default
    page.onConsoleMessage = function (msg) {
        console.log(msg);   // Pass all page logs to stdout
    
        if (msg && msg.indexOf("##jasmine.reportRunnerResults") !== -1) {
            phantom.exit();
        }
    };
    //Open the website with the teamcity reporter
    page.open(url, function (status) {
        //Page is loaded!
        if (status !== 'success') {
            console.log('Unable to load the address!');
        } else {
            //Using a delay to make sure the JavaScript is executed in the browser
            window.setTimeout(function () {
                page.render("output.png");
                phantom.exit();
            }, 200);
        }
    });
    
  • /script/teamcity/teamcity_reporter.html: HTML page which is being rendered in the PhantomJS browser. Tells Jasmine which tests it shall execute and registers TeamCity reporter with Jasmine
    
    
    
        
        Console Reporter Spec
    
        
        
        
        
        
    
        
        
    
        
        
    
    
    
    
    
        
    
    
    
    
  • /script/teamcity/jasmine.teamcity_reporter.js: hands over spec results to TeamCity build server
    (function() {
        if (! jasmine) {
            throw new Exception("jasmine library does not exist in global namespace!");
        }
    
        /**
         * Basic reporter that outputs spec results to for the Teamcity build system
         *
         * Usage:
         *
         * jasmine.getEnv().addReporter(new jasmine.TeamcityReporter());
         * jasmine.getEnv().execute();
         */
        var TeamcityReporter = function() {
            this.started = false;
            this.finished = false;
        };
    
        TeamcityReporter.prototype = {
            reportRunnerResults: function(runner) { },
    
            reportRunnerStarting: function(runner) { },
    
            reportSpecResults: function(spec) { },
    
            reportSpecStarting: function(spec) { },
    
            reportSuiteResults: function(suite) {
                var results = suite.results();
                var path = [];
                while(suite) {
                    path.unshift(suite.description);
                    suite = suite.parentSuite;
                }
                var description = path.join(' ');
    
                this.log("##teamcity[testSuiteStarted name='" + this.escapeTeamcityString(description) + "']");
    
                var outerThis = this;
                var eachSpecFn = function(spec){
                    if (spec.description) {
                        outerThis.log("##teamcity[testStarted name='" + outerThis.escapeTeamcityString(spec.description) + "' captureStandardOutput='true']");
                        var specResultFn = function(result){
                            if (!result.passed_) {
                                outerThis.log("##teamcity[testFailed name='" + outerThis.escapeTeamcityString(spec.description) + "' message='|[FAILED|]' details='" + outerThis.escapeTeamcityString(result.trace.stack) + "']");
                            }
                        };
    
                        for (var j = 0, jlen = spec.items_.length; j < jlen; j++) {
                            specResultFn(spec.items_[j]);
                        }
                        outerThis.log("##teamcity[testFinished name='" + outerThis.escapeTeamcityString(spec.description) + "']");
                    }
                };
                for (var i = 0, ilen = results.items_.length; i < ilen; i++) {
                    eachSpecFn(results.items_[i]);
                }
    
                this.log("##teamcity[testSuiteFinished name='" + outerThis.escapeTeamcityString(description) + "']");
            },
    
            log: function(str) {
                var console = jasmine.getGlobal().console;
                if (console && console.log) {
                    console.log(str);
                }
            },
    
            hasGroupedConsole: function() {
                var console = jasmine.getGlobal().console;
                return console && console.info && console.warn && console.group && console.groupEnd && console.groupCollapsed;
            },
    
            escapeTeamcityString: function(message) {
                if(!message) {
                    return "";
                }
    
                return message.replace(/\|/g, "||")
                    .replace(/\'/g, "|'")
                    .replace(/\n/g, "|n")
                    .replace(/\r/g, "|r")
                    .replace(/\u0085/g, "|x")
                    .replace(/\u2028/g, "|l")
                    .replace(/\u2029/g, "|p")
                    .replace(/\[/g, "|[")
                    .replace(/]/g, "|]");
            }
        };
    
        function suiteResults(suite) {
            console.group(suite.description);
            var specs = suite.specs();
            for (var i in specs) {
                if (specs.hasOwnProperty(i)) {
                    specResults(specs[i]);
                }
            }
            var suites = suite.suites();
            for (var j in suites) {
                if (suites.hasOwnProperty(j)) {
                    suiteResults(suites[j]);
                }
            }
            console.groupEnd();
        }
    
        function specResults(spec) {
            var results = spec.results();
            if (results.passed() && console.groupCollapsed) {
                console.groupCollapsed(spec.description);
            } else {
                console.group(spec.description);
            }
            var items = results.getItems();
            for (var k in items) {
                if (items.hasOwnProperty(k)) {
                    itemResults(items[k]);
                }
            }
            console.groupEnd();
        }
    
        function itemResults(item) {
            if (item.passed && !item.passed()) {
                console.warn({actual:item.actual,expected: item.expected});
                item.trace.message = item.matcherName;
                console.error(item.trace);
            } else {
                console.info('Passed');
            }
        }
    
        // export public
        jasmine.TeamcityReporter = TeamcityReporter;
    })();
    
  • /spec/javascripts/jasmine_spec_suite.js: Lists the jasmine specs which shall be tested by TeamCity
    // This is a manifest file that'll be compiled into application.js, which will include all the
    // files listed below.
    // Any JavaScript/Coffee file within this directory, lib/assets/javascripts,
    // vendor/assets/javascripts, or vendor/assets/javascripts of plugins,
    // if any, can be referenced here using a relative path.
    // It's not advisable to add code directly here, but if you do, it'll appear at the bottom of the
    // the compiled file.
    //
    // WARNING: THE FIRST BLANK LINE MARKS THE END OF WHAT'S TO BE
    // PROCESSED, ANY BLANK LINE SHOULD GO AFTER THE REQUIRES BELOW.
    //
    // add FACTORIES
    //= require ./factories/dog_factory
    // add SPECS
    //= require fat_dog_spec
    //= require_self
    
  • /Rakefile: task which compiles coffeescript files to one big js file
      desc "Asset Precompile for Coffeescript Jasmine Tests"
      task :jasmine_asset_precompile => ["assets:precompile"] do
        # compile JS specs after creating assets
        compile_jasmine_coffeescripts
      end
    
      # compile all JS specs to one big js file in /spec/javascript/assets dir
      def compile_jasmine_coffeescripts
        require 'sprockets'
        output_path     = Rails.root.join("spec/javascripts/assets/")
        env             = Sprockets::Environment.new(Rails.root)
        test_suite_name = "jasmine_spec_suite.js"
    
        test_suite_path = Rails.root.join("spec/javascripts/")
        env.append_path(File.join(test_suite_path))
    
        puts "Compiling JS Test Suite '#{test_suite_name}'."
        compiler = Sprockets::StaticCompiler.new(env,
                                                 File.join(output_path),
                                                 [test_suite_name],
                                                 :digest => nil,
                                                 :manifest => false)
        compiler.compile
      end
    




Setup Teamcity for Jasmine Testing

  • Create a new build step to compile all coffeescript src files to one big application.js at the build server.

    jasmine_build_step.png


    New build step: Runner type: Rake
    Rake task: projectname:jasmine_asset_precompile
    Additional rake command line parameters: RAILS_ENV=test RAILS_GROUPS=assets
  • New build step: JS Jasmine tests
    Command executable: phantomjs
    Command parameters: script/teamcity/test_runner.js teamcity.build.workingDir

By passing the teamcity.build.workingDir to the test_runner.js in the build step configuration (s.a.) we can use the jasmine gem installed in our Rails app from within our TeamCity installation without the need to install a separate jasmine instance on our TeamCity server.

Now for a real example. Let's quickly add a dummy test /spec/javascripts/jasmine_example_spec.js.coffee to our project to break our build:

describe 'A jasmine example suite', ->
  it 'should obviously fail', ->
    expect(true).toBe false # should fail

Do not forget to add the new spec file to your jasmine meta suite /spec/javascripts/assets/jasmine_spec_suite.js and commit the changes.

TeamCity detects the changes and the next available build agent starts testing. Few moments later TeamCity turns red. That's what we expected!

Jasmine-TC-Fail.png

We don't like red builds. Let's fix it!

describe 'A jasmine example suite', ->
  it 'should work flawlessly', ->
    expect(true).toBe true

Commit the file, wait some moments and watch how TeamCity magically changes from red to green. Awesome! Due to the good integration of the JS test reporter you can use the statistics, test history and similiar features of TeamCity as you are used to for your JavaScript tests.

jasmine-TC-green.png

We now have a working continuous integration system for jasmine tests in Rails applications on a TeamCity Server. We get detailed reports for test results - not only for classical RSpec tests but also for Jasmine specs written in plain Javascript or Coffeescript.

One problem still waits to be solved: If jasmine specs fail they're correctly reported as "Failed Tests". However TeamCity doesn't show a stacktrace. If you have an idea how to improve the output please leave a post.

Setup buildagents for CasperJS Testing

Additionally to their normal setup buildagents only need PhantomJS (see above)

Setup Teamcity for CasperJS Testing

  • Create a new build step in TeamCity to execute the Casper JS tests:
    New build step: Runner type: Rake
    Step name: Run CasperJS Tests
    Rake task: projectname:run_casper_js_tests
    Bundler: [x] bundle exec

When TeamCity executes this Rake task it creates an XML result file in the project log dir named 'casper_spec_output.xml'. All we need to do to finish our configuration is to add a build step to our TeamCity project configuration to evaluate this file:

Build Configuration --> Build Steps --> Additional Build Features --> XML report processing --> Add build feature
Report type: Ant JUnit
Monitoring rules: log/casper_spec_output.xml
Verbose output: [x]
Further information: http://confluence.jetbrains.com/display/TCD7/XML+Report+Processing

That's it. Let's add an example (failing) CasperJS test (written in CoffeeScript) to our project and commit it: We create a file /spec/javascripts/casperjs/dummy_spec.js.coffee which opens Google in a Browser (PhantomJS).

casper.start "http://www.google.de/", () ->
  @test.assertTitle  "Bing",                   "Google is called Bing!?" # This should fail
  @test.assertExists "form[action='/search']", "Found Google main form"

casper.run () ->
  @test.done 2

Commit it and watch your continuous integration system work for you. Our buildagents execute the CasperJS tests on commit - TeamCity reports the results and turns red!

Google-CasperJS-red.png

Next let's fix the test. Change the code and commit it:

casper.start "http://www.google.de/", () ->
  @test.assertTitle  "Google",                 "Google is called Google"
  @test.assertExists "form[action='/search']", "Found Google main form"

casper.run () ->
  @test.done 2

As always TeamCity detects the change and the next available build agent starts testing. TeamCity turns from red to green! Well done!

Google-CasperJS-Green.png




Conclusion

Manually executing JavaScript/CoffeeScript test suites on your local system is not enough. With CI systems like TeamCity you can fully automate the execution of tests on the build server. Initially some manual configuration is necessary but you will benefit from a reliable Continuous Integration System coupled with a whole bunch of sophisticated JavaScript test frameworks and tools using PhantomJS.

Remember: You are not limited to CasperJS and Jasmine. There are so many more!