Custom tests reports with TestNG Listeners, Apache Velocity and a bit of CSS

In your opinion, the original reports of TestNG are (pick one or more options):

  1. showing too much information,
  2. showing not enough information,
  3. not showing what you want to show,
  4. require too many clicks to find out what has gone wrong,
  5. not very pretty,

Because of this, you finally decide to create your own reports. Good, then read this post, and you will learn about one possible way of achieving this.

You will find much more about testing in my book
"Practical Unit Testing
with TestNG and Mockito"

Real life scenario

The problem I face is the following. I have some integration tests written with TestNG, and I want to share the results of these tests with our partners from the project. I want a test report, that will make everyone get instantly a clear picture of what is the status of the integration. The tests are grouped in few suites/projects and run independently in OSGi environment.

The resulting report provides information on:

  1. deployed bundles,
  2. tests execution results (brief - only information on failed/skipped tests),
  3. additional info - time, environment etc.,
  4. links to all possible logs,
    • original TestNG reports, so you can check the details of failed tests,
    • server logs,
    • some custom reports - e.g. report on times of tests execution (per test method)

In this post I will present how to achieve point 2 - a custom report on test execution of single test suite. Having this information, you will be able to create much more complex reports (like the one described above).

Elements

We need three things to make it work. First, we need to obtain information about the tests. This is easy if you use TestNG's listeners. The second point is to create reports without cluttering the code with presentation details. This is crucial, especially if you aim at more then one report format (I do - I want reports in plain txt for quick developers check, and visually stunning html reports). The solution here is to use Velocity Templates, that will decouple your code from presentation issues. The last element of the puzzle is to make the reports look nice - in my case, I will use CSS to achieve this result.

Ah, the fourth element of course ! Build script, yes, we need that too. We will use Gradle for this.

So, we know what will be used, now let's put all the elements of the puzzle together.

The reports

Let us decide first, what element should the report contain. I would like them to show me:

  1. number of tests that passed, failed and were skipped,
  2. if there were any failed tests:
    1. there should be a well-visible warning,
    2. names of failed tests methods (along with names of test classes) should be printed
  3. if there were any skipped tests:
    1. names of skipped tests methods (along with names of test classes) should be printed
  4. if all tests passed, there should be a well-visible green tick

Of course it is easy to add other information, for example - date of tests execution, environment (e.g. Java/OS version).

Please notice the source code attached at the end of this text.

Velocity templates + CSS

So, let us write the velocity templates first - for txt and html version. Both versions will be very similar. Of course, the txt report will not contain some graphical elements.

Both templates use:

  • some basic information regarding tests (variables: totalNb, failedNb etc.)
  • maps of type Map<String, List<String>>; each entry in this map contains the name of the class and list of methods (that failed in case of <code>failedTests or were skipped in case of skippedTests). BTW. As you can see the map-iteration is possible, but rather cumbersome with Velocity.

test_summary.txt.vm

Even if you are new to Velocity, you will easily understand this "code". It is enough to know, that words prefixed with $ are variables (that are retrieved from the VelocityContext) and expression prefixed with # are control statements.

I present a part of the template file. as the rest is very similar.

all tests: $totalNb
failed: $failedNb
passed: $passedNb
skipped: $skippedNb

#if( $failedNb > 0 )
FAILED TESTS:
#foreach( $failedClass in $failedTests.keySet() )
$failedClass
#foreach ( $failedMethod in $failedTests.get($failedClass))
$failedMethod
#end
#end
#end

[...] same for skipped tests

test_summary.html.vm

HTML version is very similar, the only difference is, that every element is inside a div or span, and CSS classes are added, so it will be easy to change the output. Only a fragment presented below.

#if( $failedNb > 0)
<div class="testSummaryFailed">
#else
#if( $failedNb == 0 && $skippedNb > 0)
<div class="testSummarySkipped">
#else
<div class="testSummaryPassed">
    #end
#end
<span class="summary">all tests: $totalNb</span>

#if( $failedNb > 0 )
<span class="failedBad">
#else
<span class="failed">
#end
failed: $failedNb</span>

// [...] same for passed and skipped goes here - not presented

#if( $failedNb > 0 )
<div class="failedTests"><span class="failedTestsTitle">FAILED TESTS</span>
#foreach( $failedClass in $failedTests.keySet() )
<span class="failedTestClass">$failedClass</span>
#foreach ( $failedMethod in $failedTests.get($failedClass))
<span class="failedTestMethod">$failedMethod</span>
#end
#end
</div>
#end

// [...] same for skipped goes here - not presented

</div>

CSS for html report

The html report needs some CSS styles so it doesn't look this ugly. Nothing fancy here. Below, a small snippet, so you know what it is like (see the attached source code).

I'm not a webmaster, so I'm pretty sure the CSS can be done better. Ah, I don't care - it works for me. :)

/***************************************
* green tick, red x or question mark ? *
***************************************/
div.testSummaryFailed, div.testSummaryPassed, div.testSummarySkipped {
background-position: right top;
background-repeat: no-repeat;
}
div.testSummaryFailed {
background-image: url("img/tests-failed.png");
}
div.testSummaryPassed {
background-image: url("img/tests-passed.png");
}
div.testSummarySkipped {
background-image: url("img/tests-skipped.png");
}
/******************
* all other stuff *
******************/
.testSummaryFailed span {
display: block;
}
.failedBad, .passedOk, .skippedBad {
font-weight: bold;
}
.failedBad, .skippedBad {
color: red;
}
.passedOk {
color: green;
}
div.failedTests {
font-size: 90%;
margin: 2ex;
padding: 1ex;
border: 1px dotted #C00;
}
div.skippedTests span, div.failedTests span {
display: block;
}

etc. etc.

TestNG Listeners + Velocity templates

Now, there are only two things left. First, write a test listener, that will pass the results to Velocity, so it can produce the reports. Second, make TestNG use this listener.

CustomReportsListener

public class TestListener implements ITestListener {

  private Map<String, List<String>> failedTests = ...
  private Map<String, List<String>> skippedTests = ...

  public void onFinish(ITestContext testContext) {
    failedTests = generateResultMap(testContext.getFailedTests());
    skippedTests = generateResultMap(testContext.getSkippedTests());
    try {
      printTestsInfo(testContext);
    } catch (IOException e) { ... }
  }

  private Map<String, List<String>> generateResultMap(IResultMap results) {
    ... gets information on failed and skipped tests from
    ... IResultMap object, and stores them in failedTests and
    ... skippedTests map respectively
  }

  private void printTestsInfo(ITestContext testContext) throws IOException {
    int passedNb = testContext.getPassedTests().size();
    int failedNb = testContext.getFailedTests().size();
    int skippedNb = testContext.getSkippedTests().size();

    VelocityContext context = new VelocityContext();
    context.put("totalNb", passedNb + skippedNb + failedNb);
    context.put("passedNb", passedNb);
    context.put("failedNb", failedNb);
    context.put("skippedNb", skippedNb);
    context.put("failedTests", failedTests);
    context.put("skippedTests", skippedTests);

    VelocityEngine ve = new VelocityEngine();
    ve.setProperty("file.resource.loader.class",
      ClasspathResourceLoader.class.getName());
    try {
      ve.init();
      writeFile(ve, context, "templates/summary_txt.vm",
        "test_summary.txt");
      writeFile(ve, context, "templates/summary_html.vm",
        "test_summary.html");
    } catch (Exception e) { ... }
  }

  private void writeFile(VelocityEngine ve, VelocityContext context,
      String vmPath, String fileName)
      throws ResourceNotFoundException,
      ParseErrorException, Exception {
    Template t = ve.getTemplate(vmPath);
    StringWriter writer = new StringWriter();
    t.merge(context, writer);
    FileWriter fileWriter = new FileWriter(fileName, true);
    fileWriter.append(writer.toString());
    fileWriter.flush();
    fileWriter.close();
  }

...

}

Binding it all together - build script

There are many ways to achieve this depending. I use Gradle, so the build script is very concise and elegant (see below).

In the snippet below you can see two interesting thinsg: declaration of listeners in test task, and report task, which simply copies styles and images.

usePlugin 'java'

repositories {
  mavenCentral()
}

dependencies {
  compile 'org.testng:testng:5.10:jdk15'
  compile 'org.apache.velocity:velocity:1.6.2'
}

test {
  useTestNG()
  options {
    listeners << 'pl.kaczanowscy.tomek.testng.TestListener'
  }
}

task report (type: Copy) {
  from "src/main/resources"
  into testResultsDir
  exclude "templates/*.vm"
}

...and why is TestNG dependency in compile group instead of testCompile ? Because TestListener needs it, and TestListener resides in src/main/java. But yes, this is weird - usually TestNG dependency would be in testCompile group.

The final result

To see the final result execute from the command line:

gradle test
gradle report

and then

less build/test-results/test_summary.txt

or

firefox build/test-results/test_summary.html

I wanted both versions to be very readable. That I only need to look at it, and I know instantly what has gone wrong.

text report

all tests: 9
failed: 3
passed: 4
skipped: 2

FAILED TESTS:
pl.kaczanowscy.tomek.testng.TestA
        failedA_2
        failedA_1
pl.kaczanowscy.tomek.testng.TestB
        failedB_1

SKIPPED TESTS:
pl.kaczanowscy.tomek.testng.TestA
        skippedA
pl.kaczanowscy.tomek.testng.TestB
        skippedB

html report

Final remarks

The simple tools and techniques presented in this blog post can be used to create any test report. The only limit is your imagination. ;)

AttachmentSize
2009_11_testng_listeners.tar_.gz33.03 KB

Send the attachments in .zip file

Hi,

It would be great if you can please attach the files in .zip format.

Thanks!

REgarding ReportNG

HI,
Is there a way to customize report obtained using ReportNG

Hi Tomek, I'm always

Hi Tomek,

I'm always interested in finding out better ways to present the TestNG HTML reports, so please let me know if you ever decide to write more extensive HTML reports..

Thanks!

--
Cédric

Usually using txt versions

Hello Cedric,

nice to see you here. :)

As for the HTML reports I don't use them very often. In fact, I usually check testng-results.xml and don't care about rest.
But if I ever go back to HTML reports and have some new ideas, I will surely let you know.

--
Tomek

Adnan Turic

Hi, do you have some experience with hudson - testng report customization?

Regards

Hi Adnan, do you have

Hi Adnan,

do you have something specific in mind when you say hudson - testng report customization? We have done some modifications (not report customization though) to the Hudson TestNG plugin to get it working according to our organization needs.

Probably, we can try out something if you give your requirements.

I can be reached at shri.naresh@gmail.com

- Naresh

no Hudson experience, sorry

no Hudson experience, sorry

Please comment using