The goal of this presentation

The original goal of this presentation was to persuade my teammates to switch to TestNG. :) But then I decided to publish it and to persuade you, dear reader, that TestNG is worth migrating to (especially if you still use JUnit).

Yes, I’m biased

Ok, I’m biased, I admit. I like TestNG, I support it, I even wrote a book about it. Yes. Still, I have some arguments in favour of TestNG, so maybe these few slides are worth reading.

Not bashing!

My goal is to inform you about the advantages of TestNG, not to bash JUnit! I feel indebted to this great project! However, when choosing technology for my next project, I want to select the best one available. So, sorry JUnit…

Why TestNG?

My (biased) observations

People are moving from JUnit to TestNG once they start to write integration and end-to-end tests (e.g. with Selenium). Apparently, on unit tests level you can put up with JUnit deficiencies, but once you go outside the unit tests realm, JUnit stands in your way.

While better support for integration and end-to-end tests is definitely true, not many people know that TestNG offers more than JUnit also in case of unit tests.

Let the code speak!

Ok, now I will show you some code snippets so you can judge for yourself if TestNG is worth your time.

I do not describe them in detail, please consult TestNG documentation for more information.

Concurrency

No need to create own Thread objects! Use annotations and that is it!

An example of running same test method simultaneously (20 times, by 3 threads):

@Test(threadPoolSize = 3, invocationCount = 20)
public void concurrencyTest() {
    System.out.print(" " + Thread.currentThread().getId());
}

Result:

13 12 13 11 12 13 11 12 13 12 11 13 12 11 11 13 12 11 12 13

Parametrized tests

In general, it is a good practice, to test your code with different sets of parameters:

expected values: sqrt(4), sqrt(9),
boundary values: sqrt(0),
strange/unexpected values: sqrt(-1), sqrt(3)

A good testing framework should make it simple, and avoid copy&paste coding.

Alas, the default JUnit implementation of the parametrized tests is rather unusable… In short, JUnit expects all test methods to be parameter-less. Because of this constraint, parametrized tests with JUnit looks awkward, because parameters are passed by constructor and kept as fields!

You can see some examples here, here or here.

From what I have noticed, people who work with JUnit usually test their code with only one set of parameters. I suspect this is because parametrized tests are unusable with JUnit. And this is bad, because such tests are really useful.

If you really, really have to use JUnit (do you?) then at least us JUnit Params!

Parametrized tests

Parametrized tests are very simple with TestNG (no constructor awkwardness).

You can have as many data providers in one class as you wish. You can reuse them (call them from other classes), and you can make them "lazy", so each set of parameters is created when required.

Once you migrate to TestNG you will notice that you start writing a lot of parametrized tests. Because they are so very useful!

Parametrized tests

Ok, so this is how it looks with TestNG. There is a test method (shouldAddSameCurrencies()) which has some "testing logic" and a data provider method (getMoney()) which provides data. That is all.

@DataProvider
private static final Object[][] getMoney(){
    return new Object[][] {
        {new Money(4, "USD"), new Money(3, "USD"), 7},
        {new Money(1, "EUR"), new Money(4, "EUR"), 5},
        {new Money(1234, "CHF"), new Money(234, "CHF"), 1468}};
}

@Test(dataProvider = "getMoney")
public void shouldAddSameCurrencies(Money a, Money b,
        int expectedResult) {
    Money result = a.add(b);
    assertEquals(result.getAmount(), expectedResult);
}

Parametrized tests

You also get a very detailed information about the test results (in this case thanks to reasonable implementation of Money class toString() method), e.g.:

BTW. not sure if you noticed, but parametrized tests are perfect for verifying if all your "utils" methods work - you know, all this date/strings/numbers handling/transforming methods.

Parametrized tests

TestNG does not care how many parametrized tests you have in one class. See below for an example of MoneyTest which verifies both the constructor and add() method using data providers.

@DataProvider
private static final Object[][] getMoney(){
    return new Object[][] {
        {new Money(4, "USD"), new Money(3, "USD"), 7},
        {new Money(1, "EUR"), new Money(4, "EUR"), 5},
        {new Money(1234, "CHF"), new Money(234, "CHF"), 1468}};
}

@Test(dataProvider = "getMoney")
public void shouldAddSameCurrencies(Money a, Money b,
        int expectedResult) {
    Money result = a.add(b);
    assertEquals(result.getAmount(), expectedResult);
}

@DataProvider
private static final Object[][] getInvalidAmount(){
    return new Integer[][] {{-12387}, {-5}, {-1}};
}

@Test(dataProvider = "getInvalidAmount", expectedExceptions = IllegalArgumentException.class)
public void shouldThrowIAEForInvalidAmount(int invalidAmount) {
    Money money = new Money(invalidAmount, VALID_CURRENCY);
}

@DataProvider
private static final Object[][] getInvalidCurrency(){
    return new String[][] {{null}, {""}};
}

@Test(dataProvider = "getInvalidCurrency", expectedExceptions = IllegalArgumentException.class) 
public void shouldThrowIAEForInvalidCurrency(String invalidCurrency) {
    Money money = new Money(VALID_AMOUNT, invalidCurrency);
}

Test Fixture Setting

TestNG offers well-thought @Before… and @After… annotations (on suite, class, group and method level).

An example below shows that a server (something heavy) is created only once before a test suite, and client (something small and cheap to create) is created before each test method.

@Test
public class TestFixtureTest {

    private Server server;
    private Client client;

    @BeforeSuite(timeOut = 1000)
    public void setUpServer() {
        server = new Server();
        server.start();
    }

    @BeforeMethod
    public void setUpClient() { client = new Client(); }

    // some test methods here

    @AfterSuite
    public void shutDownServer() { server.stop(); }
}

Parameters passing

Want to:

configure your tests so they e.g. invoke GUI methods on different servers/ports?
run tests using mysql database instead of h2?
run GUI tests with Opera, Firefox, Chrome (or even IE 6)?

It is pretty simple to pass parameters to test methods (example copied directly from TestNG documentation):

@Parameters({ "datasource", "jdbcDriver" })
@BeforeMethod
public void beforeTest(String ds, String driver) {
    m_dataSource = ...;
    m_jdbcDriver = driver;
}

Parameters can come for example from configured Maven Surefire Plugin.

Groups of tests

each test class (and method) can belong to multiple groups,
it is pretty easy to run only selected group:
- e.g. you can run only tests from groups "smoke", or only "gui" tests,
- it is possible to define dependencies between methods/groups (see next slide).
all done with annotations (e.g. @Test(groups = "smoke"))

This is also supported by tools - e.g. it is pretty simple to instruct Maven to do it (see here).

Dependencies between tests

Rather bad practice with unit tests,

but very important for integration and end-to-end tests, e.g.:

fail fast:
- run Selenium tests only if application was deployed properly,
- run full system tests only if smoke tests passed,
logical dependencies between tests:
- execute shouldDeleteUserFromDatabase test only if shouldAddUserToDatabase worked

Fail fast means that the feedback will be much quicker in case of failed tests.
Logical dependencies gives you a much more realistic error information - you learn that 1 tests has failed and 99 has been skipped, which is much easier to fix than the information about 100 failed tests (OMG! what’s going on!? All tests failed…. aaarrhghg!).

Dependencies between tests

Example copied from TestNG documentation.

Method method1 will be skipped (will not run) if serverStartedOk fails.

@Test
public void serverStartedOk() {}

@Test(dependsOnMethods = { "serverStartedOk" })
public void method1() {}

Dependencies between tests

Example copied from TestNG documentation.

Method method1 will be skipped (will not run) if any method of group init fails.

@Test(groups = { "init" })
public void serverStartedOk() {}

@Test(groups = { "init" })
public void initEnvironment() {}

@Test(dependsOnGroups = { "init.* })
public void method1() {}

Listeners, Reporters

It is pretty easy to write your own listeners (which act in runtime) and reporters (which are invoked after the execution).

Example output of listener which prints test result, class and method name, and execution time of each test method:

OK       FirstTest           fifthTestMethod            1944
OK       FirstTest           firstTestMethod            3799
OK       FirstTest           fourthTestMethod           1920
OK       FirstTest           secondTestMethod           4891
OK       FirstTest           thirdTestMethod            2963
OK       SecondTest          methodA                    3525
OK       SecondTest          methodB                    1390
FAILED   SecondTest          methodC                     117
SKIPPED  SecondTest          methodD                      23
OK       SecondTest          methodE                    4552

This feature is not very useful with unit-tests but can be very handy to report progress/results of integration and end-to-end tests.

There is more of this

There are few more interesting features of TestNG (e.g. factories) but lets skip it.

Who is the boss?

images/google_trends.png

(chart from Google trends)

Well, as you can see TestNG is not really closing the gap. It rather seems to find its niche and sit there. Well, it is time to change it! :)

But who is the boss now?

All data gathered on 4th April 2012.

	JUnit	TestNG
releases since 2009 (not counting betas)	3 - from 4.8.2 to 4.10 (see JUnit announcements)	13 - from 5.12 to 6.4 (see TestNG Changelog)
mailing list	JUnit mailing list, members: 8511 emails since January 2012: ~25	TestNG mailing list, members: 2858, emails since January 2012: ~560, threads: 115

One could say, this all means only that JUnit development has been finished, all documentation has been written, all questions answered and easy to find by googling, all issues solved, … well, actually no. It rather looks, hm… a little bit abandonned.

Documentation

TestNG offers much better documentation
- even though there are numerous articles about different versions and features of JUnit you will find it hard to find a comprehensive resource on the net which explains it thoroughly.
- TestNG documentation is in very good shape, and always up-to date (seriously, it is!)

Have no fear

Yes, with TestNG you still can:

run tests with Maven, Ant, Gradle, …
execute tests with your IDE: Eclipse, IntelliJ IDEA, …
use Jenkins, Sonar, Cobertura, Mockito, FEST Fluent Assertions, Hamcrest, …
enjoy Spring support for testing,
etc., etc.

How to migrate from JUnit to TestNG?

There is Eclipse converter which does a decent job of changing imports etc. so you do not have to do it by hand
TestNG plugin for IntelliJ IDEA also provides a "Convert JUnit test to TestNG" feature
There might be an issue with objects creation, because TestNG does not create test classes before every test
- This means you need to enclose creation of objects within some method.
Another - minor - issue is with assertions, because TestNG uses different order of arguments (first actual, then expected) than JUnit (first expected, then actual)
- If you use matcher libraries, you do not have to change anything

The End

Great things are done by a series of small things brought together.

— Vincent Van Gogh

It is many small things which sum up and give you a much better experience.

TestNG offers support for all kind of tests. It is great, and still getting better with each new release.

…is TestNG a silver bullet? No, but close to it. :)

P.S. Want to know more about TestNG? Read the book.