Interested in TestNG?
See my book
"Practical Unit Testing
with TestNG and Mockito"

Introduction

Below you will find the results of an online survey on TestNG.

I did not find the way to share the results of survey in any civilized manner (Google Docs, aren't you missing and important feature?!) so I need to paste them as screenshots. Sorry. If you want to see original spreadsheet, go to https://docs.google.com/spreadsheet/ccc?key=0Ar3dS2idvVMLdE12WEJRY3VKb3N...

And one more thing - results lie when it comes to the number of "other" answers. Sometimes there was less of them then indicated by charts. Not sure why - maybe because of blank answers.

Results

50 people responded to online survey on TestNG.

There were also few other reasons to use TestNG mentioned by responders:

• I am forced to use it
• It allows for sequenced tests
• It seems to fit best with selenium

"I am forced to use it" is definitely my favorite. :)

---

Seems like we all like new stuff. No one pointed to anything older than 5.x, and most of use are using 6.x. A lot of people is using the latest version. Nothing surprising here, because in general TestNG is backward compatible, so not a big issue to upgrade to the latest one.

No one "always uses the latest beta", which is rather reasonable. We take beta only when testing some new features.

---

Java, Java, Java, boring... :) One poor soul still on Java 1.4. Some Groovy users, and no Scala at all.

---

IDEs rules, and so does Eclipse. More Maven than Ant.

In case of running TestNG tests, some people mentioned also:

• CI server
• Jenkins
• Gradle
• jruby version of rake via exposed ant task

I'm really disappointed with lack of Gradlers (except for myself!).

---

Regarding the features of TestNG responders also pointed out the following:

• Before Class
• Method Selector

Some comments to the most often used features:

• (38 + 22) Passing parameters to tests (using testng.xml or DataProviders) is number 1. Not surprised with DataProviders results - they are really cool, and at the same time very easy to use.
• (38) A little surprise with Groups being so often used. I do not use them for unit tests but only for integration or end-to-end tests, so I did not think they are so popular.
• (34) Tests/Groups dependencies - definitely one of the selling points of TestNG. Especially useful to implement a "fail fast" approach which is so very useful for integration and end-to-end tests.
• (22) Custom listeners / reporters. Interesting, so many of use takes adavantage of TestNG hooks allowing to use tweak the output of tests.
• (20) Parallel tests execution. Yeah. We write a lot of code which is being executed by many threads, so nothing strange that so many of us use this feature.

---

Here we can see that TestNG is much more than unit testing frameworks.

In case of this question responders mentioned many specific testing frameworks and tools they use:

• end to end tests with MicroFocus Silk4J
• DBUnit
• system test on a telecom SUT
• DB validation with data providers in groovy
• we also run tests that exist in other frameworks
• massive tests, random tests, ad-hoc tests
• functional system tests
• regression tests
• Blitzer to carry out performance tests

---

Few comments on this:

• Code coverage tools used only by ~50% of us? I find it surprising. I would expect it to be much more. Is it because a lot of tests we run with TestNG are integration/end-to-end tests which makes it harder to measure code coverage?
• Test doubles are our friends. Seems like this testing technique is commonly used by TestNG users.
• Quite a lot of people mentioned ReportNG. Apparently the aesthethic look of the default TestNG reports is not enough.
• Web tests seems to be very popular with TestNG.

Responders mentioned the following "other" tools they use together with TestNG:

• Cuanto
• STAF
• xpath
• firebug

---

JUnit and then almost nothing. The only "other" answer was:

• NUnit

---

Nothing beats the original documentation (btw. why 8% of us do not use it?). Other online sources are also popular. In "other" category the folowing responses were given:

• browsing sources
• Just reading the mailing list
• video source
• Trial and Error

---

One additional comment regarding the TestNG documentation:

• So-so, but is most of the time quite updated

---

Test-first or code-first? Based on the given answers, it seems there are more options:

• Depends
• I am a tester so its always code first (which I don't write ;) )
• Varies with new code, plus a goal to add tests to large existing codebase.

And by the way, test-first is more popular than I expected it to be. Still, less popular than code-first.

---

And finally the last questions:

Forgotten Questions

Things I should have asked about:

• do you modify directly testng.xml?
• are you a boy or a girl?
• who are you - developer, tester, etc.
• what do you like / dislike about the TestNG documentation?
• missing features of TestNG
• free space for expressing your opinions/expectations/comments on TestNG

Well, next year maybe. :)

Thank You!

Thank you once again for responding to this survey.