Summary of the work of the Referee Commission

For many, TestingCup 2018 is just a memory but The Referee Commission is still working long after the light goes out. It is time for summaries and reflections.

The work of the jury lasts for almost a whole year. We have started just after the edition in Gdańsk and now we are already thinking about the next edition. Our goal is to provide you with an interesting and exciting challenge in the form of MrBuggy, at the same time, the tool that will help to select the best testers. By analyzing the surveys and your opinions, you appreciated our work and you liked the application. Maybe not everyone remembers, but in the last year we did a small research of which edition of the competition you liked best and most of you pointed Wrocław. So we decided to develop that application and we turned single-screen tasks into a series of tasks that formed a logical process. This is how the application for managing orders and test projects was created. At the initial stages, we divided defects into single-screen and process, but we rejected this concept of classification as illegible.

The enthusiastic evaluation of MrBuggy itself makes us very happy but we also heard interesting opinions comparing versions 5 and 6 of the application. MrBuggy 5 was complicated, very difficult to understand, let alone to test that is why it reflected the natural environment of each tester. MrBuggy 6 was just right. You could quickly find out what it is and what it is for and check all functions within 3 hours. So we depart from faithful presentation of everyday work for fair play rules.

Easy app and its large number of hidden bugs translated into a record number of reports – over 2300. And despite (or thanks to) lack of specification of requirements. Thus, a record from the previous year was broken, where reporting of non-defects was not punished. In 2017, non-defects accounted for about 30% of all reports, in 2018 it was less than 10% of reports. The changed convention for charter-based exploration testing has not been seen negatively – who knows, maybe this change of convention also translated into record reports of defects and finding all those identified as critical.

In the 2017 edition you had two additional options:

  • evaluation of yourself and other participants
  • “challenge” the Commission’s assessment

This year, we gave up those options mainly due to your feedback after the previous edition about drawing your attention away from the conference and extending the stress of the competition for two days. However, we are considering a return to this concept and at least a partial implementation in the following years.

During 2018 edition, once again and with the help of our sponsor, TomTom we distributed 4 special prizes: for the most interesting and unique bug reporting (for both individual and team category). It was not an easy task to select the winners for the reward for the bug report because we wanted to choose the two best, original and, at the same time, important reports for the application.

Alexander Zaleski won the prize (as an individual contestant) for an alternative approach to found by almost everyone “Critical exception Xcwne-02302” and U nas działa \ Comarch SA (team), who where it was necessary to give the arrow down press the up arrow and get the script exception.
Before the competition, we knew 41 defects, and you managed to find 48 more. Because there were many defects related to the same functionalities, they were grouped under one defect corresponding to the bug, and further reports were marked as duplicates. An example was the various reports for field validation rules where validation did not work. For the duplicates 0 points were given.

We were experimenting with the reference documents of reports this year, that was given with the application and it received very positive opinions from you in the surveys. Oddly, for the first time, no team has scored a set of points (35) for the report. This may be due to the specificity of reporting in exploratory testing which this year appeared for the first time in the history of our event.

You know the winners, but at the end we would like to tell you what contributed to the good result. This year teams and individual contestants won thanks to:

  • greater focus on defect reporting, not writing reports; due to the large number of serious defects, reports played a secondary role,
  • they did not focus on simple form of defects, but analyzed the application logic; there were the highest rated defects,
  • they widely described “controversial” defects; sometimes in exploratory testing you have to not only show the problem, but also justify why it is a defect,
  • they gave up reporting bugs, which can not be unambiguously described as a defect and it is difficult to determine if this should work this way.

Despite the fact that everyone seems to be happy with the 2018 edition, as the Commission, we will look for further improvements and new ideas. The driving force of TestingCup is the ability to experiment and discover new areas. Positive change is our basic driving force in this case. Join us during the 2019 edition and see what we will prepare for you!

TestingCup 2018 in numbers:

  • 75 teams, 26 individual contestants (including one absent)

Below you can find the number of reports in each category (after the evaluation of the jury):

  • 118 suggestions (for 0 points)
  • 289 duplicates (for 0 points)
  • 39 incomprehensible (for 0 points)
  • 209 non-defects (for -5 points)
  • 114 not reproducible (for 0 points)
  • 9 typos (for 0 points)
  • 240 language problems / utility / UX (for 1 point)
  • 3 defects of error messages (for 3 points)
  • 715 security reports / defect reports in a not critical functional area (for 5 points)
  • 490 functional defects (for 7 points)
  • 105 defects: critical functional error (for 10 points)

Visit: where you can find the application itself, list of the defects and 3 exemplary reports.

We hope you will enjoy MrBuggy7 as well!

Referee Commision: Grzegorz Dawid, Łukasz Gałuszka, Natalia Krawczyk-Grzegorzewicz, Kamila Mrozek, Dariusz Olszewski, Radek Smilgin, Maciej Zaborowski, Michał Zacharuk.