Black box or unit testing
by Harun Yayli on Saturday, November 24th, 2007 at 3:56 pm under Ideas, PHP, Programming, Testing
Nobody denies the benefits of testing and especially unit testing.
However I’m still having hard time understanding the baised testing methodology of phpunit or other automated unit testing frameworks.
In test driven development, programmer writes a test then fills in the code to make the test pass. I find this extremely baised.
Alternatively, I find coding baised test cases that can cover the code 100% gives the developers a false sense of safety. This is rather a phylosophycal discussion than practical.
For complex scenarios of testing, can unit testing cover everything efficiently? How efficient is to find and write the correct test cases that test everything throughly? Does any company have the time and money for this (or they are willing to pay for their testers to write code for 100% coverage) ?
I really like the idea of php unit and automated tests. However for web, I’m still for the black box testing, if there is a good way of analysing the efficiency of blackbox tests.
Feel free to comment.
Updated: typos
Recent Entries
- memcache.php can delete keys now
- memcache.php is now part of pecl/memcache
- memcache.php goes PECL
- memcache.php stats like apc.php
- oci_bind_by_name maxlength is not so optional
- Is Sun going to buy PHP too?(PHP Quebec 2008)
- PHP APC apc_shm_create error on CLI
- Facebook’s Buggy Spam Detection
- Is it Firefox or Zend Debugger? Cookie Standards
- ezComponents ready for prod?
Check out the Selenium portion of PHPUnit or SimpleTest for black box testing web UIs. It’s pretty much the only option right now.
As for whether or not TDD can be successful, check out the tests in PHPT (available from https://svn.phpt.info/Core/trunk/) for a project that was done entirely with TDD. As for cost, the money is spent, it’s a matter of when. For a project with a long life, it’s definitely worth spending money up front for a better project. If it’s short-lived and won’t be around in a year, there may be no need to have a project of that quality.
November 24th, 2007 at 9:08 pmI cannot agree with this. Let’s take pearweb (http://pear.php.net) for instance, we used to have some errors when executing queries for the bug report database simply because there was no unittests. The idea behind the unit tests are actually much more than only testing functionalities, but also test behaviors, query results, etc. With our unit tests, when we developped new features, we knew which collateral damages happened with our changes by having a few failing tests.
A few companies I have done consulting at had the same mindset than yours. Why should we need unit testing when we can simply test input and output (blackbox). Well after a while after having implemented unit testing, the managers realized that unit testing does a WHOLE lot more than only a few simple things. It actually helps the developers commit code that works and that doesn’t break other parts of their website.
Those same companies that had bad code, and written unit tests had to refactor their code at some point, well, with unit tests, it allowed them to refactor the way their core modules were working without breaking other parts and making sure everything was still working fine. After this, of course they had to hire new people because the companies were growing, and companies that don’t have unittests, usually (I’d say 90%) do not have any documentation. At least, with the unit-tests, the new developer had an idea of what was expected from the code, and how to achieve it.
Other example, in a well designed web application with a modular architecture (MVC for instance), each modules would have a certain amount of tests to prove that it behaves exactly like it should (input,output,sql queries expectations, . Each module is independant and each module has it’s own behavior which is proven by the unit tests.
Some will say unit testing are only a matter of religion, just like the language you use, but it is far more different than that. There’s a reason why all languages have their own unit testing frameworks. It helps development for both single and multi workers teams.
I like unit testing.
November 24th, 2007 at 9:27 pmCode coverage indeed doesn’t guarantee anything except for the fact that it shows the product developers care about improving their tests. A framework or application with code coverage info available is usually preferred over one which doesn’t provide code coverage information at all. The fact that each line of the code is covered provides _some_ kind of security but, of course, there might be other flaws, in the business logic, for example that these tests don’t necessarily reflect, unless written well.
In my opinion a company always pays the price for testing: either in advance to pay coding the tests, or afterwards to fix the bugs and to add new functions to the application which create undesirable results and nobody knows why. Tests not only test; they also ensure your application is written in such a manner (preferably based on design patterns) which promotes extensibility and maintainability.
Generally, I believe that today it’s not mandatory yet that each application have its own unit and/or web tests but quality programming teams definitely go for it. On the other hand, if you are working on any reusable product, like a framework, it is essential that you provide enough tests to guarantee the integrity of your product.
November 25th, 2007 at 9:12 amBlack box testing can be automated too. You can’t manually beat running 3000 automated tests in 10 seconds on any web application. See http://simpletest.org/en/web_tester_documentation.html
November 25th, 2007 at 9:16 amThink of it this way - you have the exterior (your black box testing, perhaps with Selenium RC or something like HttpUnit) and the interior. The interior, like it or not, influences the exterior. However if the interior is untested (repeatable testing) how can you relate a black box issue to something internally? How does a missing form field tell you where in the code the problem is? You can make guesses - but guesses mean you need to go and hunt for it (which can take a long time - at €xx per hour).
That’s one reason for unit testing, tdd, bdd etc. You can use them to write internal code to a specified behaviour which improves overall design. If something goes off the rails, your UT/BDD framework should issue an informative report (errors, exceptions, file names and line numbers). What do web tests give you? Web testing isn’t a great way of testing underlying code in any detail.
The way I prefer to think of things, is that TDD/BDD gets me the low level code, which is covered well, and provides reasonable confidence in its working state. Web testing ensures that code is producing the output and behaviour which the client is paying us for (i.e. web tests are best for Acceptance Testing).
It’s not an option A vs option B question - you really should be doing both.
Finally, such tests are not written by dedicated testers (much as they’d moan about it) but by the developers themselves writing the code, HTML and javascript. Is test writing efficient? It’s been proposed it adds 10-20% additional development time depending on your experience - however at the end of the day (the bottom line for profitability) it cuts down maintenance, bugs, design changes, etc. These IMO normally end up being very costly. I think of it as paying in the short term, to reduce the long term costs (and let me take on more work since I’m not bogged down in maintanance!).
November 26th, 2007 at 7:40 amThank you everyone for sharing their thoughts.
Actually the questions I’ve asked were the questions I’m frequently asked. I agree with Pádraic that you need to do both(unit and blackbox). Also I agree with Norbert that code coverage is not the 100% solution.
The real question is, in the end, it’s the poor project leaders problem that the code is working or not, in an environment where they have no control on tests. Pádraic is right, it’s the testers that should write the test units. So this means the project managers/leaders or responsibles have no control on the test and the quality of the tests, which is an equivalent to black box testing from the developer point of view.
With this idea in mind, I see code coverage to test the tests.
At least a metric to test the tests.
Soon, I’ll be coming up with a PEAR package (if accepted) to automate the code coverage test/reporting/analysis on PHP.
I hope you guys find it useful.
November 26th, 2007 at 10:18 am