Automated Testing is Hard – And How to Mitigate That

Automated testing is inherently difficult.

First, automated tests are themselves computer programming that also must run reliably and correctly. Verifying that the tests themselves work as intended becomes harder as the quantity and complexity of test programming increases.

Second, the need for testing extends not only over relatively narrowly defined functions, but also across systems. Systems may include browsers, front end code, session tracking, user privileges, API’s, databases, back end code and simulating human interaction. Many parts of such systems run asynchronously to each other creating challenges on setting appropriate timeout periods or in identifying signals which indicate when the next step of the test may be taken. Furthermore, successful completion of suites of automated testing may be set as a programmed requirement for program code to be copied from development to quality assurance environments.

Third, developing tests is time consuming. Not only is writing test programs a complex and iterative task like other programming tasks, but running the tests between iterations can take an annoyingly long time which further drags out test development.

These fundamental challenges are mitigated by making automated testing part of the development process from the beginning.

First, the programming team chooses libraries and environments with testability in mind. They look at testing best practices as well as programming best practices.

Second, testing costs are kept at a minimum when early design and programming is done to make automated testing easier.

Finally, early inclusion of automated testing in the development process allows for better assessment of test development capabilities and costs.

. . .

To see 3 examples of automated testing over narrowly defined functions see below. All examples are written by this author as unpaid studies and have the M.I.T. source code license.

1.)  Using Recursion – Sometimes an Elegant Solution

Source Code:
https://bseaver.jsbin.com/gahuqeh/edit?js,output
See JavaScript panel and scroll to:
* function automatedTests(sourceArray, testArray) and
* // Automated testing

Comments:
Automated testing clearly enhances value of proposed solution and may be done with few lines of code and no additional library for testing.

2.) A PHP website with Google Sheets as the Content Management System

Source Code:
https://github.com/bseaver/PHP-Put-Drive-CMS/blob/master/tests/SomePasswordTest.php

Comments:
Data driven automated testing using PHPUnit library. See other PHPUnit tests in:
https://github.com/bseaver/PHP-Put-Drive-CMS/tree/master/tests

3.) DOS Batch scripting to make sure ports are open (or closed)

Source Code:
https://github.com/bseaver/Demonstration-Projects/blob/master/mini/portsopen/portsopen_auto_test.bat

Comments:
Even the venerable DOS Batch scripting language can be used to do automated testing.