Unittesting KTL Python

The KTL Python unittest suite is contained in a single file, test_ktl. It has four different classes of tests it performs, based on how invasive it can be in a test environment:

When more information is specified on the command line, test_ktl always invokes the less-demanding tests before moving on to the next set. In addition to the command-line flags driving the above, test_ktl takes three additional flags:

  • -d: Turn on verbose debug output by invoking ktl.loglevel() with ktl.LOG_DEBUG
  • -p: Turn on profiling, using the cProfile module
  • -h: Print information on command-line options, and exit

Note

test_ktl does not probe the internal KTL/C layer module directly, it limits itself to the ktl module. It would be sensible to first assert that the KTL/C layer is behaving in a satisfactory fashion before moving on to the ktl Python module, but the unit tests are not presently constructed to support this approach.

General principles

Everything that can be tested should be tested. New functionality in the ktl module should not be considered complete until a test is written to probe all available failure modes. test_ktl could also be used for test-driven development: write the test first, enumerating all the expected behavior, and only then begin writing the implementation. Most importantly, when bugs are discovered in KTL Python, it is extremely valuable to construct or augment a unit test to trigger that bug: the developer benefits immediately, in that they now have an easy way to reproduce the bug while fixing the module; future developers also benefit, in that all subsequent modifications will be tested for a failure mode that was subtle enough to slip past thorough testing.

The test_ktl suite is also excellent at highlighting unintended consequences of minor changes. When making changes to KTL Python, it is to the developer’s benefit to run test_ktl early and often. This is most easily accomplished with the help of a dedicated test service; examples include the dummy service from svn/kroot/kss/dummy, and the pie service from cvs/lroot/pie.

Before trying to use KTL Python with a given KTL service at the application level, it can be informative to first run test_ktl against that service, especially if KTL Python is not already in production use with that underlying KTL client implementation. KTL Python is fairly aggressive in its adherence to KTL standards, and while it is deliberately constructed to maximize its compatibility, it can exercise services in ways unexpected by a specific KTL implementation; it is not uncommon for one-off client implementations to be written with a narrow scope, with a specific client application in mind, and specific (frequently limited) use of the KTL API.

This brings up another excellent use of the test_ktl suite: testing KTL client library implementations, and the KTL/C layer itself. Because KTL Python invokes so many different KTL/C functions, test_ktl is very helpful in triggering memory leaks and other bugs in client library implementations.

Without a service

In the absence of any additional flags, test_ktl will probe the internal functions of KTL Python, and ensure that functions requiring a KTL service reject bad input for service names.

With a service

If a service name is specified, test_ktl will open and close the service, probe service-level functions, and ensure that functions requiring a KTL keyword reject bad input for keyword names.

Read-only keywords

If a keyword name is specified, test_ktl will attempt to query all available metadata for the keyword, and read its value. test_ktl will will perform many ktl.Keyword.read() operations as quickly as it can, in an effort to shake loose potential race conditions.

Read/write keywords

If a keyword value is specified, test_ktl will attempt to write the value to the designated keyword, and will interleave write operations with both the new and the old values to test the behavior of keyword monitoring and callbacks. Similar to the read-only test above, test_ktl will also attempt to shake loose race conditions with rapid invocations of ktl.Keyword.write(). When selecting a keyword to test, the tester is encouraged to select a keyword that quickly completes write requests, if for no other reason than to reduce the amount of time spent waiting for the unit tests to complete.

Example use

A typical invocation of test_ktl may look like:

./test_ktl -s pie2 -k string -v unittesting

…and here is sample output:

Exercise ktl log mechanisms ... ok
Exercise ktl log levels ... ok
Legacy debug log levels ... ok
Query KTL Python version number ... ok
1344287457.714173 E: Test message from KTL Python unittest
ok
Exceptions due to absence of ktl service ...
ktl_load_open.c, 293: shareable library lib__bad_service_name___keyword.so[.0.0] not found.
ktl_open.c, 406: can't open shareable library: __bad_service_name___keyword.
ktl_load_open.c, 293: shareable library lib__bad_service_name___keyword.so[.0.0] not found.
ktl_open.c, 406: can't open shareable library: __bad_service_name___keyword.
ktl_load_open.c, 293: shareable library lib__bad_service_name___keyword.so[.0.0] not found.
ktl_open.c, 406: can't open shareable library: __bad_service_name___keyword.
ktl_load_open.c, 293: shareable library lib__bad_service_name___keyword.so[.0.0] not found.
ktl_open.c, 406: can't open shareable library: __bad_service_name___keyword.
ktl_load_open.c, 293: shareable library lib__bad_service_name___keyword.so[.0.0] not found.
ktl_open.c, 406: can't open shareable library: __bad_service_name___keyword.
ok
Open and close KTL service ... ok
List KTL keywords ... ok
List KTL file descriptors ... ok
Create Service object ... ok
Exceptions from bad encapsulated KTL reads ... ok
Exceptions from bad KTL reads ... ok
Exceptions from bad KTL datatype queries ... ok
Exceptions from bad KTL units queries ... ok
Exceptions from bad KTL writes ... ok
Exceptions from bad Keyword objects ... ok
Self-contained KTL reads ... ok
KTL range query ... ok
KTL servers query ... ok
KTL datatype query ... ok
KTL units query ... ok
KTL reads ... ok
Create KTL Keyword object ... ok
KTL Keyword history ... ok
KTL Keyword capabilities ... ok
Acquire KTL Keyword from Service object ... ok
KTL Keyword object metadata ... ok
KTL reads via Keyword object ... ok
KTL Keyword object comparisons ... ok
Rapid, repetitive KTL reads via Keyword object ... ok
Store original Keyword value ... ok
Callbacks via Keyword object ... ok
Monitoring via Keyword object ... ok
Monitoring all Keywords in a Service ... ok
KTL monitoring and callbacks (ascii) ... ok
KTL monitoring and callbacks (binary) ... ok
KTL writes (ascii) ... ok
KTL writes (binary) ... ok
KTL writes via Keyword object ... ok
KTL Keyword deadlock ... ok
KTL Keyword history comparisons ... ok
KTL Keyword[key] = value ... ok
Rapid, repetitive KTL writes ... ok

----------------------------------------------------------------------
Ran 42 tests in 18.832s

OK