.. _unittesting: Unittesting KTL Python ====================== The KTL Python unittest suite is contained in a single file, ``test_ktl``. It has four different classes of tests it performs, based on how invasive it can be in a test environment: * :ref:`Tests without a live service ` * :ref:`Tests with a live service ` (``-s servicename``) * :ref:`Read-only tests with a live service and keyword ` (``-k keywordname``) * :ref:`Read/write tests with a live service and keyword ` (``-v keywordvalue``) When more information is specified on the command line, ``test_ktl`` always invokes the less-demanding tests before moving on to the next set. In addition to the command-line flags driving the above, ``test_ktl`` takes three additional flags: * ``-d``: Turn on verbose debug output by invoking :func:`ktl.loglevel` with :attr:`ktl.LOG_DEBUG` * ``-p``: Turn on profiling, using the :mod:`cProfile` module * ``-h``: Print information on command-line options, and exit .. note:: ``test_ktl`` does not probe the internal KTL/C layer module directly, it limits itself to the :mod:`ktl` module. It would be sensible to first assert that the KTL/C layer is behaving in a satisfactory fashion before moving on to the :mod:`ktl` Python module, but the unit tests are not presently constructed to support this approach. General principles ------------------ Everything that *can* be tested *should* be tested. New functionality in the :mod:`ktl` module should not be considered complete until a test is written to probe all available failure modes. ``test_ktl`` could also be used for test-driven development: write the test first, enumerating all the expected behavior, and only then begin writing the implementation. Most importantly, when bugs are discovered in KTL Python, it is extremely valuable to construct or augment a unit test to trigger that bug: the developer benefits immediately, in that they now have an easy way to reproduce the bug while fixing the module; future developers also benefit, in that all subsequent modifications will be tested for a failure mode that was subtle enough to slip past thorough testing. The ``test_ktl`` suite is also excellent at highlighting unintended consequences of minor changes. When making changes to KTL Python, it is to the developer's benefit to run ``test_ktl`` early and often. This is most easily accomplished with the help of a dedicated test service; examples include the *dummy* service from ``svn/kroot/kss/dummy``, and the *pie* service from ``cvs/lroot/pie``. Before trying to use KTL Python with a given KTL service at the application level, it can be informative to first run ``test_ktl`` against that service, especially if KTL Python is not already in production use with that underlying KTL client implementation. KTL Python is fairly aggressive in its adherence to KTL standards, and while it is deliberately constructed to maximize its compatibility, it can exercise services in ways unexpected by a specific KTL implementation; it is not uncommon for one-off client implementations to be written with a narrow scope, with a specific client application in mind, and specific (frequently limited) use of the KTL API. This brings up another excellent use of the ``test_ktl`` suite: testing KTL client library implementations, and the KTL/C layer itself. Because KTL Python invokes so many different KTL/C functions, ``test_ktl`` is very helpful in triggering memory leaks and other bugs in client library implementations. .. _without-service: Without a service ----------------- In the absence of any additional flags, ``test_ktl`` will probe the internal functions of KTL Python, and ensure that functions requiring a KTL service reject bad input for service names. .. _with-service: With a service -------------- If a service name is specified, ``test_ktl`` will open and close the service, probe service-level functions, and ensure that functions requiring a KTL keyword reject bad input for keyword names. .. _ro-keyword: Read-only keywords ------------------ If a keyword name is specified, ``test_ktl`` will attempt to query all available metadata for the keyword, and read its value. ``test_ktl`` will will perform many :func:`ktl.Keyword.read` operations as quickly as it can, in an effort to shake loose potential race conditions. .. _rw-keyword: Read/write keywords ------------------- If a keyword value is specified, ``test_ktl`` will attempt to write the value to the designated keyword, and will interleave write operations with both the new and the old values to test the behavior of keyword monitoring and callbacks. Similar to the read-only test above, ``test_ktl`` will also attempt to shake loose race conditions with rapid invocations of :func:`ktl.Keyword.write`. When selecting a keyword to test, the tester is encouraged to select a keyword that quickly completes write requests, if for no other reason than to reduce the amount of time spent waiting for the unit tests to complete. Example use ----------- A typical invocation of ``test_ktl`` may look like:: ./test_ktl -s pie2 -k string -v unittesting ...and here is sample output:: Exercise ktl log mechanisms ... ok Exercise ktl log levels ... ok Legacy debug log levels ... ok Query KTL Python version number ... ok 1344287457.714173 E: Test message from KTL Python unittest ok Exceptions due to absence of ktl service ... ktl_load_open.c, 293: shareable library lib__bad_service_name___keyword.so[.0.0] not found. ktl_open.c, 406: can't open shareable library: __bad_service_name___keyword. ktl_load_open.c, 293: shareable library lib__bad_service_name___keyword.so[.0.0] not found. ktl_open.c, 406: can't open shareable library: __bad_service_name___keyword. ktl_load_open.c, 293: shareable library lib__bad_service_name___keyword.so[.0.0] not found. ktl_open.c, 406: can't open shareable library: __bad_service_name___keyword. ktl_load_open.c, 293: shareable library lib__bad_service_name___keyword.so[.0.0] not found. ktl_open.c, 406: can't open shareable library: __bad_service_name___keyword. ktl_load_open.c, 293: shareable library lib__bad_service_name___keyword.so[.0.0] not found. ktl_open.c, 406: can't open shareable library: __bad_service_name___keyword. ok Open and close KTL service ... ok List KTL keywords ... ok List KTL file descriptors ... ok Create Service object ... ok Exceptions from bad encapsulated KTL reads ... ok Exceptions from bad KTL reads ... ok Exceptions from bad KTL datatype queries ... ok Exceptions from bad KTL units queries ... ok Exceptions from bad KTL writes ... ok Exceptions from bad Keyword objects ... ok Self-contained KTL reads ... ok KTL range query ... ok KTL servers query ... ok KTL datatype query ... ok KTL units query ... ok KTL reads ... ok Create KTL Keyword object ... ok KTL Keyword history ... ok KTL Keyword capabilities ... ok Acquire KTL Keyword from Service object ... ok KTL Keyword object metadata ... ok KTL reads via Keyword object ... ok KTL Keyword object comparisons ... ok Rapid, repetitive KTL reads via Keyword object ... ok Store original Keyword value ... ok Callbacks via Keyword object ... ok Monitoring via Keyword object ... ok Monitoring all Keywords in a Service ... ok KTL monitoring and callbacks (ascii) ... ok KTL monitoring and callbacks (binary) ... ok KTL writes (ascii) ... ok KTL writes (binary) ... ok KTL writes via Keyword object ... ok KTL Keyword deadlock ... ok KTL Keyword history comparisons ... ok KTL Keyword[key] = value ... ok Rapid, repetitive KTL writes ... ok ---------------------------------------------------------------------- Ran 42 tests in 18.832s OK