Overview and basic design

To boil it down to a single sentence, keygrabber makes a complete record of all available telemetry exposed by KTL services. Keygrabber is a compromise, seeking a balance between high performance and perfect fidelity, while allowing users to mine the recorded data to resolve any problems whose solutions can be determined (or at least suggested) by KTL keyword data alone.

A few example uses: a broad selection of records from a specific time can be used to fabricate a FITS header on-demand; a deep inspection of a few keywords from an extended time series can be used to perform trend analysis; when troubleshooting a problem, inspection of discrete values may provoke new questions for new values, all of which are similarly available for granular acquisition.

For further reading on application design, please refer to SPIE 84510Y, A Complete History of Everything (Lanclos 2012), a copy of which is freely available on Kyle Lanclos’s personal website.

What is KTL?

The Keck Task Library (KTL) is an application programming interface (API) that revolves around keyword/value pairs. A service is a collection of keywords, each of which can be read, written to, or monitored for changes. A minority of keywords are configured as read-only or write-only, depending on the needs of the KTL service.

One or more software daemons will be responsible for handling keyword queries; these daemons are referred to as dispatchers. If a dispatcher is offline, that service’s keywords (or at least a portion of them) will not be available.

Keywords can be one of a few different basic types, allowing for native representation of strings, numbers, and numeric arrays. All keyword values can be formatted in two ways: an ascii representation, and a binary representation. For floating point numbers, the difference between the two could be the number of recorded decimal places, or it could be the same number represented in completely different ways; for example, the binary representation for a telescope pointing keyword may be in radians, while the ascii representation could be hexagesimal. For boolean or enumerated types, the binary representation is an integer, while the ascii representation is a string; for a boolean keyword, the binary values may be 0 and 1, coresponding to False and True.

What is a broadcast?

When monitoring a keyword, KTL describes it as expressing interest in broadcast events; this reflects its age more than a desire for unique nomenclature. If KTL had been created 15 years later, it would instead refer to subscribing to publications.

A single broadcast (publication) of a single KTL keyword is a value, usually one temporally relevant to when the broadcast occurred. Broadcasts are generated at the sole discretion of the dispatcher for the KTL service, and may be delayed arbitrarily depending on how the dispatcher chooses to batch its generation of broadcasts. For example, some dispatchers are known to broadcast all of their keyword values once per second, regardless of whether they have changed.

Keygrabber will record the ascii and binary representations of the keyword value, as well as a precise timestamp for the processing of the keyword broadcast. In some cases, a KTL client library may set the timestamp directly; if present, KTL Python (and by extension, keygrabber) will use that timestamp rather than set one upon processing the broadcast. The significance and accuracy of the timestamp set by a KTL client library is wholly at the discretion of the KTL client implementation.

Each broadcast will be recorded as a single row in a database table dedicated to that specific service.

The significance of the KTL implementation

Some features of KTL are optional, and their implementation is not universal; some features are essential, but their style of implementation may vary. KTL Python papers over these differences wherever possible, establishing a truly unified interface that keygrabber can leverage without implementation-specific knowledge.

Naturally, not everything can be swept under the rug.

If a KTL service does not support broadcasts, KTL Python will transparently poll individual keywords for new values. This can be resource intensive, which is why all modern KTL implementations support broadcasts.

Some KTL client library implementations do not support all keyword types. One example is the KTL RPC framework, which (as of this writing) does not support boolean, enumerated, mask, or array keyword types. While array types are rarely used, the absence of enumerated and mask types will affect how keygrabber chooses what to record (see the What gets recorded section below).

Some implementations may not be ready for the intense level of scrutiny performed by KTL Python and keygrabber. Many KTL implementations only concern themselves with a limited range of client applications, in some cases, client applications that themselves have been customized to expect the specific behavior of a specific implementation. KTL Python will probe the compatibility of any service it is asked to inspect, and expects honest answers from the KTL service; this may well tickle branches of code that were not previously exercised by the original client applications. Even if a KTL implementation does strictly adhere to the KTL API, the sheer volume of initial requests issued by keygrabber may overwhelm either the protocol used to communicate with a dispatcher, or the dispatcher itself.

In all cases, advance testing is strongly advised before keygrabber is deployed against production systems.

How keygrabber interacts with KTL services

On startup, keygrabber will parse its command-line arguments and (if specified) its configuration file. Refer to the documention on Keygrabber configuration for more details on command-line options and configuration file syntax. The resulting configuration will dictate which KTL services will be monitored, whether there are any keywords that should be ignored, and which keywords should be monitored as heartbeats. A heartbeat keyword is one that broadcasts at predictable intervals, and will be used by keygrabber to assert whether a service is online, and as a test keyword when assessing whether an automatic reconnection to a KTL service is required.

Once configured, keygrabber will open a connection to every monitored service. It will inspect every available keyword: ignored keywords will be ignored, as will keywords that cannot be read. All other keywords will be monitored for broadcast events; as broadcasts arrive, they are appended to internal queues for future processing.

Once at this stage, keygrabber’s interaction with the KTL service is very limited: receive broadcasts, queue broadcasts; repeat indefinitely.

What gets recorded

The fidelity with which keygrabber records broadcasts is configurable. By default, it will record all broadcasts of string, boolean, enumerated, and mask keyword types, with the assumption that all such broadcasts represent significant deviations from previous broadcasts. Numeric keyword types, which includes integers, floats, and doubles, are (again, by default) recorded with reduced fidelity, with the assumption that the absence of one or more broadcasts of a continuously varying numerical value does not represent a significant loss of continuity in the data set. When in this mode, the number of times that keygrabber discards a broadcast for a given keyword will be recorded in the database; regardless of the keyword type, keygrabber will also record a discrete count of the number of repeated broadcasts of identical keyword values.

Each monitored service will have its own table in the database; refer to the Database schema section for details.