Retrieving data

Any PostgreSQL client application can directly access the data stored in the relational database. Any text-parsing application can read the archival logs. The database schema is documented below; for users familiar with command-line KTL tools such as show and modify, the use of gshow is recommended, as it will translate a basic query on the command line into the proper format to retrieve the data from a keyword history database.

For users interested in turning keygrabber data into FITS headers, the keyheader tool abstracts away all of the database interaction.

Database schema

Within the configured database, a single table will be created for each service monitored by keygrabber. Each individual table has the following structure:

Column	Type	Description
time	double precision	UNIX timestamp
keyword	text	KTL keyword name
binvalue	text	Binary keyword value
ascvalue	text	ASCII keyword value
repeated	integer	Count of repeated broadcasts for this keyword value
discarded	integer	Count of discarded broadcasts that arrived too quickly between the recording of this keyword value and the subsequent keyword value in the database

Here are some sample records:

       time       |  keyword  |     binvalue     |     ascvalue     | repeated | discarded
------------------+-----------+------------------+------------------+----------+-----------
 1329178093.22804 | DISPSTA   | 0                | Ready            | 0        | 0
 1329178093.23297 | DISPSTOP  | 0                | No               | 0        | 0
 1329178093.29152 | TEMP      | 15.5             | 15.5             | 0        | 0
 1329178093.29531 | UPTIME    | Heartbeat active | Heartbeat active | 0        | 0
 1329178094.62863 | SETPOINT  | 0.0              | 0.0              | 0        | 0
 1329179561.85276 | TEMP      | 15.4             | 15.4             | 0        | 0
 1329179562.37642 | TEMP      | 15.5             | 15.5             | 0        | 0

Using gshow

Beyond replicating the basic functionality of show and cshow, gshow can also query a keyword history database. Such queries are triggered by supplying one or more -date arguments on the command line. This usurps the regular control flow in the script; rather than issue requests via the regular KTL machinery, gshow queries the history database according to the parameters provided, then resumes its normal execution path using the query results as its data source. The format of the -date argument is any date string recognized by the Tcl clock scan function. Examples:

gshow -s hamiodine SETPOINT TEMP -date "September 27, 2011 11:23"
gshow -s hamiodine SETPOINT TEMP -date "September 27, 2011 11:23" -date "September 27, 2011 12:23"
gshow -s hamiodine SETPOINT TEMP -date "September 27, 2011 11:23" -window 1h
gshow -s hamiodine SETPOINT TEMP -date "September 27, 2011 11:23" -window 1h -csv 1min
gshow -s hamiodine SETPOINT TEMP -date "2 days ago"
gshow -s hamiodine SETPOINT TEMP -date "yesterday"
gshow -s hamiodine SETPOINT TEMP -date "11AM" -date "3PM"

All three of these queries are retrieving the SETPOINT and TEMP keywords from the hamiodine service. The first query will retrieve all results from the date specified until now; the second query will retrieve only the results between the two dates specified. The third query, using the -window option, is another way to retrieve the same time range. The fourth query will output the results in a comma-separated values (.csv) format, only outputting one value per minute of available data. Subsequent queries demonstrate a limited number of the many alternate ways that dates can be specified.

For further information, refer directly the gshow documentation, which one can review by running either of the following commands:

gshow -h
gshow -H

FITS headers with keyheader

The keyheader command-line tool will generate valid FITS header segments defined by a flat-text configuration file. A typical invocation of keyheader will specify a named snapshot of keywords to retrieve, and a timestamp for a snapshot of those keyword values. It is thus possible to retrieve any FITS header segment at any time after the keygrabber data is in place; for example, a data taking daemon may have three blocks defined in the configuration file:

[ExposureBegin]
[ExposureEnd]
[ReadoutEnd]

With appropriate timestamps in-hand, the data taking daemon could then request each of the three snapshots with three separate invocations of keyheader, all right before writing the final FITS file to disk:

keyheader -c ./keyheader.conf -s ExposureBegin -t 12345678
keyheader -c ./keyheader.conf -s ExposureEnd   -t 12345679
keyheader -c ./keyheader.conf -s ReadoutEnd    -t 12345680

One major benefit of constructing FITS headers in this fashion is to decouple the collection of data from the generation of the FITS header. By doing so, the data taking daemon need not be in the business of monitoring KTL services; that is all abstracted away by leveraging keygrabber. And, as long as the data was properly stored, the FITS headers can be regenerated at any time, days, months, or years after the exposure was taken.

The keyheader configuration file affords substantial flexibility in how the final FITS header segment will be formatted. A sample configuration file follows:

# The configuration file is broken into named sections, with each section
# labelled in the following manner:
#
#       [label]
#
# Configuration directives are of the form:
#
#       Option = value
#       Option:  value
#
# Section names are case sensitive. Option names are case-insensitive; values
# are case sensitive. The hash (#) character starts a comment, as does a
# semi-colon (;); only the semi-colon may be used for inline comments.
#
# For more information on the parsing of the file, see the Python documentation:
#
#       http://docs.python.org/2/library/configparser.html


# The [database] section contains parameters that will be used to access
# the PostgreSQL database containing all of the stored keyword broadcasts.

# [database]
# username = turk
# database = keywordlog
# hostname = localhost
# password =
# timeout  = 2


# The [heartbeats] block identifies heartbeat keywords for individual
# KTL services. These should be the same heartbeat keywords used by the
# monitoring keygrabber instance; while it is possible to discover such
# keywords at runtime, the SQL queries to do so are expensive.
#
# The [heartbeats] block may be included in a FITS header, like any other
# named group of keywords; unlike other named groups, there are no formatting
# options.
#
# If a configured heartbeat is not "beating" for a given service, comments
# will be added after any FITS header card that references a potentially
# stale keyword value.
#
# There are no default values for this block. Keyword values may be
# separated by commas, whitespace, or both, and may use common glob
# wildcards (? or *).

# [heartbeats]
# service: HEARTBEAT1, HEARTBEAT2, HEARTBEAT3
# apfmot:  DISP0CLK DISP1CLK DISP9CLK
# apfmon:  DISP?CLK
# eosdome: UPTIME


# Further blocks designate named groups. These groups could have meaningful
# names, like [ExposureBegin] and [ExposureEnd]. The intent is that these
# blocks could be queried as discrete units: for example, requesting the
# [ExposureEnd] block with a timestamp corresponding to the end of an exposure.
#
# In practice, it is likely that the configuration file would only contain
# sections corresponding to the beginning and end of an exposure.

# For all assignments, the left-hand-side specifies the keyword, and the
# right-hand-side describes how the data will be inserted into the FITS
# header.
#
# The left-hand-side is specified as a service.KEYWORD pair, or as a
# comment.something pair, indicating that the record should be included
# as a FITS comment. The left-hand-side should be unique within a
# configuration block; if it is not, only the last value specified will
# be visible to the application.
#
# For comments, everything on the right-hand-side is included as the
# comment text. The trailing '.something' is necessary in order for
# the left-hand-side to be unique. If only one comment is present,
# it can safely use the left-hand-side 'comment'.
#
# For non-comment records, the available fields on the right-hand-side
# are as follows:
#
#       name=           Keyword name, as inserted into the FITS header.
#       binary=         If True, use the binary value of the keyword
#                       instead of the ascii value.
#       format=         'string' or 'number'. If not specified, the
#                       default behavior is based on the keyword type.
#                       To record FITS-native boolean values (T/F), do
#                       not set the format= field, instead set binary=True.
#       comment=        FITS card comment
#
# Values may also be quoted after any fashion acceptable to Python's
# language parser: single quotes, double quotes, or a sequence of three
# of either character: ' " ''' """
#
# Any field that is not strictly alpha-numeric (plus dashes and underscores)
# should be quoted. The only field likely to require quoting is the comment
# field.


[ExposureEnd]
comment.1: The following records correspond to the end of the exposure,
comment.2: as recorded in the DATE-END record.

eosdome.TEMPNOW1: name=TEMPLEV1 comment="[degC] Dome temperature, 1st level"
eosdome.TEMPNOW3: name=TEMPLEV3 comment="[degC] Dome temperature, 3rd level"

comment.end: No further records from the end of the exposure.


[SpectrographState]
comment: The following records encapsulate the instrument state.
apfmot.ADCNAM: name=ADCNAM comment="ADC named position"
apfmot.ADCVAX: name=ADCVAX binary=True comment="[deg] ADC elevation angle"