Integrating Event Data with AstroGrid with STAP

Introduction

The IVOA's VOEvent standard allows astronomical, solar, and STP events to be described in an XML format. VOEvents can be used by instrument teams notifying the community of new events, telescope operators publishing follow-up observations of related events, and users searching for VO data related to specific events.

Because time ranges are inherent to event descriptions, the AstroGrid Simple Time Access Protocol (STAP) web service implementation has been used to provide a queriable event interface through the client-side Workbench and VOExplorer tools.

Contents

Aims of Hands on Session

  • Familiarise oneself with the "internals" of a VOEvent description.
  • Examine the existing VOEvent publishing feeds.
  • Attach to a VOEvent feed in a programmatic fashion to capture events.
  • Store captured events in a MySQL database
  • Configure and register a STAP service (deployed to Tomcat) to provide access the stored events
  • Use the AstroGrid VODesktop to query the stored data.

Where you see the do the following symbol there are some exercises for you to do.

Accessing VOEvents

Although one of the primary motivations behind VOEvent was to enable the triggering of robotic telescopes, it is often interesting to astronomers to be directly aware and able to read the published events. The VOEvent community has come up with a rich variety of publishing methods.

Exploring VOEvent feeds

do the following
Look at the available event feeds;

Note that there is also the STAP interface in VODesktop - we will come to that below.

The VOEvent Standard

The VOEvent Standard can be found at http://www.ivoa.net/Documents/latest/VOEvent.html and the schema for a VOEvent can be found at http://www.ivoa.net/xml/VOEvent/VOEvent-v1.1.xsd.
do the following(optional)

  • download schema
  • try to create a voevent packet in your favourite xml aware editor
    • what is the smallest valid VOEvent document?

Publishing events to a VOEvent feed

VOEvent packets can be desseminated through a VOEvent broker, such as the eSTAR broker at vo.astro.ex.ac.uk. See the VOEvent sourceforge project to download perl libraries. Also see attached code for a test VOEvent client and test VOEvent broker pair.

There are several ways to generate VOEvent XML files:

  • fill out the eSTAR manual injection form (available to registered users only)
  • write a script that inserts event metadata into a VOEvent XML template
  • write a script to parse tabular catalogues into a VOEvent XML template
  • try the IDL SolarSoft vobs/ontology/export_event.pro function
  • write event metadata directly into a database, then export each database entry to VOEvent XML template

    Publication of the VOEvent message to the Event Backbone network can be done in three ways:

  • Via the eSTAR manual injection form (available to registered users only)
  • Via TCP to the eSTAR Event Broker
  • Via SOAP to the eSTAR Event Broker

    Parsing a VOEvents into a database

    Setting up the perl software

    1. download the VOEvent parsing module
    2. install the module into your perl distribution.
      • if you have write access to the perl installation then probably the neatest way to install is using CPAN
      • if you do not have write access to your perl installation then
        • create a new directory to work in - call this $WORKDIR
        • download module from http://search.cpan.org/CPAN/authors/id/A/AA/AALLAN/Astro-VO-VOEvent-0.9.4.tar.gz
        • unpack the tar file tar xvf Astro-VO-VOEvent-0.9.4.tar.gz - this will create a $WORKDIR/Astro-VO-VOEvent-0.9.4
        • create a new empty directory under your working directory $WORKDIR/perlmodule
        • cd into $WORKDIR/Astro-VO-VOEvent-0.9.4
        • perl Makefile.PL PREFIX=$WORKDIR/perlmodule
        • make install
        • set enviroment variable PERL5LIB =$WORKDIR/perlmodule/lib/perl5/site_perl/
        • test that you have what you need with perl -e "use Astro::VO::VOEvent;"

    Event metadata can be placed directly into a database. In this case, follow the instructions in the "create event tables" step. Upload data into MySQL using a form, a script using INSERT INTO, or mysqlimport. Alternatively, VOEvent metadata can be parsed from packets provided by a live VOEvent feed. In this case, follow both the "create event tables" and "parse packets from a VOEvent feed" steps.

    Create event tables

    Create a table for event metadata including event ivorn ("id" attribute from the <VOEvent> tag) url (location of the VOEvent packet as an XML file) ra and dec if applicable, at least one of start time, stop time, or peak time, event concept ("flare", "grb", etc) and name ("GRB 070616") if available, name and email of VOEvent author, params (comma-separated list of <What> param tag name, value, and unit if available: "rate=12.4 ct/s"), and refs (comma-separated list of <What> reference URL tag name and values: "photometry data=http://www.server.com/photometry.fits").

    For example, create this MySQL table:

    mysql> create table ogle (ivorn varchar(200), url varchar(200), ra float, dc float, start datetime, stop datetime, instant datetime, concept varchar(50), name varchar(50), contactname varchar(50), contactemail varchar(50), params varchar(500), refs varchar(500));

    mysql> describe ogle;

    +--------------+--------------+------+-----+---------+-------+

    Field Type Null Key Default Extra
    +--------------+--------------+------+-----+---------+-------+
    ivorn varchar(200) YES   NULL  
    url varchar(200) YES   NULL  
    ra float YES   NULL  
    dc float YES   NULL  
    start datetime YES   NULL  
    stop datetime YES   NULL  
    instant datetime YES   NULL  
    concept varchar(50) YES   NULL  
    name varchar(50) YES   NULL  
    contactname varchar(50) YES   NULL  
    contactemail varchar(50) YES   NULL  
    params varchar(500) YES   NULL  
    refs varchar(500) YES   NULL  
    +--------------+--------------+------+-----+---------+-------+

    Parse packets from a VOEvent feed

    do the following
    Skip this step if you do not wish to harvest VOEvents from live feeds into your database. Note that you need to edit the file to put in your database connection details.

    # USER CONFIGURATION - CUSTOMIZE THE FOLLOWING 4 VALUES my $dbuser="stapuser"; my $dbpasswd="stappwd"; my $dbhost="localhost"; my $db="stap";

    The event_clientDB.pl code attached can be executed as a background process to parse GCN, OGLE, or SDSS events from the eSTAR VOEvent broker at vo.astro.ex.ac.uk. The machine hosting this client must have outbound access to port 8099. Added security: this access can be optionally restricted to the eSTAR broker's IP address (144.173.229.22). This restriction is recommended since the event client has write access to your database. Customize this client to search for other event feeds by adding further elsif statements to the code beginning at line 121, "# Parse ivorn and select $table based on ivorn". Make sure that you have created a database table to hold the new events. Example client running as a background process, connected to eSTAR broker:

    % perl event_clientDB.pl -host estar6.astro.ex.ac.uk -port 8099

    Manually inject some packets

    As you can see from http://estar6.astro.ex.ac.uk/ogle/cgi-bin/timeline.cgi OGLE events only occur every few days, so we will be lucky to harvest even one event into our database during the tutorial. You can use the script parseVOEventDB.pl to insert events directly into the database.

    Deploying a STAP web service on top of an event database

    Instructions:

    1. Pre-requisites: Tomcat (5.5+), java (1.4+), MySQL (any recent version), a jar file containing the appropriate MySQL JDBC driver in $CATALINA_HOME/common/lib.
    2. Create a MySQL database to hold one or more event tables.
    3. Create a read-only MySQL user with select access to all tables that should be visible to your STAP web service(s).
      • you can skip this step and use the root user for this tutorial - for a properly deployed service a user with read-only privileges should be created for extra security.
    4. Store event parameters in a table with one row per event.
      • At a minimum, include at least one column for time information (start time, end time, or peak time)
      • If VOEvent packets are available for individual events, include a column for URLs pointing to each event's packet.
    5. Download the AstroGrid STAP service astrogrid-stap.war file. Rename the .war file to stap.war and copy it into $CATALINA_HOME/webapps
    6. Configure the web service's $CATALINA_HOME/webapps/stap/WEB-INF/classes/astrogrid.properties file.
      • Don't forget to specify the name of the event database, host machine and port, read-only username and password, access URL, and full SQL statement.
        • "accessURL" can point to a web page describing the events if VOEvent packets are not available for each event.
        • Two SQL statements can be configured: full.sql.syntax (for time-only queries) and full.sql.syntax.pos (for time plus cone search queries). Both statements may query over one or more tables.
        • Note: the current implementation of VOExplorer expects VOEvent param elements to be returned as a comma-separated string of key / value pairs (e.g., "flux=1.5 keV" or "xray class=A"). The same is true of reference URLs (e.g., "light curve="http://www.lightcurvesrus.com/path/to/lightcurve.fits"). This can be accomplished with one column each for comma-separated params and references, or by creating a comma-separated string from individual param (or reference) columns. * In addition to the 7 mandatory parameters specified in the config file, an unlimited number of optional parameters may be returned to the user. These parameters and corresponding db columns can be specified as parameters results.more.0, results.more.1. (Note that these parameter names should start from 0, not 1.)
    7. Configure the registry template $CATALINA_HOME/webapps/stap/WEB-INF/classes/regmetadata.properties file. Next, view the registry entry at http://localhost:8080/stap/MetaDataXML0_10.jsp. Use this entry to register the service with an IVOA compatible registry.
      • Be sure to include one <subject> line with value "voevent". Multiple subject lines can be used to increase filtering capability within VOExplorer.
    8. Restart the web application (or restart Tomcat - whichever is less hassle).
    9. Test the STAP deployment - perform a time query inside the Workbench application's HelioScope tool or VOExplorer's AstroScope tool and see if your STAP service is returned.

    Querying the Events with VODesktop

    • screenshot of VODesktop's AstroScope showing VOEvent services:
      voevent_astroscope.jpg

    Future work

    AstroGrid's VOEvent STAP services can be queried through HelioScope and the new VOExplorer tool. HelioScope returns event timestamps with URLs pointing to VOEvent XML packets. VOExplorer provides additional functionality: users can filter VOEvent STAP services by event type, query events by time or a combination of time and cone search, view matching event parameters in a tooltip, and open or save event reference files such as light curves, images, movies, and tables.

    Simple Event Access Protocol (SEAP) will replace STAP as the query mechanism for VOEvents. A SEAP working draft is in progress by Matthew Graham, Elizabeth Auden, and Phil Warner at http://wiki.astrogrid.org/bin/view/Astrogrid/SimpleEventAccessProtocol - all comments, suggestions, and proposed use cases are welcome.

    In addition, a proposal for VOTimeSeries is underway by several IVOA astronomers.This schema will allow providers of transient data to encapsulate time series data in a simple XML format.

    Software

    • stap.war The stap server to be installed under Tomcat.
    • Other scripts - voevent-scripts.zip
      • VOEvent client and broker pair for local testing: see attached event_client.pl and event_test.pl from eSTAR
      • VOEvent parsing library customized to extract metadata required for STAP service: see enclosed VOEventSTAP.pm.
      • VOEvent client customized to parse VOEvent packet, upload metadata to db for STAP service, and (optionally) write VOEvent XML file to local web directory: see attached voevent_clientDB.pl
    • This code uses the eSTAR Sourceforge Astro::VO::VOEvent module code, see the VOEvent sourceforge project to download perl modules.
  • Topic attachments
    I Attachment Action Size Date Who Comment
    elseproperties astrogrid.properties manage 7.6 K 25 Jun 2008 - 13:02 PaulHarrison properties file for the stap server
    elseproperties regmetadata.properties manage 8.8 K 25 Jun 2008 - 13:04 PaulHarrison registry properties for stap server
    elsewar stap.war manage 3710.7 K 26 Jun 2008 - 12:46 PaulHarrison Astrogrid STAP server war file
    zipzip voevent-scripts.zip manage 11.8 K 26 Jun 2008 - 12:39 PaulHarrison demo scripts
    Topic revision: r6 - 26 Jun 2008 - 12:46:54 - PaulHarrison
     
    This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
    Ideas, requests, problems regarding TWiki? Send feedback