DcaJune2008AstrogridTimeNotes < EuroVODCA

EuroVODCA Web>DcaJune2008Workshop>DcaJune2008AstrogridTime>DcaJune2008AstrogridTimeNotes (26 Jun 2008, PaulHarrison) (raw view)
EditAttach
---+ Integrating Event Data with AstroGrid with STAP

---++ Introduction

The IVOA's <a href="http://www.ivoa.net/Documents/latest/VOEvent.html">VOEvent standard</a> allows astronomical, solar, and STP events to be described in an XML format. VOEvents can be used by instrument teams notifying the community of new events, telescope operators publishing follow-up observations of related events, and users searching for VO data related to specific events. 

Because time ranges are inherent to event descriptions, the AstroGrid Simple Time Access Protocol (STAP) web service implementation has been used to provide a queriable event interface through the client-side Workbench and VOExplorer tools.

---++ Contents

%TOC{depth="2"}%

---++ Aims of Hands on Session
   * Familiarise oneself with the "internals" of a VOEvent description.
   * Examine the existing VOEvent publishing feeds.
   * Attach to a VOEvent feed in a programmatic fashion to capture events.
   * Store captured events in a MySQL database
   * Configure and register a STAP service (deployed to Tomcat) to provide access the stored events 
   * Use the AstroGrid VODesktop to query the stored data.
   
Where you see the <img src="%ATTACHURLPATH%/exercise.gif" width='124' height='62'  alt="do the following"/> symbol there are some exercises for you to do.


---++ Accessing VOEvents

Although one of the primary motivations behind VOEvent was to enable the triggering of robotic telescopes, it is often interesting to astronomers to be directly aware and able to read the published events. The VOEvent community has come up with a rich variety of publishing methods.

---+++ Exploring VOEvent feeds 
<img src="%ATTACHURLPATH%/exercise.gif" width='124' height='62'  alt="do the following"/><br/>
Look at the available event feeds;

   * an RSS feed <a href="http://www.estar.org.uk/voevent/eSTAR/eSTAR.rdf" class="external text" title="http://www.estar.org.uk/voevent/eSTAR/eSTAR.rdf" rel="nofollow"><img src="http://www.estar.org.uk/wiki/uploads/8/8b/Xml-button.jpg" alt="Xml-button.jpg"></a><a href="http://www.estar.org.uk/voevent/eSTAR/eSTAR.rdf" class="external text" title="http://www.estar.org.uk/voevent/eSTAR/eSTAR.rdf" rel="nofollow"><img src="http://www.estar.org.uk/wiki/uploads/0/09/Rss.png" alt="Rss.png"></a><span> eSTAR native (including OGLE messages) at http://www.estar.org.uk/voevent/eSTAR/eSTAR.rdf
 - look at this in your favourite newsreader;
   * EStar event timeline viewer http://estar6.astro.ex.ac.uk/ogle/cgi-bin/timeline.cgi
   * Google Earth (Sky mode) VOEvents http://services.google.com/earth/kmz/voe_vent_net_n.kmz
   * A directory full of real events from the OGLE feed can be found at http://www.estar.org.uk/voevent/eSTAR/uk.org.estar/pl.edu.ogle/.
   * The http://voeventnet.org/ site provides a good starting point to explore the current event feeds.

_Note_ that there is also the STAP interface in VODesktop - we will come to that below.

---++ The VOEvent Standard


The VOEvent Standard can be found at http://www.ivoa.net/Documents/latest/VOEvent.html and the
 schema for a VOEvent can be found at http://www.ivoa.net/xml/VOEvent/VOEvent-v1.1.xsd.<br/>
 <img src="%ATTACHURLPATH%/exercise.gif" width='124' height='62'  alt="do the following"/>(optional)
   * download schema
   * try to create a voevent packet in your favourite xml aware editor
      * what is the smallest valid VOEvent document? 

---++ Publishing events to a VOEvent feed

VOEvent packets can be desseminated through a VOEvent broker, such as the eSTAR broker at vo.astro.ex.ac.uk. See the <a href="http://sourceforge.net/projects/voevent/">VOEvent sourceforge project</a> to download perl libraries. Also see attached code for a  test VOEvent client and test VOEvent broker pair.

There are several ways to generate VOEvent XML files:
<li>fill out the eSTAR <a href="http://www.estar.org.uk/wiki/index.php/VOEvent">manual injection form</a> (available to registered users only)
<li>write a script that inserts event metadata into a VOEvent XML template
<li>write a script to parse tabular catalogues into a VOEvent XML template
<li>try the IDL SolarSoft vobs/ontology/export_event.pro function
<li>write event metadata directly into a database, then export each database entry to VOEvent XML template

Publication of the VOEvent message to the Event Backbone network can be done in three ways:
<li>Via the eSTAR <a href="http://www.estar.org.uk/wiki/index.php/VOEvent">manual injection form</a> (available to registered users only)
<li>Via TCP to the eSTAR <a href="http://www.estar.org.uk/wiki/index.php/ESTAR_Event_Broker">Event Broker</a>
<li>Via SOAP to the eSTAR  <a href="http://www.estar.org.uk/wiki/index.php/ESTAR_Event_Broker">Event Broker</a>

---++ Parsing a VOEvents into a database

---+++ Setting up the perl software

   1 download the VOEvent parsing module
   1 install the module into your perl distribution.
      * if you have write access to the perl installation then probably the neatest way to install is using CPAN
      * if you do not have write access to your perl installation then
         * create a new directory to work in - call this $WORKDIR
         * download module from http://search.cpan.org/CPAN/authors/id/A/AA/AALLAN/Astro-VO-VOEvent-0.9.4.tar.gz
         * unpack the tar file =tar xvf Astro-VO-VOEvent-0.9.4.tar.gz= - this will create a $WORKDIR/Astro-VO-VOEvent-0.9.4
         * create a new empty directory under your working directory $WORKDIR/perlmodule
         * cd into $WORKDIR/Astro-VO-VOEvent-0.9.4
         * =perl Makefile.PL PREFIX=$WORKDIR/perlmodule=
         * make install
         * set enviroment variable PERL5LIB=$WORKDIR/perlmodule/lib/perl5/site_perl/
         * test that you have what you need with =perl -e "use Astro::VO::VOEvent;"=

Event metadata can be placed directly into a database. In this case, follow the instructions in the "create event tables" step. Upload data into MySQL using a form, a script using INSERT INTO, or <tt>mysqlimport</tt>. Alternatively, VOEvent metadata can be parsed from packets provided by a live VOEvent feed. In this case, follow both the "create event tables" and "parse packets from a VOEvent feed" steps.

---+++ Create event tables

Create a table for event metadata including event ivorn ("id" attribute from the &lt;VOEvent&gt; tag) url (location of the VOEvent packet as an XML file) ra and dec if applicable,  at least one of start time, stop time, or peak time,  event concept ("flare", "grb", etc) and name ("GRB 070616") if available, name and email of VOEvent author, params (comma-separated list of &lt;What&gt; param tag name, value, and unit if available: "rate=12.4 ct/s"), and refs (comma-separated list of &lt;What&gt; reference URL tag name and values: "photometry data=http://www.server.com/photometry.fits").

For example, create this MySQL table:

<tt>mysql> create table ogle (ivorn varchar(200), url varchar(200), ra float, dc float, start datetime, stop datetime, instant datetime, concept varchar(50), name varchar(50), contactname varchar(50), contactemail varchar(50), params varchar(500), refs varchar(500));

mysql> describe ogle;
</tt>

+--------------+--------------+------+-----+---------+-------+
| Field        | Type         | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+-------+
| ivorn        | varchar(200) | YES  |     | NULL    |       |
| url          | varchar(200) | YES  |     | NULL    |       |
| ra           | float        | YES  |     | NULL    |       |
| dc           | float        | YES  |     | NULL    |       |
| start        | datetime     | YES  |     | NULL    |       |
| stop         | datetime     | YES  |     | NULL    |       |
| instant      | datetime     | YES  |     | NULL    |       |
| concept      | varchar(50)  | YES  |     | NULL    |       |
| name         | varchar(50)  | YES  |     | NULL    |       |
| contactname  | varchar(50)  | YES  |     | NULL    |       |
| contactemail | varchar(50)  | YES  |     | NULL    |       |
| params       | varchar(500) | YES  |     | NULL    |       |
| refs         | varchar(500) | YES  |     | NULL    |       |
+--------------+--------------+------+-----+---------+-------+

---+++ Parse packets from a VOEvent feed

<img src="%ATTACHURLPATH%/exercise.gif" width='124' height='62'  alt="do the following"/><br/>
Skip this step if you do not wish to harvest VOEvents from live feeds into your database. Note that you need to edit the file to put in your database connection details.

<code>
# USER CONFIGURATION - CUSTOMIZE THE FOLLOWING 4 VALUES
my $dbuser="stapuser";
my $dbpasswd="stappwd";
my $dbhost="localhost";
my $db="stap";
</code>

The <tt>event_clientDB.pl</tt> code attached can be executed as a background process to parse GCN, OGLE, or SDSS events from the eSTAR VOEvent broker at vo.astro.ex.ac.uk. The machine hosting this client must have outbound access to port 8099. <b>Added security:</b> this access can be optionally restricted to the eSTAR broker's IP address (144.173.229.22). This restriction is recommended since the event client has write access to your database. Customize this client to search for other event feeds by adding further <tt>elsif</tt> statements to the code beginning at line 121, "# Parse ivorn and select $table based on ivorn".  Make sure that you have created a database table to hold the new events. Example client running as a background process, connected to eSTAR broker: 

<tt>% perl event_clientDB.pl -host estar6.astro.ex.ac.uk -port 8099</tt> 

---+++ Manually inject some packets

As you can see from http://estar6.astro.ex.ac.uk/ogle/cgi-bin/timeline.cgi OGLE events only occur every few days, so we will be lucky to harvest even one event into our database during the tutorial. You can use the script
=parseVOEventDB.pl= to insert events directly into the database.


---++ Deploying a STAP web service on top of an event database

<b>Instructions:</b>
   1 Pre-requisites: Tomcat (5.5+), java (1.4+), MySQL (any recent version), a jar file containing the appropriate MySQL JDBC driver in $CATALINA_HOME/common/lib.
   1 Create a MySQL database to hold one or more event tables. 
   1 Create a read-only MySQL user with select access to all tables that should be visible to your STAP web service(s).
      * you can skip this step and use the root user for this tutorial - for a properly deployed service a user with read-only privileges should be created for extra security.
   1 Store event parameters in a table with one row per event. 
      * At a minimum, include at least one column for time information (start time, end time, or peak time)
      * If VOEvent packets are available for individual events, include a column for URLs pointing to each event's packet.
   1 Download the AstroGrid STAP service astrogrid-stap.war file. Rename the .war file to =stap.war= and copy it into =$CATALINA_HOME/webapps=
   1  Configure the web service's $CATALINA_HOME/webapps/stap/WEB-INF/classes/astrogrid.properties file.
      * Don't forget to specify the name of the event database, host machine and port, read-only username and password, access URL, and full SQL statement.
         * "accessURL" can point to a web page describing the events if VOEvent packets are not available for each event.
         * Two SQL statements can be configured: full.sql.syntax (for time-only queries) and full.sql.syntax.pos (for time plus cone search queries). Both statements may query over one or more tables.
         * Note: the current implementation of VOExplorer expects VOEvent param elements to be returned as a comma-separated string of key / value pairs (e.g., "flux=1.5 keV" or "xray class=A"). The same is true of reference URLs (e.g., "light curve="http://www.lightcurvesrus.com/path/to/lightcurve.fits"). This can be accomplished with one column each for comma-separated params and references, or by creating a comma-separated string from individual param (or reference) columns.
     * In addition to the 7 mandatory parameters specified in the config file, an unlimited number of optional parameters may be returned to the user. These parameters and corresponding db columns can be specified as parameters results.more.0, results.more.1. (Note that these parameter names should start from 0, not 1.) 
   1 Configure the registry template $CATALINA_HOME/webapps/stap/WEB-INF/classes/regmetadata.properties file. Next, view the registry entry at http://localhost:8080/stap/MetaDataXML0_10.jsp. Use this entry to register the service with an IVOA compatible registry.
      * Be sure to include one &lt;subject&gt; line with value "voevent". Multiple subject lines can be used to increase filtering capability within VOExplorer.
   1 Restart the web application (or restart Tomcat - whichever is less hassle).
   1 Test the STAP deployment - perform a time query inside the Workbench application's HelioScope tool or VOExplorer's AstroScope tool and see if your STAP service is returned.

---++ Querying the Events with VODesktop

   * screenshot of VODesktop's AstroScope showing VOEvent services: <br />
     <img src="http://wiki.astrogrid.org/pub/Astrogrid/VoEventSTAPtutorial/voevent_astroscope.jpg" alt="voevent_astroscope.jpg" width='996' height='703' />

---++ Future work

AstroGrid's VOEvent STAP services can be queried through HelioScope and the new VOExplorer tool. HelioScope returns event timestamps with URLs pointing to VOEvent XML packets. VOExplorer provides additional functionality: users can filter VOEvent STAP services by event type, query events by time or a combination of time and cone search, view matching event parameters in a tooltip, and open or save event reference files such as light curves, images, movies, and tables.

Simple Event Access Protocol (SEAP) will replace STAP as the query mechanism for VOEvents. A SEAP working draft is in progress by Matthew Graham, Elizabeth Auden, and Phil Warner at http://wiki.astrogrid.org/bin/view/Astrogrid/SimpleEventAccessProtocol - all comments, suggestions, and proposed use cases are welcome.

In addition, a proposal for VOTimeSeries is underway by several IVOA astronomers.This schema will allow providers of transient data to encapsulate time series data in a simple XML format.

---++ Software

   * <b>[[%ATTACHURLPATH%/stap.war][stap.war]]</b> The stap server to be installed under Tomcat.
   * Other scripts -  [[%ATTACHURL%/voevent-scripts.zip][voevent-scripts.zip]]
      * <b>VOEvent client and broker pair for local testing:</b> see attached event_client.pl and event_test.pl from eSTAR
      * <b>VOEvent parsing library customized to extract metadata required for STAP service:</b> see enclosed !VOEventSTAP.pm.
      * <b>VOEvent client customized to parse VOEvent packet, upload metadata to db for STAP service, and (optionally) write VOEvent XML file to local web directory:</b> see attached voevent_clientDB.pl
   * This code uses the eSTAR Sourceforge Astro::VO::VOEvent module code, see the <a href="http://sourceforge.net/projects/voevent/">VOEvent sourceforge project</a> to download perl modules.
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
properties	astrogrid.properties	r1	manage	7.6 K	25 Jun 2008 - 11:02	PaulHarrison	properties file for the stap server
properties	regmetadata.properties	r1	manage	8.8 K	25 Jun 2008 - 11:04	PaulHarrison	registry properties for stap server
war	stap.war	r2 r1	manage	3710.7 K	26 Jun 2008 - 10:46	PaulHarrison	Astrogrid STAP server war file
zip	voevent-scripts.zip	r3 r2 r1	manage	11.8 K	26 Jun 2008 - 10:39	PaulHarrison	demo scripts
Topic revision: r6 - 26 Jun 2008 - PaulHarrison
EuroVODCA
Links
Euro-VO
Euro-VO AIDA
IVOA
VOTech
Communication Network Development in FP6
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback