Publishing images with SAADA

Software configuration and installation

Java 1.5

Java JDK 1.5 can be installed from the workshop DVD.

Tomcat web server

Install Tomcat 5.0 from the workshop DVD.

Aladin

Download the workshop-customized Aladin.jar

PostgreSQL

Setup for Linux and Mac OS

First download and install PostgreSQL :

• Linux users : choose appropriate package from http://www.postgresql.org/ftp/binary/v8.2.4/linux/
• Mac users : download Postgres for Mac from http://sourceforge.net/project/showfiles.php?group_id=133151&package_id=147280

During the installation, a user postgres will be created. This user owns the database system.

Detailed instructions about setting up PostgreSQL can be found here : http://amwdb.u-strasbg.fr/saada/article.php3?id_article=108
Following is a summary of these instructions. We assume an account called johndoe, whose HOME directory is /home/johndoe/ . We also assume that Postgres has been installed in a repertory POSTGRES.
You might want to grant johndoe to execute some commands as postgres by adding the following line in /etc/sudoers.

johndoe     ALL=(postgres) /usr/bin/initdb, /usr/bin/psql, /usr/bin/pg_ctl

You need first to initialize your database. From your home directory, create a directory postgresdata :

mkdir /home/johndoe/postgresdata

and change user owner to postgres :

sudo chown -R postgres postgresdata

This postgresdata directory is a placeholder for Postgres tables.
Then, type the following command to initialize Postgres database :

sudo -u postgres initdb -D /home/johndoe/postgresdata

You are ready to start the DBMS :

sudo -u postgres pg_ctl -D /home/johndoe/postgresdata -l /home/johndoe/postgresdata/postmaster.log start

If something went wrong, have a look in /home/johndoe/postgresdata/postmaster.log to solve the problem.
Otherwise, you can try to connect to the DB using the interactive client psql :

sudo -u postgres psql

Type \q to quit psql.
We will now change Postgres security settings, in order to authorize any user to connect to the DBMS.
First, stop the database :

sudo -u postgres pg_ctl -D /home/johndoe/postgresdata stop

Edit the /home/johndoe/postgresdata/pg_hba.conf file and modify it as follows :

# TYPE  DATABASE    USER   CIDR-ADDRESS   METHOD
# "local" is for Unix domain socket connections only
local   all         all           trust
host    all         all    127.0.0.1/32   trust

Save your changes, and restart Postgres :

sudo -u postgres pg_ctl -D /home/johndoe/postgresdata -l /home/johndoe/postgresdata/postmaster.log start

Try to connect as user postgres on database postgres :

psql -U postgres postgres

Make sure this step works properly before going further.
If yes, create a role for johndoe allowing him to login to Postgres and to create databases (don't forget the final ';') :

 postgres=> CREATE ROLE johndoe CREATEDB LOGIN;
 postgres=>\q

We can now try to create and drop a database as johndoe :

createdb TEST
psql TEST
postgres=>\q
dropdb TEST

If you succeeded, Postgresql setup is fine for using SAADA.

Setup for Windows

Retrieve the Windows installation package from the workshop DVD (in directory CDS/softwares/postgresql/windows) or from http://wwwmaster.postgresql.org/download/mirrors-ftp?file=%2Fbinary%2Fv8.2.4%2Fwin32%2Fpostgresql-8.2.4-1.zip

During the installation, you will be prompted for a password for the postgres user. Please write this password down, as it will be needed in the next steps, to connect to the DMBS for instance.
Once installed, Postgres can be easily started up and shut down from the Postgres item in the Startup-->Programs menu.

Postgres comes along with an administration tool called pgAdmin . Launch pgAdmin, startup Postgres if not already done, connect to the server (right-click on item), and create a role with your Windows username allowed to login and create databases (LOGIN and CREATEDB attributes).
Edit the pg_hba.conf file (accessible from Startup-->Programs-->PostgreSQL-->Config files) and modify it as follows :

# TYPE  DATABASE    USER   CIDR-ADDRESS   METHOD
# "local" is for Unix domain socket connections only
local   all         all                   trust
host    all         all    127.0.0.1/32   trust

Add POSTGRES\bin to the PATH environment variable.

SAADA

SAADA introduction.

Download latest SAADA release corresponding to your platform from http://amwdb.u-strasbg.fr/saada/article.php3?id_article=100
If you already installed SAADA, replace it with the latest version which has new features we will use during this hands-on session.
Installation is pretty straightforward, don't forget to fill the Tomcat path field.
When prompted for the Saada operator name, enter your username, for instance johndoe .

Once you have installed SAADA, use the DEMO to check your setup.

The detailed instructions are available here.

• Make sure the Postgresql DBMS is running
• Run the script ${SAADAHOME}/demo/scripts/buildDemo (buildDemo.bat for Windows users) : this will populate your DEMO database
• Make sure Tomcat is running (TOMCAT_HOME/bin/startup.sh)
• Connect to http://localhost:8080/DEMO
• Click on Database Map and browse the DEMO collection

Publishing test data to the VO

1. Description of test dataset

We will work on a set of 2MASS and DENIS images centered on extended galaxies in the southern hemisphere.

Extract the images from the workshop_CDS_images.tgz file (located in the workshop DVD in CDS/saada_images/data/) :

tar zxvf workshop_CDS_images.tgz

Images will be located in a directory called CDS-data

You will have to ingest this set of images in SAADA, and build a SIAP (Simple Image Access Protocol) service to access these data.

2. Saada configuration

There are a few steps before uploading your data.

2.1 Creation of a SaadaDB

Details on this page

• Create two directories named saadadbs and repository
• Type createdb galaxies to create the database in Postgres (or via pgAdmin under Windows)
• Then create the saadadb
  • Chdir to the directory SAADA_HOME/bin/
  • Run the script newDB (newDB.bat for Windows users)
  • In the GUI, type galaxies for new Saada database and DBMS database name
  • Enter your login name in DBMS user name
  • Set web server host name to http://localhost:8080/galaxies
  • Fill the root directory and repository with the full paths to directories saadadbs and repository
  • Click on Create Saada Database
  • Exit the GUI

2.2 Creation of a Collection

• Chdir to $SAADADBS/galaxies/bin/
• Run the script saadadbgui (saadadbgui.bat for Windows users)
• Go to the Collections tab
• Set Collection name to nir (for Near InfraRed )
• Click the Create button

2.3 Creation of Configurations

We will create 3 different configurations corresponding to the 3 classes of data we will ingest.

1st configuration : cfg_denis_I

• Go to the Images tab, and Creation sub-tab
• Enter cfg_denis_I in the Product name for this configuration field
• Set In collection to nir
• Set Identification of the files matching ... to By filename match (RegExp ) , and enter I.* . This will select all filenames starting with I
• In Optional mapping, fill Keywords used as instance name to 2MUNUMBR
• Select One class for all product files as Class mapping strategy
• Set Class name to denis_I
• Click on Save

2nd configuration : cfg_denis_J

• Go to the Images tab, and Creation sub-tab
• Enter cfg_denis_J in the Product name for this configuration field
• Set In collection to nir
• Set Identification of the files matching ... to By filename match (RegExp ) , and enter J.* . This will select all filenames starting with J
• In Optional mapping, fill Keywords used as instance name to 2MUNUMBR
• Select One class for all product files as Class mapping strategy
• Set Class name to denis_J
• Click on Save

3rd configuration : cfg_denis_K

• Go to the Images tab, and Creation sub-tab
• Enter cfg_denis_K in the Product name for this configuration field
• Set In collection to nir
• Set Identification of the files matching ... to By filename match (RegExp ) , and enter K.* . This will select all filenames starting with K
• In Optional mapping, fill Keywords used as instance name to 2MUNUMBR
• Select One class for all product files as Class mapping strategy
• Set Class name to denis_K
• Click on Save

4th configuration : cfg_twomass

• Go to the Images tab, and Creation sub-tab
• Enter cfg_twomass in the Product name for this configuration field
• Set In collection to nir
• Set Identification of the files matching ... to By filename match (RegExp ) , and enter [0-3].* . This will select all filenames starting with K
• In Optional mapping, fill Keywords used as instance name to STRIP_ID
• Select One class for all product files as Class mapping strategy
• Set Class name to twomass
• Click on Save

We are now ready to ingest the images data.

3. Ingesting data

• Go to the Images tab, and Loading sub-tab
• Select the CDS-data directory where all the image are stored
• Select successively the four previously created configurations, and press Load

4. Acces data with a web form

Before accessing our data from the Saada web interface, we need to deploy the service to Tomcat :

• Run $SAADADBS/galaxies/bin/tomcatdeploy
• Make sure Tomcat is running ( launch $TOMCAT/bin/startup.sh )

• Open a browser, and go to http://localhost:8080/galaxies/
• This is the home page of the web interface of your SaadaDB
• Click on Database Map to visualize all ingested images (with thumbnails)
• In the map, click on a collection to see corresponding images
• If clicking on the Det green button in the first, you will access to the details of the corresponding image
• Clicking on the VO cyan button to visualize the image in the Aladin applet
• You may also download the FITS file by clicking on the orange button

5. SIAP in VO interface

• From your SaadaDB home page (http://localhost:8080/galaxies/), click on VO portal
• This web form allows you to perform SIAP queries on images data previously ingested
• In the SIAP queries panel, set RA to 13 36 55, DEC to -29 49 52 , Collections to [nir] and press Submit SIA query . The result is a VOTable file conforming to the SIAP standard, describing the list of available images for this region, and the URL to access them.
• You have also the possibility to restrict your query to a given class, by setting for instance Collections to [nir](twomass)

6. Simple Image Access in Aladin

We will now see how to access our SIAP service from a VO tool : Aladin.

• Open the SaadaDB home page ( (http://localhost:8080/galaxies/)) in a browser, and click on VO portal
• In the SIA query panel, set Collections to [nir], choose register and press Submit SIA query
• The result is a GLU record, a description of our service which will allow Aladin to query it
• Select in your browser lines starting with a '#', and copy/paste them to a new text file named myservers.txt
• Start Aladin by typing java -jar Aladin.jar -beta -glufile=myservers.txt. The -glufile parameter points toward a file describing additional servers
• Once Aladin started, click on Load
• In the right most column, click on Saada SIA-->Saada SIA galaxies
• Set the target to M83 for instance and press the Submit button
• The list of available images for this region appears in Aladin
• Click on one of the item of the list, and press LOAD to load the image in Aladin
• You have successfully queried your SIA service from Aladin !

In the next section, we will learn how to register our SIAP service, so that it can be automatically discovered and accessed from VO client tools, without prior knowledge of its existence.

7. Registering in the test registry

• Open the siap.xml template file in a text editor. We will modify this template to adjust it to our service
• Change the title
• Change the shortName
• IMPORTANT : change the identifier to ivo://org.astrogrid.demo/saada/siap/your-last-name . This identifier has to be unique.
• IMPORTANT : change the accessURL with the base URL of your SIAP service. Replace localhost with the IP of your laptop (find it by typing ifconfig ). The accessURL should look like http://131.176.165.61:8080/galaxies/sia?collection=[nir]&
• Once the template file has been modified, point your browser to http://beacon3.star.le.ac.uk:8080/astrogrid-registry/
• Set current contract to 1.0
• Click Enter resource
• Enter login=astrogrid, password=W6G117
• We will use the Upload from a local file option. Check the Validate radio button, enter the path to your modified siap.xml file, and press Upload
• If no error message appears, your service has been successfully registered. It should appear now in the list of resources available from the Registry browser (Browse link in Investigate menu)

8. Consume service in Aladin

Once your service has been registered in the workshop test registry, VO client tools will be able to automatically discover it.

• Launch Aladin by typing java -jar Aladin.jar -beta
• Click on Load
• Go in the All-VO tab
• Press on the Detailed list button
• The new window lists services entries coming from the NVO registry
• Click on Query workshop registry and click on Refresh
• The workshop registry will then be queried, and discovered services will appear with a pink background
• Unselect all services, then select SIAP services of your interest (for instance the one you registered previously)
• In the Aladin server selector, set the target to NGC4945, and press Submit
• Load one or more images of your interest

Building your own SIAP service

Repeat above steps with your own image data if you wish!

  Inclusion and usage of characterisation metadata inside the SaadaDB "galaxy" database

1. Introduction

Characterization data model IS NOT currently used by the SIA protocol, in contradiction with the situation of SSA which is fully benefiting from the Spectrum/char DM description.

We choose to bypass this restriction by linking in Saada a catalog describing the images characterization to the Image collection itself....

2. The characterization class for 2MASS images:

2.1 Building Characterization file in VOTABLE

Utypes are pointers to a specific datamodel. Saada doens't manage utypes yet; However we can choose to use utypes strings as names in our catalog.

Running the perl script SCR2MASS in the CDS-data directory is creating in the Charac subdirectory a VOTABLE file containing characterization information for each fits file ... This script can be adapted to DENIS headers or your own data to create a characterization table for each of your image files.

2.2 Downloading the xml files if 2.1 doesn't work.

Extract the Charac directory from Charac.zip and put it in the CDS-data directory

2.3 Creation of a char Configuration

We will create a configurationscorresponding to the characterization class of data.

• call again the saadadbgui of galaxy database
• Go to the Sources tab, and Creation sub-tab
• Enter cfg_twomass_char in the Product name for this configuration field
• Set In collection to nir
• Set Identification of the files matching ... to By filename match (RegExp) , and enter .*.xml . This will select all filenames starting with xml extension
• In Optional mapping for tables, don't fill anything
• In Optional mapping for table entries, just put Ra and Dec in the two first fields. Saada will find itself its way with the UCDs.
• Select One class for all product files as Class mapping strategy
• Set Class name to twomass_char
• Click on Save

2.4 Loading char Files

• Go to Sources Loading
• choose cfg_twomass_char for configuration
• open CDS-data/Charac directory where all the Characterization tables are.
• click on the load button.

2.5 looking at Characterization information

• Don't forget to run tomcatdeploy in the bin sub-directory of the galaxy database.
• Look again at Galaxy service into your Web Browser. Reload database map.
• There are now instances in the two table columns (Table and Entry). Click

on the Entry and obtain the table. Look at the details of some rows.

• You are now facing a specific implementation of characterization metadata for one of the TWO MASS images. The DataSetName field gives you the name of this file.

2.6 Going from characterization information to characterized Images

• Saada provides a Relation facility which should allow to hook the characterization record to the associated Image Record. As it is not working today, we will make this link by hand. For this pick up the content of the "DataSetName" field.

Go to SaadaDB Home and choose the HTML editor. Select "Images" for the category of searched data Go to the "Additional constraint on Saada Parameters". Look for the Saada attributes and select "Product File" . Paste the file name you picked up previously in the value1 field Clik on "Reset Query" on the bottom of the page and then on the Apply button of the "Additional constraint on Saada Parameters" section. An SQL query is then created. Click on the red "Submit query button" and obtain your 2MASS file.

-- ThomasBoch - 21 Jun 2007

• SAADA_introduction.pdf: SAADA introduction

• siap.xml: SIAP template XML registry Resource