TWiki
>
EuroVODCA Web
>
DcaJune2008Workshop
>
DcaJune2008ESOMEx
(26 Jun 2008,
BrunoRino
)
(raw view)
E
dit
A
ttach
---+ Metadata extraction: from FITS files to databases with MEx Note: this hands-on session runs only in the morning; in the afternoon you will deploy [[DcaJune2008ESADALToolKit][DALToolKit]] to create data access services using your database. ---++ Abstract Publishing images and spectra in a VO-compliant format is a two step procedure: first making sure all data is valid and described in a common, homogeneous way, and then providing query interfaces to the data, using the validated data descriptions. MEx is a tool to aid the first step (_metadata ingestion_) where the data descriptions are extracted from FITS headers into a data repository. A set of required and optional metadata are defined for each type of data, and are extracted from the FITS files through mapping rules. The data repository (most commonly a database) separates the metadata ingestion from the second step, the query service. The goal of this session is to load a simple !MySQL database and deploy [[DcaJune2008ESADALToolKit][DALToolKit]] for providing SIA and SSA interfaces to the data. Participants should bring their own data to ingest, data of which they have a good understanding (of the meaning of the FITS keywords). When a custom database structure should be supported, a small amount of programming might be required. ---+++ External References ---+++ Advisors (ESO) * Remco Slijkhuis * Bruno Rino * Jean-Christophe Malapert ---+++ Software Requirements * [[DcaJune2008Software][From workshop requirements]] * Java 1.5 * !MySQL * tomcat * More * N/A --- ---++ Download http://www.euro-vo.org/dcaworkshop2008/HandsOn/mex_daltoolkit/ ---++ Bug report and feature requests http://mex-eso.blogspot.com/ ---++ Session outline Publishing data in the VO as a two-step procedure: * gathering metadata (knowledge of the data, e.g. FITS keywords) in a homogeneous fashion: MEx _(ESO)_ * building a (web) service that searches data using the metadata and allows access to the data: !DALToolKit _(ESA)_ The session is organized in two parts: * a first one using ESO data * a second one using your own data. ---+++ 1st run: demo data, demo database * (one time only) setup a database * create SQL tables * gathering meta-data into the database * map FITS keywords to "concepts" AKA model items: Mapping Editor * get a sample FITS header * define mappings for (at least) the required metadata * test mappings against sample FITS file header * ingest metadata into database: MEx * execute mappings against all the FITS files * build service * MEx creates default !DALToolKit configuration ---+++ 2nd run: your data, your database * your data: * the Mapping Editor, revisited * several types of data in one package: Catalogue Builder * your data definition * model items configuration * your own mapedit * your database: * MEx scripting * adapt !DALToolKit configuration ---+++ Steps for using Simple Image Access Protocol ---++++ Setup a database make sure the database is running and connect yourself to the database server as "superuser" <verbatim> mysql -u root -p </verbatim> Now, create a database: <verbatim> create database esodata; </verbatim> And finally, create a SIA table in the esodata database: <verbatim> use esodata; source samples/db/sia.sql </verbatim> Check the table structure: <verbatim> show tables; desc SIA; </verbatim> ---++++ get a FITS header Extract a FITS header using the following command: <verbatim> java -jar lib/fitshead-1.0.jar -x 0 samples/images/GOODS_ISAAC_03_H_V1.5.fits > sampleheader.txt </verbatim> ---++++ define/test mappings * Open your favorite browser and go to the mapping editor interface: [[http://vops1.hq.eso.org:8080/mapedit][http://vops1.hq.eso.org:8080/mapedit]] * upload the model item list specific for this workshop: config/modelitem_definitions.txt * select data type: image.reduced * upload your fits header using *upload (text)* * click "Validate" to use the default rules * edit mapping rules that generate errors or warnings * fix errors * *Note: min and max RA and Dec are mandatory for !DALToolKit* * when all are valid, download mappings file ---++++ Ingest set-up a directory for sharing in tomcat: unzip samples/files.zip into $CATALINA_BASE/webapps Ingest data in the database: * edit mex configuration config/mex-daltookit.properties (e.g. db password, folder to copy data to) * run mex on files + mappings file <verbatim> java -jar mex-java.jar --type SIA -d samples/images -m samples/mappings/isaac.txt </verbatim> ---++++ build service * edit !DALToolKit configuration if needed * deploy !DALToolKit service <verbatim> cd DALToolkit vim build.properties.local ant deploy SIAP-v1.0-mex </verbatim> * visit service * http://localhost:8080/SIAP-v1.0-mex/ ---+++ Tips for using Simple Spectral Access Protocol v0.1 * Database creation script: <verbatim> samples/db/ssa-0.1.sql </verbatim> * Example FITS header extraction: <verbatim> java -jar lib/fitshead-1.0.jar -x 0 samples/spectra/esa/3C273_0136550101.fits > sampleheader.txt </verbatim> * Data type for mapedit: * spectrum.1d.ssa01 * Ingestion using provided mappings: <verbatim> java -jar mex-java.jar --type SSA-0.1 -d samples/spectra/esa -m samples/mappings/esa-0.1.txt </verbatim> * !DALToolKit service config: =SSAP-v0.1-mex= * !DALToolKit service: * http://localhost:8080/SSAP-v0.1-mex/ ---+++ Tips for using Simple Spectral Access Protocol v1.0 * Database creation script: <verbatim> samples/db/ssa.sql </verbatim> * Example FITS header extraction: <verbatim> java -jar lib/fitshead-1.0.jar -x 0 samples/spectra/fors2/GOODS_FORS2_GDS_J033202.99-274301.2_904509_V2.0.fits > sampleheader.txt </verbatim> * Data type for mapedit: * spectrum.1d.ssa10 * Ingestion using provided mappings: <verbatim> java -jar mex-java.jar --type SSA-1.0 -d samples/spectra/fors2 -m samples/mappings/fors2.txt </verbatim> * !DALToolKit service config: =SSAP-v1.0-mex= * !DALToolKit service: * http://localhost:8080/SSAP-v1.0-mex/
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r16
<
r15
<
r14
<
r13
<
r12
|
B
acklinks
|
V
iew topic
|
Ra
w
edit
|
M
ore topic actions
Topic revision: r16 - 26 Jun 2008
-
BrunoRino
EuroVODCA
Log In
or
Register
EuroVODCA Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
Help
Webs
EuroVOAIDA
AidaVOWS2009
VOSchool09
VOSchool10
EuroVODCA
Main
Sandbox
TWiki
Links
Euro-VO
Euro-VO AIDA
IVOA
VOTech
Communication Network Development in FP6
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback