Developer’s corner

VOSpace

Introduction

From the VOSpace document : “VOSpace is the IVOA interface to distributed storage. It specifies how VO agents and applications can use network attached data stores to persist and exchange data in a standard way.”

In the past, we have worked on VOSpace 1.15 version associated with iRODS as backend. This version was SOAP oriented and is not maintained. We are now working on the implementation of the IVOA VOSpace 2.O As the 2.0 version is a working draft we cannot provide a stable release but if you want to make some tests it is possible to try a beta release with iRODS or a local file system as backend.

The source code of the full VOSpace version 2.0 toolkit will be provided after the IVOA recommendation process.

We had also worked on a SOAP implementation of the VOSpace 1.15 recommendation but we provide no support and no maintenance for this version.

Our VOSpace 2.0 toolkit is independant from the storage system and it is possible to write an interface for a large kind of it (iRODS (provided), local file system (provided), DBs, etc.)

CDS efforts

Technical side

VOSpace 2.0 toolkit is developed in Java and is based on Restlet.

Why a flexible VOSpace implementation ?

Providing an access to a VOSpace or giving an access to data through the VOSpace protocol is not the same ! In the first case you must assume the avaibility of the user data. In the second case it is more an access to existing data stored through an another way than the VOSpace protocol.

The VOSpace 2.0 toolkit provides a complete VOSpace over iRODS for example. But it is also possible to use it as a layer over any kind of storage system.

We have separated the VOSpace layer from the backend layer which connects to the storage system.

Backends

Why a flexible backend ?

Main data centres will probably develop their own VOSpace implementation which will be mainly adapted to their own data storage system. Other people will need to implement quickly a VOSpace layer over their specific data storage system. Our VOSpace 2.0 toolkit is developped for our own needs (one way to access data stored with iRODS iRODS) but it is flexible concerning the backend. It means that it will be possible for example to implement a VOSpace layer on any kind of data storage system.

How to write a new backend ?

We propose a GenericBackend interface which must be implemented. It enables the VOSpace layer over the data storage system. VOSpace implementation metadata must also be fixed (server name, port, properties, views, etc.).

LocalFileSystemBackend and iRODSBackend will be delivered with the implementation release.

Ongoing work

  • tutorial about how to implement a backend
  • packaging
  • linking to last iRODS release

References

3 presentations in other communities done in 2009

  • In December 2009 at Queen Mary University of London, Interoperability of Digital Repositories meeting
  • In June 2009 at a CNES workshop about data preservation
  • In February 2009 at a joint IN2P3 – DICE iRODS workshop

iRODS main page illustration points to the VOSpace-iRODS article https://www.irods.org/

IVOA Reference

The IVOA VOSpace last recommendation is available at http://www.ivoa.net/Documents/VOSpace

The following text has been extracted from the IVOA VOSpace document :

(you can refer to the whole document for more information about VOSpace)

What is VOSpace ?

VOSpace is the IVOA interface to distributed storage. It specifies how VO agents and applications can use network attached data stores to persist and exchange data in a standard way.

A VOSpace web service is an access point for a distributed storage network. Through this access point, a client can:

  • add or delete data objects
  • manipulate metadata for the data objects
  • obtain URIs through which the content of the data objects can be accessed

VOSpace does not define how the data is stored or transferred, only the control messages to gain access. Thus, the VOSpace interface can readily be added to an existing storage system.

When we speak of “a VOSpace”, we mean the arrangement of data accessible through one particular VOSpace service.

Each data object within a VOSpace service is represented as a node and has a description called a representation. A useful analogy to have in mind when reading this document is that a node is equivalent to a file.

Nodes in VOSpace have unique identifiers expressed as URIs in the 'vos' scheme, as defined below.

VOSpace 2.0 does not introduce any new functionality to that already offered by prior (SOAP-based) versions of the interface (VOSpace 1.1) but defines a RESTful binding for the interface.

How does it works ?

A typical use case for VOSpace is uploading a local data file to a remote VOSpace service. This is a two-stage process: creating a description of the data file (representation) in the VOSpace including any metadata (its properties) that they want to associate with it, e.g. MIME type and defining the transfer operation that will actually see the data file bytes uploaded to the VOSpace service. The order of the processes should not matter. The user may want to create the representation first and then perform the transfer or transfer the bytes first and then update the representation with the appropriate metadata.

Let's consider the first sequence: the user provides a XML description of the data file which they HTTP PUT to the appropriate VOSpace URI - this will be the HTTP identifier for the data file in the VOSpace, e.g. http://nvo.caltech.edu/vospace/myData/table123. The description will resemble this:

<node xmlns=“http://www.ivoa.net/xml/VOSpaceTypes-v2.0

   xmlns:vost="http//www.ivoa.net/xml/VOSpaceType-v2.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
uri="vos://nvo.caltech!vospace/mytable1"
xsi:type="vost:UnstructuredDataNode">  
  <properties> 
      <property uri="ivo://ivoa.net/vospace/core#mimetype">text/xml</property>     
  </properties> 

</node>

The service will reply with an amended version of the representation containing service-specific details in addition to the information supplied by the user. These will include data formats that the service can handle for the type of node created in the VOSpace, third-party interfaces (capabilities) to the data that the service offers and system metadata.

The user will then describe the data format (the view) they want to use in uploading the file, e.g. VOTable, and the transport protocol (the protocol) that they want to employ to upload the file, e.g. HTTP PUT. This will result in the HTTP POSTing of a XML description of the transfer request to the appropriate VOSpace URI, e.g. http://nvo.caltech.edu/vospace/myData/table123/transfers. The description will resemble this:

<transfer xmlns=“http://www.ivoa.net/xml/VOSpaceTypes-v2.0

	xmlns:vost="http://www.ivoa.net/xml/VOSpaceTypes-v2.0"
	xmlns:xsd="http://www.w3.org/2001/XMLSchema"  
	xmlns:xsi="http://www.w3.org/2001/XMLSchema?instance"> 
  <direction>pushToVoSpace</direction> 
  <view uri="ivo://ivoa.net/vospace/core#votable"/> 
  <protocol uri="ivo://ivoa.net/vospace/core#http?put"/>  

</transfer>

The service will reply with the URL that the user will HTTP PUT the data file to, e.g. http://nvo.caltech.edu/bvospace/myData/table123/transfers/147516ab. The user will then use a regular HTTP client to transfer (PUT) the local file to the specified endpoint. This illustrates an important point about VOSpace - it is only concerned with the management of data storage and transfer. A client negotiates the details of a data transfer with a VOSpace service but the actual transfer of bytes across a network is handled by other tools.

Similarly, when a user wants to retrieve a data file from a VOSpace service, they will specify the data format (view) they want to use in downloading the file, e.g. VOTable, and the transport protocol (the protocol) that they want to employ to download the file, e.g. HTTP GET, and HTTP POST a XML description of this transfer request to the appropriate VOSpace URI - the transfer URI for the node in the VOSpace, e.g. http://nvo.caltech.edu/vospace/myDataNode/table123/transfers. The description will resemble this:

<transfer xmlns=“http://www.ivoa.net/xml/VOSpaceTypes-v2.0

	xmlns:vost="http://www.ivoa.net/xml/VOSpaceTypes-v2.0"
	xmlns:xsd="http://www.w3.org/2001/XMLSchema"  
	xmlns:xsi="http://www.w3.org/2001/XMLSchema?instance"> 
  <direction>pullFromVoSpace</direction> 
  <view uri="ivo://ivoa.net/vospace/core#votable"/> 
  <protocol uri="ivo://ivoa.net/vospace/core#http?get"/>  

</transfer>

The service will reply with the URL for the user to use, e.g. http://nvo.caltech.edu/vospace/myDataNode/table123/transfers/3df89ab4. The user can then download the data file by pointing an HTTP client (e.g. web browser) at the specified endpoint.

European projects support

A part of the developments has been done in the frame of the European EuroVO VOTECH and EuroVO AIDA projetcs. Presentations have been done in the frame of the EuroVO ICE project.

vospace.txt · Last modified: 2012/05/10 18:08 by administrator

Thanks for acknowledging the CDS developer’s resources (libraries, source code, etc.)

© UDS/CNRS

Contact