DataTurbine Sink

Introduction

A DataTurbine Sink (also refereed to as an ‘off-ramp’) is simply a program that takes data from a DataTurbine Server and utilizes it, for example brings it up in Matlab or Real-time Data Viewer or puts it into a relational database or file for permanent storage.

Just like a source, a sink runs independently from the server as a separate application and uses the network to communicate. It can run on the same machine as the server or on a machine across the world.

The Sink’s Perspective

From the sink’s point of view it no longer needs to know where the data came from or how it got there. It can query all the sources and channels to find out what is available or specify a single channel via its name and name of its source.

The data is heterogeneous and the sink could access any type of data seamlessly. It makes the decision on how to display and interpret the data via its data type (byte array, 32-bit float, 32-bit int, etc) as well as theĀ MIME Type specified by the sink.

A sink can issue a request to pull data from the server in a timeframe. A sink could also subscribeto a specific set of channels getting data as it becomes available.

Example: For example a sink could get a listing of all the sources available on a server pick only the temperature channels, perform some analysis and based on the result bring up the images for the corresponding channels at significant time indexes

Common Types of Sinks

  • Viewer: An application that can be used to access and interact with the streaming data
    Ex: Real-time Data Viewer (RDV), Google Earth, etc…
  • Web Server: An application that serves the data as web content for public display
    Ex: Graphs on a public web site
  • Analysis: Takes the data and performs some kind of manual or automated analysis
    Ex: Matlab, R, ESPER, etc..
  • Export: Exports the data into a file or set of files for distribution or integration
    Ex: CSV files, Excel, etc…
  • Storage: Permanent storage in a database or as a series of files.
    Ex: Storage in a relational database
  • Other: Easy to code any kind of sink that utilizes the data

Practical Example (Continued):

Going back to the example used in the source. Imagine a simple meteorological tower that measures temperature and humidity on top of a hill. Nearby is a field station that is also measuring temperature. We put this data into DataTurbine on a laptop at the field station and now want to view it and make sure that it is placed in permanent storage.

  1. Start a DataTurbine server on the laptop (rbnb.jar)
  2. Start a source on the laptop reading data from the meteorological tower
  3. Start a source on the laptop reading data from the field station
  4. Start a sink to view the data as it is collected in real-time. In this case we will use Real-time Data Viewer (RDV)
  5. Start a sink to put the data into permanent storage in a MySQL database.

Our laptop would now have five independent lightweight programs running (1 server, 2 sources, 2 sinks). We will probably keep the server, sources, and the permanent storage sink running at all times. But we will start and stop the viewer sink as we need it.

Now we have a very basic but complete deployment running. But we are not sharing the data and not really utilizing the power of a real-time system (Aside from viewing the data as it is collected). Fear not this will be discussed in further sections as we build on our example.

Open Source DataTurbine Initiative © 2016 Frontier Theme