How and Where To Use DataTurbine

If you’re new to DataTurbine, you’re perhaps wondering how and where it’s useful, and if it’d be useful for your own work. This document aims to answer those questions for you, and give you some ideas. Before I dive into what it cando, let’s focus a bit on systems you might have encountered, so that you know what’s different. How is DataTurbinedifferent than other systems:

  • The ‘network ring buffer’ idea lets you separate data sources, and also lets you mix and match different rates. Unlike an Enterprise Message Service (EMS) system, clients can get old or new data, as required, and can look at data more than once.
  • Unlike a database, you can subscribe to a data feed, and get efficient, low-latency propagation with high throughput (tens of megabytes per second, tens to hundreds of clients)
  • Unlike a filesystem, there are no locks to worry about, byte ordering issues, but you can still random-access address the data streams.
  • Unlike a simple TCP or multicast connection, you can have different rates. Sources can ‘burst’ data, that clients can let pile up and process as able and interested. Clients can also choose which sources to monitor, so they needn’t waste bandwidth seeing everything.

Since DataTurbine is Java-based, it runs anywhere there’s a JVM, from Gumstix to 64-bit servers. If your system can’t run Java, you can still stream data using a proxy (several are provided, contact us for details) or via DataTurbine’s WebDAV filesystem interface. Our best display client is called RDV, and its a rich-client Java application. There are web interfaces for data, but none are as capable, so in general Java is useful-to-essential for serious use of DataTurbine. DataTurbine is more useful in applications that are

  • Distributed
  • Heterogenous
  • Collaborative

Let me explain what I mean.

  • Since DataTurbine adds a flexible-length buffer and access to your design, it adds capabilities like TiVo-style browsing, replay, rewinding, and more. However, if everything is simple, local, and point-to-point then perhaps DataTurbine might not be helpful.
  • Since its a TCP/IP-based network application, it makes a wonderful abstraction layer for heterogenous devices; once data is sent to DataTurbine the underlying hardware and software are totally abstracted. The clients don’t need to know or care. However, if you just have one device type, then abstraction is less useful.
  • I posit that DataTurbine is most useful when you’re collaborating on science, especially with remote users. It lets anyone (who you permit) view data, watch video, annotate an ongoing experiment and zip through data with ease. All of those capabilities are great for a single lab, but they’re superb when, say, you want to view the experiments’ progress from your iPhone, or let your remote collaborator control the mass spec from their office.
  • DataTurbine’s ability to mix and match numbers and video, all synchronized, is quite valuable. You can not onlysee the video feed, but you also see what was happening on the other sensors or feeds when it happened. Automatically, all the time, no extra effort required.

Here are some examples where DataTurbine is a big win:

  • The Network for Earthquake Engineering and Simulation(NEES) uses it for distributed experiments, often spanning thousands of miles and several days. For example, the structural lab at UMN collaborates with researchers in Puerto Rico, who can observe, annotate and control without frequent flyer miles to Minnesota. UMN uses LabWindows-based data acquisition systems, Axis network cameras, streaming audio servers (custom code based around Windows and the Java Media Framework), as well as Java-controlled consumer digital cameras on remotely-controlled imaging towers.
  • We’re using DataTurbine to demonstrate streaming numeric & video data from the Santa Margarita Ecological Reserve and a coral reef in Taiwan. This uses LabVIEW CompactRIO data acquisition, a variety of sensors, numerous network video systems and at least four different operating systems, and that’s before you include the network gear!
  • Insight Racing is using it for video onboard their autonomous car. Video from Axis 206M 1.3 megapixel cameras streams to an onboard Linux-based video server for processing over Ethernet.
  • NASA uses RBNB for telemetry as part of their “Intelligent Network Data Server” or INDS at their Dryden Lab.

Here are some more ideas for places where DataTurbine is a useful capability:

  • Stream processing and event detection. You can easily interface to DataTurbine from Matlab, givng you the ability to write programs that process and analyze streams of data in real time. Think video processing, complex algorithmic event detection, anything you can do with the vast cornucopia of Matlab code or toolboxes. Output can be sent right back to DataTurbine as another data stream, viewable by anything in the system.
  • Event markers and annotation. We have a capability to annotate the DataTurbine that we call ‘Event markers.’ These are timestamped bits of XML, stored in the DataTurbine as a text channel and part of the record. You can use them as notes or annotation, either human or programmatic, and are great for situational knowledge or just understanding an experiment afterwards. For example, ‘hydraulics now online,’ ‘sensor X on box Y is faulted and offline’ or ‘I think we just saw beam failure on cantilever 3’ are all examples of possible markers. These can be generated and viewed in RDV or your own interface.

I hope this has helped give you some ideas of where DataTurbine wins. DataTurbine is under the Apache 2.0 license, but we’d appreciate greatly the courtesy of an email telling us how you’re using it, any feedback or features you’d like to see.


Open Source DataTurbine Initiative © 2016 Frontier Theme