Planning your Network
One of the benefits of using DataTurbine is the flexibility it provides in sensor deployments. DataTurbine is light weight and portable and can be used on anything from a single computer and sensor, to a massive multi server network covering dozens of sites and hundreds of sensors.
Before attempting to implement DataTurbine for your project I highly recommend you read this section and plan out an implementation that would be suitable and scalable for your particular application.
The Basics Blocks
- Server – Refers to a DataTurbine server (an instance of rbnb.jar)
- Source – a program that injects data into a DataTurbine server (Often referred to as an ‘on-ramp’)
Ex: Sensor, Camera, Application that generates data
- Sink – a program that pulls data from a DataTurbine server (Often referred to as an ‘off-ramp’)
Ex: Real Time Data Viewer, Web Page, Database off-ramp
The fundamental building blocks of a DataTurbine deployment are servers that manage data streams, sources that write data, and sinks that read data. Each of these blocks are separate lightweight program and each can run on any computer anywhere connected via a network or the internet.
Some applications can function as both sources and sinks. An example of a hybrid sink/source would be an analysis program that reads data performs some kind of summary statistics and then adds a derived channel back to the server. Hybrid sources do not change the data itself but can provide metadata and derived data for a given deployment.
Connecting Them Together
The pieces are connected via local networks or internet using the IP address of the device they are running on. The default port for DataTurbine is 3333, but this can be reconfigured (for example in order to run multiple servers). The sources and sinks communicate with a server via TCP on this port so correct network permissions and port forwarding have to be in place.
Alternatively, data can be pulled into the DT server via HTTP using the HttpMonitor application (which will typically not require any firewall changes as port 80 outbound access is common).
Multiple servers can be used for buffering and analysis. A common configuration style would use multiple servers, one at each field site, and a central server that is open to public access. Servers can mirror the data either programmatically or manually through the admin utility. When a mirror copies the specified data channels to another server.
Push Mirror: A push mirror turns a server into a source pushing data to another server as it becomes available.
Pull Mirror: A pull mirror acts like a sink it forces a server to read data from another server
Push mirrors are typically used when collecting data. Each field station would push its data to a central server. Pull mirrors on the other hand tend to be used for data sharing. An institution would pull the data from a public server to use in its analysis and visualization.