WebScan / CloudTurbine FAQ

The “How it Works” graphic on the CloudTurbine site is great, but can you clarify a few CloudTurbine API questions?

– If there is doc somewhere that I’ve missed (e.g. CTserver, CTreader javadoc)?

CloudTurbine at this point (3/16) is still in a prototype development stage, with complete documentation pending upcoming efforts.

– Do I understand correctly that
– CT data must follow the convention:
    folder named for CT source (e.g. FOCE-PH)
        folder(s) named with timestamps (Unix epoch milliseconds)
             data file(s) named with channel name


– zip files are the mechanism for binning CT source data by time
– zip files are named by the first time stamp they contain
– the binning interval doesn’t have to be constant
– zipping the data is optional


– zipping data improves CTserver throughput (CTserver uses ZIP file name to search for data, instead of traversing the (unzipped) timestamp folders)?

Zipping is probably no faster (perhaps less so) than “loose” files. What zipping mainly does is clump lots of little files together such that file-sharing services can keep up with them. E.g. a file sharing service may have no problem uploading one 100KB file per second, but wouldn’t come close to dealing with 1000x100B files per second.

– Does CloudTurbine (CTserver) work with any arbitrary data format (text, binary)?

Yes, the fundamental concept is to write, transport, and read time-series data in files. The content of the files is whatever files can hold. That said, various viewers only understand certain types of data.

– is the requirement that the data channel files are in folders named with a timestamp?

Per above, yes.

– WebScan only works with selected numeric, audio and image formats

Yes, WebScan is a viewer that gets data from CloudTurbine files via the CTserver application. Note that CTserver itself is just one of many potential CloudTurbine applications, one that converts CT files to HTTP format. Note also that WebScan is compatible with DataTurbine via its “WebTurbine” HTTP interface (they share a compatible HTTP syntax by design).

– Does CTserver accept direct CSV data input (from applications)?
– What is the record format? (is there javadoc for CTserver API?)
– what is the header format (“DateTime”, channel names)?
– Is the time field required to be first for CSV input?
– What date/time formats are supported?

A complete answer to the above would require some extensive documentation that does not yet exist. The CTlogger and CT2CSV apps can put and get CSV data to/from CloudTurbine respectively. Then in turn CTserver could convert CT to HTTP to Webscan.

In general, CTserver utilizes the CTlib.jar libary (CTreader) to fetch data. It “understands” the following formats:
– raw binary (bytearray)
– binary numeric per file suffix rule (.f32, .f64, .i32, etc)
– ascii numeric parsed as a double
– image data (.jpg only)
– audio data (.pcm, .wav format)

– Does CTserver provide CSV file output (to applications) by requesting a list of channels? one channel at a time?

One channel at a time. Not an inherent limitation of CT, but difficult to handle via HTTP (and also limited to be compatible with DataTurbine/WebTurbine).

– Is the data channel txt record format a signed double precision number (double)?


– The data channel file must be in a folder named with a timestamp, and the filename == channel name?


– the the “txt” extension required? (I’m guessing it isn’t based on what I’ve seen)

No. In fact, I think a .txt suffix would indicate a string (non-numeric).

– What image/video/audio formats is WebScan able to display (jpg, cpm, gif?, tif?, wav?, mp3?…)

jpg, wav (more eventually would be nice). Can use ffmpeg third party utility to convert perhaps.

– Can applications write .bin format files directly, or must they use CTwriter?

Applications are free to write their own files, so long as they abide by the Time/Name rules. The image-writing script is an example via simple shell script. Reading is more complicated than writing (e.g. to merge multiple records). Right now the CTlib.jar in Java is the only CT file-API, but eventually the plan would be to support Python, C# etc.

Open Source DataTurbine Initiative © 2016 Frontier Theme