Big Data

Generation of big data​

Due to the high sampling rate of 50 MSamples/sec. the QASS measuring systems are capable of generating, storing, processing and analyzing a large amount of data to generate, store, process and analyze in parallel. The processes are recorded and made available down to the microsecond. In combination with the SQL database of the measuring system, the possibilities of sensor fusion and the use of Python, unimagined possibilities arise in the possibilities in dealing with big data.​


Structure-borne sound spectrogram: 12.5 million data points/second

 

Data acquisition

  • The QASS measuring card integrated in the Optimizer4D can digitize up to 4 analogue inputs, whereby the maximum sampling rate is divided between the number of channels. 
  • In addition to the 4 16-bit channels, there is a 24-bit channel with with a maximum of 4MSamples/s is available.
  • All channels can be subjected to an FFT in real time so that a time and frequency data stream can be made available for each channel at the end.
  • For the FFTs, 1024 measured values can be transformed into a spectrum every 41 µs, resulting in spectrograms with a high temporal resolution of 25 kHz and a frequency resolution of 512 bands.
  • With the 24-bit data stream, the high amplitude resolution comes first, but we still like to convert at 4MSamples/s in order to record every change.
  • Here, the FFTs can calculate a new spectrum every 32 µs.​

 

Data sources and transfer methods​

  • The QASS structure-borne sound sensors in combination with our broadband preamplifiers provide measurement data up to over 100 MHz, i.e. sufficient bandwidth for our FFTs.
  • The QASS Barkhausen sensors, or µMag sensors, have integrated preamplifiers that are tuned primarily to the 24-bit converter and 4MSamples/s.​
  • The data rate of the spectrally analysed magnetic field data is usually 32 MBytes/s.
  • In addition, the µMag sensor has integrated measuring amplifiers for measuring the current and voltage of the excitation coils, the temperature in the measuring tip, as well as field measuring coils for measuring the excitation field strength.
  • This data is transferred to the measuring device via USB.
  • Laser line scanners provide high-resolution distance data, e.g. in the micrometre range, with a data rate in the gigabit range. This data is usually read in via TCP/IP. 
    The same applies to various industrial cameras.
  • Video data streams can be linked and correlated with the time series data or with the FFT-transformed data.​
  • PLCs and other fieldbus systems can be addressed via OPC UA, for example, and their statuses can be included.
  • Databases: In addition to sensor data, databases can be queried and the results integrated into the data stream.
  • Files can serve as data suppliers.​
  • Virtual files can be created as a continuous data source.

Sensors/Fusion

 Example: Detection of surface defects using data from a laser line scanner, 1000 points/line, 4000 lines per second




Example: Detection of individual components in bulk material, point cloud from TOF camera (time of flight) 

Data processing

Database logs are created for each recording, time stamps, measurement parameters, compression settings and any results of pre-evaluations. results of pre-evaluations.

In addition to sensor data, databases can be queried and the results integrated into the data stream. PLCs and other fieldbus systems can be addressed via OPC UA and their statuses can be included. If data is generated continuously, the system can save such external sources in corresponding data streams; if data is generated sporadically, it can be logged as process events in the database.

Depth memory

  • Fast and large memory modules (write rate >400MB/s) complement the fast data sampling.​
  • 2.5" SSDs and M.2 modules can be supplemented with compatible modules at any time.
  • 2TB - 8TB SSDs with enormously high rewritability rates to withstand long-term measurements and long-term storage for years. 
  • 9100 TBW (Terrabytes written) means a continuous storage duration of 2106 days at a permanent data rate of 50MB/s.​

Current hardware​

  • The Optimiser4D is equipped with 2.5 TB flash memory as standard.
  • 512 GB system memory in the form of a memory card and​
  • 2 TB as 2.5" SSD as high-availability storage with a total write rate of total write rate of approx. 5200 TBW.​
  • The systems can also be equipped with 8 TB SSDs in 2.5" format with a write rate of approx. 9100 TBW.
  • Each Optimiser4D can be operated with at least 2 SSDs can be operated.
  • Network storage can also be used as backend storage can also be used as backend storage.