ILC - International Linear Collider

Accessing DAQ Data using Matlab

This is a revised description of how to retrieve and use DAQ data from within Matlab. The original WikiPage introduction can be found here.

Getting started

To get started, you will need the following:

These two example scripts are functional, so download them into your directory on the DESY network, start matlab, and type 'daq_example_script'.

Examine the scripts and note how daq_example_script.m calls the function daq_fetch_v2() with the following input arguments:

  • Tstart - the start time of the desired time frame
  • Tstop - the end time of the desired time frame
  • dchan - a list of DAQ channels being requested
  • ddir - the full path of the directory where the DAQ repository resides
  • flag - a flag that can be used to determine missing channel data

In daq_example_script, the output arguments of daq_fetch_v2 are assigned to the following variables:

  • DAQ_labels - a list of the channels returned, which is identical to the list provided (dchan) plus an additional channel named 'time' appended to the end
  • DAQ_data - an array of cells containing the requested data

Later sections will explain these inputs and describe how to interpret the returned data and use it in your matlab script.

DAQ Data Directories, Time-of-Interest, DAQ Channel Names

To describe what data you want to retrieve from the DAQ archive, you must specify the directory where the DAQ data resides, the time frame of data that you want retrieved, and the channel names desired.

DAQ Directory Location

You must know the full path name where the DAQ directory resides that contains the DAQ data for your desired time frame. Due to the voluminous storage requirements of DAQ data, file systems are periodically moved around to support operational demands. At the time of this writing, the following DAQ directories were available for 9ma data on ‘flashuser2’ (Note: not all directories are available from ttfsvr3 or gansvr2):

  • September ’08 DAQ data : /daq/CCEXP/EXP/linac
  • January ’09 DAQ data: /daq/CCEXP/EXP/linac
  • September ’09 DAQ data: /daq_data/ilc/LINAC

Time Frame of Interest

You must also specify the exact time frame of DAQ data that you wish to retrieve. The format used in the above example script is:

  • Tstart = datenum('09-17-2009 15:30:00');
  • Tstop = datenum('09-17-2009 15:31:00'); %CAUTION: Be careful not to ask for too much data!

The directory and the time frame specified must be consistent; in other words the files containing the DAQ data for the specified time frame must reside in the DAQ directory specified. There is a perl script that can be used to summarize the valid time frames contained within a particular directory. On the DESY network, type the following command:

  ~narnold/scripts/ –d <directory_path>

Here is a real example with real output:

  flashuser1{narnold}1: ~narnold/scripts/ -d /daq/CCEXP/EXP/linac
      Run 2698  started 2008-09-25 22:59:56  ended 2008-09-26 14:59:16
      Run 3398  started 2009-01-06 12:37:03  ended 2009-01-06 12:53:16
      Run 3404  started 2009-01-09 13:59:06  ended 2009-01-12 07:59:34
      Run 3410  started 2009-01-14 16:00:21  ended 2009-01-15 06:59:20

If specifying the directory /daq/CCEXP/EXP/linac, the specified time frame must fall within the listed runs.

DAQ Channel Names

The DAQ system is highly configurable such that channels can be added or deleted quite easily by restarting the DAQ (which creates a new “run #”). The most exhaustive tool to extract valid DAQ channels (for a given run or time frame) is the DAQ Data GUI. This java-based tool extracts the channel names specific to a specified time frame and allows basic viewing of the data. The tool must be run on the DESY network so it has access to the DAQ directories. It is available from the FLASH control screens.

Another tool to locate channel names of interest is a relational database application that stores the DAQ channel names with some ancillary data about those channels (e.g. # of samples, names of the multiple dimensions, an associated system, etc). Such channel meta-data will become more useful as one works with the DAQ data. The initial implementation of this database reflects the DAQ channels that were being recorded at the end of the September 2009 9mA Run (Run #4453). Therefore, there may be some channels in the database that do not exist in earlier runs and/or some channels from earlier runs that don't exist in the database. The DAQ Channel Database can be perused (here).

These two tools complement one another to allow you to find specific channel names and relevant information about those channels.

Interpretation of DAQ Data (returned by daq_fetch_v2.m)

There are several different 'types' of DAQ channels and the interpretation of the data returned by daq_fetch_v2.m varies according to these types. A 'fast' channel is recorded every linac pulse (e.g. RF measurements and diagnostics) and a 'slow' channel is recorded much less frequently, sometimes as slow as once per minute (e.g. room temperatures, magnet current readbacks, etc). Some channels record an entire waveform (spectra) of data each time it is stored and others a single floating point value (scalars). The DAQ Channel Database identifies each channel as one of the following types:

  • F:spectra - Fast spectra channels store a waveform for each linac pulse (5 Hz)
  • F:scalar - Fast scalar channels store a single value for each linac pulse (5 Hz)
  • F:bitmap - Fast bitmap channels store a single integer value for each linac pulse (5 Hz) where each bit represents a separate signal, primarily used to record Beam Loss Monitor trips.
  • S:scalar:U - Slow scalar:U channels store a single value at a predefined 'Update Rate'. The update rate may vary by channel and is typically on the order of 60 seconds.
  • S:scalar:E - Slow scalar:E channels store a single value whenever that value changes (E stands for environment). This may be as often as every few seconds (e.g. a vacuum reading) or many days (e.g. a motor position).
  • S:scalar:EU - Slow scalar:EU channels store a single value whenever that value changes AND at a predefined 'Update Rate'. This ensures that a value is stored at least at the update rate … and maybe more often.

The following sections describe how to interpret the data returned by daq_fetch_v2 for each type of channel.

Spectra Data (F:spectra)

The most common type of DAQ data is ‘spectra’, which means an array of data is captured for each pulse. The length of the spectra (# of samples per pulse) and the starting time of the spectra (sampling offset) vary for different channels. These parameters are available from the DAQ Channel Database. For spectra data, the following syntax is used to extract the desired data from the DAQ_data array:

    DAQ_data{c}(pulse#, sample#, dimension#)
         c = index of channel name in channel name list
         pulse# = 1:n, where n equals the number of pulses returned
         sample# = 1:s, where s equals the number of samples in the spectra
         dimension# = 1:d, where d = 1 + # of dimensions stored for that channel
             d=1 - array of sample #'s
             d=2 - first data array
             d=3 - second data array

A single DAQ channel may contain multiple “dimensions” of spectra data (referred to by 'd' above). Two common examples of this are the BPM signals which contain two dimensions (x & y), and certain RF signals which also contain two dimensions (I & Q). The data dimensions for the DAQ channels are documented in the DAQ Channel Database.

         DAQ_data{5}(1:3,:,2) % the entire spectra for the first 3 pulses of
                                the 1st data dimension (e.g. X if a BPM)
                                of the 5th channel in the list

         DAQ_data{5}(:,700:1200,3) % samples 700 thru 1200 for every pulse
                                     returned of the 2nd data dimension 
                                     (e.g. Y if a BPM)

Scalar Data (F:scalar, F:bitmap, S:scalar:E/U/EU)

NOTE: the daq_read_svr was modified on 04-Dec-2009 to improve the reading of this data type.

The data for scalar channels is returned in a structure, so extracting the data for a given pulse is different than spectra channels. The following code example assumes that the 4th channel in the channel list is a scalar channel:

  result1 = cell2mat(DAQ_data{4});  %Convert cell data to a structure
  % result1 = nx1 structure array with fields dtype and data where n = # of pulses
  scalarValues = []; % returns nx1 array of data where n = # of pulses

F:bitmaps (e.g. Fault Words from BIS)

NOTE: the daq_read_svr was modified on 11-Nov-2009 to accommodate this data type.

The example below shows how one can view the individual bits of the fault words stored from the BIS. Since these values are scalars, the same technique is used as described above. The following example code extracts and displays the fault word:

  result5 = cell2mat(DAQ_data{5});  % extract structured data from cell
  data5 = [];  % assign structure field named 'data' to an array
  for i = 1:numel(data5)   % print array contents as decimal and binary
       fprintf('Word Value - %d; Binary - %s\n', data5(i), dec2bin(data5(i)));

Each entry in the data5 array is for an individual pulse. The time of each pulse is recorded in the 'time' channel appended to the channel list (see 'time' below). A complete matlab routine that displays the fault words with the time stamps can be found here: test_bis_channels.m.

Testing for Scalar vs Spectra

It may be necessary to determine programmatically whether a channel is a scalar type or a spectra type. This can be done by examining the dimensions of the cell returned by daq_fetch_v2. A spectra will have three dimensions (pulse #, sample #s, data dimensions). The following example code loops through all the channel data, finds channels that are scalars, and extracts the data from the scalar structure and stores it in the original array:

for i=1:numel(DAQ_labels)-1       % for each channel except the last (time)
  if numel(size(DAQ_data{i})) < 3 % if # of dimensions less than 3, then scalar 
    result1 = cell2mat(DAQ_data{i});  % extract structure
    DAQ_data{i} = [];     % replace original cell with data array


daq_fetch_v2 returns an additional channel (appended to the original channel list) called 'time'. The time structure is an n x 2 array, where n, once again, equals the number of pulses returned from the DAQ repository. The first element is the wall clock time (and date) of each pulse in unix format (i.e. # of seconds since Jan 1, 1970). The second element is the event # (or pulse #) for each pulse returned in the data (the event number is an integer that continually increments on every LINAC pulse). Here is some example code of how to use these arrays:

   % Convert unix time array to matlab //datenum// format by 
   % dividing by the # of seconds in a day (86400),
   % adding the number of days between 1/1/0000 & 1/1/1970 (719529),
   % and adding two hours (2/24) to shift to Hamburg time:
   Tmat = DAQ_data{end}(:,1)/86400 + 719529 + 2/24;
   % Convert to datestr ...
   Tstring = datestr(Tmat, 'dd-mmm-yyyy HH:MM:SS.FFF');
   % Print out Event # and timestamp (including milliseconds)
   for i=1:5 
       fprintf('Event # %d occurred at %s\n', DAQ_data{end}(i,2), Tstring(i,:));
   Event # 86215432 occurred at 15-Sep-2009 01:30:10.095
   Event # 86215433 occurred at 15-Sep-2009 01:30:10.296
   Event # 86215434 occurred at 15-Sep-2009 01:30:10.499
   Event # 86215435 occurred at 15-Sep-2009 01:30:10.715
   Event # 86215436 occurred at 15-Sep-2009 01:30:10.907

DOI (Data of Interest) - A Matlab-based GUI for Retrieving DAQ Data

The Data of Interest (DOI) Tool is a Matlab-based GUI application which provides an interactive environment for fetching “data of interest” from the DAQ. The data retrieved can be stored in a local file for additional analysis or be immediately passed on to an analysis script. Key features of the DOI tool are:

  • Entry fields for Tstart, Tend, DAQ Directory
  • Specify channels to be retrieved by using predefined “channel group” files
  • Estimates amount of data being requested
  • On daq_fetch error, searches for the channel that is missing data
  • Store results in a local file (for interactive analysis)
  • Provides a framework for writing analysis programs that are “time frame independent”
  • Introspection of analysis scripts to discover channels used by script (i.e. selecting an analysis script puts those channels on the list to be fetched)

A user guide for the DOI Tool can be found here.

More Examples

Examples of typical analysis codes are described below. Please feel free to add your own in order to share what you have learned.

Eliminating 500KHz Noise on the Vector Sum Signals

Finding Sample #'s with Beam (using a toroid)

Finding Sample #'s with Beam (using the laser setup)

Extracting Sample #'s in Common Between Two Channels

Converting I/Q to Phase/Amplitude

— Last edited by Ned Arnold 19 November 2009

9ma/daq_page_v2.txt · Last modified: 2009/12/04 17:18 by 9ma
Recent changes RSS feed