IO Schema¶

The output of Chimbuko’s AD module is serialized in JSON format. This includes the streaming output sent between the parameter server and the visualization module, and the contents of the provenance database. In this section we provide the schema for these data.

Provenance Database Schema¶

Main database¶

Below we describe the JSON schema for the anomalies and normalexecs collections of the main database component of the provenance database.

Function event schema¶

This section describes the JSON schema for the anomalies and normalexecs collections. The fields of the JSON object are bolded, and a brief description follows the colon (:).

Function execution “events” in Chimbuko are labeled by a unique (for each process) string of following form “$RANK:$IO_STEP:$IDX” (eg “0:12:225”), where RANK, IO_STEP and IDX are the MPI rank, the io step and an integer index, respectively, and $VAL indicates the numerical value of the variable VAL. We will refer to such a string as an “event label” below.

{ start of schema

“__id”: Record index assigned by Sonata,
“call_stack”:    Function execution call stack (starting with anomalous function execution),
[
{
“entry”: timestamp of function execution entry ,
“exit”: timestamp of function execution exit (0 if has not exited at time of write) ,
“fid”: Global function index (can be used as a key instead of function name),
“func”: function name,
“event_id”: Event label (see above),
“is_anomaly”: True/false depending on whether event is anomalous (applies only to executions that have exited by time of write)
},
….
],
“counter_events”: [  An array of counter data received on the specific process thread during function execution
{
“counter_idx”: An index used internally to index counters,
“counter_name”: A string describing the counter,
“counter_value”: The value of the counter (integer),
“pid”: process index,
“rid”: process rank,
“tid”: process thread,
“ts”: timestamp
},
…
],
“entry”: Timestamp of function execution entry,
“exit”: Timestamp of function execution exit,
“event_id”: Event label (see above),
“fid”: Global function index (can be used as a key instead of function name),
“func”: function name,
“algo_params”:   The parameters used by the outlier detection algorithm to classify this event (format is algorithm dependent, see below),
“is_gpu_event”: true or false depending on whether function executed on a GPU
“gpu_location”: if a GPU event, a JSON description of the context (see below), otherwise null,
“gpu_parent”: if a GPU event, a JSON description of the parent CPU function (see below), otherwise null,
“pid”: process index,
“rid”: process rank,
“tid”: thread index
“hostname”: The hostname of the node on which the application was executing
“runtime_exclusive”: Function execution time exclusive of children,
“runtime_total”: Function total execution time,
“io_step”: IO step index,
“io_step_tstart”: Time of start of IO step,
“io_step_tend”:  Time of end of IO step,
“outlier_score”: The anomaly score of the execution (algorithm dependent),
“event_window”: Capture of function executions and MPI comms events occuring in window around anomaly on same thread (object)
{
“exec_window”: The function executions in the window arranged in order of their entry time (array)
[
{
“entry”: timestamp of function execution entry ,
“exit”: timestamp of function execution exit (0 if has not exited at time of write) ,
“fid”: Global function index (can be used as a key instead of function name),
“func”: function name,
“event_id”: Event label (see above),
“parent_event_id”: Event label of parent function execution,
“is_anomaly”: True/false depending on whether event is anomalous (applies only to executions that have exited by time of write)
},
…
],
“comm_window”: The MPI communications in the window (array)
[
{
type: SEND or RECV,
pid: process index,
rid: rank of current process,
tid: thread idx,
src: message origin rank,
tar: message target rank,
bytes: message size,
tag: an integer tag associated with the message,
timestamp: time MPI function executed,
execdata_key: the ID label of the parent function
},
…
]
} end of event_window object

} end of schema

For the SSTD (original) algorithm, the algo_params field has the following format:

{

“accumulate”: not used at present,
“count”: number of times function encountered (global),
“kurtosis”: kurtosis of distribution,
“maximum”: maximum of distribution,
“mean”: mean of distribution,
“minimum”: minimum of distribution,
“skewness”: skewness of distribution,
“stddev”: standard deviation of distribution

}

The schema for the gpu_location field is as follows:

{

“context”: GPU device context (NVidia terminology),
“device”: GPU device index,
“stream”: GPU device stream (NVidia terminology),
“thread”: virtual thread index assigned to this context/device/stream by Tau

}

and for the gpu_parent field:

{

“event_id”: The event label (see above) of the parent function execution,
“tid”: Thread index for CPU parent function,
“call_stack”:    Parent function call stack (starting with parent function execution),
[
{
“entry”: timestamp of function execution entry ,
“exit”: timestamp of function execution exit (0 if has not exited at time of write) ,
“fid”: Global function index (can be used as a key instead of function name),
“func”: function name,
“event_id”: The event label
},
….
]

}

Note that Tau considers a GPU device/context/stream much in the same way as a CPU thread, and assigns it a unique index. This index is the “thread index” for GPU events.

Metadata schema¶

Metadata are stored in the metadata collection in the following JSON schema:

{

“descr”: String description (key) of metadata entry
“pid”: Program index from which metadata originated,
“rid”: Process rank from which metadata originated,
“tid”: Process thread associated with metadata,
“value”: Value of the metadata entry,
“__id”: Record index assigned by Sonata*

}

Note that the tid (thread index) for metadata is usually 0, apart from for metadata associated with a GPU context/device/stream, for which the index is the virtual thread index assigned by Tau to the context/device/stream.

Global database¶

Below we describe the JSON schema for the func_stats and counter_stats collections of the global database component of the provenance database.

Function profile statistics schema¶

func_stats contains aggregated profile information for all functions. The JSON schema is as follows:

{

‘app’: program index,

‘fid’: global function index,

‘name’: function name,

‘exclusive’: Statistics of runtime exclusive of children

{

‘accumulate’: unused,
‘count’: total function executions,
‘kurtosis’: kurtosis of function exclusive time distribution,
‘maximum’: maximum function exclusive time,
‘mean’: average function exclusive time,
‘minimum’: minimum function exclusive time,
‘skewness’: skewness of function exclusive time distribution,
‘stddev’: standard deviation of function exclusive time distribution,

},

‘inclusive’: Statistics of runtime inclusive of children

{

‘accumulate’: unused,
‘count’: total function executions,
‘kurtosis’: kurtosis of function inclusive time distribution,
‘maximum’: maximum function inclusive time,
‘mean’: average function inclusive time,
‘minimum’: minimum function inclusive time,
‘skewness’: skewness of function inclusive time distribution,
‘stddev’: standard deviation of function inclusive time distribution,

},

‘stats’: Statistics on function anomalies per timestep observed in run to-date

{

‘accumulate’: total number of anomalies observed for this function,
‘count’: number of timesteps data colected for,
‘kurtosis’: kurtosis of distribution of anomalies/step,
‘maximum’: maximum anomalies/step,
‘mean’: average anomalies/step,
‘minimum’: minimum anomalies/step,
‘skewness’: skewness of distribution of anomalies/step,
‘stddev’: standard deviation distribution of anomalies/step,

}

Counter statistics schema¶

The counter_stats collection has the following schema:

{

‘app’: Program index,

‘counter’: Counter description,

‘stats’: Global aggregated statistics on counter values since start of run,

{

‘accumulate’: Unused,
‘count’: Number of times counter appeared,
‘kurtosis’: kurtosis of distribution of value,
‘maximum’: maximum value,
‘mean’: average value,
‘minimum’: minimum value,
‘skewness’: skewness of distribution of values,
‘stddev’: standard deviation of distribution of values
}

}

Parameter Server Streaming Output¶

Every IO frame the AD instances send three pieces of information to the pserver:

For every function execution in the IO frame the inclusive and exclusive runtime and the number of anomalies for this function. These are aggregated over all IO frames and ranks on the parameter server and represent the function profile.
The total number of anomalies detected in the IO frame.
Statistics on the values of each counter over the IO step (e.g. for a memory usage counter this would be the mean, std.dev., etc of the memory usage over the IO frame. These are aggregated over all IO frams and ranks on the parameter server.

The parameter server optionally sends data to an external webserver as JSON-formatted packets via http using libcurl at some fixed frequency (independent of the frequency of IO steps in the trace data collection). This communication is handled by the PSstatSender class. The data packet is a JSON object comprising two payloads: anomaly_stats and counter_stats. Note, counters are integer valued quantities that are typically hardware counters but include information on MPI communications packet sizes, etc. The data packet JSON object has the following schema:

{

‘anomaly_stats’: Statistics of anomalies (object with schema given below). This field will not appear if no data has been received from the AD instances since the last send

‘counter_stats’: Statistics of counter values aggregated over all ranks (array). This field will not appear if no counters were ever collected

[

{

‘app’: Program index,

‘counter’: Counter description,

‘stats’: Global aggregated statistics on counter values since start of run,

{

‘accumulate’: Unused,
‘count’: Number of times counter appeared,
‘kurtosis’: kurtosis of distribution of value,
‘maximum’: maximum value,
‘mean’: average value,
‘minimum’: minimum value,
‘skewness’: skewness of distribution of values,
‘stddev’: standard deviation of distribution of values

}

},

…

]

}

Note that the anomaly_stats entry will only be present if data has been received from the AD instances since the last send, and the counter_stats array will only appear if counters have ever been collected.

The schema for the ‘anomaly_stats’ object is as follows:

{

‘created_at’: timestamp given in milliseconds relative to epoch,

‘anomaly’: Statistics on anomalies by process/rank (array)

[

{

‘data’: Number of anomalies and anomaly time window for process/rank broken down by io step (array)

[

{

‘app’: Program index,
‘max_timestamp’: Latest time of anomaly in io step,
‘min_timestamp’: Earliest time of anomaly in io step,
‘n_anomalies’: Number of anomalies in io step,
‘rank’: Process rank,
‘stat_id’: A string label of the form “$PROGRAM ID:$RANK” (eg “0:12”),
‘step’: io step index,
‘outlier_scores’: Statistics on the outlier scores for the outliers collected in this step,
{
‘accumulate’: The sum of the outlier scores,
‘count’: The number of outliers,
‘kurtosis’: kurtosis of distribution of scores,
‘maximum’: maximum score,
‘mean’: mean score,
‘minimum’: minimum score,
‘skewness’: skewness of scores,
‘stddev’: std.dev of scores
},

},

…

],

‘key’: A string label of the form “$PROGRAM ID:$RANK” (eg “0:12”),

‘stats’: *Statistics on anomalies on this process/rank over all steps to date

{

‘accumulate’: Total anomalies,
‘count’: Number of io steps included in statistics,
‘kurtosis’: kurtosis of distribution of anomalies,
‘maximum’: maximum number of anomalies observed,
‘mean’: average number of anomalies observed,
‘minimum’: minimum number of anomalies observed,
‘skewness’: skewness of distribution of anomalies,
‘stddev’: standard deviation of distribution of anomalies

}

},

…

], end of anomaly array

‘anomaly_metrics’:

[

{

‘app’: Application,
‘rank’: Program rank,
‘fid’: function ID,
‘fname’: funciton name,
‘_id’: a global index to track each (app, rank, func), for internal use,
‘new_data’: Statistics of anomaly metrics aggregated over multiple IO steps since the last pserver->viz send
{
‘first_io_step’: first io step in sum
‘last_io_step’: last io step in sum
‘max_timestamp’: max timestamp of last IO step of this period
‘min_timestamp’: min timestamp of first IO step of this period
‘severity’: RunStats assigned severity
‘score’: RunStats assigned score
‘count’: RunStats count
}
‘all_data’: Statistics of anomaly metrics aggregated since the beginning of the run
{
‘first_io_step’: first io step in sum
‘last_io_step’: last io step in sum
‘max_timestamp’: max timestamp of last IO step since start of run
‘min_timestamp’: min timestamp of first IO step since start of run
‘severity’: RunStats assigned severity
‘score’: RunStats score
‘count’: RunStats count
}

}

], end of anomaly_metrics

‘func’: Statistics on anomalies broken down by function, collected over entire run to-date (array)

[

{