ADmM: Anomaly Detection for Microservice Systems with Incomplete Metrics

Introduction

We propose an anomaly detection model, ADmM, which integrates logs, metrics, and traces to address the above challenge posed by incomplete metric data. ADmM first extracts template-level and semantic-level features from multimodal inputs. Then, a multi-scale autoencoder module is applied to impute missing metrics. For anomaly detection, the model represents microservice dependencies as a directed acyclic graph and leverages a graph neural network to learn generative patterns of normal samples. By measuring the deviation between observed values and predicted values, ADmM assigns anomaly scores to identify anomalies.

Implementation

Environments

torch==1.12.0 scikit-learn==1.0.2 scikit-network==0.28.2 pandas==1.3.5 numpy==1.21.6 gensim==4.2.0 drain3==0.9.11 dgl==1.1.0+cu116

Data Processing Guide

This document provides step-by-step instructions for processing the SN and TT datasets. The core objective is to convert raw observational data into structured feature vectors (chunks) through feature extraction.

1. Download and Extract Datasets

Download the SN and TT datasets from the following link:

🔗 Zenodo - SN & TT Datasets

Download the GAIA dataset from the following link：

https://github.com/CloudWise-OpenSource/GAIA-DataSet

Extract the contents into the dataset folder. Ensure that the dataset paths in the code are correctly configured.

2. Data Preprocessing

Run the preprocess,py script to clean and standardize the data format:

This step ensures consistency across different data sources. You can inspect the generated output to understand the expected data format.

3. Feature Extraction and Processing

Once preprocessing is complete, proceed with data transformation using the process.py script:

A chunk represents a single data sample. The transformation process includes the following steps:

3.1 Log Data Processing

• Train a Drain3 model and a FastText word embedding model.
• Save the trained models in the drain and w2v_model directories.
• Use the trained Drain3 model to parse log data and generate log feature vectors.

3.2 Metric Data Processing

• Extract relevant metric values.
• Normalize the extracted metrics for consistency.

3.3 Trace Data Processing

• Identify service call relationships within trace data.
• Compute call latencies based on trace spans.
• The logic for extracting call dependencies is implemented in the script.

4. Summary

Follow these steps sequentially to ensure proper data processing. The key focus is transforming raw textual observability data into structured feature vectors (chunks) for further analysis.

Model Training

run python main.py