Metrics Guide

This document covers the metrics collection, ML training data, and persistence systems in the core module.

Overview

FUSION collects comprehensive statistics during simulation:

+-------------------+     +-------------------+     +-------------------+
|   SimStats        | --> | StatsPersistence  | --> |   JSON/CSV File   |
| (collect metrics) |     | (save to file)    |     |   (output)        |
+-------------------+     +-------------------+     +-------------------+

+-------------------+     +-------------------+
| MLMetricsCollector| --> |   CSV File        |
| (ML training data)|     |   (training data) |
+-------------------+     +-------------------+

Key distinction:

metrics.py / SimStats: Runtime statistics collection and blocking calculations
persistence.py / StatsPersistence: Saving and loading statistics to files
ml_metrics.py / MLMetricsCollector: SL/USL training data (NOT RL metrics)

Important

The organization of metrics.py (2,027 lines) will be improved in a future version. The file should be split into focused modules. This project is open source and welcomes contributions to help with this refactoring.

metrics.py - Statistics Collection

The SimStats class is the central statistics collector for all simulation metrics.

SimStats Class

Location:: fusion/core/metrics.py
Size:: 2,027 lines (needs refactoring)
Key Class:: SimStats

Initialization:

from fusion.core import SimStats

stats = SimStats(
    engine_props=engine_props,
    sim_info="experiment_001",
)

# Initialize for new iteration
stats.init_iter_stats()

Key methods:

Method	Description
`init_iter_stats()`	Initialize statistics for new iteration
`iter_update()`	Process single request outcome
`calculate_blocking_statistics()`	Compute blocking rates after iteration
`calculate_confidence_interval()`	Check if results are statistically significant
`finalize_iteration_statistics()`	Complete iteration and prepare for next

Metrics Tracked

Blocking Metrics:

# Request blocking
stats_props.simulation_blocking_list: list[float]  # Per-iteration blocking rates
stats_props.block_reasons_dict: dict[str, int]     # Reasons for blocking

# Bit-rate blocking
stats_props.simulation_bitrate_blocking_list: list[float]

# Bandwidth-specific blocking
stats_props.bandwidth_blocking_dict: dict[str, int]  # Blocked per bandwidth class

Resource Utilization:

# Spectrum usage
stats_props.cores_dict: dict[int, int]              # Allocations per core
stats_props.link_usage_dict: dict[str, dict]        # Per-link usage
stats_props.frag_dict: dict[str, float]             # Fragmentation scores

# Lightpath utilization
stats_props.lp_bw_utilization_dict: dict[str, float]
stats_props.sim_lp_utilization_list: list[float]

Path Metrics:

stats_props.hops_list: list[float]      # Hop counts
stats_props.lengths_list: list[float]   # Path lengths in km
stats_props.route_times_list: list[float]  # Routing computation time

Signal Quality:

stats_props.snr_list: list[float]       # SNR values
stats_props.crosstalk_list: list[float] # Crosstalk values

Transponders:

stats_props.transponders_list: list[int]  # Transponders per request

Survivability (v6.0+):

stats_props.protection_switchovers: int    # Successful switchovers
stats_props.protection_failures: int       # Both paths failed
stats_props.failure_induced_blocks: int    # Dropped due to failures
stats_props.switchover_times: list[float]  # Switchover timestamps

Usage Flow

# 1. Create stats collector
stats = SimStats(engine_props, sim_info)

# 2. For each iteration
for iteration in range(max_iters):
    stats.init_iter_stats()

    # 3. Process each request
    for request in requests:
        # ... process request ...
        stats.iter_update(
            request_data=request_dict,
            sdn_data=sdn_props,
            network_spectrum_dict=spectrum_dict,
        )

    # 4. Calculate iteration statistics
    stats.calculate_blocking_statistics()
    stats.finalize_iteration_statistics()

    # 5. Check convergence
    if stats.calculate_confidence_interval():
        break  # Statistically significant

# 6. Save results
persistence = StatsPersistence(engine_props, sim_info)
persistence.save_stats(stats_dict, stats.stats_props, blocking_stats)

Confidence Interval Calculation

FUSION uses confidence intervals to determine when results are statistically significant:

def calculate_confidence_interval(self) -> bool:
    """
    Calculate 95% confidence interval for blocking probability.

    Returns True if CI width is below threshold (results are reliable).
    """
    blocking_list = self.stats_props.simulation_blocking_list

    if len(blocking_list) < 3:
        return False  # Need at least 3 samples

    mean = statistics.mean(blocking_list)
    stdev = statistics.stdev(blocking_list)
    n = len(blocking_list)

    # 95% CI using t-distribution
    t_value = 1.96  # For large n
    ci_half_width = t_value * (stdev / math.sqrt(n))

    # Check if CI is narrow enough
    ci_percent = (ci_half_width / mean) * 100 if mean > 0 else float('inf')
    return ci_percent < self.engine_props.get('ci_target', 5.0)

StatsProps Class

The StatsProps class holds all collected statistics:

class StatsProps:
    """Statistics collection container."""

    def __init__(self) -> None:
        # Snapshot tracking
        self.snapshots_dict: dict[int, dict] = {}

        # Resource usage
        self.cores_dict: dict[int, int] = {}
        self.link_usage_dict: dict[str, dict] = {}
        self.frag_dict: dict[str, float] = {}

        # Blocking
        self.block_reasons_dict: dict[str, int] = {
            "distance": 0,
            "congestion": 0,
            "xt_threshold": 0,
            "failure": 0,
        }

        # Per-simulation metrics
        self.simulation_blocking_list: list[float] = []
        self.simulation_bitrate_blocking_list: list[float] = []

        # ... many more fields

persistence.py - Data Storage

The StatsPersistence class handles saving and loading simulation statistics.

StatsPersistence Class

Location:: fusion/core/persistence.py
Size:: 254 lines
Key Class:: StatsPersistence

Purpose:

Save statistics to JSON files
Load previously saved statistics
Format data for backward compatibility

Usage:

from fusion.core import StatsPersistence

# Create persistence handler
persistence = StatsPersistence(
    engine_props=engine_props,
    sim_info="experiment_001",
)

# Save statistics
persistence.save_stats(
    stats_dict={"key": "value"},
    stats_props=stats.stats_props,
    blocking_stats={
        "block_mean": 0.05,
        "block_variance": 0.001,
        "block_ci": 0.002,
        "iteration": 10,
    },
)

# Load statistics
loaded = persistence.load_stats("path/to/file.json")

Output File Format

Statistics are saved in JSON format:

{
  "sim_end_time": "0115_14_30_45_123456",
  "blocking_mean": 0.05,
  "blocking_variance": 0.001,
  "ci_rate_block": 0.002,
  "ci_percent_block": 4.0,
  "bit_rate_blocking_mean": 0.03,
  "lightpath_utilization": 0.75,
  "link_usage": {
    "(0, 1)": {"total_allocations": 100}
  },
  "iter_stats": {
    "1": {
      "trans_mean": 2.5,
      "trans_min": 1,
      "trans_max": 5,
      "hops_mean": 3.2,
      "lengths_mean": 450.5,
      "sim_block_list": [0.04, 0.05, 0.06],
      "mods_used_dict": {"QPSK": 50, "16-QAM": 30}
    }
  }
}

Output File Location

Files are saved to:

{PROJECT_ROOT}/{base_file_path}/output/{sim_info}/{thread_num}/{erlang}_erlang.json

Example:

/project/data/output/experiment_001/s1/300_erlang.json

Difference: metrics.py vs persistence.py

Aspect	metrics.py (SimStats)	persistence.py (StatsPersistence)
Purpose	Collect and calculate statistics	Save and load statistics
Timing	During simulation	End of simulation
Data	Live, mutable	Final, serialized
Format	Python objects	JSON/CSV files
Scope	Statistics, CI, blocking rates	File I/O operations

ml_metrics.py - ML Training Data

The MLMetricsCollector collects training data for supervised and unsupervised learning models.

Important

Despite the “ML” naming, this module is specifically for supervised learning (SL) and unsupervised learning (USL) data collection. Reinforcement learning (RL) metrics are handled separately in fusion.rl.

Note

This file is new and not heavily tested yet. Contributions welcome!

MLMetricsCollector Class

Location:: fusion/core/ml_metrics.py
Size:: 163 lines
Key Class:: MLMetricsCollector

Purpose:

Collect features for ML model training
Track network state at allocation time
Generate training datasets

Usage:

from fusion.core import MLMetricsCollector

# Enable in config
engine_props['output_train_data'] = True

# Create collector
ml_metrics = MLMetricsCollector(
    engine_props=engine_props,
    sim_info="experiment_001",
)

# After each allocation
ml_metrics.update_train_data(
    old_request_info_dict=original_request,
    request_info_dict=processed_request,
    network_spectrum_dict=spectrum_dict,
    current_transponders=num_transponders,
)

# Save at end
ml_metrics.save_train_data(
    iteration=current_iter,
    max_iterations=max_iters,
)

Training Data Features

The collector generates these features:

Feature	Description
`old_bandwidth`	Original request bandwidth before slicing
`path_length`	Physical path length in km
`longest_reach`	Maximum reach of available modulation
`average_congestion`	Average spectrum utilization on path
`num_segments`	Number of lightpaths (transponders) used

Output Format

Training data is saved to CSV:

{PROJECT_ROOT}/data/output/{sim_info}/{erlang}_train_data.csv

old_bandwidth,path_length,longest_reach,average_congestion,num_segments
100,450.5,1200,0.45,1
200,800.2,1000,0.62,2
400,350.1,2000,0.38,3

When Data is Saved

Simulation statistics (via StatsPersistence) can be saved at configurable intervals using the save_step parameter, not just at the end.

ML training data (via MLMetricsCollector) is saved only on the last iteration by default to capture the complete dataset:

def save_train_data(self, iteration: int, max_iterations: int, ...):
    if iteration == (max_iterations - 1):
        # ML training data saves on last iteration by default
        save_df = pd.DataFrame(self.train_data_list)
        save_df.to_csv(output_path, index=False)

Best Practices

Collecting Custom Metrics

To add custom metrics:

Add field to StatsProps class:

# In properties.py
class StatsProps:
    def __init__(self):
        # ... existing fields ...
        self.my_custom_list: list[float] = []

Update in SimStats.iter_update():

# In metrics.py
def iter_update(self, request_data, sdn_data, ...):
    # ... existing updates ...
    self.stats_props.my_custom_list.append(custom_value)

Include in StatsPersistence._prepare_iteration_stats():

# In persistence.py
def _prepare_iteration_stats(self, stats_props, iteration):
    # ... existing code ...
    if stats_props.my_custom_list:
        iter_stats["my_custom_mean"] = mean(stats_props.my_custom_list)

Performance Considerations

Memory Usage - Lists grow during simulation - Consider periodic aggregation for long simulations - Clear lists between iterations when possible
Computation - Confidence interval calculation can be expensive - Only calculate when needed (not every request) - Use numpy for large array operations
File I/O - JSON files can be large for long simulations - Consider compression for large datasets - Save incrementally for crash recovery

Future Improvements

The following improvements are planned:

Split metrics.py into focused modules:
- blocking_stats.py - Blocking calculations
- resource_stats.py - Resource utilization
- survivability_stats.py - Recovery metrics
- confidence.py - Statistical analysis
Add streaming output for long simulations
Improve ML features for better model training