RL Arguments/Config Package

At a Glance

Purpose:

Configuration constants, algorithm definitions, and observation spaces

Location:

fusion/modules/rl/args/

Key Files:

general_args.py, observation_args.py, constants.py

Prerequisites:

None (reference module)

This package defines the configuration constants used throughout the RL module. It’s a reference module - you import from it, but rarely need to modify it.

What This Module Provides

+------------------------------------------------------------------+
|                    ARGS MODULE CONTENTS                          |
+------------------------------------------------------------------+
|                                                                  |
|  Algorithm Definitions       Observation Spaces      Defaults    |
|  ---------------------       ------------------      --------    |
|  - Which algorithms          - obs_1 through obs_8  - Epsilon    |
|    are valid                 - Feature lists        - Learning   |
|  - Path vs Core vs DRL       - Validation sets        rate       |
|  - Decay strategies                                 - Batch size |
|                                                                  |
+------------------------------------------------------------------+

Quick Reference

Valid Algorithms

from fusion.modules.rl.args import (
    VALID_PATH_ALGORITHMS,
    VALID_CORE_ALGORITHMS,
    VALID_DRL_ALGORITHMS,
)

# Path selection: all algorithms
VALID_PATH_ALGORITHMS = [
    "q_learning",
    "epsilon_greedy_bandit",
    "ucb_bandit",
    "ppo",
    "a2c",
    "dqn",
    "qr_dqn",
]

# Core selection: traditional only
VALID_CORE_ALGORITHMS = [
    "q_learning",
    "epsilon_greedy_bandit",
    "ucb_bandit",
]

# Deep RL only
VALID_DRL_ALGORITHMS = [
    "ppo",
    "a2c",
    "dqn",
    "qr_dqn",
]

Observation Spaces

from fusion.modules.rl.args import OBS_DICT

# obs_1: Minimal - just routing
OBS_DICT["obs_1"] = ["source", "destination"]

# obs_4: Standard - routing with constraints
OBS_DICT["obs_4"] = [
    "source", "destination",
    "request_bandwidth", "holding_time"
]

# obs_8: Complete - all features
OBS_DICT["obs_8"] = [
    "source", "destination",
    "request_bandwidth", "holding_time",
    "slots_needed", "path_lengths",
    "paths_cong", "available_slots",
    "is_feasible"
]

Understanding Observation Spaces

The RL module defines 8 observation space configurations, from minimal to complete:

Space	Features	Use Case
obs_1	source, destination	Minimal routing, small state space
obs_2	bandwidth	Bandwidth-aware routing
obs_3	holding_time	Time-sensitive routing
obs_4	bandwidth, holding_time	Standard routing (recommended start)
obs_5	slots_needed, path_lengths	Resource-aware routing
obs_6	paths_cong	Congestion-aware routing
obs_7	available_slots	Availability-aware routing
obs_8	is_feasible	Complete (action masking friendly)

Choosing an Observation Space

# For simple experiments, start with obs_4
engine_props["obs_space"] = "obs_4"

# For production/complex scenarios, use obs_8
engine_props["obs_space"] = "obs_8"

# Check what features are included
from fusion.modules.rl.args import OBS_DICT
features = OBS_DICT["obs_8"]
print(f"obs_8 includes: {features}")

Feature Descriptions

Feature	Description
source	Source node ID (normalized to [0, 1])
destination	Destination node ID (normalized to [0, 1])
request_bandwidth	Requested bandwidth (normalized)
holding_time	Connection duration (normalized)
slots_needed	Spectrum slots required per path
path_lengths	Physical length of each candidate path
paths_cong	Congestion level per path [0, 1]
available_slots	Available slots ratio per path [0, 1]
is_feasible	Boolean mask indicating feasible paths

Using Constants in Your Code

Validating Algorithm Names

from fusion.modules.rl.args import VALID_PATH_ALGORITHMS

def setup_agent(algorithm: str):
    if algorithm not in VALID_PATH_ALGORITHMS:
        raise ValueError(
            f"Invalid algorithm '{algorithm}'. "
            f"Valid options: {VALID_PATH_ALGORITHMS}"
        )
    # ... proceed with setup

Checking Algorithm Category

from fusion.modules.rl.args import VALID_DRL_ALGORITHMS

def is_deep_rl(algorithm: str) -> bool:
    """Check if algorithm is a deep RL method."""
    return algorithm in VALID_DRL_ALGORITHMS

# Usage
if is_deep_rl("ppo"):
    # Use SB3 for training
    pass
else:
    # Use traditional RL update
    pass

Using Episodic Strategies

from fusion.modules.rl.args import EPISODIC_STRATEGIES

# EPISODIC_STRATEGIES = ["exp_decay", "linear_decay"]

def should_update_epsilon_at_episode_end(strategy: str) -> bool:
    """Check if strategy updates at episode boundaries."""
    return strategy in EPISODIC_STRATEGIES

The New Constants Module

For new code, constants.py provides type-safe enums and better organization:

Using Enums (Type-Safe)

from fusion.modules.rl.args.constants import (
    AlgorithmType,
    EpisodicStrategy,
    ObservationFeature,
)

# Type-safe algorithm checking
def setup_algorithm(algo_type: AlgorithmType):
    if algo_type == AlgorithmType.PPO:
        return setup_ppo()
    elif algo_type == AlgorithmType.Q_LEARNING:
        return setup_q_learning()

# Using enum values with string-based APIs
algorithm_name = AlgorithmType.PPO.value  # "ppo"

Available Enums

AlgorithmType:

class AlgorithmType(Enum):
    Q_LEARNING = "q_learning"
    EPSILON_GREEDY_BANDIT = "epsilon_greedy_bandit"
    UCB_BANDIT = "ucb_bandit"
    PPO = "ppo"
    A2C = "a2c"
    DQN = "dqn"
    QR_DQN = "qr_dqn"

EpisodicStrategy:

class EpisodicStrategy(Enum):
    EXPONENTIAL_DECAY = "exp_decay"
    LINEAR_DECAY = "linear_decay"

ObservationFeature:

class ObservationFeature(Enum):
    SOURCE = "source"
    DESTINATION = "destination"
    REQUEST_BANDWIDTH = "request_bandwidth"
    HOLDING_TIME = "holding_time"
    SLOTS_NEEDED = "slots_needed"
    PATH_LENGTHS = "path_lengths"
    PATH_CONGESTION = "paths_cong"
    AVAILABLE_SLOTS = "available_slots"
    IS_FEASIBLE = "is_feasible"

Default Configurations

from fusion.modules.rl.args.constants import (
    DEFAULT_EPSILON,
    DEFAULT_EPSILON_MIN,
    DEFAULT_EPSILON_DECAY,
    DEFAULT_LEARNING_RATE,
    DEFAULT_DISCOUNT_FACTOR,
    DEFAULT_BATCH_SIZE,
    DEFAULT_BUFFER_SIZE,
    DEFAULT_OBSERVATION_SPACE,
)

# Defaults:
# DEFAULT_EPSILON = 1.0
# DEFAULT_EPSILON_MIN = 0.01
# DEFAULT_EPSILON_DECAY = 0.995
# DEFAULT_LEARNING_RATE = 0.001
# DEFAULT_DISCOUNT_FACTOR = 0.99
# DEFAULT_BATCH_SIZE = 32
# DEFAULT_BUFFER_SIZE = 10000
# DEFAULT_OBSERVATION_SPACE = "obs_4"

Algorithm Registry

The registry_args.py module provides a registry for DRL algorithms:

from fusion.modules.rl.args.registry_args import get_algorithm_registry

registry = get_algorithm_registry()

# Registry structure:
# {
#     "ppo": {"class": PPO, "load": None, "setup": setup_ppo},
#     "a2c": {"class": A2C, "load": None, "setup": setup_a2c},
#     "dqn": {"class": DQN, "load": None, "setup": setup_dqn},
#     "qr_dqn": {"class": QrDQN, "load": None, "setup": setup_qr_dqn},
# }

# Get setup function for an algorithm
setup_fn = registry["ppo"]["setup"]
model = setup_fn(env, config)

Extending the Args Module

Tutorial: Adding a New Observation Feature

Step 1: Add to ObservationFeature enum

In constants.py:

class ObservationFeature(Enum):
    # ... existing features ...
    LATENCY = "latency"  # New feature

Step 2: Add to VALID_OBSERVATION_FEATURES

In observation_args.py:

VALID_OBSERVATION_FEATURES: set[str] = {
    # ... existing features ...
    "latency",
}

Step 3: Create new observation space using it

In observation_args.py:

OBS_DICT["obs_9"] = [
    "source",
    "destination",
    "request_bandwidth",
    "holding_time",
    "latency",  # New feature
]

Step 4: Implement feature extraction

In your observation builder (e.g., adapter or environment):

if "latency" in features:
    obs["latency"] = self._compute_latency(paths)

Tutorial: Adding a New Algorithm

Step 1: Add to AlgorithmType enum

In constants.py:

class AlgorithmType(Enum):
    # ... existing algorithms ...
    SAC = "sac"  # Soft Actor-Critic

Step 2: Add to algorithm lists

In general_args.py:

VALID_PATH_ALGORITHMS: list[str] = [
    # ... existing algorithms ...
    "sac",
]

VALID_DRL_ALGORITHMS: list[str] = [
    # ... existing algorithms ...
    "sac",
]

Step 3: Add to registry (if DRL)

In registry_args.py:

from fusion.modules.rl.algorithms.sac import SAC
from fusion.modules.rl.utils.setup import setup_sac

_algorithm_registry = {
    # ... existing algorithms ...
    "sac": {
        "class": SAC,
        "load": None,
        "setup": setup_sac,
    },
}

Backward Compatibility Notes

The args module maintains backward compatibility through:

1. Abbreviated names: paths_cong instead of path_congestion

2. Dual exports: OBS_DICT and OBSERVATION_SPACE_DEFINITIONS

3. Alias variables: VALID_DEEP_REINFORCEMENT_LEARNING_ALGORITHMS = VALID_DRL_ALGORITHMS

When writing new code:

• Use the full names when possible (e.g., OBSERVATION_SPACE_DEFINITIONS)

• Use enums from constants.py for type safety

• Import from the package level (from fusion.modules.rl.args import ...)

File Reference

fusion/modules/rl/args/
|-- __init__.py          # Package exports
|-- README.md            # Module documentation
|-- general_args.py      # Algorithm lists, strategies
|-- observation_args.py  # Observation space definitions
|-- constants.py         # Enums, defaults (new code)
`-- registry_args.py     # DRL algorithm registry

What to import:

# Most common imports (from package)
from fusion.modules.rl.args import (
    VALID_PATH_ALGORITHMS,
    VALID_DRL_ALGORITHMS,
    EPISODIC_STRATEGIES,
    OBS_DICT,
    VALID_OBSERVATION_FEATURES,
)

# For new code (type-safe enums)
from fusion.modules.rl.args.constants import (
    AlgorithmType,
    EpisodicStrategy,
    ObservationFeature,
    DEFAULT_EPSILON,
    DEFAULT_LEARNING_RATE,
)

# For algorithm setup
from fusion.modules.rl.args.registry_args import get_algorithm_registry

Related Documentation

• RL Algorithms Package - Algorithm implementations that use these constants

• RL Agents Package - Agents that validate against these constants

• RL Adapter Package - Adapter that uses observation space definitions