name	fabric-perfsonar
description	Set up perfSONAR network monitoring with testpoints, archives, and dashboards on FABRIC
allowed-tools	["Read","Grep","Glob","Write","Edit","Bash"]

Instructions

When invoked, generate code to set up perfSONAR network monitoring on FABRIC nodes. perfSONAR measures network performance metrics including throughput, latency, RTT, MTU, and traceroute between endpoints.

Deployment options:

Docker Testpoint — Lightweight container-based deployment
Native Install with perfsonar-extensions — Full stack with pSConfig, Grafana, and archive (recommended)
Ad-Hoc Tests — Quick one-off measurements with pScheduler

Guide the user on which deployment model to use and what tests to configure.

API Reference

perfSONAR Test Types

Test Type	Tool	Measures
throughput	`iperf3`	Bandwidth between endpoints
latency / latencybg	`owamp`	One-way delay
rtt	`ping`	Round-trip time
trace	`traceroute`/`tracepath`	Network path
disk-to-disk	`nuttcp`	Disk-to-disk transfer rate

pSConfig Mesh Configuration

perfSONAR tests are coordinated via pSConfig (perfSONAR Configuration) mesh files that define which nodes test to which others and what tests to run. The psconfig_builder.py tool from perfsonar-extensions generates these configurations.

Patterns

Full Stack perfSONAR with perfsonar-extensions (Recommended)

from fabrictestbed_extensions.fablib.fablib import FablibManager

fablib = FablibManager()

TOTAL_NODES = 2
slice_name = "perfsonar-monitoring"

sites = fablib.get_random_sites(TOTAL_NODES)

slice = fablib.new_slice(name=slice_name)

for i, s in enumerate(sites):
    prefix = "central" if i == 0 else "remote"

    net = slice.add_l3network(name=f"{s}-l3", type="IPv4")

    node = slice.add_node(
        name=f"{prefix}-{s}",
        site=s,
        image="default_ubuntu_24",
        cores=16,
        ram=32,
        disk=100,
    )

    iface = node.add_component(model="NIC_Basic", name="nic1").get_interfaces()[0]
    iface.set_mode('auto')
    net.add_interface(iface)
    node.add_route(subnet=fablib.FABNETV4_SUBNET, next_hop=net.get_gateway())

slice.submit()

# Configure /etc/hosts for name resolution
etc_hosts = ""
for node in slice.get_nodes():
    node_ip = node.get_interface(network_name=f"{node.get_site()}-l3").get_ip_addr()
    etc_hosts += f"{node_ip} {node.get_name()}\n"

for node in slice.get_nodes():
    node.execute(f'sudo sh -c \'echo "{etc_hosts}" >> /etc/hosts\'')

# Install perfSONAR stack using perfsonar-extensions
for node in slice.get_nodes():
    node.execute(
        "git clone https://github.com/kthare10/perfsonar-extensions.git "
        "/home/ubuntu/perfsonar-extensions"
    )
    node.execute(
        "cd perfsonar-extensions/native/ && sudo scripts/perfsonar-install.sh",
        quiet=True,
        output_file=f"{node.get_name()}.log",
    )

# Allow Logstash IPs for cross-node archive access
ip_map = {}
central_host = ""
for node in slice.get_nodes():
    ip_addr = node.get_interface(network_name=f"{node.get_site()}-l3").get_ip_addr()
    ip_map[node.get_name()] = ip_addr
    if "central" in node.get_name():
        central_host = node.get_name()

all_ips = list(ip_map.values())
for node in slice.get_nodes():
    node.execute(
        f'cd perfsonar-extensions/native/ && '
        f'sudo scripts/allow_logstash_ips.sh {" ".join(map(str, all_ips))}'
    )

# Configure pSConfig mesh
for node in slice.get_nodes():
    all_hosts = [x for name, ip in ip_map.items() for x in (name, str(ip))]

    if central_host in node.get_name():
        # Central node: no tests, just archive
        cmd = (
            f"cd perfsonar-extensions/native/ && python3 psconfig/psconfig_builder.py "
            f"--base_config_file psconfig/base_psconfig.json "
            f"--output_file psconfig/psconfig.json --no_add_tests "
            f'--host_list {" ".join(map(str, all_hosts))}'
        )
    else:
        # Remote node: tests to central, with central archive
        cmd = (
            f"cd perfsonar-extensions/native/ && python3 psconfig/psconfig_builder.py "
            f"--base_config_file psconfig/base_psconfig.json "
            f"--output_file psconfig/psconfig.json "
            f"--remote {ip_map[central_host]} "
            f'--host_list {node.get_name()} {ip_map[node.get_name()]} '
            f'{central_host} {ip_map[central_host]}'
        )

    node.execute(cmd)
    node.execute("cd perfsonar-extensions/native/ && sudo psconfig validate psconfig/psconfig.json")
    node.execute("cd perfsonar-extensions/native/ && sudo psconfig publish psconfig/psconfig.json")
    node.execute('sudo psconfig remote add "https://localhost/psconfig/psconfig.json"')

Docker Testpoint Deployment

slice = fablib.new_slice(name="perfsonar-docker")

sites = fablib.get_random_sites(count=2)

node1 = slice.add_node(name="ps-node1", site=sites[0], cores=4, ram=8, disk=50, image="default_ubuntu_24")
node1.add_fabnet(net_type="IPv4")

node2 = slice.add_node(name="ps-node2", site=sites[1], cores=4, ram=8, disk=50, image="default_ubuntu_24")
node2.add_fabnet(net_type="IPv4")

slice.submit()

# Install Docker and run perfSONAR testpoint
for node in slice.get_nodes():
    commands = [
        "sudo apt-get update -qq",
        "sudo apt-get install -y -qq docker.io",
        "sudo systemctl enable docker",
        "sudo systemctl start docker",
        "sudo docker run -d --name perfsonar-testpoint "
        "--net=host --privileged "
        "-v /var/run "
        "perfsonar/testpoint:latest",
    ]
    for cmd in commands:
        node.execute(cmd)

    print(f"perfSONAR testpoint running on {node.get_name()}")

Running Ad-Hoc Tests with pScheduler

node1 = slice.get_node(name="ps-node1")
node2 = slice.get_node(name="ps-node2")

node2_ip = node2.get_interface(network_name="FABNET").get_ip_addr()

# Throughput test
stdout, stderr = node1.execute(
    f"pscheduler task throughput --dest {node2_ip} --duration PT10S"
)
print("Throughput result:", stdout)

# Latency test (RTT)
stdout, stderr = node1.execute(
    f"pscheduler task rtt --dest {node2_ip} --count 10"
)
print("RTT result:", stdout)

# Traceroute
stdout, stderr = node1.execute(
    f"pscheduler task trace --dest {node2_ip}"
)
print("Trace result:", stdout)

Grafana Dashboard for Results

# If using perfsonar-extensions, Grafana is installed automatically.
# Access it via the central node's management IP:

central_node = slice.get_node(name=central_host)
print(f"Grafana: http://[{central_node.get_management_ip()}]:3000")
print("Default credentials: admin/admin")

Useful Verification Commands

# Check pScheduler status
node.execute("pscheduler troubleshoot")

# View scheduled tests
node.execute("pscheduler schedule")

# Check archiving status (last 30 minutes)
node.execute("pscheduler archiving-summary PT30M")

# Validate pSConfig
node.execute("psconfig validate /path/to/psconfig.json")

Important Notes

Nodes need at least 16 cores and 32 GB RAM for the full perfsonar-extensions stack
Docker testpoint is lighter-weight (4 cores, 8 GB RAM minimum)
FABnet (L3) connectivity required for cross-site measurements
pScheduler port 443 must be accessible between testpoints
Use central and remote naming convention for the perfsonar-extensions setup
The central node collects archives from all remote nodes