OVS Database and Flow Analysis

OVS Database and Flow Analysis

This skill provides detailed analysis of Open vSwitch (OVS) data collected via sosreport. It operates in four modes:

1. Default mode: Full analysis - conf.db + all text files (requires ovsdb-tool, falls back to text files if not available)
2. Database mode (--db): Database only - analyze conf.db (requires ovsdb-tool)
3. Text files mode (--flows-only): Text files only - no ovsdb-tool needed
4. Query mode (--query): Run custom OVSDB JSON queries (requires ovsdb-tool)

When to Use This Skill

Use this skill when:

• Troubleshooting packet drops - Find drop flows being hit
• Analyzing flow performance - View top flows by packet count
• Analyzing OVS configuration from a sosreport
• Troubleshooting bridge, port, or interface issues
• Investigating DPDK vs kernelspace datapath
• Reviewing tunnel configurations (VXLAN, Geneve, GRE)
• Checking port statistics (drops, errors, traffic)
• Mapping Kubernetes pods to OVS interfaces

Prerequisites

Default mode (full analysis):

• ovsdb-tool must be installed (from openvswitch package)
• Falls back to --flows-only if ovsdb-tool not found
• Sosreport with sos_commands/openvswitch/ directory and conf.db

Database mode (--db):

• ovsdb-tool must be installed (from openvswitch package)
  • Check: which ovsdb-tool
  • Fedora/RHEL: sudo dnf install openvswitch
  • Ubuntu/Debian: sudo apt install openvswitch-common
• Sosreport with conf.db file

Text files mode (--flows-only):

• No special tools needed - works out of the box!
• Sosreport with sos_commands/openvswitch/ directory

Query mode (--query):

• ovsdb-tool must be installed
• Incompatible with --flows-only

File Locations in Sosreport

The sosreport collects:

sosreport-hostname-date/
├── etc/openvswitch/conf.db              (OVS database - for --db mode)
├── var/lib/openvswitch/conf.db          (alternate location)
└── sos_commands/openvswitch/            (default mode uses these)
    ├── ovs-vsctl_-t_5_show              (topology)
    ├── ovs-vsctl_-t_5_list_bridge       (bridge details)
    ├── ovs-vsctl_-t_5_list_interface    (interface details)
    ├── ovs-vsctl_-t_5_list_Open_vSwitch (system info)
    ├── ovs-ofctl_dump-flows_<bridge>    (OpenFlow entries)
    ├── ovs-ofctl_dump-ports_<bridge>    (Port statistics)
    ├── ovs-appctl_coverage.show         (internal stats)
    ├── ovs-appctl_upcall.show           (datapath health)
    ├── ovs-appctl_tnl.ports.show_-v     (tunnel ports)
    └── ...

Analysis Script

The analysis script is located at:

<plugin-root>/skills/ovs-db-analysis/scripts/analyze_ovs_db.py

Implementation Steps

Step 1: Locate the Analysis Script

SCRIPT_PATH=$(find ~ -name "analyze_ovs_db.py" -path "*/sosreport/skills/ovs-db-analysis/scripts/*" 2>/dev/null | head -1)

if [ -z "$SCRIPT_PATH" ]; then
    echo "ERROR: analyze_ovs_db.py script not found."
    exit 1
fi

Step 2: Run the Analysis

# Default: Full analysis - conf.db + text files (requires ovsdb-tool)
python3 "$SCRIPT_PATH" /path/to/sosreport-hostname-date/

# Analyze from archive
python3 "$SCRIPT_PATH" /path/to/sosreport-hostname-date.tar.xz

# Database only mode (requires ovsdb-tool)
python3 "$SCRIPT_PATH" /path/to/sosreport/ --db

# Text files only mode (no ovsdb-tool needed)
python3 "$SCRIPT_PATH" /path/to/sosreport/ --flows-only

Step 3: Custom Queries (Optional, requires --db)

For specific investigations, use raw OVSDB queries:

# Query all bridges
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Bridge", "where":[], "columns":["name","datapath_type","fail_mode"]}]'

# Query VXLAN interfaces
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Interface", "where":[["type","==","vxlan"]], "columns":["name","options"]}]'

# Query interfaces with errors
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Interface", "where":[], "columns":["name","error"]}]'

Analysis Output

The default mode analyzes:

OpenFlow Analysis Features

OVN Internal Tables (Ignore Drops in These)

When analyzing drop flows in OVN-managed bridges (br-int), ignore drops in table 44 and tables 64-87 as these are internal OVN mechanics, not security policy or real packet drops.

⚠️ Note: OVN and OVS are complex systems and table mappings can change between releases. Always verify drop analysis against the specific OVN version in use. When in doubt, check the OVN source code for your release.

Relevant drop tables for troubleshooting:

• Table 9: Ingress ACLs - actual policy drops (look for reg0=0x8000)
• Table 0: Initial classification drops
• Tables < 64: Generally meaningful drops

Common Analysis Scenarios

1. Troubleshooting Packet Drops

# Quick analysis (default mode, no ovsdb-tool)
python3 "$SCRIPT_PATH" /path/to/sosreport/

Look for:

• ACTIVE DROP FLOWS section showing flows dropping packets
• PORT STATISTICS showing ports with drops/errors

2. Checking DPDK Configuration

# Default mode shows DPDK info from text files
python3 "$SCRIPT_PATH" /path/to/sosreport/

# Or use database mode for custom queries
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Open_vSwitch", "where":[], "columns":["dpdk_initialized","other_config"]}]'

3. Investigating Pod Connectivity

The interface analysis shows pod-to-OVS mapping:

• Interfaces with external_ids containing iface-id
• Interface names ending with _h (veth host side)

4. Datapath Health Check

The upcall stats show:

DATAPATH FLOW TABLE HEALTH
  Current flows: 155 / 200,000 (0.1% used)
  Average: 156, Max seen: 215
  ✓ Flow table healthy

If usage > 90%, flows are being evicted too aggressively.

Output Example

================================================================================
OVS ANALYSIS - sosreport-hostname-2024-01-15
================================================================================
Mode: Text file analysis (no ovsdb-tool required)

================================================================================
OVS SYSTEM INFORMATION
================================================================================

  Field                     Value
  ------------------------- --------------------------------------------------
  OVS Version               "3.3.4-62.el9fdp"
  DB Version                "8.5.0"
  System Type               rhcos
  System Version            "4.16"
  DPDK Initialized          false
  Datapath Types            [netdev, system]

  External IDs:
    hostname: master2.example.com
    ovn-encap-ip: 10.32.110.5
    ovn-encap-type: geneve

================================================================================
OVS TOPOLOGY
================================================================================

  System UUID: 7e9a3f70-86fa-4578-a849-4fd807a64a10
  Total Bridges: 2

  Bridge: br-ex
    ports: 3
      internal: br-ex
      patch: patch-br-ex-to-br-int
      system: nm-bond

  Bridge: br-int
    fail_mode: secure
    datapath_type: system
    ports: 12
      geneve: 9 ports
      internal: ovn-k8s-mp0, br-int
      patch: patch-br-int-to-br-ex

================================================================================
OPENFLOW ANALYSIS
================================================================================

  Bridge: br-int
  ----------------------------------------------------------------------
  Total flows: 2,017
  Flows with hits: 318
  Drop flows: 150 (9 actively dropping)

  ⚠️  ACTIVE DROP FLOWS (9):
    table=40, priority=0, packets=8,105
    table=79, priority=100, packets=1,356
      match: ip,reg14=0x2,metadata=0x5,dl_src=00:62:0b:ea:b5:e0

--------------------------------------------------------------------------------
DATAPATH FLOW TABLE HEALTH
--------------------------------------------------------------------------------

  Current flows: 155 / 200,000 (0.1% used)
  Average: 156, Max seen: 215
  ✓ Flow table healthy

Error Handling

See Also

• OVN Database Analysis - For OVN NB/SB databases
• sosreport openvswitch plugin - What sosreport collects