Troubleshooting

Every device fails authentication at once

If it worked yesterday, or works on another machine, check these in order.

1. config.yaml is missing. The most common cause, and it produces no error message. Netpilot silently falls back to built-in defaults (src/config.py:73) whose credential slots are named network_1 through network_4. Your real secrets are named something else, so every lookup misses.

Test-Path config.yaml

config.yaml is commonly git-ignored because it names your credential slots and secrets paths, so a fresh clone or a new machine starts with no config and no warning. Restore it. See the config.yaml reference.

2. Credentials were created by a different Windows account. DPAPI uses CurrentUser scope, so another account gets a decryption failure that looks exactly like a wrong password. This is why a setup that works by hand fails as a scheduled task under a service account.

3. Slot names don’t match the files. Every name in credentials.sets needs a matching {slot}.txt or CRED_{SLOT} variable. Missing slots are skipped silently, so a typo looks identical to a deliberate omission.

4. Secrets directory mismatch. setup_credentials.ps1 writes to a hardcoded D:\Prod_Automation\env\prod\secrets_v3 regardless of config.yaml. Point paths.secrets_dir_windows there, or set SECRETS_DIR, which overrides both.

Authentication failed on some devices

Cause: Wrong credential set for those specific devices, or a DPAPI file encrypted by a different Windows user.

Fix: Check which credential set those devices accept. If the file was created by a different user:

Caution

Re-running setup_credentials.ps1 overwrites existing credential files. Make sure you have the passwords ready before running.

cd src
.\setup_credentials.ps1

Always run as the same Windows user that runs Netpilot.

Discovery or capture finds zero devices and exits instantly

Cause: --role is an exact match, not a substring or prefix, and it defaults to core.

Fix: Pass the roles your inventory actually uses.

python -m src --discover-lldp                        # matches role=core ONLY
python -m src --discover-lldp --role cisco_core,alcatel_core   # what you probably meant

The same applies to --capture-core-stack. --vendor is different: it is a case-insensitive substring test, so --vendor alcatel matches alcatel_wlc too.

The run looks hung

Cause: --workers defaults to 1, so everything runs strictly sequentially. Three hundred devices at roughly 15 seconds each is over an hour. Unreachable devices are much worse, because each burns the full retry and backoff cost across every configured credential set before failing.

Fix: Pass --workers 30, or set execution.max_workers in config.yaml. A progress line prints as each host completes, so you can tell a slow run from a stuck one.

Note that --health-check ignores --workers entirely and always runs one device at a time.

Status is PARTIAL

Cause: The device connected and authenticated, then rejected one or more config commands. It is neither a success nor a connection failure. Almost always a firmware syntax difference — see the Alcatel notes below.

Fix: Read POST_CHECK/{hostname}.cfg to see what the device actually has, then correct the command file or add a role override at config/{vendor}/{role_suffix}/{task}.txt.

—dry-run prints devices but no commands

Cause: Expected. preview_commands looks for command files at the legacy flat path config/{vendor}_{task}.txt, while the real push reads the nested config/{vendor}/{task}.txt (src/utils.py:499).

Fix: Use --dry-run to check the device list, and --validate to check that command files resolve.

Compliance results look too clean

Cause: --baseline reads saved configs from BACKUP/, not live devices. Devices unreachable during the backup run are not evaluated at all, rather than counted as failures.

Fix: Check how many devices the backup job actually captured before trusting the summary.

Error reading SSH protocol banner

Every row comes back blank — blank hostname, blank serial, stack_count=1 — while the run still prints a summary that looks successful. Blank hostname is the tell: it is read from the CLI prompt, so it is only ever empty when SSH never opened.

Cause 1 — legacy SSH server rejects paramiko’s identification string. Paramiko announces itself as SSH-2.0-paramiko_<version>. Alcatel AOS6 (OS6450/OS6350, fronting OpenSSH_5.0) and Cisco IOS XR close the socket without sending their own banner when they see it. The giveaway: logging in manually with an OpenSSH client works from the same machine, seconds apart.

Fix: Netpilot announces OpenSSH_9.5 by default (connection.ssh_client_id). If a device still refuses, try another value:

connection:
  ssh_client_id: "OpenSSH_8.9p1"

Confirm the device’s own SSH version first:

ssh -v admin@10.0.0.1 2>&1 | Select-String "remote software version"

Cause 2 — transient network issue. Retry only the failed devices:

python -m src --task SNMPV3 --commands snmpv3 --retry --workers 10

Command read timeout

Cause: Device is slow to respond to commands.

Fix: Netpilot already uses send_command_timing for Alcatel devices. If timeouts persist, reduce --workers to lower concurrent load on the device.

Failed to read credential files

Cause: Credential file missing or encrypted by a different Windows user account.

Fix: Re-run setup_credentials.ps1 as the correct user. See Credential setup.

No IP column found

Cause: Status CSV passed to --status-file does not have a host or ip column.

Fix: Add a host column to your CSV, or rename the existing IP column to host.

1344 invalid devices (or similar count)

Cause: Device hostnames contain parentheses or other special characters that earlier versions rejected.

Fix: Upgrade to the latest release — parentheses are stripped automatically in update_inventory.py.

---

Alcatel-specific notes

No configure terminal mode

AOS OmniSwitches use direct commands only. Do not add configure terminal to Alcatel command files.

CERTIFY mode banner

Some switches print Setting CERTIFY Timeout for 800 seconds on login. Netpilot handles this automatically — no action needed.

poe_switch vs dataswitch

Older firmware (poe_switch role) uses different syntax:

Feature	alcatel_dataswitch	alcatel_poeswitch
AES	sha256+aes256	sha256+aes
Trap	snmp authentication trap enable	snmp authentication-trap enable

Using the wrong file causes silent misconfiguration. Verify the role in devices.csv.

AOS-W WLC syntax

WLCs use auth-prot sha / priv-prot aes — not Cisco-style auth sha / priv aes. See the Alcatel WLC vendor page.

AOS-W paging and version capture

show inventory paginates on AOS-W. Netpilot quits the pager rather than waiting for a timeout. AOS-W also has no Software Version: line in the shape other platforms use, so version is recovered from the inventory Description field.

WLCs have no virtual-chassis concept, so stack capture always produces exactly one row with switch_number of 1.

---

Blank serial or version in stack capture

version and uptime capture is best-effort. If the version command fails on a model, serial and model still land and those two columns come out blank.

If serial is blank too, the chassis command itself failed. Classic StackWise Cisco switches where show module is not the right command fall back to show switch, which returns a member count only, without per-unit serials.

Interrupting a long run

Safe. With --apply, discovery results are streamed to disk as each source device finishes, so Ctrl+C keeps everything already found, and re-running skips what is already recorded.

--export-atlas is the exception: it is written once at the end, so an interrupted run produces no Atlas file.