Debugging processes and the daemon
Inspecting a single process
verdi process status <PK> # call stack and where execution stopped
verdi process report <PK> # log messages emitted during execution
verdi process show <PK> # inputs, outputs, exit code
verdi node show <PK> # node attributes and extras
For a full provenance dump including input/output files:
verdi process dump <PK> # dump a process and its provenance
For CalcJobs specifically, jump to the remote working directory on the HPC (requires SSH access, which may not be available for sandboxed agents):
verdi calcjob gotocomputer <PK>
Inspecting the daemon
# storage (PostgreSQL or SQLite) + daemon & broker (RabbitMQ, if configured):
verdi status
# tail daemon logs in real-time
# best with a single daemon worker, multiple workers garble output:
verdi daemon logshow
# requeue processes stuck after a daemon crash (stop the daemon first):
verdi process repair
Common failure modes
- Process stuck in
waiting: usually means the daemon lost track of it after a crash or restart. Runverdi process repairto requeue. - Process state inconsistent with node attributes : check whether
seal()has been called; only_updatable_attributescan change on a storedProcessNodebefore sealing. presto-marked test failures : these use an in-memorySqliteTempBackend, so the bug is in the code, not in service configuration.- Daemon subprocess killed on shutdown : daemon-launched subprocesses must pass
start_new_session=Trueor they inherit the daemon's signal handling and die with it.
Interactive inspection
verdi shell # interactive IPython shell with AiiDA loaded
verdi devel run-sql "SELECT ..." # run raw SQL against the profile database (USE WITH CAUTION)
Useful patterns inside verdi shell:
from aiida.orm import QueryBuilder, Node, load_node
# Find nodes by type
qb = QueryBuilder().append(Node, filters={'node_type': {'like': 'data.core.dict.%'}})
# Inspect a specific node
node = load_node(<PK>)
node.base.attributes.all # all stored attributes
node.base.extras.all # all extras (mutable even after storing)
node.base.repository.list_object_names() # files in the node's repository
Related source
- Process runner:
src/aiida/engine/runners.py - Daemon client:
src/aiida/engine/daemon/client.py - CalcJob exec manager (file copying, job submission, retrieval):
src/aiida/engine/daemon/execmanager.py - Transport tasks (submit, update, retrieve):
src/aiida/engine/processes/calcjobs/tasks.py