kabishou11 Dify troubleshooting
Instructions
Use this when Dify is up but something fails. Changing workers/timeouts/.env: Dify compose and config. Intranet/SSRF: Dify intranet. Plugin uv: Dify plugin install. Canvas/DSL: Dify apps and workflows. Prefixes: Dify API catalog.
Decide where it broke
| Symptom | Likely layer |
|---|---|
Connection refused on :80 |
nginx / compose / dockerd |
/install loops |
admin not created |
401 Invalid encrypted data |
password not Base64 |
401 CSRF / unauthorized |
missing CSRF on GET too, or session ~1h expired |
Unauthenticated / is 307 /signin |
1.17 WebApp; follow redirects or open /signin |
Server console API URL is not configured |
web missing SERVER_CONSOLE_API_URL=http://api:5001 |
nginx 502 host not found in upstream "api_websocket" |
websocket container down during nginx reload — start it first |
Agent missing from GET /apps |
roster is GET /agent |
Plugin red / uv exit status 1 |
daemon cannot reach PyPI / local index |
| UI "N failed tasks" but list ok | stale tasks; POST .../plugin/tasks/delete_all |
| Provider missing | plugin not local runtime ready |
| Upload 413 | nginx body size and UPLOAD_FILE_* |
.env changed, container unchanged |
1.17: api/worker/web/plugin_daemon/sandbox load .env; nginx/ssrf/weaviate/db still need listed keys. Optional knobs live in docker/envs/*.env. Recreate the service (nginx NGINX_* ≠ reload). |
HTTP / external KB 502 to 10./192.168. |
Squid SSRF; NO_PROXY on api+worker and/or SSRF_PROXY_ALLOW_PRIVATE_IPS as CIDR list (not true) |
MILVUS_USER is required |
set MILVUS_USER/MILVUS_PASSWORD in compose and .env |
minimax_group_id |
old MiniMax models need Group ID, or delete that default |
| File preview broken in plugins | INTERNAL_FILES_URL=http://api:5001 |
| Logged out / file URLs die | SECRET_KEY changed after boot (also kills model credential decrypt) |
| After reboot, nothing listens | nested boxes: manual dockerd, then compose up |
Draft save 400 / draft_workflow_not_sync |
stale workflow hash |
Draft save 400 environment_variables extra_forbidden |
1.17 dropped top-level environment_variables on graph sync. Send graph+features+hash; env edits use environment_variable_patch |
| Canvas "同步数据中" / React #130 | Socket.IO /socket.io/ missing, NEXT_PUBLIC_SOCKET_URL=localhost, or DSL nodes lack top-level type: custom |
| Plugin icons 503 | nginx console_limit burst; give /plugin/icon its own location without limit |
| Recreate api/web then 502 | nginx cached upstream IP → nginx -s reload |
403 /rbac /billing RAG publish |
community feature gate |
/agent 404 vs empty app list |
Studio /agent ≠ agent-chat |
Bearer on /console/api |
wrong surface |
Invalid upload file |
see service API (user + key + /v1 upload) |
| External KB low scores / zero recall | RAGFlow /dify/retrieval has no rerank; disable Dify score_threshold |
| External KB path 404 | endpoint must be .../dify because Dify appends /retrieval |
| LLM empty text | thinking model max_tokens too small (use ≥16384) or missing user message |
Tool Unknown error |
tool_name is not the OpenAPI operationId |
| Publish "无效的变量" on Loop | break_conditions missing id/varType, or they reference child outputs |
| Publish "视觉变量不能为空" | vision.enabled without configs |
| Publish "Rerank 模型不能为空" | rerank missing provider/model (UI) vs reranking_* (engine) — write all four |
credentials is not initialized |
model row on the wrong provider, or orphan provider_models after daemon restart |
CONSOLE_API_URL points at api:5001 in the browser |
leave CONSOLE_API_URL/APP_API_URL empty (relative via nginx) |
403 on /rbac |
community |
Empty /workflow-app-logs or /workflow-runs |
default triggered_from=debugging; cleanup already ran |
/workflow-app-logs 400 on a chat app |
route is mode=workflow |
Agent debug blocking 400 |
Studio is SSE-only |
Hit-test empty records |
embedding down / threshold / still indexing |
| Long workflow dies ~6 min | GUNICORN_TIMEOUT still 360 |
| Long workflow dies ~15s in code | SANDBOX_WORKER_TIMEOUT |
| Celery pegs all CPUs | CELERY_AUTO_SCALE without CELERY_MAX_WORKERS in worker env |
| Schedule never fires | beat down or worker -Q dropped schedule_* |
| 413 PDF but Dify limit is 20MB | nginx client_max_body_size still 100M / bind-mount not edited |
.env 200M nginx, live 100M |
listed compose env is not what nginx serves |
| New box canvas sync hang | NEXT_PUBLIC_SOCKET_URL still the old host |
| LLM 502 from a node | host not on NO_PROXY / Squid 5s |
Compose health
sudo docker compose ps
sudo docker compose logs --tail=80 api plugin_daemon nginx api_websocket
curl -sS http://127.0.0.1/console/api/setup
curl -sS http://127.0.0.1/console/api/version
# Socket.IO (expect 101 or 426, not 308)
curl -s -o /dev/null -w "%{http_code}" -H "Upgrade: websocket" -H "Connection: Upgrade" \
"http://127.0.0.1/socket.io/?EIO=4&transport=websocket"
Never compose down -v. After --force-recreate, reload nginx only if api_websocket is already up.
Env injection
Official 1.17: api / worker / beat / web / plugin_daemon / sandbox have env_file including ./.env. Custom clones often list the same keys in compose environment: (listed value wins) or bind-mount nginx/squid (then NGINX_* / SSRF_* never reach the process). Always docker exec <svc> printenv KEY. Procedure: Dify compose and config.
Plugins / models / RAG
Installed ≠ credentials saved. high_quality needs embedding. External KB: Dify knowledge bases. MiniMax is a cloud API — on an air-gap, remove it as the workspace default.
Nested Docker
whiteout … operation not permitted → fuse-overlayfs. ICC broken → bridge-nf-call-iptables.
Examples
New-box smoke (read-only on a live clone; apply only on the new compose dir):
curl -sS http://127.0.0.1/console/api/setup
curl -sS http://127.0.0.1:8001/v1/models
docker exec docker-api-1 printenv GUNICORN_TIMEOUT WORKFLOW_MAX_EXECUTION_TIME NO_PROXY
docker exec docker-nginx-1 nginx -T | grep -E 'client_max_body_size|proxy_read_timeout'
Performance Notes
Cap json-file in compose (50m × 3). Old containers may have unlimited logs until recreated; truncate *-json.log, do not restart dockerd to apply daemon log-opts.
Troubleshooting
The table above is the symptom index. Compose knobs: Dify compose and config.
Do not
Set DEPLOYMENT_EDITION=ENTERPRISE. Open container egress with iptables to "fix" marketplace. Delete volumes/ to fix one plugin. Put host passwords or SECRET_KEY into skills/git. Do not retune a live production stack as if it were empty.
Field-proven (2026-09-01)
| Symptom | Cause | Fix |
|---|---|---|
| Browser: workflow output file download says "cannot extract file"/404 | FILES_URL is a container-private origin → signed links bound to it for external viewers |
split FILES_URL (browser origin) vs INTERNAL_FILES_URL (http://api:5001), recreate api; see compose-and-config |
PaddleOCR tool: Invalid file URL '/files/...' |
tool got an origin-free URI; SDK tries httpx.get on it |
set INTERNAL_FILES_URL + patch plugin utils to prefix (prefer INTERNAL) |
| PaddleOCR-VL: job 422 "OCR服务请求失败" | outputFormats=markdown not supported by VL model |
remove the param |
Canvas publish: tool panel "cannot be empty" (e.g. force_text_value, title) |
parameters split by value kind instead of schema form; form=form must live in tool_configurations, form=llm in tool_parameters |
distribute by plugin yaml parameters[].form |
draft/run: client JSON parse fails though status: succeeded |
response is SSE | parse data: lines, read workflow_finished |
| LLM→code JSON parse yields empty arrays | model emitted unescaped quotes / duplicated bare key "k","k": |
chained repair in the code node (see apps-and-workflows DSL proven facts) |
| KB retrieval node empty but hit-test fine on one box | node multiple_retrieval_config still references the source box's reranker/embedding |
rewrite node config (weights) to this box; empty config silently returns [] |
| Knowledge retrieval minutes-slow intermittently | hosted embedding free tier throttling (burst) | retry later / switch embedding provider; keep retrieval timeouts sane; verify with hit-test timing |
no available node, plugin runtime not found right after daemon restart |
plugin runtime still initializing | wait for local runtime ready in daemon logs (~1 min) |
| Symptom | Cause | Fix |
|---|---|---|
| Plugin tool returns garbage for text-PDFs (digits/symbols only) | PyMuPDF mis-renders embedded CJK fonts | switch to pypdfium2 (page.render(scale=150/72).to_pil()); verify by exporting the rendered PNG |
| Service API key "disappeared" / run & test logs gone after shipping a new version | the app was deleted and re-imported — POST /apps/imports mints a new app_id; keys, site, logs, triggers are bound to the old id |
re-create the key on the new id and re-point callers; never ship versions this way — sync draft in place on the SAME id then publish (see apps-and-workflows red line) |
no available node, plugin runtime not found / api ConnectError ... plugin_daemon |
call landed in a plugin_daemon restart window | retry after daemon logs local runtime ready; transient by design |
| Canvas checklist "合同文件/文件 不能为空" on tool nodes | tool params placed in the wrong bucket | split by schema form: llm→tool_parameters, form→tool_configurations (see apps-and-workflows) |
| Bumped plugin version, behavior unchanged | workspace still bound to old installation | uninstall + reinstall same-id (see plugin-install) |
Messages.create() got an unexpected keyword argument 'temperature' (minimax) |
plugin 0.0.27 switched to Anthropic-compatible channel; SDK rejects temperature/top_p kwargs |
downgrade to 0.0.26 (pack from daemon cwd old dir, uninstall 0.0.27 preserve_credentials:true, install 0.0.26); regression = only nodes with sampling params fail, nodes without them work fine (source: _functional.py:97-101 sampling tuple) |