mirivlad/ducklm
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add DuckLM service scripts and utility model benchmark

Browse Source
This commit is contained in:
mirivlad
2026-05-22 07:27:59 +08:00
parent e6b82f0376
commit ff98224eb6
24 changed files with 2492 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/bench/utility_model_bench_20260522_044407.json
+80
View File
@@ -0,0 +1,80 @@
[
{
"model": "Qwen3.6-35B nonMTP GPU baseline",
"quality": 0.971,
"avg_latency_seconds": 17.935,
"avg_tokens_per_second": 4.51,
"cases": [
{
"role": "action",
"case": "direct_answer_no_tools",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 15.321,
"completion_tokens": 45,
"tokens_per_second": 2.94,
"content_preview": "{\n \"kind\": \"action_directive\",\n \"intent\": \"direct_answer\",\n \"risk_level\": \"low\",\n \"actions\": []\n}\n"
},
{
"role": "action",
"case": "read_specific_file",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 19.638,
"completion_tokens": 81,
"tokens_per_second": 4.12,
"content_preview": "{\n \"kind\": \"action_directive\",\n \"intent\": \"read_file\",\n \"risk_level\": \"low\",\n \"actions\": [\n {\n \"tool\": \"file_read\",\n \"args\": {\n \"path\": \"CURRENT_STATE.md\"\n }\n }\n ]\n}\n"
},
{
"role": "memory_policy",
"case": "store_user_preference",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 18.421,
"completion_tokens": 88,
"tokens_per_second": 4.78,
"content_preview": "{\n \"should_store\": true,\n \"memory_type\": \"preference\",\n \"summary\": \"User prefers responses in Russian and requires explicit confirmation before executing sudo commands.\",\n \"importance\": 0.9,\n \"scope\": \"global\",\n \"metadata\": {\n \"language\": \"ru\",\n \"security_policy\": \"sudo_confirmation_required\"\n }\n}"
},
{
"role": "memory_policy",
"case": "ignore_trivial_tool_call",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 14.978,
"completion_tokens": 61,
"tokens_per_second": 4.07,
"content_preview": "{\n \"should_store\": false,\n \"memory_type\": \"note\",\n \"summary\": \"Routine execution of pwd command returning /tmp/project.\",\n \"importance\": 0.1,\n \"scope\": \"conversation\",\n \"metadata\": {}\n}"
},
{
"role": "recall",
"case": "select_relevant_memory",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 15.038,
"completion_tokens": 66,
"tokens_per_second": 4.39,
"content_preview": "{\n \"relevant_ids\": [\n \"m1\"\n ],\n \"reasoning\": \"Memory m1 directly addresses the user's preference regarding the execution of sudo commands, which is the core of the query. Memories m2 and m3 are unrelated to sudo or command execution preferences.\"\n}"
},
{
"role": "summary",
"case": "preserve_decisions",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 9.991,
"completion_tokens": 44,
"tokens_per_second": 4.4,
"content_preview": "1. WebChat работает на 8000, llama-server на 8081.\n2. Для внешних путей нужен approval.\n3. allow_forever хранится по normalized action hash."
},
{
"role": "critic",
"case": "reflection_quality",
"score": 0.8,
"note": "missing=['lesson']",
"elapsed_seconds": 32.16,
"completion_tokens": 220,
"tokens_per_second": 6.84,
"content_preview": "**Critic Reflection: WebChat API Incident**\n\n**1. Risk Assessment**\n\n* **Operational Fragility (High):** The incident revealed a critical dependency on manual intervention. If the API crashes or the server reboots, WebChat will silently fail or return errors until an engineer notices and manually restarts the service. This creates a \"single point of failure\" in the operational process.\n* **Meaning Time to Recovery (MTTR) Variance:** Recovery time is currently dependent on human availability "
}
]
}
]
docs/bench/utility_model_bench_20260522_044407.md
+21
View File
@@ -0,0 +1,21 @@
# Utility Role Model Benchmark
Scope: service roles only (`action`, `memory_policy`, `recall`, `summary`, `critic`).
The main user-facing thinker is not evaluated for replacement here.
| Model | Quality | Avg latency, s | Avg tok/s | Notes |
| --- | ---: | ---: | ---: | --- |
| Qwen3.6-35B nonMTP GPU baseline | 0.97 | 17.93 | 4.51 | critic/reflection_quality: missing=['lesson'] |
## Case Details
### Qwen3.6-35B nonMTP GPU baseline
| Role | Case | Score | Latency, s | tok/s | Note |
| --- | --- | ---: | ---: | ---: | --- |
| action | direct_answer_no_tools | 1.00 | 15.32 | 2.94 | ok |
| action | read_specific_file | 1.00 | 19.64 | 4.12 | ok |
| memory_policy | store_user_preference | 1.00 | 18.42 | 4.78 | ok |
| memory_policy | ignore_trivial_tool_call | 1.00 | 14.98 | 4.07 | ok |
| recall | select_relevant_memory | 1.00 | 15.04 | 4.39 | ok |
| summary | preserve_decisions | 1.00 | 9.99 | 4.40 | ok |
| critic | reflection_quality | 0.80 | 32.16 | 6.84 | missing=['lesson'] |
docs/bench/utility_model_bench_20260522_050427.json
+392
View File
@@ -0,0 +1,392 @@
[
{
"model": "Qwen3.6-35B nonMTP GPU baseline",
"quality": 0.971,
"avg_latency_seconds": 17.94,
"avg_tokens_per_second": 4.51,
"cases": [
{
"role": "action",
"case": "direct_answer_no_tools",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 15.307,
"completion_tokens": 45,
"tokens_per_second": 2.94,
"content_preview": "{\n \"kind\": \"action_directive\",\n \"intent\": \"direct_answer\",\n \"risk_level\": \"low\",\n \"actions\": []\n}\n"
},
{
"role": "action",
"case": "read_specific_file",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 19.612,
"completion_tokens": 81,
"tokens_per_second": 4.13,
"content_preview": "{\n \"kind\": \"action_directive\",\n \"intent\": \"read_file\",\n \"risk_level\": \"low\",\n \"actions\": [\n {\n \"tool\": \"file_read\",\n \"args\": {\n \"path\": \"CURRENT_STATE.md\"\n }\n }\n ]\n}\n"
},
{
"role": "memory_policy",
"case": "store_user_preference",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 18.533,
"completion_tokens": 88,
"tokens_per_second": 4.75,
"content_preview": "{\n \"should_store\": true,\n \"memory_type\": \"preference\",\n \"summary\": \"User prefers responses in Russian and requires explicit confirmation before executing sudo commands.\",\n \"importance\": 0.9,\n \"scope\": \"global\",\n \"metadata\": {\n \"language\": \"ru\",\n \"security_policy\": \"sudo_confirmation_required\"\n }\n}"
},
{
"role": "memory_policy",
"case": "ignore_trivial_tool_call",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 15.0,
"completion_tokens": 61,
"tokens_per_second": 4.07,
"content_preview": "{\n \"should_store\": false,\n \"memory_type\": \"note\",\n \"summary\": \"Routine execution of pwd command returning /tmp/project.\",\n \"importance\": 0.1,\n \"scope\": \"conversation\",\n \"metadata\": {}\n}"
},
{
"role": "recall",
"case": "select_relevant_memory",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 15.085,
"completion_tokens": 66,
"tokens_per_second": 4.38,
"content_preview": "{\n \"relevant_ids\": [\n \"m1\"\n ],\n \"reasoning\": \"Memory m1 directly addresses the user's preference regarding the execution of sudo commands, which is the core of the query. Memories m2 and m3 are unrelated to sudo or command execution preferences.\"\n}"
},
{
"role": "summary",
"case": "preserve_decisions",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 9.954,
"completion_tokens": 44,
"tokens_per_second": 4.42,
"content_preview": "1. WebChat работает на 8000, llama-server на 8081.\n2. Для внешних путей нужен approval.\n3. allow_forever хранится по normalized action hash."
},
{
"role": "critic",
"case": "reflection_quality",
"score": 0.8,
"note": "missing=['lesson']",
"elapsed_seconds": 32.088,
"completion_tokens": 220,
"tokens_per_second": 6.86,
"content_preview": "**Critic Reflection: WebChat API Incident**\n\n**1. Risk Assessment**\n\n* **Operational Fragility (High):** The incident revealed a critical dependency on manual intervention. If the API crashes or the server reboots, WebChat will silently fail or return errors until an engineer notices and manually restarts the service. This creates a \"single point of failure\" in the operational process.\n* **Meaning Time to Recovery (MTTR) Variance:** Recovery time is currently dependent on human availability "
}
]
},
{
"model": "Menlo_Lucy-Q4_K_M CPU",
"quality": 0.771,
"avg_latency_seconds": 4.406,
"avg_tokens_per_second": 16.21,
"cases": [
{
"role": "action",
"case": "direct_answer_no_tools",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 3.228,
"completion_tokens": 31,
"tokens_per_second": 9.6,
"content_preview": "{\"kind\": \"action_directive\", \"intent\": \"answer_question\", \"risk_level\": \"low\", \"actions\": []}"
},
{
"role": "action",
"case": "read_specific_file",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 3.031,
"completion_tokens": 48,
"tokens_per_second": 15.84,
"content_preview": "{\"kind\": \"action_directive\", \"intent\": \"file_read\", \"risk_level\": \"low\", \"actions\": [{\"tool\": \"file_read\", \"args\": {\"path\": \"CURRENT_STATE.md\"}}]}\n\n"
},
{
"role": "memory_policy",
"case": "store_user_preference",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 3.62,
"completion_tokens": 54,
"tokens_per_second": 14.92,
"content_preview": "{\n \"should_store\": true,\n \"memory_type\": \"preference\",\n \"summary\": \"User prefers Russian and requires sudo confirmation.\",\n \"importance\": 0.7,\n \"scope\": \"global\",\n \"metadata\": {}\n}"
},
{
"role": "memory_policy",
"case": "ignore_trivial_tool_call",
"score": 0.3,
"note": "stored_trivial={'should_store': True, 'memory_type': 'fact', 'summary': 'Password was successfully launched and user was informed.', 'importance': 0.7, 'scope': 'global', 'metadata': {}}",
"elapsed_seconds": 3.192,
"completion_tokens": 58,
"tokens_per_second": 18.17,
"content_preview": "{\n \"should_store\": true,\n \"memory_type\": \"fact\",\n \"summary\": \"Password was successfully launched and user was informed.\",\n \"importance\": 0.7,\n \"scope\": \"global\",\n \"metadata\": {}\n}\n"
},
{
"role": "recall",
"case": "select_relevant_memory",
"score": 0.3,
"note": "wrong_ids=[]",
"elapsed_seconds": 3.737,
"completion_tokens": 60,
"tokens_per_second": 16.05,
"content_preview": "{\n \"relevant_ids\": [],\n \"reasoning\": \"The query is about how the user wants to run sudo, but none of the provided memories are related to sudo or user preferences for running commands. The memories are about SQLite and weather answers, which are unrelated to the query.\"\n}"
},
{
"role": "summary",
"case": "preserve_decisions",
"score": 0.8,
"note": "missing=['approval']",
"elapsed_seconds": 3.334,
"completion_tokens": 61,
"tokens_per_second": 18.29,
"content_preview": "1. WebChat работает на портах 8000, а LLaMA-Server на 8081. \n2. Для внешних путей требуется подтверждение. \n3. allow_forever хранится по hash-коду действий."
},
{
"role": "critic",
"case": "reflection_quality",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 10.697,
"completion_tokens": 220,
"tokens_per_second": 20.57,
"content_preview": "Okay, the user wants me to reflect on the risk and reusable lessons from fixing the WebChat issue. Let me break this down.\n\nFirst, the root cause was the API not running. They manually started the API but didn't have a unified service script. So the main risk here is that without a unified script, there's a chance the API might not be running consistently or could be misconfigured.\n\nReusability is key here. Maybe we can create a script that starts the API and monitors its status. That way, if th"
}
]
},
{
"model": "Qwen3.5-9B-GLM5.1-Distill-v1-Q4_K_M CPU",
"quality": 0.4,
"avg_latency_seconds": 61.939,
"avg_tokens_per_second": 2.56,
"cases": [
{
"role": "action",
"case": "direct_answer_no_tools",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 68.075,
"completion_tokens": 72,
"tokens_per_second": 1.06,
"content_preview": "{\"kind\": \"action_directive\", \"intent\": \"answer\", \"risk_level\": \"high\", \"actions\": []}"
},
{
"role": "action",
"case": "read_specific_file",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 72.149,
"completion_tokens": 86,
"tokens_per_second": 1.19,
"content_preview": "{\"kind\": \"action_directive\", \"intent\": \"read file\", \"risk_level\": \"low\", \"actions\": [{\"tool\": \"file_read\", \"args\": {\"path\": \"CURRENT_STATE.md\"}}]}\n"
},
{
"role": "memory_policy",
"case": "store_user_preference",
"score": 0.0,
"note": "invalid_json: Expecting value: line 1 column 1 (char 0)",
"elapsed_seconds": 67.764,
"completion_tokens": 180,
"tokens_per_second": 2.66,
"content_preview": ""
},
{
"role": "memory_policy",
"case": "ignore_trivial_tool_call",
"score": 0.0,
"note": "invalid_json: Expecting value: line 1 column 1 (char 0)",
"elapsed_seconds": 64.653,
"completion_tokens": 160,
"tokens_per_second": 2.47,
"content_preview": ""
},
{
"role": "recall",
"case": "select_relevant_memory",
"score": 0.0,
"note": "invalid_json: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)",
"elapsed_seconds": 59.451,
"completion_tokens": 160,
"tokens_per_second": 2.69,
"content_preview": "{\n"
},
{
"role": "summary",
"case": "preserve_decisions",
"score": 0.2,
"note": "missing=['8000', '8081', 'approval', 'allow_forever']",
"elapsed_seconds": 47.05,
"completion_tokens": 180,
"tokens_per_second": 3.83,
"content_preview": ""
},
{
"role": "critic",
"case": "reflection_quality",
"score": 0.6,
"note": "missing=['risk', 'lesson']",
"elapsed_seconds": 54.433,
"completion_tokens": 220,
"tokens_per_second": 4.04,
"content_preview": ""
}
]
},
{
"model": "X-Coder-SFT-Qwen3-8B.Q6_K CPU",
"quality": 0.757,
"avg_latency_seconds": 60.117,
"avg_tokens_per_second": 2.51,
"cases": [
{
"role": "action",
"case": "direct_answer_no_tools",
"score": 0.0,
"note": "invalid_json: Expecting ',' delimiter: line 13 column 6 (char 632)",
"elapsed_seconds": 121.049,
"completion_tokens": 180,
"tokens_per_second": 1.49,
"content_preview": "{\n \"kind\": \"action_directive\",\n \"intent\": \"answer\",\n \"risk_level\": \"low\",\n \"actions\": [\n {\n \"tool\": \"coder\",\n \"args\": {\n \"task_description\": \"Объясните коротко, что такое DuckLM\",\n \"language\": \"python\",\n \"context\": \"Пользователь запросил краткое описание DuckLM. DuckLM - это проект, который развивается сообществом, направленный на создание открытой и этичной языковой модели. Мы можем сказать, что это модель на основе Llama, созданная Meta, но адаптированная "
},
{
"role": "action",
"case": "read_specific_file",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 37.562,
"completion_tokens": 134,
"tokens_per_second": 3.57,
"content_preview": "{\n \"kind\": \"action_directive\",\n \"intent\": \"read_current_state\",\n \"risk_level\": \"low\",\n \"actions\": [\n {\n \"tool\": \"file_read\",\n \"args\": {\n \"path\": \"CURRENT_STATE.md\"\n },\n \"reason\": \"We need to read the content of CURRENT_STATE.md to provide a summary of the project status.\"\n }\n ],\n \"expected_observations\": [\n \"file_content\"\n ],\n \"stop_reason\": \"After reading the file, we can directly generate the response without further tools.\"\n}\n"
},
{
"role": "memory_policy",
"case": "store_user_preference",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 66.979,
"completion_tokens": 80,
"tokens_per_second": 1.19,
"content_preview": "{\n \"should_store\": true,\n \"memory_type\": \"preference\",\n \"summary\": \"User prefers responses in Russian and explicit sudo confirmation.\",\n \"importance\": 0.9,\n \"scope\": \"global\",\n \"metadata\": {\n \"language\": \"Russian\",\n \"sudo_policy\": \"explicit_confirmation_required\"\n }\n}\n"
},
{
"role": "memory_policy",
"case": "ignore_trivial_tool_call",
"score": 0.3,
"note": "stored_trivial={'should_store': True, 'memory_type': 'event', 'summary': 'User executed pwd command and received /tmp/project as output.', 'importance': 0.8, 'scope': 'conversation', 'metadata': {}}",
"elapsed_seconds": 21.772,
"completion_tokens": 62,
"tokens_per_second": 2.85,
"content_preview": "{\n \"should_store\": true,\n \"memory_type\": \"event\",\n \"summary\": \"User executed pwd command and received /tmp/project as output.\",\n \"importance\": 0.8,\n \"scope\": \"conversation\",\n \"metadata\": {}\n}\n"
},
{
"role": "recall",
"case": "select_relevant_memory",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 58.656,
"completion_tokens": 90,
"tokens_per_second": 1.53,
"content_preview": "{\n \"relevant_ids\": [\"m1\"],\n \"reasoning\": \"The query is about how the user wants to run 'sudo'. The only memory that directly relates to this is memory m1, which states that the user does not want to run sudo without explicit confirmation. The other memories (m2 and m3) are about unrelated topics (database and weather). Therefore, only m1 is relevant.\"\n}\n"
},
{
"role": "summary",
"case": "preserve_decisions",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 53.245,
"completion_tokens": 180,
"tokens_per_second": 3.38,
"content_preview": "We are going to compress the task context into 3 key points while preserving the decisions and outcomes.\n\nKey Points:\n1. **Service Endpoints**: \n - WebChat is running on port 8000.\n - llama-server is running on port 8081.\n\n2. **External Paths Approval**:\n - Any external path that requires access must be approved. This implies that we need to have a mechanism (like a firewall or access control list) to check and approve external requests.\n\n3. **Forever Allowance Storage**:\n - The `allow_f"
},
{
"role": "critic",
"case": "reflection_quality",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 61.555,
"completion_tokens": 220,
"tokens_per_second": 3.57,
"content_preview": "We are reflecting on the task: fixing the WebChat which had an API that wasn't running.\n The root cause was that the API was not running. We started it manually but there was no unified service script.\n\n Reflection:\n - Risk: The manual start might be error-prone and not scalable. If multiple components, hard to manage.\n - Waste: We ended up having to start manually, which is not efficient and might lead to repeated issues.\n - Reusable Lesson: We should create a unified service script that starts"
}
]
},
{
"model": "gemma-4-E4B-it-Q4_K_M CPU",
"quality": 0.971,
"avg_latency_seconds": 21.235,
"avg_tokens_per_second": 5.36,
"cases": [
{
"role": "action",
"case": "direct_answer_no_tools",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 35.717,
"completion_tokens": 53,
"tokens_per_second": 1.48,
"content_preview": "{\n \"kind\": \"action_directive\",\n \"intent\": \"answer_directly\"\n ,\n \"risk_level\": \"low\"\n ,\n \"actions\": []\n}\n"
},
{
"role": "action",
"case": "read_specific_file",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 13.325,
"completion_tokens": 88,
"tokens_per_second": 6.6,
"content_preview": "{\n \"kind\": \"action_directive\",\n \"intent\": \"tool_use\",\n \"risk_level\": \"low\",\n \"actions\": [\n {\n \"tool\": \"file_read\",\n \"args\": {\n \"path\": \"CURRENT_STATE.md\"\n }\n }\n ]\n}\n"
},
{
"role": "memory_policy",
"case": "store_user_preference",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 27.13,
"completion_tokens": 98,
"tokens_per_second": 3.61,
"content_preview": "{\n \"should_store\": true,\n \"memory_type\": \"preference\",\n \"summary\": \"Пользователь требует, чтобы ответы всегда были на русском языке и чтобы не запускать sudo без явного подтверждения.\",\n \"importance\": 0.95,\n \"scope\": \"global\",\n \"metadata\": {\n \"task_id\": \"task_pref\"\n }\n}\n"
},
{
"role": "memory_policy",
"case": "ignore_trivial_tool_call",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 10.229,
"completion_tokens": 90,
"tokens_per_second": 8.8,
"content_preview": "{\n \"should_store\": false,\n \"memory_type\": \"note\",\n \"summary\": \"The 'pwd' command was executed, and the output was sent to the user.\",\n \"importance\": 0.1,\n \"scope\": \"conversation\",\n \"metadata\": {\n \"task_id\": \"task_tmp\"\n }\n}\n"
},
{
"role": "recall",
"case": "select_relevant_memory",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 19.391,
"completion_tokens": 62,
"tokens_per_second": 3.2,
"content_preview": "{\n \"relevant_ids\": [\n \"m1\"\n ],\n \"reasoning\": \"Воспоминание m1 напрямую отвечает на вопрос о том, как пользователь хочет, чтобы запускался sudo (с требованием отдельного подтверждения).\"\n}\n"
},
{
"role": "summary",
"case": "preserve_decisions",
"score": 1.0,
"note": "ok",
"elapsed_seconds": 14.371,
"completion_tokens": 88,
"tokens_per_second": 6.12,
"content_preview": "Вот сжатый контекст в 3 пунктах:\n\n1. **Порты:** WebChat использует порт 8000, llama-server — 8081.\n2. **Безопасность:** Для внешних путей требуется предварительное одобрение (approval).\n3. **Хранение разрешений:** `allow_forever` сохраняется на основе хеша нормализованного действия."
},
{
"role": "critic",
"case": "reflection_quality",
"score": 0.8,
"note": "missing=['lesson']",
"elapsed_seconds": 28.484,
"completion_tokens": 220,
"tokens_per_second": 7.72,
"content_preview": "## DuckLM: Critic Reflection\n\n**Task:** Fix WebChat.\n**Observed Root Cause:** API was not running.\n**Action Taken:** Started API manually.\n**Observed Deficiency:** Lack of a unified service script.\n\n---\n\n### 🔍 Reflection Analysis\n\n#### 1. Risk Assessment (What went wrong/could go wrong?)\n\n* **Operational Risk (High):** The immediate risk was service unavailability (WebChat down). Manually starting the API is a brittle, high-touch workaround. If the system restarts, or if the API needs to be re"
}
]
}
]
docs/bench/utility_model_bench_20260522_050427.md
+69
View File
@@ -0,0 +1,69 @@
# Utility Role Model Benchmark
Scope: service roles only (`action`, `memory_policy`, `recall`, `summary`, `critic`).
The main user-facing thinker is not evaluated for replacement here.
| Model | Quality | Avg latency, s | Avg tok/s | Notes |
| --- | ---: | ---: | ---: | --- |
| Qwen3.6-35B nonMTP GPU baseline | 0.97 | 17.94 | 4.51 | critic/reflection_quality: missing=['lesson'] |
| Menlo_Lucy-Q4_K_M CPU | 0.77 | 4.41 | 16.21 | memory_policy/ignore_trivial_tool_call: stored_trivial={'should_store': True, 'memory_type': 'fact', 'summary': 'Password was successfully launched and user was informed.', 'importance': 0.7, 'scope': 'global', 'metadata': {}}; recall/select_relevant_memory: wrong_ids=[]; summary/preserve_decisions: missing=['approval'] |
| Qwen3.5-9B-GLM5.1-Distill-v1-Q4_K_M CPU | 0.40 | 61.94 | 2.56 | memory_policy/store_user_preference: invalid_json: Expecting value: line 1 column 1 (char 0); memory_policy/ignore_trivial_tool_call: invalid_json: Expecting value: line 1 column 1 (char 0); recall/select_relevant_memory: invalid_json: Expecting property name enclosed in double quotes: line 1 column 2 (char 1) |
| X-Coder-SFT-Qwen3-8B.Q6_K CPU | 0.76 | 60.12 | 2.51 | action/direct_answer_no_tools: invalid_json: Expecting ',' delimiter: line 13 column 6 (char 632); memory_policy/ignore_trivial_tool_call: stored_trivial={'should_store': True, 'memory_type': 'event', 'summary': 'User executed pwd command and received /tmp/project as output.', 'importance': 0.8, 'scope': 'conversation', 'metadata': {}} |
| gemma-4-E4B-it-Q4_K_M CPU | 0.97 | 21.23 | 5.36 | critic/reflection_quality: missing=['lesson'] |
## Case Details
### Qwen3.6-35B nonMTP GPU baseline
| Role | Case | Score | Latency, s | tok/s | Note |
| --- | --- | ---: | ---: | ---: | --- |
| action | direct_answer_no_tools | 1.00 | 15.31 | 2.94 | ok |
| action | read_specific_file | 1.00 | 19.61 | 4.13 | ok |
| memory_policy | store_user_preference | 1.00 | 18.53 | 4.75 | ok |
| memory_policy | ignore_trivial_tool_call | 1.00 | 15.00 | 4.07 | ok |
| recall | select_relevant_memory | 1.00 | 15.09 | 4.38 | ok |
| summary | preserve_decisions | 1.00 | 9.95 | 4.42 | ok |
| critic | reflection_quality | 0.80 | 32.09 | 6.86 | missing=['lesson'] |
### Menlo_Lucy-Q4_K_M CPU
| Role | Case | Score | Latency, s | tok/s | Note |
| --- | --- | ---: | ---: | ---: | --- |
| action | direct_answer_no_tools | 1.00 | 3.23 | 9.60 | ok |
| action | read_specific_file | 1.00 | 3.03 | 15.84 | ok |
| memory_policy | store_user_preference | 1.00 | 3.62 | 14.92 | ok |
| memory_policy | ignore_trivial_tool_call | 0.30 | 3.19 | 18.17 | stored_trivial={'should_store': True, 'memory_type': 'fact', 'summary': 'Password was successfully launched and user was informed.', 'importance': 0.7, 'scope': 'global', 'metadata': {}} |
| recall | select_relevant_memory | 0.30 | 3.74 | 16.05 | wrong_ids=[] |
| summary | preserve_decisions | 0.80 | 3.33 | 18.29 | missing=['approval'] |
| critic | reflection_quality | 1.00 | 10.70 | 20.57 | ok |
### Qwen3.5-9B-GLM5.1-Distill-v1-Q4_K_M CPU
| Role | Case | Score | Latency, s | tok/s | Note |
| --- | --- | ---: | ---: | ---: | --- |
| action | direct_answer_no_tools | 1.00 | 68.08 | 1.06 | ok |
| action | read_specific_file | 1.00 | 72.15 | 1.19 | ok |
| memory_policy | store_user_preference | 0.00 | 67.76 | 2.66 | invalid_json: Expecting value: line 1 column 1 (char 0) |
| memory_policy | ignore_trivial_tool_call | 0.00 | 64.65 | 2.47 | invalid_json: Expecting value: line 1 column 1 (char 0) |
| recall | select_relevant_memory | 0.00 | 59.45 | 2.69 | invalid_json: Expecting property name enclosed in double quotes: line 1 column 2 (char 1) |
| summary | preserve_decisions | 0.20 | 47.05 | 3.83 | missing=['8000', '8081', 'approval', 'allow_forever'] |
| critic | reflection_quality | 0.60 | 54.43 | 4.04 | missing=['risk', 'lesson'] |
### X-Coder-SFT-Qwen3-8B.Q6_K CPU
| Role | Case | Score | Latency, s | tok/s | Note |
| --- | --- | ---: | ---: | ---: | --- |
| action | direct_answer_no_tools | 0.00 | 121.05 | 1.49 | invalid_json: Expecting ',' delimiter: line 13 column 6 (char 632) |
| action | read_specific_file | 1.00 | 37.56 | 3.57 | ok |
| memory_policy | store_user_preference | 1.00 | 66.98 | 1.19 | ok |
| memory_policy | ignore_trivial_tool_call | 0.30 | 21.77 | 2.85 | stored_trivial={'should_store': True, 'memory_type': 'event', 'summary': 'User executed pwd command and received /tmp/project as output.', 'importance': 0.8, 'scope': 'conversation', 'metadata': {}} |
| recall | select_relevant_memory | 1.00 | 58.66 | 1.53 | ok |
| summary | preserve_decisions | 1.00 | 53.24 | 3.38 | ok |
| critic | reflection_quality | 1.00 | 61.55 | 3.57 | ok |
### gemma-4-E4B-it-Q4_K_M CPU
| Role | Case | Score | Latency, s | tok/s | Note |
| --- | --- | ---: | ---: | ---: | --- |
| action | direct_answer_no_tools | 1.00 | 35.72 | 1.48 | ok |
| action | read_specific_file | 1.00 | 13.32 | 6.60 | ok |
| memory_policy | store_user_preference | 1.00 | 27.13 | 3.61 | ok |
| memory_policy | ignore_trivial_tool_call | 1.00 | 10.23 | 8.80 | ok |
| recall | select_relevant_memory | 1.00 | 19.39 | 3.20 | ok |
| summary | preserve_decisions | 1.00 | 14.37 | 6.12 | ok |
| critic | reflection_quality | 0.80 | 28.48 | 7.72 | missing=['lesson'] |
docs/how_to_run.md
+37 -18
View File
@@ -16,34 +16,53 @@ cp .env.example .env
The default `DUCK_MAIN_MODEL_PATH` points to `./models/Qwen3.6/nonMTP/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`.
3. Start `llama-server`:
3. Start DuckLM:
```bash
bash scripts/llama/start_main.sh start
bash scripts/duck.sh start
```
This starts both processes:
- `llama-server` on `http://127.0.0.1:8081/v1`
- DuckLM API/WebChat on `http://127.0.0.1:8000/`
Useful process commands:
```bash
bash scripts/llama/start_main.sh status
bash scripts/llama/start_main.sh logs --follow
bash scripts/llama/start_main.sh restart
bash scripts/llama/start_main.sh stop
bash scripts/duck.sh status
bash scripts/duck.sh logs --follow
bash scripts/duck.sh restart
bash scripts/duck.sh stop
```
4. Start DuckLM API:
```bash
python -m duck_core.api
```
5. Open WebChat:
4. Open WebChat:
```text
http://127.0.0.1:8000/
```
6. Send a task:
Low-level llama-only commands are still available when needed:
```bash
bash scripts/llama/start_main.sh status
bash scripts/llama/start_main.sh logs --follow
```
MTP/speculative variant:
```bash
bash scripts/duck.sh stop
bash scripts/duck-mtp.sh start
bash scripts/duck-mtp.sh status
bash scripts/duck-mtp.sh logs --follow
```
`duck-mtp.sh` keeps DuckLM on `http://127.0.0.1:8000/` and starts the MTP-backed
`llama-server` on the normal role endpoint `http://127.0.0.1:8081/v1`, so
`config/models.yaml` does not need to change.
5. Send a task:
```bash
curl -X POST http://127.0.0.1:8000/v1/chat \
@@ -51,21 +70,21 @@ curl -X POST http://127.0.0.1:8000/v1/chat \
-d '{"message":"Скажи коротко, что ты DuckLM","workspace":"./workspace","debug":true}'
```
7. Inspect events:
6. Inspect events:
```bash
curl http://127.0.0.1:8000/v1/tasks/<task_id>/events
```
8. Approvals:
7. Approvals:
```bash
curl http://127.0.0.1:8000/v1/approvals/pending
```
9. Stop services:
8. Stop services:
```bash
bash scripts/llama/start_main.sh stop
bash scripts/duck.sh stop
docker compose -f docker-compose.memory.yml down
```
docs/web_api.md
+15
View File
@@ -23,3 +23,18 @@ GET /v1/experience
GET /v1/experience/{id}
GET /v1/memory/search?q=...
```
Chat requests accept optional `reasoning`:
```json
{
"message": "hello",
"reasoning": "auto"
}
```
Allowed values:
- `auto`: use the `llama-server` default.
- `on`: pass `enable_thinking=true` for the thinker response.
- `off`: pass `enable_thinking=false` and `thinking_budget_tokens=0`.