memory-lancedb-pro Plugin Maintenance Guide
Overview
memory-lancedb-pro is an enhanced long-term memory plugin for OpenClaw. It replaces the built-in memory-lancedb plugin with advanced retrieval capabilities, multi-scope memory isolation, and a management CLI.
Repository: https://github.com/win4r/memory-lancedb-pro
License: MIT | Language: TypeScript (ESM) | Runtime: Node.js via OpenClaw Gateway
Architecture
┌─────────────────────────────────────────────────────────┐
│ index.ts (Entry Point) │
│ Plugin Registration · Config Parsing · Lifecycle Hooks │
└────────┬──────────┬──────────┬──────────┬───────────────┘
│ │ │ │
┌────▼───┐ ┌────▼───┐ ┌───▼────┐ ┌──▼──────────┐
│ store │ │embedder│ │retriever│ │ scopes │
│ .ts │ │ .ts │ │ .ts │ │ .ts │
└────────┘ └────────┘ └────────┘ └─────────────┘
│ │
┌────▼───┐ ┌─────▼──────────┐
│migrate │ │noise-filter.ts │
│ .ts │ │adaptive- │
└────────┘ │retrieval.ts │
└────────────────┘
┌─────────────┐ ┌──────────┐
│ tools.ts │ │ cli.ts │
│ (Agent API) │ │ (CLI) │
└─────────────┘ └──────────┘
File Reference (Quick Navigation)
| File |
Purpose |
Key Exports |
index.ts |
Plugin entry point. Registers with OpenClaw Plugin API, parses config, mounts lifecycle hooks |
memoryLanceDBProPlugin (default), shouldCapture, detectCategory |
openclaw.plugin.json |
Plugin metadata + full JSON Schema config with uiHints |
— |
package.json |
NPM package. Deps: @lancedb/lancedb, openai, @sinclair/typebox |
— |
cli.ts |
CLI: memory-pro list/search/stats/delete/delete-bulk/export/import/reembed/migrate |
createMemoryCLI, registerMemoryCLI |
src/store.ts |
LanceDB storage layer. Table creation, FTS indexing, CRUD, vector/BM25 search |
MemoryStore, MemoryEntry, loadLanceDB |
src/embedder.ts |
Embedding abstraction. OpenAI-compatible API, task-aware, LRU cache |
Embedder, createEmbedder, getVectorDimensions |
src/retriever.ts |
Hybrid retrieval engine. Full scoring pipeline |
MemoryRetriever, createRetriever, DEFAULT_RETRIEVAL_CONFIG |
src/scopes.ts |
Multi-scope access control |
MemoryScopeManager, createScopeManager |
src/tools.ts |
Agent tool definitions: memory_recall/store/forget/update/stats/list |
registerAllMemoryTools |
src/noise-filter.ts |
Noise filter for low-quality content |
isNoise, filterNoise |
src/adaptive-retrieval.ts |
Skip retrieval for greetings, commands, emoji |
shouldSkipRetrieval |
src/migrate.ts |
Migration from legacy memory-lancedb |
MemoryMigrator, createMigrator |
scripts/jsonl_distill.py |
JSONL session distillation script (Python) |
— |
Core Subsystem Reference
For detailed deep-dives into each subsystem, read the appropriate reference file:
- Retrieval Pipeline (scoring math, RRF fusion, reranking, all scoring stages): See references/retrieval_pipeline.md
- Storage & Data Model (LanceDB schema, FTS indexing, CRUD, vector dim): See references/storage_and_schema.md
- Embedding System (providers, task-aware API, caching, dimensions): See references/embedding_system.md
- Plugin Lifecycle & Config (hooks, registration, config parsing): See references/plugin_lifecycle.md
- Scope System (multi-scope isolation, agent access, patterns): See references/scope_system.md
- Tools & CLI (agent tools, CLI commands, parameters): See references/tools_and_cli.md
- Common Gotchas & Troubleshooting: See references/troubleshooting.md
Development Workflows
Adding a New Embedding Provider
- Check if it's OpenAI-compatible (most are). If so, no code change needed — just config
- If the model is not in
EMBEDDING_DIMENSIONS map in src/embedder.ts, add it
- If the provider needs special request fields beyond
task and normalized, extend buildPayload() in src/embedder.ts
- Test with
embedder.test() method
- Document the provider in README.md table
Adding a New Rerank Provider
- Add provider name to
RerankProvider type in src/retriever.ts
- Add case in
buildRerankRequest() for request format (headers + body)
- Add case in
parseRerankResponse() for response parsing
- Add to
rerankProvider enum in openclaw.plugin.json
- Test with actual API calls — reranker has 5s timeout protection
Adding a New Scoring Stage
- Create a
private apply<StageName>(results: RetrievalResult[]): RetrievalResult[] method in MemoryRetriever
- Add corresponding config fields to
RetrievalConfig interface
- Insert the stage in the pipeline sequence in both
hybridRetrieval() and vectorOnlyRetrieval()
- Add defaults to
DEFAULT_RETRIEVAL_CONFIG
- Add JSON Schema fields to
openclaw.plugin.json
- Pipeline order: Fusion → Rerank → Recency → Importance → LengthNorm → TimeDecay → HardMin → Noise → MMR
Adding a New Agent Tool
- Create
registerMemory<ToolName>Tool() in src/tools.ts
- Define parameters with
Type.Object() from @sinclair/typebox
- Use
stringEnum() from openclaw/plugin-sdk for enum params
- Always validate scope access via
context.scopeManager
- Register in
registerAllMemoryTools() — decide if core (always) or management (optional)
- Return
{ content: [{ type: "text", text }], details: {...} }
Adding a New CLI Command
- Add command in
registerMemoryCLI() in cli.ts
- Pattern:
memory.command("name <args>").description("...").option("--flag", "...").action(async (args, opts) => { ... })
- Support
--json flag for machine-readable output
- Use
process.exit(1) for error cases
- CLI is registered via
api.registerCli() in index.ts
Modifying Auto-Capture Logic
shouldCapture(text) in index.ts controls what gets auto-capturedMEMORY_TRIGGERS regex array defines trigger patterns (supports EN/CJK)detectCategory(text) classifies captures as preference/fact/decision/entity/other- Auto-capture runs in
agent_end hook, limited to 3 per turn
- Duplicate detection threshold: cosine similarity > 0.95
Modifying Auto-Recall Logic
- Auto-recall uses
before_agent_start hook (OFF by default)
shouldSkipRetrieval() from src/adaptive-retrieval.ts gates retrieval- Injected as
<relevant-memories> XML block with UNTRUSTED DATA warning
sanitizeForContext() strips HTML, newlines, limits to 300 chars per memory- Max 3 memories injected per turn
Key Design Decisions
- autoRecall defaults to OFF — prevents model from echoing injected memory context
- autoCapture defaults to ON — transparent memory accumulation
- sessionMemory defaults to OFF — raw session summaries degrade retrieval quality; use JSONL distillation instead
- LanceDB dynamic import — loaded asynchronously to avoid blocking; cached in singleton promise
- Startup checks are fire-and-forget — gateway binds HTTP port immediately; embedding/retrieval tests run in background with 8s timeout
- Daily JSONL backup — 24h interval, keeps last 7 files, runs 1 min after start
- BM25 score normalization — raw BM25 scores are unbounded, normalized with sigmoid:
1 / (1 + exp(-score/5))
- Update = delete + re-add — LanceDB doesn't support in-place updates
- ID prefix matching — 8+ hex char prefix resolves to full UUID for user convenience
- CJK-aware thresholds — shorter minimum lengths for Chinese/Japanese/Korean text (4–6 chars vs 10–15 for English)
- Env var resolution —
${VAR} syntax resolved at config parse time; gateway service may not inherit shell env
Testing
- Smoke test:
node test/cli-smoke.mjs
- Manual verification:
openclaw plugins doctor, openclaw memory-pro stats
- Embedding test:
embedder.test() returns { success, dimensions, error? }
- Retrieval test:
retriever.test() returns { success, mode, hasFtsSupport, error? }
