Rust Self-Learning Memory System

Open Issues

Production Ready • 99.3% Test Pass Rate • 92.5% Coverage • Zero Clippy Warnings

NEW: Semantic Pattern Search & Recommendations 🔍

A self-learning episodic memory system with semantic pattern search, embeddings, MCP server, and secure code execution sandbox.

Overview • Features • Quick Start • Documentation • Contributing • Quality Gates • License

</div>

Overview

The Rust Self-Learning Memory System provides persistent memory across agent interactions through a comprehensive MCP (Model Context Protocol) server with secure code execution. It captures, stores, and learns from episodic experiences to improve future performance.

Current Status (v0.1.15):

Production-ready episodic memory management system for AI agents
9 workspace members: memory-core, memory-storage-turso, memory-storage-redb, memory-mcp, memory-cli, test-utils, benches, examples, tests
632 Rust source files with ~140,000 lines of code
811+ lib tests with 99.5% pass rate
92.5% test coverage across all modules
10-100x performance improvements over baseline measurements
Zero clippy warnings with strict linting rules
Multi-provider semantic embeddings with OpenAI, Mistral, and local backends
Dual storage backends: Turso for durability, redb for cache
6-layer security sandbox in Wasmtime for safe code execution
Phase 2 Turso Optimization: 100% complete (connection pooling, adaptive sizing, compression, adaptive TTL)
CI Status: ALL PASSING (Nightly Full Tests FIXED in #283)
MCP Token Optimization: 98% token reduction for tool discovery

Architecture:

memory-core: Core memory operations, pattern extraction, and reward scoring
memory-storage-turso: Primary database storage (libSQL)
memory-storage-redb: Fast embedded cache layer
memory-mcp: MCP server with secure WASM sandbox
memory-cli: Full-featured command-line interface (9 commands, 9 aliases)
test-utils: Shared testing utilities
benches: Comprehensive benchmark suite
examples: Usage examples and demonstrations

Tech Stack: Rust/Tokio + Turso/libSQL + redb cache + Wasmtime WASM + optional embeddings (OpenAI, Mistral, local)

Features

🧠 Episodic Memory

Complete episode lifecycle (start → execute → score → learn → retrieve)
Detailed execution step logging with tool usage tracking
Intelligent reward scoring with efficiency and quality bonuses
Automatic reflection generation for learning

📚 Multiple Storage Backends

Turso Cloud: Remote libSQL database (default)
redb Cache: Fast embedded key-value storage
Local SQLite: Local file-based database (fallback)
Automatic caching with TTL-based invalidation

🎯 Pattern Recognition & Semantic Search

Four pattern types: ToolSequence, DecisionPoint, ErrorRecovery, ContextPattern
NEW: Semantic pattern search with natural language queries
NEW: Intelligent pattern recommendations for tasks using multi-signal ranking
NEW: Cross-domain pattern discovery to find analogous patterns
Async pattern extraction with queue-based workers
Pattern effectiveness tracking with decay over time
Multi-signal ranking: semantic similarity, context match, effectiveness, recency, success rate
Minimum success rate filtering (default 70%)

🔒 Secure Code Sandbox

Wasmtime WASM sandbox for safe code execution
Resource limits (timeout, memory, CPU)
Defense-in-depth security with access controls
Support for concurrent executions (20 parallel by default)

📊 Advanced Analysis

Statistical analysis (ETS forecasting, MSTL decomposition)
Anomaly detection and changepoint analysis
Time series forecasting with configurable horizon
Causal inference for pattern relationships

🔍 MCP Server

Standard MCP protocol implementation (v2024-11)
8 MCP tools for memory operations, pattern search, and code execution
NEW: search_patterns - Semantic pattern search with configurable ranking
NEW: recommend_patterns - Task-specific pattern recommendations
Progressive tool disclosure based on usage
Execution monitoring and metrics tracking
Wasmtime-based WASM sandbox for secure code execution

🛠️ Full-Featured CLI

9 main commands for episode, pattern, and storage management
9 command aliases for rapid development workflow
Episode management (create, list, search, complete)
Pattern analysis and effectiveness tracking
Storage operations (sync, vacuum, health checks)
Backup and restore capabilities
Monitoring and metrics export
Multiple output formats (human, JSON, YAML)

🌐 Multi-Provider Embeddings

OpenAI embeddings integration (text-embedding-3-small, text-embedding-3-large, ada-002)
Mistral AI embeddings integration
Local CPU-based embeddings
Semantic search with cosine similarity
Automatic embedding caching and batch processing

🛡️ Quality Assurance

Automated quality gates (>90% coverage)
Comprehensive test suite across all crates (811+ lib tests)
Security auditing for sandbox operations
Performance benchmarks with regression detection
Zero clippy warnings policy
Pre-commit and post-commit hooks for code quality

Quick Start

🔍 Pattern Search Example (NEW in v0.1.13)

use memory_core::{SelfLearningMemory, TaskContext, ComplexityLevel};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let memory = SelfLearningMemory::new();
    
    // Search for patterns using natural language
    let context = TaskContext {
        domain: "web-api".to_string(),
        language: Some("rust".to_string()),
        framework: None,
        complexity: ComplexityLevel::Moderate,
        tags: vec!["rest".to_string(), "async".to_string()],
    };
    
    let results = memory.search_patterns_semantic(
        "How to handle API rate limiting with retries",
        context,
        5  // limit
    ).await?;
    
    for result in results {
        println!("Pattern: {:?}", result.pattern);
        println!("Relevance: {:.2}", result.relevance_score);
        println!("Success Rate: {:.1}%", result.pattern.success_rate() * 100.0);
    }
    
    // Get task-specific recommendations
    let recommendations = memory.recommend_patterns_for_task(
        "Build async HTTP client with connection pooling",
        context,
        3
    ).await?;
    
    for rec in recommendations {
        println!("Recommended: {:?}", rec.pattern);
    }
    
    Ok(())
}

Try it yourself: cargo run --example pattern_search_demo

Documentation: See memory-core/PATTERN_SEARCH_FEATURE.md for complete API reference and examples.

Prerequisites

Rust 1.85+ (stable, 2024 edition)
SQLite (for local development)
Optional: Turso CLI (for cloud database)

Installation

# Clone the repository
git clone https://github.com/d-o-hub/rust-self-learning-memory.git
cd rust-self-learning-memory

# Build the project
cargo build --release

# Run tests
cargo test --all
# Or use nextest for faster tests
cargo nextest run --all

# Run quality gates
./scripts/quality-gates.sh

Local Database Setup

# Quick setup with the provided script
./scripts/setup-local-db.sh

# Or manual setup
cp .env.example .env
mkdir -p ./data ./backups

Basic Usage

Setup Configuration

# Run interactive configuration wizard
memory config wizard

# Follow the prompts to configure:
# - Database (local SQLite or remote Turso)
# - Storage (cache size, TTL, connection pool)
# - CLI (output format, progress bars, batch size)

# Validate configuration
memory config validate

# Check configuration status
memory config check

NEW: Configuration Wizard - Interactive step-by-step setup with sensible defaults and validation.

CLI Interaction

# Create an episode
memory-cli episode create --task "Implement user authentication" --context '{"language": "rust", "domain": "auth"}'

# List episodes
memory-cli episode list --limit 10

# Retrieve relevant context
memory-cli episode search "authentication" --limit 5

# NEW: Search patterns semantically
memory-cli pattern search --query "How to build REST API" --domain web-api --limit 5

# NEW: Get pattern recommendations
memory-cli pattern recommend --task "Build async HTTP client" --domain web-api --limit 3

# Analyze patterns
memory-cli pattern list --min-confidence 0.8

MCP Server

# Start the MCP server
cargo run --bin memory-mcp-server

# Or run with custom config
cargo run --bin memory-mcp-server -- --config mcp-config-memory.json

Programmatic Usage

use memory_core::{SelfLearningMemory, TaskContext, TaskType};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let memory = SelfLearningMemory::new(Default::default()).await?;

    let context = TaskContext {
        language: "rust".to_string(),
        domain: "web".to_string(),
        tags: vec!["api".to_string()],
    };

    let episode_id = memory.start_episode(
        "Build REST API endpoint".to_string(),
        context,
        TaskType::CodeGeneration,
    ).await;

    Ok(())
}

Documentation

Document	Description
Configuration Wizard	Interactive setup guide
Configuration Guide	Complete configuration options
Database Setup	Local database configuration
Quality Gates	Automated quality standards
YAML Validation	Configuration validation strategy
Testing Guide	Testing infrastructure and strategies
Contributing	Development workflow
Security	Security policies and practices
Deployment	Deployment strategies
Release Engineering	Release workflow and automation

Agent Documentation

Document	Description
Building the Project	Build commands and setup
Running Tests	Testing strategies and coverage
Code Conventions	Rust idioms and patterns
Service Architecture	System design and components
Database Schema	Data structures and relationships
Communication Patterns	Inter-service communication

Crate Documentation

Crate	Description
memory-core	Core episodic learning system
memory-mcp	MCP server with secure sandbox
memory-cli	Command-line interface
memory-storage-turso	Turso/libSQL storage backend
memory-storage-redb	redb cache backend

Quality Gates

The project maintains high quality standards through automated quality gates:

Gate	Threshold	Description
Test Coverage	> 90%	Line coverage across all crates
Pattern Accuracy	> 70%	Pattern recognition accuracy
Code Complexity	Avg < 10	Average cyclomatic complexity
Security	0 vulns	Zero critical/high/medium vulnerabilities
Linting	0 warnings	Zero clippy warnings
Formatting	100%	All code rustfmt compliant
Performance	< 10% regression	No performance degradation

Run quality gates locally:

./scripts/quality-gates.sh

For more details, see Quality Gates Documentation.

Feature Flags

Enable optional features via Cargo:

# Basic features (default)
cargo build

# All features
cargo build --all-features

# Specific features
cargo build --features openai
cargo build --features mistral
cargo build --features local-embeddings
cargo build --features embeddings-full

Available Features:

openai: OpenAI API embeddings support
mistral: Mistral AI embeddings support
local-embeddings: CPU-based local embeddings
embeddings-full: All embedding providers (openai + mistral)
mcp: MCP server tools and protocol support
sandbox: Wasmtime sandbox for code execution

Configuration

Environment Variables

# Turso Cloud (default)
TURSO_DATABASE_URL=libsql://your-db.turso.io
TURSO_AUTH_TOKEN=your-auth-token

# Local SQLite (fallback)
LOCAL_DATABASE_URL=sqlite:./data/memory.db
MEMORY_REDB_PATH=./data/memory.redb

# Cache settings
MEMORY_MAX_EPISODES_CACHE=1000
MEMORY_CACHE_TTL_SECONDS=3600

# Sandbox settings
MCP_USE_WASM=true
JAVY_PLUGIN=./memory-mcp/javy-plugin.wasm

TOML Configuration

[database]
turso_url = "libsql://your-db.turso.io"
turso_token = "your-auth-token"
redb_path = "memory.redb"

[storage]
max_episodes_cache = 1000
cache_ttl_seconds = 3600
pool_size = 10

[sandbox]
max_execution_time_ms = 5000
max_memory_mb = 128
max_cpu_percent = 50
allow_network = false
allow_filesystem = false

[cli]
default_format = "human"
progress_bars = true
batch_size = 100

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      Memory CLI                              │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │   Episode   │  │  Pattern    │  │   Storage   │         │
│  │ Management  │  │  Analysis   │  │ Operations  │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
└─────────────────────────────────────────────────────────────┘
                               │
┌─────────────────────────────────────────────────────────────┐
│                     Memory MCP Server                        │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │  MCP Tools  │  │  WASM       │  │  Advanced   │         │
│  │  Interface  │  │  Sandbox    │  │  Analysis   │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
└─────────────────────────────────────────────────────────────┘
                               │
┌─────────────────────────────────────────────────────────────┐
│                     Memory Core                              │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │   Episode   │  │   Pattern   │  │   Reward    │         │
│  │ Management  │  │ Extraction  │  │   Scoring   │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
└─────────────────────────────────────────────────────────────┘
                               │
         ┌─────────────────────┼─────────────────────┐
         │                     │                     │
┌───────▼────────┐  ┌────────▼────────┐  ┌────────▼────────┐
│ Turso Storage  │  │  Redb Cache     │  │  In-Memory      │
│                │  │                 │  │                 │
│ libSQL/Remote  │  │   Fast Access   │  │  Temporary      │
└────────────────┘  └─────────────────┘  └─────────────────┘

MCP Server Tools

The MCP server exposes the following tools:

query_memory

Query episodic memory for relevant past experiences based on task type, domain, and query text.

analyze_patterns

Analyze patterns from past episodes to identify successful strategies and recommendations.

execute_agent_code

Execute TypeScript/JavaScript code in a secure WASM sandbox with resource limits.

Advanced Pattern Analysis

Statistical analysis, forecasting, anomaly detection, and causal inference on time series data from memory episodes.

Performance

All operations meet or exceed performance targets:

Operation	Target (P95)	Typical Performance
Episode Creation	< 50ms	~2.5 µs (19,531x faster)
Step Logging	< 20ms	~1.1 µs (17,699x faster)
Episode Completion	< 500ms	~3.8 µs (130,890x faster)
Pattern Extraction	< 1000ms	~10.4 µs (95,880x faster)
Memory Retrieval	< 100ms	~721 µs (138x faster)
WASM Execution	< 200ms	~50-200ms (typical)

Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Workflow

Fork the repository
Create a feature branch: git checkout -b feature-name
Make your changes
Run quality gates: ./scripts/quality-gates.sh
Submit a pull request

Code Standards

Follow Rust idioms
Maintain 90%+ test coverage
Run cargo fmt and cargo clippy before committing
Document public APIs
Write descriptive commit messages

Quality Requirements

All tests must pass
No clippy warnings
90%+ test coverage
Security audit must pass
Performance benchmarks must not degrade >10%

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

Acknowledgments

libSQL for the embedded database
redb for the embedded key-value store
tokio for asynchronous runtime
Turso for the cloud database service
Wasmtime for the secure WASM runtime
Javy for JavaScript compilation

Documentation • GitHub