Open Source · Apache 2.0

LangSmith + Judgment Memory

LangSmith traces what your agent did. Judgment Memory learns what it should do next — and proves it's safe to let it. Three lines of code. One pip install.

pip install graph-attention-engine[langchain]

LangSmith Observability

  • Trace every tool call, chain, retrieval
  • Latency, token count, error rate
  • Debug and replay runs
  • Evaluate with scoring functions

Judgment Memory Compounding

  • Score decisions against learned centroids
  • Learn from verified outcomes (not just traces)
  • Conservation law gates automation expansion
  • Decision #1,000 is measurably better than #1

LangSmith's traces are the input. JM's learned centroids are the output that improves.

Integration

Add to any LangChain agent — three lines

# Your existing LangSmith setup (unchanged)
import os
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_PROJECT"] = "my-rag-pipeline"

from langchain.agents import AgentExecutor

# ← Add Judgment Memory (3 lines)
from gae.integrations.langchain import JudgmentMemoryCallback

jm = JudgmentMemoryCallback(
    categories=["retrieval", "generation", "routing"],
    actions=["use", "skip", "escalate"],
    factors=["relevance", "confidence", "recency",
             "source_trust", "complexity"],
    state_path="./jm_state.json",
)

# Both callbacks run together
result = executor.invoke(
    {"input": "What's our Q3 compliance status?"},
    config={"callbacks": [jm]},
)

# After verification
jm.verify(correct=True)

# Check status
jm.status()
# → {"decisions": 247, "accuracy": 0.84, "conservation": "GREEN"}

LangSmith traces automatically via env vars. JM scores via the callback. Both run on the same agent invocation. State persists to jm_state.json — pick up where you left off across restarts.

Production patterns

Three scenarios where JM compounds on LangSmith traces

Pattern 1

RAG Pipeline — which retrieval strategy works for YOUR queries?

LangSmith traces 1,000 retrievals. You see latency and token counts. But after 1,000 queries, which strategy (dense, sparse, hybrid, rerank) actually produces correct answers for which query types? LangSmith can't tell you — it doesn't learn from outcomes.
categories = ["compliance", "product", "financial", "technical"]
actions    = ["dense_only", "sparse_only", "hybrid", "hybrid_rerank"]
factors    = ["query_complexity", "corpus_coverage",
              "recency_requirement", "precision_requirement",
              "query_length", "entity_density"]
After 1,000 queries
compliance→ hybrid_rerank (p=0.273)
product→ dense_only (p=0.274)
financial→ hybrid (p=0.272)
technical→ sparse_only (p=0.270)
dominant factorsrecency, complexity, entity_density
JM learned: compliance needs reranking. Product lookups are faster AND more accurate with dense-only. Stop reranking product queries — save latency, gain accuracy.
Pattern 2

Multi-Agent Router — which agent should handle this?

You have 4 agents (research, code, data, writing). A router picks one per request. LangSmith traces all 1,500 routing decisions. But the router still uses Day 1 rules.
categories = ["code_question", "research", "data_analysis",
              "content_creation", "ambiguous"]
actions    = ["research_agent", "code_agent",
              "data_agent", "writing_agent"]
factors    = ["has_code_keywords", "needs_web_search",
              "has_data_reference", "creative_intent",
              "technical_depth", "user_history_code",
              "user_history_research", "request_length"]
After 1,500 decisions
Day 1 accuracy66%
Day 1500 accuracy80%
improvement+14pp
research_agent86.3% (387 routed)
code_agent78.4% (380 routed)
data_agent78.0% (378 routed)
writing_agent75.8% (355 routed)
The router learned YOUR users' request patterns. Routing accuracy improved 14pp without changing a single routing rule — centroids absorbed the pattern from verified feedback.
Pattern 3

Alert Triage — should this auto-resolve?

Your agent triages production alerts. LangSmith traces 2,000 triage decisions. But the auto-resolve threshold is static. After 500 correct auto-resolves for disk_space alerts, the system still asks a human every time.
categories = ["disk_space", "memory_leak", "api_latency",
              "cert_expiry", "deploy_failure", "security_scan"]
actions    = ["auto_resolve", "escalate_l1",
              "escalate_l2", "page_oncall"]
factors    = ["severity", "recurrence_count", "blast_radius",
              "time_of_day", "similar_resolution",
              "source_reliability", "sla_remaining_pct"]
After 2,000 alerts
disk_space auto-resolve92% accurate (294/321)
cert_expiry auto-resolve95% accurate (196/207)
api_latency auto-resolve59% accurate (60/101)
security_scan auto-resolve40% accurate (2/5)
memory_leak auto-resolve56% accurate (45/81)
JM learned the difference. disk_space and cert_expiry earned autonomy (92%, 95%). security_scan did not (40%). Each expansion step bounded by the conservation law — θmin = 23.53/(α×V). The system can improve. It cannot silently degrade.
The difference

What each system gives you after 1,000 decisions

LangSmith

  • 1,000 traces with full metadata
  • P95 latency: 2.1s → 1.8s (cache hit)
  • Token usage trending down 12%
  • Error rate: 3.2% (up from 2.8%)
  • Three root causes identified in trace

Observability. What happened.

Judgment Memory

  • Compliance → always rerank (91% vs 55%)
  • Product → skip reranking (94%, 3× faster)
  • Routing accuracy: 66% → 80% (+14pp)
  • disk_space auto-resolve: safe (92%)
  • security_scan: NOT safe (40%) — gate held

Judgment. What to do differently, and why.

LangSmith tells you what happened.
Judgment Memory tells you what to do next — and proves it's safe.
They're complementary. LangSmith's traces are the raw signal. JM's centroids are the compounding output.
Under the hood

What happens on each decision

①
Agent calls tool
LangSmith traces it
②
JM scores against
learned centroids
③
Human or check
verifies outcome
④
Centroids update
Conservation recalculates
# What compounds in the state file:

# Centroid geometry — 4 categories × 4 actions × 6 factors = 96 values
# Each value learned from YOUR verified outcomes
# Day 1: uniform [0.5, 0.5, 0.5, 0.5, 0.5, 0.5]
# Day 1000: shaped [0.82, 0.31, 0.74, 0.91, 0.45, 0.67]
#                   ↑ this shape IS the judgment

# Conservation gate
# θ_min = 23.53 / (α × V)
# The system can improve. It cannot silently degrade.
# 3 independent math reviewers · adversarial: 100% rejected

# DK weights — per-factor trust
# "source_trust" has weight 0.91 (your data confirms it matters)
# "query_length" has weight 0.14 (noise for your domain)

The state file is portable. Save it from a LangChain callback, load it in the MCP server, or move it between environments. The judgment carries.

Also available

MCP Server — for Claude, Cursor, Windsurf

Same engine, different integration surface. Six tools over Model Context Protocol:

# Claude Desktop / Cursor config:
{
  "mcpServers": {
    "rgi-judgment-memory": {
      "command": "rgi-mcp-server",
      "args": [
        "--categories", "retrieval,generation,routing",
        "--actions", "use,skip,escalate",
        "--factors", "relevance,confidence,recency,source_trust,complexity",
        "--state", "./jm_state.json"
      ]
    }
  }
}
6 MCP tools
rgi_scoreScore a decision → action + probabilities
rgi_verifyLearn from outcome → centroids update
rgi_conservationSafety gate → RED / AMBER / GREEN
rgi_decisionsDecision history with outcomes
rgi_accuracyPer-category accuracy + IKS
rgi_explainFactor contributions + reasoning trace
pip install graph-attention-engine[mcp]

State files are interchangeable between LangChain and MCP integrations.

Try it

Run the examples — no API key needed

# Install
pip install graph-attention-engine

# Run all three patterns (uses only GAE, no LLM)
python examples/langsmith_jm_examples.py

# Run the MCP quickstart
python examples/mcp_quickstart.py

# Check system status
python examples/status_dashboard.py
1,280
tests · 0 failures
v0.7.26
on PyPI
Apache 2.0
your data · your machine

GitHub → ArindamBanerji/graph-attention-engine

Arindam Banerji · dakshineshwari.net
RGI · Dakshineshwari LLC · October 2026 · Pre-revenue · All results from simulation runs on synthetic data