TWDAgentsHub · April 2026

Agent Runtime — Kiến trúc tổng quan & Sales-v0 flow

Snapshot trạng thái hiện tại của apps/api/src/agent-runtime/ — single sole orchestration path sau khi xoá legacy orchestrator (Phase 01). Bao gồm: layered architecture, tích hợp ERP qua @twd/erp-adapters, MCP tools layer, và 3 trigger flows của Sales-v0 (cron-apollo, event-rfq, message-telegram).

01 Tiers tổng thể

Toàn bộ stack chạy trong NestJS modular monolith (@twd/api). Frontend (admin-ui + chat widget) gọi REST/WebSocket; channels (Telegram, SendGrid) đi qua webhook; tất cả hợp nhất tại Agent Runtime.

flowchart TB
  %% ① Clients
  subgraph CLIENT["① Clients"]
    direction LR
    USR["End users
(Telegram · Web · Email)"] OPS["Operators
(Admin UI)"] end %% ② Channels subgraph CH["② Channels"] direction LR REST["REST API"] TGC["Telegram"] SGC["SendGrid"] end %% ③ Agent Runtime — the brain subgraph RT["③ Agent Runtime"] direction LR TRG["Triggers
cron · event · message · webhook"] EXE["Executor
workflow + step pipeline"] REG["Registries
agents · steps · tools"] TRG --> EXE REG -. lookup .-> EXE end %% ④ Capability layers subgraph CAP["④ Capabilities"] direction LR TOOLS["Tools
(MCP · Native · Memory)"] ADP["ERP Adapters
(@twd/erp-adapters)"] LLM["LLM
(tier router)"] end %% ⑤ External subgraph EXT["⑤ External services"] direction LR ERP[("Twendee ERP")] APO[("Apollo.io")] EMAIL[("SendGrid · Gmail")] AI[("OpenAI · Gemini · Ollama")] end %% ⑥ Platform subgraph PLT["⑥ Platform"] direction LR PG[("PostgreSQL
multi-tenant + RLS")] RD[("Redis
queues + cache")] SEC["Auth · Encryption ·
Metering · Observability"] end CLIENT ==> CH ==> RT ==> CAP ADP --> ERP ADP --> APO ADP --> EMAIL LLM --> AI RT -.-> PLT classDef ch fill:transparent,stroke:#0f766e,color:inherit; classDef rt fill:transparent,stroke:#b45309,color:inherit; classDef tl fill:transparent,stroke:#4d7c0f,color:inherit; classDef ad fill:transparent,stroke:#7c3aed,color:inherit; class REST,TG,SG,UI ch; class TR,EX,SE,REG rt; class TI,TR2 tl; class APOL,CRM,EM,OTH ad;
Channels

Inbound

REST /api/v1/chat, Telegram webhook, SendGrid inbound parse, Admin UI. Mỗi channel có adapter riêng + idempotency service.

Runtime

Agent Runtime

Trigger bus → Workflow Executor → Step Executor. Không còn orchestrator cũ — agent definitions là code (defineAgent) + override DB agent_workflows.

Adapters

ERP / 3rd-party

@twd/erp-adapters exposes ports (CRM, Email, Calendar, Enrichment). Resolve per-tenant qua SalesIntegrationFactoryService.bundleFor(ctx).

02 Agent Runtime — bên trong

Một agent là object code-defined (defineAgent) gồm identity, triggers[], tools[], channels[], defaultBudget. Mỗi trigger có pipeline[] (step list).

flowchart LR
  AG["defineAgent
id identity triggers
tools channels budget"] --> AR["AgentRegistryService
boot scan"] AR --> TBUS["Trigger Bus"] TBUS --> CRON["CronScheduler
BullMQ repeat + LISTEN
agent_workflows_changed"] TBUS --> EVT["EventBus
emit fanout matched agents"] TBUS --> MSG["MessageRouter
team lead · mention · keywords · fallback"] TBUS --> WHK["Webhook controller"] CRON --> Q["trigger-exec queue
BullMQ"] EVT --> Q MSG --> Q WHK --> Q Q --> P["TriggerExecProcessor"] P --> WF["WorkflowExecutor
runPipeline steps ctx"] WF --> SX["StepExecutor"] SX --> PR["ParamResolver
trigger.x · steps.id.y"] SX --> BG["BudgetGuard
tokens · cost · duration"] SX --> OBS["ObservabilityInjector
pipeline_events"] SX --> SR["StepRegistry
handler dispatch"] SR --> DATA["data steps
salesRunCampaignsBatch
apolloEnrichLeads
erpQueryLead
erpUpsertContacts"] SR --> LLMS["llm steps
defaultChatLoop
clarifyRequirements
draftProposal ..."] SR --> OUT["output steps
telegramNotify · emitEvent"] classDef rt fill:transparent,stroke:#b45309,color:inherit; classDef tr fill:transparent,stroke:#0f766e,color:inherit; classDef st fill:transparent,stroke:#4d7c0f,color:inherit; class AG,AR,WF,SX,P rt; class TBUS,CRON,EVT,MSG,WHK,Q tr; class DATA,LLMS,OUT,SR,PR,BG,OBS st;
Hot reload schedule: Postgres trigger (migration 0038) NOTIFY trên kênh agent_workflows_changed; CronScheduler debounce 250ms rồi reload. Tenant đổi cron schedule qua DB / Admin UI / psql, có hiệu lực ~250ms — không cần restart app.

03 4 loại trigger

  • cronschedule CRON expr · scope: 'per-tenant' | 'global' · BullMQ repeat job. Job name ${agentId}:${triggerId}. Fanout ra trigger-exec queue (1 job per tenant).
  • eventInternal pub/sub qua EventBusService.emit(name, payload, ctx). JobId dedupe = name + sha(payload). Filter: equality dot-path. Ví dụ rfqReceived, salesLeadsImported.
  • messageChat message từ Telegram/REST. MessageRouterService resolve theo: (1) team lead → (2) mention @agent → (3) keywords → (4) fallback no-constraint agent.
  • webhookWebhookTriggerController nhận POST từ third-party, match theo path → enqueue trigger-exec.

04 Tools & ERP integration

Step handler không gọi adapter trực tiếp. Mọi tác vụ chạm thế giới ngoài đi qua ToolInvokerService.invoke(qualifiedName, params, mcpCtx) — tra ToolRegistry, validate Zod, dispatch tool.rawHandler (KHÔNG phải chat handler).

flowchart TB
  STEP["Step handler
e.g. salesRunCampaignsBatch"] -->|"invoke qualifiedName params ctx"| TI["ToolInvokerService"] TI --> ZV["Zod validate
tool.inputSchema"] TI --> AUDIT["Audit log
tool.invoked"] TI --> RAW["tool.rawHandler
parsed, ctx"] RAW --> RES["AdapterResolver ctx
SalesIntegrationFactoryService.bundleFor"] RES -->|"per-tenant"| BUNDLE["SalesAdapterBundle
apollo · crm · email"] BUNDLE --> APO["Apollo client
ISalesEnrichmentPort"] BUNDLE --> CRMA["TwendeeCRMAdapter
ICRMAdapter
via TwendeeHttpClient"] BUNDLE --> EMP["Email port
SendGrid · Gmail OAuth"] CRMA -->|"HTTPS"| CRMERP[("Twendee ERP
REST API")] APO -->|"HTTPS"| APOAPI[("Apollo.io API")] EMP -->|"HTTPS"| SGAPI[("SendGrid /
Gmail SMTP")] subgraph TR2["ToolRegistry contents"] direction LR M["MCP tools
sales.campaign_*
sales.crm_*"] N["Native dynamic tools
SQL · HTTP scripts"] MEM["Memory tools"] end TI -. lookup .- TR2 classDef st fill:transparent,stroke:#4d7c0f,color:inherit; classDef ad fill:transparent,stroke:#7c3aed,color:inherit; classDef erp fill:transparent,stroke:#b45309,color:inherit; class STEP,TI,RAW,ZV,AUDIT,M,N,MEM st; class RES,BUNDLE,APO,CRMA,EMP ad; class CRMERP,APOAPI,SGAPI erp;

Cách giao tiếp ERP

  • Package@twd/erp-adapters (in-repo)
  • PatternPorts (interfaces) + Adapters (impl). CRM/Email/Calendar/Enrichment/Slack/PM ports.
  • CRM implTwendeeCRMAdapter qua TwendeeHttpClient — REST HTTPS sang Twendee ERP.
  • ResolveSalesIntegrationFactoryService.bundleFor(McpContext) trả bundle theo tenantId.
  • SecretsAesEncryptionService · tenant_*_providers tables · resolve runtime, không cache module-level.

3rd-party đang tích hợp

  • Apollo.ioLead search & enrichment — ISalesEnrichmentPort
  • Twendee CRMDeals, contacts, campaigns, comments — ICRMAdapter
  • SendGridInbound parse webhook + outbound send — channel + email port
  • Gmail OAuthPer-user OAuth send (PR #24)
  • TelegramBot Channel — webhook + Bot API qua TelegramApiService; idempotency + queue
  • LLM ProvidersOpenAI · Gemini · Ollama qua Vercel AI SDK + LlmProviderFactory tier router

05 Sales Agent v0 — agent đầu tiên trên runtime mới

Định nghĩa tại apps/api/src/agent-runtime/agents/sales-v0.agent.ts. 3 triggers song song, mỗi trigger có pipeline riêng. Tools whitelist ở agent level: ['erpQueryLead']. Channel: telegram. Feature flag: SALES_V0_RUNTIME_ENABLED.

defineAgent({
  id: 'sales-v0',
  identity: { name: 'Sales Agent v0', persona: 'qualify leads · draft proposals · query CRM' },
  triggers: [ cronApollo, eventRfq, messageTelegram ],
  tools: ['erpQueryLead'],
  channels: ['telegram'],
  defaultBudget: { maxLLMCalls: 5, maxTokens: 10000, maxCostUsd: 1.0, maxDurationMs: 120_000 },
  enabled: true,
})

5a. cron-apollo — daily 8am ICT lead generation

Schedule 0 1 * * 1-5 (08:00 ICT Mon-Fri). Per-tenant scope. Budget: 200 LLM / 800 tool / 200K tokens / $5 / 5 min.

sequenceDiagram
  autonumber
  participant CRON as BullMQ repeat
  participant FAN as CronFanoutProcessor
  participant Q as trigger exec queue
  participant WF as WorkflowExecutor
  participant BATCH as salesRunCampaignsBatch
  participant TI as ToolInvoker
  participant CRM as Twendee CRM
  participant APO as Apollo
  participant LLM as LLM smart tier
  participant TG as Telegram

  CRON->>FAN: fire sales v0 cron apollo
  FAN->>Q: enqueue per tenant jobs
  Q->>WF: runPipeline batch notify fanout

  WF->>BATCH: handler ctx params
  BATCH->>TI: campaign_list_active
  TI->>CRM: GET active campaigns
  CRM-->>BATCH: up to 5 campaigns

  loop per campaign
    BATCH->>TI: campaign_search_apollo_leads
    TI->>APO: POST mixed_people search with ICP filter
    APO-->>BATCH: leads
    BATCH->>TI: campaign_create_lead concurrency 5
    TI->>CRM: upsert contact and link campaignId
    BATCH->>LLM: draft personalized email
    BATCH->>TI: campaign_save_email_template
    TI->>CRM: save PENDING_REVIEW draft
  end

  BATCH-->>WF: perCampaign reportHtml upsertedIds
  WF->>TI: telegramNotify chatId from agent_workflows
  TI->>TG: sendMessage reportHtml
  Note over TG: onError continue
  WF->>WF: emitEvent salesLeadsImported with upsertedIds
Tenant override: chatId: 0 trong code chỉ là fallback. Tenants override schedule + chatId + ICP filter qua row agent_workflows. Hot-reload qua LISTEN/NOTIFY.

5b. event-rfq — RFQ inbound flow

Subscribe event rfqReceived. Concurrency 4. Budget: 4 LLM / 8K tokens / $0.20 / 60s.

flowchart LR
  PROD["Producer
REST controller / ERP webhook /
another agent"] -->|"emit rfqReceived"| EB["EventBus"] EB -->|"jobId via sha payload
match subscribers"| Q["trigger exec queue"] Q --> WF["WorkflowExecutor"] WF --> S1["clarify
step clarifyRequirements
rfq from trigger.rfq"] S1 --> S2["draft
step draftProposal
rfq + clarifications"] S2 --> S3["notify
step telegramNotify
onError continue"] S3 --> TG[("Telegram chat")] classDef st fill:transparent,stroke:#4d7c0f,color:inherit; classDef tr fill:transparent,stroke:#0f766e,color:inherit; class S1,S2,S3 st; class EB,Q,WF tr;

5c. message — Telegram chat fallback

Match { mention: false } — fallback agent cho mọi message không có @mention. Concurrency 8. Budget: 1 LLM / 2K tokens / $0.05 / 20s.

sequenceDiagram
  autonumber
  participant U as User
  participant TG as Telegram Bot API
  participant WH as TelegramController
  participant IDM as Idempotency
  participant MR as MessageRouter
  participant Q as trigger exec queue
  participant WF as WorkflowExecutor
  participant CHAT as defaultChatLoop
  participant TI as ToolInvoker
  participant CRM as Twendee CRM
  participant LLM as LLM

  U->>TG: message
  TG->>WH: POST telegram webhook
  WH->>IDM: dedupe by update id
  WH->>MR: dispatch tenant channel text conversationId
  MR->>MR: resolve agent by lead mention keywords fallback
  MR->>Q: enqueue sales v0 message trigger
  Q->>WF: runPipeline chat
  WF->>CHAT: handler ctx
  CHAT->>LLM: chat with tool erpQueryLead
  alt tool call
    LLM-->>CHAT: tool_call erpQueryLead args
    CHAT->>TI: invoke erp_query_lead args mcpCtx
    TI->>CRM: GET leads
    CRM-->>CHAT: result
    CHAT->>LLM: continue with tool result
  end
  LLM-->>CHAT: final reply
  CHAT->>TG: channels reply conversationId text
  TG->>U: bot message

06 Takeaways để nắm hệ thống

Mental model 1 dòng

Trigger → Pipeline of Steps → ToolInvoker → Per-tenant Adapter Bundle → 3rd-party / ERP. Tất cả còn lại (registry, budget, observability, retry, hot-reload) là plumbing quanh trục này.

Điểm vào khi đọc code

  1. agent-runtime/agents/sales-v0.agent.ts — definition
  2. agent-runtime/triggers/* — cron / event / message
  3. agent-runtime/execution/workflow-executor.ts + step-executor.ts
  4. agent-runtime/steps/data/sales-run-campaigns-batch.step.ts — step composite phức tạp nhất
  5. agent-runtime/tools/tool-invoker.service.ts — bridge sang MCP tool layer
  6. packages/erp-adapters/src/* — ports + adapters

Câu hỏi mở

  • Producer cụ thể của event rfqReceived hiện tại — REST controller riêng hay parse SendGrid inbound? (chưa verify trong scan).
  • Sales-v0 còn đang feature-flagged (SALES_V0_RUNTIME_ENABLED) — đã bật trên production tenants nào?
  • Webhook trigger: ngoài Telegram/SendGrid hiện đã có third-party nào dùng WebhookTriggerController chưa?