01 Tiers tổng thể
Toàn bộ stack chạy trong NestJS modular monolith (@twd/api). Frontend (admin-ui + chat widget) gọi REST/WebSocket; channels (Telegram, SendGrid) đi qua webhook; tất cả hợp nhất tại Agent Runtime.
flowchart TB
%% ① Clients
subgraph CLIENT["① Clients"]
direction LR
USR["End users
(Telegram · Web · Email)"]
OPS["Operators
(Admin UI)"]
end
%% ② Channels
subgraph CH["② Channels"]
direction LR
REST["REST API"]
TGC["Telegram"]
SGC["SendGrid"]
end
%% ③ Agent Runtime — the brain
subgraph RT["③ Agent Runtime"]
direction LR
TRG["Triggers
cron · event · message · webhook"]
EXE["Executor
workflow + step pipeline"]
REG["Registries
agents · steps · tools"]
TRG --> EXE
REG -. lookup .-> EXE
end
%% ④ Capability layers
subgraph CAP["④ Capabilities"]
direction LR
TOOLS["Tools
(MCP · Native · Memory)"]
ADP["ERP Adapters
(@twd/erp-adapters)"]
LLM["LLM
(tier router)"]
end
%% ⑤ External
subgraph EXT["⑤ External services"]
direction LR
ERP[("Twendee ERP")]
APO[("Apollo.io")]
EMAIL[("SendGrid · Gmail")]
AI[("OpenAI · Gemini · Ollama")]
end
%% ⑥ Platform
subgraph PLT["⑥ Platform"]
direction LR
PG[("PostgreSQL
multi-tenant + RLS")]
RD[("Redis
queues + cache")]
SEC["Auth · Encryption ·
Metering · Observability"]
end
CLIENT ==> CH ==> RT ==> CAP
ADP --> ERP
ADP --> APO
ADP --> EMAIL
LLM --> AI
RT -.-> PLT
classDef ch fill:transparent,stroke:#0f766e,color:inherit;
classDef rt fill:transparent,stroke:#b45309,color:inherit;
classDef tl fill:transparent,stroke:#4d7c0f,color:inherit;
classDef ad fill:transparent,stroke:#7c3aed,color:inherit;
class REST,TG,SG,UI ch;
class TR,EX,SE,REG rt;
class TI,TR2 tl;
class APOL,CRM,EM,OTH ad;
Inbound
REST /api/v1/chat, Telegram webhook, SendGrid inbound parse, Admin UI. Mỗi channel có adapter riêng + idempotency service.
Agent Runtime
Trigger bus → Workflow Executor → Step Executor. Không còn orchestrator cũ — agent definitions là code (defineAgent) + override DB agent_workflows.
ERP / 3rd-party
@twd/erp-adapters exposes ports (CRM, Email, Calendar, Enrichment). Resolve per-tenant qua SalesIntegrationFactoryService.bundleFor(ctx).
02 Agent Runtime — bên trong
Một agent là object code-defined (defineAgent) gồm identity, triggers[], tools[], channels[], defaultBudget. Mỗi trigger có pipeline[] (step list).
flowchart LR AG["defineAgent
id identity triggers
tools channels budget"] --> AR["AgentRegistryService
boot scan"] AR --> TBUS["Trigger Bus"] TBUS --> CRON["CronScheduler
BullMQ repeat + LISTEN
agent_workflows_changed"] TBUS --> EVT["EventBus
emit fanout matched agents"] TBUS --> MSG["MessageRouter
team lead · mention · keywords · fallback"] TBUS --> WHK["Webhook controller"] CRON --> Q["trigger-exec queue
BullMQ"] EVT --> Q MSG --> Q WHK --> Q Q --> P["TriggerExecProcessor"] P --> WF["WorkflowExecutor
runPipeline steps ctx"] WF --> SX["StepExecutor"] SX --> PR["ParamResolver
trigger.x · steps.id.y"] SX --> BG["BudgetGuard
tokens · cost · duration"] SX --> OBS["ObservabilityInjector
pipeline_events"] SX --> SR["StepRegistry
handler dispatch"] SR --> DATA["data steps
salesRunCampaignsBatch
apolloEnrichLeads
erpQueryLead
erpUpsertContacts"] SR --> LLMS["llm steps
defaultChatLoop
clarifyRequirements
draftProposal ..."] SR --> OUT["output steps
telegramNotify · emitEvent"] classDef rt fill:transparent,stroke:#b45309,color:inherit; classDef tr fill:transparent,stroke:#0f766e,color:inherit; classDef st fill:transparent,stroke:#4d7c0f,color:inherit; class AG,AR,WF,SX,P rt; class TBUS,CRON,EVT,MSG,WHK,Q tr; class DATA,LLMS,OUT,SR,PR,BG,OBS st;
agent_workflows_changed; CronScheduler debounce 250ms rồi reload. Tenant đổi cron schedule qua DB / Admin UI / psql, có hiệu lực ~250ms — không cần restart app.
03 4 loại trigger
- cron
scheduleCRON expr ·scope: 'per-tenant' | 'global'· BullMQ repeat job. Job name${agentId}:${triggerId}. Fanout ra trigger-exec queue (1 job per tenant). - eventInternal pub/sub qua
EventBusService.emit(name, payload, ctx). JobId dedupe =name + sha(payload). Filter: equality dot-path. Ví dụrfqReceived,salesLeadsImported. - messageChat message từ Telegram/REST.
MessageRouterServiceresolve theo: (1) team lead → (2) mention@agent→ (3) keywords → (4) fallback no-constraint agent. - webhook
WebhookTriggerControllernhận POST từ third-party, match theo path → enqueue trigger-exec.
04 Tools & ERP integration
Step handler không gọi adapter trực tiếp. Mọi tác vụ chạm thế giới ngoài đi qua ToolInvokerService.invoke(qualifiedName, params, mcpCtx) — tra ToolRegistry, validate Zod, dispatch tool.rawHandler (KHÔNG phải chat handler).
flowchart TB STEP["Step handler
e.g. salesRunCampaignsBatch"] -->|"invoke qualifiedName params ctx"| TI["ToolInvokerService"] TI --> ZV["Zod validate
tool.inputSchema"] TI --> AUDIT["Audit log
tool.invoked"] TI --> RAW["tool.rawHandler
parsed, ctx"] RAW --> RES["AdapterResolver ctx
SalesIntegrationFactoryService.bundleFor"] RES -->|"per-tenant"| BUNDLE["SalesAdapterBundle
apollo · crm · email"] BUNDLE --> APO["Apollo client
ISalesEnrichmentPort"] BUNDLE --> CRMA["TwendeeCRMAdapter
ICRMAdapter
via TwendeeHttpClient"] BUNDLE --> EMP["Email port
SendGrid · Gmail OAuth"] CRMA -->|"HTTPS"| CRMERP[("Twendee ERP
REST API")] APO -->|"HTTPS"| APOAPI[("Apollo.io API")] EMP -->|"HTTPS"| SGAPI[("SendGrid /
Gmail SMTP")] subgraph TR2["ToolRegistry contents"] direction LR M["MCP tools
sales.campaign_*
sales.crm_*"] N["Native dynamic tools
SQL · HTTP scripts"] MEM["Memory tools"] end TI -. lookup .- TR2 classDef st fill:transparent,stroke:#4d7c0f,color:inherit; classDef ad fill:transparent,stroke:#7c3aed,color:inherit; classDef erp fill:transparent,stroke:#b45309,color:inherit; class STEP,TI,RAW,ZV,AUDIT,M,N,MEM st; class RES,BUNDLE,APO,CRMA,EMP ad; class CRMERP,APOAPI,SGAPI erp;
Cách giao tiếp ERP
- Package
@twd/erp-adapters(in-repo) - PatternPorts (interfaces) + Adapters (impl). CRM/Email/Calendar/Enrichment/Slack/PM ports.
- CRM impl
TwendeeCRMAdapterquaTwendeeHttpClient— REST HTTPS sang Twendee ERP. - Resolve
SalesIntegrationFactoryService.bundleFor(McpContext)trả bundle theo tenantId. - Secrets
AesEncryptionService·tenant_*_providerstables · resolve runtime, không cache module-level.
3rd-party đang tích hợp
- Apollo.ioLead search & enrichment —
ISalesEnrichmentPort - Twendee CRMDeals, contacts, campaigns, comments —
ICRMAdapter - SendGridInbound parse webhook + outbound send — channel + email port
- Gmail OAuthPer-user OAuth send (PR #24)
- TelegramBot Channel — webhook + Bot API qua
TelegramApiService; idempotency + queue - LLM ProvidersOpenAI · Gemini · Ollama qua Vercel AI SDK +
LlmProviderFactorytier router
05 Sales Agent v0 — agent đầu tiên trên runtime mới
Định nghĩa tại apps/api/src/agent-runtime/agents/sales-v0.agent.ts. 3 triggers song song, mỗi trigger có pipeline riêng. Tools whitelist ở agent level: ['erpQueryLead']. Channel: telegram. Feature flag: SALES_V0_RUNTIME_ENABLED.
defineAgent({
id: 'sales-v0',
identity: { name: 'Sales Agent v0', persona: 'qualify leads · draft proposals · query CRM' },
triggers: [ cronApollo, eventRfq, messageTelegram ],
tools: ['erpQueryLead'],
channels: ['telegram'],
defaultBudget: { maxLLMCalls: 5, maxTokens: 10000, maxCostUsd: 1.0, maxDurationMs: 120_000 },
enabled: true,
})
5a. cron-apollo — daily 8am ICT lead generation
Schedule 0 1 * * 1-5 (08:00 ICT Mon-Fri). Per-tenant scope. Budget: 200 LLM / 800 tool / 200K tokens / $5 / 5 min.
sequenceDiagram
autonumber
participant CRON as BullMQ repeat
participant FAN as CronFanoutProcessor
participant Q as trigger exec queue
participant WF as WorkflowExecutor
participant BATCH as salesRunCampaignsBatch
participant TI as ToolInvoker
participant CRM as Twendee CRM
participant APO as Apollo
participant LLM as LLM smart tier
participant TG as Telegram
CRON->>FAN: fire sales v0 cron apollo
FAN->>Q: enqueue per tenant jobs
Q->>WF: runPipeline batch notify fanout
WF->>BATCH: handler ctx params
BATCH->>TI: campaign_list_active
TI->>CRM: GET active campaigns
CRM-->>BATCH: up to 5 campaigns
loop per campaign
BATCH->>TI: campaign_search_apollo_leads
TI->>APO: POST mixed_people search with ICP filter
APO-->>BATCH: leads
BATCH->>TI: campaign_create_lead concurrency 5
TI->>CRM: upsert contact and link campaignId
BATCH->>LLM: draft personalized email
BATCH->>TI: campaign_save_email_template
TI->>CRM: save PENDING_REVIEW draft
end
BATCH-->>WF: perCampaign reportHtml upsertedIds
WF->>TI: telegramNotify chatId from agent_workflows
TI->>TG: sendMessage reportHtml
Note over TG: onError continue
WF->>WF: emitEvent salesLeadsImported with upsertedIds
chatId: 0 trong code chỉ là fallback. Tenants override schedule + chatId + ICP filter qua row agent_workflows. Hot-reload qua LISTEN/NOTIFY.
5b. event-rfq — RFQ inbound flow
Subscribe event rfqReceived. Concurrency 4. Budget: 4 LLM / 8K tokens / $0.20 / 60s.
flowchart LR PROD["Producer
REST controller / ERP webhook /
another agent"] -->|"emit rfqReceived"| EB["EventBus"] EB -->|"jobId via sha payload
match subscribers"| Q["trigger exec queue"] Q --> WF["WorkflowExecutor"] WF --> S1["clarify
step clarifyRequirements
rfq from trigger.rfq"] S1 --> S2["draft
step draftProposal
rfq + clarifications"] S2 --> S3["notify
step telegramNotify
onError continue"] S3 --> TG[("Telegram chat")] classDef st fill:transparent,stroke:#4d7c0f,color:inherit; classDef tr fill:transparent,stroke:#0f766e,color:inherit; class S1,S2,S3 st; class EB,Q,WF tr;
5c. message — Telegram chat fallback
Match { mention: false } — fallback agent cho mọi message không có @mention. Concurrency 8. Budget: 1 LLM / 2K tokens / $0.05 / 20s.
sequenceDiagram
autonumber
participant U as User
participant TG as Telegram Bot API
participant WH as TelegramController
participant IDM as Idempotency
participant MR as MessageRouter
participant Q as trigger exec queue
participant WF as WorkflowExecutor
participant CHAT as defaultChatLoop
participant TI as ToolInvoker
participant CRM as Twendee CRM
participant LLM as LLM
U->>TG: message
TG->>WH: POST telegram webhook
WH->>IDM: dedupe by update id
WH->>MR: dispatch tenant channel text conversationId
MR->>MR: resolve agent by lead mention keywords fallback
MR->>Q: enqueue sales v0 message trigger
Q->>WF: runPipeline chat
WF->>CHAT: handler ctx
CHAT->>LLM: chat with tool erpQueryLead
alt tool call
LLM-->>CHAT: tool_call erpQueryLead args
CHAT->>TI: invoke erp_query_lead args mcpCtx
TI->>CRM: GET leads
CRM-->>CHAT: result
CHAT->>LLM: continue with tool result
end
LLM-->>CHAT: final reply
CHAT->>TG: channels reply conversationId text
TG->>U: bot message
06 Takeaways để nắm hệ thống
Mental model 1 dòng
Trigger → Pipeline of Steps → ToolInvoker → Per-tenant Adapter Bundle → 3rd-party / ERP. Tất cả còn lại (registry, budget, observability, retry, hot-reload) là plumbing quanh trục này.
Điểm vào khi đọc code
agent-runtime/agents/sales-v0.agent.ts— definitionagent-runtime/triggers/*— cron / event / messageagent-runtime/execution/workflow-executor.ts+step-executor.tsagent-runtime/steps/data/sales-run-campaigns-batch.step.ts— step composite phức tạp nhấtagent-runtime/tools/tool-invoker.service.ts— bridge sang MCP tool layerpackages/erp-adapters/src/*— ports + adapters
Câu hỏi mở
- Producer cụ thể của event
rfqReceivedhiện tại — REST controller riêng hay parse SendGrid inbound? (chưa verify trong scan). - Sales-v0 còn đang feature-flagged (
SALES_V0_RUNTIME_ENABLED) — đã bật trên production tenants nào? - Webhook trigger: ngoài Telegram/SendGrid hiện đã có third-party nào dùng
WebhookTriggerControllerchưa?