System Architecture & Data Flow
About This Page
What: System boundaries and tech stack, layered relationships among 32 microservices, database architecture and sync pipelines, inter-service communication protocols and latency analysis, the full path a Deposit/Withdrawal takes from initiation to completion (which services and tables it touches), Bank Statement collection frequencies, request tracing examples, monitoring and alerting, and common latency bottlenecks with troubleshooting approaches Audience: Product managers and Operations staff who need to understand the full technical landscape Prerequisites: Getting StartedReading time: 12 minutes Owner: Deposit & Withdrawal Product Team
Key takeaway: The system comprises 32 microservices organized into three tiers — a PHP business logic layer, a Go/Python bank integration layer, and a Python SBA Orchestration layer. Understanding these tiers is the key to locating issues and assessing the impact of changes.
Tech Stack Overview
| Tier | Technology | Components | PM Focus |
|---|---|---|---|
| Business Logic | PHP/Lumen 5.5 | Deposit (deposit/), Withdrawal (withdraw/), Bank Card (bankcard_service/), Cash (cash/) | Core business APIs, latency-sensitive — directly affects App response time |
| Bank Integration | Go + Python hybrid | Go: BOCHK statements, CGB FPS, SCB, blacklist/whitelist, Airstar gateway; Python: SBA Orchestration, HSBC, ICBC, CMB, CMBC | Multi-language heterogeneous bank Channels — you need to check the right language stack's logs when troubleshooting |
| Communication | Protobuf RPC + HTTP REST | SRPC (internal sync), HTTP (CRM/external), SM2 Socket (CMB/CMBC) | SRPC timeout → individual Deposit/Withdrawal stuck; HTTP timeout → CRM operation failure |
| Bank Protocols | B2E XML / MT910 / SFTP+GPG / SM2 encryption / REST API | One protocol per bank | Different protocols determine Bank Statement retrieval speed and automation level |
| Storage | MySQL (sharded) + Elasticsearch (Canal sync) + Redis | Primary DB sharded into 100 tables, ES full-text search, Redis cache + locks | CRM search uses ES (second-level latency), not direct database queries |
| Messaging | Redis event queues + RabbitMQ | Internal async events + cross-service decoupling | Queue backlog → batch Deposit/Withdrawal delays |
32 Microservices by Category
Full Service Inventory
Core Business (4)
| Service | Language | Port | Responsibility | External Interface |
|---|---|---|---|---|
deposit/ | PHP | 20001 | Deposit applications, Bank Statements, Matching, Approval, in-transit funds, reconciliation | SRPC + HTTP REST |
withdraw/ | PHP | 20002 | Withdrawal applications, 3-step Approval (Audit → Confirm → Remittance), bank Channels | SRPC + HTTP REST |
cash/ | PHP | 20003 | Cash service API and batch processing | HTTP REST |
bankcard_service/ | PHP | 20005 | Full Bank Card lifecycle management (binding/verification/unbinding) | HTTP REST |
Bank Integration Layer (13)
| Service | Language | Bank | Responsibility | Communication | Call Direction |
|---|---|---|---|---|---|
bochk_flow_go/ | Go | BOCHK | B2E protocol parsing, statement conversion | B2E XML | moomoo → Bank (active pull) |
bochk_deposit_mgr/ | Python | BOCHK | Deposit Orchestration management | SRPC | Internal call |
bochk_fts_mgr/ | Python | BOCHK | FTS (remittance) management | SRPC | Internal call |
bochk_relay/ | Python | BOCHK | Request forwarding proxy | HTTP | Internal forwarding |
hsbc_bank_flow_service/ | Python | HSBC | MT910 statement parsing, SFTP pull, GPG encryption | SFTP+GPG | Bank → moomoo (push + pull) |
hsbc_edda_report/ | Python | HSBC | eDDA report download | SFTP | moomoo → Bank (scheduled pull) |
sba_hsbc_eddi/ | Python | HSBC | eDDA/eDDI Deposit direct debit integration | REST API | moomoo → Bank (initiate + poll) |
sba_hase_eddi/ | Python | Hang Seng | eDDA/eDDI Deposit direct debit integration | REST API | moomoo → Bank (initiate + poll) |
scb_service/ | Go | SCB | FPS RPC interface | SRPC | moomoo → Bank |
cgb_fps_service/ | Go | CGB | FPS real-time transfer | SM2 encryption | moomoo → Bank |
icbc_be_relay/ | Python | ICBC | Bank-enterprise relay | RSA signature verification | moomoo → Bank |
cmb_stock_trans/ | Python | CMB | BST transfer (Socket binary bidirectional link) | SM2 Socket | Bidirectional real-time |
ms_stock_bank_transaction/ | Python | CMBC | BST transfer | SM2 Socket | Bidirectional real-time |
SBA Orchestration Layer (3)
| Service | Language | Responsibility | State Count |
|---|---|---|---|
sba_deposit_system/ | Python | Deposit Orchestration: create → review → Risk Control change → credit, supports 6 Deposit modes | 6 |
sba_cash_withdraw_system/ | Python | Withdrawal Orchestration: freeze → deduct → transfer → release, supports 4 phases and 17 states | 17 |
sba_fund_withdraw/ | Python | Fund Withdrawal: money market fund redemption → Withdrawal one-click Orchestration | — |
Risk Control (3)
| Service | Language | Responsibility | Operations Impact |
|---|---|---|---|
hk-deposit-blacklist-go/ | Go | Deposit blacklist (bulk add/remove/query) | Hit blocks the Deposit; Procedure → manual_confirm |
hk-deposit-whitelist-go/ | Go | Deposit whitelist | Whitelisted users take the fast track, skipping manual review |
hk-withdraw-blacklist-go/ | Go | Withdrawal blacklist | Hit sends the Withdrawal into the manual review queue |
Other (9)
| Service | Language | Responsibility | Status |
|---|---|---|---|
bank-card/ | PHP | Bank Card base API | Legacy service, gradually migrating to bankcard_service |
cash_service/ | PHP | Cash service extension | Active |
refund/ | PHP | Refund processing | Active |
airstar_service/ | Go | Airstar bank payment gateway (Mandate authorization / Deposit / Withdrawal) | Active — core of Airstar BST |
sync_bst_data_helper/ | Python | CMB abnormal statement recovery | Compensation service — used when CMB Socket disconnects |
tools-pic-upload/ | Go | Voucher image upload | Active |
bank_conn_doc/ | - | Bank integration specification documents (HSBC/BOCHK/Hang Seng/ICBC/EWB/SPD) | Documentation repository |
procedure_system/ | Python | Procedure Orchestration system (SBA core framework) | Active — the underlying engine of SBA |
Database Architecture
Three Core Databases
| Database | Owning Service | Core Tables | Sharding Strategy | Estimated Volume |
|---|---|---|---|---|
| Primary DB (cash_deposit / cash_withdraw) | deposit/, withdraw/, cash/, refund/ | applys, flows, matches, tasks, queue | applys sharded by uid modulo 100; flows sharded by bank type + month | Millions/month |
| web_account | bankcard_service/ | bank_card, bank_info, bank_card_certificate, bank_card_asb_bst, bank_card_edda | Not sharded | Hundreds of thousands |
| SBA Procedure DB | procedure_system/, sba_*_system/ | proc_info, proc_cash_deposit, proc_cash_withdraw, flow | Not sharded | Millions/month |
The three databases are physically isolated; services communicate via RPC calls rather than direct cross-database queries.
Impact of Sharding on Operations
Deposit application table applys is sharded into 100 tables by user ID (applys_00 through applys_99). CRM search is unaffected (uses ES), but if you need to query the database directly for troubleshooting, you must first calculate the table suffix from the user ID: uid % 100.
Bank Statement table flows is sharded by bank + month (e.g., flows_bochk_202604); cross-month queries require specifying the correct month.
Withdrawal task table tasks is not sharded, but for large data volumes, queries use indexes on created_at + status.
SBA Procedure table proc_info is not sharded; queries use id or nn_uid + biz_type.
Data Sync Pipeline (Canal CDC → Elasticsearch)
The CRM Operations console supports fuzzy search by card number, username, Bank Statement reference, etc. — this does not query the database directly but rather Elasticsearch. Data is synced in real time via Canal (Alibaba's open-source MySQL Binlog subscription tool):
PM perspective: CRM search results are near real-time (second-level latency). If a search returns "not found," it does not necessarily mean the data doesn't exist — it may be a Canal sync delay. Waiting a few seconds and searching again usually resolves it.
Canal-Synced Entities
| Entity | Source Table | ES Index Purpose | Supported Search Fields | CRM Search Entry Point |
|---|---|---|---|---|
ApplySync | Deposit applications applys | CRM Deposit list search | uid, Bank Card number, amount, status, creation time | Deposit Management → Search |
FlowSync | Bank Statements flows | CRM statement search, Matching troubleshooting | statement reference, amount, bank, currency, credit time | Statement Management → Search |
TaskSync | Withdrawal tasks | CRM Withdrawal list search | uid, target card number, amount, status, creation time | Withdrawal Management → Search |
FileSync | Voucher files | CRM voucher search | filename, upload time, associated uid | Voucher Management |
DepositSbaListSync | Deposit SBA association | Deposit–Procedure association query | apply_id, procedure_id | Deposit Details → SBA Association |
ApplyAdditionSync | Deposit supplementary info | Supplementary search fields | remarks, additional tags | Deposit Details → Supplementary Info |
Troubleshooting steps when a record cannot be found:
- Wait 5–10 seconds and search again (Canal sync delay)
- Try different search criteria (e.g., use uid instead of card number)
- Check Canal Consumer logs for errors
- Query the database directly to confirm whether the data exists
Inter-Service Communication
| Protocol | Use Case | Typical Call | Characteristics | PM Significance |
|---|---|---|---|---|
| SRPC (Protobuf RPC) | Core services ↔ SBA / Risk Control | deposit/ calls CashDepositCreate | Synchronous, strongly typed, millisecond-level | Timeout directly affects user wait time |
| HTTP REST | Bank Card service / Airstar gateway / CRM | CRM calls /bank_card/get_list | Standard Web API | Easy to monitor and debug |
| SM2 Socket | CMB/CMBC BST | Binary bidirectional link | Persistent connection, real-time push | Bank-side results arrive in seconds |
| REST API (polling) | Airstar BST / HSBC & Hang Seng eDDA | AsbBstCreateResultJob polling | Requires active querying | Longer user wait time |
| RabbitMQ | Trade notification → Risk Control; cross-service async | RmqReportMsg triggers Risk Control update | Async decoupling | Queue backlog affects batch processing |
Impact of Communication Methods on Latency
| Scenario | Communication Path | Typical Latency | Latency Reason | Impact Scope |
|---|---|---|---|---|
| BST Deposit (CMB/CMBC) | SM2 Socket real-time push | Seconds | Bank actively pushes result | Single |
| BST Deposit (Airstar) | REST API + polling | Seconds to minutes | Polls up to 60 times | Single |
| BST Withdrawal (Airstar) | REST API + polling | Seconds to minutes | Polls up to 10 times | Single |
| eDDA Direct Debit Deposit | REST API + polling | T+0 to T+1 | Bank processes asynchronously | Single |
| Standard Deposit Matching | Scheduled task | Every 3 minutes | Matching engine runs on schedule | Batch |
| Online Banking Withdrawal | Manual operation | Hours to days | Depends on Operations completing transfer via online banking | Single |
| CRM Search | Canal CDC → ES | Seconds | Binlog sync delay | CRM |
PM perspective: If a Deposit is experiencing unusual delay, the bottleneck may be in an SRPC synchronous call (e.g., SBA Procedure creation timeout) or Canal async queue backlog. SRPC issues affect individual transactions; queue backlog affects batches.
Deposit Data Flow: How Funds Arrive
Standard Deposit (User Perspective)
| Step | User Action | User Perception | Expected Duration |
|---|---|---|---|
| 1 | Select Bank Card in App, get Deposit account info | Sees moomoo receiving account number | Instant |
| 2 | Transfer to moomoo account via banking app / online banking | Bank shows transfer successful | 1–5 minutes |
| 3 | Wait for system to match and credit | App shows "Deposit processing" | 3 minutes to hours |
| 4 | Deposit complete | App balance increases, notification received | — |
System Perspective
Core Tables in the Deposit Data Path
| Phase | Table Name | Sharding Rule | Description | Operations View Location |
|---|---|---|---|---|
| 1. User application | applys_{00-99} | uid % 100 | One row = one Deposit application | CRM → Deposit Management |
| 2. Bank Statement | flows_{bankType}_{YYYYMM} | Bank type + month | One row = one Bank Statement entry | CRM → Statement Management |
| 3. Matching | matches_{shard} | — | Association record between Flow and Apply | CRM → Matching Records |
| 4. Deposit task | deposits_{shard} | — | Matched Deposit tasks | CRM → Deposit List |
| 5. Anomaly | abnormal_deposits | Not sharded | Abnormal statements tracked separately | CRM → Abnormal Deposits |
| 6. SBA | proc_info + proc_cash_deposit | Not sharded | Procedure tracking records | CRM → SBA Management |
Full state machine and field descriptions for Deposit Procedures → SBA Funds Orchestration - Deposit Procedure State Machine
Withdrawal Data Flow: How Funds Are Sent Out
Withdrawal (User Perspective)
| Step | User Action | User Perception | Expected Duration |
|---|---|---|---|
| 1 | Select Withdrawal Bank Card and amount in App | — | Instant |
| 2 | Confirm Withdrawal | Available balance decreases (frozen) | Instant |
| 3 | Wait for system processing | App shows "Withdrawal processing" | BST: seconds / Online banking: hours |
| 4 | Withdrawal complete | Receives Withdrawal success notification | — |
| 5 | Bank arrival | Bank Card receives funds | 1–3 business days |
System Perspective
Core Tables in the Withdrawal Data Path
| Phase | Table Name | Description | Operations View Location |
|---|---|---|---|
| 1. Create task | tasks | Withdrawal task core record | CRM → Withdrawal Management |
| 2. Operation log | flows | Operation log for each Approval step (staff_id, action, step) | CRM → Operation Records |
| 3. Event queue | queue | Async events (BST sync, high-risk checks, etc.) | Internal system |
| 4. SBA tracking | sba_list | Withdrawal Task ↔ SBA Procedure association | CRM → SBA Status |
| 5. BST settings | auto_settings | Auto-Approval balance/limit config by bank × currency | CRM → BST Settings |
| 6. Bank Statement | cmb_list / ms_list | Withdrawal records initiated from the CMB/CMBC bank side | CRM → BST Withdrawal |
Full state machine for Withdrawal Procedures (17 states) and the freeze-transfer-release pattern → SBA Funds Orchestration - Withdrawal Procedure State Machine
Bank Statement Collection Frequency
Different banks have vastly different statement retrieval methods and frequencies — this directly affects Deposit crediting speed:
| Bank | Collection Method | Frequency | Notes | Expected Deposit Crediting | Automation Level |
|---|---|---|---|---|---|
| BOCHK | B2E active pull | 3x daily (06:00/07:00/08:00) + 2-hour conversion | TodayActivity → AcctStatement → AcctActivity | T+0 (same day) | Semi-auto |
| HSBC | MT910 real-time push + SFTP pull | Real-time + batch fallback | Transaction Reference Number deduplication | Minutes | Fully auto |
| ICBC | Bank-enterprise active pull | Daily real-time pull | Query by date range, RSA signature verification | T+0 | Semi-auto |
| CMB | Bidirectional link passive receive | Event-driven (real-time) | Received in real time when bank initiates | Seconds | Fully auto |
| CMBC | Bidirectional link passive receive | Event-driven (real-time) | Received in real time when bank initiates | Seconds | Fully auto |
| Airstar | REST API polling | Polling (up to 60 times) | Polls for result after Deposit creation | Seconds to minutes | Fully auto |
| Hang Seng | eDDA direct debit + REST API | Poll after initiating debit | Processed via sba_hase_eddi | T+0 to T+1 | Fully auto |
| EWB | CSV + BAI2 file | On-demand upload | CSV deduplication + BAI2 name supplement | Manual processing | Manual |
| Other banks | Deposit Matching task scan | Every 3 minutes | CCB Asia, DBS, ZA Bank, etc. | 3 minutes to hours | Semi-auto |
Statement retrieval delay ≠ crediting delay
Statement retrieval is only the first step of the Deposit process. Afterward, the Matching engine must compare records (every 3 minutes), SBA creates a Procedure, Risk Control changes are applied, and more. BST Channels skip the Matching step (directly creating a Procedure), which is why they credit the fastest.
Cumulative latency estimates by stage:
| Deposit Channel | Statement Retrieval | Matching | SBA + Risk Control | Total |
|---|---|---|---|---|
| BST (CMB/CMBC) | Seconds | Skipped | Milliseconds | Seconds |
| BST (Airstar) | Seconds to 3 min | Skipped | Milliseconds | Seconds to 3 min |
| eDDA (Hang Seng/HSBC) | T+0 to T+1 | Skipped | Milliseconds | T+0 to T+1 |
| Standard Deposit (HSBC) | Minutes | 3 min | Milliseconds | 3–10 minutes |
| Standard Deposit (BOCHK) | 3x daily | 3 min | Milliseconds | Hours |
| Standard Deposit (Other) | 3 min | 3 min | Milliseconds | 6–30 minutes |
Request Tracing
Deposit Trace Example: A CMB BST Deposit
User initiates BST Deposit of 10,000 HKD via CMB banking app
↓
① cmb_stock_trans receives SM2 Socket push (< 1 sec)
→ Decrypts message, extracts: card number, amount, currency, market
↓
② deposit/ receives Deposit notification (SRPC, < 100ms)
→ Creates Apply record (applys_xx table)
→ Blacklist/whitelist check (parallel RPC)
↓
③ sba_deposit_system/ creates Procedure (SRPC, < 100ms)
→ Writes proc_info + proc_cash_deposit
→ Auto-review (deposit_type = TRANS_AUTO)
↓
④ Risk Control system asset change (SRPC, < 200ms)
→ Balance increases by 10,000 HKD
↓
⑤ Procedure → end_ok(deposit_ok) (< 50ms)
→ Writes back Apply status → DONE
→ Canal syncs to ES (< 3 sec)
↓
⑥ User sees balance increase in App
Total elapsed: ~1–3 secondsWithdrawal Trace Example: An Online Banking Withdrawal
User initiates 50,000 HKD Withdrawal to Hang Seng Bank via moomoo App
↓
① withdraw/ creates Task (< 100ms)
→ Blacklist check
→ Bank Card verification (bankcard_service/ HTTP)
→ Step 1 Audit: auto-pass (standard template)
→ Step 2 Confirm: parameter validation
→ Step 3 Remittance: create SBA Procedure
↓
② sba_cash_withdraw_system/ freezes funds (< 200ms)
→ Risk Control system freezes 50,000 HKD
→ Procedure: new(freeze) → new(waiting) → new(blank)
↓
③ Deduction (< 200ms)
→ Risk Control system formally deducts 50,000 HKD
→ Procedure: pending(deduct_done)
↓
④ Enter manual transfer (Hang Seng uses corporate online banking)
→ Procedure: pending(transfer_manual)
→ **Awaiting Operations to complete transfer via Hang Seng corporate online banking**
↓
⑤ After Operations completes transfer on online banking (hours to days)
→ Click "Confirm Transfer Complete" in CRM
→ SetManualTransferDone
→ Procedure: end_ok(transfer_done)
↓
⑥ User receives funds 1–3 business days later
Total elapsed: hours to 1 business day (depends on Operations processing speed)Monitoring & Alerting
Core Monitoring Metrics
| Metric | Monitoring Method | Alert Threshold | Impact |
|---|---|---|---|
| Deposit crediting time | Time from SBA Procedure creation to end_ok | BST > 5 min / Standard > 1 hour | User experience |
| Withdrawal processing time | Time from Task creation to end_ok | BST > 10 min / Online banking > 24 hours | User experience |
| Matching engine execution | Scheduled task execution logs | Not executed for > 6 minutes | Batch Deposit delays |
| Canal sync delay | ES latest record time vs MySQL latest record time | Delay > 30 seconds | CRM cannot find latest data |
| SM2 Socket connection | Heartbeat detection (CMB/CMBC) | Heartbeat timeout > 60 seconds | BST Channel unavailable |
| Polling task timeout | Job retry_count | Deposit > 60 / Withdrawal > 10 | Airstar BST transaction stuck |
| Freeze not released | Procedure duration in pending(unfreeze) | > 5 minutes | User funds stuck |
| Reversal failure | Procedure duration in pending(return) | > 5 minutes | User funds "disappear" |
Alert Notification Methods
| Severity | Alert Method | Response SLA | Typical Scenario |
|---|---|---|---|
| P0 Critical | Feishu group + phone call | Within 15 minutes | Freeze not released, reversal failure, SM2 Socket disconnection |
| P1 Important | Feishu group | Within 30 minutes | Polling timeout, Matching engine stopped, prolonged Canal sync delay |
| P2 Normal | Feishu group | Within 2 hours | Individual Deposit/Withdrawal delay exceeds expectations |
Detailed alert rule configuration, scheduled task inventory, and Dashboard URLs → Scheduled Tasks & Monitoring
Common Latency Bottlenecks & Troubleshooting
When Deposits or Withdrawals experience delays, the issue typically lies in one of the following areas:
| # | Bottleneck | Symptom | Troubleshooting Approach | Impact Scope | Severity |
|---|---|---|---|---|---|
| 1 | Bank Statement not received | Apply created but no Matching statement | Check the corresponding bank integration service logs; confirm whether the bank side has completed the transfer | Single | P2 |
| 2 | Matching engine delay | Statement in database but not matched to Apply | Check whether the Matching scheduled task is running normally (every 3 minutes) | Batch | P1 |
| 3 | SBA Procedure creation timeout | Match successful but Procedure not created | Check whether SBA service is healthy, whether SRPC is timing out | Single | P1 |
| 4 | Risk Control change failed | Procedure created successfully but status is end_reject | Check the Procedure's reason field | Single | P2 |
| 5 | Canal sync delay | CRM cannot find existing records | Wait a few seconds and search again; check whether Canal Consumer is running normally | CRM | P2 |
| 6 | BST polling timeout | Airstar BST Deposit/Withdrawal stuck in processing | Check polling task retry count (Deposit 60 / Withdrawal 10) | Single | P1 |
| 7 | Scheduled script not running | Withdrawal Procedure stuck at new(waiting) | Check whether the Withdrawal scheduled script is executing normally | Batch | P0 |
| 8 | Manual transfer not confirmed | Withdrawal Procedure stuck at pending(transfer_manual) | Remind Operations to confirm after completing the transfer via online banking | Single | P2 |
| 9 | SM2 Socket disconnection | CMB/CMBC BST Channel completely unavailable | Check cmb_stock_trans / ms_stock_bank_transaction service status | Channel-wide | P0 |
| 10 | RabbitMQ backlog | Multiple Deposits/Withdrawals delayed in batch | Check queue depth and consumer status | Batch | P1 |
Quick Troubleshooting Decision Tree
Inter-System Call Relationships
After Reading
| I want to... | Go to |
|---|---|
| Understand what the system covers and what it doesn't | Getting Started - 30-Second Overview |
| Dive deeper into SBA Orchestration | SBA Funds Orchestration |
| Dive deeper into the Bank Card system | Bank Cards & Authorization |
| See integration methods for each bank | Bank Capability Matrix |
| See alerts and scheduled tasks | Scheduled Tasks & Monitoring |
Was this page helpful?