Bank Statement Collection
About This Page
What: How each bank's statements reach the system, flow table structure and sharding strategy, collection cron jobs, and flow type codes Audience: Product managers who need to understand "where the matching engine's input comes from" Prerequisites: Deposit Methods OverviewReading time: 5 minutes Owner: Deposit Product Manager
Key Takeaways: Bank statement collection is the input source for the matching engine. Different banks have vastly different collection methods — from real-time push in seconds to batch pulls only 3 times daily — directly determining the user's deposit waiting time.
Quick Navigation — What you might want to do:
- User says "money hasn't arrived," want to check if the statement came in? -> Summary Comparison Table (see bank latencies)
- Want to understand how a specific bank's statements come in? -> Statement Collection by Bank
- Want to know how the flow table is sharded? -> Flow Table Structure & Sharding
- Want to onboard a new bank's statement collection? -> Change Guide - Adding a Bank
- Want to see collection cron jobs? -> Collection Cron Jobs
Why This Page Exists
The core prerequisite for Push mode (FPS, online banking, ATM, etc.) deposits: bank statements must first reach the system before the matching engine can work.
Different banks' statement arrival methods vary dramatically — from real-time push in seconds to batch pulls only 3 times daily. This directly determines the waiting time from user transfer to seeing "deposit successful." Understanding statement collection mechanisms is the foundation for diagnosing "money hasn't arrived" issues and evaluating bank onboarding proposals.
eDDA/BST Don't Go Through Statement Collection
Pull mode (eDDA) and Direct mode (BST) deposits do not rely on statement collection — the system initiates the debit/transfer itself and knows when it completes. This page only covers Push mode statement collection.
Statement Collection by Bank
Real-time Push
HSBC — MT910 Real-time Notification
| Dimension | Description |
|---|---|
| Protocol | SWIFT MT910 format, transmitted via SFTP + GPG encryption |
| Arrival Latency | Seconds — pushed immediately after bank credits |
| Collection Service | hsbc_bank_flow_service (Python) |
| TransType | 218 (HSBC) |
| Deduplication | Transaction Reference Number (:20: field) unique constraint |
MT910 is the SWIFT standard "credit confirmation" message. The system periodically downloads GPG-encrypted files from SFTP, decrypts and parses the MT910 format, extracting transaction reference number, amount, currency, payer name, and other fields.
Key Parsing Fields:
:20:-> Transaction reference number (deduplication identifier):32A:-> Date + Currency + Amount (compound field: YYMMDD + CCY + Amount):50K:-> Payer name (used for name matching):72:-> Remarks (supplementary information)
SCB / CGB — FPS API Real-time
| Dimension | SCB | CGB |
|---|---|---|
| Protocol | Webhook callback + REST API | REST API batch/single |
| Arrival Latency | Seconds | Seconds |
| Collection Service | scb_service (Go) | cgb_fps_service (Go) |
| TransType | 219 (SC_SUBACC) | FPS related |
| Queue Mechanism | DB Queue (9 Job Handlers) | DB Queue (4 Job Handlers) |
SCB actively pushes transaction events to the system via Webhook. The system persists the event first then enqueues for processing (ProcessWebhookEvent -> Save -> Enqueue). CGB submits batch or single payment tasks via REST API, and the system polls for results via query Jobs.
CMB/MSB BST — Socket Bidirectional Link
| Dimension | Description |
|---|---|
| Protocol | Binary Socket bidirectional link, message-level encryption |
| Arrival Latency | Seconds — event-driven, bank actively pushes |
| Collection Service | cmb_stock_trans / ms_stock_bank_transaction |
| TransType | 101 (MSB BST), 102 (CMB BST), 304 (Airstar BST) |
BST (Bank-Securities Transfer) communicates via long-connection Socket. After the bank completes fund transfer, it actively pushes the result through the established Socket link. No system polling required.
Note the Distinction
BST statements are part of Direct mode and don't go through the matching engine. However, BST statements are still written to the flows table for reconciliation and monitoring purposes.
Scheduled Pull
BOCHK — B2E Batch Pull
| Dimension | Description |
|---|---|
| Protocol | B2E (Business-to-Enterprise) XML-RPC |
| Arrival Latency | 3 pulls per day + ~2 hours conversion processing |
| Collection Service | bochk_flow_go (Go) |
| TransType | 301 (BOC) |
| Deduplication | Pre-query DB to skip existing records |
BOCHK B2E is the slowest statement source. The system actively pulls 3 times daily:
| Batch | Time | Content |
|---|---|---|
| TodayActivity | ~06:00 | Today's settled transactions |
| AcctStatement | ~07:00 | Previous day's account statement |
| AcctActivity | ~08:00 | Recent 2 days' transaction details (including counterparty info) |
After pulling, ~2 hours of flow conversion (TrdRecordFlowConverter / TransactionRecordFlowConverter) is needed to standardize raw records into boc_bank_flow format before writing to the flows table.
PM Focus: BOCHK deposit "slow crediting" mainly comes from statement collection delay, not the matching engine. This is why BOCHK's matching time window is -3~+4 days (2 days wider than the standard -3~+2).
ICBC Asia — Banking API Polling
| Dimension | Description |
|---|---|
| Protocol | Bank-enterprise direct REST API, RSA signature verification |
| Arrival Latency | Minutes |
| Collection Service | icbc_be_relay (Python) |
| TransType | 202 (ICBCASIA) |
ICBC Known System Limitations
ICBC is the second largest deposit source (about 12.3%), but has the most system limitations among all banks. PMs and operations should pay special attention to these known issues:
| Limitation | Impact | Mitigation |
|---|---|---|
| No unique transaction ID | Deduplication can only rely on composite fields (amount+date+card number+remarks), higher duplicate risk than other banks | Matching engine adds extra deduplication validation logic |
| T-1 late statements | Some statements arrive the day after the transaction date (not real-time) | Matching time window extended by 1 extra day |
| Missing balance check for refund/reversal | Cannot automatically verify securities account balance sufficiency before reversal | Operations must manually check balance before reversal |
| Unstable statement format | ICBC occasionally adjusts statement message format without prior notice | Statement parser maintains compatibility redundancy |
Extra Regression Required for ICBC Changes
Due to the above limitations, any changes involving matching rules or statement parsing require extra regression testing specifically for ICBC — particularly deduplication logic and T-1 statement scenarios. For ICBC-related troubleshooting -> Deposit Troubleshooting - ICBC Duplicate Statements
Other API Polling
| Bank | Collection Service | TransType | Arrival Latency |
|---|---|---|---|
| DBS | dbs_service | 208 | Minutes |
| CCB Asia | API polling | 206 | Minutes |
| ZA Bank | API polling | 221 | Minutes |
| Airstar | API callback | 304 | Seconds |
Manual Upload
EWB — CSV + BAI2 Files
| Dimension | Description |
|---|---|
| Protocol | Operations manually upload CSV / BAI2 files to OA backend |
| Arrival Latency | Depends on operations, typically 1-2 times daily |
| TransType | 217 (EWB_SUBACC) |
CSV provides basic transaction information; BAI2 supplements payer name and other fields needed for matching. After upload, the system automatically parses, deduplicates, and writes to the flows table.
Summary Comparison Table
| Bank | Collection Method | Protocol | Arrival Latency |
|---|---|---|---|
| HSBC | Real-time push | MT910 via SFTP | Seconds |
| SCB | Real-time push | Webhook + API | Seconds |
| CGB | Real-time push | FPS REST API | Seconds |
| CMB/MSB | Real-time push | Socket bidirectional link | Seconds |
| Airstar | API callback | REST API | Seconds |
| ICBC | Scheduled pull | Banking API | Minutes |
| DBS | Scheduled pull | API | Minutes |
| CCB | Scheduled pull | API | Minutes |
| ZA Bank | Scheduled pull | API | Minutes |
| BOCHK | Batch pull | B2E XML-RPC | Hours |
| Hang Seng | Special interface | — | Minutes~hours |
| EWB | Manual upload | CSV + BAI2 | Depends on operations |
Bank Statement Arrival Latency Visualization
From user transfer to statement entering the system, the waiting experience varies dramatically:
| Latency Level | Banks | Typical Wait | User Perception |
|---|---|---|---|
| Seconds | HSBC(MT910), SCB(Webhook), CGB(FPS API), Airstar(API) | <1 minute | "I just transferred and it's already showing" |
| Minutes | ICBC(Banking API), DBS(API), CCB(API), ZA(API) | 5~30 minutes | "It arrived after a short while" |
| Hours | BOCHK(B2E 3x daily) | 2~6 hours | "Transferred in the morning, arrived in the afternoon" |
| Uncertain | Hang Seng(special interface), EWB(manual upload) | Hours~next day | "Needs to wait for operations to process" |
PM Focus: When a user complains "money hasn't arrived," first check which bank — if it's BOCHK, 2~6 hours of waiting is normal, not a system failure.
If Requirements Change: Statement Collection Related
| Change | Location | Notes |
|---|---|---|
| Add new bank statement collection | New Go/Python service + crontab config | Refer to Deposit Change Guide - Scenario 7 |
| Modify B2E pull frequency (BOCHK) | bochk_flow_go/conf/conf.toml | Adjust cron expression |
| Modify matching frequency | deposit/doc/crontab.sh -> corresponding match:* | Refer to Deposit Change Guide - Scenario 6 |
| Replace SFTP/API credentials | Configuration file of corresponding bank collection service | Usually requires ops deployment |
| Modify Flow Converter frequency (BOCHK) | bochk_flow_go cron config | Currently every 2 hours |
Statement Collection Exception Troubleshooting
When statements are not coming in or processing abnormally, follow this path.
Quick Diagnosis: Why Statements Haven't Arrived
Step-by-Step Runbook: Received "Bank Statement Not Coming In" Alert
Minute 1 — Assess impact scope
- Review alert content: which bank? How many pending statements are queued?
- In OA statement list, filter by that bank, check the arrival time of the most recent statement
- If most recent statement was a few minutes ago -> possible false alarm (statements coming in normally, just matching delayed)
- If most recent statement was hours ago -> confirm collection is interrupted
Minutes 2~5 — Locate collection service 5. Check if the corresponding collection service process is alive:
- HSBC:
hsbc_bank_flow_service(Python) - SCB:
scb_service(Go) - CGB:
cgb_fps_service(Go) - BOCHK:
bochk_flow_go(Go) - ICBC:
icbc_be_relay(Python)
- Check recent service logs -> any connection timeouts, auth failures, parse exceptions
Minutes 5~10 — Attempt recovery 7. If service is down -> restart service, observe 1~2 cycles for new statements 8. If auth/certificate expired -> need to update credentials (contact bank relationship team for new credentials) 9. If bank side didn't push data -> not a system failure, record and contact bank relationship team
Minutes 10~15 — Compensation processing 10. After service recovery, check if any statements were missed during interruption (compare with bank reconciliation statement) 11. If missed -> BOCHK can manually trigger a supplemental pull; other banks need to contact the bank to re-push
Not recovered after 15 minutes -> Escalate to technical operations on-call
Step-by-Step Runbook: Matching Cron Job Not Executing
Minute 1 — Confirm the issue
- Check if the corresponding
match:{bank}Cron is running - Check the execution time and result of the most recent matching job
Minutes 2~5 — Investigate cause 3. Is the Cron process alive? If the Cron scheduler itself is down -> all banks' matching will stop 4. Check if a matching job is running too long (over 3 minutes), causing the next round to be skipped 5. Check if DB connection is normal — matching jobs need simultaneous access to flows and applys tables
Minutes 5~10 — Recovery 6. Restart Cron scheduler 7. Manually trigger one matching job, confirm successful execution 8. Observe whether the next cycle automatically recovers
Not recovered after 10 minutes -> Escalate to technical operations
Flow Table Structure & Sharding
Sharding Strategy
Flow tables are sharded by bank type + month:
flows_{trans_type}_{YYYYMM}Example: flows_218_202604 = April 2026 HSBC MT910 statements.
Why shard this way:
- By
trans_type: Different banks have very different flow fields; separating allows independent index optimization per bank - By month: High volume of statements; monthly archiving allows historical months to be progressively frozen
Configuration Location
Sharding logic: deposit/src/app/Business/Flow.php:80 Auto-create when table doesn't exist: CREATE_WHEN_TABLE_NOT_FOUND config
Flow Status Lifecycle
| Status Code | Meaning | Description |
|---|---|---|
| 0 | Pending | Statement stored, awaiting matching engine processing |
| 1 | Processed | Successfully matched and deposit task created |
| 2 | Error | Problem statement requiring manual intervention |
| 3 | Locked | Operations locked, temporarily excluded from matching (e.g., waiting for user supplementary info) |
| 4 | In-transit | Transfer confirmed but statement info incomplete, awaiting completion |
| 9 | Soft deleted | Marked as deleted (not physically deleted) |
Flow Disposition Results
When a statement is processed, the system records the disposition method:
| Result Code | Meaning | Trigger Scenario |
|---|---|---|
| 0 | System matched | Matching engine auto-matched successfully |
| 1 | Manually added | Operations manually associated in OA backend |
| 2 | Deposit initiated | Operations directly initiated deposit from statement |
| 3 | Refund | Funds corresponding to the statement need to be returned (see Refund & Reversal) |
| 4 | Other | Non-deposit statements (e.g., interest, fees) |
| 5 | Emergency processed | Processed through abnormal deposit flow |
| 6 | Auto-deposited | Matching engine auto-matched + auto-credited |
Configuration Location
Status codes: deposit/src/app/Business/BankFlow.php:59-64 Result codes: deposit/src/app/Business/BankFlow.php:80-87
Collection Cron Jobs
Matching Crons
The matching engine scans each bank's pending statements every 3 minutes:
| Job Name | Frequency | Bank | Description |
|---|---|---|---|
match:ccbasia | Every 3 min | CCB Asia | CCB statement matching |
match:ewb | Every 3 min | EWB | East West Bank matching |
match:boc | Every 3 min | BOCHK | BOCHK B2E matching |
match:hangseng | Every 3 min | Hang Seng | Hang Seng direct connection matching |
match:hsbc | Every 3 min | HSBC | HSBC MT910 matching |
match:main icbc-new | Every 3 min | ICBC | ICBC new version matching |
match:main za-sub-account | Every 3 min | ZA Bank | ZA sub-account matching |
Monitoring Crons
| Job Name | Frequency | Description |
|---|---|---|
monitor:flow-monitor | Every 30 min (07:00~23:00) | Detect statement backlog — alert when pending statements exceed threshold |
abnormal-deposit:search | Every 30 min | Scan for unmatched orphan statements |
abnormal-deposit:update-status | Every 3 min | Update abnormal deposit status |
Reconciliation Crons
| Job Name | Frequency | Description |
|---|---|---|
bank_reconciliation:generate | 16:15 daily | Generate today's reconciliation report (CNH/HKD/USD) |
bank_reconciliation:generate --date=yesterday USD | 09:05 daily | Generate yesterday's USD reconciliation (due to USD T+1 settlement) |
crmbos_reconciliation:generate | Hourly on the hour | CRM-BOS inter-system reconciliation |
Configuration Location
All Cron definitions: deposit/doc/crontab.sh
Bank Statement Type Codes
Different banks' statements carry different type fields identifying the transfer method. The matching engine selects different matching rules based on type:
BOCHK
| Type | Remarks Keyword | Meaning | Matching Rule |
|---|---|---|---|
| Transfer | FPS | FPS transfer | Real-time matching window |
| Transfer | E-BANKING TRANSFER | Online banking transfer | Standard matching window |
| Transfer | CHATS | CHATS clearing transfer | Standard matching window |
| Transfer | REMIT IN | Inward remittance | Cross-region tolerance |
| Transfer | CBS TRANSFER | CBS system transfer | Standard matching window |
| Clearing Cheque | — | Cheque clearing | Standard matching window |
Hang Seng
| Type Code | Meaning | Matching Rule |
|---|---|---|
| ATM | ATM deposit/transfer | Standard matching |
| GT | Counter transfer | Standard matching |
| WY | Online banking transfer | Standard matching |
| ZP | Cheque | Standard matching |
| BP | Bill Payment | Standard matching |
ICBC
| Type Keyword | Meaning |
|---|---|
| 匯款存入 | Remittance deposit |
| FPS 轉賬 | FPS transfer |
| 網上轉賬存款 | Online banking deposit |
| 櫃員機跨行轉賬存款 | ATM interbank transfer deposit |
| 支票 | Cheque deposit |
Connection to the Matching Engine
Once a statement is written to the flows table with status = 0 (PROCESSING), it will be scanned in the next 3-minute matching cycle.
Detailed matching engine logic -> Matching & Auto-crediting
Common Misconceptions
| Misconception | Fact |
|---|---|
| "Statements arriving means immediate crediting" | Statement arrival is just the first step. Still need to wait for the matching engine's next 3-minute cycle to scan, and then match successfully before crediting. Minimum one matching cycle wait |
| "BOCHK slow deposits are the matching engine's fault" | No. BOCHK is slow because B2E only pulls 3 times daily + 2 hours conversion time. Statements haven't even entered the system; the matching engine has nothing to match |
| "EWB statements are automatically collected" | No. EWB requires operations to manually upload CSV/BAI2 files. If operations hasn't uploaded, there are no statements in the system |
| "BST and eDDA also need statement collection" | No. BST and eDDA are system-initiated debits; the system knows the results itself. Statement collection on this page only covers Push mode |
| "All banks have the same statement format" | Completely different. Every bank has different fields, encoding, and time formats, so each bank has independent collection services and parsing logic |
After Reading
| I want to... | Go to |
|---|---|
| Understand matching logic after statement arrival | Matching & Auto-crediting |
| Troubleshoot "statement not in system" | Deposit Troubleshooting |
| Look up TransType and flow status codes | Deposit Quick Reference |
| See detailed integration rules for a specific bank | Bank Capability Matrix |
| Understand the collection service's position in the architecture | System Architecture & Data Flow |