Scheduled Tasks & Monitoring

About This Page

What: Complete list of scheduled tasks (name / frequency / responsibility / dependencies) and Monitoring Alert rules with thresholds Audience: Product managers and Operations staff who need to look up automated system tasks and Alert rules Prerequisites: Architecture & Data FlowReading time: 4 minutes Owner: Technical Operations Lead

Key takeaway: The deposit and withdrawal system relies heavily on scheduled tasks to drive asynchronous workflows. Monitoring Alerts cover four categories — Bank Statement collection, Matching engine, bank callbacks, and balance circuit breaker. A hung task is the most common root cause of anomalies.

---

Scheduled Tasks Overview

The deposit and withdrawal system relies heavily on scheduled tasks to drive asynchronous workflows. They are grouped into four business domains:

Domain	Task Count	Core Responsibility
BST (Bank-Securities Transfer)	~7	Deposit/Withdrawal instruction dispatch, result polling, status sync
Deposit Matching	~3	Bank Statement Matching, anomaly detection, timeout handling
eDDA/eDDI	~4	Authorization polling, Direct Debit instruction result queries
Reconciliation	~2	Daily reconciliation, discrepancy detection

---

BST Scheduled Tasks

Airstar BST Tasks

Job Name	Responsibility	Trigger	Retry	Notes
AsbBstCreateJob	Create deposit instruction	Triggered by user deposit	—	Calls Airstar API to send deposit request
AsbBstCreateResultJob	Poll deposit result	Scheduled polling	Up to 60 times	Queries Airstar API for deposit status
AsbBstDepositJob	Process deposit crediting	Polling returns SUCCESS	—	Executes SBA crediting orchestration
AsbBstTransfer	Withdrawal transfer + polling	Triggered by Approval pass	Up to 10 times	Sends withdrawal request and polls result
SyncAsbBstWithdraw	Withdrawal fallback sync	Scheduled	2-hour window	Fallback after AsbBstTransfer times out

Airstar deposit chain: AsbBstCreateJob → AsbBstCreateResultJob (60 polls) → AsbBstDepositJob

Airstar withdrawal chain: AsbBstTransfer (10 polls) → SyncAsbBstWithdraw (2h fallback)

CMB / Minsheng BST Tasks

Service Name	Bank	Responsibility	Notes
cmb_stock_trans	CMB	Bank-Securities Transfer	SM2-encrypted Socket, real-time bidirectional push
ms_stock_bank_transaction	Minsheng	Bank-Securities Transfer	SM2-encrypted Socket, real-time bidirectional push

CMB/Minsheng receive results in real time via a bidirectional Socket link — no polling tasks are needed. Results are pushed directly over the same connection.

For detailed BST task chains and bank comparisons → Mainland Bank BST Overview: Scheduled Tasks

---

Deposit Matching Scheduled Tasks

Task	Frequency	Responsibility	Notes
Auto-Matching round	Every 3 minutes	Execute Bank Statement–application Matching	Runs 5-dimension Matching on newly arrived Flows and pending Applies
AbnormalDepositJob	Scheduled	Abnormal deposit detection	Detects Flows that remain unmatched for an extended period; triggers notification
Auto-rejection on timeout	Scheduled	Overdue application handling	Automatically rejects unmatched applications after the configured number of days

Bank Statement arrival frequency by bank:

Bank	Statement Method	Arrival Frequency
HSBC	MT910 real-time push	Seconds
Citi / GDB	FPS API	Seconds
BOCHK	B2E batch pull	3 times daily
Other banks	Online banking transfer notification	Minutes to hours

For detailed Matching logic → Matching & Auto-Crediting

Auto-Matching Operating Window

Auto-Matching does not run 24/7. The HKD/CNH auto-Matching window is Monday 07:00 to Saturday 10:00; USD is daily 09:00-16:00. Bank Statements arriving outside the window only enter smart reminders (24/7) and are not auto-credited. See Manual Matching Guide: Auto-Matching vs Smart Reminders for details.

---

ICBC Bank Statement Monitoring

ICBC Bank Statement system uses a dual-AZ disaster recovery architecture with unique Monitoring requirements.

Dual-AZ Architecture

Deployment Group	Dedicated Line	Notes
Production group	Line 1	Primary Bank Statement pull (high frequency)
Canary group (backup)	Line 2	Backup Bank Statement pull (low frequency)

Disaster Recovery Limitation

The two dedicated lines cannot be switched — the production group can only use Line 1, and the canary group can only use Line 2. If either line fails, the corresponding deployment group cannot switch to the other. The system mitigates this via the Bank Statement management service's automatic routing.

Five Types of Production Issues and Handling

Type	Symptoms	Cause	Resolution
1. Customer refund	Both production and canary groups Alert	Customer requested a refund after transfer; bank deleted the Bank Statement	Most common; run the Alert handling script (auto-generates SQL script)
2. Refund script failure	Alert but the one-click handling script fails	Canary group pulls slowly; no subsequent Bank Statement after the refunded one; script stalls	Manually investigate canary group Bank Statement status
3. Duplicate Bank Statements	Only production group Alerts; canary group normal	Two Bank Statements have identical time, amount, and remarks; system cannot distinguish them	Modify close_balance; emergency-credit one of the Bank Statements
4. Bank pushes anomalous statement	Only production group Alerts; canary group normal	Bank first pushes an incorrect Bank Statement (balance same as previous entry), then deletes it	Verification: compare production/canary group statements line by line + compare against bank statement; delete unused statements then restart
5. Cross-day statement pushed late	Both production and canary groups Alert	Bank pushes T-1 day's Bank Statements after midnight	Modify previous day's close_balance; emergency-credit the missing Bank Statements

Key Concepts

Concept	Description
Balance verification	Each Bank Statement is verified on insertion: previous balance + current credit - current debit = current balance. When a refund causes a statement to be deleted, the balance chain breaks and triggers an Alert
is_used field	Indicates whether a Bank Statement has been used for Matching. After a full re-pull, statements with is_used=0 are unmatched
Production vs canary group differences	The production group pulls more frequently (potentially per-entry), while the canary group pulls less often (may receive multiple entries at once). They behave differently in refund and duplicate statement scenarios

ICBC Alert Handling Runbook

Minute 1 — Identify the Alert type:

Alert Source	Most Likely Type	Next Step
Production + canary group Alert simultaneously	1. Customer refund (most common)	Run the auto-handling script
Only production group Alerts	3. Duplicate statements or 4. Anomalous bank statement	Compare both groups' statements line by line
Auto-script execution failed	2. Refund script stalled	Manually investigate canary group status

Minutes 2-5 — Execute resolution:

• Type 1 (Customer refund):

  1. Confirm the flow_id in the Alert
  2. Run the auto-handling script (it auto-generates and executes corrective SQL)
  3. Verify: balance chain is restored; Alert auto-clears

• Type 3/4 (Production group only):

  1. Compare same-day statements between production/canary groups in the database, sorted by created_at
  2. Identify the differing statement → determine if it is a duplicate (Type 3) or a ghost statement (Type 4)
  3. Type 3: modify one entry's close_balance; emergency-credit
  4. Type 4: compare against bank statement to confirm → delete anomalous statements with is_used=0

If not resolved after 10 minutes → escalate to the on-duty technical Operations staff

Diagnostic Commands and Log Keywords

When an ICBC Alert is received, use these keywords to locate the issue:

Scenario	Service	Log Keywords	Notes
Balance chain break	icbc_be_relay	balance mismatch / close_balance	Statement deletion caused balance discontinuity
Duplicate statements	icbc_be_relay	duplicate flow / same remarks	Identical amount + time + remarks
Pull timeout	icbc_be_relay	timeout / connection refused	Dedicated line connection interrupted
Canary group delay	icbc_be_relay	canary lag / sync delay	Canary group statements arriving slower than production
Cross-day missing	icbc_be_relay	missing T-1 / late push	Post-midnight pull did not include T-1 day statements

Quick Verification

To compare production and canary group statements: sort same-day statements by created_at in the database. If a statement only exists in the production group → likely an anomalous bank push (Type 4). If the canary group has fewer statements than production → likely a canary group pull delay (Type 2).

---

BOCHK Dual-Path Bank Statement Monitoring

BOCHK Bank Statements come from two independent data sources that require correlated Monitoring.

Dual-Path Architecture

Data Source	Type	Latency	Information Included
Bank-Enterprise Service	Real-time API	Seconds	Transaction amount, currency, time; some include customer name
FTP Report S16	Scheduled pull	~30 minutes	Supplements customer name + remitter account number

Why Dual-Path Is Needed

Most Bank Statements from the Bank-Enterprise Service lack the customer name, which is a core condition for auto-Matching. The FTP Report S16 supplements this critical information.

FTP Report Operating Hours

Dimension	Hours
Bank-side service hours	Mon-Sat 8:00-20:00
Futu-side service hours	24/7 (pulls around the clock)

Report File Identification

Version	Filename Format	Notes
S16 (current)	AGSA/CNo+UC01.TXT, UC02.TXT, UC03.TXT...	One file per time period
S14 (legacy)	AGSA/CNo+AC1.TXT, AC2.TXT, AC3.TXT...	Deprecated; for rollback reference only

Bank Statement Correlation Rules

How to correlate S16 report statements with Bank-Enterprise real-time statements:

1. Take the "Transaction Reference Number" from S16
2. Remove the first 5 characters
3. Strip leading zeros from the remainder → yields seq_no_s16
4. Take seq_no from the Bank-Enterprise real-time statement and strip leading zeros
5. If seq_no_s16 == seq_no → same statement; can be correlated and updated

Update logic:

Bank-Enterprise Statement Status	S16 Update Content
Already has a name (en_name)	Update balance and bank account number
No name (en_name is empty)	Update name, balance, and bank account number

Monitoring Points

Monitor Item	Trigger Condition	Resolution
S16 report delay	No new report received for over 1 hour	Check FTP connection status; confirm whether within bank service hours
Statement correlation failure	S16 reference number cannot be matched to a Bank-Enterprise statement	Check reference number parsing logic; bank may have introduced an unknown format
Name not updated	Bank-Enterprise statement lacks a name for an extended period	Check whether S16 is being pulled normally; confirm whether the report contains the statement

---

eDDA / eDDI Scheduled Tasks

eDDA (authorization) and eDDI (Direct Debit instruction) are both used for deposit via Direct Debit — they are not withdrawal protocols.

Task	Bank	Responsibility	Notes
eDDA authorization polling	HSBC/Hang Seng	Query eDDA authorization status	Polls bank for confirmation after user initiates authorization
eDDI deposit instruction query	HSBC/Hang Seng	Query eDDI Direct Debit result	Queries whether the Direct Debit instruction was successfully credited
eDDI deposit result processing	HSBC/Hang Seng	Deposit crediting	Executes crediting after confirming Direct Debit success
eDDA status sync	HSBC/Hang Seng	Periodic authorization status sync	Detects whether authorization changes have occurred on the bank side

Difference Between eDDA/eDDI and BST Polling

eDDA/eDDI polling checks bank confirmation actions (whether authorization succeeded, whether Direct Debit completed), whereas BST Airstar polling checks transaction execution results (whether deposit/withdrawal completed). eDDA/eDDI polling frequency is lower; BST Airstar polling frequency is higher with explicit retry limits.

---

Concurrent Alert Triage

When the on-duty Operations staff receives multiple Alerts simultaneously, handle them in this priority order:

Priority	Criteria	Typical Alerts	Why This Priority
P0	Funds moved but status not updated	Freeze not released, withdrawal sync timeout (2h exhausted)	Fund status inconsistency; may lead to duplicate withdrawals or locked user funds
P1	Channel completely down	Socket disconnected (CMB/Minsheng), API connectivity failure (Airstar)	Affects all subsequent transactions; impact expands every minute
P2	Single transaction anomaly	Deposit/Withdrawal polling timeout, deposit refund REFUNDED	Affects a single customer; system fallback mechanisms are usually running
P3	Trend-based warning	Daily circuit breaker triggered, cumulative alarm, Approval backlog	No immediate impact, but will worsen if not addressed

Alerts That Can Be Safely Deferred

• BST Alerts on non-trading days → confirm it is a false trigger (BST does not process transactions on weekends)
• Canary group delay < 5 minutes (ICBC) → known behavior; canary group pull frequency is inherently lower
• bank-side initiation volume spike → first observe whether it is a normal business peak (e.g. start of month / payday)

---

Monitoring Alert Rules

Runbook Format Note

The following Alert rules are organized in Runbook format. Upon receiving an Alert, follow the flow: Severity → First responder → Resolution steps → Escalation criteria. See the Alert Escalation Matrix.

BST Alerts

Airstar BST:

#	Alert Item	Trigger Condition	Severity	Resolution
1	Authorization creation timeout	Mandate PROCESSING exceeds threshold	High	Check Airstar API connectivity
2	Authorization failure rate spike	FAIL count exceeds limit within time window	High	Investigate bank-side anomaly
3	Deposit creation failure	AsbBstCreateJob fails consecutively	High	Check request parameters and API
4	Deposit polling timeout	60 polls exhausted	High	Manually query bank side
5	Deposit refund	Received REFUNDED	High	Reversal + notify user
6	Withdrawal polling timeout	10 polls exhausted	High	Enters SyncAsbBstWithdraw
7	Withdrawal sync timeout	2h window exhausted	Critical	Manually confirm bank-side status
8	Freeze not released	Withdrawal failed but freeze not rolled back	Critical	Manually release freeze
9	Daily circuit breaker triggered	Cumulative count reaches stop threshold	Medium	Assess whether to resume
10	Cumulative alarm triggered	Cumulative count reaches alarm threshold	Medium	Operations to monitor
11	Duplicate transaction detected	Duplicate ID	High	Check deduplication logic
12	API connectivity failure	Response timeout / error rate increase	Critical	Contact Airstar technical support
13	Anomalous bank-side initiation volume	source=2 volume spike	Medium	Investigate business fluctuation

CMB / Minsheng BST:

Alert Item	Trigger Condition	Severity	Resolution
Socket connection dropped	Connection to bank exit_server interrupted	Critical	Auto-switch to backup server; Alert simultaneously
Timeout rate increase	-5 callback code ratio exceeds threshold	High	Check network and bank-side status
Rejection rate increase	-6 callback code ratio exceeds threshold	High	Contact bank to confirm rejection reason

Deposit Alerts

Alert Item	Trigger Condition	Severity	Resolution
Bank Statement backlog	Unmatched Flow count exceeds threshold	High	Check whether the Matching task is running normally
Abnormal deposit	AbnormalDepositJob detects anomaly	Medium	Operations to review abnormal deposits
Auto-crediting failure	Matching succeeded but crediting failed	High	Check SBA orchestration status

Withdrawal Alerts

Alert Item	Trigger Condition	Severity	Resolution
Approval backlog	Pending task count exceeds threshold	Medium	Increase Operations capacity
Large withdrawal	Single amount exceeds Monitoring threshold	Medium	Operations to review and confirm
Reversal operation	A withdrawal has been reversed	High	Confirm Reversal reason

---

Daily Operations Timeline

BST and other bank channels each have their own processing windows. Below is the typical trading day timeline:

Time	Event	Notes
07:00	CMB/Minsheng BST processing window opens
08:00	Airstar BST processing window opens
07:00	eDDI (HSBC/Hang Seng) processing window opens
09:00-10:00	BOCHK B2E first batch of Bank Statements arrives
11:00	Withdrawal batch cutoff	Withdrawals submitted after 11:00 go into the next-day batch
After 11:00	Withdrawal file export	Transfer files exported by method and uploaded to online banking
13:00-14:00	BOCHK B2E second batch of Bank Statements arrives
Afternoon	Check deposits	Staff visit branches to deposit checks
16:00	eDDI processing window closes
17:00-18:00	BOCHK B2E third batch of Bank Statements arrives
22:00	Airstar BST processing window closes
Next day 04:00	CMB/Minsheng BST processing window closes
Early next morning	Daily reconciliation tasks run

Non-Trading Days

BST does not process transactions on weekends and public holidays (per HKEX calendar). eDDI operates on business days only. FPS is theoretically 24/7 but Operations Approval is only available on business days.

---

Alert Escalation Matrix

When an Alert is triggered, follow this escalation flow:

Severity	Response SLA	First Responder	Escalation Criteria	Escalation Target
Critical	15 minutes	On-duty Operations	Cannot recover within 15 minutes	→ Technical Operations + corresponding bank service developer
High	30 minutes	On-duty Operations	Cannot identify root cause within 30 minutes	→ Deposit/Withdrawal product manager
Medium	2 hours	On-duty Operations	Cannot resolve within the business day	→ Operations Lead

Critical Alerts (15-minute response required)

• Socket connection dropped (CMB/Minsheng BST)
• API connectivity failure (Airstar)
• Withdrawal sync timeout (2h window exhausted)
• Freeze not released (withdrawal failed but funds still frozen)

High Alerts (30-minute response required)

• Bank Statement backlog (pending statement count exceeds threshold)
• Deposit/Withdrawal polling timeout
• BST timeout rate / rejection rate spike
• Auto-crediting failure
• Reversal operation
• Deposit refund (REFUNDED)

Medium Alerts (2-hour response required)

• Abnormal deposit detected
• Approval backlog
• Large withdrawal
• Cumulative alarm / daily circuit breaker triggered

If Requirements Change: Adjusting Monitoring Alert Configuration

Code locations:

• Deposit Monitoring tasks: deposit/doc/crontab.sh (all Cron definitions)
• Bank Statement backlog Monitoring: deposit/src/app/Console/Commands/Monitor/FlowMonitor.php
• BST Approval Monitoring: withdraw/src/app/Console/Commands/Monitor/AutoBsApprovalMonitor.php
• auto_settings Alerts: withdraw/src/app/Business/AutoSetting.php → autoOut() method (auto-Alerts when balance is insufficient)
• Alert delivery: via Feishu messages (sendFeishu() method); Alert recipient configuration is in auto_settings.alarm_staff_name

Common change scenarios:

• Modify Bank Statement backlog threshold → modify threshold constants in FlowMonitor.php
• Modify Alert recipients → modify alarm_staff_name via the CRM auto-withdrawal settings page, or use the AutoSetting::setAlarmStaff() API
• Add a new Monitoring dimension → create a new Monitor Command class and register it in crontab.sh
• Modify Alert interval → modify AUTO_SETTING_ALARM_INTERVAL (currently 1 hour, to avoid duplicate Alerts)

Documentation Maintenance Reminder

The following information is prone to becoming outdated. We recommend verifying it quarterly:

Check Item	Verification Method	Owner
Bank maintenance window times	Confirm with bank liaison whether changes have occurred	Bank liaison
eDDA supported bank list	Compare against actual HSBC/Hang Seng channel configurations	Deposit product manager
Withdrawal Channel fees	Compare against the bank's latest fee schedule	Withdrawal Operations Lead
Alert thresholds	Compare against actual values in FlowMonitor.php / auto_settings	Technical Operations
FPS ID and payee name	Confirm with bank whether updates have been made	Bank liaison

---

After Reading

I want to...	Go to
See bank-level details for BST tasks	Mainland Bank BST Overview
Understand deposit Matching logic	Matching & Auto-Crediting
See the operational guide after an Alert triggers	Manual Matching Guide
Look up error codes and status codes	Unified Error Code Center
Learn about CRM module operations	CRM Operations Guide
See the eDDA troubleshooting workflow	eDDA Troubleshooting Guide

Was this page helpful?