Architecture and High-Level Design
Solution Context, Components, Integrations and Quality Attributes
ICICI Direct • Interactive Brokers • Revolut
Figure 1. Target architecture and principal trust boundaries.
| Design intent Provide a secure personal portfolio hub that accommodates three incompatible source-integration patterns while maintaining cached availability and simple operations. |
|---|
1. Architecture overview
The system uses a browser-facing Oracle Ubuntu platform as the consolidation layer. Nginx terminates HTTPS, serves static pages and proxies API traffic to a Flask service managed by PM2. Broker data is normalised into JSON snapshots. Market prices are obtained server-side, cached, and combined with holdings data for dashboards.
2. Logical layers
| Layer | Components | Responsibilities |
|---|---|---|
| Presentation | $hail's Dashboard, ICICI page, ibkr.html, Revolut.html, upload pages | Navigation, status, cards, charts, tables and controlled user actions. |
| Edge / web | Nginx, TLS, basic authentication, route-specific agent exception | Static delivery, reverse proxy, authentication boundary and HTTPS. |
| Application | Flask breeze-api, Revolut processor/upload blueprint, IBKR agent-control blueprint | Broker orchestration, validation, secure actions, job state and snapshot publication. |
| Local IBKR integration | TWS/IB Gateway, export_ibkr.py, ibkr_windows_agent.py | Authenticated local socket access, portfolio export and secure outbound upload. |
| Data | Snapshot JSON, external price JSON, mappings, logs, backups | State, auditability, last-known-good values and recovery. |
| Scheduling | Ubuntu cron | Weekday price refreshes at configured UTC times. |
| External services | ICICI Breeze, Twelve Data, Alpha Vantage | Portfolio data and supported market prices/FX. |
3. Integration patterns
ICICI Direct: synchronous API
The browser calls Oracle /api routes; Flask authenticates to Breeze using server-side environment values and returns portfolio JSON.
IBKR: asynchronous command and agent
The browser queues a job. A Windows agent polls outbound over HTTPS, claims the job, calls TWS locally and uploads the resulting snapshot.
Revolut: secure file ingestion
The browser uploads CSV files to Flask. Private upload directories and a full-history processor produce a normalised snapshot.
Pricing: scheduled cache
Cron invokes Python scripts. Providers are called server-side, and the browser reads only cached JSON.
4. Deployment topology
| Node | Runtime / assets | Connectivity |
|---|---|---|
| Oracle Ubuntu VM | Nginx, Flask, PM2, Python virtual environment, cron, /var/www/html, private app directories | Inbound HTTPS; outbound HTTPS to broker/price APIs. |
| Windows laptop(s) | TWS, Python, export script, polling agent, private config | Local TWS socket; outbound HTTPS to Oracle agent endpoints. |
| Browser device | Modern browser | HTTPS to Oracle; may be different from TWS laptop. |
| External providers | Breeze, Twelve Data, Alpha Vantage | HTTPS API calls from Oracle only. |
5. Security architecture
- HTTPS protects browser, Oracle and agent traffic.
- Website authentication protects interactive pages and standard API routes.
- Admin PIN authorises state-changing dashboard actions.
- A separate high-entropy agent token authenticates /api/ibkr-agent/* requests.
- Provider keys and Breeze credentials live in a protected .env outside the web root.
- Uploaded Revolut CSV files remain outside /var/www/html with restrictive permissions.
- Nginx disables website Basic Authentication only for the narrow agent endpoint prefix; Flask token validation remains mandatory.
- Backups are created before snapshot replacement.
6. Data flow summary
| Flow | Source → Destination | Payload | Control |
|---|---|---|---|
| ICICI read | Browser → Nginx → Flask → Breeze | JSON request/response | Site auth; Breeze server credentials |
| IBKR request | Browser → Flask state file | Job ID/status | Site auth + admin PIN |
| IBKR agent | Windows agent → Flask | Claim/progress/upload | Private agent token |
| Revolut upload | Browser → Flask private upload directory | Two CSV files | Site auth + admin PIN + validation |
| Price refresh | Cron → provider APIs → cache files | Quote/FX JSON | Server-side API keys + throttling |
| Dashboard display | Browser → static snapshots/caches | JSON | Site authentication |
7. Availability and resilience
- Dashboards consume last-known-good snapshots/caches, allowing reads during broker/provider outages.
- Provider failures are recorded without deleting prior valid prices.
- A failed Revolut rebuild does not publish partial output.
- IBKR snapshot and external-price scripts create backups before replacement.
- Separated holdings and pricing workflows reduce coupling.
8. Capacity and constraints
Twelve Data Basic is used within its published credit limits for supported US equities, FX and crypto. Alpha Vantage free access is treated as scarce and reserved primarily for mapped UK/EU symbols. TWS API access requires a running TWS or IB Gateway process and network connectivity from the local agent to that process.
9. High-level operational schedule
| Job | Schedule | Outcome |
|---|---|---|
| refresh_revolut_prices.py | 22:30 UTC, Monday–Friday | Update Revolut supported prices and FX cache. |
| refresh_ibkr_prices.py | 22:40 UTC, Monday–Friday | Update IBKR price cache and enrich snapshot values. |
| IBKR holdings refresh | On demand | TWS agent creates and uploads a current snapshot. |
| Revolut holdings refresh | On demand | Full-history CSV upload rebuilds snapshot. |
10. Key design decisions
| Decision | Rationale | Trade-off |
|---|---|---|
| JSON snapshots instead of a database | Simple, inspectable and suitable for personal scale. | Limited querying/concurrency. |
| Polling Windows agent | Works behind home NAT without opening inbound laptop ports. | Agent and TWS must be running for holdings refresh. |
| Server-side price caching | Protects keys, controls quotas and improves dashboard speed. | Prices are as fresh as the scheduled/manual refresh. |
| Full Revolut rebuild | Reduces duplicate/delta reconciliation errors. | Processing grows with statement length. |
| Provider fallback | Maintains usability when coverage or quota fails. | Data freshness may vary by symbol. |
References
- Interactive Brokers API home: https://www.interactivebrokers.com/campus/ibkr-api-page/ibkr-api-home/
- Interactive Brokers TWS API introduction: https://interactivebrokers.github.io/tws-api/introduction.html
- Interactive Brokers TWS API connectivity: https://interactivebrokers.github.io/tws-api/connection.html
- Twelve Data pricing: https://twelvedata.com/pricing
- Alpha Vantage support and limits: https://www.alphavantage.co/support/
- Alpha Vantage API documentation: https://www.alphavantage.co/documentation/
- Ubuntu CronHowto: https://help.ubuntu.com/community/CronHowto