Baobab

Each of the four publishing institutions gets its own pipeline, and GSE and Bank of Ghana each need two, because their data types don't share a shape. The cost is six things to maintain instead of one clever adapter. The alternative, a single generic ingestion abstraction, would have meant bending an Excel bond-trade parser and a PDF inflation-report parser into the same interface for no reason except that both eventually become a row in Postgres. Building six honest pipelines was the less clever and more correct choice.

A single 8 PM UTC orchestrator scans the last 90 business days in Postgres for missing dates per data type before it dispatches anything, and caches the result in Redis with a two-hour TTL, deliberately shorter than the run interval so a stale cache can't hide a real gap for long. This exists because a nightly cron job that assumes it ran successfully last night is the single most common way market data goes silently stale. A Sunday 2 AM subprocess backfill covers gaps that fall outside the daily CSV window.

The scraping worker itself runs at Celery concurrency 1. Multiple concurrent Playwright browser instances contending for the same upstream page produced enough flakiness that trading it for slower, serial, reliable scraping was the right call.

_resolve_holding_price is the single function both the advisor UI and the external partner endpoint call to price a holding, deliberately shared so the two surfaces cannot disagree about what a stock is worth. When a live price is unavailable it returns null rather than falling back to a stale or estimated number: callers have to surface the gap to the user instead of quietly inventing a price. That is a slower failure mode for the UI to handle, and the correct one for a system representing client money.

Kofi's current guardrail against off-scope questions is a system prompt: an enumerated set of prohibitions and a scripted redirect to a human advisor. That is not a hard boundary, it is a strongly worded instruction, and the honest next step is a classifier with a deterministic refusal path in front of generation. The mitigating fact is that Kofi's knowledge base holds no client data, only platform and FAQ documents, so there is no cross-client retrieval risk even if the guardrail is talked around.

The GSE price API doesn't error when called from an AWS IP range, it hangs, so the Celery task fetching stock prices simply timed out and stock data went stale with no exception anywhere to alert on.

The first fix was the one that could ship within a day. The API answers browsers from any origin, so live price synchronisation moved to the client: requests came from real user browsers instead of a datacentre, and prices were flowing again. That only works while someone has the app open, though, so the next three days went into getting the server its data back by engineering around the block. A Cloudflare Worker acting as an authenticated proxy came first, on the assumption the block was on AWS's address range rather than on the requester; Cloudflare's edge addresses were refused too. A scheduled GitHub Actions workflow came next, fetching from a runner outside AWS and posting the prices to a sync endpoint. That did not hold either.

What closed it was a conversation: contacting the provider directly, explaining what was being built and how the feed was being used, and they whitelisted the server's address. Server-side fetching became the primary path again, with the browser relay kept as a fallback. A proxy or a relay is a more expensive way to get refused when a block is deliberate, and sending an email was the cheaper, more durable fix, reached for last rather than first. Building the fix also surfaced a second problem, a nightly snapshot job that had been failing silently behind the same block and nobody had noticed. One outage during this period produced 19 failure emails in a morning before a 30-minute cooldown was added to the alerting.

Two endpoints called the valuation service with no check that the requesting user actually owned the portfolio being valued, so any authenticated user could read any portfolio by incrementing an integer ID. Ron found this himself, after it had already shipped, by re-reading his own code rather than through a report or a scan. The fix returns 404 rather than 403 on an unauthorised request, specifically so a probing attacker can't use the response code to enumerate which portfolio IDs exist.

Behind that sat a second, deeper bug: portfolios.user_id had no foreign key constraint and a default of 1, so any insert that omitted a user_id silently attributed the portfolio to user 1 instead of failing. The fix added the foreign key and a NOT NULL constraint, dropped the default, and removed four service-function parameter defaults of = 1 that had been quietly relying on it. It shipped behind a read-only preflight script and an atomic, idempotent migration, because a migration that touches ownership on financial records in production does not get a second, casual attempt.

Four publishing institutions, six independent pipelines, each with its own parser and its own way of avoiding duplicate writes when a job reruns: file-level tracking tables for repeating publications, a UNIQUE(url) constraint for news articles, upserts everywhere else.

GSE listed equities come from the Kwayisi API live during market hours, with Playwright and Scrapy used for historical backfill, polled every 5 minutes. Bank of Ghana forex, 19 currency pairs, and Bank of Ghana T-bills and bonds both arrive nightly at 8 PM, the forex as a CSV export and the rates scraped. GFIM secondary bond trades are parsed nightly from an Excel export. Ghana Statistical Service CPI and GDP, and the Bank of Ghana bulletin's cocoa, gold, and Brent crude figures, are parsed from PDFs on the 5th of each month. Ghana business news arrives hourly over RSS.

There is no holiday calendar in the scheduling, only weekend exclusion, so a public holiday during market hours is treated as a normal trading day until proven otherwise by the gap detector.