// WRITING · 2026.04.18
Market data pipelines with fastapi + vector dbs
streaming tick data into embeddings, and what breaks when you try to query it in real time.
Financial data is a firehose with a deadline. Ticks arrive faster than you can think, and a query that takes 400ms too long is a query that answered the wrong market.
The shape of the pipeline
- Ingest — a FastAPI service pulls tick and order-book data, validated on the way in with Pydantic so malformed payloads never reach storage.
- Embed — windows of market state are turned into vectors capturing regime, volatility, and momentum.
- Serve — a vector database answers “what past states look like now?” in single-digit milliseconds.
What breaks
Everything, eventually — but predictably:
- Backpressure. When embedding lags ingestion, queues grow without bound. You need to drop or batch, and you need to decide which on purpose.
- Staleness. A real-time query against a five-second-old index is a lie you told yourself. Freshness has to be a measured number, not a hope.
- Schema drift. Pydantic models are the contract. Version them, or a silent field rename corrupts a week of embeddings.
Real-time is not a speed problem. It is a discipline problem — every stage has to fail loudly instead of quietly falling behind.