I work in a market data team that captures and stores market data, calculates statistics and then provides both the raw data and stats to our clients according to their needs. We handle both real-time and historical data. In an ideal world, you will never have to worry about your historical data but many times, …
Category Archives: q/kdb
How data is saved in a historical database
Real-time databases are easy to understand as everything is happening in memory. Historical databases are a different story. Recently, I have been getting much more involved with hdbs at work and just the way they work is incredible and fascinating to me. If you are serious about kdb+, you should make sure to know how …
Continue reading “How data is saved in a historical database”
Overview of kdb+ architecture
So far, I have covered quite a few intermediate topics without covering the basics. In this post, I would like to take a step back and talk about general architecture of kdb+. A simple kdb+ setup includes a feed handler (fh), a ticker plant (tp), a real-time database (rdb), and a historical database (hdb). These processes work …
Enumeration in kdb+
I was planning on writing a blog post on how to save data to disk from a q process but then realized that I need to first cover the enumeration process because any data you wish to write to disk must be enumerated if it has a sym column. If you try to save a table …
Subscribing to a table in ticker plant
I am currently working on writing a stats process that will calculate some stats such as high, low, avg, min, max price in real-time. I have a whole stack running (fh/tp/rdb/hdb) where the fh regularly sends updates to my ticker plant. You can see my previous post about how to code that. As I started …
Coding a sample feed handler
Feed handlers sound scary. They are often written in some language other than q (or at least the ones I was exposed to initially) such as java which is a pain to look at once you get used to a compact language like q. I always avoided feed handlers until the day I read a …
Tables, keyed tables and dictionaries
When I first started learning q, I had a difficult time understanding the differences between tables, keyed tables and dictionaries. The differences seemed very subtle at the time. Just recently, I was explaining a colleague (java developer with some exposure to q/kdb) how you can check meta of a table, look up the keys and …
Using Attributes in kdb+
One of the biggest advantages of kdb+ is fast data retrieval. As a client, you simply run a qsql command (or some API) and you are given thousands and thousands of rows of data within seconds. How does kdb+ do that? There are many factors that contribute to high retrieval speed and one of them …