History of Kdb+/q Database by kx.com
Contents
- Timeseries of Kdb - Where kdb came from
- Arthur Whitney - One man and his many languages
- Future of kdb+/q
Timeseries of Kdb+ q
- 1964 APL A Programming Language invented by Ken Iverson
- 1988 A+ - language developed by Arthur Whitney at MS
- 1993 K - A terse variant of APL developed by Arthur Whitney
- 1998 Kdb+ - In-memory column based database
- 2003 q - Wrapper around k providing readability
Kdb/q's origin is an obscure academic language and a story of progress towards increasing user friendliness
through the efforts of one Arthur Whitney working at various financial firms. In the 1960's Iverson developed a radical new array-orientated programming language called APL, unfortunately the notation was incredibly terse, for example:
and a special keyboard was required to enter code. It was ahead of its time in many ways but its lack of user friendliness has seen its use decline.
Morgan Stanley then needed APL for its new sunos systems so Arthur and a team wrote A+.
In 1993 Arthur decided to leave MS and form a new company kx together with Janet Lustgarten, they signed a contract with
UBS to provide them with K, a variant of APL that used only characters available on standard keyboards to increase user friendliness. K looked like this:
life:{3=a-x*4=a:2{+x-2{|1_0+’:x,1#x}/x}/x}
In 1998 the exclusive contract with UBS expires and kx launches kdb, a column-oriented database for timeseries data.
Banks were starting to increasingly store and analyse tick data so it became increasingly popular in the financial community, however its lack of user friendliness still hindered it. In 2003
Arthur developed a new language q that sits ontop of k and provides english words as functions:
life:{3=a-x*4=a:2 sum/2(1 0 -1 rotate'\:)/x}
OK its still not perfect but there are two english words to make it less scary and I assure you to a kdb programmer this is readable and
that's where we are at today (till Arthur comes out with his next language.)
Arthur Whitney
Arthur has been the lead developer of A+, k and q over a career stretching from Morgan Stanley to UBS to jointly forming the company KX with Janet Lustgarten in 1993. Since then every four or five years Arthur has brought forth an innovative new language. This is sometimes why you'll hear q referred to as k4 as there were in fact 3 distinct releases of k. Details on Arthur are somewhat scarce as he seems to prefer being locked away writing code. However the following articles are available:
- A Conversation with Arthur Whitney - 2009 interview with ACM
- Janet Lustgarten Interview, CEO at KX Systems
Future of kdb+/q
What innovation will kx.com come out with next? Where is development headed?
Unfortunately kx is a rather closed company and doesn't reveal much about its plans, recent major changes include:
- 2.5/2.6 Memory Management Changes, garbase collector added and unused memory returned
- 2.7 Built in file compression
- 3.0 Move to 64 bit indexing to support partitioned tables
With Arthur off working on kparc.com (a k based OS to compete against android) we will probably see more incremental improvements rather than innovative leaps. We would really like to see some major chances taken by kx to increase the use of kdb outside finance as its a beautiful database and language that could solve many problems that nosql is only now getting around to. However we will have to wait and see.
This tutorial is a small subsection of one module from our full kdb+ training courses. If you are new to kdb+, you will find our Introductory kdb+ training course the best way to master kdb+.
