Kdb database limits and errors

Table of Limits and Errors
Limitations - Look at each in turn, see what it is and workarounds.
Limits Summary

Table of Limit Errors

You may encounter these error messages when working in kdb, here's what they mean:

limit	Attempt to create list longer than allowable maximum
params	Too many parameters (8 max)
branch	A branch (if;do;while;$[.;.;.]) more than 255 byte codes away
rank	Invalid number of arguments
constants	Too many constants (max 96)
globals	Too many global variables (32 max)
locals	Too many local variables (24 max)
mlim	More than 999 nested columns in splayed table
glim	`g# limit, kdb is currently limited to 99 concurrent `g#'s
stop	A client query took longer than the time allowed.
abort	The maximum memory that kdb was allowed, was exceeded.

There are also a number of architectural limitations namely:

2 GB IPC object transfer size limit
Number of file handles permitted by OS limits all operations with on disk tables.
You can only have 56 enums

Limitations

'limit error

The limit error is either caused by trying to create too large a list (2 billion items usually) or trying to transfer an object to/from a kdb process that is over 2GB in size. e.g.: q) / list is too large q)0W#2 'limit q)-22!til 280100200 / size in bytes 2240801614 q)til 280100200 / can generate ok 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20.. q)h:hopen 5000 q)h "til 280100200" / cant retrieve from remote process 'limit

The 2GB limit is because internally integers are used for storage and addressing and 2^31 is 2,147,483,648. You will sometimes see this error when retrieving a large of results with many columns. The only workaround is to transfer the table in multiple requests.

'constants 'locals 'global error

You have tried to declare too many variables at a given scope. Kdb stores functions as bytecode, this code is only built to handle reasonable limits on the number of variables. If we try to create too many variables at a given scope we get an error, e.g. too many function local variables:

q) / long function definition		
q)"f:{",(raze ("a",/:string[til[24]]),\:":1; "),"}"
"f:{a0:1; a1:1; a2:1; a3:1; a4:1; a5:1; a6:1; a7:1; a8:1; a9:1; a10:1; a11:1;..
q) / works ok for 23
q)value "f:{",(raze ("a",/:string[til[23]]),\:":1; "),"}"
q) / fails with 24
q)value "f:{",(raze ("a",/:string[til[24]]),\:":1; "),"}"
'locals

The general solution is to encapsulate data into objects (lists, tables, dictionaries) rather than having many separate variables

'params error

Params error is caused when you try to create a function with more than 8 parameters. To us this sounds like a reasonable tradeoff to make functions readable. If you find your hitting this limit, try to group the arguments into cohedive objects (lists/dictionaries).

		
q)f:{[a;b;c;d;e;f;g;h] g*100}
q)f:{[a;b;c;d;e;f;g;h;i] g*100}
'params

q) / rewrite to group common variables into a dictionary
q)f:{[pVol; pSize; pLimit; pHeight; portSize; portValue; portTime;h;i] g*100}
'params
q)p:`vol`size`limit`height!(1 ; 2; 200; 3)
q)port:`height`size`value!(`large;`expensive; 08:00t)
q)f:{ [p; port; h; i] h*100}

'branch error

This error is caused when a branch like an if condition, $ if-else condition, while, do etc. has too many statements in it. If you receive this error you should modularise your code into more functions, both to fix the error and to make your code more readable. (See our guide on q coding standards)

	
q)"f:{ if[1b; ",(raze 122#enlist "1;"),"]}"
"f:{ if[1b; 1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1..
q)value "f:{ if[1b; ",(raze 122#enlist "1;"),"]}"
q)value "f:{ if[1b; ",(raze 132#enlist "1;"),"]}"
'branch

'rank error

Rank errors happen when you attempt to call a function and supply too many parameters. There is also a particularly tricky variation of this where you try to use default parameters to a qsql query. The parser gets confued and does not recognide them as parameters. As shown below:

/ simple case of too many arguments
q)f:{x+1}
q)f[8;9]
'rank
		
q)/ tricky case where parser is getting confused by default parameters.		
q)t:([] a:1 2 3; b:4 5 6)
q)f:{select from t where a=x,b=y}
q)f[1;4]
'rank

q) / force it to work by specifying them
q)f:{[p;q] select from t where a=p,b=q}
q)f[1;4]
a b
---
1 4

'stop error

Using a -T setting, A maximum time allowed for a remote query can be configured.
In this example we call a function that exceeds that timeout and throws a stop error:

Server q -p 5000 -T 2

>q -p 5000 -T 2
KDB+ 3.1 Copyright (C) 1993-2013 Kx Systems

q)/ slow function to simulate doing work
q)f:{do[x*1500; r:x xexp 1.1]; r}
q)\t f 1000
685
q)\t f 6000
3992
q)/ Can break the 2 second limit at server console

Client










q)h:hopen 5000
q)h "f 1000"
1995.262
q)h "f 6000"
'stop
q)/ long client queries receive a stop error

'abort error

A query used more memory than the maximum memory permitted, as set using the -w kdb command line option.
An error is thrown and the kdb process is stopped.

>q -p 5000 -T 2 -w 128
KDB+ 3.1 2013.09.05 Copyright (C) 1993-2013 Kx Systems

q){show "variable needs ",string[8*x%1024]," MB"; a::til x*1024;} each `int$3 xexp 7+til 9
"variable needs 17.0859 MB"
"variable needs 51.2578 MB"
"variable needs 153.773 MB"
-w abort

Limits Summary

Many errors are caused by poorly written code, we recommend following the q language style guidelines which should eliminate most these problems. Other errors are reported due to OS limits, these are usually a hint that you are using kdb for a problem that it is not designed to solve. You can alter the OS to handle these cases but you may increasingly struggle to get the performance you want. That leaves a very few unfortunately tricky cases where kdb has sharp edges where it's basically user beware