Extend Kdb using Compiled C DLL Example

It is possible to import C functions directly for use within the kdb process by dynamically loading modules. Below we will look at an example of compiling, importing and executing C functions on windows for kdb 3.0+.

We will be adding two functions myavg, mysum which provide moving average and moving sum functions respectively. Yes, kdb does already mavg but read till the end and I think you will find the performance comparison interesting. The Interfacing Kdb with C and Extending with C page on the code KX website documents the C API extensively, we will give details for the core parts used in our example but you will want to read those pages thoroughly at a later stage.

Compiling a C DLL on Windows
Loading a module
Executing our C Code

Download Example C Code

Example of kdb C DLL Compilation

Compiling a C DLL on Windows

The steps to compile and load a C DLL are as follows:

Download kdb C example code. It contains the following files:
1. The latest version of header files provided by kx. k.h and and q.lib for windows 64.
2. mymoving.c - C code that defines our actual functions
3. mymoving.def - Declares the functions we will be providing
Compile the code using comprun.bat this requires MS Visual Studio 2010 installed
1. It uses "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" amd64
  to set the required environment variables for our compilation.
2. Runs the command line compiler: cl /LD /DKXVER=3 mymoving.c mymoving.def q.lib
3. Launch q running the script load-functions.q so that our DLL functions are loaded and defined.
Now within q we can call our mytop, myavg functions.

comprun.bat

CALL "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" amd64 
cl /LD /DKXVER=3 mymoving.c mymoving.def q.lib
q load-functions.q

mymoving.c

// Find the moving average for a given window size and array of nums
K myavg(K window, K nums) {
	
	// initialize variables
	long wsize = window->j;
	long i=0;
	K res = ktn(KF, nums->n);
	double total;
	long count;

	F* resf = kF(res);
	F* numsf = kF(nums);

	// check type is float
	if( (nums->t != KF) || (window->t != -KJ)) {
		printf("invalid params");
		return krr("wrong type params");
	}
	
	// find moving average of first window (special case)
	total = numsf[0];
	resf[0] = total;
	count = 2;
	for(i=1; i < wsize; i++) {
		total += numsf[i];
		resf[i] = total / count++;
	}
	
	// move window along by adding next number and removing earliest from average.
	for(i=wsize; i < nums->n; i++) {
		total = total - numsf[i-wsize]+numsf[i];
		resf[i] = total / wsize;
	}

	return res;
}

Loading a module

load-functions.q

/ from the DLL mymoving load the functions and assign to named variables
/ The 2: is used for loading
/ 2 at the end specifies the number of arguments
mysum:`mymoving 2:(`mysum;2)
myavg:`mymoving 2:(`myavg;2)

show "myavg[3; 6 7 5 8 2.]";
myavg[3; 6 7 5 8 2.]

In this q script we import functions from our DLL and assign them to variable names:
mymoving is the name of our DLL which is in the current directory, you could place this in c:/q/w64 or specify a full path.
2: is the kdb function for loading native code.
The right hand side e.g. (`mysum;2) specifies a function that exists and the number of arguments it accepts.

Executing our C Code

q)myavg[3; 10 20 30 40 41 42 43 44.]
10 15 20 30 37 41 42 43f
q)mavg[3; 10 20 30 40 41 42 43 44.]
10 15 20 30 37 41 42 43f

q)mysum[3; 10 20 30 40 41 42 43 44.]
10 30 60 90 111 123 126 129f
q)msum[3; 10 20 30 40 41 42 43 44.]
10 30 60 90 111 123 126 129f

q)l:100200?100.
q)myavg[10;l]~mavg[10;l]
1b
q)myavg[60;l]~mavg[60;l]
1b

q)\t do[100; myavg[60;l]]
90
q)\t do[100; mavg[60;l]]
4618

As you can see we get the same results using either our own function or kdb's builtin mavg. However our function is 46x quicker (users on linux report a much smaller 3x speed difference). There are a number of reasons why this is (nulls and partitioned data) but that's a topic for another post.

If your having problems compiling or want advice either contact us or ask in our forums. We also provide kdb consulting services and have particular expertise in writing kdb+tick subscribers and feedhandlers.