Kdb Parallel Processing - peach and .Q.fc

Parallel processing can be used to speed up calculations

q)/ slow function that allows us to simulate doing work q)f:{do[x*1500; r:x xexp 1.1]; r}; q)a:10*(100#1),100#4 q)f each a 12.58925 12.58925 12.58925 12.58925 12.58925 12.58925 12.58925 12.58925 12.58.. q)\t f each a 2177 q)\s 2 q)/ The peach adverb is used for parallel execution of a function over data q)/ as long as we started kdb with -s 2 q)/ Results from the peach function are copied q)\t f peach a 1996 q)(f each a)~(f peach a) 1b q)/ so as we can see the parallel processing has provided a significant speedup

The function must be costly enough to justify the messaging overhead

q)\t do[5000; {x+1} each a] 115 q)\t do[5000; {x+1} peach a] 308

When the function is fast the overhead of passing the data to the threads outweighs the parallelisation benefits

Take account of how work is being distributed

q)\t {f x; show x} peach 10 10 10000 10000 10 10 10000 10000 7976 q)\t {f x; show x} peach 10 10000 10 10000 10 10 10000 10000 10021 q)\t .Q.fc[{r:f each x; show x; r}] 10 10 10000 10000 10 10 10000 10000 10055 q)\t .Q.fc[{r:f each x; show x; r}] 10 10000 10 10000 10 10000 10 10000 7992

• PEACH - When 2 slaves, assigns items at 0,2,4,6.. to one thread. 1,3,5,7 to another.
• .Q.FC - Cuts vector into s equal sized continuous pieces, where s is number slaves passes whole vector piece at once to each thread

Performance for semi-realistic data sizes

q)a:10*(100#1),100#4 q)a 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10.. q)b:10*200#1 4 q)b 10 40 10 40 10 40 10 40 10 40 10 40 10 40 10 40 10 40 10 40 10 40 10 40 10 40.. q)c:10*1+200?4 q)c 10 10 30 20 30 20 30 40 30 20 10 40 20 30 30 20 10 20 30 10 20 20 30 40 10 20.. q)sum each (a;b;c) 5000 5000 4630 q) q)r1:{system "ts f each ",x} each "abc" q)r2:{system "ts f peach ",x} each "abc" q)r3:{system "ts .Q.fc[{f each x}] ",x} each "abc" q)r:flip `method`data`time`space!flip (raze 3#/:`each`peach`Qfc),'(9#`a`b`c),'r1,r2,r3 q) q)update time%min time, space%min space from r method data time space ----------------------------- each a 1.186237 2.848485 each b 1.193337 2.848485 each c 1.097215 2.848485 peach a 1.091207 3.612121 peach b 1.250137 2.848485 peach c 1 2.848485 Qfc a 1.364282 1 Qfc b 1.10213 1.763636 Qfc c 1 1.763636

Conclusion

1. each was consistantly slow
2. peach was faster except on data b.
3. .Q.fc was slightly faster still except on a.

Careful sending unbalanced workloads to peach / .Q.fc. would be nice if they were smarter to batch into groups and queue them to each processor

.Q.fc appears to use less memory but this test isn't really representative in real world usage I've seen .Q.fc explode memory wise particularly when the vectors its passing to threads are just over a power of 2 as the memory manager then allocates basically double the space needed. peach is less likely to explode like that.

The normal rule of using vector operations still applies. (prefer .Q.fc)

q)g:{do[10000;x:sqrt x xexp 1.9999]; x} q)\t r:g each 2000#999 7546 q)\t r1:g peach 2000#999 6578 q)r~r1 1b q)\t r2: g 2000#999 2875 q)r~r2 1b q)\t r3:.Q.fc[g] 2000#999 1687 q)r~r3 1b

Parallelising on different functions/data is possible

q)/ typical peach format is f peach d q)/ where f is a function that is applied the same to every piece of data q)f:{do[x*1000; r:x xexp 1.1]; r}; q)d:100?100 q)\t f each d 3796 q)\t f peach d 3359 q)/ what if we have multiple functions we want to apply to a single piece of data? q)h1:{do[19000*x; r:sqrt x xexp 2.0001]; r} q)h2:{do[19000*x; r:sqrt x xexp 1.9999]; r} q)/ multiple instruction - single data q)data:100; q)\t r1:(h1 data; h2 data) 2360 q)\t r2:{x data} peach (h1;h2) 1984 q)r1~r2 1b q)/ so applying diff functions in parallel to same data is possible and faster q)/ multiple instruction - diff data q)d1:100; q)d2:200; q)\t r1:(h1 d1; h2 d2) 3546 q)\t r2:{(x[0]) x 1} peach ((h1;d1);(h2;d2)) 2874 q)r1~r2 1b q)/ so you can basically parallelise any functions and data q)/ it just isn't pretty