(kdb+) Software Development Essentials

The below article is intended as bullet points to be given as a talk. Enough people asked for a summary that we posted it here, hopefully you find it useful or get a chance to attend the talk itself.

My names Ryan Hamilton, I've been building systems, mainly kdb+ in banks for 13 years. Given the multiple kdb teams and 100s of kdb developers I've interacted with, I want to record the essentials that I think most (kdb) developers would benefit from:

Having seen how many people arrive at kdb+, having had little training in software development or technology.
This talk contains some of the info I think that would most help them.

Users
Team
Continuous Deployment

Users

Before you even join the company or team or as shortly as possible after, you need to make sure there are
People who care and can generate profit from good market data access.
There's no point nor rewarding feeling for building a system to find it unused, unloved or wasted. A car is the sum of it's parts.

Typically you will work with:

5-15 Knowledgeable quants
2-10 App Teams with technical developers
Risk / Compliance / ?
Time-Wasters

Team

Want good developers, not 10x coders
Ownership of product
Trust within team to say when there’s a problem
Clearly agreed goals
Metrics to measure ourselves and keep us honest
The ability to say “no”. this needs ingrained within the team
Management buy-in helps to prevent distractions
Hiring kdb+ skills is difficult

Continuous Deployment

Code is read 10x more than it’s written
The earlier a problem is found
the cheaper it is to fix
Code Reviews are primarily to share knowledge and reach consensus and shared ownership
CI - Continuous Integration at minimum, essential to
- Attract/train/keep good team members
- Keep moving at speed as system scales (modularity)
Invest as much as reasonably possible optimizing the appropriate part of your pipeline. Every developer MUST be able to run new code in a repeatable environment.

Feedback

The common thread linking Users/Teams and Continuous deployment is that it's about getting and using feedback as early as possible.

The way I think of it is, that we are a team in a rocket looking for a habitable planet but we don't know where it is. The best approach then is to head in the best direction you currently know, continuously look out the window, perform radar sweeps and re-adjust your direction. You need trust within the team to let each person get on with their area and the safety feeling so every member feels OK to mention if they see an asteroid.

Notice in this scenario:

Management is useless UNLESS they have intelligence on what's outside your radar :)
You want to have fun as a team, as you will be stuck in close proximity for years

Rules of the game

Where possible the team itself decides all rules and goals
Real Developers, Take development all the way
Relationship building -> requirements -> coding -> deployment -> demos -> <- user feedback
No developer may code until they can describe one user.
Whatever you measure, will improve. We will measure something. Team decides what is measured and what is important
All code is reviewed. Part of dev responsibility to bring team mate on the journey together.
If code isn’t used or functionality doesn’t meet success criteria we will remove it.

Secret Weapons

These are things that require significant effort or cost but end up returning a huge amount of value:

Full Integration Tests - We were only able to support complex database configurations, keep them working and keep moving as every database type had a full suite of tests. This can be massively expensive to retrofit but if you consider yourself as going to work here forever, it's the only sane way to work.
Keep It Simple - Say no to users/management. Be careful where you decide to add complexity, 3x more careful if it's exposed complexity.
Keep Smelly Working Code - Say no to ourselves. We kept some awful code and hacks. In one case we added a huge hack ontop of an existing mess,
(A huge bash script to allow deploys over multiple servers) BUT it rarely changed after that one big hack. It just worked and was barely changed since. barely anyone even looked at it.
Non-Dev work can pay hugely - In a world full of people that love "coding". If you can commit yourself to being a developer and doing parts others may not enjoy, those parts have larger returns.
- e.g. We delivered training in-house. We didn't really enjoy it but it built a relationship with our users, trained them to do things better and meant they asked us questions in future.
- e.g. The best code is NO-code. When we asked users to pay a cost to have something developed OR insisted they give us a firm commitment of when and how they would use it, quite often they said no, don't build it actually. I won't need it. We saved a LOT of effort.
Super-Gateway - Users loved this.
S3 Storage - The secret is, a lot of old data isn't used. S3 didn't let us use old data, it removed the need for us to ask users which data we could delete. Especially in a bank where it's hard to directly bill your users, no customer will ever want to delete data, why would they?
Good artists copy; great artists steal - A lot of the time our best "ideas" were what we found the quants already doing but that we could bring deep technical skills to and make 10x better and more reliable

Costly Mistakes

Complex Packaging - Twice I helped build a maven based kdb packager. It worked great to allow integration into existing standard bank build systems. Over time it cost a lot of effort to keep "compile" time down. (testing time is fine). If I could do it again, I'd try to remove all steps that rearrange q files, separate packages and just have one bug package.
Wasted Sales Effort - I pitched our system to other teams many many times. Once in particular I just failed to realize the incentives of those involved was not aligned at all. Even if I'd been selling the best system ever, their motives didn't allow to using it.
Hiring / Training - I failed to implement a continuous hiring pipeline. I can only apologies to my successor.