My wife manages the family’s short-term apartment rental business in Budapest and maintains a detailed Google Sheet for all bookings. This weekend, she asked me if I could help calculate a monthly utilization and a profit share report for this year (2024). Calculating these metrics can be cumbersome with traditional data analytics software even for moderately sized datasets. This is especially true when bookings span multiple months and pricing varies based on factors like weekends. The request made me contemplate the beauty and expressiveness of q programming language and inspired me to write this article.
There are several IDEs available for kdb+, including the soon-to-be-released Visual Studio KX plugin, KX Developer/Analyst, Studio for kdb+, etc. Often, you find yourself working in a terminal and would simply like to execute some function on a remote kdb+ process. Typically, you start a kdb+ process, create a connection handler (h) and send the command synchronously.
Fiona is a developer with limited kdb+ experience, often finds herself facing a challenge. She needs to execute a function (stored procedure) on a remote kdb+ server maintained by her IT people. Armed with VS Code’s kdb+ plugin (or other tools like KX Developer or kdb+ Studio) she successfully connects to the kdb+ server and inputs the function name, sets the parameters, and hits the run command. However, instead of the expected outcome, she’s greeted by a puzzling 'type error. Just a 'type error, nothing else. Fiona, like many kdb+ users, grapples with debugging and spends significant time tinkering with parameter values until success.
Implementing concurrent execution of a function across several kdb+ processes simultaneously is an interesting topic worth investigating. While this requirement might not be a common use case, it becomes essential in scenarios such as evaluating storage performance scaling under simultaneous disk read operations within a storage performance tool - see nano storage performance tool that triggered this small research.
The first function that you learn on your way to the functional programming world is map. It applies a unary/monadic function to each element of a list and returns the list of results.
Comma-separated text files (CSV) are the most fundamental format for data processing. All programming languages and software that support working with relational data, also provide some level of CSV handling. You can persist and process data without installing a database management system. Often you don’t need a full-blown DBMS with all its features, like handling transactions and concurrent/remote access, indexing, etc… The lightweight CSV format allows for easy processing and sharing of the captured information.
Python is a popular programming language that is easy to learn, efficient and enjoys the support of a large and active community. It is a general-purpose language with libraries specialized for various areas, including web development, scripting, data science, and DevOps.
In this article, I take a simple, real-life problem and analyze different solutions in Python, BigQuery and Q. The problem is magical in the sense that unraveling it leads us to discover nice areas of these two programming languages including vector operations, Einstein summation, adverbs and functional form of select statements. Each solution has lessons to learn that deepens our IT knowledge, especially if we consider performance aspects as well. I believe that people understand programming concepts better via examples.
Python and Q are two world-class, dynamic programming languages with many similarities and some interesting differences. Python is a general language with specialized libraries for different fields like web and mobile development or data analysis. Q is primarily used in the financial industry for data analysis and developing trading algorithms. On the Wall Street people often use these tools side-by-side, sporadically with R and Matlab.The symbioses of Python and Q developers demanded new tools to bridge the gap. Q users can import embedPy to call Python functions which opens the world to a rich set of machine learning algorithms. Python fans can experience the performance gain of Q by loading PyQ and bringing the Python and Q interpreters into the same process so that code written in either of the languages operates on the same data. Libraries embedPy and PyQ reduce integration costs by offering two languages in one interpreter. Still, to become a successful developer, you need to know what expression corresponds to your codeline in the other language. This article helps you with this.