Subscribing to a message broker in q/kdb through embedPy

Update: As of May 2020, there is now an official Solace interface for kdb+ which can be used, instead of embedPy, to interact with Solace’s PubSub+ broker.

You have undoubtedly heard of messaging platforms such as kafka, solace and rabbitMQ that allow applications to communicate with each other.  Messaging platforms have existed for several years now but with recent rise in streaming data and remote cloud applications, messaging platforms have become extremely helpful in both capturing this data and communicating with cloud applications.

When it comes to messaging, I have seen very few applications or libraries available in q/kdb+ that do so and the reason is probably because it’s so easy to open a handle to host/port and send a message. Additionally, a lot of existing kdb+ setups are local, meaning that they are hosted in local data centers and don’t require too much communication with other applications. With more and more applications being deployed to the cloud (AWS, Google Cloud, Azure, etc), it has become increasingly important for applications to communicate with each other. This coupled with multiple applications (especially IoT devices) streaming data in real-time to other applications means that we need to be able to push and receive messages from popular message brokers.

I noticed that there is already a kafka client available in q. I wanted to create something which uses an open protocol and can be used with any messaging platform. I picked MQTT which is a lightweight network protocol commonly used for publishing and subscribing to messages. Next, I wanted to start sending/receiving messages quickly. That’s where Solace comes in to the picture.

In this post, I will be covering:

1. how to use Solace’s PubSub+ Cloud offering,
2. how to run python code from q using embedPy, and
3. how to send and receive messages from Solace using MQTT protocol

Getting started with Solace PubSub+ Cloud

If enough sexual stimulation viagra from usa is present, then ED can be possible. Some of cialis professional for sale these problems are strictly physical ones, but depression was also reported. And while I tend to like Haley more than I like Pioli, I might have come to the same conclusion and fired tadalafil best buy Haley. Here are some daily habits of men and women causes lack of well-being, dysphoric mood, bone loss, cognition changes, physical fatigue, muscle loss insomnia, pain, urinary complaints and sexual dysfunction. super viagra

If you have worked in the financial industry, you have probably heard of Solace. It’s an enterprise grade messaging platform that is commonly used to transfer high volume of data such as market data but has also expanded out into other usecases such as powering Daimler’s connected car initiative and Singapore’s Smart City. What I really like about Solace is that is supports all major open protocols such as AMPQ, JMS, MQTT, websockets and REST which makes it really easy to migrate to or off of Solace.

Solace has different offerings with their main product being on-site appliances that handle your messages and also a containerized software broker. However, given the shift to cloud infrastructure, Solace allows users to use messaging as a service in any of the major cloud platforms! The service is called Solace PubSub+ Cloud and offers a free tier plan which was enough for me to test my code. Despite being someone who doesn’t have much experience with messaging platforms, I was able to spin up an instance within seconds.

You can follow these setups to set up a message broker:

1. Click on “+” to spin up a new instance.
2. Select the free plan, name your service and then pick your cloud platform. I selected AWS.
3. Click on ‘Start Service’.

That’s it! Your instance is now up and running!

Before we go, let’s get some important connection details.

Go to ‘Connect’ tab to get your connection details such as username, password, and host urls. Even though the page says REST API, these details are valid for MQTT too.

In my case, my details are: username: solace-cloud-client, password: hidden, host: <host>.messaging.solace.cloud

Now go to top right and click on ‘Manage Service’. This will open up a new window.

Click on ‘Connectivity’ tab and scroll down to ‘MQTT’. The port number listed below ‘Plain Text Enabled’ will be used to connect to Solace via MQTT protocol. In my case, port is 20678.

Now, let’s go back to our original instance page and click on ‘Try Me’. Click on the radio button below to Connect to the service.

Scroll down and in the ‘Consumer’ section, create a new topic. I created a topic called ‘q-mqtt’ for this example.

You can use the left side, where it says ‘Producer’, to send test messages to this topic.

Once you hit ‘Publish’, the message will appear on right side.

Great! We have confirmed that our instance works. Now, we move on to the code to connect to Solace.

q wrapper on top of paho-mqtt library

Few months ago, kx announced something really powerful: embedPy. embedPy allows you to run python code from q which means, all of sudden, you have the capability to leverage python’s countless libraries and run them from q. I did just that. Since Solace supports MQTT, I found a python library called paho-mqtt which is extremely easy to use. You can find great documentation for the library here but all we care about right now is to install the library which can be done using this command:

pip install paho-mqtt

I have created a module called mqtt.q which you can find on my github.

Here is how you can start a subscriber and receive messages:

$ q mqtt.q 
q)connect["<host>.messaging.solace.cloud";20678;"solace-cloud-client";"<password>"] 
q)Connected to broker 
subscribe["q-mqtt"] 
Subscribing to: q-mqtt q)start_session[] 
Message received: Second test message

You can also send messages:

$ q mqtt.q
q)connect["<host>.messaging.solace.cloud";20678;"solace-cloud-client";"<password>"]
q)Connected to broker
publish["q-mqtt";"Third test message"]
Sending message: Third test message

You can see on the browser that the message was sent correctly:

I have written another function called publish_q which is written in q. This function runs a simple CURL command to submit a POST message via REST API. Note that the host and port are different for REST API.

Here is how you can use the function:

publish_q["<host>.messaging.solace.cloud:20682";"Fourth test message";"q-mqtt";"solace-cloud-client";"<password>"]

As you can see, it’s really easy to send and receive messages thanks to paho-mqtt, embedPy and Solace! I wanted to show you how easy it is to take a python library and wrap it in q thanks to embedPy. Since this code uses MQTT, you can use it with any message platform that supports MQTT. However, I highly encourage you to check out Solace PubSub+ Cloud! You can learn more about Solace by taking this free Udemy course and/or by visiting their dev portal.

You can find the code on my github.