Queue Processors

Queues - Introduction

In school we learned about queues. A queue is a line of people waiting for something.

In computer science a queue is a list of items that are waiting to be processed.

2 types of queues were discussed in school.

  • LIFO - Last In First Out
  • FIFO - First In First Out

In the real world, most of the queues are FIFO.

There are a few benefits of working with queues.

  • You can have one process add items to a queue. You can have another processes to read and process the items on the queue.
  • The second process can be running on a different machine.
  • You can setup multiple processes to add to the queue. You can setup multiple processes to read from the queue.
  • You an move the processes that read from the queue to different machines - thus allowing you to scale out.

When you start designing with a queue system, you need to think about the following.

  • Should you use your own queue system, or use a third party queue system.

One thing about a third party system is that it will have the same complexities as a database server, or a web server.

I took a look at a few queue systems:

  • Solace - https://solace.com/ - This is where I did my most work. I studied the pages, I watched a course from Udemy. I setup a Solace Server on my home computer using Docker. I used AI to help setup a demo. An integration from my checkbook program to Solace, then from Solace to the GL program. Then integrations to send a message from GL to Solace, then from Solace to my checkbook program.

    Solace Notes

  • Apache Kafka - https://kafka.apache.org/ - I started studying this. I watched a 6 hours course from Pluralsight. I setup a Kafka server on my home computer. I installed Kafka on my home computer - using Docker. What I learned is this is a very complex system. I did not get very far. More studying is needed.

  • AI mentioned lots of other systems.

What I learned was you send a message to a queue. You can think of a message as having at least 2 parts.

  • A queue name.
  • A message

When you abstract the message to a few parts, you start to realize that you have a few operations

  • connect to the queue server.
  • send a message to the queue.
  • read a message from the queue.

If the server fails, or is not suitable for the job, you can change the server. and really the only thing you need to change is the connection processing.

So I think the idea of setting up a queue server is a good idea.