An experimental pub/sub client and server project.
willdot.net
1# Message Broker
2
3Message broker is my attempt at building client / server pub/sub system written in Go using plain `net.Conn`.
4
5I decided to try to build one to further my understanding of how such a common tool is used in software engineering.
6
7I realised that I took for granted how complicated other pub / sub message brokers were such as NATs, RabbitMQ and Kafka. By creating my own, I hope to dive into and understand more of how message brokers work.
8
9## Concept
10
11The server accepts TCP connections. When a connection is first established, the client will send an "action" message which determines what type of client it is; subscribe or publish.
12
13### Subscribing
14Once a connection has declared itself as a subscribing connection, it will need to then send the list of topics it wishes to initally subscribe to. After that the connection will enter a loop where it can then send a new action; subscribe to new topic(s) or unsubscribe from topic(s).
15
16### Publishing
17Once a subscription has declared itself as a publisher, it will enter a loop where it can then send a message for a topic. Once a message has been received, the server will then send it to all connections that are subscribed to that topic.
18
19### Sending data via a connection
20
21When sending a message representing an action (subscribe, publish etc) then a uint16 binary message is sent.
22
23When sending any other data, the length of the data is to be sent first using a binary uint32 and then the actual data sent afterwards.
24
25## Running the server
26
27There is a server that can be run using `docker-compose up message-server`. This will start a server running listening on port 3000.
28
29## Example clients
30There is an example application that implements the subscriber and publishers in the `example` directory.
31
32Run `go build .` to build the file.
33
34When running the example there are the following flags:
35
36`publish` : settings this to true will allow messages to be sent every 500ms as well as consuming
37`consume-from` : this allows you to specify what message to start from. If you don't set this or set it to be -1, you will start consuming from the next sent message.