When your servo motor project meets microservices: why you need a message queue and our practice

Picture this: you are debugging a robotic arm, with each joint controlled by an independent servo motor. These motors need to work together in real time, but the instructions come from different microservices - one calculates the trajectory, one handles sensor feedback, and another manages the safety logic. Suddenly, a certain service slows down for a few milliseconds due to high load, and the entire operation begins to freeze or even become out of sync. At that moment, did you think that it would be great if these services could make conversations more "smooth"?

This is actually not just a problem encountered by mechanical control. Any system split into microservices faces similar challenges: How to communicate between services reliably and efficiently? How to ensure that data is not lost, the order is not messed up, and that it can withstand sudden traffic?

We hit this wall a few years ago when we were upgrading an automated sorting line. Traditional methods, such as direct HTTP calls, become fragile as services increase. When one service goes down, other services on the link are also affected; as soon as peak requests come, the system starts queuing up, and real-time performance is out of the question. We need a mechanism that is more like a "conveyor belt" - put the data on it and let the places where it is needed access it at their own pace without blocking each other.

So, we started looking and eventually introduced Kafka.

What is Kafka? Think of it as a high-speed, smart assembly line

If you use a familiar scene as an analogy: Kafka is like the never-ending conveyor belt in a factory. Each microservice can be regarded as a workstation. Station A puts the semi-finished products (messages) on the conveyor belt, and stations B and C take them away for processing at any time as needed. The conveyor belt itself continues to run. Even if a certain station is temporarily repaired, the semi-finished products will not disappear, but will just wait quietly on the line until a station takes over.

This solves several headaches:

• decoupling: Services no longer "call" directly, but are handed over via a conveyor belt. The sender drops the message and walks away without knowing who will handle it or when.
• buffer: When the upstream suddenly generates a large amount of data (such as a sudden high-frequency signal from a sensor), the conveyor belt can temporarily carry it, and the downstream can digest it according to its own capabilities to avoid being overwhelmed.
• reliability: Messages are persisted on the conveyor belt, and even if some services are restarted, the data will not be lost.
• sequentiality: On a conveyor belt, the order of items is fixed. This is critical for sequence-sensitive data such as motor control instructions.

How do we use it? A non-linear attempt

At the beginning, some people in the team were worried: "Is this too complicated? We have to maintain a new set of middleware." But in practice, we found that it was more like building blocks than building a skyscraper.

We didn’t start out with a complete overhaul. Instead, we chose a link with the most obvious pain point to test the waters - the status monitoring of the robotic arm. It turns out that status data is pushed from one service to another and is often lost due to network jitters, causing the monitoring panel to be "fragmented" from time to time. We simply established a Kafka Topic (which can be understood as a dedicated conveyor belt), let the status publishing service write data here, and the monitoring service subscribes to it. Almost immediately, the fragmentation problem disappeared. Because even if the monitoring service is temporarily offline, it can still read all unprocessed status updates from the breakpoint after restarting.

This small success gave us confidence. Then, we gradually migrated the order instruction flow and alarm event flow. Gradually, the data flow of the entire system becomes clearly visible, like looking at a real-time logistics map. It is clear at a glance which lines are congested and which lines are idle.

Some real fragments and feelings

Of course there was some exploration in the process. For example, we once struggled with "How many conveyor belts (Topics) should be divided into?" Later, we found that just like in a workshop, all parts will not be mixed on one line, and it is more reasonable to manage them separately by data domain and urgency. Motor control instructions require low latency and require a separate high-speed line; log and audit data are large in volume but have low real-time requirements, so use another high-capacity line. Each goes his own way without interfering with each other.

Some people also ask: "What if the conveyor belt itself breaks?" Kafka's distributed design makes it inherently redundant. Data will be backed up on multiple nodes. If a single node fails, other nodes can take over immediately and the service will be almost imperceptible. It's like putting a spare engine in the transmission.

Looking back now, the introduction of Kafka did not make the system more "complex", but instead made the originally invisible chaos orderly. The tense direct coupling relationship between microservices has been relaxed. Each service can focus more on its own logic, and the flexibility of the entire architecture has been significantly enhanced. When the production line needs to be expanded and a new inspection station is added, we only need to subscribe the new service to the corresponding Topic, which is as simple as adding a new station next to the conveyor belt, without disturbing other parts at all.

Let’s talk about choices

There are many message queue tools on the market, why choose Kafka? For us, we value its balanced performance in high throughput, persistence, and sequence guarantees. It may not be the one with the flashiest features, but it’s solid enough for industrial scenarios that require stable processing of data streams. Of course, no tool is a silver bullet, and its learning curve and operation and maintenance costs require investment. But if your microservices also start to "act uncoordinated" due to communication problems, it may be worth considering installing a set of such "intelligent conveyor belts" on the system.

After all, good technology should be like reliable mechanical transmission—quiet, precise, and making complex collaboration effortless.

Established in 2005,kpowerhas been dedicated to a professional compact motion unit manufacturer, headquartered in Dongguan, Guangdong Province, China. Leveraging innovations in modular drive technology,kpowerintegrates high-performance motors, precision reducers, and multi-protocol control systems to provide efficient and customized smart drive system solutions.kpowerhas delivered professional drive system solutions to over 500 enterprise clients globally with products covering various fields such as Smart Home Systems, Automatic Electronics, Robotics, Precision Agriculture, Drones, and Industrial Automation.