building microservices design for resilience

When your mechanical project starts to get angry

This kind of thing is not uncommon, right? We often spend a lot of time selecting motors and adjusting mechanical structures, but it is easy to ignore those invisible code services. They are like the nervous system, which cannot send out instructions, and no matter how good the "muscles" are, they cannot move.

So the question is: How to make these microservices more reliable in an industrial environment? How to make them, like servo motors, able to quickly correct and keep working even if they encounter interference?

It’s better to think differently: design microservices as mechanical components

Think about how you choose a servo motor: you will look at torque, speed, accuracy, environmental adaptability, heat dissipation, and protection level. Because you know that there are vibrations, temperature differences, and dust in the workshop. What you choose is not just a motor, but a partner that can continue to work.

The same goes for microservices. It should not just be able to run, but should be able to maintain functionality despite network jitters, data surges, and local failures. This is what we often call "resilience". It's a bit like a redundant design in machinery - when a gear wears out, the backup structure immediately catches up and the system doesn't stop.

kpowerWhen helping customers with integration, we often encounter such needs. Someone asked: "We use very good motors and controllers, why is the system still unstable?" After checking, it is often that the calls between services are too "fragile": a timeout, and the entire chain collapses.

What to do? There are several directions you can consider:

• Timeout and retry, but be smart. Just like a servo receiving a pulse signal, if it does not receive it once, it will wait for the next cycle. The same should be true for microservice calls. But simply retrying can make the problem worse - if the downstream service is really down, retrying repeatedly will only make things worse. A back-off mechanism must be added, such as waiting for 2 seconds, waiting for 4 seconds... and setting an upper limit. Don't let one service slow down the entire system.

  

• Fail quickly and then take a detour. If a sensor on the assembly line fails, you will temporarily block it and use other data to make calculations. Microservices can also set "circuit breakers": after several consecutive failures, the calls are temporarily cut off and the preset results are directly returned, such as default values ​​or cached data. Give downstream services breathing time while ensuring that the main line process can still run.

• Allow services to survive independently. Good mechanical modules can be disassembled and tested individually. Microservices should also try to reduce external dependence as much as possible. For example, by caching commonly used data locally, even if the database is temporarily unavailable, the core functions can still be maintained for a while. This is like having a built-in backup power supply for the servo. When the main power supply is cut off, it can still maintain its position.

This sounds complicated, do you really want to do it yourself?

If you have someone on your team who understands software architecture, you can try it yourself. But challenges in the industrial field are often more specific: your microservices may need to communicate directly with the PLC, process motor feedback data in real time, and ensure response times at the millisecond level. This is not something that just a general framework can solve.

kpowerThe best approach is to find inspiration from the experience of hardware integration. We have noticed that mechanical systems that operate stably often have these characteristics:

• state observable: Just like motors have temperature sensors and vibration monitoring, microservices should also expose health status and load data so that people can understand them at a glance.
• Decoupling between modules: Conveyor belt control and robot control should be separated as much as possible. Even if one party has a problem, the other party can continue.
• Allow local degradation: When there is a problem with a non-core function, the system can temporarily reduce the accuracy or skip minor steps to ensure that the main line continues.

These principles are transferred to software design and become practical ideas for building resilient microservices. It is not about reinventing the wheel, but about adding some "buffers" and "safety valves" to the existing architecture.

So, where exactly do you start?

If you are planning a new project or revamping an existing production line, you can try this:

1. First draw a service dependency diagram, just like a mechanical assembly diagram, marking which services call which. Find the single point of vulnerability—that “core component” that too many services depend on.

2. Add protection to key services, such as the order dispatch service, which will stop the entire production line as soon as it stops. Set circuit breakers, caches, and downgrade strategies for it. Even if it is not used temporarily, reserve the interface first.

3. Unified monitoring and logging: All services output logs in the same format, with timestamps, service names, and key data. In this way, no matter where there is a problem, you can quickly locate it by following the clues just like checking for a motor fault.

4. Regularly conduct "fault drills" to artificially simulate network delays and database timeouts to see how the system will react. It is better to detect problems in advance than to receive an alarm call in the middle of the night after going online.

Say something honest

Microservices in an industrial environment do not need to pursue Internet-level high concurrency. But it is resilient enough - because if the production line is stopped for one hour, the loss may be hundreds of thousands.

Good resilient design will not make the system complicated and difficult to understand. On the contrary, it will make operation and maintenance easier: when a problem occurs, the system itself can alleviate part of the impact and buy you time to troubleshoot. Just like a well-tuned machine, if something goes wrong in a certain link, it will still keep running and a warning light will light up to remind you.

kpowerWhen serving customers, I found that the root cause of many stability problems is neither hardware nor software, but the connection between the two. Design microservices as part of a mechanical system, and treat it with the same pragmatic attitude as you treat steering gears and servo motors. You will find that many questions already have answers.

After all, reliability in the workshop never depends on luck.

Established in 2005, Kpower has been dedicated to a professional compact motion unit manufacturer, headquartered in Dongguan, Guangdong Province, China. Leveraging innovations in modular drive technology, Kpower integrates high-performance motors, precision reducers, and multi-protocol control systems to provide efficient and customized smart drive system solutions. Kpower has delivered professional drive system solutions to over 500 enterprise clients globally with products covering various fields such as Smart Home Systems, Automatic Electronics, Robotics, Precision Agriculture, Drones, and Industrial Automation.