Ensuring System Stability: Exploring Degradation Drill Strategies and Practices for DB, Cache, Message Queue, and ES

What is Middleware Degradation Drill?

Middleware degradation drill refers to the process of intentionally causing middleware failures in production or simulation environments to test the system's emergency response capabilities and fault recovery mechanisms. This helps validate the robustness of existing systems and enhances the operations team's ability to handle and respond to faults.

Key Targets for Middleware Degradation

Database

When the database service is unavailable, the system can maintain basic functionality by switching to read-only mode, using a backup database, or utilizing temporary storage mechanisms.

Working Principle:

1. Read-Only Mode: Switch to read-only mode when the database is not writable to ensure that query operations remain unaffected.
2. Backup Database: Utilize master-slave replication or multi-master architecture to automatically switch to a backup database when the primary database fails.
3. Temporary Storage: Temporarily store critical data locally or in cache when the database is completely unavailable, synchronizing it back to the database once it is restored.

Cache

When the cache service is unavailable, the system can reduce its dependency on the cache through degradation strategies, directly accessing the database or limiting certain non-critical functions.

Working Principle:

1. Direct Database Access: Access the database directly when the cache fails, which may increase response time but ensures data availability.
2. Cache Strategy Adjustment: Adjust cache expiration times and strategies to reduce dependency on the cache, prioritizing core business cache.

Message Queue

When the message queue is unavailable, the system can maintain business continuity by caching messages, using backup queues, or directly processing some critical messages.

Working Principle:

1. Message Persistence: Temporarily store messages locally or in the database when the message queue is unavailable, processing them once the queue is restored.
2. Backup Queue: Configure a backup message queue to automatically switch to it when the primary queue is unavailable, ensuring reliable message delivery.
3. Critical Message Processing: Prioritize processing critical business messages during faults to ensure the normal operation of core business functions.

Search Engine

When the search engine service (e.g., ElasticSearch) is unavailable, the system can provide basic search functionality or delay search services through degradation strategies.

Working Principle:

1. Basic Search Functionality: Provide simplified basic search functions when the search engine is unavailable, such as replacing the ES service with database full-text search.
2. Delayed Search: Process search requests after the search service is restored, notifying users that search results will be delayed.