The LSM-Tree structure is a multi-layer architecture consisting of the core memtable and SSTFile , which is very efficient for importing business data. The data addition, deletion , and modification operations are first updated to the memtable in an append manner , and then the SSTFile is merged through the Flush and Compaction operations .
Figure 4 LSM-Tree structure and merging process
As shown in Figure 4, this process involves merging and sorting SSTFiles at different levels , resulting in a large number of I/O operations and CPU calculations. Especially after a large number of update and delete operations, the compaction operation will cause storage space expansion and fluctuating consumption of I/O and CPU resources. This is because kazakhstan phone number data the compaction operation needs to apply for a new SSTFile first , and then delete the old SSTFile after completing the data merge , which means that the storage space requirement may exceed the space occupied by the original data, or even double.
In scenarios where storage space is limited, the compression rate of the LSM-Tree-based compression solution is difficult to determine, because the storage space is constantly changing dynamically, and the storage cost should include the maximum value of the space occupied.
Mainstream storage engines have their own advantages and disadvantages, and each has certain limitations in specific scenarios. Therefore, balancing the multiple factors affecting the business system, such as improving storage utilization and reducing the pressure of frequent transformation, is the key to solving the dilemma of business scenarios.
If the compression scheme decides whether to compress or not based solely on whether the data resides in memory or on the hard disk, then when processing extremely large amounts of data, although compression can save storage space, decompressing the data and loading it into memory may cause a decrease in business read and write performance, thereby increasing the risk of memory expansion.