🌐 中文 English
This article mainly introduces the basics of Linux zero-copy and file warming.
Zero-Copy
When sending a file over the network, it generally goes through the following steps:
- The process switches to kernel mode.
- Read the file content from disk into the kernel buffer.
- Copy the content from the kernel buffer to the user buffer.
- Copy the content from the user buffer to the network buffer.
- Send the content from the network buffer over the network.
- The process switches back to user mode and releases resources.
Overall, this process is very time-consuming and resource-intensive because it involves multiple copies, i.e., a large amount of cached I/O operations. In some scenarios, the number of copies is indeed excessive, leading to the concept of zero-copy.
Zero-Copy via sendFile
The steps are:
- The process switches to kernel mode.
- Read the file content from disk into the kernel buffer.
- Copy the content from the kernel buffer to the network buffer.
- Send the content from the network buffer over the network.
- The process switches back to user mode and releases resources. As you can see, in this process, there is no copying to the user buffer. Instead, the content from the kernel buffer is directly copied to the network buffer and then sent over the network. This reduces two round-trip copies, improving efficiency.
Zero-Copy via sendFile + gather
The steps are:
- The process switches to kernel mode.
- Read the file content from disk into the kernel buffer.
- Send the content from the kernel buffer directly over the network.
- The process switches back to user mode and releases resources. As you can see, in this process, data no longer needs to be copied to the socket buffer (network buffer). Instead, the content from the kernel buffer is sent directly over the network. This reduces three copies, further improving efficiency.
Suitable Scenarios for Zero-Copy
Zero-copy is suitable for scenarios where there is no need to parse, modify, or back up the content of the disk file, as the data is never copied to user space and cannot be parsed.
File Warming
File warming refers to preloading files into memory to avoid reading from disk every time a file is accessed. This can significantly improve file read performance, especially for frequently accessed files or in high-concurrency scenarios, effectively reducing I/O latency.
mmap + File Warming
In RocketMQ’s file warming process, mmap maps the kernel read buffer corresponding to the disk file and the user’s cache to the same address. Operations performed by user space on this cached data directly affect the kernel’s read buffer, essentially sharing the same physical memory. This reduces copying from kernel space to user space, allowing user space to directly manipulate the data.
However, physical memory resources are not actually allocated at this stage. Only when the process accesses the memory and finds no data will a page fault interrupt occur, triggering resource allocation. This page fault interrupt is a system call involving context switching, which is time-consuming and can cause performance fluctuations for RocketMQ’s message writing operations.
Therefore, RocketMQ adopts file warming, which involves traversing each page of the currently mapped file and writing a zero byte to trigger the page fault interrupt system call. This pre-allocates memory and locks part or all of the process’s address space in physical memory, preventing it from being swapped to swap space.
Summary
Zero-copy and file warming are important techniques for improving file read performance, reducing I/O latency, and enhancing file read efficiency.
