Impact of Indirection Unit on Write Amplification in SSDs
Developers typically think of SSDs as a black box which will store any IO that is coming its way into a non-volatile memory (such as NAND). Even though the part about storing the IO to the non-volatile memory is true, the way it achieves it depends on various implementation details and parameters. These parameters can have different side effects on performance, endurance, etc.
One such parameter we will explore in this article is the Indirection Unit and how it impacts the device’s endurance based on Write Amplification.
First, let us see what is Write Amplification and then discuss about Indirection Unit and its impact.
Write Amplification:
Write Amplification (WA) happens in SSD when the actual amount of written physical data is more than the amount of logical data that is written by the host.
WAF is the mathematical representation of this phenomenon, describing the ratio of physical writes to logical writes. Let us say the host wrote 4KB, and the SSD has to write 16KB to accommodate that operation; then the WAF will be 4. WAF has a direct impact on the lifetime of the SSDs as more WAF leads to more writes to the underlying media.
WAF on the device can be calculated as follows:
# io_len: size of the IO from the host
# io_extra: extra IO incurred by the SSD for a given io_len
WAF = (io_len + io_extra) / io_len
End-End WAF is a culmination of different WAFs0:
WAFTotal = WAF_App * WAF_Device
# Splitting WAF_Device further:
WAFTotal = WAF_App * WAF_SSD * WAF_IU
WAF contribution happens because of different factors. We will look
into the impact of the Indirection unit(WAF_IU) on WAF as part of this article.
Indirection Unit:
SSDs maintain a Logical to Physical mapping(L2P) table to map a logical block to an underlying physical NAND block in its RAM(device RAM and not host RAM1). Logically contiguous blocks do not translate to physical contiguous blocks. This is similar to how virtual memory works in an Operating system.
If the mapping granularity is 4KB, then a 4KB logical block corresponds to a 4KB physical block. Whenever a block is written to the device, a new mapping is created in RAM, and it is used again to find the corresponding physical block when a read happens.
 |
|---|
| SSD Logical to Physical mapping |
Having a mapping table will incur some RAM costs for the device. Assuming a 1:1 L2P with 4KB Logical block size will require at least 256MB of RAM for an SSD of size 256GB.
4KB LBA size in 256 GB SSD = 64M entries (256GB / 4KB)
Each entry could be 32 bits.
64M * 4 bytes = 256 MB
The amount of RAM is directly proportional to the size of the SSD. Extrapolating the same math for 64TB SSD will result in having a whooping 64GB of RAM in the device to hold the mapping table.
High-capacity SSDs have already started to appear in the market3, and device vendors need to use new techniques to keep the RAM under control for mapping table to reduce cost, etc.
One technique that device vendors actively use to reduce RAM
footprint is to increase the mapping ratio or the Indirection Unit. Instead of having 1:1
mapping, device could have n:1 L2P mapping, where n > 1. RAM footprint
is inversely proportional to n i.e., multiple logical blocks could
have 1 physical mapping as follows:
 |
|---|
| Mapping table with 16k Indirection Unit (4:1) |
Even though increasing the logical block size above 4KB is an option, backward compatibility with the host will not make the transition easy4. Solidigm’s high-capacity drive has an Indirection Unit of 16k for an SSD with 61.44TB capacity3.
The following section discusses the impact of increasing Indirection Unit(IU) on WAF.
Indirection Unit impact on WAF:
As increasing the IU is inevitable for high-capacity SSDs, evaluating its effects on WAF is essential. As multiple LBAs map to a single physical block, IO writes that are smaller than IU will incur a Read-Modify-Write(RMW) that leads to WAF. RMW has to read the old data, merge the new data, and write it back to the media.
 |
|---|
| Read-Modify-Write operation2 |
Optimal write should align and be a multiple of IUs. RMW negatively impacts the performance and lifetime of the SSD due to extra writes incurred.
Quantifying IU WAF:
WAF_IU can be easily quantified by monitoring the IO write patterns coming
from the host.
On a 16k IU drive, io spanning from offset 12k to 32k (io_len of 20k)
will incur an extra_io of 12k due to RMW as explained before. The
resulting WAF_IU is 1.6. ASCII art explaining the workload:
0 4 8 12 16 20 24 28 32
|--------|--------|--------|--------|--------|--------|--------|--------|.. LBA space
<--------------------------------------------> io_len
<-#######################-> extra_io
<-----------------------------------------------------------------------> total_io
WAF_IU can be calculated as follows(code gist here):
# io_offset: IO offset from the host
# io_len: size of the IO from the host
# IU: Indirection Unit
total_io = (round_up((io_offset + io_len), IU) - round_down((io_offset), IU))
WAF = total_io / io_len
One interesting observation that the above formula indicates that the extra IO due to IU is caused due to the unalignment in the either ends of an IO.
 |
|---|
| Worst case WAF_IU for different IO length |
The plot above shows the worst case WAF_IU for different IO length on a 16k IU device. If the IO size from the host much greater than the IU, then the impact on WAF due to IU drastically reduces. The biggest impact on WAF due to IU happens when the IO size is smaller than the IU of the device.
Takeaways:
- Indirection units will increase to reduce cost for high-capacity SSDs.
- The indirection unit of the device has an impact on total WAF.
- Impact of Indirection Unit on WAF is highest when IO size is smaller than the Indirection unit and lowest when the IO size is higher than the Indirection unit.
- The host can avoid the WAF due to IU by aligning and sending IO writes that are a multiple of IU to the device.
References:
0 Real Life workloads allow more efficient data granularity and enable very large SSD capacities
1 There are HMB SSDs which store the L2P table in Host RAM but they are not yet widely used
2 Achieving Optimal Performance & Endurance on Coarse Indirection Unit SSDs