arm64/mm: Implement clear_ptes() to optimize exit, munmap, dontneed
With the core-mm changes in place to batch-clear ptes during zap_pte_range(), we can take advantage of this in arm64 to greatly reduce the number of tlbis we have to issue, and recover the lost performance in exit, munmap and madvise(DONTNEED) incured when adding support for transparent contiguous ptes. If we are clearing a whole contpte range, we can elide first unfolding that range and save the tlbis. We just clear the whole range. The following microbenchmark results demonstate the effect of this change on madvise(DONTNEED) performance for large pte-mapped folios. madvise(dontneed) is called for each page of a 1G populated mapping and the total time is measured. 100 iterations per run, 8 runs performed on both Apple M2 (VM) and Ampere Altra (bare metal). Tests performed for case where 1G memory is comprised of pte-mapped order-9 folios. Negative is faster, positive is slower, compared to baseline upon which the series is based: | dontneed | Apple M2 VM | Ampere Altra | | order-9 |-------------------|-------------------| | (pte-map) | mean | stdev | mean | stdev | |---------------|---------|---------|---------|---------| | baseline | 0.0% | 7.9% | 0.0% | 0.0% | | before-change | -1.3% | 7.0% | 13.0% | 0.0% | | after-change | -9.9% | 0.9% | 14.1% | 0.0% | The memory is initially all contpte-mapped and has to be unfolded (which requires tlbi for the whole block) when the first page is touched (since the test is madvise-ing 1 page at a time). Ampere Altra has high cost for tlbi; this is why cost increases there. The following microbenchmark results demonstate the recovery (and overall improvement) of munmap performance for large pte-mapped folios. munmap is called for a 1G populated mapping and the function runtime is measured. 100 iterations per run, 8 runs performed on both Apple M2 (VM) and Ampere Altra (bare metal). Tests performed for case where 1G memory is comprised of pte-mapped order-9 folios. Negative is faster, positive is slower, compared to baseline upon which the series is based: | munmap | Apple M2 VM | Ampere Altra | | order-9 |-------------------|-------------------| | (pte-map) | mean | stdev | mean | stdev | |---------------|---------|---------|---------|---------| | baseline | 0.0% | 6.4% | 0.0% | 0.1% | | before-change | 43.3% | 1.9% | 375.2% | 0.0% | | after-change | -6.0% | 1.4% | -0.6% | 0.2% | Tested-by:John Hubbard <jhubbard@nvidia.com> Signed-off-by:
Ryan Roberts <ryan.roberts@arm.com>
Loading
Please register or sign in to comment