mm: Avoid calling page allocator from apply_to_page_range()
Lazy mmu mode applies to the current task and permits pte modifications
to be deferred and updated at a later time in a batch to improve
performance. apply_to_page_range() calls its callback in lazy mmu mode
and some of those callbacks call into the page allocator to either
allocate or free pages.
This is problematic with CONFIG_DEBUG_PAGEALLOC because
debug_pagealloc_[un]map_pages() calls the arch implementation of
__kernel_map_pages() which must modify the ptes for the linear map.
There are two possibilities at this point:
- If the arch implementation modifies the ptes directly without first
entering lazy mmu mode, the pte modifications may get deferred until
the existing lazy mmu mode is exited. This could result in taking
spurious faults for example.
- If the arch implementation enters a nested lazy mmu mode before
modification of the ptes (many arches use apply_to_page_range()),
then the linear map updates will definitely be applied upon leaving
the inner lazy mmu mode. But because lazy mmu mode does not support
nesting, the remainder of the outer user is no longer in lazy mmu
mode and the optimization opportunity is lost.
So let's just ensure that the page allocator is never called from within
lazy mmu mode. New "_nolazy" variants of apply_to_page_range() and
apply_to_existing_page_range() are introduced which don't enter lazy mmu
mode. Then users which need to call into the page allocator within their
callback are updated to use the _nolazy variants.
Signed-off-by:
Ryan Roberts <ryan.roberts@arm.com>
Loading
Please register or sign in to comment