mm: wake kcompactd before kswapd's short sleep

When kswapd goes to sleep it checks if the node is balanced and at first
it sleeps only for HZ/10 time, then rechecks if the node is still
balanced and nobody has woken it during the initial sleep.  Only then it
goes fully sleep until an allocation slowpath wakes it up again.

For higher-order allocations, waking up kcompactd is done only before
the full sleep.  This turns out to be an issue in case another
high-order allocation fails during the initial sleep.  It will wake
kswapd up, however kswapd considers the zone balanced from the order-0
perspective, and will just quickly try to sleep again.  So if there's a
longer stream of high-order allocations hitting the slowpath and waking
up kswapd, it might never actually wake up kcompactd, which may be
considered a regression from kswapd-based compaction.  In the worst
case, it might be that a single allocation that cannot direct
reclaim/compact itself is waking kswapd in the retry loop and preventing
kcompactd from being woken up and unblocking it.

This patch makes sure kcompactd is woken up in such situations by simply
moving the wakeup before the short initial sleep.  More efficient
solution would be to wake kcompactd immediately instead of kswapd if the
node is already order-0 balanced, but in that case we should also move
reset_isolation_suitable() call to kcompactd so it's not adding to the
allocator's latency.  Since it's late in the 4.6 cycle, let's go with
the simpler change for now.

Fixes: accf62422b3a ("mm, kswapd: replace kswapd compaction with waking up kcompactd")
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: David Rientjes <rientjes@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

diff --git a/mm/vmscan.c b/mm/vmscan.c
index c638b28310fc39b306abca39d8cec1ff339ce434..142cb61f4822454bf3819a4c10c4e8b479a70ec8 100644 (file)
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3318,6 +3318,20 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int order,
        /* Try to sleep for a short interval */
        if (prepare_kswapd_sleep(pgdat, order, remaining,
                                                balanced_classzone_idx)) {
+               /*
+                * Compaction records what page blocks it recently failed to
+                * isolate pages from and skips them in the future scanning.
+                * When kswapd is going to sleep, it is reasonable to assume
+                * that pages and compaction may succeed so reset the cache.
+                */
+               reset_isolation_suitable(pgdat);
+
+               /*
+                * We have freed the memory, now we should compact it to make
+                * allocation of the requested order possible.
+                */
+               wakeup_kcompactd(pgdat, order, classzone_idx);
+
                remaining = schedule_timeout(HZ/10);
                finish_wait(&pgdat->kswapd_wait, &wait);
                prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
@@ -3341,20 +3355,6 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int order,
                 */
                set_pgdat_percpu_threshold(pgdat, calculate_normal_threshold);
 
-               /*
-                * Compaction records what page blocks it recently failed to
-                * isolate pages from and skips them in the future scanning.
-                * When kswapd is going to sleep, it is reasonable to assume
-                * that pages and compaction may succeed so reset the cache.
-                */
-               reset_isolation_suitable(pgdat);
-
-               /*
-                * We have freed the memory, now we should compact it to make
-                * allocation of the requested order possible.
-                */
-               wakeup_kcompactd(pgdat, order, classzone_idx);
-
                if (!kthread_should_stop())
                        schedule();