powerpc: add pte_free_defer() for pgtables sharing page

Add powerpc-specific pte_free_defer(), to free table page via call_rcu().
pte_free_defer() will be called inside khugepaged's retract_page_tables()
loop, where allocating extra memory cannot be relied upon. This precedes
the generic version to avoid build breakage from incompatible pgtable_t.

This is awkward because the struct page contains only one rcu_head, but
that page may be shared between PTE_FRAG_NR pagetables, each wanting to
use the rcu_head at the same time. But powerpc never reuses a fragment
once it has been freed: so mark the page Active in pte_free_defer(),
before calling pte_fragment_free() directly; and there call_rcu() to
pte_free_now() when last fragment is freed and the page is PageActive.

Link: https://lkml.kernel.org/r/[e-mail]
Suggested-by: Jason Gunthorpe <[e-mail]>
Signed-off-by: Hugh Dickins <[e-mail]>
Cc: Alexander Gordeev <[e-mail]>
Cc: Alistair Popple <[e-mail]>
Cc: Aneesh Kumar K.V <[e-mail]>
Cc: Anshuman Khandual <[e-mail]>
Cc: Axel Rasmussen <[e-mail]>
Cc: Christian Borntraeger <[e-mail]>
Cc: Christophe Leroy <[e-mail]>
Cc: Christoph Hellwig <[e-mail]>
Cc: Claudio Imbrenda <[e-mail]>
Cc: David Hildenbrand <[e-mail]>
Cc: "David S. Miller" <[e-mail]>
Cc: Gerald Schaefer <[e-mail]>
Cc: Heiko Carstens <[e-mail]>
Cc: Huang, Ying <[e-mail]>
Cc: Ira Weiny <[e-mail]>
Cc: Jann Horn <[e-mail]>
Cc: Kirill A. Shutemov <[e-mail]>
Cc: Lorenzo Stoakes <[e-mail]>
Cc: Matthew Wilcox (Oracle) <[e-mail]>
Cc: Mel Gorman <[e-mail]>
Cc: Miaohe Lin <[e-mail]>
Cc: Michael Ellerman <[e-mail]>
Cc: Mike Kravetz <[e-mail]>
Cc: Mike Rapoport (IBM) <[e-mail]>
Cc: Minchan Kim <[e-mail]>
Cc: Naoya Horiguchi <[e-mail]>
Cc: Pavel Tatashin <[e-mail]>
Cc: Peter Xu <[e-mail]>
Cc: Peter Zijlstra <[e-mail]>
Cc: Qi Zheng <[e-mail]>
Cc: Ralph Campbell <[e-mail]>
Cc: Russell King <[e-mail]>
Cc: SeongJae Park <[e-mail]>
Cc: Song Liu <[e-mail]>
Cc: Steven Price <[e-mail]>
Cc: Suren Baghdasaryan <[e-mail]>
Cc: Thomas Hellström <[e-mail]>
Cc: Vasily Gorbik <[e-mail]>
Cc: Vishal Moola (Oracle) <[e-mail]>
Cc: Vlastimil Babka <[e-mail]>
Cc: Will Deacon <[e-mail]>
Cc: Yang Shi <[e-mail]>
Cc: Yu Zhao <[e-mail]>
Cc: Zack Rusin <[e-mail]>
Cc: Zi Yan <[e-mail]>
Signed-off-by: Andrew Morton <[e-mail]>

arch/powerpc/include/asm/pgalloc.h

···
 	pte_fragment_free((unsigned long *)ptepage, 0);
 }
 
+/* arch use pte_free_defer() implementation in arch/powerpc/mm/pgtable-frag.c */
+#define pte_free_defer pte_free_defer
+void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable);
+
 /*
  * Functions that deal with pagetables that could be at any level of
  * the table need to be passed an "index_size" so they know how to

arch/powerpc/mm/pgtable-frag.c

···
 	return __alloc_for_ptecache(mm, kernel);
 }
 
+static void pte_free_now(struct rcu_head *head)
+{
+	struct page *page;
+
+	page = container_of(head, struct page, rcu_head);
+	pgtable_pte_page_dtor(page);
+	__free_page(page);
+}
+
 void pte_fragment_free(unsigned long *table, int kernel)
 {
 	struct page *page = virt_to_page(table);
···
 
 	BUG_ON(atomic_read(&page->pt_frag_refcount) <= 0);
 	if (atomic_dec_and_test(&page->pt_frag_refcount)) {
-		if (!kernel)
-			pgtable_pte_page_dtor(page);
-		__free_page(page);
+		if (kernel)
+			__free_page(page);
+		else if (TestClearPageActive(page))
+			call_rcu(&page->rcu_head, pte_free_now);
+		else
+			pte_free_now(&page->rcu_head);
 	}
 }
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable)
+{
+	struct page *page;
+
+	page = virt_to_page(pgtable);
+	SetPageActive(page);
+	pte_fragment_free((unsigned long *)pgtable, 0);
+}
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */