Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux

AutoFDO: Enable machine function split optimization for AutoFDO

Enable the machine function split optimization for AutoFDO in Clang.

Machine function split (MFS) is a pass in the Clang compiler that
splits a function into hot and cold parts. The linker groups all
cold blocks across functions together. This decreases hot code
fragmentation and improves iCache and iTLB utilization.

MFS requires a profile so this is enabled only for the AutoFDO builds.

Co-developed-by: Han Shen <shenhan@google.com>
Signed-off-by: Han Shen <shenhan@google.com>
Signed-off-by: Rong Xu <xur@google.com>
Suggested-by: Sriraman Tallam <tmsriram@google.com>
Suggested-by: Krzysztof Pszeniczny <kpszeniczny@google.com>
Tested-by: Yonghong Song <yonghong.song@linux.dev>
Tested-by: Yabin Cui <yabinc@google.com>
Tested-by: Nathan Chancellor <nathan@kernel.org>
Reviewed-by: Kees Cook <kees@kernel.org>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

authored by

Rong Xu and committed by
Masahiro Yamada
2fd65f7a 0847420f

+8 -1
+6 -1
include/asm-generic/vmlinux.lds.h
··· 556 556 __cpuidle_text_end = .; \ 557 557 __noinstr_text_end = .; 558 558 559 + #define TEXT_SPLIT \ 560 + __split_text_start = .; \ 561 + *(.text.split .text.split.[0-9a-zA-Z_]*) \ 562 + __split_text_end = .; 563 + 559 564 #define TEXT_UNLIKELY \ 560 565 __unlikely_text_start = .; \ 561 566 *(.text.unlikely .text.unlikely.*) \ ··· 587 582 ALIGN_FUNCTION(); \ 588 583 *(.text.asan.* .text.tsan.*) \ 589 584 *(.text.unknown .text.unknown.*) \ 585 + TEXT_SPLIT \ 590 586 TEXT_UNLIKELY \ 591 587 . = ALIGN(PAGE_SIZE); \ 592 588 TEXT_HOT \ 593 589 *(TEXT_MAIN .text.fixup) \ 594 590 NOINSTR_TEXT \ 595 591 *(.ref.text) 596 - 597 592 598 593 /* sched.text is aling to function alignment to secure we have same 599 594 * address even at second ld pass when generating System.map */
+2
scripts/Makefile.autofdo
··· 10 10 11 11 ifdef CLANG_AUTOFDO_PROFILE 12 12 CFLAGS_AUTOFDO_CLANG += -fprofile-sample-use=$(CLANG_AUTOFDO_PROFILE) -ffunction-sections 13 + CFLAGS_AUTOFDO_CLANG += -fsplit-machine-functions 13 14 endif 14 15 15 16 ifdef CONFIG_LTO_CLANG_THIN ··· 18 17 KBUILD_LDFLAGS += --lto-sample-profile=$(CLANG_AUTOFDO_PROFILE) 19 18 endif 20 19 KBUILD_LDFLAGS += --mllvm=-enable-fs-discriminator=true --mllvm=-improved-fs-discriminator=true -plugin-opt=thinlto 20 + KBUILD_LDFLAGS += -plugin-opt=-split-machine-functions 21 21 endif 22 22 23 23 export CFLAGS_AUTOFDO_CLANG