Analyze command failed with errors

[QGrain]

Hello, I am trying to set up and run KernelGPT. But when following the README, I encountered two issues.

# My envs:
ubuntu 22.04 in docker
clang-14.0.0 with libclang-dev
syzkaller-related packages are installed

1. Failed to compile kernel 6.6.12 with allyesconfig. I also tried upgrading clang to 17.0.6 and compiling allysconfig again, but it also failed with the same output. (Then I compiled it with a syzbot .config)

...
  CC      drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp2_execution.o
  CC      drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp2_transition.o
  AR      drivers/gpu/drm/nouveau/built-in.a
  AR      drivers/gpu/drm/amd/amdgpu/built-in.a
  AR      drivers/gpu/drm/built-in.a
  AR      drivers/gpu/built-in.a
  AR      drivers/built-in.a
make[1]: *** [/root/local_kernels/linux-6.6.12-allyes/Makefile:1913: .] Error 2
make: *** [Makefile:234: __sub-make] Error 2

2. Failed to execute ./analyze -p /my/path/compile_commands.json. (Note: compile_commands.json is generated with the syzbot .config rather than allyesconfig)

... After running several hours ...
...
fatal error: too many errors emitted, stopping now [-ferror-limit=]
/root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/mkpiggy.c
/root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/early_serial_console.c
/root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/ident_map_64.c
/root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/idt_64.c
error: unsupported option '--mrelax-relocations'
error: unknown argument: '-cc1'
error: unknown argument: '-triple'
error: unknown argument: '-clear-ast-before-backend'
error: unknown argument: '-main-file-name'
error: unknown argument: '-mrelocation-model'
error: unknown argument: '-pic-level'
error: unknown argument: '-pic-is-pie'
error: unknown argument: '-mframe-pointer=none'
error: unknown argument: '-mconstructor-aliases'
error: unknown argument: '-funwind-tables=2'
error: unknown argument: '-target-cpu'
error: unknown argument: '-tune-cpu'
error: unknown argument: '-internal-isystem'
error: unknown argument: '-internal-isystem'
error: unknown argument: '-internal-isystem'
error: unknown argument: '-internal-externc-isystem'
error: unknown argument: '-internal-externc-isystem'
error: unknown argument: '-internal-externc-isystem'
fatal error: too many errors emitted, stopping now [-ferror-limit=]
/root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/cpuflags.c
/root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/acpi.c
/root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/pgtable_64.c
/root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/mkpiggy.c
/root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/misc.c
In file included from arch/x86/boot/compressed/idt_64.c:5:
In file included from arch/x86/boot/compressed/misc.h:42:
./include/linux/acpi.h:37:10: fatal error: 'acpi/acpi_io.h' file not found
#include <acpi/acpi_io.h>
         ^~~~~~~~~~~~~~~~
In file included from arch/x86/boot/compressed/acpi.c:3:
In file included from arch/x86/boot/compressed/misc.h:42:
In file included from ./include/linux/acpi.h:13:
In file included from ./include/linux/resource_ext.h:11:
In file included from ./include/linux/slab.h:16:
In file included from ./include/linux/gfp.h:7:
./include/linux/mmzone.h:24:10: fatal error: 'linux/local_lock.h' file not found
#include <linux/local_lock.h>
         ^~~~~~~~~~~~~~~~~~~~
error: error opening 'arch/x86/boot/.cpu.o.d': No such file or directory
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/arch/x86/boot/cpu.c.
error: unsupported option '--mrelax-relocations'
error: unknown argument: '-cc1'
error: unknown argument: '-triple'
error: unknown argument: '-clear-ast-before-backend'
error: unknown argument: '-main-file-name'
error: unknown argument: '-mrelocation-model'
error: unknown argument: '-pic-level'
error: unknown argument: '-pic-is-pie'
error: unknown argument: '-mframe-pointer=none'
error: unknown argument: '-mconstructor-aliases'
error: unknown argument: '-funwind-tables=2'
error: unknown argument: '-target-cpu'
error: unknown argument: '-tune-cpu'
error: unknown argument: '-internal-isystem'
error: unknown argument: '-internal-isystem'
error: unknown argument: '-internal-isystem'
error: unknown argument: '-internal-externc-isystem'
error: unknown argument: '-internal-externc-isystem'
fatal error: too many errors emitted, stopping now [-ferror-limit=]
error: unsupported option '--mrelax-relocations'
error: unknown argument: '-cc1'
error: unknown argument: '-triple'
error: unknown argument: '-clear-ast-before-backend'
error: unknown argument: '-main-file-name'
error: unknown argument: '-mrelocation-model'
error: unknown argument: '-pic-level'
error: unknown argument: '-pic-is-pie'
error: unknown argument: '-mframe-pointer=none'
error: unknown argument: '-mconstructor-aliases'
error: unknown argument: '-funwind-tables=2'
error: unknown argument: '-target-cpu'
error: unknown argument: '-tune-cpu'
error: unknown argument: '-internal-isystem'
error: unknown argument: '-internal-isystem'
error: unknown argument: '-internal-isystem'
error: unknown argument: '-internal-externc-isystem'
error: unknown argument: '-internal-externc-isystem'
error: unknown argument: '-internal-externc-isystem'
fatal error: too many errors emitted, stopping now [-ferror-limit=]
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-icade.c.
error: error opening 'drivers/hid/.hid-holtek-kbd.o.d': No such file or directory
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-holtek-kbd.c.
error: error opening 'drivers/hid/.hid-lenovo.o.d': No such file or directory
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-lenovo.c.
error: error opening 'init/.version-timestamp.o.d': No such file or directory
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/init/version-timestamp.c.
error: error opening 'drivers/hid/.hid-lg4ff.o.d': No such file or directory
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-lg4ff.c.
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-saitek.c.
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-retrode.c.
error: error opening 'drivers/hid/.hid-roccat-common.o.d': No such file or directory
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-roccat-common.c.
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-roccat-savu.c.
error: error opening 'drivers/hid/.hid-roccat-koneplus.o.d': No such file or directory
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-roccat-koneplus.c.
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-roccat-ryos.c.
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/cmdline.c.
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/acpi.c.
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/arch/x86/boot/compressed/idt_64.c.
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-rmi.c.
error: error opening 'drivers/hid/.hid-tivo.o.d': No such file or directory
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-tivo.c.
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-uclogic-core.c.
1 error generated.
Error while processing /root/local_kernels/linux-6.6.12/drivers/hid/hid-wiimote-core.c.

Is there any difference between my envs and yours?

[YangChenyuan]

Hi @QGrain,
Thank you for your interest, and I apologize for the delayed response.

Regarding your first issue about the build errors: This could indeed be kernel-related. We've previously encountered similar build failures that required manual kernel code patches. I'm currently testing a build with Linux v6.6 and will update you with the results.

As for your second point about build command failures: You're correct that certain files may fail with our current approach. We're exploring the use of make XXX.o as an alternative to direct clang commands. If this proves more reliable, we'll update our implementation accordingly.

I'll follow up once I have more concrete information about the Linux v6.6 build.

[pikaball]

I also met the second problem, and my solution is running analyze under the linux kernel source tree:

cd /path/to/linux
../spec-gen/analyze -p compile_commands.json

the include paths in compile_commands.json are relative paths, I guess this is the reason

[Lelouny]

@pikaball i followed your solution and it works，thanks！but i encountered a new issue now when i run python process_output.py --linux-path /path/to/linux --usage:

Traceback (most recent call last):
  File "process_output.py", line 227, in <module>
    process_usage()
  File "process_output.py", line 201, in process_usage
    for line in file_path.read_text().splitlines():
  File "/usr/lib/python3.8/pathlib.py", line 1236, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/usr/lib/python3.8/pathlib.py", line 1222, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/usr/lib/python3.8/pathlib.py", line 1078, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/li/labdir/linux-6.7/usage.jsonl'

Have you encountered this issue?

[pikaball]

@pikaball i followed your solution and it works，thanks！but i encountered a new issue now when i run python process_output.py --linux-path /path/to/linux --usage:

Traceback (most recent call last):
File "process_output.py", line 227, in
process_usage()
File "process_output.py", line 201, in process_usage
for line in file_path.read_text().splitlines():
File "/usr/lib/python3.8/pathlib.py", line 1236, in read_text
with self.open(mode='r', encoding=encoding, errors=errors) as f:
File "/usr/lib/python3.8/pathlib.py", line 1222, in open
return io.open(self, mode, buffering, encoding, errors, newline,
File "/usr/lib/python3.8/pathlib.py", line 1078, in _opener
return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/li/labdir/linux-6.7/usage.jsonl'
Have you encountered this issue?

yes but not so clear about it.
the usage program generate usage.jsonl when I try to get compile_commands.json from the whole kernel with this command: make CC=clang compile_commands.json -j $(nproc)
as I compile the subdir with the following commands, it performs like you mentioned

make clean && make mrproper
# get .config ...
make CC=clang HOSTCC=clang olddefconfig
make CC=clang HOSTCC=clang prepare
bear -- make CC=clang HOSTCC=clang drivers/usb/ -j$(nproc)

I try to use uasge.jsonl generating from whole kernel in my work that focus on subdir, but don't know if it's the correct way

[Lelouny]

@pikaball i followed your solution and it works，thanks！but i encountered a new issue now when i run python process_output.py --linux-path /path/to/linux --usage:
Traceback (most recent call last):
File "process_output.py", line 227, in
process_usage()
File "process_output.py", line 201, in process_usage
for line in file_path.read_text().splitlines():
File "/usr/lib/python3.8/pathlib.py", line 1236, in read_text
with self.open(mode='r', encoding=encoding, errors=errors) as f:
File "/usr/lib/python3.8/pathlib.py", line 1222, in open
return io.open(self, mode, buffering, encoding, errors, newline,
File "/usr/lib/python3.8/pathlib.py", line 1078, in _opener
return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/li/labdir/linux-6.7/usage.jsonl'
Have you encountered this issue?

yes but not so clear about it. the usage program seems to generate usage.jsonl only when I compile the whole kernel as I compile the subdir with the following commands, it performs like you mentioned

make clean && make mrproper
# get .config ...
make CC=clang HOSTCC=clang olddefconfig
make CC=clang HOSTCC=clang prepare
bear -- make CC=clang HOSTCC=clang drivers/usb/ -j$(nproc)

I try to use uasge.jsonl generating from whole kernel in my work that focus on subdir, but don't know if it's the correct way

In fact, i compiled the whole kernel（Linux6.7）with syzbot.config. I didn't use allyesconfig because make failed. Can i know which version of Linux kernel and what .config do you use？Did you generate usage.jsonl successfully when you compile the whole kernel?

[pikaball]

@pikaball i followed your solution and it works，thanks！but i encountered a new issue now when i run python process_output.py --linux-path /path/to/linux --usage:
Traceback (most recent call last):
File "process_output.py", line 227, in
process_usage()
File "process_output.py", line 201, in process_usage
for line in file_path.read_text().splitlines():
File "/usr/lib/python3.8/pathlib.py", line 1236, in read_text
with self.open(mode='r', encoding=encoding, errors=errors) as f:
File "/usr/lib/python3.8/pathlib.py", line 1222, in open
return io.open(self, mode, buffering, encoding, errors, newline,
File "/usr/lib/python3.8/pathlib.py", line 1078, in _opener
return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/li/labdir/linux-6.7/usage.jsonl'
Have you encountered this issue?

yes but not so clear about it. the usage program seems to generate usage.jsonl only when I compile the whole kernel as I compile the subdir with the following commands, it performs like you mentioned

make clean && make mrproper
# get .config ...
make CC=clang HOSTCC=clang olddefconfig
make CC=clang HOSTCC=clang prepare
bear -- make CC=clang HOSTCC=clang drivers/usb/ -j$(nproc)

I try to use uasge.jsonl generating from whole kernel in my work that focus on subdir, but don't know if it's the correct way

In fact, i compiled the whole kernel（Linux6.7）with syzbot.config. I didn't use allyesconfig because make failed. Can i know which version of Linux kernel and what .config do you use？Did you generate usage.jsonl successfully when you compile the whole kernel?

I've correct my answer above, sry:)
I used make compile_commands.json but not bear -- make when processing the whole kernel
the version is v6.11, config from https://syzkaller.appspot.com/text?tag=KernelConfig&x=fd7ef62cf6a59063

[YangChenyuan]

Thanks so much for your interest and helpful comments!!

python process_output.py --linux-path /path/to/linux --usage failed

You have to run ./usage -p /path/to/linux/compile_commands.json to create the usage.jsonl, which collects all the usage to the handler, like

{"alias":"pipefifo_fops","filename":"/home/chenyuan/projects/kgpt-uiuc/linux/fs/pipe.c:882","name":"get_pipe_inode","source":"static struct inode * get_pipe_inode(void)\n{\n\tstruct inode *inode = new_inode_pseudo(pipe_mnt->mnt_sb);\n\tstruct pipe_inode_info *pipe;\n\n\tif (!inode)\n\t\tgoto fail_inode;\n\n\tinode->i_ino = get_next_ino();\n\n\tpipe = alloc_pipe_info();\n\tif (!pipe)\n\t\tgoto fail_iput;\n\n\tinode->i_pipe = pipe;\n\tpipe->files = 2;\n\tpipe->readers = pipe->writers = 1;\n\tinode->i_fop = &pipefifo_fops;\n\n\t/*\n\t * Mark the inode dirty from the very beginning,\n\t * that way it will never be moved to the dirty\n\t * list because \"mark_inode_dirty()\" will think\n\t * that it already _is_ on the dirty list.\n\t */\n\tinode->i_state = I_DIRTY;\n\tinode->i_mode = S_IFIFO | S_IRUSR | S_IWUSR;\n\tinode->i_uid = current_fsuid();\n\tinode->i_gid = current_fsgid();\n\tsimple_inode_init_ts(inode);\n\n\treturn inode;\n\nfail_iput:\n\tiput(inode);\n\nfail_inode:\n\treturn NULL;\n}"}

Note: This process relies on the preceding step, python process_output.py --linux-path /path/to/linux, which collects handler names into the file handler_names.txt (see https://github.com/ise-uiuc/KernelGPT/blob/main/spec-gen/analyzer/usage.cpp#L124 and https://github.com/ise-uiuc/KernelGPT/blob/main/spec-gen/analyzer/process_output.py#L66).

I didn't use allyesconfig because make failed

Here is the patch for the kernel code (commit d2f51b3516dade79269ff45eae2a7668ae711b25):

diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index f558c054cf6e..6064895daece 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -658,7 +658,7 @@ static void tcp_options_write(struct tcphdr *th, struct tcp_sock *tp,
 			memset(ptr, TCPOPT_NOP, sizeof(*ptr));
 			ptr++;
 		}
-out_ao:
+out_ao:;
 #endif
 	}
 	if (unlikely(opts->mss)) {
diff --git a/sound/soc/codecs/aw88399.c b/sound/soc/codecs/aw88399.c
index ce30bc7cdea9..47fe38d0def5 100644
--- a/sound/soc/codecs/aw88399.c
+++ b/sound/soc/codecs/aw88399.c
@@ -438,7 +438,7 @@ static int aw_dev_set_vcalb(struct aw88399 *aw88399)
 	if (ret)
 		return ret;
 
-	vsense_select = vsense_select & (~AW88399_VDSEL_MASK);
+	vsense_select = vsense_value & (~AW88399_VDSEL_MASK);
 
 	ret = aw88399_dev_get_icalk(aw88399, &icalk);
 	if (ret) {

Please note that when generating the specification, we use allyesconfig to cover as much code as possible.

[Lelouny]

@YangChenyuan Thanks for your reply！I have already run preceding steps in README, including:

• patched the kernel with the patch given by you and make kernel sucessfully with allyesconfig
• run ./usage -p /path/to/linux/compile_commands.json which didn't have obvious errors but failed to create usage.jsonl

So when running python process_output.py --linux-path /path/to/linux --usage , it still failed with No such file or directory: '/home/li/labdir/linux-6.7/usage.jsonl'. Could you give me more advice?

[YangChenyuan]

Do you have handler_names.txt under the analyzer directory? Could you show me what you have under this directory?

[selfbypass]

Hi,

I also met some errors, like
"error: unknown warning option '-Wno-format-truncation-non-kprintf'; did you mean '-Wno-empty-translation-unit'? [-Werror,-Wunknown-warning-option]",
"./include/linux/mm_types.h:1068:4: error: unknown attribute 'randomize_layout' ignored [-Werror,-Wunknown-attributes]".
Linux kernel version is 6.13.6.

[YangChenyuan]

Hi,

I also met some errors, like "error: unknown warning option '-Wno-format-truncation-non-kprintf'; did you mean '-Wno-empty-translation-unit'? [-Werror,-Wunknown-warning-option]", "./include/linux/mm_types.h:1068:4: error: unknown attribute 'randomize_layout' ignored [-Werror,-Wunknown-attributes]". Linux kernel version is 6.13.6.

Hi, could you provide me with more context? Are you compiling the kernel with allyesconfig by Clang?

[selfbypass]

Hi,
I also met some errors, like "error: unknown warning option '-Wno-format-truncation-non-kprintf'; did you mean '-Wno-empty-translation-unit'? [-Werror,-Wunknown-warning-option]", "./include/linux/mm_types.h:1068:4: error: unknown attribute 'randomize_layout' ignored [-Werror,-Wunknown-attributes]". Linux kernel version is 6.13.6.

Hi, could you provide me with more context? Are you compiling the kernel with allyesconfig by Clang?

Yeah, but I used Clang-18 to build the kernel and I switched to Clang-14 for analysis. Will it cause errors? I am building other kernel versions with Clang-14.

[YangChenyuan]

Hi,
I also met some errors, like "error: unknown warning option '-Wno-format-truncation-non-kprintf'; did you mean '-Wno-empty-translation-unit'? [-Werror,-Wunknown-warning-option]", "./include/linux/mm_types.h:1068:4: error: unknown attribute 'randomize_layout' ignored [-Werror,-Wunknown-attributes]". Linux kernel version is 6.13.6.

Hi, could you provide me with more context? Are you compiling the kernel with allyesconfig by Clang?

Yeah, but I used Clang-18 to build the kernel and I switched to Clang-14 for analysis. Will it cause errors? I am building other kernel versions with Clang-14.

I suspect this might be causing an issue. It would be better to use the same clang version since we rely on the commands from the kernel build process in our analysis. Please let me know if you encounter any other issues after switching to the same version.