Open Source Summit North America 2022

Linux Kernel has two types of panics, namely, hard panics (Aiee!) and soft panics (Oops!). A soft panic occurs when kernel encounters a fault or exception in code, and then it dumps the stack trace in the debug console. The stack dump contains the processor status and the CPU register values when the panic occurred. Any developer must have a good understanding of the target architecture and Linux kernel internals to perform the root cause analysis of an Oops and to debug the issue. This talk is intended for developers who have just begun their journey in Linux Kernel Development. A general methodology of debugging a kernel soft panic is discussed in this talk which can be followed by anyone as starting steps. To achieve this, a simple soft panic is triggered and then a standard approach is followed, which involves understanding the kernel stack dump, back tracing to the faulty code from dump, deciding on which tool to use for debugging depending on the location of faulty code. This talk also discusses several exception templates with examples and guides to understand their types and category for accelerated debugging. The main tools discussed in this paper are, printk, oops tracing, ksymoops, KDB and KGDB.