Linux Boot sequence on ARM CPU

Bootloader preparations

Before jumping to kernel entry point boot loader should do at least the following:
1. Setup and initialise the RAM.
2. Initialise one serial port.
3. Detect the machine type.
4. Setup the kernel tagged list.
5. Call the kernel image.
 CPU register settings
  r0 = 0,
  r1 = machine type number discovered in (3) above.
  r2 = physical address of tagged list in system RAM, or
       physical address of device tree block (dtb) in system RAM
 

Low level kernel init

Kernel entry point is arch/arm/kernel/head.S:stext
At this point we save values from boot loader, check that CPU is in correct state, enable low level debug if enabled. Prepare MMU and enable it. Call trace is below.
1) ./arch/arm/kernel/head.S:stext()
//For early printk example look the trace below. For stages w/o MMU you should just set omap_uart_phys as one defined in uboot config file.
   ./arch/arm/kernel/head-common.S:__error_p() ->  printascii() -> addruart_current() ->
        arch/arm/mach-omap2/include/mach/debug-macro.S:addruart()
2) ./arch/arm/kernel/head.S:__enable_mmu()
3) ./arch/arm/kernel/head.S:__turn_mmu_on()
4) ./arch/arm/kernel/head-common.S:__mmap_switched()
5) init/main.c:start_kernel()

Low level debug

To enable console output as soon as possible, CONFIG_EARLY_PRINTK should be enabled.
Then correct physical and virtual address should be set up (for example see [2]). At this point kernel just writes output symbols directly to specific addresses instead of using kernel log daemon.
If UART address values are set up properly you can use assembler routines like ./arch/arm/kernel/head-common.S:printascii() from assembler code. From kernel code you can use early_printk() routine, which actually use same assembler code to throw output symbols though console.

Single thread kernel initialization

Code is in init/main.c:kernel_start() does the following things:
1) Obtain CPU id
2) Initialize runtime locking correctness validator
3) Initialize object tracker. (initialize the hash buckets and link the static object pool objects into the poll list)
4) Set up the the initial canary   (GCC stack protector support. Stack protector works by putting predefined pattern at the start of  the stack frame and verifying that it hasn’t been overwritten when returning from the function.  The pattern is called stack canary and gcc expects it to be defined by a global variable called “__stack_chk_guard” on ARM.  This unfortunately means that on SMP we cannot have a different canary value per task.
5) Initialize cgroups at system boot, and initialize any subsystems that request early init. (cgroups (control groups) is a Linux kernel feature to limit, account and isolate resource usage (CPU, memory, disk I/O, etc.) of process groups.)
==== Disable IRQ ====
6) initialize the tick control (Register the notifier with the clockevents framework)
7) Activate the first processor.
8) Initialize page address pool
9) Setup architecture
   a) Setup CPU configuration and CPU initialization
   b) Setup machine device tree (tags)
   c) Parse early parameters
   d) Initialize mem blocks
   e) sets up the page tables, initialises the zone memory maps, and sets up the zero page, bad page and bad page tables.
   f)  Unflatten device tree
   g) Store callbacks from machine description
   h) Init other CPUs if necessary
   i) reserves memory area given in “crashkernel=” kernel command line parameter. The memory reserved is used by a dump capture kernel when primary kernel is crashing.
   j) Initialize TCM memory (Tightly-coupled Memory, memory which resides directly on the processor of a computer)
   k) Early trap initialization
   l) Call machine early_init routine (if exists)
10) Setup init mm owner and cpumask
11) Store command line (We need to store the untouched command line for future reference. We also need to store the touched command line since the parameter  parsing is performed in place, and we should allow a component to store reference of name/value for future reference.)
12) Save nubmber of CPU IDs
13) SMP percpu area setup
14) Run arch-specific boot CPU hooks (??? SMP staff)
15) Build zone lists
16) Initialize page allocation
17) Parse early parameters (earlycon, console)
18) Parse other parameters
19) Initialize jump_labels
20) Setup log buffer
21) Initialize pid hash table
22) early initialization of vfs caches
23) Sort the kernel’s built-in exception table
24) trap initialization (not implemeted for ARM)
25) Set up kernel memory allocators
26) Set up the scheduler prior starting any interrupts (such as the timer interrupt). Full topology setup happens at smp_init() time – but meanwhile we still have a functioning scheduler.
==== Disable preemption ====
27) initialize idr cache (Small id to pointer translation service.)
28) Initialize performance events core
29) Initialize RCU (Read-Copy Update mechanism for mutual exclusion)
30) Initialize radix tree
31) Early IRQ init (init some links before init_ISA_irqs())
32) IRQ init
33) Initialize priority search tree (A clever mix of heap and radix trees forms a radix priority search tree which is useful for storing intervals.)
34) Init timers
35) Init HR timers
36) Init  soft IRQ
37) Initializes the clocksource and common timekeeping values
38) Set machine timer as a system one and initialize it
39) Initialize simple kernel profiler
40) Register CPUs going up/down notifiers
====Enable IRQ====
41) Late initialize of kmem cache
42) Initialize console
43) Fall with panic here if needed
44) Run lock dependency validator
45) Run locking API test suite
46) Check initrd was not overwritten (if needed)
47) Initialize page cgroup
48) Enable debug page allocation
49) Initialize debug memory objects (Called after the kmem_caches are functional to setup a dedicated cache pool, which has the SLAB_DEBUG_OBJECTS flag set. This flag prevents that the debug code is called on kmem_cache_free() for the debug tracker objects to avoid recursive calls.)
50) Initialize kmemleaks (Kmemleak provides a way of detecting possible kernel memory leaks in a way similar to a tracing garbage collector with the difference that the orphan objects are not freed but only reported via /sys/kernel/debug/kmemleak. A similar method is used by the Valgrind tool (memcheck –leak-check) to detect the memory leaks in user-space applications.)
51) Allocate per cpu pagesets and initialize them.
52) Numa policy initialization (Non Uniform Memory Access policy)
53) Run late time init if provided (Machine specific ?)
54) Initialize schedule clock
55) Calibrating delay loop
56) Initialize pid hash table
57) anon_vma_init (?)
58) initialise the credentials stuff
59) Initialize fork (Allocate space for task structures )
60) Prepare proc caches (allocate memory for fork)
61) Allocate kernel buffer
62) Initialize the key management state.
63) Initialize security framework
64) Late gdb initialization
65) Initialize VFS caches
66) Initialize signals
67) Initialize page write-back
68) Initialize proc FS (if enabled)
69) Initialize cgroups (Register cgroup filesystem and /proc file, and initialize any subsystems that didn’t request early init.
70) Initialize top_cpuset and the cpuset internal file system
71) early initialization of taskstat (Export per-task statistics to userland)
72) Initialize per-task delay accounting
73) Check write buffer bugs
74) Early initialization of ACPI
75) Late initialization of SFI (Simple Firmware Interface)
76) Ftrace initialization
77) Do the rest non-__init’ed, we’re now alive (Create bunch of threads and call schedule to get things moving) Code in init/main.c:rest_init() routine.
   a) Create kernel init thread.
   b) Create kthreadd thread
   c) Prepare scheduler
==== Enable preemption ====
   d) call schedule()

Late kernel initialization

kthread() thread

1) Setup a clean context for our children to inherit
2) If kthread_create_list empty just reschedule
3) Create kernel thread for every task in kthread_create_list
4) Go back to step 2

kernel_init() thread

1) Wait for kernel thread daemon initialization completion
2) Setup init permissions:
    a) init can allocate pages on any node
    b) init can run on any cpu
3) Prepare CPUs for smp
4) Do pre SMP init calls
5) Init lockup detector
6) Enable SMP
7) Initialize SMP support in scheduler
8) Initialize devices in init/main.c:do_basic_setup() (Ok, the machine is now initialized. None of the devices have been touched yet, but the CPU subsystem is up and running, and memory and process management works. Now we can finally start doing some real work..)
    a) Finish top cpuset after cpu, node maps are initialized
    b) Initialize user mode helper
    c) Initialize shmem
    d) Initialize drivers
    e) Initialize /proc/irq handling code
    f) Call all constructor functions linked into the kernel
    g) usermodehelper_enable – allow new helpers to be started again
    h) Do init calls
9) Open the /dev/console on the rootfs
10) check if there is an early userspace init.  If yes, let it do all the work
11) Run late init (Ok, we have completed the initial bootup, and we’re essentially up and running. Get rid of the initmem segments and start the user-mode stuff..)
    a) finish all async __init code before freeing the memory
    b) Free init memory
    c) Mark readonly data as RO
    d) Set system state to  Running
    e) Try to execute userspace init command