malloc triggers panic instead of returning NULL on allocation failure (OOM)

[nuczyc]

Describe the bug

When malloc is called with a size that cannot be satisfied by the allocator (causing an Out-Of-Memory condition), the system panics. The C standard specifies that malloc should return NULL on failure, allowing the application to handle the error gracefully.

To Reproduce

1. Compile the program and run.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>

/*
 * PoC for potential arithmetic overflow in ArceOS kernel memory initialization
 * 
 * The crash occurs in axhal/src/mem.rs:56 when calculating .rodata section size:
 *   size: _erodata as usize - _srodata as usize
 * 
 * This is a kernel boot-time initialization issue where linker symbols define
 * section boundaries. The panic happens if _erodata < _srodata or if the
 * difference exceeds usize capacity.
 * 
 * NOTE: This crash is NOT directly reachable from user space because:
 * 1. It occurs during kernel initialization, not in response to syscalls
 * 2. The symbols (_erodata, _srodata) are fixed at link time
 * 3. User applications cannot trigger kernel re-initialization
 * 
 * This PoC attempts to stress the memory subsystem in case there are related
 * user-reachable paths or if the vulnerability has a user-space component.
 */

int main() {
    printf("PoC: Attempting to trigger memory-related kernel behavior\n");
    
    // Allocate large memory chunks to stress the kernel
    size_t sizes[] = {
        1024 * 1024,        // 1MB
        10 * 1024 * 1024,   // 10MB
        100 * 1024 * 1024,  // 100MB
        1024 * 1024 * 1024  // 1GB
    };
    
    void* ptrs[4];
    
    for (int i = 0; i < 4; i++) {
        printf("Allocating %zu bytes...\n", sizes[i]);
        ptrs[i] = malloc(sizes[i]);
        if (ptrs[i]) {
            memset(ptrs[i], 0xAA, sizes[i]);
            printf("  Success: allocated at %p\n", ptrs[i]);
        } else {
            printf("  Failed to allocate\n");
        }
    }
    
    // Try mmap with large sizes
    printf("\nAttempting large mmap allocations...\n");
    void* mmap_ptr = mmap(NULL, 2UL * 1024 * 1024 * 1024, 
                         PROT_READ | PROT_WRITE,
                         MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (mmap_ptr != MAP_FAILED) {
        printf("  mmap success: %p\n", mmap_ptr);
        memset(mmap_ptr, 0x55, 1024 * 1024); // Touch first MB
        munmap(mmap_ptr, 2UL * 1024 * 1024 * 1024);
    } else {
        printf("  mmap failed\n");
    }
    
    // Clean up
    for (int i = 0; i < 4; i++) {
        if (ptrs[i]) {
            free(ptrs[i]);
        }
    }
    
    printf("\nPoC completed. The actual crash occurs in kernel init code\n");
    printf("and is not directly triggerable from user space.\n");
    
    return 0;
}

Environment

• ArceOS version: [Introduce max-cpu-num config item, remove all direct usages of axconf… · arceos-org/arceos@bcc354a](bcc354a)

Logs

SeaBIOS (version 1.16.3-debian-1.16.3-2)


iPXE (https://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+06FCAA90+06F0AA90 CA00
                                                                               


Booting from ROM..TSC frequency: 4000 MHz

       d8888                            .d88888b.   .d8888b.
      d88888                           d88P" "Y88b d88P  Y88b
     d88P888                           888     888 Y88b.
    d88P 888 888d888  .d8888b  .d88b.  888     888  "Y888b.
   d88P  888 888P"   d88P"    d8P  Y8b 888     888     "Y88b.
  d88P   888 888     888      88888888 888     888       "888
 d8888888888 888     Y88b.    Y8b.     Y88b. .d88P Y88b  d88P
d88P     888 888      "Y8888P  "Y8888   "Y88888P"   "Y8888P"

arch = x86_64
platform = x86-pc
target = x86_64-unknown-none
build_mode = release
log_level = warn
smp = 1

PoC: Attempting to trigger memory-related kernel behavior
Allocating 1048576 bytes...
  Success: allocated at 0xffff80000033e018
Allocating 10485760 bytes...
  Success: allocated at 0xffff80000053e018
Allocating 104857600 bytes...
[  0.009926 0 axruntime::lang_items:5] panicked at library/alloc/src/alloc.rs:437:13:
memory allocation of 104857608 bytes failed