[axlibc] malloc panics instead of returning NULL when allocation size causes LayoutError

[nuczyc]

Describe the bug

When malloc is called with an extremely large size (e.g., close to SIZE_MAX), the system panics instead of returning NULL.

arceos/ulib/axlibc/src/malloc.rs

Line 29 in bcc354a

let layout = Layout::from_size_align(size + CTRL_BLK_SIZE, 8).unwrap();

To Reproduce

1. Compile the program and run.

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <limits.h>

int main() {
    // This PoC attempts to trigger an arithmetic overflow in the kernel's page allocator
    // The overflow occurs in axalloc/src/page.rs:61 in the size() function:
    //   self.num_pages * PAGE_SIZE
    // When num_pages is very large, this multiplication can overflow usize
    
    printf("Attempting to trigger arithmetic overflow in page allocator...\n");
    
    // Try to allocate a size that would cause overflow in page calculation
    // We need a size that's close to SIZE_MAX but still requires multiple pages
    size_t large_size = SIZE_MAX - 4095;  // Just under SIZE_MAX, will need multiple pages
    
    printf("Attempting malloc with size: %zu bytes\n", large_size);
    
    // This malloc call should eventually reach the kernel's page allocator
    // The allocator will calculate num_pages = ceil(size / PAGE_SIZE)
    // Then size() will compute num_pages * PAGE_SIZE, which may overflow
    void *ptr = malloc(large_size);
    
    if (ptr != NULL) {
        printf("Allocation succeeded unexpectedly\n");
        free(ptr);
    } else {
        printf("Allocation failed as expected\n");
    }
    
    // Try with calloc as well - it might take a different code path
    printf("\nAttempting calloc with large count and size...\n");
    size_t count = SIZE_MAX / 4096;
    size_t elem_size = 4096;
    
    void *ptr2 = calloc(count, elem_size);
    if (ptr2 != NULL) {
        printf("Calloc succeeded unexpectedly\n");
        free(ptr2);
    } else {
        printf("Calloc failed as expected\n");
    }
    
    return 0;
}

Environment

• ArceOS version: [Introduce max-cpu-num config item, remove all direct usages of axconf… · arceos-org/arceos@bcc354a](bcc354a)

Logs

SeaBIOS (version 1.16.3-debian-1.16.3-2)


iPXE (https://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+06FCAA90+06F0AA90 CA00
                                                                               


Booting from ROM..TSC frequency: 4000 MHz

       d8888                            .d88888b.   .d8888b.
      d88888                           d88P" "Y88b d88P  Y88b
     d88P888                           888     888 Y88b.
    d88P 888 888d888  .d8888b  .d88b.  888     888  "Y888b.
   d88P  888 888P"   d88P"    d8P  Y8b 888     888     "Y88b.
  d88P   888 888     888      88888888 888     888       "888
 d8888888888 888     Y88b.    Y8b.     Y88b. .d88P Y88b  d88P
d88P     888 888      "Y8888P  "Y8888   "Y88888P"   "Y8888P"

arch = x86_64
platform = x86-pc
target = x86_64-unknown-none
build_mode = release
log_level = warn
smp = 1

Attempting to trigger arithmetic overflow in page allocator...
Attempting malloc with size: 18446744073709547520 bytes
[  0.004838 0 axruntime::lang_items:5] panicked at ulib/axlibc/src/malloc.rs:29:67:
called `Result::unwrap()` on an `Err` value: LayoutError