Skip to content

add support for linux-arm64 #1138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

add support for linux-arm64 #1138

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
2105dc3
add support for linux-arm64
nipeone Mar 27, 2025
bc4dde8
update compile.yml
nipeone Mar 27, 2025
aeef2eb
update compile.yml DGGML_CPU_ARM_ARCH=armv8-a
nipeone Mar 27, 2025
80d75d9
update runtime.targets
nipeone Mar 27, 2025
5996b40
Merge branch 'SciSharp:master' into master
nipeone Apr 2, 2025
6f8b7ce
Merge branch 'SciSharp:master' into master
nipeone Apr 7, 2025
31c1218
Merge branch 'SciSharp:master' into master
nipeone Apr 11, 2025
6becd43
Merge branch 'SciSharp:master' into master
nipeone Apr 18, 2025
c62980f
Apply suggestions from code review
martindevans Apr 21, 2025
dfb3cc9
Update .github/workflows/compile.yml
martindevans Apr 21, 2025
47f90c4
Merge branch 'master' into master
martindevans Apr 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 26 additions & 7 deletions .github/workflows/compile.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,25 @@ jobs:
include:
- build: 'noavx'
defines: '-DGGML_AVX=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF'
os: ubuntu-24.04
arch: x64
- build: 'avx2'
defines: ''
os: ubuntu-24.04
arch: x64
- build: 'avx'
defines: '-DGGML_AVX2=OFF'
os: ubuntu-24.04
arch: x64
- build: 'avx512'
defines: '-DGGML_AVX512=ON'
runs-on: ubuntu-24.04
os: ubuntu-24.04
arch: x64
- build: 'aarch64'
defines: '-DGGML_NATIVE=OFF -DGGML_CPU_AARCH64=ON -DGGML_CPU_ARM_ARCH=armv8-a'
os: ubuntu-24.04-arm
arch: arm64
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
with:
Expand All @@ -52,28 +64,28 @@ jobs:
- uses: actions/upload-artifact@v4
with:
path: ./build/bin/libllama.so
name: llama-bin-linux-${{ matrix.build }}-x64.so
name: llama-bin-linux-${{ matrix.build }}-${{ matrix.arch }}.so
if-no-files-found: error
- uses: actions/upload-artifact@v4
with:
path: ./build/bin/libggml.so
name: ggml-bin-linux-${{ matrix.build }}-x64.so
name: ggml-bin-linux-${{ matrix.build }}-${{ matrix.arch }}.so
if-no-files-found: error
- uses: actions/upload-artifact@v4
with:
path: ./build/bin/libggml-base.so
name: ggml-base-bin-linux-${{ matrix.build }}-x64.so
name: ggml-base-bin-linux-${{ matrix.build }}-${{ matrix.arch }}.so
if-no-files-found: error
- uses: actions/upload-artifact@v4
with:
path: ./build/bin/libggml-cpu.so
name: ggml-cpu-bin-linux-${{ matrix.build }}-x64.so
name: ggml-cpu-bin-linux-${{ matrix.build }}-${{ matrix.arch }}.so
if-no-files-found: error
- name: Upload Llava
uses: actions/upload-artifact@v4
with:
path: ./build/bin/libllava_shared.so
name: llava-bin-linux-${{ matrix.build }}-x64.so
name: llava-bin-linux-${{ matrix.build }}-${{ matrix.arch }}.so
if-no-files-found: error

compile-musl:
Expand Down Expand Up @@ -601,7 +613,7 @@ jobs:
- name: Rearrange Files
run: |
# Make all directories at once
mkdir --parents deps/{noavx,avx,avx2,avx512,musl-noavx,musl-avx,musl-avx2,musl-avx512,osx-arm64,osx-x64,osx-x64-rosetta2,cu11.7.1,cu12.2.0,vulkan,android-arm64-v8a,android-x86,android-x86_64}
mkdir --parents deps/{noavx,avx,avx2,avx512,linux-arm64,musl-noavx,musl-avx,musl-avx2,musl-avx512,osx-arm64,osx-x64,osx-x64-rosetta2,cu11.7.1,cu12.2.0,vulkan,android-arm64-v8a,android-x86,android-x86_64}

# Linux
cp artifacts/ggml-bin-linux-noavx-x64.so/libggml.so deps/noavx/libggml.so
Expand All @@ -628,6 +640,13 @@ jobs:
cp artifacts/llama-bin-linux-avx512-x64.so/libllama.so deps/avx512/libllama.so
cp artifacts/llava-bin-linux-avx512-x64.so/libllava_shared.so deps/avx512/libllava_shared.so

# Arm64
cp artifacts/ggml-bin-linux-aarch64-arm64.so/libggml.so deps/linux-arm64/libggml.so
cp artifacts/ggml-base-bin-linux-aarch64-arm64.so/libggml-base.so deps/linux-arm64/libggml-base.so
cp artifacts/ggml-cpu-bin-linux-aarch64-arm64.so/libggml-cpu.so deps/linux-arm64/libggml-cpu.so
cp artifacts/llama-bin-linux-aarch64-arm64.so/libllama.so deps/linux-arm64/libllama.so
cp artifacts/llava-bin-linux-aarch64-arm64.so/libllava_shared.so deps/linux-arm64/libllava_shared.so

# Musl
cp artifacts/ggml-bin-musl-noavx-x64.so/libggml.so deps/musl-noavx/libggml.so
cp artifacts/ggml-base-bin-musl-noavx-x64.so/libggml-base.so deps/musl-noavx/libggml-base.so
Expand Down
22 changes: 22 additions & 0 deletions LLama/LLamaSharp.Runtime.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,28 @@
</None>


<None Include="$(MSBuildThisFileDirectory)runtimes/deps/linux-arm64/libllama.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-arm64/native/libllama.so</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/linux-arm64/libggml.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-arm64/native/libggml.so</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/linux-arm64/libggml-base.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-arm64/native/libggml-base.so</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/linux-arm64/libggml-cpu.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-arm64/native/libggml-cpu.so</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/linux-arm64/libllava_shared.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-arm64/native/libllava_shared.so</Link>
</None>


<None Include="$(MSBuildThisFileDirectory)runtimes/deps/cu11.7.1/libllama.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-x64/native/cuda11/libllama.so</Link>
Expand Down
42 changes: 29 additions & 13 deletions LLama/Native/Load/NativeLibraryUtils.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,19 +88,28 @@ internal static IntPtr TryLoadLibrary(NativeLibraryConfig config, out INativeLib
// On other platforms (Windows, Linux), we need to load the CPU backend from the specified AVX level directory
// We are using the AVX level supplied by NativeLibraryConfig, which automatically detects the highest supported AVX level for us

// ggml-cpu
dependencyPaths.Add(Path.Combine(
$"runtimes/{os}/native/{NativeLibraryConfig.AvxLevelToString(library.Metadata.AvxLevel)}",
$"{libPrefix}ggml-cpu{ext}"
));

// ggml-cuda
if (library.Metadata.UseCuda)
dependencyPaths.Add(Path.Combine(currentRuntimeDirectory, $"{libPrefix}ggml-cuda{ext}"));

// ggml-vulkan
if (library.Metadata.UseVulkan)
dependencyPaths.Add(Path.Combine(currentRuntimeDirectory, $"{libPrefix}ggml-vulkan{ext}"));
if (os == "linux-arm64"){
dependencyPaths.Add(Path.Combine(
$"runtimes/{os}/native",
$"{libPrefix}ggml-cpu{ext}"
));
}
else{
// ggml-cpu
dependencyPaths.Add(Path.Combine(
$"runtimes/{os}/native/{NativeLibraryConfig.AvxLevelToString(library.Metadata.AvxLevel)}",
$"{libPrefix}ggml-cpu{ext}"
));

// ggml-cuda
if (library.Metadata.UseCuda)
dependencyPaths.Add(Path.Combine(currentRuntimeDirectory, $"{libPrefix}ggml-cuda{ext}"));

// ggml-vulkan
if (library.Metadata.UseVulkan)
dependencyPaths.Add(Path.Combine(currentRuntimeDirectory, $"{libPrefix}ggml-vulkan{ext}"));
}

}
}

Expand Down Expand Up @@ -218,6 +227,13 @@ public static void GetPlatformPathParts(OSPlatform platform, out string os, out

if (platform == OSPlatform.Linux)
{
if(System.Runtime.Intrinsics.Arm.ArmBase.Arm64.IsSupported){
// linux arm64
os = "linux-arm64";
fileExtension = ".so";
libPrefix = "lib";
return;
}
if(RuntimeInformation.RuntimeIdentifier.ToLower().StartsWith("alpine"))
{
// alpine linux distro
Expand Down
16 changes: 11 additions & 5 deletions LLama/Native/Load/NativeLibraryWithAvx.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,17 @@ public IEnumerable<string> Prepare(SystemInfo systemInfo, NativeLogConfig.LLamaL
private string? GetAvxPath(SystemInfo systemInfo, AvxLevel avxLevel, NativeLogConfig.LLamaLogCallback? logCallback)
{
NativeLibraryUtils.GetPlatformPathParts(systemInfo.OSPlatform, out var os, out var fileExtension, out var libPrefix);
var avxStr = NativeLibraryConfig.AvxLevelToString(avxLevel);
if (!string.IsNullOrEmpty(avxStr))
avxStr += "/";
var relativePath = $"runtimes/{os}/native/{avxStr}{libPrefix}{_libraryName.GetLibraryName()}{fileExtension}";
return relativePath;
if (os != "linux-arm64"){
var avxStr = NativeLibraryConfig.AvxLevelToString(avxLevel);
if (!string.IsNullOrEmpty(avxStr))
avxStr += "/";
var relativePath = $"runtimes/{os}/native/{avxStr}{libPrefix}{_libraryName.GetLibraryName()}{fileExtension}";
return relativePath;
} else {
var relativePath = $"runtimes/{os}/native/{libPrefix}{_libraryName.GetLibraryName()}{fileExtension}";
return relativePath;
}

}
}
#endif
Expand Down
8 changes: 7 additions & 1 deletion LLama/runtimes/build/LLamaSharp.Backend.Cpu.nuspec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,13 @@
<file src="runtimes/deps/avx512/libllama.so" target="runtimes\linux-x64\native\avx512\libllama.so" />
<file src="runtimes/deps/avx512/libllava_shared.so" target="runtimes\linux-x64\native\avx512\libllava_shared.so" />

<file src="runtimes/deps/musl-noavx/libggml.so" target="runtimes\linux-musl-x64\native\noavx\libggml.so" />
<file src="runtimes/deps/linux-arm64/libggml.so" target="runtimes\linux-arm64\native\libggml.so" />
<file src="runtimes/deps/linux-arm64/libggml-base.so" target="runtimes\linux-arm64\native\libggml-base.so" />
<file src="runtimes/deps/linux-arm64/libggml-cpu.so" target="runtimes\linux-arm64\native\libggml-cpu.so" />
<file src="runtimes/deps/linux-arm64/libllama.so" target="runtimes\linux-arm64\native\libllama.so" />
<file src="runtimes/deps/linux-arm64/libllava_shared.so" target="runtimes\linux-arm64\native\libllava_shared.so" />

<file src="runtimes/deps/musl-noavx/libggml.so" target="runtimes\linux-musl-x64\native\noavx\libggml.so" />
<file src="runtimes/deps/musl-noavx/libggml-base.so" target="runtimes\linux-musl-x64\native\noavx\libggml-base.so" />
<file src="runtimes/deps/musl-noavx/libggml-cpu.so" target="runtimes\linux-musl-x64\native\noavx\libggml-cpu.so" />
<file src="runtimes/deps/musl-noavx/libllama.so" target="runtimes\linux-musl-x64\native\noavx\libllama.so" />
Expand Down