ghostmirror v0.18.5

Introduction:

We've all experienced moments when there seemed to be no packages to update, only to realize days later that the issue was an out-of-sync mirror.
Despite the numerous tools available on Arch Linux for managing mirrors, none of them fully met my expectations.
So, I set out to solve this problem.
While writing the software, I realized that by continuously adding features, ghostmirror has become a great tool for the mirror maintainers themselves.
In fact, besides displaying mirrors that have errors, it can investigate and show the possible causes that generated the error.

Description:

What does GhostMirror do?
It compares the mirror databases with the local database and provides a detailed description of whether the mirror's packages are more or less up-to-date compared to our local database.
It can analyze the mirrors and display in-depth errors or the names of packages that are not updated.
Thanks to the custom sorting mode, it can create a list of mirrors based on each user's needs.
If you don't have time to update the mirror list manually, by adding a single command-line argument, the systemd service will be automatically activated. Feel free to forget about the mirrors.

Install:

You can use your favorite AUR helper to install it.

$ yay -S ghostmirror

otherwise you can find PKGBUILD in distro dir

$ cd distro && makepkg -sirc

Usage:

You can use the software in three ways: manually, automatically, or for investigation.

Manually

In this mode, you will perform all the steps manually.

First step

in the first step you need to search a good quantity of mirrors.
-P for get progress and output colors, -o for get output table.
-c for select country, for example Italy,Germany,France
-l where do you save list
-L max numbers of output mirror in list
-S sort mode, first add state, in this mode remove error mirror, after you can add outofdate, in this mode display first the mirror sync, can also add morerecent to ensure you never go out of sync, and at the end, you can add ping to try to prioritize the closest ones.

$ ghostmirror -PoclLS Italy,Germany,France ./mirrorlist.new 30 state,outofdate,morerecent,ping

Second step

Now, instead of taking the mirrors from the global list, we will better evaluate the mirrors found in the first step.
While the first step will be performed only once or rarely, this step will be the one you repeat periodically.
-P -o -l is same the previous step
-m tell software to use a local mirror list, and -u for use only uncommented mirror
-s for apply a real test for mirror speed
-S change a sort mode, remove ping, add estimated for get more stable mirror and speed for reorder speed

$ ghostmirror -PmuolsS  ./mirrorlist.new ./mirrorlist.new light state,outofdate,morerecent,estimated,speed

Last step

now you save your old mirrorlist

# cp /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.bak

now save new mirrorlist

# cp ./mirrorlist.new /etc/pacman.d/mirrorlist

Direct replace mirrorlist

If you don't want to create a temporary file and prefer to modify the mirrorlist directly, you can run ghostmirror with sudo on the mirrorlist itself.
the first and second step are same but with sudo and different mirror file
only first time, for generate personal mirrorlist

$ sudo ghostmirror -PoclLS Italy,Germany,France /etc/pacman.d/mirrorlist 30 state,outofdate,morerecent,ping

every time you need refresh

$ sudo ghostmirror -PmuolsS  /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist light state,outofdate,morerecent,estimated,speed

Automatically

In this mode, you will perform the second step automatically.

Prepare

you need manual make dir for a new location of mirrorlist, need location where user can edit this without root privilege

$ mkdir ~/.config/ghostmirror

now you need to inform pacman where you have stored mirrorlist
edit file /etc/pacman.conf, search and replace this line and change with your username.

[core]
Include = /home/<username>/.config/ghostmirror/mirrorlist

[extra]
Include = /home/<username>/.config/ghostmirror/mirrorlist

First step

same manually mode you need create a big good mirrorlist

$ ghostmirror -PoclLS Italy,Germany,France ~/.config/ghostmirror/mirrorlist 30 state,outofdate,morerecent,ping

Last step

the arguments,lsS, you pass to ghostmirror at this point it's saved for auto reuse it in service.
so, the only difference with manually second step is -D option, this enable ghostmirror.timer and loginctl linger

$ ghostmirror -PoDumlsS  ~/.config/ghostmirror/mirrorlist ~/.config/ghostmirror/mirrorlist light state,outofdate,morerecent,estimated,speed

Use of systemd

show the timer

$ systemctl --user list-timers

force start ghostmirror before timer ellapsed

$ systemctl --user start ghostmirror.service

End

Now you can forget about mirrors forever.

Investigation

It's be simple, add -i option and it's show the list of error server with motivation.

Reference

the software accept long options with -- and assign value without space but with =

--option
--option value
--option=value

accept short option with - or multiple option, followed by value

-o
-o value
-abc valueA valueB valueC

All Options

-a --arch

select arch, default 'x86_64'

-m --mirrorfile

use mirror file instead of downloading mirrorlist, without -m and -u ghostmirror download mirrorlist and search in all mirror.
you can use -m /etc/pacman.d/mirrorlist.pacnew if you not want downloading mirrorlist but used local list.

-c --country

select country from mirrorlist.
if mirrorlist is not same the default mirrorlist and not have ##Country or mixed mirror, the selection country fail

-C --country-list

show all possibile country

-u --uncommented

use only uncommented mirror, by default use commented and uncommented mirror.
with -m is simple to check local mirrolist -mu /etc/pacman.d/mirrorlist.

-d --downloads

set numbers of parallel download, default '4'.
if you abuse the download is more simple to fail, 1,4,8,16 is good value, >16 you can try but is not very affidable

-O --timeout

set connection timeout in seconds for not reply mirror

-p --progress

show progress, default false

-P --progress-colors

same -p but with -o add color table

-o --output

enable table output, with -P display with colors

-s --speed

test speed for downloading:
light: download one small package ~6MiB
normal: download light + normal package ~250Mib
heavy: download light+normal+heavy packege ~350MiB, total download >500Mib

-S --sort

sort result for any of fields display in table, mutiple fields supported.
country and mirror is sorted by name
proxy first false, last true
state first success last error
outofdate, retry and ping, display first less value
uptodate, morerecent, sync, speed and estimated, display first great value

-l --list

save new mirrorlist in file passed as argument.
special name, stdout, can be used for write to stdout file.

-L --max-list

set max numbers of output mirrors

-T --type

select mirrors type, http,https,all

-i --investigate

search mirror errors to detect the problem.
can select mode: error, outofdate, all.
error: investigate only on error.
outofdate: investigate only on outofdate package.
all: same passing -i error,outofdate

-D --systemd

auto manager systemd.timer.
when you pass this option the software activate login linger if not ebabled.
auto create ghostmirror.service and ghostmirror.timer
the config.service start ghostmirror in the same mode you haved executed it, with only differences that need -l.
for exaples if you execute: -DmuldsS 16 light estimated,speed
the service is always start with 16 parallel downloads, speed light and estimated,speed sort.
for change you can simple repeat a command.
the expire timer is the first element in table and is dinamic, can change every time.

-t --time

in systemd timer whend extimate date is ellapsed can set a time. Set specific hh:mm:ss when start service, default 00:00:00.
validate input with systemd-analyzer calendar hh:mm:ss before use

-f --fixed-time

in systemd timer use fixed time instead of estimated time.
validate input with systemd-analyzer calendar hh:mm:ss before use

-h --help

display this

Manual Build

This section is for developer

Require:

libcurl, zlib, systemd-libs

Build:

$ meson setup build
$ cd build
$ ninja

Debug:

for enable very verbose output.

$ meson configure -Debug=4

warning this is only for contributor, enable auto versioning and auto push

$ meson configure -Developer=true

Bug:

try to writing many