Dust never finishes

[sasq64]

Did cargo install and ran it in my home directory.
Quickly counted up to

Indexing: . 2835562 files, 299G ... /

but then doesn't seem to get any further. Interrupted it after 5 minutes.

[Amr-Shams]

could u use dust --print-errors

[sankalp-khare]

Same issue on v1.2.0 (installed using homebrew).
Tried with dust --print-errors but it kept spinning and no error details got printed:

❯ dust --print-errors
Indexing: . 4584637 files, 286G ... /   # the spinner keeps rotating, but nothing gets printed

I built v1.1.1 locally and tested it on the same filesystem, and the issue does not happen.

• to be specific, v.1.1.1 handles the Interrupted system call (os error 4) and completes execution, see Dust never finishes #495 (comment)

Looks like a regression introduced in v1.2.0.

@sasq64 do you mind trying this on your end? Build with tag v.1.1.1 code and see if the issue is reproducible?

Is there a way I could get dust to print what file it is currently processing? I don't see a --verbose equivalent in the flags.

[Amr-Shams]

well there is no verbose in the flags
yet i have tried to add it locally to my version-tbh i don't want this flag to be there for many reasons
fork my repo I have added the following changes to the latest commit

[sasq64]

My suspicion was that it tried to access a folder that it didn't have permissions for, because I was using a new terminal emulator on macOS when I got the error...

[Amr-Shams]

interesting but with the flag print-errors is enabled it should show this before the spawn

[sankalp-khare]

... because I was using a new terminal emulator on macOS when I got the error...

@sasq64 are you saying the issue happens only with your new terminal emulator, but not with your previous one?

[sasq64]

I have not had it happen again, but I know that if you scan your home directory from a newly installed application on macOS, it will pop up dialogs asking for permissions to read various directories.

[sankalp-khare]

@Amr-Shams thanks for your commit adding -V option!
Using the release executable built from Amr-Shams@419e669 I can see the issue:

Scanning: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/1.bp.blogspot.com/-McqWCErHM6s/XADb1-1fVqI/AAAAAAAAMsY/9u0U361v_MMHaVVPZfwBhm3jEFajYQPBACK4BGAYYCw/s1600
    File: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/1.bp.blogspot.com/-McqWCErHM6s/XADb1-1fVqI/AAAAAAAAMsY/9u0U361v_MMHaVVPZfwBhm3jEFajYQPBACK4BGAYYCw/s1600/Slide003.png (size: 176128)
Indexing: . 4586481 files, 287G ... |Error reading directory ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A: Interrupted system call (os error 4)
Indexing: . 4586481 files, 287G ... -

is where it's getting stuck. I am, however, able to traverse the directory using find without seeing any error:

❯ find './Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A'
./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A
./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in
./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0
./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https
./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com
./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer
./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/86
./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/86/ba
./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/86/ba/86ba3be1-a8ca-429c-ae8a-625620d6d5e3
./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/86/ba/86ba3be1-a8ca-429c-ae8a-625620d6d5e3/russian_family_3.jpg

Am also able to stat it

❯ stat './Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A'
  File: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A
  Size: 96        	Blocks: 0          IO Block: 4096   directory
Device: 1,13	Inode: 6070627     Links: 3
Access: (0755/drwxr-xr-x)  Uid: (  501/ sankalp)   Gid: (   20/   staff)
Access: 2022-11-09 15:22:02.786003347 +0530
Modify: 2022-11-09 15:14:25.624805205 +0530
Change: 2022-11-09 15:14:25.624805205 +0530
 Birth: 2022-11-09 15:14:25.624485331 +0530

And most stunning is that I am able to run dust -V on the directory without facing any error:

❯ code/github.com/Amr-Shams/dust/target/release/dust !$ -V
code/github.com/Amr-Shams/dust/target/release/dust './Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A' -V
Scanning: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A
Scanning: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in
Scanning: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0
Scanning: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https
Scanning: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com
Scanning: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer
Scanning: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/86
Scanning: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/86/ba
Scanning: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/86/ba/86ba3be1-a8ca-429c-ae8a-625620d6d5e3
    File: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/th-thumbnailer.cdn-si-edu.com/ztcTnL0_Q_OJVCbNLuuU9Lc_r3A/fit-in/1072x0/https/tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/86/ba/86ba3be1-a8ca-429c-ae8a-625620d6d5e3/russian_family_3.jpg (size: 20480)
20K                   ┌── russian_family_3.jpg                │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ │ 100%
20K                 ┌─┴ 86ba3be1-a8ca-429c-ae8a-625620d6d5e3  │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ │ 100%
20K               ┌─┴ ba                                      │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ │ 100%
20K             ┌─┴ 86                                        │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ │ 100%
20K           ┌─┴ filer                                       │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ │ 100%
20K         ┌─┴ tf-cmsv2-smithsonianmag-media.s3.amazonaws.com│███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ │ 100%
20K       ┌─┴ https                                           │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ │ 100%
20K     ┌─┴ 1072x0                                            │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ │ 100%
20K   ┌─┴ fit-in                                              │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ │ 100%
20K ┌─┴ ztcTnL0_Q_OJVCbNLuuU9Lc_r3A                           │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ │ 100%

Moved up a few directories and am able to see the same error now on a different subpath:

Indexing: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18875 files, 2.3G ... -Error reading directory ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/images.newindianexpress.com/uploads/user/ckeditor_images/article/2017: Interrupted system call (os error 4)
Indexing: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18875 files, 2.3G ... |Error reading directory ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/media.distractify.com/brand-img/fqMhhITij/0x0: Interrupted system call (os error 4)
Error reading directory ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/medianama.com/wp-content: Interrupted system call (os error 4)
Error reading directory ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/www.jansatta.com/wp-content/uploads/2022/05: Interrupted system call (os error 4)
Error reading directory ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/ninjadoge24.github.io/assets: Interrupted system call (os error 4)
Indexing: ./Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18875 files, 2.3G ... /

[sankalp-khare]

This error is intermittent on the same directory. I don't have any Pocket process running => I don't think any changes are happening real-time to the directory or its contents. Yet repeated invocations of dust on the dir sometimes run into this error and sometimes don't.

[sankalp-khare]

When the "interrupted system call (os error 4)" error happens,

v1.1.1 prints it, moves past it and prints the expected final output:

❯ target/release/dust ~/Library/Containers/com.readitlater.PocketMac/Data/Library/Application\ Support/Pocket/offline/cache0/RIL_assets
Unknown Error: Interrupted system call (os error 4)
 16M     ┌── wp-content                     │██                                                                                                                                                                        │   1%
 16M   ┌─┴ canadatibet.com                  │██                                                                                                                                                                        │   1%
 16M   │   ┌── static                       │██                                                                                                                                                                        │   1%
 16M   │ ┌─┴ buzzfeed-static                │██
... (so on)

but v.1.2.0 gets stuck at Indexing:

❯ dust ~/Library/Containers/com.readitlater.PocketMac/Data/Library/Application\ Support/Pocket/offline/cache0/RIL_assets
Indexing: /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18883 files, 2.3G ... /

same thing with Amr-Shams@419e669 --verbose:

Indexing: /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18883 files, 2.3G ... \Error reading directory /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/static01.nyt.com/images/2019/05/17: Interrupted system call (os error 4)
Indexing: /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18883 files, 2.3G ... /

[sankalp-khare]

I added some print statements to trace the execution, and it looks to me like it gets stuck in handle_error_and_retry()

dust/src/dir_walker.rs

Line 290 in 702f0f0

fn handle_error_and_retry(failed: &Error, dir: &Path, walk_data: &WalkData) -> bool {

called at

dust/src/dir_walker.rs

Line 264 in 702f0f0

if handle_error_and_retry(&failed, &dir, walk_data) {

Indexing: /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18880 files, 2.3G ... \Error reading directory /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/media.wired.com/photos/6345b2232b2823c1af218031/169/w_80025252Ch_45025252Cc_limit: Interrupted system call (os error 4)
We've entered handle_error_and_retry
We've entered the Interrupted error handling block
Indexing: /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18880 files, 2.3G ... /Error reading directory /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/e4p7c9i3.stackpathcdn.com/wp-content/plugins/related-posts-thumbnails/assets: Interrupted system call (os error 4)
We've entered handle_error_and_retry
Error reading directory /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/www.livelaw.in/h-upload/2019: Interrupted system call (os error 4)
We've entered handle_error_and_retry
Error reading directory /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/static01.nyt.com/images/2020/05/07/science/00VIRUS-HISTORY1: Interrupted system call (os error 4)
We've entered handle_error_and_retry
Indexing: /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18880 files, 2.3G ... \

What's curious to me is that in the 2nd, 3rd, and 4th case of entering handle_error_and_retry, each of which happen after Error reading directory <dir>: Interrupted system call (os error 4), it does not enter the Interrupted error handling block... 🤔

It make sense then that we don't see the panic at

dust/src/dir_walker.rs

Line 310 in 702f0f0

panic!("Multiple Interrupted Errors occurred while scanning filesystem. Aborting");

because that block isn't executing more than 1x...

[sankalp-khare]

Using executable compiled from sankalp-khare@86861fb

I'm convinced that the walk_data.errors.lock().unwrap() call is getting stuck:

Indexing: /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18650 files, 2.2G ... -Error reading directory /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/media.wired.com/photos: Interrupted system call (os error 4)
> We've entered handle_error_and_retry
> We've made it past the walk_data.errors.lock().unwrap()
>> We've entered the Interrupted error handling block
Indexing: /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18650 files, 2.2G ... |Error reading directory /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets/res.cloudinary.com/dety84pbu/image/upload: Interrupted system call (os error 4)
> We've entered handle_error_and_retry
Indexing: /Users/sankalp/Library/Containers/com.readitlater.PocketMac/Data/Library/Application Support/Pocket/offline/cache0/RIL_assets 18650 files, 2.2G ... \

walk_data.errors.lock().unwrap() works fine the very first time, thereafter it seems like gets stuck.
I'm not at all familiar with the concurrency situation in the execution of this logic, but I think what's happening above, in sequence, is:

(there are two occurrences of walk_data.errors.lock().unwrap() in handle_error_and_retry(), I will call them unwrap1

dust/src/dir_walker.rs

Line 291 in 702f0f0

let mut editable_error = walk_data.errors.lock().unwrap();

and unwrap2

dust/src/dir_walker.rs

Line 307 in 702f0f0

let mut editable_error = walk_data.errors.lock().unwrap();

in order of their occurrence in the function)
(I use the word "thread" because I don't know what abstraction exists / what term is used in Rust for this)

1. Thread 1 enters handle_error_and_retry() and goes past unwrap1 (the first, and only successful walk_data.errors.lock().unwrap())
2. Thread 1 gets stuck at unwrap2, in the Interrupted error match case
3. Thread 2 enters handle_error_and_retry() and gets stuck at unwrap1

Hope this helps figure out the issue.

[bootandy]

coo, this is interesting, I'll look into this soon.

thanks for the investigation so far

[bootandy]

(there are two occurrences of walk_data.errors.lock().unwrap() in handle_error_and_retry(), I will call them unwrap1

Yeah that looks a bit sus