only one sccache instance connecting a single REDIS backend can work correctly

[diablohsh]

I test SSCACHE with two identical CMAKE clang++ project in two different Docker Container connecting to a single REDIS instance. If i execute the two CMAKE simultaneously, one of them will exit with error. What did i do wrong?

[Xuanwo]

Can you share the detailed error message?

[Xuanwo]

You can start a instance with SCCACHE_LOG=debug SCCACHE_START_SERVER=1 SCCACHE_NO_DAEMON=1 sccache which will print log to console directly.

[diablohsh]

stvhuang@stvhuang-PC6:~/test$ SCCACHE_LOG=debug SCCACHE_START_SERVER=1 SCCACHE_NO_DAEMON=1 SCCACHE_REDIS=redis://:[email protected]:6380 ./sccache 
[2023-03-22T04:51:45Z DEBUG sccache::config] Attempting to read config file at "/home/stvhuang/.config/sccache/config"
[2023-03-22T04:51:45Z DEBUG sccache::config] Couldn't open config file: failed to open file `/home/stvhuang/.config/sccache/config`
[2023-03-22T04:51:45Z INFO  sccache::server] start_server: port: 4226
[2023-03-22T04:51:45Z INFO  sccache::server] No scheduler address configured, disabling distributed sccache
[2023-03-22T04:51:45Z DEBUG sccache::cache::cache] Init redis cache with url redis://:[email protected]:6380
[2023-03-22T04:51:45Z DEBUG opendal::services] service=redis operation=read path=.sccache_check range=0- -> started
[2023-03-22T04:51:45Z DEBUG opendal::services] service=redis operation=read path=.sccache_check range=0- -> got reader
[2023-03-22T04:51:45Z DEBUG opendal::services] service=redis operation=read path=.sccache_check has_read=13 -> dropped reader
[2023-03-22T04:51:45Z DEBUG opendal::services] service=redis operation=write path=.sccache_check size=13 -> started
[2023-03-22T04:51:45Z DEBUG opendal::services] service=redis operation=write path=.sccache_check has_read=13 -> consumed reader fully
[2023-03-22T04:51:45Z DEBUG opendal::services] service=redis operation=write path=.sccache_check size=13 -> written
[2023-03-22T04:51:45Z INFO  sccache::server] server has setup with ReadWrite
[2023-03-22T04:51:45Z INFO  sccache::server] server started, listening on port 4226
[2023-03-22T04:51:50Z DEBUG sccache::server] handle_client: compile
[2023-03-22T04:51:50Z DEBUG sccache::compiler::compiler] Found clang++
[2023-03-22T04:51:50Z DEBUG sccache::server] check_compiler: Supported compiler
[2023-03-22T04:51:50Z DEBUG sccache::server] parse_arguments: Ok: ["-g", "-MD", "-MT", "CMakeFiles/test.dir/test.cc.o", "-MF", "CMakeFiles/test.dir/test.cc.o.d", "-o", "CMakeFiles/test.dir/test.cc.o", "-c", "/home/stvhuang/test/test.cc"]
[2023-03-22T04:51:50Z DEBUG sccache::compiler::compiler] [test.cc.o]: get_cached_or_compile: ["-g", "-MD", "-MT", "CMakeFiles/test.dir/test.cc.o", "-MF", "CMakeFiles/test.dir/test.cc.o.d", "-o", "CMakeFiles/test.dir/test.cc.o", "-c", "/home/stvhuang/test/test.cc"]
[2023-03-22T04:51:50Z DEBUG sccache::compiler::gcc] arch args before rewrite: []
[2023-03-22T04:51:50Z DEBUG sccache::compiler::gcc] cmd after -arch rewrite: Some("/usr/bin/clang++" "-x" "c++" "-E" "-P" "/home/stvhuang/test/test.cc" "-MD" "-MF" "CMakeFiles/test.dir/test.cc.o.d" "-MT" "CMakeFiles/test.dir/test.cc.o" "-g")
[2023-03-22T04:51:50Z DEBUG sccache::compiler::compiler] [test.cc.o]: generate_hash_key took 0.033 s
[2023-03-22T04:51:50Z DEBUG opendal::services] service=redis operation=read path=4/e/1/4e185bc971a30fd7257301411a9ace354c459e947813265e801bb3ff789063ac range=0- -> started
[2023-03-22T04:51:50Z DEBUG opendal::services] service=redis operation=read path=4/e/1/4e185bc971a30fd7257301411a9ace354c459e947813265e801bb3ff789063ac range=0- -> got reader
[2023-03-22T04:51:50Z DEBUG opendal::services] service=redis operation=read path=4/e/1/4e185bc971a30fd7257301411a9ace354c459e947813265e801bb3ff789063ac has_read=939 -> dropped reader
[2023-03-22T04:51:50Z DEBUG sccache::compiler::compiler] [test.cc.o]: Cache hit in 0.000 s
[2023-03-22T04:51:50Z DEBUG sccache::server] compile result: cache hit

I think i found the answer, the key observation is that two different WSL in same host will communicate with each the other, i can do --stop-server in WSL instance 2 to stop the daemon in WSL instance 1. The whole thing happened as below:

1. in WSL instance 1, after a successful build, the SCCACHE daemon 1 is started
2. in WSL instance 2 on the same computer, the second build found that there is a SCCACHE daemon running, it communicate with the daemon and ask for the compiled file.
3. the daemon downloaded the compiled file from REDIS and stored the file in WSL instance 1. the WSL instance 2 got nothing.

[Xuanwo]

Great investigation! However, it seems to be a WSL issue within the same host that cannot be resolved by sccache. One possible solution is to deploy sccache-dist.

[diablohsh]

is it possible to add a "client only mode" to sccache without a running deamon in the future?

[Xuanwo]

No such plan for now.