Source code not found warning after upgrading to rebar 3.14.3 #2494

oleglite · 2021-02-08T13:47:50Z

oleglite
Feb 8, 2021

I am trying to upgrade to the latest version of rebar3, but I am getting following messages during release:

===> Warnings generating release:
*WARNING* prometheus: Source code not found: prometheus_boolean.erl
*WARNING* prometheus: Source code not found: prometheus_counter.erl
*WARNING* prometheus: Source code not found: prometheus_gauge.erl

I found out that this warning comes from https://github.com/erlang/otp/blob/OTP-21.3/lib/sasl/src/systools_make.erl#L1151
And may be caused be prometheus lib structure:
https://github.com/deadtrickster/prometheus.erl/tree/v4.6.0/src

$ tree lib/prometheus/src/
lib/prometheus/src/
|-- collectors
|   |-- mnesia
|   |   `-- prometheus_mnesia_collector.erl
|   `-- vm
|       |-- prometheus_vm_dist_collector.erl
|       |-- prometheus_vm_memory_collector.erl
|       |-- prometheus_vm_statistics_collector.erl
|       `-- prometheus_vm_system_info_collector.erl
|-- contrib
|   |-- prometheus_http.erl
|   |-- prometheus_mnesia.erl
|   `-- prometheus_test_instrumenter.erl
|-- formats
|   |-- prometheus_protobuf_format.erl
|   `-- prometheus_text_format.erl
|-- metrics
|   |-- prometheus_boolean.erl
|   |-- prometheus_counter.erl
|   |-- prometheus_gauge.erl
|   |-- prometheus_histogram.erl
|   `-- prometheus_summary.erl
|-- model
|   |-- prometheus_model.erl
|   |-- prometheus_model.proto
|   `-- prometheus_model_helpers.erl
|-- prometheus.app.src
|-- prometheus.erl
|-- prometheus_buckets.erl
|-- prometheus_collector.erl
|-- prometheus_format.erl
|-- prometheus_instrumenter.erl
|-- prometheus_metric.erl
|-- prometheus_metric_spec.erl
|-- prometheus_misc.erl
|-- prometheus_registry.erl
|-- prometheus_sup.erl
`-- prometheus_time.erl

My question is it a problem with prometheus lib, with my rebar config or something else? How can I fix this warning?

I reproduced it with bare umbrella project:

$ ./rebar3 --version
rebar 3.14.3 on Erlang/OTP 21 Erts 10.3
$ ./rebar3 new umbrella myproj
===> Writing myproj/apps/myproj/src/myproj_app.erl
===> Writing myproj/apps/myproj/src/myproj_sup.erl
===> Writing myproj/apps/myproj/src/myproj.app.src
===> Writing myproj/rebar.config
===> Writing myproj/config/sys.config
===> Writing myproj/config/vm.args
===> Writing myproj/.gitignore
===> Writing myproj/LICENSE
===> Writing myproj/README.md
$ cd myproj/
$ cat myproj/rebar.config
{erl_opts, [debug_info]}.
{deps, [
     {prometheus, ".*",
        {git, "https://github.com/deadtrickster/prometheus.erl.git", {tag, "v4.6.0"}}}
]}.

{relx, [{release, {myproj, "0.1.0"},
         [myproj,
          sasl,
          prometheus]},

        {mode, dev},

        %% automatically picked up if the files
        %% exist but can be set manually, which
        %% is required if the names aren't exactly
        %% sys.config and vm.args
        {sys_config, "./config/sys.config"},
        {vm_args, "./config/vm.args"}

        %% the .src form of the configuration files do
        %% not require setting RELX_REPLACE_OS_VARS
        %% {sys_config_src, "./config/sys.config.src"},
        %% {vm_args_src, "./config/vm.args.src"}
]}.

{profiles, [{prod, [{relx,
                     [%% prod is the default mode when prod
                      %% profile is used, so does not have
                      %% to be explicitly included like this
                      {mode, prod}

                      %% use minimal mode to exclude ERTS
                      %% {mode, minimal}
                     ]
            }]}]}.
$ ../rebar3 compile
===> Verifying dependencies...
===> Fetching prometheus (from {git,"https://github.com/deadtrickster/prometheus.erl.git",
                      {tag,"v4.4.0"}})
===> Fetching rebar3_archive_plugin v0.0.1
===> Fetching rebar3_elvis_plugin (from {git,"https://github.com/deadtrickster/rebar3_elvis_plugin.git",
                               "master"})
===> WARNING: It is recommended to use {branch, Name}, {tag, Tag} or {ref, Ref}, otherwise updating the dep may not work as expected.
===> Fetching katana_code v0.1.0
===> Fetching zipper v1.0.1
===> Fetching aleppo v1.0.0
===> Analyzing applications...
===> Compiling aleppo
_build/default/plugins/aleppo/src/aleppo.erl:6: Warning: record ale_context has field(s) without type information
_build/default/plugins/aleppo/src/aleppo.erl:12: Warning: missing specification for function process_file/1
_build/default/plugins/aleppo/src/aleppo.erl:15: Warning: missing specification for function process_file/2
_build/default/plugins/aleppo/src/aleppo.erl:25: Warning: missing specification for function process_tokens/1
_build/default/plugins/aleppo/src/aleppo.erl:32: Warning: missing specification for function process_tokens/2
_build/default/plugins/aleppo/src/aleppo.erl:307: Warning: missing specification for function scan_file/1

===> Compiling katana_code
===> Compiling rebar3_elvis_plugin
===> Compiling zipper
===> Analyzing applications...
===> Compiling rebar3_archive_plugin
===> Analyzing applications...
===> Compiling prometheus
===> Analyzing applications...
===> Compiling myproj
$ ../rebar3 release
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling myproj
===> Assembling release myproj-0.1.0...
===> Warnings generating release:
*WARNING* prometheus: Source code not found: prometheus_boolean.erl
*WARNING* prometheus: Source code not found: prometheus_counter.erl
*WARNING* prometheus: Source code not found: prometheus_gauge.erl
*WARNING* prometheus: Source code not found: prometheus_histogram.erl
*WARNING* prometheus: Source code not found: prometheus_http.erl
*WARNING* prometheus: Source code not found: prometheus_mnesia.erl
*WARNING* prometheus: Source code not found: prometheus_mnesia_collector.erl
*WARNING* prometheus: Source code not found: prometheus_model.erl
*WARNING* prometheus: Source code not found: prometheus_model_helpers.erl
*WARNING* prometheus: Source code not found: prometheus_protobuf_format.erl
*WARNING* prometheus: Source code not found: prometheus_summary.erl
*WARNING* prometheus: Source code not found: prometheus_test_instrumenter.erl
*WARNING* prometheus: Source code not found: prometheus_text_format.erl
*WARNING* prometheus: Source code not found: prometheus_vm_dist_collector.erl
*WARNING* prometheus: Source code not found: prometheus_vm_memory_collector.erl
*WARNING* prometheus: Source code not found: prometheus_vm_statistics_collector.erl
*WARNING* prometheus: Source code not found: prometheus_vm_system_info_collector.erl
===> Release successfully assembled: _build/default/rel/myproj

ferd · 2021-02-08T17:01:37Z

ferd
Feb 8, 2021
Maintainer

I guess part of the issue is that the beam files are flat in ebin/ but the source files are not searched recursively by systools whereas rebar3 will do so.

2 replies

oleglite Feb 15, 2021
Author

Yes, beam files are flat in /ebin and source files are not. It looks like there are two solutions:

The structure of source files in libs like prometheus must be changed to flat
systools must support recursive search

Also I reproduced it with the latest erlang 23.2.

ferd Feb 15, 2021
Maintainer

23.2 has SSL issues I believe, so you might want to update.

The systools recursive search punts this problem into OTP rather than rebar3, if it is the component that complains. the only thing rebar3 could do is flatten the source directory on behalf of users before compiling, but that would probably be annoyingly slow.

bryanhughes · 2021-02-25T22:22:36Z

bryanhughes
Feb 25, 2021

I am having a similar issue and I am not sure if there is a solution being presented that I am being too dense and missing. I am working on an opensource tool that will allow starting from a database schema, and then generate protobuffers and all the CRUDL (Create, Read, Update, Delete, List/Lookup) code. The tool then ties in with gpb to compile the protos that are generated from the schema to Erlang source codec which in turn is used by the generated code. The end result is that you run the tool against your database and you then have all the code and protos generated to seamlessly round trip from protobuffers across the wire to and from your database just using the maps/protos and back, and across your stack.

In my case, the tool repo was generated with rebar3 and is an escript. It includes an example directory with an example application also generated by rebar3 to build a working HTTP example demonstrating round tripping a proto across the wire to the database and back.

Everything compiles but does not run. My issue is a little more extensive:

example git:(main) ✗ rebar3 run
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling example
===> Assembling release example-0.1.0...
===> Warnings generating release:
*WARNING* example: Source code not found: example_app.erl
*WARNING* example: Source code not found: example_http.erl
*WARNING* example: Source code not found: example_sup.erl
*WARNING* example: Source code not found: example_utils.erl
*WARNING* example: Source code not found: public_example_a_db.erl
*WARNING* example: Source code not found: public_example_b_db.erl
*WARNING* example: Source code not found: public_example_c_db.erl
*WARNING* example: Source code not found: public_part_db.erl
*WARNING* example: Source code not found: public_part_part_db.erl
*WARNING* example: Source code not found: public_product_db.erl
*WARNING* example: Source code not found: public_product_parts_db.erl
*WARNING* example: Source code not found: test_schema_address_db.erl
*WARNING* example: Source code not found: test_schema_test_table_no_pkey_db.erl
*WARNING* example: Source code not found: test_schema_test_table_pkey_db.erl
*WARNING* example: Source code not found: test_schema_user_db.erl
*WARNING* example: Source code not found: test_schema_user_product_part_db.erl
*WARNING* stdlib: Object code (filename) out of date
*WARNING* inets: Source code not found: inets.erl
*WARNING* inets: Source code not found: inets_sup.erl
*WARNING* inets: Source code not found: inets_app.erl
*WARNING* inets: Source code not found: inets_service.erl
*WARNING* inets: Source code not found: inets_trace.erl
*WARNING* inets: Source code not found: inets_lib.erl
*WARNING* inets: Source code not found: inets_ftp_wrapper.erl
*WARNING* inets: Source code not found: httpc.erl
*WARNING* inets: Source code not found: httpc_handler.erl
*WARNING* inets: Source code not found: httpc_handler_sup.erl
*WARNING* inets: Source code not found: httpc_manager.erl
*WARNING* inets: Source code not found: httpc_profile_sup.erl
*WARNING* inets: Source code not found: httpc_request.erl
*WARNING* inets: Source code not found: httpc_response.erl
*WARNING* inets: Source code not found: httpc_sup.erl
*WARNING* inets: Source code not found: httpc_cookie.erl
*WARNING* inets: Source code not found: http_chunk.erl
*WARNING* inets: Source code not found: http_request.erl
*WARNING* inets: Source code not found: http_response.erl
*WARNING* inets: Source code not found: http_transport.erl
*WARNING* inets: Source code not found: http_util.erl
*WARNING* inets: Source code not found: http_uri.erl
*WARNING* inets: Source code not found: httpd.erl
*WARNING* inets: Source code not found: httpd_acceptor.erl
*WARNING* inets: Source code not found: httpd_acceptor_sup.erl
*WARNING* inets: Source code not found: httpd_cgi.erl
*WARNING* inets: Source code not found: httpd_connection_sup.erl
*WARNING* inets: Source code not found: httpd_conf.erl
*WARNING* inets: Source code not found: httpd_custom.erl
*WARNING* inets: Source code not found: httpd_custom_api.erl
*WARNING* inets: Source code not found: httpd_esi.erl
*WARNING* inets: Source code not found: httpd_example.erl
*WARNING* inets: Source code not found: httpd_file.erl
*WARNING* inets: Source code not found: httpd_instance_sup.erl
*WARNING* inets: Source code not found: httpd_log.erl
*WARNING* inets: Source code not found: httpd_logger.erl
*WARNING* inets: Source code not found: httpd_manager.erl
*WARNING* inets: Source code not found: httpd_misc_sup.erl
*WARNING* inets: Source code not found: httpd_request.erl
*WARNING* inets: Source code not found: httpd_request_handler.erl
*WARNING* inets: Source code not found: httpd_response.erl
*WARNING* inets: Source code not found: httpd_script_env.erl
*WARNING* inets: Source code not found: httpd_socket.erl
*WARNING* inets: Source code not found: httpd_sup.erl
*WARNING* inets: Source code not found: httpd_util.erl
*WARNING* inets: Source code not found: mod_actions.erl
*WARNING* inets: Source code not found: mod_alias.erl
*WARNING* inets: Source code not found: mod_auth.erl
*WARNING* inets: Source code not found: mod_auth_dets.erl
*WARNING* inets: Source code not found: mod_auth_mnesia.erl
*WARNING* inets: Source code not found: mod_auth_plain.erl
*WARNING* inets: Source code not found: mod_auth_server.erl
*WARNING* inets: Source code not found: mod_cgi.erl
*WARNING* inets: Source code not found: mod_dir.erl
*WARNING* inets: Source code not found: mod_disk_log.erl
*WARNING* inets: Source code not found: mod_esi.erl
*WARNING* inets: Source code not found: mod_get.erl
*WARNING* inets: Source code not found: mod_head.erl
*WARNING* inets: Source code not found: mod_log.erl
*WARNING* inets: Source code not found: mod_range.erl
*WARNING* inets: Source code not found: mod_responsecontrol.erl
*WARNING* inets: Source code not found: mod_security.erl
*WARNING* inets: Source code not found: mod_security_server.erl
*WARNING* inets: Source code not found: mod_trace.erl
*WARNING* inets: Source code not found: inets_tftp_wrapper.erl
===> Release successfully assembled: _build/default/rel/example
Error spawning "/usr/local/Cellar/erlang/23.2.3/lib/erlang/erts-11.1.7/bin/epmd" -daemon (error 10)

Note, the same is true for rebar3 release, just doesnt try to spawn epmd.

Here is my rebar3.config

{erl_opts, [debug_info,
            {i, "./_build/default/plugins/gpb/include/"},
            {i, "apps/example/include"}]}.
{deps, [{elli, {git, "https://github.com/elli-lib/elli.git", {branch, "develop"}}},
        elli_cookie,
        elli_date,
        mimerl,
        {pgo, {git, "https://github.com/bryanhughes/pgo.git", {branch, "master"}}}]}.

{plugins, [{rebar3_gpb_plugin, "2.19.0"}, {rebar3_autotest, "0.1.1"}, rebar3_run, rebar_alias, rebar3_hex]}.

{gpb_opts, [{i, "proto"},
            {o_erl, "src"},
            {o_hrl, "include"},
            use_packages,
            {module_name_suffix, "_pb"},
            {strings_as_binaries, true},
            {recursive, true},
            maps,
            type_specs]}.

{provider_hooks, [
    {pre, [
        {compile, {protobuf, compile}},
        {clean, {protobuf, clean}}
    ]}
]}.

{relx, [{release, {example, "0.1.0"}, [example, sasl]},
        {mode, dev},

        %% automatically picked up if the files
        %% exist but can be set manually, which
        %% is required if the names aren't exactly
        %% sys.config and vm.args
        {sys_config, "./config/sys.config"},
        {vm_args, "./config/vm.args"}

        %% the .src form of the configuration files do
        %% not require setting RELX_REPLACE_OS_VARS
        %% {sys_config_src, "./config/sys.config.src"},
        %% {vm_args_src, "./config/vm.args.src"}
]}.

{shell, []}.
{dist_node, [{name, '[email protected]'}]}.

{profiles, [
    {prod, [
        {include_erts, true},
        {erl_opts, [no_debug_info, warnings_as_errors]},
        {relx, [{mode, prod}]}
    ]},
    {test, [
        {deps, [meck]},
        {erl_opts, [debug_info]}
    ]}
]}.

When I investigate the _build/default/lib/example/ebin all the beams are there, but not all the sources are in _build/default/lib/example/src. In any case, the WARNINGS list pretty much all the sources in the project, including those from one of the dependencies.

I am using

Erlang/OTP 23 [erts-11.1.7] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe] [dtrace]

and

rebar 3.14.3 on Erlang/OTP 23 Erts 11.1.7

The other thing that stands out is *WARNING* stdlib: Object code (filename) out of date

Here is the debug output...
example-debug.txt

Any help is much appreciate.

Thanks!
Bryan

1 reply

bryanhughes Feb 26, 2021

I have been futzing around with trying to see if I can make the release and found that if I change {mode, dev} to {mode, prod} then it gets rid of all the warnings.

        {mode, prod},

        %% automatically picked up if the files
        %% exist but can be set manually, which
        %% is required if the names aren't exactly
        %% sys.config and vm.args
        {sys_config, "./config/sys.config"},
        {vm_args, "./config/vm.args"}

        %% the .src form of the configuration files do
        %% not require setting RELX_REPLACE_OS_VARS
        %% {sys_config_src, "./config/sys.config.src"},
        %% {vm_args_src, "./config/vm.args.src"}
]}.```

Now I just have to figure out why I am getting...

➜ example git:(main) ✗ rebar3 run
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling example
===> Assembling release example-0.1.0...
===> Release successfully assembled: _build/default/rel/example
Error spawning "/usr/local/Cellar/erlang/23.2.3/lib/erlang/erts-11.1.7/bin/epmd" -daemon (error 10)

ferd · 2021-02-26T00:30:06Z

ferd
Feb 26, 2021
Maintainer

we don't have anything to offer at this point -- none of the maintainers recently had time to solve this one yet. Thanks for the additional information though. No idea yet when I'll have the time to take a look, unfortunately, but the extra data will likely be useful.

2 replies

bryanhughes Mar 1, 2021

Bummer. Just wanted to give some more information. I have created a second new umbrella project and ran into the very same situation, but I figured out how to run rebar3 release without the warnings, and thus broken application because it fails on stdlib.

To work around, add {dev_mode, false}, to your relx config. if it is set to true, then switch it to false:

{relx, [{release, {traphic, "0.1.0"}, [traphic, sasl]},
        {mode, prod},

        %% automatically picked up if the files
        %% exist but can be set manually, which
        %% is required if the names aren't exactly
        %% sys.config and vm.args
        {sys_config, "config/sys.config"},
        {vm_args, "config/vm.args"},

        {dev_mode, false},   <--------
        {include_erts, false},

        {extended_start_script, true}
]}.

Hope this helps...

ferd Mar 1, 2021
Maintainer

Might have something to do with symlinks then, but that's a good pointer to solve it, thanks for the info

reganheath · 2023-03-27T11:08:22Z

reganheath
Mar 27, 2023

I have a similar (the same?) issue.

I have just upgraded from rebar3 v3.9.1 to rebar3 v3.15.2 (for the hex.pm TLS fix). With no other changes to OTP version (21). rebar.config or the relx.config (we use a separate file and run rebar3 release -c relx.conf) we now get these same "Source code not found" warnings.

The files it is complaining about all all external deps and the src tree/folder structure for these has not changed (they're pinned).

So.. it feels like something has changed in the defaults or behaviour of rebar3/relx to trigger these warnings?

The relx.config is minimal (replaced app names with X, Y, and Z), e.g.

{default_release, {X, "0.3.4"}}.

{release, {X, "0.3.4"}, ["X>=0.3.4",
                            "Y>=0.3.4",
                            "Z>=0.1.0",
                            sasl,
                            redbug,
                            recon,
                            tools,
                            wx,
                            observer,
                            debugger,
                            runtime_tools,
                            debugger]}.

{sys_config, "conf/sys.config"}.
{vm_args, "conf/vm.args"}.
{extended_start_script, true}.

As you can see, it has no explicit "mode" specified, nor any of the things that mode is expanded to by relx (include_src, debug_info, include_erts, dev_mode).

Inspecting our previous releases, they appears to be the result of "{mode, dev}" or equivalent.

In addition to these new warnings, the relx.config above is no longer valid.

default_release - no longer supported? (but still documented here https://github.com/erlware/relx/wiki/Configuration)

release tuple, applications list, format of applications WRT required version changed from "X>=0.3.4" to {X, "0.3.4"} (and I cannot see whether >= is still supported in some form?)

1 reply

tsloughter Mar 27, 2023
Maintainer

rebar3 upgraded to relx 4.x and this caused the changes you are experiencing.

Adding {src_tests, false} should get rid of the warnings about source code. It is also only done if include_src is true which happens with the new default of mode being dev.

The constraints like X>=0.3.4 were removed because they didn't do anything, rebar3 is in charge of what app versions are in the release. Were you seeing otherwise?

I don't recall why default_release was removed.. Thanks for calling out it is still in the wiki.

DominicGame · 2023-04-04T06:28:34Z

DominicGame
Apr 4, 2023

I have some test result about it
env:

win10
rebar 3.20.0 on Erlang/OTP 24 Erts 12.3.2

flow:

rebar3 new release test
cd test
rebar3 release as prod
rebar3 tar as prod
(move tar to some where and decompression)
./bin/test console

default config:

{undef,[{test_app,.....

when I remove otp(right?), it can work:

{relx, [{release, {test, "0.1.0"},
         [test,
          sasl]},

        {mode, dev},
        {system_libs, false},
        {include_erts, false},
        {include_src, true},

        %% automatically picked up if the files
        %% exist but can be set manually, which
        %% is required if the names aren't exactly
        %% sys.config and vm.args
        {sys_config, "./config/sys.config"},
        {vm_args, "./config/vm.args"}

        %% the .src form of the configuration files do
        %% not require setting RELX_REPLACE_OS_VARS
        %% {sys_config_src, "./config/sys.config.src"},
        %% {vm_args_src, "./config/vm.args.src"}
]}.

{profiles, [{prod, [{relx,
                     [%% prod is the default mode when prod
                      %% profile is used, so does not have
                      %% to be explicitly included like this
                      %% use minimal mode to exclude ERTS
                      {mode, minimal}
                     ]
            }]}]}.

release warning:

*WARNING* test: Source code not found: test_app.erlrelease start script created*WARNING* Not creating RELEASES file because system libs are not included in the release.

tar warning:

*WARNING* Not creating RELEASES file because system libs are not included in the release. The RELEASES file is required before release_handler can be used to install a release.

it can work with recursive dir in src

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source code not found warning after upgrading to rebar 3.14.3 #2494

{{title}}

Replies: 5 comments 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Source code not found warning after upgrading to rebar 3.14.3 #2494

oleglite Feb 8, 2021

Replies: 5 comments · 6 replies

ferd Feb 8, 2021 Maintainer

oleglite Feb 15, 2021 Author

ferd Feb 15, 2021 Maintainer

bryanhughes Feb 25, 2021

bryanhughes Feb 26, 2021

ferd Feb 26, 2021 Maintainer

bryanhughes Mar 1, 2021

ferd Mar 1, 2021 Maintainer

reganheath Mar 27, 2023

tsloughter Mar 27, 2023 Maintainer

DominicGame Apr 4, 2023

oleglite
Feb 8, 2021

Replies: 5 comments 6 replies

ferd
Feb 8, 2021
Maintainer

oleglite Feb 15, 2021
Author

ferd Feb 15, 2021
Maintainer

bryanhughes
Feb 25, 2021

ferd
Feb 26, 2021
Maintainer

ferd Mar 1, 2021
Maintainer

reganheath
Mar 27, 2023

tsloughter Mar 27, 2023
Maintainer

DominicGame
Apr 4, 2023