Issue to run jaeger-collector with ES.

[kleysonr]

I'm trying to run a new jaeger-collector using the latest docker image.

I've started a new ES instance

docker run --name elasticsearch -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.12.1

$ curl -X GET "http://10.XXX.XX.102:9200/"
{
  "name" : "d170e18717b5",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "8b88CHteQcqWKHgC7w618A",
  "version" : {
    "number" : "7.12.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "3186837139b9c6b6d23c3200870651f10d3343b7",
    "build_date" : "2021-04-20T20:56:39.040728659Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

After, I'm trying to start a new jaeger-collector:

docker run --rm -it \
       -e SPAN_STORAGE_TYPE=elasticsearch \
       -e ES_SERVER_URLS=http://10.XXX.XX.102:9200 \
       -e ES_USERNAME=elastic \
       -e LOG_LEVEL=debug \
       -p 14250:14250 -p 14269:14269 \
       jaegertracing/jaeger-collector:1.22

But getting the following error to start the collector:

2021/05/20 12:52:23 maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
{"level":"info","ts":1621515143.6990724,"caller":"flags/service.go:117","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1621515143.6991575,"caller":"flags/service.go:123","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
{"level":"info","ts":1621515143.7001376,"caller":"flags/admin.go:105","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1621515143.7002406,"caller":"flags/admin.go:111","msg":"Starting admin HTTP server","http-addr":":14269"}
{"level":"info","ts":1621515143.700584,"caller":"flags/admin.go:97","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
{"level":"fatal","ts":1621515149.8248925,"caller":"command-line-arguments/main.go:75","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: Head \"http://10.XXX.XX.102:9200\": dial tcp 10.XXX.XX.102:9200: connect: no route to host: no Elasticsearch node available","stacktrace":"main.main.func1\n\tcommand-line-arguments/main.go:75\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/[email protected]/command.go:838\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/[email protected]/command.go:943\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/[email protected]/command.go:883\nmain.main\n\tcommand-line-arguments/main.go:137\nruntime.main\n\truntime/proc.go:204"}

Am I missing something ? How to get the collector running ?

[albertteoh]

This looks like a container networking problem.

jaeger-collector is running in its own container and doesn't know how to talk to elasticsearch running in another container. You'll need to put them on the same network, either explicitly through a docker --network flag or docker-compose can do it for you implicitly.

With the former approach using docker, please try:

$ docker network create mynet
$ docker run --name elasticsearch --network=mynet -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.12.1
$ docker run --rm -it \
       -e SPAN_STORAGE_TYPE=elasticsearch \
       -e ES_SERVER_URLS=http://elasticsearch:9200 \
       -e ES_USERNAME=elastic \
       -e LOG_LEVEL=debug \
       -p 14250:14250 -p 14269:14269 \
       --network=mynet \
       jaegertracing/jaeger-collector:1.22

[kleysonr]

@albertteoh creating the docker network it worked.

But that's makes no sense. What if the elastic is running in another server ? Or maybe even not running in a docker container ?

How can I get the collector working if it is not able to connect in the elastic in a non-docker network ?

[kleysonr]

@albertteoh Should I open an issue for this ?

[yurishkuro]

There is no issue from Jaeger side. Docker containers run in their own networking namespace making them invisible to other processes unless you take special steps to overcome that. If you run ES and Jaeger directly on bare bones server, you will be able to connect them just fine.