Errors when displaying logs and backup details

[abn-financial-com]

Hi,
I have installed velero (v1.9) and the local-volume-provider (0.3.3) in my k8s cluster. I am able to do backups and restores successfully. But I have some issues when I am checking logs or backup details. For example:
$ velero backup logs backup-41 An error occurred: Get "http://172.29.3.166:3000/nfs-snapshots/backups/backup-41/backup-41-logs.gz?expires=2022-07-27T15%3A22%3A58.124Z&signature=KCvo8QFpUqtp_QlIbQZB03_dbLA=": dial tcp 172.29.3.166:3000: connect: connection refused

$ velero backup describe backup-41 --details ... Resource List: <error getting backup resource list: Get "http://172.29.3.166:3000/nfs-snapshots/backups/backup-41/backup-41-resource-list.json.gz?expires=2022-07-27T15%3A24%3A39.114Z&signature=JyObtPAybGAXCH1_uJSf8XVMa48=": dial tcp 172.29.3.166:3000: connect: connection refused>

The URLs mentioned in the error message work from a pod within the k8s cluster but not from the velero client. The error is raised by the local-volume-provider sidecar.

Did someone already encounter the same issue ?
Let me know if you need more input.

Thank you.

[EugenMayer]

Running into the same issue. Did you ever resolve this one @abn-financial-com

The IP that is referenced in the error log is the velero pod ip, so i assume that the local-volume-provider sidecar does host the (its the fileserver https://github.com/replicatedhq/local-volume-provider/blob/main/cmd/local-volume-fileserver/main.go) app to offer those logs ass you mentioned

• we might be able to expose that port using end endpoint + service + ingress (that be doable)
• Not sure how to tell velero to properly generate the URL using a different ip / host

[EugenMayer]

Would assume that publicUrl would be what we need to set ( e.g. by https://velero.io/docs/main/contributions/minio/#accessing-logs-with-an-https-endpoint ) but at least for me, that does not work. Still, the URL addressed the internal pod ip

[EugenMayer]

Looking at the code i found

local-volume-provider/pkg/plugin/plugin.go

Line 254 in 978608c

signedUrl := url.URL{

, so basically by setting POD_IP on the sidecar pod, we can change the host.

Problems are

• POD_IP is also used to to bind the server at

  local-volume-provider/pkg/plugin/kubernetes.go

  Line 167 in 978608c

  if !containerHasEnvVar(veleroContainer, "POD_IP") {

  so overriding the POD_IP will make this an issue (deal breaker)
• the schema will be forced to http

Currently, the only way is, patching the code, introducing the ENV vars, defaulting to POD_IP for backwards compat.

Though, considering the activity in the repo, i doubt anything is going to get merged ;/

[EugenMayer]

I could create an ingress and test it, but i get a 400 when i use the path/params of the intenal URL and use it with the domain. Response is 400, which matches

local-volume-provider/cmd/local-volume-fileserver/main.go

Line 48 in 978608c

if !valid {

means that my signature is invalid.

Looking at the sign code it becomes clear, that the domain part is part of the signature, thus we would need to match the code to generate it for the ingress domain to begin with

[EugenMayer]

Implemented / fixed via #84

[kaovilai]

@EugenMayer FYI velero plans for fixing access to these long term is in vmware-tanzu/velero#6167

[EugenMayer]

@kaovilai considering the value of log/describe/--detail and how long the issue is waiting for any movement on the velero side, i would rather solve this locally in local-volume-provider for now, and then remove it when (and if) the other solution hit's the deck.

in addition to that

• it was not a huge effort to implement it here. It can be easily removed again
• the feature is purely opt in
• since we are using k8s ingress, any other additional options can be used to secure access in addition to the signature like IP based access on the ingress. But i do not see a huge value to secure it tighter, the signature seems fairly secure already.