curl handles csv upload, but database returns error

[RumRogerz]

Versions:
be: 4.1.0
fe: 4.1.1

my curl command:

        -H 'x-access-token: '${JTL_REPORTS_API_KEY} \
        -F 'kpi=@"/jtl-reports/logresults-pass.jtl"' \
        -F 'environment="'my-fancy-environment"'
}

once executed, it posts a result:

{"itemId":"6bcbdf18-d62b-4f44-a695-654e4ba9d114"}

however, if i go to see the database logs, I see this error:

2022-06-15 15:58:14.316 UTC [21293] ERROR:  column s.delete_samples does not exist at character 170
2022-06-15 15:58:14.316 UTC [21293] STATEMENT:  SELECT s.threshold_error_rate as "errorRate", s.threshold_percentile as "percentile", s.threshold_throughput as "throughput", s.threshold_enabled as "thresholdEnabled", s.delete_samples as "deleteSamples"  FROM jtl.scenario as s
	    LEFT JOIN jtl.projects p ON p.id = s.project_id
	    WHERE p.project_name = $1
	    AND s.name = $2

backend logs gives out a similar error:

{"level":"info","message":"Parsed 37247 records in 6.21 seconds"}
error: column s.delete_samples does not exist
    at Parser.parseErrorMessage (/src/be/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/src/be/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/src/be/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/src/be/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (events.js:375:28)
    at addChunk (internal/streams/readable.js:290:12)
    at readableAddChunk (internal/streams/readable.js:265:9)
    at Socket.Readable.push (internal/streams/readable.js:204:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23) {
  length: 115,
  severity: 'ERROR',
  code: '42703',
  detail: undefined,
  hint: undefined,
  position: '170',
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'parse_relation.c',
  line: '3504',
  routine: 'errorMissingColumn'
}

Any help would be appreciated

I would have used your newer versions, however I noticed that if I use anything above these versions, you have a 3rd option for users (roles) which is not in the schema.db file in your repository. I'm not exactly a database guru and I have little to no idea on what to do. Essentially what happens is, if i start up all the containers and try to make a new user, it just laughs in my face with an error. Logs were revealing that there is a missing "roles" option and it left me completely lost as to what to do. So I backtracked.

[ludeknovy]

Hi @RumRogerz ! The best way would be to upgrade to latest version and resolve the "role" issue.

To start with - when you upgrade your services, do you upgrade migration as well (it should be the same version as BE) ?

[RumRogerz]

Hi @ludeknovy; that's exactly what it was - migrations.

Essentially, I took your project and ported it over to kubernetes. I had to make... "adjustments" to get it to play properly. When I was reviewing all the code and your docker-compose, I was still struggling to understand what migrations did or if it was even necessary for a kubernetes deployment. So I foolheartedly left it out.

So what I did, was used your BE image as an init container to start up the migration and then the same BE image as the main container. Once I did that - magic happened and everything started to work right out of the box.

Now I'm able to access your awesome product over https so my team can view all of QA's jmeter runs, and also use the API!

[ludeknovy]

@RumRogerz Perfect! Glad you made it work!

Would you mind sharing the tutorial on how to run the application over https in kubernetes?

[RumRogerz]

@ludeknovy

Super easy on my end. Granted, We use a few technologies for our ingress. Essentially, we have an nginx ingress controller on our kubernetes cluster that acts as a load balancer and reverse-proxy.
Things to note:

1. We use AWS, but NOT EKS. We run a customised version of k3s on simple vm's. With a bunch of IaC to manage it
2. We run cert-manager. This will give us let's encrypt certs and handles all of the auto-renewals and stuff
3. Cert manager also handles our external DNS entries - we use route53 because of AWS

Once all of that was set up, creating the ingress is easy as pie:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jtl-reports
  namespace: jtl-reports
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod-r53
    external-dns.alpha.kubernetes.io/hostname: jtl-reports.my-cool-domain.com
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
    nginx.ingress.kubernetes.io/proxy-body-size: 2048M
spec:
  rules:
  - host: jtl-reports.my-cool-domain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: jtl-reporter-frontend
            port:
              number: 2020
  tls:
  - hosts:
    - jtl-reports.my-cool-domain.com
    secretName: jtl-reports-ingress-tls

NB: I had to also add in nginx.ingress.kubernetes.io/proxy-body-size: 2048M to the annotations. The default on our nginx controller is much lower, so this option overrides that setting, and I set it to the same values found in nginx.conf in your FE deployment.

All API calls can be made just by hitting the new https url.

[ludeknovy]

Thanks a lot. I would like to add it into docs, but maybe you would like to contribute and take the credits for it?
https://github.com/ludeknovy/jtl-reporter-docs