[FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... #389

lefebsy · 2024-10-21T20:13:14Z

Description (edited) :

This is a S3 proposition of Polaris core storage implementation, copy of the aws + new parameters : endpoint, path style...

It is tested OK with :

local MinIO
Backblaze B2 (thanks to @metadaddy)
Ceph.io seems also ok thanks to @mmgaggle & @Gerrit-K tests

This should works with many S3 compatible solutions like Dell ECS, NetApp StorageGRID, etc...

By default it is trying to respect the same behavior about credentials than AWS (IAM/STS). The same dynamic policy is applied, limiting the scope to the data queried.
Otherwise if STS is not available 'skipCredentialSubscopingIndirection' = true will disabling Polaris "SubScoping" of the credentials

Let me know your opinion about this design proposal.
Thank you

curl -X POST -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" \
     -H 'Accept: application/json' -H 'Content-Type: application/json' \
     http://${POLARIS_HOST}:8181/api/management/v1/catalogs -d \
      '{
          "name": "my-s3compatible-catalog",
          "id": 100,
          "type": "INTERNAL",
          "readOnly": false,
          "properties": {
            "default-base-location": "${S3_LOCATION}"
          },
          "storageConfigInfo": {
            "storageType": "S3_COMPATIBLE",
            "allowedLocations": ["${S3_LOCATION}/"],
            "s3.endpoint": "https://localhost:9000"
          }
        }'

            # optional:
            "s3.pathStyleAccess": true / false
            "s3.region": "rack-1 or us-east-1"
            "s3.roleArn": "arn:xxx:xxx:xxx:xxxx"
            "s3.credentials.catalog.accessKeyId": "CATALOG_1_ACCESS_KEY_ENV_VARIABLE_NAME"
            "s3.credentials.catalog.secretAccessKey": "CATALOG_1_SECRET_KEY_ENV_VARIABLE_NAME"

            # optional in case STS/IAM is not available :
            "skipCredentialSubscopingIndirection": true
                # optional for skipCredentialSubscopingIndirection - served by catalog to client like spark or trino
                "s3.credentials.client.accessKeyId": "CLIENT_OF_CATALOG_1_ACCESS_KEY_ENV_VARIABLE_NAME"
                "s3.credentials.client.secretAccessKey": "CLIENT_OF_CATALOG_1_SECRET_KEY_ENV_VARIABLE_NAME"

Included Changes:

New type of storage "S3_COMPATIBLE".
Tested against MinIO with self-signed certificate
regtests/run_spark_sql_s3compatible.sh

Type of change:

Bug fix (non-breaking change which fixes an issue)
Documentation update
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

Please delete options that are not relevant.

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue

...s-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java

spec/polaris-management-service.yml

eric-maynard · 2024-11-05T08:47:39Z

polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisCredentialProperty.java

@@ -23,6 +23,8 @@ public enum PolarisCredentialProperty {
  AWS_KEY_ID(String.class, "s3.access-key-id", "the aws access key id"),
  AWS_SECRET_KEY(String.class, "s3.secret-access-key", "the aws access key secret"),
  AWS_TOKEN(String.class, "s3.session-token", "the aws scoped access token"),
+  AWS_ENDPOINT(String.class, "s3.endpoint", "the aws s3 endpoint"),
+  AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),


whether or not to use path-style access

Many S3COMPATIBLE solutions are deployed without network devices or configurations in front of them allowing support of dynamic hosts names including buckets.
TLS certificate with private AC could also be a challenge for dynamic host name. "*. domain" can also be forbidden by some enterprise security policy

Path style is useful in many cases. In ideal world I agree it should stay deprecated...

I think @eric-maynard was asking if we can change the description to something like this:

Suggested change

AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),

AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "whether or not to use path-style access"),

I also agreed that we should make sure it false by default

For IBM's watsonx.data product it is set to true by default for Minio and Ceph bucket types, reason being that it's more likely to work. Path style will work regardless of whether the customer has setup wildcard DNS, a TLS certificate with a subject-alternate-name (the wildcard), and the hostname in the zonegroup (for Ceph). Virtual host style will only work if all of those things are done.

It's not a hill I would die on, but it's worthy of consideration.

To clarify, I was referring to the Polaris scope default. Users always have the option to set it to true when creating a new catalog.

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java

collado-mike

Most of the code here seems like it should be delegated to either catalog-level properties (to init the FileIO with the right endpoint) or customizations to the way we construct the STS builder. I don't think a separate S3_COMPATIBLE StorageIntegration really needs to exist, with the only distinct feature here being the vending of raw credentials.

Personally, I don't think we ought to be vending raw credentials at all. I think that either the service should be able to send sub-scoped, time-bound credentials or, if that's not possible, the query engine ought to have direct access to the long-lived credentials through some other means.

But regardless of whether or not we vend raw credentials, allowing the user to simply declare whatever arbitrary environment variables they want to read from the server and to then get them is 100% not a safe or secure way of implementing this.

collado-mike · 2025-02-11T00:21:34Z

polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisCredentialProperty.java

+  AWS_ENDPOINT(String.class, "s3.endpoint", "the aws s3 endpoint"),
+  AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),


Typically, these are catalog-level properties, not credentials. I think it's best to avoid overloading the credential providers with the need to populate these configuration properties.

collado-mike · 2025-02-11T00:25:58Z

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java

+    propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+    propertiesMap.put(
+        PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+        storageConfig.getS3PathStyleAccess().toString());


I don't understand why you need them in this class at all. The catalog properties are used to initialize the FileIO in BasePolarisCatalog. The properties from this class are appended to those properties.

collado-mike · 2025-02-11T00:29:53Z

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java

+    if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+      LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+      clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+      clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());


This is not safe. Now, if I have privilege to create a catalog, I can simply set this skipCredentialSubcopingIndirection flag and I immediately get access to the raw credentials used to assume any role for any other catalog in the service?

No, it is not related

collado-mike · 2025-02-11T00:36:50Z

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java

+      StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+      stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));


If I read this right, this is the only code that really needs to deviate from the default S3 credential vending process, right? We could realistically refactor the default STS client to be provided by a factory that takes in the storage config. Then we don't need all this duplication

Almost yes.
If catalog creator do not want to rely on default and global Polaris service keys and want dedicated keys for each catalog, adding credential alternative system in this in factory could be useful. No ?

Security pattern like not putting all eggs in one basket

Also onPrem S3 buckets can been managed with security pattern where administrators are generating different keys for each s3 user associated with some buckets + assumeRole policy (I have this case in my company, considered as secutity segreggation pattern). In this case Polaris with central and unique keys need to be over instanciated :-( Managed at catalog level it allow to keep one Polaris service and many catalogs

//not using provider built credentials from standard AWS env var stsBuilder.credentialsProvider( StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));

collado-mike · 2025-02-11T00:42:44Z

.../java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java

+  private @Nullable String s3CredentialsCatalogAccessKeyId;
+  private @Nullable String s3CredentialsCatalogSecretAccessKey;
+  private @Nullable Boolean s3PathStyleAccess;
+  private @NotNull Boolean skipCredentialSubscopingIndirection;
+  private @Nullable String s3CredentialsClientAccessKeyId;
+  private @Nullable String s3CredentialsClientSecretAccessKey;


What is the distinction between s3CredentialsCatalogAccessKeyId and s3CredentialsClientAccessKeyId? Rather than having users set the names of environment variables, can't we use AWS profiles? That's a heck of a lot safer than allowing users to put the name of any environment variable and read it directly from the server.

Explained in the updated PR summary..
The idea was to use

'catalog' key only for polaris s3 access and sts generation

'client' keys served to spark or trino when STS are not available.

Replacing variable name by Profile name... I like it !
Yes it could probably works, harder to integrate the file in a kubernetes deployement. Secrets are not allowed in configMap, all the file content should be stored as secret and mounted as volume. Except if profile file is able to interprete variables for secret keys...

mmgaggle · 2025-02-11T05:13:08Z

Most of the code here seems like it should be delegated to either catalog-level properties (to init the FileIO with the right endpoint) or customizations to the way we construct the STS builder. I don't think a separate S3_COMPATIBLE StorageIntegration really needs to exist, with the only distinct feature here being the vending of raw credentials.

Is there a way to define a storage endpoint that is used for S3 and STS per catalog without this?

I also tend to agree about vending raw credentials, if a storage backend doesn't support IAM/STS and session policies, tough cookies. Vending scoped credentials is one of the marquee features for Polaris, and vending a static credential might give users a false sense of security.

flyrain · 2025-02-12T18:11:32Z

Vending scoped credentials is one of the marquee features for Polaris, and vending a static credential might give users a false sense of security.

We definitely want to avoid that, because handing out raw credentials is pretty much the same as making them public—any Polaris user could easily access them.

collado-mike · 2025-02-12T20:43:50Z

Is there a way to define a storage endpoint that is used for S3 and STS per catalog without this?

We do this in one of the integ tests that spins up a mock S3 container - see https://github.com/apache/polaris/blob/main/integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisSparkIntegrationTest.java#L116-L134

lefebsy · 2025-02-13T01:32:31Z

Thanks a lot, @lefebsy, for adding this! I believe this is a really useful feature—appreciate your effort. Apologies for the delay.

I agree with @collado-mike that we should avoid vending raw credentials.

You're all welcome.
I can uderstand the motivation behind not vending raw credentials. (Even though I will personally miss it strongly in my business)
I've prepared the quarkus rebase, so will add the easy suggestions of this review and of course keep only STS. Squashed some intermediate commits to be able to have a clean rebase since quarkus.

Longer modifications suggested (profile, refactor of policy methods from aws) will be for next step. Advice welcome :-)

Better descriptions typo & comments Refacoring with skipCredentialSubscopingIndirection -> finaly removed Rebase with AWS updates from main branch adding roleArn, camelCase refactoring, typo, cleaning Add default AWS credentials provider for STS Error Co-authored-by: Gerrit-K <[email protected]> Rebase from quarkus and keep only sts with some suggestions from code review helm unit test

mmgaggle · 2025-03-04T23:26:48Z

Thank you for working on this!

lefebsy · 2025-03-04T23:43:39Z

Hello,

Last refactoring vending only STS.

Support of profile to manage credentials used by catalog to communicate with S3
Refactor the duplicated functions used by 'Aws' and also by 'S3Compatible'. They have been moved to StorageUtil, here they should be imported and used by 'S3Compatible' or either by 'Aws' if this place is adopted.

curl -X POST -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" \
     -H 'Accept: application/json' -H 'Content-Type: application/json' \
     http://${POLARIS_HOST}:8181/api/management/v1/catalogs -d \
      '{
          "name": "my-s3compatible-catalog-1",
          "id": 100,
          "type": "INTERNAL",
          "readOnly": false,
          "properties": {
            "default-base-location": "${S3_LOCATION}"
          },
          "storageConfigInfo": {
            "storageType": "S3_COMPATIBLE",
            "allowedLocations": ["${S3_LOCATION}/"],
            "s3.endpoint": "https://localhost:9000"
          }
        }'

As is AWS SDK will use all default values and settings available in Polaris service, to build catalog communication to S3 endpoint

Otherwise indications can be given

          # optional - 'Indicate a AWS Profile name'
          "s3.profileName": "minio-catalog-1",

or

          # optional - 'Indicate Env variable name'
          "s3.credentials.catalog.accessKeyEnvVar": "CATALOG_S3_KEY_ID_FOR_CATALOG_1",
          "s3.credentials.catalog.secretAccessKeyEnvVar": "CATALOG_S3_KEY_SECRET_FOR_CATALOG_1",

and if helpful

          # optional
          "s3.region": "region-1",
          "s3.pathStyleAccess": true,
          "s3.roleArn": "arn:xxx:xxx:xxx:xxx:xxx"

Test script (with minio container) adapted to use 'profile' instead of 'env var'

>regtests/run_spark_sql_s3compatible.sh

collado-mike

This is a much better impl, IMO. Thanks for the work - it's a feature a lot of folks want, so I'm happy to see it implemented.

flyrain · 2025-03-05T05:56:18Z

.../java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java

+  // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+  // for allowed read and write locations for sub-scoping credentials.
+  @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;


I'd suggest remove it per discussion here, #1068 (comment). So that we don't have to introduce extra config later.

Ok I will rebase after #1068 merged

flyrain

Thanks a lot for working on it. Getting close. Left some comments. Other than the server side env variable settings, can we also have unit tests for two new classes added?

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java

flyrain · 2025-03-05T06:32:30Z

.../java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java

+    // storing properties
+    super(storageType, allowedLocations);
+    validateMaxAllowedLocations(MAX_ALLOWED_LOCATIONS);
+    this.s3PathStyleAccess = s3PathStyleAccess;


I refer to the field(s3PathStyleAccess) in this class, which can be set to null. In that case, storageConfig.getS3PathStyleAccess().toString() will throw a NPE.

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java

.../java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java

…ompatible/S3CompatibleCredentialsStorageIntegration.java Co-authored-by: Yufei Gu <[email protected]>

lefebsy · 2025-03-07T18:45:53Z

I refer to the field(s3PathStyleAccess) in this class, which can be set to null. In that case, storageConfig.getS3PathStyleAccess().toString() will throw a NPE

There is a default "false" in rest spec. But ok 👍

flyrain · 2025-03-07T19:22:37Z

I refer to the field(s3PathStyleAccess) in this class, which can be set to null. In that case, storageConfig.getS3PathStyleAccess().toString() will throw a NPE

There is a default "false" in rest spec. But ok 👍

I agreed that it's fine if the object is always created from the REST. But the class itself could be constructed internally in the future. I don't want a NPE surprise in that case.

lefebsy · 2025-03-08T12:04:30Z

Thanks a lot for working on it. Getting close. Left some comments. Other than the server side env variable settings, can we also have unit tests for two new classes added?

Is it ok for you if I do UT only for config class and move to dockerized tests for credentials class ?
I am really not good in the mock code area.

I can easily add minio to regtests docker-compose and enrich "regtests/t_spark_sql/"

My first try just learned me not to name script with "S3" inside otherwise it's trapped by AWS stop pattern and never run.

flyrain · 2025-03-09T02:04:26Z

Is it ok for you if I do UT only for config class and move to dockerized tests for credentials class ? I am really not good in the mock code area.

I can easily add minio to regtests docker-compose and enrich "regtests/t_spark_sql/"

My first try just learned me not to name script with "S3" inside otherwise it's trapped by AWS stop pattern and never run.

I'm OK with a followup PR to resolve the mock related unit tests. We can get some help from the community. Can you file an issue to track that?

lefebsy requested review from jbonofre, ashvina, RussellSpitzer, snazy, vvcephei, takidau, jackye1995, flyrain, eric-maynard, collado-mike and ebyhr as code owners October 21, 2024 20:13

lefebsy changed the title ~~add s3 compatible storage - first commit~~ [FEATURE REQUEST] On-Premise S3... #32 Oct 21, 2024

lefebsy changed the title ~~[FEATURE REQUEST] On-Premise S3... #32~~ [FEATURE REQUEST #32] On-Premise S3... Oct 21, 2024

lefebsy changed the title ~~[FEATURE REQUEST #32] On-Premise S3...~~ [FEATURE REQUEST #32] On-Premises S3... Oct 21, 2024

This comment was marked as outdated.

Sign in to view

lefebsy force-pushed the s3compatible branch from ee0a8ef to 138c1f7 Compare October 22, 2024 17:48

eric-maynard reviewed Oct 22, 2024

View reviewed changes

...s-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java Outdated Show resolved Hide resolved

eric-maynard reviewed Oct 22, 2024

View reviewed changes

...s-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java Outdated Show resolved Hide resolved

eric-maynard reviewed Oct 22, 2024

View reviewed changes

spec/polaris-management-service.yml Outdated Show resolved Hide resolved

eric-maynard reviewed Oct 22, 2024

View reviewed changes

spec/polaris-management-service.yml Outdated Show resolved Hide resolved

eric-maynard reviewed Oct 22, 2024

View reviewed changes

spec/polaris-management-service.yml Outdated Show resolved Hide resolved

eric-maynard reviewed Oct 22, 2024

View reviewed changes

spec/polaris-management-service.yml Outdated Show resolved Hide resolved

lefebsy force-pushed the s3compatible branch 2 times, most recently from dd8d860 to e2c296b Compare October 28, 2024 15:56

lefebsy requested a review from eric-maynard October 28, 2024 16:12

sungwy mentioned this pull request Oct 29, 2024

WIP: Testing out using Polaris Docker image in Integration Test Suite apache/iceberg-python#1252

Closed

lefebsy force-pushed the s3compatible branch from cb39f08 to fb42d0e Compare October 31, 2024 17:39

eric-maynard reviewed Nov 5, 2024

View reviewed changes

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java Outdated Show resolved Hide resolved

eric-maynard reviewed Nov 5, 2024

View reviewed changes

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java Outdated Show resolved Hide resolved

collado-mike requested changes Feb 11, 2025

View reviewed changes

lefebsy force-pushed the s3compatible branch from f7c3c68 to a594a38 Compare February 13, 2025 07:44

lefebsy requested review from dennishuo, dimas-b and MonkeyCanCode as code owners February 13, 2025 07:44

lefebsy requested a review from flyrain February 13, 2025 07:57

This was referenced Feb 19, 2025

Apache Polaris OSS Roadmap Proposal sfc-gh-ygu/test#1

Open

Apache Polaris OSS Roadmap 2025 v2 sfc-gh-ygu/test#2

Open

lefebsy and others added 5 commits March 4, 2025 22:36

add s3 compatible storage - first commit

d8c2079

add s3 profile

0327f15

rebase

1fa3a97

format

bd90304

lefebsy force-pushed the s3compatible branch from a594a38 to bd90304 Compare March 4, 2025 23:08

collado-mike approved these changes Mar 5, 2025

View reviewed changes

flyrain reviewed Mar 5, 2025

View reviewed changes

lefebsy and others added 2 commits March 5, 2025 18:12

Update polaris-core/src/main/java/org/apache/polaris/core/storage/s3c…

eedee82

…ompatible/S3CompatibleCredentialsStorageIntegration.java Co-authored-by: Yufei Gu <[email protected]>

Update polaris-core/src/main/java/org/apache/polaris/core/storage/s3c…

ac951e3

…ompatible/S3CompatibleCredentialsStorageIntegration.java Co-authored-by: Yufei Gu <[email protected]>

Akongstad mentioned this pull request Mar 5, 2025

(BLOCKED)Add a new polaris-boot approach that uses minio to simulate AWS S3 msc-open-metadata/polaris-boot#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... #389

[FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... #389

lefebsy commented Oct 21, 2024 •

edited

Loading

This comment was marked as outdated.

eric-maynard Nov 5, 2024

lefebsy Nov 6, 2024

flyrain Feb 9, 2025

mmgaggle Feb 12, 2025

flyrain Feb 12, 2025

collado-mike left a comment

collado-mike Feb 11, 2025

collado-mike Feb 11, 2025

collado-mike Feb 11, 2025

lefebsy Feb 12, 2025 •

edited

Loading

collado-mike Feb 11, 2025

lefebsy Feb 12, 2025 •

edited

Loading

collado-mike Feb 11, 2025

lefebsy Feb 12, 2025

mmgaggle commented Feb 11, 2025 •

edited

Loading

flyrain commented Feb 12, 2025

collado-mike commented Feb 12, 2025

lefebsy commented Feb 13, 2025 •

edited

Loading

mmgaggle commented Mar 4, 2025

lefebsy commented Mar 4, 2025 •

edited

Loading

collado-mike left a comment

flyrain Mar 5, 2025 •

edited

Loading

lefebsy Mar 5, 2025

flyrain left a comment

flyrain Mar 5, 2025

lefebsy commented Mar 7, 2025

flyrain commented Mar 7, 2025

lefebsy commented Mar 8, 2025 •

edited

Loading

flyrain commented Mar 9, 2025

	AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),
	AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "whether or not to use path-style access"),

		AWS_ENDPOINT(String.class, "s3.endpoint", "the aws s3 endpoint"),
		AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),

		StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
		stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));

[FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... #389

Are you sure you want to change the base?

[FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... #389

Conversation

lefebsy commented Oct 21, 2024 • edited Loading

Description (edited) :

Included Changes:

Type of change:

Checklist:

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

collado-mike left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lefebsy Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lefebsy Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmgaggle commented Feb 11, 2025 • edited Loading

flyrain commented Feb 12, 2025

collado-mike commented Feb 12, 2025

lefebsy commented Feb 13, 2025 • edited Loading

mmgaggle commented Mar 4, 2025

lefebsy commented Mar 4, 2025 • edited Loading

collado-mike left a comment

Choose a reason for hiding this comment

flyrain Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flyrain left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lefebsy commented Mar 7, 2025

flyrain commented Mar 7, 2025

lefebsy commented Mar 8, 2025 • edited Loading

flyrain commented Mar 9, 2025

lefebsy commented Oct 21, 2024 •

edited

Loading

lefebsy Feb 12, 2025 •

edited

Loading

lefebsy Feb 12, 2025 •

edited

Loading

mmgaggle commented Feb 11, 2025 •

edited

Loading

lefebsy commented Feb 13, 2025 •

edited

Loading

lefebsy commented Mar 4, 2025 •

edited

Loading

flyrain Mar 5, 2025 •

edited

Loading

lefebsy commented Mar 8, 2025 •

edited

Loading