Database subsetting and filters

[wwoytenko]

This discussion is created to continue about subsetting and filters raised in #17

[wwoytenko]

A short recap of #17:

• Greenmask has not supported yet database subsetting
• The subset feature is going to be released in 2024 Q2-Q3
• The temporal solution is query property usage (but it can violate PK/FK constraint)

Example:

- schema: "humanresources"
  name: "employee"
  query: "select * from humanresources.employee where birthdate > '1980-01-1';"
  transformers:
    - name: "RandomDate"
      params:
        column: "modifieddate"
        max: "2024-01-21 11:32:30.385227"
        min: "2020-01-21 11:32:30.385227"

[gregwebs]

If someone requires this feature before then they can try using datanymizer

[gregwebs]

I looked at our anonymization and we use the subsetting (dump_condition) less than a "transform_condition". A transform_condition preserves a subset of the rows as is without obfuscating them. This feature is a lot easier to implement since it is not concerned about foreign keys.

[wwoytenko]

You're right the transformation condition is much easier to implement than the dump filters with inheritance by FK/PK references. In Greenmask we just need to provide a textual field in the config that receives the go template and provide a record object in template runtime. Using template language and record objects users can write any condition. I believe we can publish this feature in the nearest release.

[gregwebs]

Congrats on putting out a new release!

If you have accepted the feature request for a conditional transform, should that be made into a github issue?

[wwoytenko]

Thank you!

It's up to you, I will try to deliver it in the next minor version 0.1.8, before major release in the April.

[gregwebs]

I think an issue is good because it's separate from the subsetting request in the title. #34

[wwoytenko]

Just wanted to announce that we have started to implement the database subset feature. The approximate release date is May-June 2024

[wwoytenko]

Caused #110

[wwoytenko]

Hi!
I am happy to announce that this thread can be closed by the #110 completion.

[wwoytenko]

The release date is tomorrow (30.08.2024) as a version v0.2b2