Skip to content

DOC-5223 RDI release notes for v1.8.0 #1549

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 13, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion config.toml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ rdi_redis_gears_version = "1.2.6"
rdi_debezium_server_version = "2.3.0.Final"
rdi_db_types = "cassandra|mysql|oracle|postgresql|sqlserver"
rdi_cli_latest = "latest"
rdi_current_version = "v1.6.7"
rdi_current_version = "1.8.0"

[params.clientsConfig]
"Python"={quickstartSlug="redis-py"}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,10 @@ Complete the following steps before installing the RDI Helm chart:
more information).

- Download the RDI Helm chart tar file from the
[Redis download center](https://redis.io/downloads/) (in the *Modules, Tools & Integration* category) .
[Redis download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-{{< rdi-version >}}.tgz) (in the *Modules, Tools & Integration* category) .

```bash
export RDI_VERSION=<version>
export RDI_VERSION={{< rdi-version >}}
wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-$RDI_VERSION.tgz
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ This guide explains how to install Redis Data Integration (RDI) on one or more V
your source database. You can also
[Install RDI on Kubernetes]({{< relref "/integrate/redis-data-integration/installation/install-k8s" >}}).

{{< note >}}We recommend you always use the latest version, which is RDI {{< field "rdi_current_version" >}}.
{{< note >}}We recommend you always use the latest version, which is RDI v{{< rdi-version >}}.
{{< /note >}}

## Create the RDI database
Expand Down Expand Up @@ -137,12 +137,12 @@ sudo ufw reload
Follow the steps below for each of your VMs:

1. Download the RDI installer from the
[Redis download center](https://redis.io/downloads/)
[Redis download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-{{< rdi-version >}}.tar.gz)
(from the *Modules, Tools & Integration* category) and extract it to your preferred installation
folder.

```bash
export RDI_VERSION=<version>
export RDI_VERSION={{< rdi-version >}}
wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-$RDI_VERSION.tar.gz
tar -xvf rdi-installation-$RDI_VERSION.tar.gz
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,12 @@ Follow the steps below to upgrade an existing
[VM installation]({{< relref "/integrate/redis-data-integration/installation/install-vm" >}})
of RDI:

1. Download the RDI installer from the [Redis download center](https://redis.io/downloads/)
1. Download the RDI installer from the [Redis download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-{{< rdi-version >}}.tar.gz)
(in the *Modules, Tools & Integration* category) and extract it to your
preferred installation folder.

```bash
export RDI_VERSION=<version>
export RDI_VERSION={{< rdi-version >}}
wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-$RDI_VERSION.tar.gz
tar -xvf rdi-installation-$RDI_VERSION.tar.gz
```
Expand Down Expand Up @@ -92,11 +92,11 @@ installation of RDI:
[Using a private image registry]({{< relref "/integrate/redis-data-integration/installation/install-k8s#using-a-private-image-registry" >}})
and add them to your local registry.

1. Download the RDI Helm chart tar file from the [Redis download center](https://redis.io/downloads/)
1. Download the RDI Helm chart tar file from the [Redis download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-{{< rdi-version >}}.tgz)
(in the *Modules, Tools & Integration* category).

```bash
export RDI_VERSION=<version>
export RDI_VERSION={{< rdi-version >}}
wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-$RDI_VERSION.tgz
```

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
---
Title: Redis Data Integration release notes 1.8.0 (May 2025)
alwaysopen: false
categories:
- docs
- operate
- rs
description: |
Enhanced RDI operator for better pipeline orchestration, resiliency, observability and flexibility;
External collector support;
Labels and annotations to RDI data plane pods;
Custom Debezium image;
Calculated TTL for target database.
linkTitle: 1.8.0 (May 2025)
toc: 'true'
weight: 984
---

{{< note >}}This minor release replaces the 1.6.7 release.{{< /note >}}

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

## Headlines

- Enhanced RDI operator for better pipeline orchestration, resiliency, observability, and flexibility. It
will also enable many new features in the near future.
- You can now use an external collector that is not managed by RDI but writes into RDI streams
(Debezium compatible).
- You can now add labels and annotations to RDI data plane pods, for example to control service
mesh features.
- RDI now uses a custom image of Debezium (based on `3.0.8.Final`) to address known vulnerabilities.
- Added support for calculated TTL for target database keys via `expire` expressions.

## Detailed changes

### Helm chart changes

- All collector and processor values that were previously under `collector`, `collectorSourceMetricsExporter`, and `processor` have been moved to `operator.dataPlane.collector` and `operator.dataPlane.processor`.
- `global.collectorApiEnabled` has been moved to `operator.dataPlane.collectorApi.enabled`, and is now a boolean value (`true` or `false`), not `"0"` or `"1"`.
- `api.authEnabled` is also now a boolean value, not `"0"` or `"1"`.
- The following values have been deprecated: `rdiMetricsExporter.service.protocol`, `rdiMetricsExporter.service.port`, `rdiMetricsExporter.serviceMonitor.path`, `api.service.name`
- You can now add custom labels and annotations to all RDI components.
- You can now disable the creation of the RDI system secrets.

### Operator Improvements

The RDI operator has been significantly enhanced in the following areas:

- **Resilience**: The operator now always maintains the desired pipeline state. Manual changes or random disruptions are reverted automatically.
- **Automatic recovery**: When a configuration issue is resolved, the entire pipeline starts automatically, eliminating the need for manual redeployment.
- **Consistency**: A pipeline that has been stopped with `stop` will remain stopped after `deploy` or `reset`, until explicitly started again.
- **Enhanced configuration**: You can now configure data plane components in ways that were previously not supported, such as adding labels and annotations.
- **External collector support**: No collector resources are created for sources of type `external`.
- **Enhanced troubleshooting**: You can now gain extra insight into the pipeline state by examining the `Pipeline` and `PipelineRelease` custom K8s resources.

### Other Features, Improvements and Enhancements

- Added `expire` expression for target output in transformation jobs.
- Addressed security vulnerabilities: TLS certificate hostname verification is now ON by default.
- Improved Helm default values while preserving `values.yaml` formatting.
- Enhanced Helm values and templates for better configuration.
- Added a script to create or update secrets when using Helm (`rdi-secret.sh` in the Helm zip file).
- Improved validation schema and ensured backward compatibility.
- Fixed compatibility issues with newer versions of `requests` and `urllib3`.
- Improved error messages for JSON schema validation.
- Improved PostgreSQL documentation for mTLS.
- Added timestamps to the `status` command.
- Fixed issues with `primary_key` and `unique_constraint` attributes in Oracle metadata.
- Added `capture.mode` to MongoDB scaffolding.
- Improved Helm TLS setup for RDI database connections.
- Enhanced error handling and validation for transformation jobs.
- Improved documentation for supported platforms and configurations.

### Fixes

- Fixed HTTP 500 error when querying columns with tables parameter.
- Improved Helm TLS setup for RDI database connections.
- Fixed keystore overwrite when using mTLS on both source and RDI DBs in the collector.

## Limitations

RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits.
1 change: 1 addition & 0 deletions layouts/shortcodes/rdi-version.html
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{- .Site.Params.rdi_current_version -}}