From 630ffb22faeec5b8cd460712f2daa7368fb8f255 Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 20 Nov 2025 12:01:00 -0500 Subject: [PATCH 1/7] add doc --- .../en/observability_pipelines/rehydration.md | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 content/en/observability_pipelines/rehydration.md diff --git a/content/en/observability_pipelines/rehydration.md b/content/en/observability_pipelines/rehydration.md new file mode 100644 index 0000000000000..6ffd2e6eaebaf --- /dev/null +++ b/content/en/observability_pipelines/rehydration.md @@ -0,0 +1,44 @@ +--- +title: Rehydration +description: Learn more about using Rehydration to pull archived logs and processing them in Observability Pipelines. +disable_toc: false +private: true +--- + +## Overview + +Organizations often store large volumes of logs in cost-efficient, long-term archives to control spend and meet compliance requirements. However, when there is a security incident, audit request, or operational investigation historical data often becomes difficult to access. Retrieving archived logs from cold storage can be slow, manual, and disruptive, requiring ad-hoc scripts, decompression, or dedicated engineering effort. + +Rehydration for Observability Pipelines removes these barriers by enabling you to quickly pull archived logs from object storage and process them through the same processors. This gives teams fast, consistent access to historical context without rebuilding workflows or modifying ingestion pipelines. This gives you fast, consistent access to historical context without rebuilding workflows or modifying ingestion pipelines. + +## How does Rehydration work? + +Rehydration provides an automated workflow for retrieving and reprocessing archived logs stored in cost-optimized object stores, such as Amazon S3, Google Cloud Storage, and Azure Blob Storage. This helps you balance storage efficiency with rapid access to historical data. + +With Rehydration, you can: + +### Retrieve archived logs on demand + +Pull only the data required for investigations, audits, troubleshooting, or pipeline testing, and eliminate long retrieval delays and manual extraction steps. + +### Target specific time ranges or event slices + +Specify the exact timeframe or subset of events needed to ensure fast access and prevent unnecessary data movement or processing. + +### Process historical logs with Observability Pipelines and Packs + +Rehydrated logs go through the same parsing, enrichment, normalization, and routing logic applied to live streams. + +This ensures: +- Consistent formatting and field extraction +- Reliable enrichment (for example, user, geo-IP, and cloud metadata) +- Uniform security and compliance controls +- Identical behavior across historical and real-time data + +### Route rehydrated data to any supported destination + +Send processed historical logs to SIEMs, data lakes, analytics platforms, threat tools, or any destination connected to Observability Pipelines. + +### Eliminate manual handling + +Rehydration provides a structured, predictable way to pull archived data back into your tooling, when you need it and at the scale you choose. You don't have to use custom scripts, manual decompression, or ad-hoc retrieval processes. From 4e2c9387b41f6e326e8e0e35520d10df8548661d Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 20 Nov 2025 13:08:42 -0500 Subject: [PATCH 2/7] updates --- .../en/observability_pipelines/rehydration.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/content/en/observability_pipelines/rehydration.md b/content/en/observability_pipelines/rehydration.md index 6ffd2e6eaebaf..6a02b18c01602 100644 --- a/content/en/observability_pipelines/rehydration.md +++ b/content/en/observability_pipelines/rehydration.md @@ -1,35 +1,36 @@ --- title: Rehydration -description: Learn more about using Rehydration to pull archived logs and processing them in Observability Pipelines. +description: Learn about using Rehydration to pull archived logs and processing them in Observability Pipelines. disable_toc: false private: true --- ## Overview -Organizations often store large volumes of logs in cost-efficient, long-term archives to control spend and meet compliance requirements. However, when there is a security incident, audit request, or operational investigation historical data often becomes difficult to access. Retrieving archived logs from cold storage can be slow, manual, and disruptive, requiring ad-hoc scripts, decompression, or dedicated engineering effort. +Rehydration for Observability Pipelines enables you to pull archived logs from object storage and process them in Observability Pipelines. This gives you consistent access to historical context without having to rebuild workflows or modify ingestion pipelines. -Rehydration for Observability Pipelines removes these barriers by enabling you to quickly pull archived logs from object storage and process them through the same processors. This gives teams fast, consistent access to historical context without rebuilding workflows or modifying ingestion pipelines. This gives you fast, consistent access to historical context without rebuilding workflows or modifying ingestion pipelines. +Organizations often store large volumes of logs in cost-efficient, long-term archives to control spend and meet compliance requirements. However, when there is a security incident, audit request, or operational investigation historical data often becomes difficult to access. Retrieving archived logs from cold storage can be slow, manual, and disruptive, requiring ad-hoc scripts, decompression, or dedicated engineering effort. Rehydration for Observability Pipelines resolves these issues. ## How does Rehydration work? -Rehydration provides an automated workflow for retrieving and reprocessing archived logs stored in cost-optimized object stores, such as Amazon S3, Google Cloud Storage, and Azure Blob Storage. This helps you balance storage efficiency with rapid access to historical data. +Rehydration provides an automated workflow for retrieving and reprocessing archived logs stored in cost-optimized object stores, such as Amazon S3, Google Cloud Storage, and Azure Blob Storage. This helps you balance storage efficiency with quick access to historical data. With Rehydration, you can: ### Retrieve archived logs on demand -Pull only the data required for investigations, audits, troubleshooting, or pipeline testing, and eliminate long retrieval delays and manual extraction steps. +Pull only the data you need for investigations, audits, troubleshooting, or pipeline testing, and eliminate long retrieval delays and manual extraction steps. ### Target specific time ranges or event slices -Specify the exact timeframe or subset of events needed to ensure fast access and prevent unnecessary data movement or processing. +Specify the exact time frame or subset of events you need to prevent moving or processing data unnecessarily. ### Process historical logs with Observability Pipelines and Packs -Rehydrated logs go through the same parsing, enrichment, normalization, and routing logic applied to live streams. +Rehydrated logs go through the same parsing, enrichment, normalization, and routing logic applied to live log streams. This ensures: + - Consistent formatting and field extraction - Reliable enrichment (for example, user, geo-IP, and cloud metadata) - Uniform security and compliance controls @@ -37,8 +38,8 @@ This ensures: ### Route rehydrated data to any supported destination -Send processed historical logs to SIEMs, data lakes, analytics platforms, threat tools, or any destination connected to Observability Pipelines. +You can send processed historical logs to SIEMs, data lakes, analytics platforms, or any Observability Pipelines destinations. ### Eliminate manual handling -Rehydration provides a structured, predictable way to pull archived data back into your tooling, when you need it and at the scale you choose. You don't have to use custom scripts, manual decompression, or ad-hoc retrieval processes. +Rehydration provides a structured, predictable way to pull archived data back into your observability platform, so you don't have to use custom scripts, manual decompression, or ad-hoc retrieval processes. From c1ec37442eda9c9343830c4d3e0b80f099480ff8 Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 20 Nov 2025 14:34:28 -0500 Subject: [PATCH 3/7] add further reading --- .../en/observability_pipelines/rehydration.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/content/en/observability_pipelines/rehydration.md b/content/en/observability_pipelines/rehydration.md index 6a02b18c01602..03a61f036c94f 100644 --- a/content/en/observability_pipelines/rehydration.md +++ b/content/en/observability_pipelines/rehydration.md @@ -3,17 +3,24 @@ title: Rehydration description: Learn about using Rehydration to pull archived logs and processing them in Observability Pipelines. disable_toc: false private: true +further_reading: +- link: "/observability_pipelines/processors/" + tag: "Documentation" + text: "Learn more about processors" +- link: "/observability_pipelines/packs/" + tag: "Documentation" + text: "Learn more about Packs" --- ## Overview Rehydration for Observability Pipelines enables you to pull archived logs from object storage and process them in Observability Pipelines. This gives you consistent access to historical context without having to rebuild workflows or modify ingestion pipelines. -Organizations often store large volumes of logs in cost-efficient, long-term archives to control spend and meet compliance requirements. However, when there is a security incident, audit request, or operational investigation historical data often becomes difficult to access. Retrieving archived logs from cold storage can be slow, manual, and disruptive, requiring ad-hoc scripts, decompression, or dedicated engineering effort. Rehydration for Observability Pipelines resolves these issues. +Organizations often store large volumes of logs in cost-efficient, long-term archives to control spend and meet compliance requirements. However, historical data often becomes difficult to access when there is a security incident, audit request, or operational investigation. Retrieving archived logs from cold storage can be slow, manual, and disruptive, requiring ad-hoc scripts, decompression, or dedicated engineering effort. Rehydration for Observability Pipelines resolves these issues. ## How does Rehydration work? -Rehydration provides an automated workflow for retrieving and reprocessing archived logs stored in cost-optimized object stores, such as Amazon S3, Google Cloud Storage, and Azure Blob Storage. This helps you balance storage efficiency with quick access to historical data. +Rehydration provides an automated workflow for retrieving and reprocessing archived logs stored in object stores, such as Amazon S3, Google Cloud Storage, and Azure Blob Storage. This helps you balance storage efficiency with quick access to historical data. With Rehydration, you can: @@ -25,7 +32,7 @@ Pull only the data you need for investigations, audits, troubleshooting, or pipe Specify the exact time frame or subset of events you need to prevent moving or processing data unnecessarily. -### Process historical logs with Observability Pipelines and Packs +### Process historical logs with Observability Pipelines Rehydrated logs go through the same parsing, enrichment, normalization, and routing logic applied to live log streams. @@ -38,8 +45,12 @@ This ensures: ### Route rehydrated data to any supported destination -You can send processed historical logs to SIEMs, data lakes, analytics platforms, or any Observability Pipelines destinations. +You can send processed historical logs to SIEMs, data lakes, analytics platforms, or any Observability Pipelines destination. ### Eliminate manual handling Rehydration provides a structured, predictable way to pull archived data back into your observability platform, so you don't have to use custom scripts, manual decompression, or ad-hoc retrieval processes. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} \ No newline at end of file From 9207f2b14cce5278c7747c6025f06b210f23ceab Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 20 Nov 2025 14:38:06 -0500 Subject: [PATCH 4/7] add preview box --- content/en/observability_pipelines/rehydration.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/content/en/observability_pipelines/rehydration.md b/content/en/observability_pipelines/rehydration.md index 03a61f036c94f..fb03d67df723b 100644 --- a/content/en/observability_pipelines/rehydration.md +++ b/content/en/observability_pipelines/rehydration.md @@ -12,6 +12,11 @@ further_reading: text: "Learn more about Packs" --- +{{< callout + btn_hidden="true" header="false">}} +Rehydration is in Preview. +{{< /callout >}} + ## Overview Rehydration for Observability Pipelines enables you to pull archived logs from object storage and process them in Observability Pipelines. This gives you consistent access to historical context without having to rebuild workflows or modify ingestion pipelines. From d05a4f3128d7bd8cd78dc51aa328fd14d709b8aa Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 20 Nov 2025 14:53:39 -0500 Subject: [PATCH 5/7] Update content/en/observability_pipelines/rehydration.md --- content/en/observability_pipelines/rehydration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/observability_pipelines/rehydration.md b/content/en/observability_pipelines/rehydration.md index fb03d67df723b..7a49361c0f7cb 100644 --- a/content/en/observability_pipelines/rehydration.md +++ b/content/en/observability_pipelines/rehydration.md @@ -21,7 +21,7 @@ Rehydration is in Preview. Rehydration for Observability Pipelines enables you to pull archived logs from object storage and process them in Observability Pipelines. This gives you consistent access to historical context without having to rebuild workflows or modify ingestion pipelines. -Organizations often store large volumes of logs in cost-efficient, long-term archives to control spend and meet compliance requirements. However, historical data often becomes difficult to access when there is a security incident, audit request, or operational investigation. Retrieving archived logs from cold storage can be slow, manual, and disruptive, requiring ad-hoc scripts, decompression, or dedicated engineering effort. Rehydration for Observability Pipelines resolves these issues. +Organizations often store large volumes of logs in cost-efficient, long-term archives to control spend and meet compliance requirements. However, historical data often becomes difficult to access when there is a security incident, audit request, or operational investigation. Retrieving archived logs from cold storage can be slow, manual, and disruptive, requiring ad-hoc scripts, decompression, or dedicated engineering effort. Rehydration for Observability Pipelines solves these issues. ## How does Rehydration work? From 028c4325edae0c4b5ffe77f2fea457f8c5c26611 Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 20 Nov 2025 14:53:57 -0500 Subject: [PATCH 6/7] Update content/en/observability_pipelines/rehydration.md --- content/en/observability_pipelines/rehydration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/observability_pipelines/rehydration.md b/content/en/observability_pipelines/rehydration.md index 7a49361c0f7cb..a2d6c79045e7a 100644 --- a/content/en/observability_pipelines/rehydration.md +++ b/content/en/observability_pipelines/rehydration.md @@ -23,7 +23,7 @@ Rehydration for Observability Pipelines enables you to pull archived logs from o Organizations often store large volumes of logs in cost-efficient, long-term archives to control spend and meet compliance requirements. However, historical data often becomes difficult to access when there is a security incident, audit request, or operational investigation. Retrieving archived logs from cold storage can be slow, manual, and disruptive, requiring ad-hoc scripts, decompression, or dedicated engineering effort. Rehydration for Observability Pipelines solves these issues. -## How does Rehydration work? +## How Rehydration works Rehydration provides an automated workflow for retrieving and reprocessing archived logs stored in object stores, such as Amazon S3, Google Cloud Storage, and Azure Blob Storage. This helps you balance storage efficiency with quick access to historical data. From 3057d9f8bea022237bd9e69bff9fd78982644305 Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 20 Nov 2025 15:52:15 -0500 Subject: [PATCH 7/7] add link to packs --- content/en/observability_pipelines/rehydration.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/en/observability_pipelines/rehydration.md b/content/en/observability_pipelines/rehydration.md index a2d6c79045e7a..7c5faf1bf683f 100644 --- a/content/en/observability_pipelines/rehydration.md +++ b/content/en/observability_pipelines/rehydration.md @@ -19,7 +19,7 @@ Rehydration is in Preview. ## Overview -Rehydration for Observability Pipelines enables you to pull archived logs from object storage and process them in Observability Pipelines. This gives you consistent access to historical context without having to rebuild workflows or modify ingestion pipelines. +Rehydration for Observability Pipelines enables you to pull archived logs from object storage and process them in Observability Pipelines, including with [Packs][1]. This gives you consistent access to historical context without having to rebuild workflows or modify ingestion pipelines. Organizations often store large volumes of logs in cost-efficient, long-term archives to control spend and meet compliance requirements. However, historical data often becomes difficult to access when there is a security incident, audit request, or operational investigation. Retrieving archived logs from cold storage can be slow, manual, and disruptive, requiring ad-hoc scripts, decompression, or dedicated engineering effort. Rehydration for Observability Pipelines solves these issues. @@ -58,4 +58,6 @@ Rehydration provides a structured, predictable way to pull archived data back in ## Further reading -{{< partial name="whats-next/whats-next.html" >}} \ No newline at end of file +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /observability_pipelines/packs/ \ No newline at end of file