add data readme

Author: emmanuelubachi

data/README.md

@@ -0,0 +1,64 @@
+# Data Workflow Process
+
+This document outlines the standard data workflow process from the "Raw" data folder to the "Final" data folder, including the "Processed" data folder. This process ensures that data is properly cleaned, transformed, and prepared for analysis.
+
+## 1. Raw Data
+
+- **Folder:** Raw
+- **Purpose:** The "Raw" folder contains the original, untouched data as received from external sources or collected internally. This data serves as the source of truth.
+
+## 2. Data Ingestion
+
+- **Task:** Move data from the "Raw" folder to the "Staging" folder for initial data ingestion and preparation.
+- **Folder:** Staging
+- **Purpose:** The "Staging" folder is used to prepare data for further processing.
+
+## 3. Data Cleaning and Transformation
+
+- **Task:** Perform data cleaning, including handling missing values, outliers, and data type conversions.
+- **Folder:** Staging
+- **Purpose:** Prepare data in the "Staging" folder for analysis by ensuring it is accurate and consistent.
+
+## 4. Data Exploration
+
+- **Task:** Explore the data in the "Staging" folder to understand its characteristics and identify potential insights.
+- **Folder:** Staging
+- **Purpose:** Gain insights into the data, which will inform further data processing steps.
+
+## 5. Intermediate Data Storage
+
+- **Task:** Move cleaned and explored data from the "Staging" folder to the "Processed" folder.
+- **Folder:** Processed
+- **Purpose:** The "Processed" folder is used to store data that has undergone initial cleaning and exploration.
+
+## 6. Additional Data Transformation
+
+- **Task:** Perform additional data transformations, such as feature engineering or aggregations, on data in the "Processed" folder.
+- **Folder:** Processed
+- **Purpose:** Create datasets in the "Processed" folder that are tailored for specific analysis goals.
+
+## 7. Data Quality Assurance
+
+- **Task:** Ensure data quality by conducting thorough quality checks and validation.
+- **Folder:** Processed
+- **Purpose:** Verify that data in the "Processed" folder is accurate and reliable for analysis.
+
+## 8. Intermediate Data Storage (Interim)
+
+- **Task:** Move data from the "Processed" folder to the "Interim" folder as needed for specific analysis steps.
+- **Folder:** Interim
+- **Purpose:** The "Interim" folder is used to store intermediate datasets that are essential for specific analysis steps.
+
+## 9. Final Data Storage
+
+- **Task:** Move the final analysis-ready data from the "Interim" folder to the "Final" folder.
+- **Folder:** Final
+- **Purpose:** The "Final" folder contains cleaned and processed data that is ready for analysis, reporting, or sharing with stakeholders.
+
+## 10. Documentation
+
+- **Task:** Document all data processing steps, transformations, and any relevant metadata.
+- **Folder:** docs
+- **Purpose:** Maintain clear documentation to ensure reproducibility and transparency in the data analysis process.
+
+This workflow provides a structured approach to preparing data for analysis, ensuring that data is accurate, cleaned, and transformed appropriately before it reaches its final state in the "Final" folder.

data/external/README.md

@@ -0,0 +1,11 @@
+### Folder: External
+
+**Purpose:**
+
+> This folder is intended for storing data from external sources. Data placed here should be in its raw, unprocessed form, as received from external providers. It serves as our source repository for external data.
+
+**Usage:**
+
+> - Data in this folder should not be modified directly.
+> - When new external data arrives, place it here for further processing.
+> - Document the data source and any relevant details in the metadata file.

data/final/README.md

@@ -0,0 +1,11 @@
+### Folder: Final
+
+**Purpose:**
+
+> The "Final" folder contains cleaned and processed data that is ready for analysis, reporting, or sharing with stakeholders. This represents the final state of our data for specific projects.
+
+**Usage:**
+
+> - Data here is considered ready for analysis.
+> - Ensure that all necessary data transformations and cleaning have been completed before moving data to this folder.
+> - Document any transformations or preprocessing steps applied to the data.

data/interim/README.md

@@ -0,0 +1,11 @@
+### Folder: Interim
+
+**Purpose:**
+
+> The "Interim" folder is used for storing intermediate datasets generated during the data preparation process. These datasets might not be the final data but are crucial for specific analysis steps or for tracking data transformations.
+
+**Usage:**
+
+> - Data here is not considered final but may be required for ongoing analysis.
+> - Document the purpose of each interim dataset and its relationship to the overall analysis process.
+> - Regularly review and clean up outdated or unused interim data.

data/lookup/README.md

@@ -0,0 +1,11 @@
+### Folder: Lookup
+
+**Purpose:**
+
+> This folder contains reference data and lookup tables. Lookup tables are used to map values from one dataset to another, often for data enrichment or creating meaningful labels.
+
+**Usage:**
+
+> - Store all reference data, such as code lists, dictionaries, or mappings, in this folder.
+> - Ensure that lookup tables are well-documented, specifying the keys and values they contain.
+> - Keep these tables up to date if any changes occur.

data/processed/README.md

@@ -0,0 +1,10 @@
+### Folder: Processed
+
+**Purpose:**
+
+> The "Processed" folder is for datasets that have gone through some initial processing but are not yet finalized. It might contain partially cleaned or transformed data.
+
+**Usage:**
+
+> - Use this folder for data that is being actively worked on but is not yet ready for analysis.
+> - Document the processing steps that have been applied and any remaining tasks.

data/raw/README.md

@@ -0,0 +1,11 @@
+### Folder: Raw
+
+**Purpose:**
+
+> The "Raw" folder stores the original, untouched data as it was received or collected from its source. It serves as the source of truth and can be used to recreate all other data processing steps.
+
+**Usage:**
+
+> - Data in this folder should not be modified or overwritten.
+> - Consider creating subfolders for different data sources or projects if needed.
+> - Document the source and collection date of the raw data.

data/staging/README.md

@@ -0,0 +1,11 @@
+### Folder: Staging
+
+**Purpose:**
+
+> The "Staging" folder is an intermediary step before data is moved to other folders like "Interim" or "Final." It's used for data that is in the process of being ingested or prepared for analysis.
+
+**Usage:**
+
+> - Data here is temporary and will be moved to other folders once it's ready.
+> - Keep this folder organized and regularly clean up data that has been processed and moved to other folders.
+> - Document the staging process for data ingestion and preparation.