Merge pull request #27 from RolnickLab/pipeline-setup

pipeline-setup

Author: f-PLT

.gemini/GEMINI.md

@@ -1 +1 @@
-You must strictly adhere to the project rules defined in `docs/agents/agent_instructions.md`. Read this file before making any significant architectural, geospatial data processing changes, or general code modifications. Use `read_file` to load `docs/agents/agent_istructions.md`.
+You must strictly adhere to the project rules defined in [docs/agents/agent_instructions.md](../docs/agents/agent_instructions.md). Read this file before making any significant architectural, geospatial data processing changes, or general code modifications. Use `read_file` to load [docs/agents/agent_instructions.md](../docs/agents/agent_instructions.md).

.github/copilot-instructions.md

@@ -1 +1 @@
-You must strictly adhere to the project rules defined in `docs/agents/agent_instructions.md`. Read this file before making any significant architectural, geospatial data processing changes, or general code modifications.
+You must strictly adhere to the project rules defined in [docs/agents/agent_instructions.md](../docs/agents/agent_instructions.md). Read this file before making any significant architectural, geospatial data processing changes, or general code modifications.

.gitignore

@@ -183,3 +183,4 @@ data/*
 scripts/config.sh
 Makefile.private
 configs/geospatial_tools_ini.yaml
+**/settings.local.json

.markdown-link-check.json

@@ -3,21 +3,6 @@
     {
       "pattern": "http://localhost"
     },
-    {
-      "pattern": "tps://github.com/RolnickLab/best-practices-documentation/tree"
-    },
-    {
-      "pattern": "tps://github.com/RolnickLab/best-practices-documentation/issues"
-    },
-    {
-      "pattern": "tps://github.com/RolnickLab/best-practices-documentation/pulls"
-    },
-    {
-      "pattern": "https://github.com/RolnickLab/best-practices-documentation/wiki"
-    },
-    {
-      "pattern": "https://github.com/RolnickLab/best-practices-documentation/wiki/DRAC-%E2%80%90-Compute-Canada"
-    },
     {
       "pattern": "https://github.com/RolnickLab/lab-advanced-template"
     },
@@ -35,6 +20,9 @@
     },
     {
       "pattern": "https://www.earthdata.nasa.gov/"
+    },
+    {
+      "pattern": "https://earthexplorer.usgs.gov"
     }
   ]
 }

CLAUDE.md

@@ -1 +1 @@
-You must strictly adhere to the project rules defined in `docs/agents/agent_instructions.md`. Read this file before making any significant architectural, geospatial data processing changes, or general code modifications.
+You must strictly adhere to the project rules defined in [docs/agents/agent_instructions.md](docs/agents/agent_instructions.md). Read this file before making any significant architectural, geospatial data processing changes, or general code modifications.

docs/agents/README.md

@@ -14,20 +14,13 @@ Yes, these instructions are more prescriptive than *current* best practices, but
 
 The `instructions/` folder contains specific skill files that guide the agent's behavior for particular tasks. Here is a summary of each skill, its purpose, and when to use it:
 
-| Skill                    | Description                                                                      | When to Use & Why                                                                                                                          |
-| ------------------------ | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
-| `analytics.md`           | Extracts truth from experimental data with statistical rigor.                    | Use for Exploratory Data Analysis (EDA) and visualization to ensure reproducibility and prevent misleading claims.                         |
-| `formal_planning.md`     | Enforces a structured planning protocol via a Formal Design Document.            | Use when explicitly asked for a plan, architecture, or proposal to map out scope, trade-offs, and steps before coding.                     |
-| `infrastructure.md`      | Manages reproducible and resilient environments/pipelines as code.               | Use for containerization (Docker), HPC/SLURM cluster setup, or CI/CD tasks to ensure fault tolerance and exact dependency pinning.         |
-| `KNOWLEDGE.md`           | A centralized repository for project-specific tribal knowledge.                  | Use to document or consult specific findings, library quirks, or architectural decisions to avoid repeating past mistakes.                 |
-| `ml.md`                  | Builds state-of-the-art, reproducible, and reliable machine learning pipelines.  | Use for ML model training, evaluation, and experiment management to guarantee strict data isolation and deterministic execution.           |
-| `orchestrator.md`        | Focuses on horizontal integration and strategic decomposition of goals.          | Use for multi-component tasks to define explicit contracts between modules and ensure end-to-end flows work correctly.                     |
-| `plan_to_tasks.md`       | Decomposes high-level plans into modular, atomic, and verifiable tasks.          | Use when transitioning from planning to execution to ensure each step has clear context, acceptance criteria, and testing protocols.       |
-| `python.md`              | Elevates Python scripts into robust, maintainable, and type-safe software.       | Use for all Python development and QA to enforce strict typing, SOLID principles, vectorization, and automated workflows.                  |
-| `root_cause_analysis.md` | Systematically diagnoses and permanently fixes software failures.                | Use when presented with a bug, traceback, or unexpected result to isolate the failure (MRE) and target the actual root cause.              |
-| `security.md`            | Identifies vulnerabilities, enforces defense-in-depth, and ensures data privacy. | Use for tasks involving authentication, untrusted input, or infrastructure to prevent injection attacks and hardcoded secrets.             |
-| `specdrivendev.md`       | Implements lightweight Spec-Driven Development to define contracts first.        | Use when starting a new feature to define data structures, signatures, and docstrings before writing logic, preventing LLM hallucinations. |
-| `systemdesign.md`        | Designs systems that are maintainable, evolvable, and robust.                    | Use for architectural decisions to enforce separation of concerns, configuration-first design, and proper dependency injection.            |
+| Skill                    | Description                                                              | When to Use & Why                                                                                                                      |
+| ------------------------ | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
+| `planning.md`            | Combined Planning, Specification (SDD), and Task Decomposition Protocol. | Use for any multi-step implementation to map out scope, define technical contracts (SDD), and break down work into atomic tasks.       |
+| `software-dev.md`        | Unified Software Development, ML, Analytics & Infrastructure Protocol.   | Use for Python development, system design, ML/geospatial processing, data analysis, and infrastructure/environment management.         |
+| `review.md`              | Harsh Security & Architecture Review Protocol.                           | Use to identify vulnerabilities (tokens, paths, SSL) and tear apart implementations for performance bottlenecks and architectural rot. |
+| `root_cause_analysis.md` | Systematically diagnoses and permanently fixes software failures.        | Use when presented with a bug, traceback, or unexpected result to isolate the failure (MRE) and target the actual root cause.          |
+| `KNOWLEDGE.md`           | A centralized repository for project-specific tribal knowledge.          | Use to document or consult specific findings, library quirks, or architectural decisions to avoid repeating past mistakes.             |
 
 ## How to use them
 
@@ -36,16 +29,36 @@ This template comes with CLAUDE.md and GEMINI.md files that essentially point to
 In practice... it's not always the case. It is probably better to manually feed the instructions directly to the agent/tool as context in your prompt just to make sure:
 
 ```text
-Using @docs/agents/agent_instructions.md, @docs/agents/instructions/python.md, and @docs/agents/instructions/systemdesign.md and @docs/agents/instructions/formal_planning.md, create a plan for a new class that will ...
+Using @docs/agents/agent_instructions.md, @docs/agents/instructions/software-dev.md, and @docs/agents/instructions/planning.md, create a plan for a new class that will ...
 ```
 
-When using models with smaller context windows, it will also be important to clear the context once in a while to ensure better results.
+It's usually a good thing to clear/compress the context once in a while to ensure better results.
 
-For example:
+### Workflow example
 
-- Create plan
-    - Manually revise plan document
-- Clear context
-- Ask agent to implement first step of the plan
-- Clear context
-- Repeat
+First, define **what** you want to do, **how** and **why**. Go in as much detail as you can. Let's call this your *preliminary design document*.
+
+You can also do this step through a chat interface with an LLM so you can brainstorm, ask questions, investigate starting points, development directions and library/tool selections.
+
+Once you have your *preliminary design document*, you are ready to start using your agent.
+
+- Activate `@docs/agents/instructions/planning.md`
+    - **Create PLAN**
+        - Ask agent to create a plan based on your *preliminary design document*
+        - Manually revise plan document
+    - **Create SPEC**
+        - Ask agent to create a specification based on the plan
+        - Manually revise the specification document
+    - **Create TASKS**
+        - Ask agent to create tasks based on the plan and specification created
+        - Manually revise the tasks
+    - **_CLEAR CONTEXT_**
+- Activate `@docs/agents/instructions/code_review.md`
+    - Ask the agent to review your planning documents
+    - **_CLEAR CONTEXT_**
+- Activate `@docs/agents/instructions/software_dev.md`
+    - Ask agent to implement first task of the plan
+    - Manual review
+    - Commit work to git
+    - **_COMPRESS CONTEXT_**
+    - Repeat for subsequent tasks