feat: add postgre and fabric mirror

.gitignore

@@ -14,3 +14,4 @@ __pycache__
 # Local-only Bicep parameter overrides
 infra/*.local.bicepparam
 infra/*.local.bicepparam.json
+copy.main.bicepparam

.gitmodules

@@ -1,3 +1,3 @@
 [submodule "submodules/ai-landing-zone"]
 	path = submodules/ai-landing-zone
-	url = https://github.com/Azure/AI-Landing-Zones.git
+	url = https://github.com/Azure/bicep-ptn-aiml-landing-zone

CHANGELOG.md

@@ -2,6 +2,20 @@
 
 All notable changes to this project will be documented in this file.
 
+## [Unreleased] - 2026-03-06
+### Added
+- Parameter to override Log Analytics workspace resource ID and output mapping for automation scripts
+- Optional `SKIP_PURVIEW_INTEGRATION` guard for Purview automation scripts (used by hooks when Purview is disabled)
+- Retry/timeout handling for AI Search public network access toggles in OneLake indexing scripts
+
+### Changed
+- Preprovision error output simplified with concise failure reason and optional verbose diagnostics
+- Main parameter file reordered into required/optional/defaulted sections with clearer comments
+- OneLake indexing scripts prefer outputs, include AAD-only auth, and handle transient 409 run conflicts
+
+### Fixed
+- Power BI headers initialization in Log Analytics linkage script to resolve workspace ID lookups
+
 ## [1.3] - 2025-12-09
 ### Added
 - Microsoft Fabric integration with automatic capacity creation and management

README.md

@@ -22,7 +22,7 @@ This accelerator extends the [AI Landing Zone](https://github.com/Azure/ai-landi
 
 ### Solution Architecture
 
-| ![Architecture](./img/Architecture/AI-Landing-Zone-without-platform.png) |
+| ![Architecture](./img/Architecture/Deploy-AI-App-in-Prod-Architecture_final.png) |
 |---|
 
 ### Key Components
@@ -39,7 +39,7 @@ This accelerator extends the [AI Landing Zone](https://github.com/Azure/ai-landi
 
 ### Additional Resources
 
-- [AI Landing Zone Documentation](https://github.com/Azure/ai-landing-zone)
+- [AI Landing Zone Documentation](https://github.com/Azure/bicep-ptn-aiml-landing-zone)
 - [Azure AI Foundry Documentation](https://learn.microsoft.com/en-us/azure/ai-foundry/)
 - [Microsoft Fabric Documentation](https://learn.microsoft.com/en-us/fabric/)
 
@@ -104,7 +104,7 @@ Follow the deployment guide to deploy this solution to your own Azure subscripti
 > **GitHub Codespaces and Dev Containers handle this automatically.**
 
 >  **Windows shell note**
-> <br/>Preprovision uses `shell: sh`. Run `azd` from Git Bash/WSL so `bash` is available, or switch the `preprovision` hook in `azure.yaml` to the provided PowerShell script if you want to stay in PowerShell.
+> <br/>Preprovision runs with PowerShell (`pwsh`) by default. Run `azd` from PowerShell 7+ (or any terminal that can invoke `pwsh`).
 
 <br/>
 
@@ -223,6 +223,7 @@ Supporting documentation
 |----------|-------------|
 | [Deployment Guide](./docs/DeploymentGuide.md) | Complete deployment instructions |
 | [Post Deployment Steps](./docs/post_deployment_steps.md) | Verify your deployment |
+| [PostgreSQL Mirroring](./docs/postgresql_mirroring.md) | Create the Fabric connection and mirror PostgreSQL |
 | [Parameter Guide](./docs/PARAMETER_GUIDE.md) | Configure deployment parameters |
 | [Quota Check Guide](./docs/quota_check.md) | Check Azure OpenAI quota availability |
 

azure.yaml

@@ -16,14 +16,8 @@ metadata:
 hooks:
   preprovision:
     # Integrated preprovision:
-    # - Runs AI Landing Zone preprovision to generate deploy/ files and Template Specs
-    # - Ensures our wrapper points to deploy/main.bicep (Template Spec-based) to avoid ARM 4MB template limit
-    # On Windows, `shell: sh` may not be available; the PowerShell script is a fallback.    
-    - shell: sh
-      run: ./scripts/preprovision-integrated.sh
-      interactive: false
-      continueOnError: true
-
+    # - Deploys the AI Landing Zone submodule separately to avoid ARM 4MB template limit
+    # PowerShell is the supported entrypoint in this repo.
     - shell: pwsh
       run: ./scripts/preprovision-integrated.ps1
       interactive: false
@@ -61,13 +55,13 @@ hooks:
       continueOnError: false
 
     # Stage 5: Purview Collection Creation
-    - run: ./scripts/automationScripts/FabricPurviewAutomation/create_purview_collection.ps1
+    - run: "$env:SKIP_PURVIEW_INTEGRATION='true'; ./scripts/automationScripts/FabricPurviewAutomation/create_purview_collection.ps1"
       interactive: false
       shell: pwsh
       continueOnError: false
 
     # Stage 6: Register Fabric as Purview Data Source
-    - run: ./scripts/automationScripts/FabricWorkspace/CreateWorkspace/register_fabric_datasource.ps1
+    - run: "$env:SKIP_PURVIEW_INTEGRATION='true'; ./scripts/automationScripts/FabricWorkspace/CreateWorkspace/register_fabric_datasource.ps1"
       interactive: false
       shell: pwsh
       continueOnError: false
@@ -77,6 +71,18 @@ hooks:
       interactive: false
       shell: pwsh
       continueOnError: false
+
+    # Stage 7.4: Prepare PostgreSQL for Fabric mirroring (server params + role)
+    - run: ./scripts/automationScripts/FabricWorkspace/Mirror/run_postgresql_mirroring_prep_with_public_access.ps1
+      interactive: false
+      shell: pwsh
+      continueOnError: false
+
+    # Stage 7.5: Create PostgreSQL Mirrored Database (if PostgreSQL is provisioned)
+    - run: ./scripts/automationScripts/FabricWorkspace/Mirror/create_postgresql_mirror.ps1
+      interactive: false
+      shell: pwsh
+      continueOnError: false
 
     # Stage 8: Setup Fabric Workspace Private Link (for VNet integration)
     - run: ./scripts/automationScripts/FabricWorkspace/SecureWorkspace/setup_fabric_private_link.ps1
@@ -133,19 +139,9 @@ hooks:
       continueOnError: false
 
     # Stage 17: Trigger Purview Scan (if Purview enabled)
-    - run: ./scripts/automationScripts/FabricPurviewAutomation/trigger_purview_scan_for_fabric_workspace.ps1
+    - run: "$env:SKIP_PURVIEW_INTEGRATION='true'; ./scripts/automationScripts/FabricPurviewAutomation/trigger_purview_scan_for_fabric_workspace.ps1"
       interactive: false
       shell: pwsh
       continueOnError: false
 
-    # Stage 18: Connect Log Analytics (placeholder)
-    - run: ./scripts/automationScripts/FabricPurviewAutomation/connect_log_analytics.ps1
-      interactive: false
-      shell: pwsh
-      continueOnError: false
 
-    # Stage 19: Clean up AI Landing Zone template specs
-    - run: ./submodules/ai-landing-zone/bicep/scripts/postprovision.ps1
-      interactive: false
-      shell: pwsh
-      continueOnError: false

docs/DeploymentGuide.md

@@ -28,7 +28,7 @@ To deploy this solution accelerator, ensure you have access to an [Azure subscri
 | Git | Latest | [Install Git](https://git-scm.com/downloads) |
 | PowerShell | 7.0+ | [Install PowerShell](https://learn.microsoft.com/powershell/scripting/install/installing-powershell) |
 
-> **Windows-specific shell requirement:** Preprovision hooks run with `shell: sh`. Install Git for Windows (includes Git Bash) **or** run `azd` from WSL/Ubuntu so `bash/sh` is on PATH. If you prefer pure PowerShell, update `azure.yaml` to point `preprovision` to the provided `preprovision.ps1`.
+> **Windows shell requirement:** Preprovision runs with PowerShell (`pwsh`). Use PowerShell 7+ so `pwsh` is on PATH.
 
 ### External Resources
 
@@ -106,7 +106,7 @@ If you're not using Codespaces or Dev Containers:
 
 4. Continue with [Deployment Steps](#deployment-steps) below
 
-> **Note (Windows):** Run `azd up` from Git Bash or WSL so the `preprovision` hook can execute. If you want to stay in PowerShell, edit `azure.yaml` to use `preprovision.ps1` instead of the `.sh` script.
+> **Note (Windows):** Run `azd up` from PowerShell 7+ so the `pwsh` preprovision hook can execute.
 
 </details>
 
@@ -152,22 +152,23 @@ Edit `infra/main.bicepparam` or set environment variables:
 | Parameter | Description | Example |
 |-----------|-------------|---------|
 | `purviewAccountResourceId` | Resource ID of existing Purview account | `/subscriptions/.../Microsoft.Purview/accounts/...` |
-| `aiSearchAdditionalAccessObjectIds` | Array of Entra object IDs to grant Search roles | `["00000000-0000-0000-0000-000000000000"]` |
-| `fabricCapacityMode` | Fabric capacity mode: `create`, `byo`, or `none` | `create` |
-| `fabricWorkspaceMode` | Fabric workspace mode: `create`, `byo`, or `none` | `create` |
-| `fabricCapacitySku` | Fabric capacity SKU (only used when `fabricCapacityMode=create`) | `F8` (default) |
-| `fabricCapacityAdmins` | Fabric capacity admin principals (UPN emails or Entra object IDs) (required when `fabricCapacityMode=create`) | `["user@contoso.com"]` |
-| `fabricCapacityResourceId` | Existing Fabric capacity ARM resource ID (required when `fabricCapacityMode=byo`) | `/subscriptions/.../providers/Microsoft.Fabric/capacities/...` |
-| `fabricWorkspaceId` | Existing Fabric workspace ID (GUID) (required when `fabricWorkspaceMode=byo`) | `00000000-0000-0000-0000-000000000000` |
-| `fabricWorkspaceName` | Existing Fabric workspace name (used when `fabricWorkspaceMode=byo`) | `my-existing-workspace` |
+| `fabricCapacityPreset` | Fabric capacity preset: `create`, `byo`, or `none` | `create` |
+| `fabricWorkspacePreset` | Fabric workspace preset: `create`, `byo`, or `none` | `create` |
+| `fabricCapacitySku` | Fabric capacity SKU (only used when `fabricCapacityPreset=create`) | `F8` (default) |
+| `fabricCapacityAdmins` | Fabric capacity admin principals (UPN emails or Entra object IDs) (required when `fabricCapacityPreset=create`) | `["user@contoso.com"]` |
+| `fabricCapacityResourceId` | Existing Fabric capacity ARM resource ID (required when `fabricCapacityPreset=byo`) | `/subscriptions/.../providers/Microsoft.Fabric/capacities/...` |
+| `fabricWorkspaceId` | Existing Fabric workspace ID (GUID) (required when `fabricWorkspacePreset=byo`) | `00000000-0000-0000-0000-000000000000` |
+| `fabricWorkspaceName` | Existing Fabric workspace name (used when `fabricWorkspacePreset=byo`) | `my-existing-workspace` |
 
 ```bash
 # Example: Set Purview account
-azd env set purviewAccountResourceId "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Purview/accounts/<account-name>"
+# (Edit infra/main.bicepparam)
+# param purviewAccountResourceId = "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Purview/accounts/<account-name>"
 
 # Example: Disable all Fabric automation
-azd env set fabricCapacityMode none
-azd env set fabricWorkspaceMode none
+# (Edit infra/main.bicepparam)
+# var fabricCapacityPreset = 'none'
+# var fabricWorkspacePreset = 'none'
 ```
 
 </details>
@@ -177,9 +178,11 @@ azd env set fabricWorkspaceMode none
 
 | Parameter | Description | Default |
 |-----------|-------------|---------|
-| `aiSearchAdditionalAccessObjectIds` | Entra ID object IDs for additional Search access | `[]` |
-| `networkIsolationMode` | Network isolation level | `AllowInternetOutbound` |
-| `vmAdminUsername` | Jump box VM admin username | `azureuser` |
+| `networkIsolation` | Enable network isolation | `false` |
+| `postgreSqlNetworkIsolation` | PostgreSQL private networking toggle (defaults to `networkIsolation`) | `networkIsolation` |
+| `useExistingVNet` | Reuse an existing VNet | `false` |
+| `existingVnetResourceId` | Existing VNet resource ID (when `useExistingVNet=true`) | `` |
+| `vmUserName` | Jump box VM admin username | `` |
 | `vmAdminPassword` | Jump box VM admin password | (prompted) |
 
 </details>
@@ -214,8 +217,8 @@ azd up
 ```
 
 This command will:
-1. Run pre-provision hooks (validate environment)
-2. Deploy all Azure infrastructure (~30-40 minutes)
+1. Run pre-provision hooks (deploy AI Landing Zone submodule)
+2. Deploy Fabric capacity and supporting infrastructure (~30-40 minutes)
 3. Run post-provision hooks (configure Fabric, Purview, Search RBAC)
 
 > **Note:** The entire deployment typically takes 45-60 minutes.

docs/PARAMETER_GUIDE.md

@@ -5,13 +5,15 @@ This guide focuses on configuration concepts for the **AI Landing Zone**.
 > **Important**: This repository deploys using Bicep parameter files, not `infra/main.parameters.json`.
 >
 > - Primary parameters file: `infra/main.bicepparam`
-> - AI Landing Zone submodule parameters file (if you deploy it directly): `submodules/ai-landing-zone/bicep/infra/main.bicepparam`
+> - AI Landing Zone submodule parameters file (if you deploy it directly): `submodules/ai-landing-zone/main.parameters.json`
 >
 > **Fabric options in this repo** are configured in `infra/main.bicepparam` via:
 > - `fabricCapacityPreset` (`create` | `byo` | `none`)
 > - `fabricWorkspacePreset` (`create` | `byo` | `none`)
 > - BYO inputs: `fabricCapacityResourceId`, `fabricWorkspaceId`, `fabricWorkspaceName`
 
+> **Deployment flow**: This repo deploys the AI Landing Zone submodule from `submodules/ai-landing-zone/main.bicep` during the preprovision hook. The single source of truth for parameters is `infra/main.bicepparam`.
+
 ## Table of Contents
 1. [Basic Parameters](#basic-parameters)
 2. [Deployment Toggles](#deployment-toggles)
@@ -151,6 +153,14 @@ Each toggle controls whether a service is created. Set to `true` to deploy, `fal
 - `buildVm: true` - For CI/CD build agents
 - `jumpVm: true` - For Windows-based management
 
+### Log Analytics (Optional)
+
+If you are using an existing Log Analytics workspace, set the resource ID in `infra/main.bicepparam`:
+
+```bicep-params
+param logAnalyticsWorkspaceResourceId = '/subscriptions/<subId>/resourceGroups/<rg>/providers/Microsoft.OperationalInsights/workspaces/<name>'
+```
+
 ### Network Security Groups
 
 ```json
@@ -427,6 +437,17 @@ az cognitiveservices account list-usage \
 
 ## Individual Service Configuration
 
+### PostgreSQL Flexible Server (Repo Wrapper)
+
+Use these in `infra/main.bicepparam` when deploying via this repo. `postgreSqlNetworkIsolation` defaults to `networkIsolation`.
+
+```bicep-params
+param deployPostgreSql = true
+param postgreSqlNetworkIsolation = networkIsolation
+```
+
+When `postgreSqlNetworkIsolation` is `false`, PostgreSQL uses public access and does not create private endpoints or private DNS resources.
+
 ### Storage Account
 
 ```json

docs/automation-outputs-mapping.md

@@ -20,8 +20,8 @@ The postprovision automation scripts consume deployment outputs via the `AZURE_O
 
 | Bicep Output | Script Variable | Used By | Purpose |
 |-------------|-----------------|---------|---------|
-| `fabricCapacityModeOut` | `fabricCapacityMode` | Multiple Fabric scripts | Whether capacity is `create`, `byo`, or `none` |
-| `fabricWorkspaceModeOut` | `fabricWorkspaceMode` | Multiple Fabric scripts | Whether workspace is `create`, `byo`, or `none` |
+| `fabricCapacityModeOut` | `fabricCapacityMode` | Multiple Fabric scripts | Resolved mode from `fabricCapacityPreset` (`create`, `byo`, `none`) |
+| `fabricWorkspaceModeOut` | `fabricWorkspaceMode` | Multiple Fabric scripts | Resolved mode from `fabricWorkspacePreset` (`create`, `byo`, `none`) |
 | `fabricCapacityId` | `FABRIC_CAPACITY_ID` | `ensure_active_capacity.ps1` | ARM resource ID of Fabric capacity |
 | `fabricCapacityResourceIdOut` | `fabricCapacityId` | `create_fabric_workspace.ps1` | Resource ID for capacity assignment |
 | `fabricWorkspaceIdOut` | `FABRIC_WORKSPACE_ID` | Multiple Fabric scripts | Existing or created Fabric workspace ID |

docs/post_deployment_steps.md

@@ -10,6 +10,7 @@ After running `azd up` or `azd provision` followed by `azd hooks run postprovisi
 |-----------|---------------|----------------|
 | Fabric Capacity | Azure Portal → Microsoft Fabric capacities | **Active** (not Paused) |
 | Fabric Workspace | [app.fabric.microsoft.com](https://app.fabric.microsoft.com) | Workspace visible with 3 lakehouses |
+| PostgreSQL Mirroring (if enabled) | Fabric → Workspace → Connections/Mirror | Connection saved and mirror running |
 | AI Foundry Project | [ai.azure.com](https://ai.azure.com) | Project accessible, models deployed |
 | AI Search Index | Azure Portal → AI Search → Indexes | `onelake-index` exists with documents |
 | Purview Scan | Purview Portal → Data Map → Sources | Fabric data source registered |
@@ -46,6 +47,18 @@ az fabric capacity resume --capacity-name <capacity-name> --resource-group <rg-n
 
 5. Open the **bronze** lakehouse and verify the `Files/documents` folder structure exists
 
+### PostgreSQL Mirroring (if enabled)
+
+Use these short steps to create the Fabric connection and enable mirroring. For full details and troubleshooting, see [PostgreSQL mirroring](./postgresql_mirroring.md).
+
+0. In **Azure Portal** → **Key Vault** → your vault → **Networking**, set **Public access** to **Allow public access from specific virtual networks and IP addresses**, add your client IP, then **Apply**. This lets you read the `fabric_user` password from the vault.
+   After you retrieve the secret, remove your IP and **Apply** again to re-lock the vault.
+1. In Fabric, open the workspace, then select **Connections** → **New** → **PostgreSQL**.
+2. Use the PostgreSQL server name, database name, and the `fabric_user` credentials stored in Key Vault.
+3. Test the connection and **Save**.
+4. In the workspace, select **New** → **Data pipeline** → **Mirror database**.
+5. Pick the PostgreSQL connection, select the target database, and **Start mirroring**.
+
 ---
 
 ## 3. Verify AI Foundry Project
@@ -106,7 +119,7 @@ If no documents appear, check:
 
 ## 6. Verify Network Isolation (if enabled)
 
-When `networkIsolationMode` is set to isolate resources:
+When `networkIsolation` is set to `true`:
 
 ### Check AI Foundry Network Settings