feat: add postgre and fabric mirror

.gitignore

@@ -14,3 +14,4 @@ __pycache__
 # Local-only Bicep parameter overrides
 infra/*.local.bicepparam
 infra/*.local.bicepparam.json
+copy.main.bicepparam

.gitmodules

@@ -1,3 +1,3 @@
 [submodule "submodules/ai-landing-zone"]
 	path = submodules/ai-landing-zone
-	url = https://github.com/Azure/AI-Landing-Zones.git
+	url = https://github.com/Azure/bicep-ptn-aiml-landing-zone

CHANGELOG.md

@@ -2,13 +2,51 @@
 
 All notable changes to this project will be documented in this file.
 
+## [2026-03-20]
+### Added
+- Read-only PostgreSQL mirroring preflight script for validating runner prerequisites before mirror setup
+- PostgreSQL mirroring follow-up wrapper to run preflight, preparation, and mirror creation as a deliberate post-deployment flow
+- Shared AI Search helper module for OneLake indexing scripts to centralize public network access toggles and tokenized REST calls
+
+### Changed
+- Repository documentation now uses Microsoft Foundry naming more consistently, including the README, deployment verification guide, and related runbooks
+- PostgreSQL mirroring guidance now treats mirroring as a follow-up step after `azd up`, with clearer public-access versus private-network paths
+- Postprovision now restores only PostgreSQL mirroring readiness preparation instead of attempting full mirror creation during the main deployment run
+- PostgreSQL infrastructure outputs now expose the intended Fabric connection identity and default authentication settings needed for mirroring setup
+- Fabric connection and workspace automation now resolve more values from deployment outputs, azd environment values, and deployed resources when transient hook context is incomplete
+- PostgreSQL mirroring scripts now support explicit connection-mode outputs, stronger credential handling, clearer network-path failures, and gateway-aware Fabric connection creation
+- Purview collection and Fabric datasource registration scripts now derive default names and deployment context more reliably from outputs and environment values
+- Fabric workspace and capacity automation now tolerate more incomplete hook context, recover more reliably from existing resources, and improve capacity/workspace lookup behavior
+- Preprovision retries the landing-zone deployment when Foundry account provisioning is still settling instead of failing immediately on transient provisioning-state errors
+- Secure REST helpers now sanitize captured response bodies before surfacing API errors in automation logs
+- Post-deployment and mirroring documentation consolidated the mirror workflow into a single primary runbook and clarified when mirroring should be deferred
+
+### Removed
+- Temporary PostgreSQL mirroring prep wrapper that toggled public access as a separate script
+- Fabric connection probe debug script and the redundant PostgreSQL mirroring opt-in guide
+
+## [2026-03-18]
+### Added
+- Parameter to override Log Analytics workspace resource ID and output mapping for automation scripts
+- Optional `SKIP_PURVIEW_INTEGRATION` guard for Purview automation scripts (used by hooks when Purview is disabled)
+- Retry/timeout handling for AI Search public network access toggles in OneLake indexing scripts
+
+### Changed
+- Preprovision error output simplified with concise failure reason and optional verbose diagnostics
+- Main parameter file reordered into required/optional/defaulted sections with clearer comments
+- OneLake indexing scripts prefer outputs, include AAD-only auth, and handle transient 409 run conflicts
+- Post-deployment steps now include Fabric mirroring checklist items and Key Vault networking guidance for retrieving the `fabric_user` password
+
+### Removed
+- Log Analytics linkage script `scripts/automationScripts/FabricPurviewAutomation/connect_log_analytics.ps1`
+
 ## [1.3] - 2025-12-09
 ### Added
 - Microsoft Fabric integration with automatic capacity creation and management
 - Microsoft Purview integration for governance and data cataloging
 - OneLake indexing pipeline connecting Fabric lakehouses to AI Search
 - Comprehensive post-provision automation (22 hooks for Fabric/Purview/Search setup)
-- New documentation: `deploy_app_from_foundry.md` for publishing apps from AI Foundry
+- New documentation: `deploy_app_from_foundry.md` for publishing apps from Microsoft Foundry
 - New documentation: `TRANSPARENCY_FAQ.md` for responsible AI transparency
 - New documentation: `NewUserGuide.md` for first-time users
 - Header icons matching GSA standard format

azure.yaml

@@ -16,14 +16,8 @@ metadata:
 hooks:
   preprovision:
     # Integrated preprovision:
-    # - Runs AI Landing Zone preprovision to generate deploy/ files and Template Specs
-    # - Ensures our wrapper points to deploy/main.bicep (Template Spec-based) to avoid ARM 4MB template limit
-    # On Windows, `shell: sh` may not be available; the PowerShell script is a fallback.    
-    - shell: sh
-      run: ./scripts/preprovision-integrated.sh
-      interactive: false
-      continueOnError: true
-
+    # - Deploys the AI Landing Zone submodule separately to avoid ARM 4MB template limit
+    # PowerShell is the supported entrypoint in this repo.
     - shell: pwsh
       run: ./scripts/preprovision-integrated.ps1
       interactive: false
@@ -77,6 +71,12 @@ hooks:
       interactive: false
       shell: pwsh
       continueOnError: false
+
+    # Stage 7.5: Prepare PostgreSQL server for Fabric mirroring readiness
+    - run: ./scripts/automationScripts/FabricWorkspace/mirror/prepare_postgresql_for_mirroring.ps1
+      interactive: false
+      shell: pwsh
+      continueOnError: false
 
     # Stage 8: Setup Fabric Workspace Private Link (for VNet integration)
     - run: ./scripts/automationScripts/FabricWorkspace/SecureWorkspace/setup_fabric_private_link.ps1
@@ -138,14 +138,4 @@ hooks:
       shell: pwsh
       continueOnError: false
 
-    # Stage 18: Connect Log Analytics (placeholder)
-    - run: ./scripts/automationScripts/FabricPurviewAutomation/connect_log_analytics.ps1
-      interactive: false
-      shell: pwsh
-      continueOnError: false
 
-    # Stage 19: Clean up AI Landing Zone template specs
-    - run: ./submodules/ai-landing-zone/bicep/scripts/postprovision.ps1
-      interactive: false
-      shell: pwsh
-      continueOnError: false

docs/ACCESSING_PRIVATE_RESOURCES.md

@@ -32,7 +32,7 @@ Once connected to the Jump VM, you can:
 - **Azure AI Search**: Manage indexes via Azure Portal
 - **Storage Account**: Browse blobs via Azure Portal or Storage Explorer
 - **Container Registry**: Push/pull images using Docker CLI
-- **AI Foundry**: Manage projects and deployments
+- **Microsoft Foundry**: Manage projects and deployments
 
 ### 3. Install Tools on Jump VM (Optional)
 

docs/DeploymentGuide.md

@@ -28,7 +28,7 @@ To deploy this solution accelerator, ensure you have access to an [Azure subscri
 | Git | Latest | [Install Git](https://git-scm.com/downloads) |
 | PowerShell | 7.0+ | [Install PowerShell](https://learn.microsoft.com/powershell/scripting/install/installing-powershell) |
 
-> **Windows-specific shell requirement:** Preprovision hooks run with `shell: sh`. Install Git for Windows (includes Git Bash) **or** run `azd` from WSL/Ubuntu so `bash/sh` is on PATH. If you prefer pure PowerShell, update `azure.yaml` to point `preprovision` to the provided `preprovision.ps1`.
+> **Windows shell requirement:** Preprovision runs with PowerShell (`pwsh`). Use PowerShell 7+ so `pwsh` is on PATH.
 
 ### External Resources
 
@@ -106,7 +106,7 @@ If you're not using Codespaces or Dev Containers:
 
 4. Continue with [Deployment Steps](#deployment-steps) below
 
-> **Note (Windows):** Run `azd up` from Git Bash or WSL so the `preprovision` hook can execute. If you want to stay in PowerShell, edit `azure.yaml` to use `preprovision.ps1` instead of the `.sh` script.
+> **Note (Windows):** Run `azd up` from PowerShell 7+ so the `pwsh` preprovision hook can execute.
 
 </details>
 
@@ -152,22 +152,23 @@ Edit `infra/main.bicepparam` or set environment variables:
 | Parameter | Description | Example |
 |-----------|-------------|---------|
 | `purviewAccountResourceId` | Resource ID of existing Purview account | `/subscriptions/.../Microsoft.Purview/accounts/...` |
-| `aiSearchAdditionalAccessObjectIds` | Array of Entra object IDs to grant Search roles | `["00000000-0000-0000-0000-000000000000"]` |
-| `fabricCapacityMode` | Fabric capacity mode: `create`, `byo`, or `none` | `create` |
-| `fabricWorkspaceMode` | Fabric workspace mode: `create`, `byo`, or `none` | `create` |
-| `fabricCapacitySku` | Fabric capacity SKU (only used when `fabricCapacityMode=create`) | `F8` (default) |
-| `fabricCapacityAdmins` | Fabric capacity admin principals (UPN emails or Entra object IDs) (required when `fabricCapacityMode=create`) | `["user@contoso.com"]` |
-| `fabricCapacityResourceId` | Existing Fabric capacity ARM resource ID (required when `fabricCapacityMode=byo`) | `/subscriptions/.../providers/Microsoft.Fabric/capacities/...` |
-| `fabricWorkspaceId` | Existing Fabric workspace ID (GUID) (required when `fabricWorkspaceMode=byo`) | `00000000-0000-0000-0000-000000000000` |
-| `fabricWorkspaceName` | Existing Fabric workspace name (used when `fabricWorkspaceMode=byo`) | `my-existing-workspace` |
+| `fabricCapacityPreset` | Fabric capacity preset: `create`, `byo`, or `none` | `create` |
+| `fabricWorkspacePreset` | Fabric workspace preset: `create`, `byo`, or `none` | `create` |
+| `fabricCapacitySku` | Fabric capacity SKU (only used when `fabricCapacityPreset=create`) | `F8` (default) |
+| `fabricCapacityAdmins` | Fabric capacity admin principals (UPN emails or Entra object IDs) (required when `fabricCapacityPreset=create`) | `["user@contoso.com"]` |
+| `fabricCapacityResourceId` | Existing Fabric capacity ARM resource ID (required when `fabricCapacityPreset=byo`) | `/subscriptions/.../providers/Microsoft.Fabric/capacities/...` |
+| `fabricWorkspaceId` | Existing Fabric workspace ID (GUID) (required when `fabricWorkspacePreset=byo`) | `00000000-0000-0000-0000-000000000000` |
+| `fabricWorkspaceName` | Existing Fabric workspace name (used when `fabricWorkspacePreset=byo`) | `my-existing-workspace` |
 
 ```bash
 # Example: Set Purview account
-azd env set purviewAccountResourceId "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Purview/accounts/<account-name>"
+# (Edit infra/main.bicepparam)
+# param purviewAccountResourceId = "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Purview/accounts/<account-name>"
 
 # Example: Disable all Fabric automation
-azd env set fabricCapacityMode none
-azd env set fabricWorkspaceMode none
+# (Edit infra/main.bicepparam)
+# var fabricCapacityPreset = 'none'
+# var fabricWorkspacePreset = 'none'
 ```
 
 </details>
@@ -177,9 +178,11 @@ azd env set fabricWorkspaceMode none
 
 | Parameter | Description | Default |
 |-----------|-------------|---------|
-| `aiSearchAdditionalAccessObjectIds` | Entra ID object IDs for additional Search access | `[]` |
-| `networkIsolationMode` | Network isolation level | `AllowInternetOutbound` |
-| `vmAdminUsername` | Jump box VM admin username | `azureuser` |
+| `networkIsolation` | Enable network isolation | `false` |
+| `postgreSqlNetworkIsolation` | PostgreSQL private networking toggle (defaults to `networkIsolation`) | `networkIsolation` |
+| `useExistingVNet` | Reuse an existing VNet | `false` |
+| `existingVnetResourceId` | Existing VNet resource ID (when `useExistingVNet=true`) | `` |
+| `vmUserName` | Jump box VM admin username | `` |
 | `vmAdminPassword` | Jump box VM admin password | (prompted) |
 
 </details>
@@ -214,8 +217,8 @@ azd up
 ```
 
 This command will:
-1. Run pre-provision hooks (validate environment)
-2. Deploy all Azure infrastructure (~30-40 minutes)
+1. Run pre-provision hooks (deploy AI Landing Zone submodule)
+2. Deploy Fabric capacity and supporting infrastructure (~30-40 minutes)
 3. Run post-provision hooks (configure Fabric, Purview, Search RBAC)
 
 > **Note:** The entire deployment typically takes 45-60 minutes.
@@ -233,7 +236,7 @@ Running postprovision hooks
   ✓ Lakehouse creation (bronze, silver, gold)
   ✓ Purview registration
   ✓ OneLake indexing setup
-  ✓ AI Foundry RBAC configuration
+  ✓ Microsoft Foundry RBAC configuration
 ```
 
 ### Step 5: Verify Deployment
@@ -265,7 +268,7 @@ Then follow the [Post Deployment Steps](./post_deployment_steps.md) to validate:
 ### Connect Foundry to Search Index
 
 1. Navigate to [ai.azure.com](https://ai.azure.com)
-2. Open your AI Foundry project
+2. Open your Microsoft Foundry project
 3. Go to **Playgrounds** → **Chat**
 4. Click **Add your data** → Select your Search index
 5. Test with a sample query

docs/PARAMETER_GUIDE.md

@@ -5,18 +5,20 @@ This guide focuses on configuration concepts for the **AI Landing Zone**.
 > **Important**: This repository deploys using Bicep parameter files, not `infra/main.parameters.json`.
 >
 > - Primary parameters file: `infra/main.bicepparam`
-> - AI Landing Zone submodule parameters file (if you deploy it directly): `submodules/ai-landing-zone/bicep/infra/main.bicepparam`
+> - AI Landing Zone submodule parameters file (if you deploy it directly): `submodules/ai-landing-zone/main.parameters.json`
 >
 > **Fabric options in this repo** are configured in `infra/main.bicepparam` via:
 > - `fabricCapacityPreset` (`create` | `byo` | `none`)
 > - `fabricWorkspacePreset` (`create` | `byo` | `none`)
 > - BYO inputs: `fabricCapacityResourceId`, `fabricWorkspaceId`, `fabricWorkspaceName`
 
+> **Deployment flow**: This repo deploys the AI Landing Zone submodule from `submodules/ai-landing-zone/main.bicep` during the preprovision hook. The single source of truth for parameters is `infra/main.bicepparam`.
+
 ## Table of Contents
 1. [Basic Parameters](#basic-parameters)
 2. [Deployment Toggles](#deployment-toggles)
 3. [Network Configuration](#network-configuration)
-4. [AI Foundry Configuration](#ai-foundry-configuration)
+4. [Microsoft Foundry Configuration](#microsoft-foundry-configuration)
 5. [Individual Service Configuration](#individual-service-configuration)
 6. [Common Customization Examples](#common-customization-examples)
 
@@ -151,6 +153,14 @@ Each toggle controls whether a service is created. Set to `true` to deploy, `fal
 - `buildVm: true` - For CI/CD build agents
 - `jumpVm: true` - For Windows-based management
 
+### Log Analytics (Optional)
+
+If you are using an existing Log Analytics workspace, set the resource ID in `infra/main.bicepparam`:
+
+```bicep-params
+param logAnalyticsWorkspaceResourceId = '/subscriptions/<subId>/resourceGroups/<rg>/providers/Microsoft.OperationalInsights/workspaces/<name>'
+```
+
 ### Network Security Groups
 
 ```json
@@ -283,11 +293,11 @@ Each toggle controls whether a service is created. Set to `true` to deploy, `fal
 
 ---
 
-## AI Foundry Configuration
+## Microsoft Foundry Configuration
 
 ### aiFoundryDefinition
 
-Controls AI Foundry hub/project and model deployments.
+Controls Microsoft Foundry account/project and model deployments.
 
 ```json
 "aiFoundryDefinition": {
@@ -304,7 +314,7 @@ Controls AI Foundry hub/project and model deployments.
 ### includeAssociatedResources
 **Type**: `boolean`  
 **Default**: `true`  
-**Description**: Create dedicated AI Search, Cosmos DB, Key Vault, and Storage for AI Foundry.
+**Description**: Create dedicated AI Search, Cosmos DB, Key Vault, and Storage for Microsoft Foundry.
 
 Set to `false` if you want to use shared resources.
 
@@ -427,6 +437,29 @@ az cognitiveservices account list-usage \
 
 ## Individual Service Configuration
 
+### PostgreSQL Flexible Server (Repo Wrapper)
+
+Use these in `infra/main.bicepparam` when deploying via this repo. `postgreSqlNetworkIsolation` defaults to `networkIsolation`.
+
+```bicep-params
+param deployPostgreSql = true
+param postgreSqlNetworkIsolation = networkIsolation
+param postgreSqlMirrorConnectionMode = 'fabricUser'
+param postgreSqlAuthConfig = {
+  activeDirectoryAuth: 'Enabled'
+  passwordAuth: 'Enabled'
+}
+```
+
+When `postgreSqlNetworkIsolation` is `false`, PostgreSQL uses public access and does not create private endpoints or private DNS resources.
+
+`postgreSqlAuthConfig` should remain set to both authentication modes enabled if you plan to configure Fabric mirroring after deployment. This ensures the server is created with password authentication available for the `fabric_user` connection instead of relying on a later hook to change the auth mode.
+
+`postgreSqlMirrorConnectionMode` controls which credential the manual Fabric PostgreSQL connection should use after deployment:
+
+- `fabricUser` uses the dedicated least-privilege mirroring user and `postgres-fabric-user-password`. This is the production-oriented default.
+- `admin` uses the PostgreSQL admin login and `postgres-admin-password`. This is intended for demo automation scenarios where you want to avoid creating a separate mirroring user.
+
 ### Storage Account
 
 ```json

docs/Required_roles_scopes_resources.md

@@ -1,4 +1,4 @@
-# Required Roles and Scopes for AI Foundry isolated network template deployment
+# Required Roles and Scopes for Microsoft Foundry isolated network template deployment
 To deploy this code, assign roles with minimal privileges to create and manage necessary Azure resources. Ensure roles are assigned at the appropriate subscription or resource group levels.
 
 ## Role Assignments: 
@@ -13,14 +13,14 @@ Be sure these resource providers are registered in your Azure subscription. To r
 
 | **Resource Type** | **Azure Resource Provider** | **Type** | **Description** |
 |-------------------|----------------------------|----------|-----------------|
-| Application Insights | Microsoft.Insights | /components | An Azure Application Insights instance associated with the Azure AI Foundry Hub |
+| Application Insights | Microsoft.Insights | /components | An Azure Application Insights instance associated with the Microsoft Foundry account |
 |Azure Log Analytics|Microsoft.OperationalInsights|/workspaces|An Azure Log Analytics workspace used to collect diagnostics|
-|Azure Key Vault|Microsoft.KeyVault|/vaults|An Azure Key Vault instance associated with the Azure AI Foundry Hub|
-|Azure Storage Account|Microsoft.Storage|/storageAccounts|An Azure Storage instance associated with the Azure AI Foundry Hub|
-|Azure Container Registry|Microsoft.ContainerRegistry|/registries|An Azure Container Registry instance associated with the Azure AI Foundry Account|
+|Azure Key Vault|Microsoft.KeyVault|/vaults|An Azure Key Vault instance associated with the Microsoft Foundry account|
+|Azure Storage Account|Microsoft.Storage|/storageAccounts|An Azure Storage instance associated with the Microsoft Foundry account|
+|Azure Container Registry|Microsoft.ContainerRegistry|/registries|An Azure Container Registry instance associated with the Microsoft Foundry account|
 |Azure AI Services|Microsoft.CognitiveServices|/accounts|An Azure AI Services as the model-as-a-service endpoint provider including GPT-4o and ADA Text Embeddings model deployments|
-|Azure Virtual Network|Microsoft.Network|/virtualNetworks|A bring-your-own (BYO) virtual network hosting a virtual machine to connect to Azure AI Foundry which will be behind a private endpoint when in network isolation mode. |
+|Azure Virtual Network|Microsoft.Network|/virtualNetworks|A bring-your-own (BYO) virtual network hosting a virtual machine to connect to Microsoft Foundry which will be behind a private endpoint when in network isolation mode. |
 |Bastion Host|Microsoft.Network||A Bastion Host defined in the BYO virtual network that provides RDP connectivity to the jumpbox virtual machine|
 |Azure NAT Gateway|Microsoft.Network|/natGateways|An Azure NAT Gateway that provides outbound connectivity to the jumpbox virtual machine|
-|Azure Private Endpoints|Microsoft.Network|/privateEndpoints|Azure Private Endpoints defined in the BYO virtual network for Azure Container Registry, Azure Key Vault, Azure Storage Account, and Azure AI Foundry Hub/Project|
+|Azure Private Endpoints|Microsoft.Network|/privateEndpoints|Azure Private Endpoints defined in the BYO virtual network for Azure Container Registry, Azure Key Vault, Azure Storage Account, and Microsoft Foundry account/project|
 |Azure Private DNS Zones|Microsoft.Network|/privateDnsZones|Azure Private DNS Zones are used for the DNS resolution of the Azure Private Endpoints|