feat: add postgre and fabric mirror by mswantek68 · Pull Request #117 · microsoft/Deploy-Your-AI-Application-In-Production

mswantek68 · 2026-03-12T23:11:42Z

Purpose

Replace sub module location, add postgreSQL provisioning, add Fabric mirror of PostgreSQL DB

Does this introduce a breaking change?

[ X] Yes
No

Golden Path Validation

[ x] I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

[ x] I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid
Follow post_deployment_steps.md for listing

Other Information

As reviewed with team on 3-10-26

…reating workspace.

…osoft/Deploy-Your-AI-Application-In-Production into feat/addPostgreFabricMirror

Copilot

Pull request overview

This PR updates the AI Landing Zone submodule source and reworks provisioning/automation to add PostgreSQL Flexible Server provisioning plus Fabric mirroring support, while improving resiliency of post-provision scripts (outputs-based configuration, retries, and safer error handling).

Changes:

Switch AI Landing Zone submodule to a new upstream repo and change preprovision to deploy the submodule directly via az deployment group create.
Add PostgreSQL Flexible Server provisioning in the wrapper Bicep, plus PowerShell automation to prep PostgreSQL for Fabric mirroring and create a mirrored database.
Improve automation scripts to prefer AZURE_OUTPUTS_JSON, add retries/guardrails, and introduce a Purview skip flag.

Reviewed changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
submodules/ai-landing-zone	Updates submodule commit pointer to the new upstream state.
scripts/preprovision-integrated.ps1	Replaces preprovision flow with direct submodule deployment + env output publishing.
scripts/automationScripts/SecurityModule.ps1	Enhances REST error sanitization and adds HTTP status/body diagnostics.
scripts/automationScripts/OneLakeIndex/setup_ai_services_rbac.ps1	Adds AI Foundry project existence checks to improve RBAC setup reliability.
scripts/automationScripts/OneLakeIndex/06_setup_ai_foundry_search_rbac.ps1	Pulls settings from outputs, discovers project, and improves additional principals parsing.
scripts/automationScripts/OneLakeIndex/05_create_onelake_indexer.ps1	Adds public access toggle + retry handling for Search requests and RBAC propagation.
scripts/automationScripts/OneLakeIndex/04_create_onelake_datasource.ps1	Same Search public-access toggle + retry layer; prefers outputs for config.
scripts/automationScripts/OneLakeIndex/03_create_onelake_index.ps1	Same Search public-access toggle + retry layer; prefers outputs for config.
scripts/automationScripts/OneLakeIndex/02_create_onelake_skillsets.ps1	Same Search public-access toggle + retry layer; prefers outputs for config.
scripts/automationScripts/OneLakeIndex/01_setup_rbac.ps1	Prefers outputs for Search/Foundry values used during RBAC.
scripts/automationScripts/FabricWorkspace/mirror/run_postgresql_mirroring_prep_with_public_access.ps1	Adds wrapper to temporarily enable public access for KV/PG during mirroring prep.
scripts/automationScripts/FabricWorkspace/mirror/prepare_postgresql_for_mirroring.ps1	Adds PostgreSQL mirroring preparation (params, roles/grants, seed table).
scripts/automationScripts/FabricWorkspace/mirror/create_postgresql_mirror.ps1	Adds Fabric API automation for creating a mirrored database once a connection exists.
scripts/automationScripts/FabricWorkspace/CreateWorkspace/register_fabric_datasource.ps1	Adds `SKIP_PURVIEW_INTEGRATION` short-circuit.
scripts/automationScripts/FabricWorkspace/CreateWorkspace/materialize_document_folders.ps1	Improves retry behavior and idempotency checks for folder materialization.
scripts/automationScripts/FabricWorkspace/CreateWorkspace/create_fabric_workspace.ps1	Moves from Power BI to Fabric API and updates admin/capacity assignment flows.
scripts/automationScripts/FabricPurviewAutomation/trigger_purview_scan_for_fabric_workspace.ps1	Adds skip flag and retry logic around scan create/run.
scripts/automationScripts/FabricPurviewAutomation/create_purview_collection.ps1	Adds skip flag to avoid running Purview setup.
scripts/automationScripts/FabricPurviewAutomation/connect_log_analytics.ps1	Removes placeholder Log Analytics linkage script.
infra/main.bicepparam	Reorganizes parameters and adds PostgreSQL + mirror-related inputs/toggles.
infra/main.bicep	Adds PostgreSQL provisioning and updates wrapper outputs for scripts.
docs/quota_check.md	Clarifies terminal requirements for quota script vs deployment.
docs/postgresql_mirroring.md	Adds end-to-end runbook for Fabric mirroring with what’s automated vs manual.
docs/post_deployment_steps.md	Adds PostgreSQL mirroring verification steps and updates network isolation wording.
docs/automation-outputs-mapping.md	Updates mapping descriptions for resolved Fabric modes.
docs/PARAMETER_GUIDE.md	Updates submodule param references and documents PostgreSQL options.
docs/DeploymentGuide.md	Updates Windows shell guidance and reflects new preprovision/deployment flow.
azure.yaml	Switches preprovision to `pwsh` and adds PostgreSQL mirroring steps.
README.md	Updates architecture image, upstream link, and adds PostgreSQL mirroring doc link.
CHANGELOG.md	Adds Unreleased notes for new behaviors and script changes.
.gitmodules	Updates submodule URL to new upstream repository.
.gitignore	Ignores a new local copy parameter file.

Comments suppressed due to low confidence (3)

scripts/preprovision-integrated.ps1:1

The GUID validation regex ^[0-9a-fA-F-]{36}$ is too permissive (it accepts many invalid GUID strings). Use [guid]::TryParse(...) (or a stricter GUID regex) to ensure AZURE_PRINCIPAL_ID / principalId is a valid GUID before relying on it for RBAC and deployment parameters.
scripts/preprovision-integrated.ps1:1
The GUID validation regex ^[0-9a-fA-F-]{36}$ is too permissive (it accepts many invalid GUID strings). Use [guid]::TryParse(...) (or a stricter GUID regex) to ensure AZURE_PRINCIPAL_ID / principalId is a valid GUID before relying on it for RBAC and deployment parameters.
scripts/preprovision-integrated.ps1:1
The Select-String pattern only matches parameters that start at column 0. If the submodule formats parameters with indentation (or has leading whitespace), valid params will be missed and silently dropped from $filtered.parameters, causing deployment failures due to missing required params. Use a pattern that allows leading whitespace (e.g., ^\\s*param\\s+...).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/automationScripts/FabricWorkspace/CreateWorkspace/create_fabric_workspace.ps1

infra/main.bicepparam

azure.yaml

infra/main.bicep

scripts/automationScripts/SecurityModule.ps1

CHANGELOG.md

Copilot

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 8 comments.

Comments suppressed due to low confidence (2)

scripts/preprovision-integrated.ps1:1

The GUID validation regex ^[0-9a-fA-F-]{36}$ is overly permissive (it accepts many non-GUID 36-char strings with hyphens in the wrong places). This can lead to invalid principalId being accepted and later causing RBAC/deployment failures. Prefer strict GUID parsing (e.g., [guid]::TryParse(...)) for validation.
scripts/preprovision-integrated.ps1:1
The GUID validation regex ^[0-9a-fA-F-]{36}$ is overly permissive (it accepts many non-GUID 36-char strings with hyphens in the wrong places). This can lead to invalid principalId being accepted and later causing RBAC/deployment failures. Prefer strict GUID parsing (e.g., [guid]::TryParse(...)) for validation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/automationScripts/FabricWorkspace/CreateWorkspace/create_fabric_workspace.ps1

...utomationScripts/FabricWorkspace/mirror/run_postgresql_mirroring_prep_with_public_access.ps1

scripts/automationScripts/SecurityModule.ps1

scripts/automationScripts/OneLakeIndex/03_create_onelake_index.ps1

scripts/automationScripts/FabricWorkspace/mirror/prepare_postgresql_for_mirroring.ps1

Copilot

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 10 comments.

Comments suppressed due to low confidence (1)

scripts/preprovision-integrated.ps1:1

The regex ^param\s+(\w+) only matches param at column 1. In Bicep, parameters can be indented and/or preceded by decorators, so this can miss valid params and produce an incomplete $filteredParams file (leading to ARM/Bicep validation errors due to missing required parameters). Use a whitespace-tolerant pattern (e.g., ^\s*param\s+(\w+)) and consider accounting for decorator lines (or using az bicep build --stdout/a safer parsing approach) so the allowlist is accurate.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/automationScripts/FabricWorkspace/CreateWorkspace/create_fabric_workspace.ps1

+  $groups = Invoke-SecureRestMethod -Uri "$apiRoot/groups?%24top=5000" -Headers $apiHeaders -Method Get -ErrorAction Stop
  $g = $groups.value | Where-Object { $_.name -eq $WorkspaceName }


scripts/automationScripts/FabricWorkspace/CreateWorkspace/create_fabric_workspace.ps1

+        }
+      }
+
+      Invoke-SecureWebRequest -Uri "$apiRoot/workspaces/$workspaceId/roleAssignments" -Method Post -Headers ($apiHeaders) -Body (@{ principal = @{ id = $pincipalId; type = 'User' }; role = 'Admin' } | ConvertTo-Json) -ErrorAction Stop


scripts/automationScripts/FabricWorkspace/mirror/prepare_postgresql_for_mirroring.ps1

+# Set server parameters for mirroring
+$changed = $false
+$needsRestart = $false


scripts/automationScripts/FabricWorkspace/mirror/prepare_postgresql_for_mirroring.ps1

+function Set-ParamValue([string]$paramName, [string]$value, [bool]$requiresRestart) {
+  $current = Get-ParamValue $paramName
+  if ($current -ne $value) {
+    Log "Setting $paramName to '$value' (was '$current')"
+    az postgres flexible-server parameter set -g $resourceGroup -s $postgreSqlServerName -n $paramName --value $value --subscription $subscriptionId 1>$null
+    $script:changed = $true
+    if ($requiresRestart) { $script:needsRestart = $true }
+  }
+}


scripts/automationScripts/FabricWorkspace/mirror/prepare_postgresql_for_mirroring.ps1

+  Set-ParamValue -paramName 'max_worker_processes' -value $targetWorkers.ToString() -requiresRestart $true
+}
+
+if ($changed -and $needsRestart) {


scripts/automationScripts/FabricWorkspace/mirror/prepare_postgresql_for_mirroring.ps1

+# Increase max_worker_processes by 3 per mirrored database
+$maxWorkers = Get-ParamValue 'max_worker_processes'
+if ($maxWorkers -and $maxWorkers -as [int]) {
+  $currentWorkers = [int]$maxWorkers
+  $targetWorkers = $currentWorkers + (3 * $MirrorCount)
+  Set-ParamValue -paramName 'max_worker_processes' -value $targetWorkers.ToString() -requiresRestart $true


azure.yaml


    # Stage 5: Purview Collection Creation
-    - run: ./scripts/automationScripts/FabricPurviewAutomation/create_purview_collection.ps1
+    - run: "$env:SKIP_PURVIEW_INTEGRATION='true'; ./scripts/automationScripts/FabricPurviewAutomation/create_purview_collection.ps1"


azure.yaml


    # Stage 6: Register Fabric as Purview Data Source
-    - run: ./scripts/automationScripts/FabricWorkspace/CreateWorkspace/register_fabric_datasource.ps1
+    - run: "$env:SKIP_PURVIEW_INTEGRATION='true'; ./scripts/automationScripts/FabricWorkspace/CreateWorkspace/register_fabric_datasource.ps1"


azure.yaml


    # Stage 17: Trigger Purview Scan (if Purview enabled)
-    - run: ./scripts/automationScripts/FabricPurviewAutomation/trigger_purview_scan_for_fabric_workspace.ps1
+    - run: "$env:SKIP_PURVIEW_INTEGRATION='true'; ./scripts/automationScripts/FabricPurviewAutomation/trigger_purview_scan_for_fabric_workspace.ps1"


CHANGELOG.md

+### Fixed
+- Power BI headers initialization in Log Analytics linkage script to resolve workspace ID lookups
+


…irror

Mike Swantek and others added 13 commits March 5, 2026 11:52

Update submodule integration flow

5a4b478

Pin ai-landing-zone submodule to v1.0.1

5467ae3

Swap in Azure AI landing zone submodule and pin to v1.0.1

8c76feb

Add postgresql provisioning and fabric automation for mirror

d167a97

doc updates and notation update postgresql mirror scripts

81d391b

remove values

e7df29b

Fabric - minor param cleanup, use Fabric API over Power BI API when c…

2daac4a

…reating workspace.

Merge branch 'feat/addPostgreFabricMirror' of https://github.com/micr…

a1f8bb7

…osoft/Deploy-Your-AI-Application-In-Production into feat/addPostgreFabricMirror

Infra - minor param fix

cf8b27d

Script modifications, script order of operations and documentation adds

c2fd16a

Merge branch 'feat/addPostgreFabricMirror' of https://github.com/micr…

8d716ad

…osoft/Deploy-Your-AI-Application-In-Production into feat/addPostgreFabricMirror

Documentation and postprovision steps updated

531dc53

Adjust post deployment step md

4ee31df

mswantek68 requested review from Copilot and sethsteenken March 12, 2026 23:11

mswantek68 requested review from Avijit-Microsoft, Prajwal-Microsoft, Roopan-Microsoft, Vinay-Microsoft, aniaroramsft and dgp10801 as code owners March 12, 2026 23:11

Copilot AI reviewed Mar 12, 2026

View reviewed changes

mswantek68 changed the title ~~feat! add postgre and fabric mirror~~ feat add postgre and fabric mirror Mar 18, 2026

mswantek68 requested a review from Copilot March 18, 2026 20:02

Copilot AI reviewed Mar 18, 2026

View reviewed changes

mswantek68 changed the title ~~feat add postgre and fabric mirror~~ feat: add postgre and fabric mirror Mar 18, 2026

update docs and post deployment steps

17ec603

mswantek68 requested a review from Copilot March 19, 2026 13:01

Copilot AI reviewed Mar 19, 2026

View reviewed changes

docs: rename AI Foundry references to Microsoft Foundry

60bcbd0

Mike Swantek added 3 commits March 20, 2026 13:20

harden Fabric mirroring automation and align docs with Microsoft Foundry

90fa399

update change log

e45220e

update parameters to add postgres with mixed auth to connect fabric m…

ba7050f

…irror

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add postgre and fabric mirror#117

feat: add postgre and fabric mirror#117
mswantek68 wants to merge 18 commits intomainfrom
feat/addPostgreFabricMirror

mswantek68 commented Mar 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		$groups = Invoke-SecureRestMethod -Uri "$apiRoot/groups?%24top=5000" -Headers $apiHeaders -Method Get -ErrorAction Stop
		$g = $groups.value \| Where-Object { $_.name -eq $WorkspaceName }

		### Fixed
		- Power BI headers initialization in Log Analytics linkage script to resolve workspace ID lookups

Conversation

mswantek68 commented Mar 12, 2026

Purpose

Does this introduce a breaking change?

Golden Path Validation

Deployment Validation

What to Check

Other Information

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants