diff --git a/docs/features/in-cluster-scanning.md b/docs/features/in-cluster-scanning.md index aae563fd..100c3f6c 100644 --- a/docs/features/in-cluster-scanning.md +++ b/docs/features/in-cluster-scanning.md @@ -187,3 +187,260 @@ container in one of the report types, you can adjust the timeout by adding: --set $reportType.timeout=3600 # 3600s = 5min ``` to your `helm update --install` command. + +## Event Watcher + +The `insights-event-watcher` is a specialized Kubernetes component that captures **real-time policy violations** during resource admission. Unlike regular policy scans that find existing violations, the event watcher captures **blocked deployments** as they happen. + +### Purpose & Data Flow: +- **Regular Kyverno Plugin**: Scans existing resources → Reports as Action Items +- **Event Watcher**: Captures blocked admissions → Reports to Admission page +- **Focus**: ValidatingAdmissionPolicy and Kyverno policy violations that **prevent** resource creation + +The component supports both local audit logs and AWS CloudWatch integration for EKS clusters. + +### Key Features + +- **ValidatingAdmissionPolicy Focus**: Primary focus on detecting policy violations that block resource installation +- **Dual Log Sources**: Supports both local audit logs (Kind/local clusters) and CloudWatch logs (EKS clusters) +- **Policy Violation Detection**: Automatically detects and processes policy violations from Kubernetes events +- **CloudWatch Integration**: Real-time processing of EKS audit logs from AWS CloudWatch +- **Insights Integration**: Sends blocked policy violations directly to Fairwinds Insights API for display on the Admission page + +### Enabling Event Watcher + +Event watcher is configured through the Helm chart: + +#### Local Mode (Kind/Local Clusters) +```yaml +insights-event-watcher: + enabled: true + logLevel: "info" + logSource: "local" + auditLogPath: "/var/log/kubernetes/kube-apiserver-audit.log" + resources: + limits: + cpu: 100m + memory: 128Mi + requests: + cpu: 50m + memory: 64Mi +``` + +#### CloudWatch Mode (EKS Clusters) +```yaml +insights-event-watcher: + enabled: true + logLevel: "info" + logSource: "cloudwatch" + cloudwatch: + enabled: true + logGroupName: "/aws/eks/production-eks/cluster" + region: "us-west-2" + filterPattern: "{ $.stage = \"ResponseComplete\" && $.responseStatus.code >= 400 && $.requestURI = \"/api/v1/*\" }" + batchSize: 100 + pollInterval: "30s" + maxMemoryMB: 512 + serviceAccount: + annotations: + eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT_ID:role/insights-watcher-cloudwatch-role" + resources: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 100m + memory: 256Mi +``` + +### Configuration Options + +| Parameter | Description | Default | +|-----------|-------------|---------| +| `insights-event-watcher.enabled` | Enable/disable event watcher | `false` | +| `insights-event-watcher.logLevel` | Log level (debug, info, warn, error) | `info` | +| `insights-event-watcher.logSource` | Log source (local, cloudwatch) | `local` | +| `insights-event-watcher.auditLogPath` | Path to audit log file (local mode) | `/var/log/kubernetes/kube-apiserver-audit.log` | +| `insights-event-watcher.cloudwatch.logGroupName` | CloudWatch log group name | - | +| `insights-event-watcher.cloudwatch.region` | AWS region for CloudWatch | - | +| `insights-event-watcher.cloudwatch.filterPattern` | CloudWatch filter pattern | - | +| `insights-event-watcher.cloudwatch.batchSize` | Events per batch | `100` | +| `insights-event-watcher.cloudwatch.pollInterval` | Poll interval | `30s` | + +### How Event Watcher Works + +1. **Log Source Selection**: Monitors either local audit logs or CloudWatch logs based on configuration +2. **Policy Violation Detection**: Identifies ValidatingAdmissionPolicy and Kyverno policy violations +3. **Event Processing**: Processes events that contain "(blocked)" indicators showing policy enforcement +4. **Insights Transmission**: Sends policy violation events to Fairwinds Insights API for display on the Admission page +5. **Health Monitoring**: Provides health check endpoints for Kubernetes monitoring + +### Monitored Policy Violations + +The event watcher specifically looks for policy violations that **block** resource creation: + +#### ValidatingAdmissionPolicy Events +```bash +Warning PolicyViolation validatingadmissionpolicy/disallow-host-path Deployment default/nginx: [disallow-host-path] fail (blocked); HostPath volumes are forbidden... +``` + +#### Kyverno Policy Events +```bash +Warning PolicyViolation deployment/nginx policy disallow-host-path/disallow-host-path fail (blocked): HostPath volumes are forbidden... +``` + +### CloudWatch Integration (EKS) + +For EKS clusters, the event watcher can process audit logs directly from CloudWatch: + +#### IAM Setup +Create an IAM role with CloudWatch logs permissions: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "logs:DescribeLogGroups", + "logs:DescribeLogStreams", + "logs:FilterLogEvents", + "logs:GetLogEvents" + ], + "Resource": [ + "arn:aws:logs:*:*:log-group:/aws/eks/*/cluster", + "arn:aws:logs:*:*:log-group:/aws/eks/*/cluster:*" + ] + } + ] +} +``` + +#### Service Account Annotation +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: insights-event-watcher + annotations: + eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT_ID:role/insights-watcher-cloudwatch-role" +``` + +### Event Watcher Operations + +#### Check Event Watcher Status +```bash +# View event watcher deployment +kubectl get deployment -n insights-agent insights-event-watcher + +# Check event watcher logs +kubectl logs -n insights-agent -l app=insights-event-watcher + +# View health check endpoints +kubectl port-forward -n insights-agent deployment/insights-event-watcher 8080:8080 +curl localhost:8080/healthz +curl localhost:8080/readyz +``` + +#### Manual Testing +```bash +# Create a policy that blocks resources +kubectl apply -f - <.success.yaml` - Resources that should **pass** validation (policy allows them) +- `.failure.yaml` - Resources that should **fail** validation (policy blocks them) +- `.testcase.yaml` - Named test cases with specific expected outcomes + +**Example Directory Structure:** +``` +kyverno-policies/ +├── require-labels.yaml # The policy file +├── require-labels.success.yaml # Resources that should pass +├── require-labels.failure.yaml # Resources that should fail +└── require-labels.testcase-edge.yaml # Additional test case +``` + +**Example Policy (`require-labels.yaml`):** +```yaml +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: require-labels +spec: + validationFailureAction: Enforce + rules: + - name: check-labels + match: + any: + - resources: + kinds: + - Pod + validate: + message: "Required labels missing" + pattern: + metadata: + labels: + app: "?*" + version: "?*" +``` + +**Success Test Case (`require-labels.success.yaml`):** +```yaml +# This pod should PASS validation (has required labels) +apiVersion: v1 +kind: Pod +metadata: + name: good-pod + labels: + app: my-app + version: v1.0.0 +spec: + containers: + - name: app + image: nginx +--- +# Another pod that should PASS +apiVersion: v1 +kind: Pod +metadata: + name: another-good-pod + labels: + app: web-server + version: v2.1.0 +spec: + containers: + - name: web + image: httpd +``` + +**Failure Test Case (`require-labels.failure.yaml`):** +```yaml +# This pod should FAIL validation (missing required labels) +apiVersion: v1 +kind: Pod +metadata: + name: bad-pod + labels: + environment: production # Missing 'app' and 'version' labels +spec: + containers: + - name: app + image: nginx +--- +# Another pod that should FAIL +apiVersion: v1 +kind: Pod +metadata: + name: incomplete-pod + labels: + app: my-app # Missing 'version' label +spec: + containers: + - name: app + image: nginx +``` + +**Named Test Case (`require-labels.testcase-edge.yaml`):** +```yaml +# Test edge case: pod with empty label values (should fail) +apiVersion: v1 +kind: Pod +metadata: + name: empty-labels-pod + labels: + app: "" # Empty value should fail + version: v1.0.0 +spec: + containers: + - name: app + image: nginx +``` + +**Validate a single policy with test resources:** + +```bash +# Validate policy against a specific test resource +insights-cli validate kyverno-policies -r require-labels.yaml -k test-pod.yaml + +# Validate policy against multiple test files +insights-cli validate kyverno-policies -r require-labels.yaml -k good-pod.yaml,bad-pod.yaml +``` + +**Validate a directory of policies (recommended):** + +```bash +# Validate all policies in directory with their test cases +insights-cli validate kyverno-policies -b ./kyverno-policies + +# This will automatically: +# 1. Find all .yaml policy files +# 2. Look for corresponding .success.yaml and .failure.yaml test files +# 3. Run validation tests and report results + +# Validate only specific policies and their test cases +insights-cli validate kyverno-policies -b ./kyverno-policies -p require-labels,disallow-privileged +``` + +**Understanding Validation Output:** + +When validation runs, you'll see output like: +``` +✓ require-labels.yaml + ✓ require-labels.success.yaml (2 resources passed as expected) + ✓ require-labels.failure.yaml (2 resources failed as expected) + ✓ require-labels.testcase-edge.yaml (1 resource failed as expected) + +✗ disallow-privileged.yaml + ✗ disallow-privileged.success.yaml (1 resource failed unexpectedly) + ✓ disallow-privileged.failure.yaml (1 resource failed as expected) +``` + +**Validate policies for a specific cluster:** + +```bash +insights-cli validate kyverno-policies --cluster production +``` + +This validates all policies that would be deployed to the cluster (with app groups applied). + +#### Namespaced Policies + +For namespaced policy kinds (`Policy`, `NamespacedValidatingPolicy`), ensure the `metadata.namespace` field is set in your policy file: + +```yaml +apiVersion: kyverno.io/v1 +kind: Policy +metadata: + name: namespace-specific-policy + namespace: production # Required for namespaced policies +spec: + # ... policy spec +``` + ### Teams Management Teams in **Fairwinds Insights** can be managed in two ways: diff --git a/docs/features/integrations.md b/docs/features/integrations.md index 5d1e2138..a12b75cd 100644 --- a/docs/features/integrations.md +++ b/docs/features/integrations.md @@ -60,10 +60,10 @@ The linked account will show as the creator of any tickets created via Insights ### Jira Custom Fields Fairwinds Insights supports advanced Jira custom field mapping, allowing you to: -* Map Jira fields to custom Insights fields -* Configure default values and validation rules -* Create context-aware field configurations -* Support 30+ Jira field types including text, select, user pickers, and more +* Map existing Jira fields to Insights custom fields for enhanced ticket creation +* Configure default values and display preferences +* Support all standard Jira field types discovered through the API +* Create sophisticated field mappings with validation and constraints See [Jira Custom Fields](/features/jira-custom-fields) for detailed configuration instructions. diff --git a/docs/features/kyverno-policies.md b/docs/features/kyverno-policies.md new file mode 100644 index 00000000..eb0828e8 --- /dev/null +++ b/docs/features/kyverno-policies.md @@ -0,0 +1,415 @@ +--- +meta: + - name: description + content: "Fairwinds Insights | Documentation: Manage and deploy Kyverno policies across your Kubernetes clusters using App Groups and Policy Mappings." +--- + +# Kyverno Policies + +## About + +Fairwinds Insights allows you to centrally manage Kyverno policies and automatically deploy them to your Kubernetes clusters. By combining [App Groups](/features/app-groups) and [Policy Mappings](/features/policy-mappings), you can create sophisticated policy deployment strategies that apply different policies to different resources across your clusters. + +### Key Benefits + +- **Centralized Management**: Manage all Kyverno policies from a single location +- **Cluster-Specific Deployment**: Automatically deploy policies to specific clusters based on App Group criteria +- **Policy Scoping**: Use App Groups to control which resources policies apply to +- **Enforcement Control**: Configure whether policies block or audit via Policy Mappings +- **Automatic Sync**: Policies are automatically synced to clusters when changes are made + +## Prerequisites + +Before managing Kyverno policies in Insights, ensure: + +1. **Kyverno is installed** in your clusters (see [Kyverno installation guide](https://kyverno.io/docs/installation/)) +2. **Kyverno plugin is enabled** in your Insights Agent (see [technical documentation](/technical-details/reports/kyverno)) +3. **Kyverno Policy Sync is enabled** for clusters where you want to deploy policies (see [Policy Sync configuration](/technical-details/reports/kyverno#kyverno-policy-sync)) + +## Creating Kyverno Policies + +Kyverno policies can be managed using the Insights CLI. This allows you to manage policies as code and keep them in version control. + +> **Note**: Currently, Kyverno policies can only be created and managed via the CLI. The Insights UI can be used to view policies, but creation and editing must be done through the CLI. + +### Via the Insights CLI + +You can manage Kyverno policies as code using the Insights CLI. This is recommended for managing policies in version control. + +See the [Insights CLI documentation](/features/insights-cli#kyverno-policies) for detailed instructions. + +**Quick example:** + +```bash +# Create a directory for your policies +mkdir -p kyverno-policies + +# Create a policy file +cat > kyverno-policies/require-resource-limits.yaml < --format yaml` + +## Troubleshooting + +### Policy Not Appearing in Cluster + +1. **Check App Group criteria**: Ensure the App Group matches the cluster +2. **Check Policy Mapping**: Verify the Policy Mapping is enabled and has the correct context +3. **Check Sync status**: Verify the policy sync has completed +4. **Check cluster configuration**: Ensure Kyverno Policy Sync is enabled for the cluster + +### Policy Not Enforcing + +1. **Check `block` setting**: Verify `block: true` is set in the Policy Mapping +2. **Check context**: Ensure the context includes `Admission` or `CI` for blocking +3. **Check Admission Controller**: Verify Admission Controller is enabled and in blocking mode +4. **Check policy spec**: Verify the policy's `validationFailureAction` is set correctly + +### Policy Validation Errors + +1. **Review error messages**: The validation provides specific error details +2. **Check Kyverno version**: Ensure your policy is compatible with your Kyverno version +3. **Check policy syntax**: Verify JSON/YAML syntax is correct +4. **Check required fields**: Ensure all required fields are present for the policy kind + +### Transformation Issues + +1. **Check App Group criteria**: Verify App Group criteria are valid +2. **Check rule names**: Rule names are automatically generated; very long names may be truncated +3. **Review transformed YAML**: Download and review the final YAML to see how policies are transformed + +## Related Documentation + +- [App Groups](/features/app-groups) - Learn how to define resource groups +- [Policy Mappings](/features/policy-mappings) - Learn how to map policies to App Groups +- [Insights CLI - Kyverno Policies](/features/insights-cli#kyverno-policies) - Manage policies as code +- [Kyverno Technical Details](/technical-details/reports/kyverno) - Setup, installation, and Policy Sync configuration +- [Event Watcher](/features/in-cluster-scanning#event-watcher) - Monitor cluster events for policy violations +- [Admission Controller](/features/admission-controller) - Enforce policies at deployment time + diff --git a/docs/features/policy-mappings.md b/docs/features/policy-mappings.md index a03ec0bf..53284f74 100644 --- a/docs/features/policy-mappings.md +++ b/docs/features/policy-mappings.md @@ -104,7 +104,7 @@ spec: ## Blocking and Enforcement The `block` configuration in a Policy Mapping only applies to Admission Controllers in Blocking mode (e.g., Passive Mode is disabled) and Repo Scans configured to fail pipelines. -- `block` is `null` or not present (default) - Insights will only block if the Action Item has a severity of High or Critical +- `block` is `null` or not present (default) - Insights will only block if the Action Item has a severity of High or Critical - in the case of Kyverno policies managed by Insights, it keeps the behaviour from the original policy - `block: true` - Insights will always block if any of the `policies` are present - `block: false` - Insights will never block (but continue to report Action Items) if any of the `policies` are present diff --git a/docs/technical-details/architecture/architecture.md b/docs/technical-details/architecture/architecture.md index 378cd520..2fdde8d8 100644 --- a/docs/technical-details/architecture/architecture.md +++ b/docs/technical-details/architecture/architecture.md @@ -29,6 +29,37 @@ open up access for any kind of network ingress. Any exceptions or additions to this flow are listed below. +## Additional Components + +### Kyverno Policy Sync +The `kyverno-policy-sync` component runs as a CronJob (default: every 5 minutes) to: +* Fetch Kyverno policies from the Insights API based on App Group and Policy Mapping configurations +* Use kubectl to apply, update, and delete policies +* Only manage policies with `insights.fairwinds.com/owned-by: "Fairwinds Insights"` annotation +* Use Kubernetes Lease-based leader election to prevent concurrent operations +* Support all Kyverno policy types (ClusterPolicy, Policy, ValidatingAdmissionPolicy, etc.) +* Provide comprehensive logging of all sync operations + +**Requirements:** +* RBAC permissions to manage all Kyverno policy types and Leases for leader election +* Network access to Insights API +* kubectl installed in the container +* Kyverno CRDs installed in the cluster + +### Event Watcher +The `insights-event-watcher` runs as a Deployment to: +* Monitor policy-related resources and events, focusing on ValidatingAdmissionPolicy violations +* Support both local audit logs (Kind/local clusters) and AWS CloudWatch logs (EKS clusters) +* Detect policy violations that block resource installation (events containing "(blocked)") +* Send policy violation events to Insights API for correlation with Action Items +* Provide health check endpoints for Kubernetes monitoring + +**Requirements:** +* RBAC permissions to watch events, ValidatingAdmissionPolicies, and Kyverno resources +* Network access to Insights API +* For CloudWatch mode: IAM permissions to access CloudWatch logs (IRSA recommended) +* Access to audit logs (local mode) or CloudWatch log groups (EKS mode) + ### Goldilocks In addition to the typical cronjob, Goldilocks will run a long lived Deployment for controlling VPA objects. Goldilocks will add a VPA (in `recommend` mode) to any incoming Deployment in order to diff --git a/docs/technical-details/reports/kyverno.md b/docs/technical-details/reports/kyverno.md index 579c5d24..04f19050 100644 --- a/docs/technical-details/reports/kyverno.md +++ b/docs/technical-details/reports/kyverno.md @@ -10,12 +10,25 @@ Refer to the [Kyverno documentation](https://kyverno.io/docs/) for using Kyverno Here is an example of installing Kyverno with the `test-org` organization and `prod` cluster: ### Install `insights-agent` with helm and enable Kyverno plugin + +**For Kyverno Policy Scanning (reports Action Items):** +```yaml +insights: + organization: "test-org" + cluster: "prod" +kyverno: + enabled: true +``` + +**For Kyverno Policy Sync (deploys policies from Insights):** ```yaml insights: organization: "test-org" cluster: "prod" kyverno: enabled: true +kyverno-policy-sync: + enabled: true ``` ```bash @@ -134,3 +147,287 @@ kubectl -n default patch deployment nginx-deployment --type JSON -p '[{"op":"rep ``` You should see `PolicyReport`s with each `Policy` name. Under the displayed column with status `fail`, you should see that those are now resolved. The corresponding Action Items should also be resolved after the `kyverno` insights job runs again. + +## Kyverno Policy Sync + +The `kyverno-policy-sync` is a **separate component** from the Kyverno report plugin. While the Kyverno plugin **scans existing policies** and reports violations as Action Items, the policy sync component **deploys and manages policies** from Insights to your clusters. + +### Key Distinction: +- **Kyverno Plugin**: Scans → Reports violations as Action Items +- **Policy Sync**: Fetches policies from Insights → Deploys to cluster + +The `kyverno-policy-sync` automatically keeps your cluster's Kyverno policies in sync with the policies defined in Fairwinds Insights. + +### Key Features + +- **Automatic Policy Synchronization**: Keeps cluster policies in sync with Insights +- **Insights-Managed Only**: Only affects policies with `insights.fairwinds.com/owned-by: "Fairwinds Insights"` annotation +- **Distributed Locking**: Uses Kubernetes Lease-based leader election for preventing concurrent operations +- **kubectl Integration**: Uses `kubectl apply` and `kubectl delete` for all policy operations +- **Multi-Resource Support**: Supports all Kyverno policy types (ClusterPolicy, Policy, ValidatingAdmissionPolicy, etc.) +- **Dry-Run Mode**: Preview changes before applying them +- **Comprehensive Logging**: Detailed audit trail of all operations + +### Enabling Policy Sync + +Policy sync is typically deployed as a CronJob through the Helm chart: + +```yaml +kyverno-policy-sync: + enabled: true + # CronJob schedule (every 5 minutes) + schedule: "*/5 * * * *" + # Dry run mode - preview changes without applying + dryRun: false + # Log level (debug, info, warn, error) + logLevel: "info" + # Resource configuration + resources: + limits: + memory: "256Mi" + cpu: "200m" + requests: + memory: "128Mi" + cpu: "100m" +``` + +### Configuration Options + +The `kyverno-policy-sync` section supports the following options: + +- `enabled`: Enable/disable policy sync (required) +- `schedule`: CronJob schedule (default: `"*/5 * * * *"`) +- `dryRun`: Preview changes without applying (default: `false`) +- `logLevel`: Log level - debug, info, warn, error (default: `"info"`) +- `resources`: Resource limits and requests for the sync job + +### Environment Variables + +The sync uses environment variables for configuration (automatically set by Helm): + +- `FAIRWINDS_INSIGHTS_HOST`: Insights API host (required) +- `FAIRWINDS_TOKEN`: Insights API token (required) +- `FAIRWINDS_ORG`: Organization name (required) +- `FAIRWINDS_CLUSTER`: Cluster name (required) +- `DRY_RUN`: Enable dry-run mode (optional, default: false) +- `LOG_LEVEL`: Log level (optional, default: info) + +### How Policy Sync Works + +1. **Fetch Policies**: Retrieves expected policies from Insights API based on: + - Organization and cluster configuration + - App Group criteria matching the cluster + - Policy Mapping configurations +2. **Compare States**: Compares expected policies with currently deployed policies +3. **Determine Actions**: Identifies policies to apply, update, or remove +4. **Acquire Lock**: Uses Kubernetes Lease-based leader election to prevent concurrent operations +5. **Execute Changes**: Applies, updates, or removes policies using kubectl commands +6. **Release Lock**: Releases the leader election lock +7. **Report Results**: Logs comprehensive results of the sync operation + +### Policy Management + +#### Policy Ownership +Only policies with the following annotation are managed by the sync: + +```yaml +metadata: + annotations: + insights.fairwinds.com/owned-by: "Fairwinds Insights" +``` + +#### Policy Operations +The sync performs operations using kubectl: + +1. **Apply**: New policies from Insights + ```bash + kubectl apply -f policy.yaml + ``` + +2. **Update**: Existing policies that have changed + ```bash + kubectl apply -f policy.yaml # Handles both create and update + ``` + +3. **Remove**: Policies no longer in Insights + ```bash + kubectl delete clusterpolicy policy-name + ``` + +#### Supported Policy Types +The sync supports all Kyverno policy resource types: +- `clusterpolicies` (ClusterPolicy) +- `policies` (Policy) +- `validatingpolicies` (ValidatingPolicy) +- `validatingadmissionpolicies` (ValidatingAdmissionPolicy) +- `clustercleanuppolicies` (ClusterCleanupPolicy) +- `imagevalidatingpolicies` (ImageValidatingPolicy) +- `mutatingpolicies` (MutatingPolicy) +- `generatingpolicies` (GeneratingPolicy) +- `deletingpolicies` (DeletingPolicy) +- `namespacedvalidatingpolicies` (NamespacedValidatingPolicy) +- `policyexceptions` (PolicyException) + +### Distributed Locking + +The sync uses Kubernetes Lease-based leader election for distributed locking: + +#### Lock Mechanism +- **Lease Name**: `kyverno-policy-sync-lock` +- **Namespace**: Current pod namespace (or `default`) +- **Lock Duration**: 15 seconds +- **Renew Deadline**: 10 seconds +- **Retry Period**: 2 seconds +- **Identity**: Pod name or job name + +#### Lock Operations +```bash +# Check lease status +kubectl get lease kyverno-policy-sync-lock -n + +# View lease details +kubectl describe lease kyverno-policy-sync-lock -n + +# Manual lease release (if needed - lease will expire automatically) +kubectl delete lease kyverno-policy-sync-lock -n +``` + +### Monitoring Policy Sync + +#### Check Sync Job Status +```bash +# View recent sync jobs +kubectl get jobs -n insights-agent -l app=kyverno-policy-sync + +# Check CronJob status +kubectl get cronjob -n insights-agent kyverno-policy-sync + +# View job logs +kubectl logs -n insights-agent -l job-name=kyverno-policy-sync- +``` + +#### Manual Policy Sync +```bash +# Trigger a manual sync +kubectl create job -n insights-agent \ + --from=cronjob/kyverno-policy-sync \ + manual-policy-sync-$(date +%s) +``` + +#### Verify Deployed Policies +```bash +# List all cluster policies +kubectl get clusterpolicy + +# Check policies managed by Insights +kubectl get clusterpolicy -l insights.fairwinds.com/owned-by="Fairwinds Insights" + +# View policy details +kubectl describe clusterpolicy +``` + +### Sync Results Logging + +The sync provides detailed logging for monitoring: + +```json +{ + "level": "info", + "msg": "Policy sync completed", + "success": true, + "duration": "5.2s", + "summary": "Applied 2, Updated 1, Removed 0, Failed 0", + "applied": ["policy1", "policy2"], + "updated": ["policy3"], + "removed": [], + "failed": [] +} +``` + +### Troubleshooting Policy Sync + +#### Common Issues + +**Leader Election Lock Exists** +Another sync operation is running or waiting for leadership: +```bash +# Check lease details +kubectl get lease kyverno-policy-sync-lock -n insights-agent -o yaml + +# View current leader +kubectl describe lease kyverno-policy-sync-lock -n insights-agent + +# Manual lease release (if confirmed safe - lease expires automatically) +kubectl delete lease kyverno-policy-sync-lock -n insights-agent +``` + +**Policy Application Failed** +Policy has syntax errors or kubectl apply fails: +```bash +# Check job logs for application errors +kubectl logs -n insights-agent -l job-name=kyverno-policy-sync- | grep -i error + +# Test policy manually +kubectl apply -f policy.yaml --dry-run=client +``` + +**API Authentication Failed** +Invalid token or permissions: +```bash +# Verify API token is configured +kubectl get secret -n insights-agent insights-token + +# Check token permissions in logs +kubectl logs -n insights-agent -l job-name=kyverno-policy-sync- | grep -i auth +``` + +**Kubernetes API Errors** +Cluster connectivity or RBAC issues: +```bash +# Check RBAC permissions +kubectl auth can-i create clusterpolicies --as=system:serviceaccount:insights-agent:kyverno-policy-sync + +# Verify network connectivity +kubectl exec -n insights-agent -- curl -I https://insights.fairwinds.com +``` + +### RBAC Requirements + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: kyverno-policy-sync +rules: +# All Kyverno policy types +- apiGroups: ["kyverno.io"] + resources: ["policies", "clusterpolicies"] + verbs: ["get", "list", "create", "update", "patch", "delete"] +- apiGroups: ["wgpolicyk8s.io"] + resources: ["validatingpolicies", "namespacedvalidatingpolicies"] + verbs: ["get", "list", "create", "update", "patch", "delete"] +- apiGroups: ["admissionregistration.k8s.io"] + resources: ["validatingadmissionpolicies"] + verbs: ["get", "list", "create", "update", "patch", "delete"] +- apiGroups: ["kyverno.io"] + resources: ["clustercleanuppolicies", "imagevalidatingpolicies", "mutatingpolicies", "generatingpolicies", "deletingpolicies", "policyexceptions"] + verbs: ["get", "list", "create", "update", "patch", "delete"] +# Lease for leader election +- apiGroups: ["coordination.k8s.io"] + resources: ["leases"] + verbs: ["get", "list", "create", "update", "patch", "delete"] +# Events for logging +- apiGroups: [""] + resources: ["events"] + verbs: ["create"] +``` + +### Best Practices + +1. **Start with Dry Run**: Enable `dryRun: true` initially to preview changes +2. **Monitor Leader Election**: Watch for leadership conflicts or extended wait times +3. **Resource Limits**: Set appropriate resource limits for the sync job +4. **Backup Policies**: Keep local backups of critical policies before enabling sync +5. **Staged Rollout**: Test policy changes in non-production clusters first +6. **Monitor Logs**: Regularly review sync logs for errors or warnings +7. **Policy Ownership**: Ensure only policies managed by Insights have the ownership annotation