Add Azure Log Analytics tools for HPCC component analysis and cost tracking

[Copilot]

Description

Tools for querying Azure Log Analytics to analyze HPCC component resource usage and correlate with infrastructure costs. Addresses the need to understand which HPCC components keep Azure VMs active and contribute to operational expenses.

Implementation

azure_log_analytics_fetch.py - KQL query tool (320 lines)

• Queries KubeNodeInventory and KubePodInventory tables
• Time range and namespace filtering
• CSV output with metadata header documenting query parameters
• Supports Azure CLI and service principal authentication

azure_log_analytics_analyze.py - Component categorization (501 lines)

• Pattern-based pod categorization into HPCC components: dali, thor, roxie, esp, eclagent, eclccserver, eclscheduler, dfuserver, sasha, dafilesrv
• Non-HPCC system pod categorization: kubernetes-system, azure-system, monitoring, logging, ingress
• Time-series output suitable for Gantt visualization
• Summary reports with component distribution and node usage statistics

test_categorization.py - Validation suite (164 lines)

• 29 test cases covering HPCC and system component patterns
• 100% pass rate

Usage

# Fetch inventory data
./azure_log_analytics_fetch.py \
  --subscription-id <sub-id> \
  --workspace-id <workspace-id> \
  --aks-name <cluster> \
  --start "2024-11-01T00:00:00Z" \
  --end "2024-11-02T00:00:00Z" \
  --output results.csv

# Analyze and categorize
./azure_log_analytics_analyze.py \
  --input results.csv \
  --output analysis.csv \
  --summary summary.txt

Output enables correlation of component lifecycles with VM costs, resource optimization analysis, and capacity planning based on historical usage patterns.

Type of change:

• This change is a new feature (non-breaking change which adds functionality).

Checklist:

• My code follows the code style of this project.
  • My code does not create any new warnings from compiler, build system, or lint.
• The commit message is properly formatted and free of typos.
  • The commit message title makes sense in a changelog, by itself.
  • The commit is signed.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly, or...
  • I have created a JIRA ticket to update the documentation.
  • Any new interfaces or exported functions are appropriately commented.
• I have read the CONTRIBUTORS document.
• The change has been fully tested:
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • I have checked that this change does not introduce memory leaks.
  • I have used Valgrind or similar tools to check for potential issues.
• I have given due consideration to all of the following potential concerns:
  • Scalability
  • Performance
  • Security
  • Thread-safety
  • Cloud-compatibility
  • Premature optimization
  • Existing deployed queries will not be broken
  • This change fixes the problem, not just the symptom
  • The target branch of this pull request is appropriate for such a change.
• There are no similar instances of the same problem that should be addressed
  • I have addressed them here
  • I have raised JIRA issues to address them separately
• This is a user interface / front-end modification
  • I have tested my changes in multiple modern browsers
  • The component(s) render as expected

Testing:

Comprehensive test suite validates component categorization logic:

• 17 HPCC component patterns (dali, thor, roxie, esp, eclagent, eclccserver, eclscheduler, dfuserver, sasha, dafilesrv)
• 10 system component patterns (kubernetes-system, azure-system, monitoring, logging, ingress)
• 2 unknown component fallback cases

Integration tested with sample AKS inventory data covering realistic deployment scenarios. Python syntax validated, no security vulnerabilities introduced.

Original prompt

write a python program that access Azure Log Analytics via KQL for a given date+time range, for a given subscription and aks name, and pulls information about nodes and pods that were running from KubeNodeInventory and KubePodInventory and outputs in a clean csv format. The output should contain details about the command ran in a comment block header.
By default it should process all namespaces, but a optional filter should allow select namespaces to be fetched.
This tool should live in tools/azure.

Write a 2nd python tool, that compliments this tool, that is designed to analyze the csv output from the 1st. This analyze tool should categorize the pods into HPCC components based on their name (from knowledge gleaned from the helm/hpcc chart). It should also categorize non-hpcc pods.

The goal of this analytic tools is to create a time-series of categorized components and what resources (nodes) they were using, such that [separatley] this could be visualized as a gantt chart or similar, and insights into what components were responsible for was keeping the VM's up (and hence costing money).

If you have any questions, ask before proceeding.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.