jan/retries-and-degradations-analysis

[jan-osch]

• Analysis of degradations and retries
  

[coderabbitai]

Walkthrough

This update introduces functionality for analyzing degradation and retry metrics over time. A new type and prompt function are added to create validated timelines for operational data analysis. A dedicated analysis module processes check results into time buckets and aggregates degradation and retry counts. In addition, the Slackbot components are updated: the check summary block interface now supports additional analysis outputs, and the check summary command concurrently executes both heatmap and retry/degradation analysis, integrating all results into the final output.

Changes

File(s)	Change Summary
src/prompts/checkly.ts	Added the DegradationAndRetriesTimeline type and the analyseDegradationAndRetriesTimeline function, which constructs a prompt (with Zod schema validation) for analyzing time-series data on degradations and retries.
src/slackbot/analysis/analyseRetriesAndDegradations.ts	Introduced new functions: analyseRetriesAndDegradations (for analyzing check result timelines) and prepareData (to aggregate results into 30-minute time buckets with degradation, retry, and failure counts).
src/slackbot/blocks/newCheckSummaryBlock.ts	Updated the CheckStats interface by adding optional retriesAnalysis and degradationsAnalysis properties, and modified the block generation to conditionally include these new sections in the failure summary.
src/slackbot/commands/check-summary.ts	Modified the checkSummary function to import and execute analyseRetriesAndDegradations concurrently with the heatmap analysis using Promise.all, then integrate the additional analysis results into the final output.

Possibly related PRs

• Use only FINAL results in the single check analysis #88: Directly relates to the new analyseDegradationAndRetriesTimeline function usage in processing analysis data.
• Account summary #68: Also connected at the code level through the integration of timeline analysis functionality in processing degradation and retry metrics.

Suggested reviewers

• aluedeke
• tnolet

✨ Finishing Touches

• 📝 Generate Docstrings

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[tnolet]

holy smokes, this feels so weird.

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

src/prompts/checkly.ts (1)

701-796: Comprehensive implementation for analyzing degradations and retries

The function creates a detailed prompt for analyzing patterns in timeline data, with clear instructions for the AI model to identify relationships between degradations, retries, and failures. The schema validation ensures consistent output format.

Consider adding JSDoc documentation for this function to improve developer experience, similar to how other prompt functions like contextAnalysisSummaryPrompt (line 586) are documented.

+/**
+ * Generates a prompt definition for analyzing degradation and retries timeline data.
+ * Creates a structured prompt for identifying patterns in degradations and retries
+ * that might serve as leading indicators of failures.
+ *
+ * @param {DegradationAndRetriesTimeline} timelineData - Timeline data containing degradations and retries
+ * @param {number} totalRetries - The total number of retries across all periods
+ * @param {number} totalDegradations - The total number of degradations across all periods
+ * @returns {PromptDefinition} A prompt definition for timeline analysis
+ */
 export function analyseDegradationAndRetriesTimeline(
   timelineData: DegradationAndRetriesTimeline,
   totalRetries: number,
   totalDegradations: number,
 ): PromptDefinition {

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 5cdde12 and a7c0b72.

📒 Files selected for processing (1)

• src/prompts/checkly.ts (2 hunks)

🧰 Additional context used

🧬 Code Definitions (1)

src/prompts/checkly.ts (1)

src/prompts/common.ts (2)

• PromptDefinition (18-25)
• defineMessagesPrompt (62-76)

⏰ Context from checks skipped due to timeout of 90000ms (2)

• GitHub Check: build
• GitHub Check: build

🔇 Additional comments (2)

src/prompts/checkly.ts (2)

691-699: Well-structured type definition for timeline data

The DegradationAndRetriesTimeline type provides a clear structure for time-series operational data, properly defining the hierarchy of period timestamps and region-specific metrics.

---

710-710: Consider previous feedback on system prompt

A previous reviewer commented "holy smokes, this feels so weird" on this exact line. While the detailed instructions in the system prompt are valuable for guiding the AI's analysis, consider reviewing this section to ensure clarity and necessity of all instructions.

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

src/prompts/checkly.ts (1)

701-796: Comprehensive prompt construction for degradation and retry analysis.

The function effectively creates a structured prompt for analyzing time-series operational data with clear instructions and expected output format. The schema validation ensures consistent response structure.

Two considerations:

1. There's no validation that totalRetries and totalDegradations are non-negative numbers.
2. The analysis is constrained to "maximum 1 sentence per field" (line 748), which might limit expression of complex patterns.

Consider adding parameter validation:

export function analyseDegradationAndRetriesTimeline(
  timelineData: DegradationAndRetriesTimeline,
  totalRetries: number,
  totalDegradations: number,
): PromptDefinition {
+  // Validate parameters
+  if (totalRetries < 0 || totalDegradations < 0) {
+    throw new Error("Total retries and degradations must be non-negative");
+  }

  const messages: CoreMessage[] = [
    // Rest of function...

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a7c0b72 and 066a5cb.

📒 Files selected for processing (4)

• src/prompts/checkly.ts (2 hunks)
• src/slackbot/analysis/analyseRetriesAndDegradations.ts (1 hunks)
• src/slackbot/blocks/newCheckSummaryBlock.ts (3 hunks)
• src/slackbot/commands/check-summary.ts (3 hunks)

🚧 Files skipped from review as they are similar to previous changes (3)

• src/slackbot/blocks/newCheckSummaryBlock.ts
• src/slackbot/commands/check-summary.ts
• src/slackbot/analysis/analyseRetriesAndDegradations.ts

🧰 Additional context used

🧬 Code Definitions (1)

src/prompts/checkly.ts (1)

src/prompts/common.ts (2)

• PromptDefinition (18-25)
• defineMessagesPrompt (62-76)

⏰ Context from checks skipped due to timeout of 90000ms (2)

• GitHub Check: build
• GitHub Check: build

🔇 Additional comments (2)

src/prompts/checkly.ts (2)

691-699: Well-defined type for timeline analysis.

The DegradationAndRetriesTimeline type is structured clearly with appropriate properties for time-based analysis of operational data across regions.

---

710-710: Address previous feedback.

I notice there was a previous comment from tnolet: "holy smokes, this feels so weird." This line is now part of your new system message. Consider reviewing and addressing the previous concern if it's still relevant to the current implementation.