From cbc13366c6c633439814d7f5ae1e7d0a98413f2b Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 2 Feb 2026 03:25:37 +0000 Subject: [PATCH] Add comprehensive documentation and examples for AI detection system Co-authored-by: blackpiglet <59276555+blackpiglet@users.noreply.github.com> --- .github/AI-DETECTION-EXAMPLES.md | 197 +++++++++++++++++++++++ .github/MAINTAINER-AI-DETECTION-GUIDE.md | 186 +++++++++++++++++++++ 2 files changed, 383 insertions(+) create mode 100644 .github/AI-DETECTION-EXAMPLES.md create mode 100644 .github/MAINTAINER-AI-DETECTION-GUIDE.md diff --git a/.github/AI-DETECTION-EXAMPLES.md b/.github/AI-DETECTION-EXAMPLES.md new file mode 100644 index 000000000..666ee7611 --- /dev/null +++ b/.github/AI-DETECTION-EXAMPLES.md @@ -0,0 +1,197 @@ +# AI Issue Detection - Examples + +This document provides examples to help understand what triggers AI detection. + +## Example 1: High AI Score (Score: 6/8) ❌ + +**This would be flagged:** + +```markdown +## Description +When deploying Velero on an EKS cluster with `hostNetwork: true`, the application fails to start. + +## Critical Problem +``` +time="2026-01-26T16:40:55Z" level=fatal msg="failed to start metrics server" +``` + +Status: BLOCKER + +## Affected Environment + +| Parameter | Value | +|----------|----------| +| Cluster | Amazon EKS | +| Velero Version | 1.8.2 | +| Kubernetes | 1.33 | + +## Root Cause Analysis + +The controller-runtime metrics uses port 8080 as a hardcoded default... + +## Resolution Attempts + +### Attempt 1: Use extraArgs +Result: Failed + +### Attempt 2: Configure metricsAddress +Result: Failed + +## Expected Permanent Solution + +Velero should: +1. Auto-detect an available port +2. Accept configuring the controller-runtime port + +## Questions for Maintainers +1. Why does controller-runtime use hardcoded 8080? +2. Is there a roadmap to support hostNetwork? + +## Labels and Metadata +Severity: CRITICAL +``` + +**Why flagged (Patterns detected: 6/8):** +- ✓ `futureDates` - References "2026-01-26" and "Kubernetes 1.33" +- ✓ `excessiveHeaders` - 8+ section headers +- ✓ `formalPhrases` - "Root Cause Analysis", "Expected Permanent Solution", "Questions for Maintainers", "Labels and Metadata" +- ✓ `aiSectionHeaders` - "## Description", "## Critical Problem", "## Affected Environment", "## Resolution Attempts" +- ✓ `perfectFormatting` - Perfect table structure +- ✓ `genericSolutions` - Mentions "auto-detect" + +--- + +## Example 2: Medium AI Score (Score: 2/8) ✅ + +**This would NOT be flagged (below threshold):** + +```markdown +**What steps did you take and what happened:** + +I'm trying to restore a backup but getting this error: +``` +error: backup "my-backup" not found +``` + +**What did you expect to happen:** +The backup should restore successfully + +**Environment:** +- Velero version: 1.13.0 +- Kubernetes version: 1.28 +- Cloud provider: AWS + +**Additional context:** +I can see the backup in S3 but Velero doesn't list it. Running `velero backup get` shows no backups. +``` + +**Why NOT flagged (Patterns detected: 2/8):** +- ✗ `futureDates` - Uses realistic versions +- ✗ `excessiveHeaders` - Only 3 headers +- ✗ `formalPhrases` - No formal AI phrases +- ✓ `excessiveTables` - Has a table but only 1 +- ✗ `perfectFormatting` - Normal formatting +- ✗ `aiSectionHeaders` - Standard issue template headers +- ✓ `excessiveFormatting` - Has code blocks +- ✗ `genericSolutions` - No generic solutions + +--- + +## Example 3: Legitimate Detailed Issue (Score: 3/8) ⚠️ + +**This would be flagged but is actually legitimate:** + +```markdown +## Problem Description + +VolumeGroupSnapshot restore fails with Ceph RBD driver. + +## Environment + +- Velero: 1.14.0 +- Kubernetes: 1.28.3 +- ODF: 4.14.2 with Ceph RBD CSI driver + +## Root Cause + +Ceph RBD stores group snapshot metadata in journal as `csi.groupid` omap key. During restore, when creating pre-provisioned VSC, the RBD driver reads this and populates `status.volumeGroupSnapshotHandle`. + +The CSI snapshot controller looks for a VGSC with matching handle. Since Velero deletes VGSC after backup, it's not found. + +## Reproduction Steps + +1. Create backup with VGS +2. Delete namespace +3. Restore backup +4. Observe VS stuck with "cannot find group snapshot" + +## Workaround + +Create stub VGSC with matching `volumeGroupSnapshotHandle` and patch status. + +## Proposed Fix + +1. Backup: Capture `volumeGroupSnapshotHandle` in CSISnapshotInfo +2. Restore: Create stub VGSC if handle exists + +## Code References + +- Ceph RBD: https://github.com/ceph/ceph-csi/blob/devel/internal/rbd/snapshot.go#L167 +- Velero deletion: https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/actions/csi/pvc_action.go#L1124 +``` + +**Why flagged (Patterns detected: 3/8):** +- ✗ `futureDates` - Uses current versions +- ✓ `excessiveHeaders` - Has 6 section headers +- ✓ `formalPhrases` - "Root Cause", "Proposed Fix" +- ✗ `excessiveTables` - No tables +- ✗ `perfectFormatting` - Normal formatting +- ✗ `aiSectionHeaders` - Technical, not generic +- ✗ `excessiveFormatting` - Reasonable formatting +- ✓ `genericSolutions` - Structured solution with code refs + +**Maintainer Action**: This is a legitimate, well-researched issue. Verify the details with the contributor and remove the `potential-ai-generated` label. + +--- + +## Example 4: Simple Valid Issue (Score: 0/8) ✅ + +**This would NOT be flagged:** + +```markdown +Velero backup fails with error: `rpc error: code = Unavailable desc = connection error` + +Running Velero 1.13 on GKE. Backups were working yesterday but now all fail with this error. + +Logs show the node-agent pod is crashing. Any ideas? +``` + +**Why NOT flagged (Patterns detected: 0/8):** +- All patterns: None detected + +--- + +## Key Takeaways + +### Will Trigger Detection ❌ +- Future dates/versions (2026+, K8s 1.33+) +- 4+ formal AI phrases +- 8+ section headers +- Perfect table formatting across multiple tables +- Generic AI section titles +- Auto-detect/generic solution patterns + +### Will NOT Trigger ✅ +- Realistic version numbers +- Actual error messages from real systems +- Normal issue formatting +- Moderate level of detail +- Standard GitHub issue template + +### May Trigger (But Legitimate) ⚠️ +- Very detailed technical analysis +- Multiple code references +- Well-structured proposals +- Extensive testing documentation + +For these cases, maintainers will verify with the contributor and remove the flag once confirmed. diff --git a/.github/MAINTAINER-AI-DETECTION-GUIDE.md b/.github/MAINTAINER-AI-DETECTION-GUIDE.md new file mode 100644 index 000000000..2f757e1f6 --- /dev/null +++ b/.github/MAINTAINER-AI-DETECTION-GUIDE.md @@ -0,0 +1,186 @@ +# Maintainer Guide: AI-Generated Issue Detection + +This guide helps Velero maintainers understand and work with the AI-generated issue detection system. + +## Overview + +The AI detection system automatically analyzes new and edited issues to identify potential AI-generated content. This helps maintain issue quality and ensures contributors verify their submissions. + +## How It Works + +### Automatic Detection + +When an issue is opened or edited, the workflow: + +1. **Analyzes** the issue body for 8 different AI patterns +2. **Calculates** an AI confidence score (0-8) +3. **If score ≥ 3**: Adds labels and posts a comment +4. **If score < 3**: Takes no action (issue proceeds normally) + +### Detection Patterns + +| Pattern | Description | Weight | +|---------|-------------|--------| +| `excessiveTables` | More than 5 markdown tables | 1 | +| `excessiveHeaders` | More than 8 section headers | 1 | +| `formalPhrases` | 4+ AI-typical phrases (e.g., "Root Cause Analysis") | 1 | +| `excessiveFormatting` | Multiple horizontal rules (---) | 1 | +| `futureDates` | Dates/versions in 2026+ or 2030s | 1 | +| `perfectFormatting` | Multiple identical table structures | 1 | +| `aiSectionHeaders` | 4+ generic AI headers (e.g., "Critical Problem") | 1 | +| `genericSolutions` | Auto-detect patterns with multiple YAML blocks | 1 | + +## Working with Flagged Issues + +### Step 1: Review the Issue + +When you see an issue labeled `potential-ai-generated`: + +1. **Read the issue carefully** +2. **Check the detected patterns** (listed in the auto-comment) +3. **Look for red flags**: + - Future version numbers (e.g., "Kubernetes 1.33") + - Future dates (e.g., "2026-01-27") + - Non-existent features or configurations + - Perfect table formatting with no actual content + - Generic solutions that don't match Velero's architecture + +### Step 2: Engage with the Contributor + +**If the issue seems legitimate but over-formatted:** + +```markdown +Thanks for the detailed report! Could you confirm: +1. Are you running Velero version X.Y.Z (you mentioned version A.B.C)? +2. Is the error message exactly as shown? +3. Have you actually tried the workarounds mentioned? + +Once verified, we'll remove the AI-generated flag and investigate. +``` + +**If the issue appears to be unverified AI content:** + +```markdown +This issue appears to contain AI-generated content that hasn't been verified. + +Please review our [AI contribution guidelines](https://github.com/vmware-tanzu/velero/blob/main/site/content/docs/main/code-standards.md#ai-generated-content) and: +1. Confirm this describes a real problem in your environment +2. Verify all version numbers and error messages +3. Remove any placeholder or example content +4. Test that the issue is reproducible + +If you can't verify the issue, please close it. We're happy to help with real problems! +``` + +### Step 3: Take Action + +**For verified issues:** +1. Remove the `potential-ai-generated` label +2. Keep or remove `needs-triage` as appropriate +3. Proceed with normal issue triage + +**For unverified/invalid issues:** +1. Request verification (see templates above) +2. If no response after 7 days, consider closing as `stale` +3. If clearly invalid, close with explanation + +## Common Patterns + +### False Positives (Legitimate Issues) + +These may trigger the detector but are usually valid: + +- **Very detailed bug reports** with extensive logs and testing +- **Technical design proposals** with multiple sections +- **Well-organized feature requests** with tables and examples + +**Action**: Engage with contributor, ask clarifying questions, remove flag if verified. + +### True Positives (AI-Generated) + +Red flags that indicate unverified AI content: + +- **Future version numbers**: "Kubernetes 1.33" (doesn't exist yet) +- **Future dates**: "2026-01-27" (if current date is before) +- **Non-existent features**: References to Velero features that don't exist +- **Generic solutions**: "Auto-detect available port" (not how Velero works) +- **Perfect formatting, wrong content**: Beautiful tables with incorrect info + +**Action**: Request verification, ask for actual environment details, consider closing if unverified. + +### Edge Cases + +**Contributor using AI as a writing assistant:** +- Issue content is verified and accurate +- Just used AI to help structure/format the report +- **Action**: This is acceptable! Remove flag if content is verified. + +**Legitimate issue that happens to match patterns:** +- Real problem with detailed analysis +- Includes proper version numbers and logs +- **Action**: Verify with contributor, remove flag once confirmed. + +## Statistics and Monitoring + +You can search for flagged issues: + +``` +is:issue label:potential-ai-generated +``` + +Monitor trends: +- High detection rate → May need to adjust thresholds +- Low detection rate → Patterns working well or need refinement + +## Adjusting the System + +### Modifying Detection Patterns + +Edit `.github/workflows/ai-issue-detector.yml`: + +```javascript +// Increase threshold to reduce false positives +if (aiScore >= 4) { // was 3 + +// Adjust pattern sensitivity +excessiveTables: (issueBody.match(/\|.*\|/g) || []).length > 8, // was 5 +``` + +### Adding New Patterns + +Add to the `aiPatterns` object: + +```javascript +// Example: Detect excessive use of emojis +excessiveEmojis: (issueBody.match(/[\u{1F300}-\u{1F9FF}]/gu) || []).length > 10, +``` + +### Disabling the Workflow + +Rename or delete `.github/workflows/ai-issue-detector.yml` + +## Best Practices + +1. **Be courteous**: Contributors may not realize their AI tool generated incorrect info +2. **Verify, don't assume**: Some detailed issues are legitimate +3. **Educate**: Point to the AI guidelines in code-standards.md +4. **Track patterns**: Note common AI-generated patterns for future improvements +5. **Iterate**: Adjust detection thresholds based on false positive rates + +## FAQ + +**Q: Should we reject all AI-assisted contributions?** +A: No! AI assistance is fine if the contributor verifies accuracy. We only flag unverified AI content. + +**Q: What if a contributor is offended by the flag?** +A: Explain it's automated and not personal. We just need verification of technical details. + +**Q: Can we automatically close flagged issues?** +A: No. Always engage with the contributor first. Some are legitimate. + +**Q: What's an acceptable false positive rate?** +A: Aim for <10%. If higher, increase the threshold from 3 to 4 or 5. + +## Support + +Questions about the AI detection system? Tag @vmware-tanzu/velero-maintainers in issue #9501.