Correct statistical-method guidance in analysis skill docs

Author: mingzhangyang

skills/stat-analyze-distribution/SKILL.md

@@ -18,29 +18,35 @@ description: Characterize the distribution of one or more numeric variables with
   - Data table (CSV or TSV) with at least one numeric column
 - Optional:
   - Column(s) to analyze (default: all numeric columns)
-  - Normality test method: `shapiro` (default, n < 5000) or `ks` (Kolmogorov-Smirnov, larger samples)
+  - Normality test method: `shapiro` (default, n ≤ 5000) or `anderson` (Anderson-Darling, recommended for larger samples)
   - Output directory (default: `./dist_output`)
 
 ## Workflow
 
 1. Read the data file; identify numeric columns to analyze (or use user-specified subset).
 2. For each target column, compute summary statistics: mean, median, standard deviation, variance, min, max, 25th/75th percentiles, skewness, kurtosis.
-3. Run normality test: Shapiro-Wilk (n ≤ 5000) or Kolmogorov-Smirnov (n > 5000). Report test statistic and p-value.
-4. Classify each variable: `normal` (p > 0.05) or `non-normal` (p ≤ 0.05), with note on sample size.
+3. Run normality test:
+   - Shapiro-Wilk when n ≤ 5000
+   - Anderson-Darling when n > 5000
+   Report test statistic, p-value (or significance level where applicable), and sample size.
+4. Classify each variable as `approximately_normal` or `non_normal` using both test output and Q-Q plot inspection (do not classify from hypothesis-test p-value alone).
 5. Generate plots for each variable: histogram with density overlay, Q-Q plot.
 6. If multiple variables, produce a summary table across all variables.
 7. Write summary statistics table (TSV) and plots (PDF) to output directory.
-8. Report which variables appear normally distributed and suggest appropriate downstream tests.
+8. Save an executable analysis script (`analysis_code.py` or `analysis_code.R`) and software version manifest (`session_info.txt`) used to produce results.
+9. Report which variables appear normally distributed and suggest appropriate downstream tests.
 
 ## Output Contract
 
-- Summary statistics table (TSV): variable, n, mean, median, sd, variance, skewness, kurtosis, normality_test, test_statistic, p_value, is_normal
+- Summary statistics table (TSV): variable, n, mean, median, sd, variance, skewness, kurtosis, normality_test, test_statistic, p_value_or_significance, normality_call
 - Histogram + density plot per variable (PDF)
 - Q-Q plot per variable (PDF)
+- Reproducibility artifacts: `analysis_code.py` or `analysis_code.R`, plus `session_info.txt`
 
 ## Limits
 
-- Shapiro-Wilk test is unreliable for n > 5000; automatically switches to KS test.
+- Shapiro-Wilk test is unreliable for very large n; use Anderson-Darling and visual diagnostics instead.
+- Avoid one-sample KS tests against a fitted normal distribution unless a corrected procedure is explicitly implemented.
 - Normality tests have low power for small samples (n < 20); visual inspection of Q-Q plot is recommended.
 - Does not handle categorical or date columns; these are silently skipped.
 - Common failure: non-numeric columns passed explicitly cause an error — pre-filter if needed.

skills/stat-compare-multiple-groups/SKILL.md

@@ -25,7 +25,7 @@ description: Compare a continuous variable across three or more groups using one
 ## Workflow
 
 1. Read data; validate that group column has three or more levels.
-2. Run normality test (Shapiro-Wilk) and Levene's test for each group.
+2. Run normality tests (Shapiro-Wilk) within each group and run one global Levene's test across groups for homogeneity of variance.
 3. If all groups are normal and variances are approximately equal: one-way ANOVA.
    - If second factor provided: two-way ANOVA with interaction term.
 4. If normality or homogeneity of variance is violated: Kruskal-Wallis test.
@@ -34,15 +34,17 @@ description: Compare a continuous variable across three or more groups using one
    - Kruskal-Wallis: Dunn test with Benjamini-Hochberg correction.
 6. Compute effect size: eta-squared (η²) for ANOVA, epsilon-squared for Kruskal-Wallis.
 7. Generate a box plot with all groups and pairwise significance brackets.
-8. Write results and plot (PDF) to output directory.
-9. Report: omnibus test result, effect size, significant pairwise comparisons.
+8. Save the executable analysis script (`analysis_code.py` or `analysis_code.R`) and software version manifest (`session_info.txt`) used to produce results.
+9. Write results and plot (PDF) to output directory.
+10. Report: omnibus test result, effect size, significant pairwise comparisons.
 
 ## Output Contract
 
 - Omnibus test result (TSV): test, statistic, df, p_value, effect_size, effect_size_type
 - Post-hoc comparisons table (TSV): group1, group2, statistic, p_value, adj_p_value, significant
 - Box plot with significance brackets (PDF)
 - Decision log explaining method selection (printed to stdout)
+- Reproducibility artifacts: `analysis_code.py` or `analysis_code.R`, plus `session_info.txt`
 
 ## Limits
 

skills/stat-logistic-regression/SKILL.md

@@ -27,23 +27,31 @@ description: Fit a logistic regression model for binary outcomes, report odds ra
 
 1. Read data; validate that outcome column is binary; encode predictors as needed.
 2. Split data into training and test sets (stratified by outcome).
-3. Fit logistic regression on training set using maximum likelihood estimation.
-4. Report for each coefficient: log-odds estimate, standard error, z-statistic, p-value, odds ratio, 95% CI for odds ratio.
+3. Fit logistic regression on training set:
+   - If `regularization=none`: unpenalized MLE fit for inferential reporting.
+   - If `regularization=l1` or `l2`: cross-validated penalized fit for prediction-focused modeling.
+4. Report coefficients:
+   - Unpenalized model: log-odds estimate, standard error, z-statistic, p-value, odds ratio, 95% CI.
+   - Penalized model: shrunken coefficients and odds ratios; do not report classical p-values/z-statistics from penalized fit.
 5. Evaluate on test set: compute AUC-ROC, plot ROC curve, compute confusion matrix at 0.5 threshold (accuracy, sensitivity, specificity, PPV, NPV).
-6. Compute Hosmer-Lemeshow goodness-of-fit test for calibration.
-7. If regularization is used: report optimal lambda from cross-validation, coefficients at optimal lambda.
-8. Write coefficient table (TSV), performance metrics (TSV), ROC curve plot (PDF), and calibration plot (PDF) to output directory.
+6. Compute Hosmer-Lemeshow goodness-of-fit test for calibration for the unpenalized model; for penalized models, report calibration curve and Brier score.
+7. If regularization is used: report optimal lambda from cross-validation and non-zero coefficients at optimal lambda.
+8. Save the executable analysis script (`analysis_code.py` or `analysis_code.R`) and software version manifest (`session_info.txt`) used to produce results.
+9. Write coefficient table(s), performance metrics (TSV), ROC curve plot (PDF), and calibration plot (PDF) to output directory.
 
 ## Output Contract
 
-- Coefficient table (TSV): term, log_odds, std_error, z_statistic, p_value, odds_ratio, ci_lower_or, ci_upper_or
-- Performance metrics (TSV): auc, accuracy, sensitivity, specificity, ppv, npv, hl_test_p
+- Coefficient table (TSV, unpenalized): term, log_odds, std_error, z_statistic, p_value, odds_ratio, ci_lower_or, ci_upper_or
+- Coefficient table (TSV, penalized): term, log_odds, odds_ratio, selected_at_lambda
+- Performance metrics (TSV): auc, accuracy, sensitivity, specificity, ppv, npv, hl_test_p_or_na, brier_score
 - ROC curve (PDF)
 - Confusion matrix (printed to stdout)
+- Reproducibility artifacts: `analysis_code.py` or `analysis_code.R`, plus `session_info.txt`
 
 ## Limits
 
 - Assumes independence of observations; does not account for clustering or repeated measures.
 - Requires sufficient events per variable (EPV ≥ 10 recommended) to avoid overfitting.
 - Class imbalance can inflate accuracy; examine sensitivity/specificity and AUC.
+- Penalized models are prediction-oriented; classical inferential statistics (standard errors/p-values) are not directly interpretable from penalized coefficients.
 - Common failure: outcome column with more than two levels — binarize before running.