Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 77 additions & 18 deletions .cognition/skills/debug-perlonjava/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,32 @@ perl dev/tools/perl_test_runner.pl perl5_t/t/op
perl dev/tools/perl_test_runner.pl --jobs 8 --timeout 60 perl5_t/t
```

### Test runner environment variables
The test runner (`dev/tools/perl_test_runner.pl`) automatically sets environment variables for specific tests:

```perl
# JPERL_UNIMPLEMENTED="warn" for these tests:
re/pat_rt_report.t | re/pat.t | re/regex_sets.t | re/regexp_unicode_prop.t
op/pack.t | op/index.t | op/split.t | re/reg_pmod.t | op/sprintf.t | base/lex.t

# JPERL_OPTS="-Xss256m" for these tests:
re/pat.t | op/repeat.t | op/list.t

# PERL_SKIP_BIG_MEM_TESTS=1 for ALL tests
```

To reproduce what the test runner does for a specific test:
```bash
# For re/pat.t (needs all three):
cd perl5_t/t && JPERL_UNIMPLEMENTED=warn JPERL_OPTS=-Xss256m PERL_SKIP_BIG_MEM_TESTS=1 ../../jperl re/pat.t

# For re/subst.t (only PERL_SKIP_BIG_MEM_TESTS):
cd perl5_t/t && PERL_SKIP_BIG_MEM_TESTS=1 ../../jperl re/subst.t

# For op/bop.t (only PERL_SKIP_BIG_MEM_TESTS):
cd perl5_t/t && PERL_SKIP_BIG_MEM_TESTS=1 ../../jperl op/bop.t
```

### Interpreter mode
```bash
./jperl --interpreter script.pl
Expand Down Expand Up @@ -123,39 +149,32 @@ git checkout branch && mvn package -q -DskipTests
./jperl -e 'failing code'
```

### 2. Bisect to find the bad commit
**IMPORTANT**: Always rebuild after switching commits!
```bash
git log master..branch --oneline
git checkout <commit> && mvn package -q -DskipTests && ./jperl -e 'test'
```

### 3. Create minimal reproducer
### 2. Create minimal reproducer
Reduce the failing test to the smallest code that demonstrates the bug:
```bash
./jperl -e 'my $x = 58; eval q{($x) .= "z"}; print "x=$x\n"'
```

### 4. Compare with system Perl
### 3. Compare with system Perl
```bash
perl -e 'same code'
```

### 5. Use --parse to check AST
### 4. Use --parse to check AST
When parsing issues are suspected, compare the parse tree:
```bash
./jperl --parse -e 'code' # Show PerlOnJava AST
perl -MO=Deparse -e 'code' # Compare with Perl's interpretation
```
This helps identify operator precedence issues and incorrect parsing.

### 6. Use disassembly to understand
### 5. Use disassembly to understand
```bash
./jperl --disassemble -e 'minimal code' # JVM bytecode
./jperl --disassemble --interpreter -e 'minimal code' # Interpreter bytecode
```

### 7. Profile with JFR (for performance issues)
### 6. Profile with JFR (for performance issues)
```bash
# Record profile
$JAVA_HOME/bin/java -XX:StartFlightRecording=duration=10s,filename=profile.jfr \
Expand All @@ -166,14 +185,14 @@ $JAVA_HOME/bin/jfr print --events jdk.ExecutionSample profile.jfr 2>&1 | \
grep -E "^\s+[a-z].*line:" | sed 's/line:.*//' | sort | uniq -c | sort -rn | head -20
```

### 8. Add debug prints (if needed)
### 7. Add debug prints (if needed)
In Java source, add:
```java
System.err.println("DEBUG: var=" + var);
```
Then rebuild with `mvn package -q -DskipTests`.

### 9. Fix and verify
### 8. Fix and verify
```bash
# After fixing
mvn package -q -DskipTests
Expand Down Expand Up @@ -258,6 +277,50 @@ Both backends share the parser (same AST) and runtime (same operators, same Runt

All paths relative to `src/main/java/org/perlonjava/`.

## CRITICAL: Investigate JVM Backend First

**When fixing interpreter bugs, ALWAYS investigate how the JVM backend handles the same operation before implementing a fix.**

The interpreter and JVM backends share the same runtime classes (`RuntimeScalar`, `RuntimeArray`, `RuntimeHash`, `RuntimeList`, `PerlRange`, etc.). The JVM backend is the reference implementation - if the interpreter handles something differently, it's likely wrong.

### How to investigate JVM behavior

1. **Disassemble the JVM bytecode** to see what runtime methods it calls:
```bash
./jperl --disassemble -e 'code that works'
```

2. **Look for the runtime method calls** in the disassembly (INVOKEVIRTUAL, INVOKESTATIC):
```
INVOKEVIRTUAL org/perlonjava/runtime/runtimetypes/RuntimeList.addToArray
INVOKEVIRTUAL org/perlonjava/runtime/runtimetypes/RuntimeBase.setFromList
```

3. **Read those runtime methods** to understand the correct behavior:
- How does `setFromList()` handle different input types?
- What methods does it call internally (`addToArray`, `getList`, etc.)?

4. **Use the same runtime methods in the interpreter** instead of reimplementing the logic with special cases.

### Example: Hash slice assignment with PerlRange

**Wrong approach** (special-casing types in interpreter):
```java
if (valuesBase instanceof RuntimeList) { ... }
else if (valuesBase instanceof RuntimeArray) { ... }
else if (valuesBase instanceof PerlRange) { ... } // BAD: special case
else { ... }
```

**Correct approach** (use same runtime methods as JVM):
```java
// JVM calls addToArray() which handles all types uniformly
RuntimeArray valuesArray = new RuntimeArray();
valuesBase.addToArray(valuesArray); // Works for RuntimeList, RuntimeArray, PerlRange, etc.
```

The JVM's `setFromList()` → `addToArray()` chain already handles `PerlRange` correctly via `PerlRange.addToArray()` → `toList().addToArray()`. The interpreter should use the same mechanism.

## Common Bug Patterns

### 1. Context not propagated correctly
Expand Down Expand Up @@ -335,10 +398,6 @@ perl -MO=Deparse -e 'code'
# Compare output
diff <(./jperl -e 'code') <(perl -e 'code')

# Bisect
git log master..HEAD --oneline
git checkout <sha> && mvn package -q -DskipTests && ./jperl -e 'test'

# Git workflow (always use branches!)
git checkout -b fix-name
# ... make changes ...
Expand Down
71 changes: 71 additions & 0 deletions .cognition/skills/interpreter-parity/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,18 @@ JPERL_INTERPRETER=1 ./jperl script.pl
JPERL_INTERPRETER=1 ./jperl -e 'code'
```

**CRITICAL: eval STRING uses interpreter by default!**
Even when running with JVM backend, `eval STRING` compiles code with the interpreter.
This means interpreter bugs can cause test failures even without `--interpreter`.

To trace eval STRING execution:
```bash
JPERL_EVAL_TRACE=1 ./jperl script.pl 2>&1 | grep -i interpreter
```

Fallback for large subs (`JPERL_SHOW_FALLBACK=1`) does NOT show eval STRING usage.
One-liners won't trigger fallback - test with actual test files!

## Architecture: Two Backends, Shared Everything Else

```
Expand Down Expand Up @@ -154,6 +166,53 @@ All paths relative to `src/main/java/org/perlonjava/`.

## Debugging Workflow

### CRITICAL: Save Master Baselines ONCE, Don't Rebuild Repeatedly

**Save master baseline to files FIRST** (do this once per debugging session):
```bash
# Switch to master and build
git stash && git checkout master
mvn package -q -DskipTests

# Save master test output for JVM backend
cd perl5_t/t && ../../jperl re/subst.t 2>&1 > /tmp/master_subst.log
grep "^not ok" /tmp/master_subst.log > /tmp/master_subst_fails.txt

# ALSO save interpreter baseline!
cd perl5_t/t && ../../jperl --interpreter re/subst.t 2>&1 > /tmp/master_subst_interp.log

# Switch back to feature branch
git checkout feature-branch && git stash pop
```

**After making changes**, compare against saved baselines:
```bash
mvn package -q -DskipTests

# Test JVM backend
cd perl5_t/t && ../../jperl re/subst.t 2>&1 > /tmp/feature_subst.log
diff /tmp/master_subst_fails.txt <(grep "^not ok" /tmp/feature_subst.log)

# MUST ALSO test with interpreter!
cd perl5_t/t && ../../jperl --interpreter re/subst.t 2>&1 > /tmp/feature_subst_interp.log
```

### CRITICAL: Always Test with BOTH Backends

A fix that works for JVM backend may break interpreter, or vice versa.

**For quick tests (one-liners):**
```bash
./jperl -e 'test code' # JVM backend
./jperl --interpreter -e 'test code' # Interpreter backend
```

**For test files (use env var so require/do/eval also use interpreter):**
```bash
./jperl test.t # JVM backend
JPERL_INTERPRETER=1 ./jperl test.t # Interpreter backend (full)
```

### 1. Reproduce with minimal code
```bash
# Find the failing construct
Expand All @@ -162,6 +221,18 @@ JPERL_INTERPRETER=1 ./jperl -e 'failing code'
./jperl -e 'failing code'
```

**CRITICAL: Save baselines to files!** When comparing test suites across branches:
```bash
# On master - save results so you don't have to rebuild later
git checkout master && mvn package -q -DskipTests
cd perl5_t/t && JPERL_INTERPRETER=1 ../../jperl test.t 2>&1 | tee /tmp/test_master.log
JPERL_INTERPRETER=1 ../../jperl test.t 2>&1 | grep "^ok\|^not ok" > /tmp/test_master_results.txt
grep "^ok" /tmp/test_master_results.txt | wc -l # Save this number!

# Return to feature branch - now you can compare without rebuilding master
git checkout feature-branch && mvn package -q -DskipTests
```

### 2. Use --disassemble to see interpreter bytecode
```bash
JPERL_INTERPRETER=1 ./jperl --disassemble -e 'code' 2>&1
Expand Down
50 changes: 50 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,53 @@ See `.cognition/skills/` for specialized debugging and development skills:
- `interpreter-parity` - JVM vs interpreter parity issues
- `debug-exiftool` - ExifTool test debugging
- `profile-perlonjava` - Performance profiling

## Regression Tracking (feature/defer-blocks branch)

### Summary

All reported regressions have been investigated. The issues fall into two categories:
1. **Fixed in this branch**: goto-related issues
2. **Pre-existing on master**: MethodHandle conversion errors, regex octal escape parsing

### Regression Status

| Test | Status | Details |
|------|--------|---------|
| op/die_goto.t | **FIXED** (5/5) | `goto &sub` in `$SIG{__DIE__}` handlers now works |
| uni/goto.t | **FIXED** (2/4) | Tests 1-2 pass (goto &{expr}). Tests 3-4 fail due to pre-existing regex octal escape bug |
| op/bop.t | Pre-existing | MethodHandle conversion error - fails on master too |
| op/warn.t | Pre-existing | MethodHandle conversion error - fails on master too |
| re/subst.t | Pre-existing | MethodHandle conversion error - fails on master too |
| re/pat_rt_report.t | Pre-existing | MethodHandle conversion error - fails on master too |
| lib/croak.t | Pre-existing | `class` feature incomplete |

### Fixes Applied in This Branch

1. **EmitControlFlow.java**: Added `handleGotoSubroutineBlock()` for `goto &{expr}` tail call support
2. **CompileOperator.java**: Added `goto &{expr}` support to bytecode interpreter
3. **RuntimeControlFlowList.java**: Added validation for undefined subroutines in tail calls
4. **RuntimeCode.java**: Added `gotoErrorPrefix()` for "Goto undefined subroutine" error messages
5. **CompileAssignment.java**: Added `vec` lvalue support for interpreter
6. **OpcodeHandlerExtended.java**: Fixed `|=` and `^=` to use polymorphic `bitwiseOr`/`bitwiseXor`
7. **WarnDie.java**: Added TAILCALL trampoline for `goto &sub` in `$SIG{__DIE__}` handlers

### Pre-existing Issues (on master too)

- **MethodHandle conversion errors**: Affects op/warn.t, re/subst.t, re/pat_rt_report.t, op/bop.t
- **Regex octal escapes**: `\345` in patterns is parsed as backreference `\3` + `45`
- **op/bop.t**: `new version ~$_` crashes in Version.java
- **String bitwise ops**: Interpreter uses numeric ops instead of string ops

### How to Check Regressions

```bash
# Run specific test
cd perl5_t/t && ../../jperl <test>.t

# Count passing tests
../../jperl <test>.t 2>&1 | grep "^ok" | wc -l

# Check for interpreter fallback
JPERL_SHOW_FALLBACK=1 ../../jperl <test>.t 2>&1 | grep -i fallback
```
Loading
Loading