Skip to content

fix: correct Token.Position.Offset when YAML contains comments#867

Open
Yanhu007 wants to merge 1 commit intogoccy:masterfrom
Yanhu007:fix/comment-token-offset
Open

fix: correct Token.Position.Offset when YAML contains comments#867
Yanhu007 wants to merge 1 commit intogoccy:masterfrom
Yanhu007:fix/comment-token-offset

Conversation

@Yanhu007
Copy link
Copy Markdown

Fixes #856

Problem

Token.Position.Offset is incorrect when YAML contains comments. Each preceding comment shifts subsequent token offsets by -1:

tokens := lexer.Tokenize("# comment1\n# comment2\nkey: value")
// "key" token:
//   Offset: 21 (actual)
//   Offset: 23 (expected, 1-indexed)
//   Shift: -1 per comment = -2

Root Cause

In scanComment, the # character is consumed with s.progress(ctx, 1) which advances the source index (ctx.idx) but does not update s.offset. This creates a cumulative offset drift of -1 per comment.

Fix

Replace s.progress(ctx, 1) with s.progressOnly(ctx, 1), which advances both the source index and the offset counter.

All existing tests pass.

When scanning a comment, the "#" character was consumed with
s.progress() which advances the source index but does not update
s.offset. This caused all subsequent token offsets to be shifted
by -1 for each preceding comment.

Replace s.progress(ctx, 1) with s.progressOnly(ctx, 1) to also
advance the offset counter when skipping the "#" character.

Fixes goccy#856
@codecov-commenter
Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.65%. Comparing base (edee2f9) to head (e2cb4e1).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #867   +/-   ##
=======================================
  Coverage   80.65%   80.65%           
=======================================
  Files          22       22           
  Lines        6845     6845           
=======================================
  Hits         5521     5521           
  Misses        905      905           
  Partials      419      419           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Token.Position.Offset is incorrect when YAML contains comments

2 participants