Skip to content

Commit 24d9870

Browse files
authored
fix: parser improvements for v0.0.40 (#22)
* fix: support private/protected/public def syntax parsing (#17) Add visibility modifier support for method definitions: - Parse `private def`, `protected def`, `public def` syntax - Extract visibility info and include in parse result - Preserve visibility keywords during type erasure - Add VISIBILITY_PATTERN constant for reuse Fixes #17 * fix: skip heredoc content during parsing (#18) Add HeredocDetector to identify heredoc regions and skip them during parsing. This prevents def patterns inside heredoc from being incorrectly recognized as method definitions. - Add HeredocDetector class with detect() and inside_heredoc?() methods - Support <<DELIM, <<-DELIM, <<~DELIM, <<'DELIM', <<"DELIM" syntax - Integrate heredoc detection in Parser.parse() - Add comprehensive tests for HeredocDetector Fixes #18 * feat: support visibility modifiers in RBS generation Add private/protected visibility prefix to generated RBS method signatures. Both TRuby::RBSGenerator and TRuby::IR::RBSGenerator now output correct RBS format like 'private def method: () -> Type'. Related to #17 * feat: add visibility support to RBS generation with E2E tests - Pass visibility from parser to IR MethodDef nodes - Add private prefix to RBS output (RBS doesn't support protected) - Add E2E tests for visibility in RBS and compiled Ruby - Update unit tests for RBS generator visibility handling RBS only supports private, not protected (ruby/rbs#579) Related to #17 * fix: skip =begin/=end block comments during parsing Extend HeredocDetector to also detect Ruby block comments. Parser now correctly ignores def patterns inside =begin/=end blocks. Related to #18 * refactor: remove legacy compiler path Remove legacy components that are no longer needed: - Remove TRuby::RBSGenerator (use IR::RBSGenerator) - Remove LegacyCompiler class - Remove LegacyParser class - Remove use_ir parameter from Compiler (always use IR) - Remove use_combinator parameter from Parser (always enabled) This simplifies the codebase by eliminating dual-path maintenance. * fix: preserve default values when erasing type annotations (#21)
1 parent a9d85aa commit 24d9870

File tree

13 files changed

+557
-340
lines changed

13 files changed

+557
-340
lines changed

lib/t_ruby.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,14 +11,14 @@
1111

1212
# Basic components
1313
require_relative "t_ruby/type_alias_registry"
14+
require_relative "t_ruby/heredoc_detector"
1415
require_relative "t_ruby/body_parser"
1516
require_relative "t_ruby/parser"
1617
require_relative "t_ruby/union_type_parser"
1718
require_relative "t_ruby/generic_type_parser"
1819
require_relative "t_ruby/intersection_type_parser"
1920
require_relative "t_ruby/type_erasure"
2021
require_relative "t_ruby/error_handler"
21-
require_relative "t_ruby/rbs_generator"
2222
require_relative "t_ruby/declaration_generator"
2323
require_relative "t_ruby/compiler"
2424
require_relative "t_ruby/lsp_server"

lib/t_ruby/compiler.rb

Lines changed: 29 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -7,16 +7,17 @@ module TRuby
77
# \p{L} matches any Unicode letter, \p{N} matches any Unicode number
88
IDENTIFIER_CHAR = '[\p{L}\p{N}_]'
99
METHOD_NAME_PATTERN = "#{IDENTIFIER_CHAR}+[?!]?".freeze
10+
# Visibility modifiers for method definitions
11+
VISIBILITY_PATTERN = '(?:(?:private|protected|public)\s+)?'
1012

1113
class Compiler
12-
attr_reader :declaration_loader, :use_ir, :optimizer
14+
attr_reader :declaration_loader, :optimizer
1315

14-
def initialize(config = nil, use_ir: true, optimize: true)
16+
def initialize(config = nil, optimize: true)
1517
@config = config || Config.new
16-
@use_ir = use_ir
1718
@optimize = optimize
1819
@declaration_loader = DeclarationLoader.new
19-
@optimizer = IR::Optimizer.new if use_ir && optimize
20+
@optimizer = IR::Optimizer.new if optimize
2021
@type_inferrer = ASTTypeInferrer.new if type_check?
2122
setup_declaration_paths if @config
2223
end
@@ -42,16 +43,16 @@ def compile(input_path)
4243
source = File.read(input_path)
4344

4445
# Parse with IR support
45-
parser = Parser.new(source, use_combinator: @use_ir)
46-
parse_result = parser.parse
46+
parser = Parser.new(source)
47+
parser.parse
4748

4849
# Run type checking if enabled
49-
if type_check? && @use_ir && parser.ir_program
50+
if type_check? && parser.ir_program
5051
check_types(parser.ir_program, input_path)
5152
end
5253

5354
# Transform source to Ruby code
54-
output = @use_ir ? transform_with_ir(source, parser) : transform_legacy(source, parse_result)
55+
output = transform_with_ir(source, parser)
5556

5657
# Compute output path (respects preserve_structure setting)
5758
output_path = compute_output_path(input_path, @config.ruby_dir, ".rb")
@@ -63,11 +64,7 @@ def compile(input_path)
6364
if @config.compiler["generate_rbs"]
6465
rbs_path = compute_output_path(input_path, @config.rbs_dir, ".rbs")
6566
FileUtils.mkdir_p(File.dirname(rbs_path))
66-
if @use_ir && parser.ir_program
67-
generate_rbs_from_ir_to_path(rbs_path, parser.ir_program)
68-
else
69-
generate_rbs_file_to_path(rbs_path, parse_result)
70-
end
67+
generate_rbs_from_ir_to_path(rbs_path, parser.ir_program)
7168
end
7269

7370
# Generate .d.trb file if enabled in config (legacy support)
@@ -87,25 +84,17 @@ def compile(input_path)
8784
def compile_string(source, options = {})
8885
generate_rbs = options.fetch(:rbs, true)
8986

90-
parser = Parser.new(source, use_combinator: @use_ir)
91-
parse_result = parser.parse
87+
parser = Parser.new(source)
88+
parser.parse
9289

9390
# Transform source to Ruby code
94-
ruby_output = @use_ir ? transform_with_ir(source, parser) : transform_legacy(source, parse_result)
91+
ruby_output = transform_with_ir(source, parser)
9592

9693
# Generate RBS if requested
9794
rbs_output = ""
98-
if generate_rbs
99-
if @use_ir && parser.ir_program
100-
generator = IR::RBSGenerator.new
101-
rbs_output = generator.generate(parser.ir_program)
102-
else
103-
generator = RBSGenerator.new
104-
rbs_output = generator.generate(
105-
parse_result[:functions] || [],
106-
parse_result[:type_aliases] || []
107-
)
108-
end
95+
if generate_rbs && parser.ir_program
96+
generator = IR::RBSGenerator.new
97+
rbs_output = generator.generate(parser.ir_program)
10998
end
11099

111100
{
@@ -134,7 +123,7 @@ def compile_to_ir(input_path)
134123
end
135124

136125
source = File.read(input_path)
137-
parser = Parser.new(source, use_combinator: true)
126+
parser = Parser.new(source)
138127
parser.parse
139128
parser.ir_program
140129
end
@@ -380,10 +369,10 @@ def setup_declaration_paths
380369
@declaration_loader.add_search_path("./lib/types")
381370
end
382371

383-
# Transform using IR system (new approach)
372+
# Transform using IR system
384373
def transform_with_ir(source, parser)
385374
ir_program = parser.ir_program
386-
return transform_legacy(source, parser.parse) unless ir_program
375+
return source unless ir_program
387376

388377
# Run optimization passes if enabled
389378
if @optimize && @optimizer
@@ -396,33 +385,15 @@ def transform_with_ir(source, parser)
396385
generator.generate_with_source(ir_program, source)
397386
end
398387

399-
# Legacy transformation using TypeErasure (backward compatible)
400-
def transform_legacy(source, parse_result)
401-
if parse_result[:type] == :success
402-
eraser = TypeErasure.new(source)
403-
eraser.erase
404-
else
405-
source
406-
end
407-
end
408-
409388
# Generate RBS from IR to a specific path
410389
def generate_rbs_from_ir_to_path(rbs_path, ir_program)
390+
return unless ir_program
391+
411392
generator = IR::RBSGenerator.new
412393
rbs_content = generator.generate(ir_program)
413394
File.write(rbs_path, rbs_content) unless rbs_content.strip.empty?
414395
end
415396

416-
# Legacy RBS generation to a specific path
417-
def generate_rbs_file_to_path(rbs_path, parse_result)
418-
generator = RBSGenerator.new
419-
rbs_content = generator.generate(
420-
parse_result[:functions] || [],
421-
parse_result[:type_aliases] || []
422-
)
423-
File.write(rbs_path, rbs_content) unless rbs_content.empty?
424-
end
425-
426397
def generate_dtrb_file(input_path, out_dir)
427398
dtrb_path = compute_output_path(input_path, out_dir, DeclarationGenerator::DECLARATION_EXTENSION)
428399
FileUtils.mkdir_p(File.dirname(dtrb_path))
@@ -511,7 +482,8 @@ def erase_parameter_types(source)
511482
result = source.dup
512483

513484
# Match function definitions and remove type annotations from parameters
514-
result.gsub!(/^(\s*def\s+#{TRuby::METHOD_NAME_PATTERN}\s*\()([^)]+)(\)\s*)(?::\s*[^\n]+)?(\s*$)/) do |_match|
485+
# Also supports visibility modifiers: private def, protected def, public def
486+
result.gsub!(/^(\s*#{TRuby::VISIBILITY_PATTERN}def\s+#{TRuby::METHOD_NAME_PATTERN}\s*\()([^)]+)(\)\s*)(?::\s*[^\n]+)?(\s*$)/) do |_match|
515487
indent = ::Regexp.last_match(1)
516488
params = ::Regexp.last_match(2)
517489
close_paren = ::Regexp.last_match(3)
@@ -558,10 +530,13 @@ def remove_param_types(params_str)
558530
params.join(", ")
559531
end
560532

561-
# Clean a single parameter (remove type annotation)
533+
# Clean a single parameter (remove type annotation, preserve default value)
562534
def clean_param(param)
563-
# Match: name: Type or name (supports Unicode identifiers)
564-
if (match = param.match(/^(#{TRuby::IDENTIFIER_CHAR}+)\s*:/))
535+
# Match: name: Type = value (with default value)
536+
if (match = param.match(/^(#{TRuby::IDENTIFIER_CHAR}+)\s*:\s*.+?\s*(=\s*.+)$/))
537+
"#{match[1]} #{match[2]}"
538+
# Match: name: Type (without default value)
539+
elsif (match = param.match(/^(#{TRuby::IDENTIFIER_CHAR}+)\s*:/))
565540
match[1]
566541
else
567542
param
@@ -580,11 +555,4 @@ def erase_return_types(source)
580555
result
581556
end
582557
end
583-
584-
# Legacy Compiler for backward compatibility (no IR)
585-
class LegacyCompiler < Compiler
586-
def initialize(config)
587-
super(config, use_ir: false, optimize: false)
588-
end
589-
end
590558
end

lib/t_ruby/heredoc_detector.rb

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# frozen_string_literal: true
2+
3+
module TRuby
4+
# Detects regions that should be skipped during parsing:
5+
# - Heredoc content
6+
# - Block comments (=begin/=end)
7+
class HeredocDetector
8+
# Heredoc start patterns:
9+
# <<IDENTIFIER, <<-IDENTIFIER, <<~IDENTIFIER
10+
# <<'IDENTIFIER', <<"IDENTIFIER"
11+
HEREDOC_START_PATTERN = /<<([~-])?(['"]?)(\w+)\2/
12+
13+
# Detect all skippable ranges in lines (heredocs and block comments)
14+
# @param lines [Array<String>] source lines
15+
# @return [Array<Range>] content ranges to skip (0-indexed)
16+
def self.detect(lines)
17+
ranges = []
18+
i = 0
19+
20+
while i < lines.length
21+
line = lines[i]
22+
23+
# Check for =begin block comment
24+
if line.strip == "=begin"
25+
start_line = i
26+
i += 1
27+
28+
# Find =end
29+
while i < lines.length
30+
break if lines[i].strip == "=end"
31+
32+
i += 1
33+
end
34+
35+
# Range covers from =begin to =end (inclusive)
36+
ranges << (start_line..i) if i < lines.length
37+
# Check for heredoc
38+
elsif (match = line.match(HEREDOC_START_PATTERN))
39+
delimiter = match[3]
40+
squiggly = match[1] == "~"
41+
start_line = i
42+
i += 1
43+
44+
# Find closing delimiter
45+
while i < lines.length
46+
# For squiggly heredoc or dash heredoc, delimiter can be indented
47+
# For regular heredoc, delimiter must be at line start
48+
if squiggly || match[1] == "-"
49+
break if lines[i].strip == delimiter
50+
elsif lines[i].chomp == delimiter
51+
break
52+
end
53+
i += 1
54+
end
55+
56+
# Range covers content lines (after start, up to and including end delimiter)
57+
ranges << ((start_line + 1)..i) if i < lines.length
58+
end
59+
60+
i += 1
61+
end
62+
63+
ranges
64+
end
65+
66+
# Check if a line index is inside any skippable region
67+
# @param line_index [Integer] line index to check
68+
# @param heredoc_ranges [Array<Range>] ranges from detect()
69+
# @return [Boolean]
70+
def self.inside_heredoc?(line_index, heredoc_ranges)
71+
heredoc_ranges.any? { |range| range.include?(line_index) }
72+
end
73+
end
74+
end

lib/t_ruby/ir.rb

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -778,7 +778,8 @@ def build_method(info)
778778
name: info[:name],
779779
params: params,
780780
return_type: info[:return_type] ? parse_type(info[:return_type]) : nil,
781-
body: body
781+
body: body,
782+
visibility: info[:visibility] || :public
782783
)
783784
end
784785

@@ -1051,7 +1052,8 @@ def visit_method_def(node)
10511052
end
10521053

10531054
return_type ||= "untyped"
1054-
emit("def #{node.name}: (#{params}) -> #{return_type}")
1055+
visibility_prefix = format_visibility(node.visibility)
1056+
emit("#{visibility_prefix}def #{node.name}: (#{params}) -> #{return_type}")
10551057
end
10561058

10571059
def visit_class_decl(node)
@@ -1092,6 +1094,17 @@ def visit_instance_variable(node)
10921094
def emit(text)
10931095
@output << ((" " * @indent) + text)
10941096
end
1097+
1098+
def format_visibility(visibility)
1099+
# RBS only supports private visibility, not protected
1100+
# See: https://github.com/ruby/rbs/issues/579
1101+
case visibility
1102+
when :private
1103+
"private "
1104+
else
1105+
""
1106+
end
1107+
end
10951108
end
10961109

10971110
#==========================================================================

lib/t_ruby/lsp_server.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -787,7 +787,7 @@ def generate_semantic_tokens(text)
787787
lines = text.split("\n")
788788

789789
# Parse the document to get IR
790-
parser = Parser.new(text, use_combinator: true)
790+
parser = Parser.new(text)
791791
parse_result = parser.parse
792792
parser.ir_program
793793

0 commit comments

Comments
 (0)