Unicode characters breaks YARD generation

[thomthom]

Unicode characters in the Ruby or README files is breaking YARD with incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError

When I used the superscript 3 character (³) in my files, YARD will fail to generate correct output.

I have verified that I saved the files with UTF-8 encoding by inspecting the files with a hex-editor:

Steps to reproduce

.yardopts

--title "test³"
--markup markdown
*.rb
-
README.md

hello.rb

module Hello
    # hi³ there.
    def self.hi
        345
    end
end

README.md

# Hello World³

Actual Output

C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/layout.erb:21:in `_erb_cache_5': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:289:in `erb'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/setup.rb:62:in `layout'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:369:in `render_section'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:261:in `block (2 levels) in run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:258:in `each'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:258:in `block in run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:400:in `add_options'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:257:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:136:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:52:in `block in serialize_index'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/engine.rb:123:in `block in with_serializer'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/logging.rb:82:in `capture'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/engine.rb:121:in `with_serializer'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:51:in `serialize_index'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:68:in `serialize_file'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:11:in `block in init'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:10:in `each'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:10:in `each_with_index'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:10:in `init'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:195:in `initialize'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:131:in `new'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:136:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/engine.rb:105:in `generate'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/yardoc.rb:357:in `run_generate'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/yardoc.rb:267:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/command.rb:14:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/command_parser.rb:72:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/command_parser.rb:54:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/bin/yard:13:in `<top (required)>'
        from C:/Ruby32-x64/bin/yard:32:in `load'
        from C:/Ruby32-x64/bin/yard:32:in `<main>'

If I remove the superscript 3 from the README the output becomes:

C:\Users\tthomas2\Desktop\yard-bug>yard doc --debug
[debug]: Parsing ["*.rb"] with `ruby` parser
[debug]: Parsing hello.rb
[debug]: Serializing to .yardoc/objects/root.dat
[debug]: Re-generating object ...
[debug]: Re-generating object Hello...
[debug]: Generating asset js/jquery.js
[debug]: Serializing to doc/js/jquery.js
[debug]: Generating asset js/app.js
[debug]: Serializing to doc/js/app.js
[debug]: Generating asset js/full_list.js
[debug]: Serializing to doc/js/full_list.js
[debug]: Generating asset css/style.css
[debug]: Serializing to doc/css/style.css
[debug]: Generating asset css/common.css
[debug]: Serializing to doc/css/common.css
[debug]: Generating asset css/full_list.css
[debug]: Serializing to doc/css/full_list.css
[debug]: Generating asset class_list.html
[debug]: Serializing to doc/class_list.html
[debug]: Generating asset method_list.html
[debug]: Serializing to doc/method_list.html
[debug]: Generating asset file_list.html
[debug]: Serializing to doc/file_list.html
[debug]: Generating asset frames.html
[debug]: Serializing to doc/frames.html
[debug]: Serializing to doc/_index.html
[debug]: Serializing to doc/index.html
[debug]: Serializing to doc/file.README.html
[debug]: Serializing to doc/top-level-namespace.html
[error]: Exception occurred while generating 'Hello.html'
[error]: CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
[error]: Stack trace:
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/layout.erb:21:in `_erb_cache_5'
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:289:in `erb'
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/setup.rb:62:in `layout'
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:369:in `render_section'
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:261:in `block (2 levels) in run'
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:258:in `each'

Files:           1
Modules:         1 (    1 undocumented)
Classes:         0 (    0 undocumented)
Constants:       0 (    0 undocumented)
Attributes:      0 (    0 undocumented)
Methods:         1 (    0 undocumented)
 50.00% documented

The documentation "completes" but with an inline error message that doesn't stop the process. This time it's the superscript 3 in hello.rb that breaks.

I'm wondering if YARD is loading text from templates as ASCII and thus failing to combine UTF-8 and ASCII strings.

Expected Output

Expected YARD to generate the documentation without errors.

Environment details:

• OS: Windows 10
• Ruby version (ruby -v): ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt]
• YARD version (yard -v): yard 0.9.34

I have read the Contributing Guide.

[lsegal]

I can't seem to reproduce this on ruby 3.3.x with yard 0.9.35. Is this still an issue for you?