Lexical scope document control directives

[tompng]

Currently, document control directives such as :stopdoc:, :startdoc:, :enddoc:, :nodoc: are buggy.
Documentation and implementation are both wrong. Nobody knows the specification of these directives.
We need to make it clear what the expected specification is, what we are making and fixing.

My proposal is to introduce a lexical scope document control that follows class/module nesting.

Q: Isn't it already lexical scope?

Not actually.

• RDoc stores document_self and document_children to a code object. It's not controlled in parsing phase.
• I think RDoc is trying to emulate lexical scope with wrong mechanism with many bugs.
• Basically enclosed in file scope, but it sometimes leaks (:stopdoc: can prevent indexing a class documented in another file #398).

So we need to implement a real and consistent lexical scope.

Q: Shouldn't it be a macro/preprocessor like `#ifdef` `#endif` instead of lexical scope?

If these directives are normally written without indent, then it makes sense, but it's not.
Preferred indented usage indicates that it is related to class/module's nesting scope.

# Preprocessor-style startdoc/stopdoc. But we don't write like this.
class A
  class B
# :startdoc:
    class C
      X = 1
# :stopdoc:
      Y = 2
    end
  end
end

Make :stopdoc: and :startdoc: lexical scope.

Don't modify container code object's state, just skip documenting in stopdoc region.
When reached an end that close a class/module, it also closes :startdoc: or :stopdoc: in the class/module scope.
This is similar to current behavior but clearly defines what happen if there is no corresponding :startdoc:.

class A
  # :stopdoc:
  class B
    # (stopdoc applied)
    # :startdoc:
    # (startdoc applied)
  end # (close startdoc at L5)
  # (stopdoc applied)
end # (close stopdoc at L2)

Existing bugs

class A
  # :stopdoc:
  class B
    def nodoc; end
  end
  # :startdoc:
end

class A
  class B
    # not documented
    def shoulddoc; end
  end
end

Change :enddoc: to stop documenting in the current lexical scope

It will be almost the same as :stopdoc: but can't do startdoc.
Note that :enddoc: is rarely used (in ruby/*), mostly used at toplevel (in rails/rails).

class A
  class B
    # :enddoc:
    class C
      # (document disabled here too)
    end
  end # (enddoc ends here)
end

class B
  # (document enabled)
  # (originally, class B is marked as document_done)
end

Existing bugs and ambiguity

Existing bug

class A
  class B
    def f; end
  end
  # :enddoc:
  class B # B is hidden from documentation
    def g; end
  end
end

If it's not lexical scope, I think it's not clear whether the following should be.

class A::B
  def f; end
  # :enddoc:
  def g; end
end

class A
  class B
    # documented?
    def h; end
    class C # How about this?
    end
  end
end
class A::B::C # and this?
end

Make :doc: override :stopdoc: (optional)

This will expand use case of :doc: directive. It's just an enhancement idea. Not important.

class A
  private def f; end # :doc: (original usage)
  # :stopdoc:
  ...
  def f; end # :doc: (additional usage)
  ...
  # :startdoc:
end

Change :nodoc:all to globally mark code object as nodoc

Document for the same code object in any other files will be ignored.
Also applies something like :enddoc: to this scope. All method def including unrelated open class inside here is just ignored.
I think this will cover most usecase of :nodoc: to hide internal-use classes/modules.

# a.rb
class Internal # :nodoc: all
  class ::A; end # ignored
end

class A # (documented)
end

# b.rb
class Internal
  # (not documented)
end

Change :nodoc: to mark lexical scope as nodoc

Similar to :enddoc:. For compatibility, :nodoc: can't mark code object as hidden from document.

class Array # :nodoc:
  include CoreExt # (not documented)
  class B; end # (not documented)
  def f; end # (not documented)
end

class Array
  # (this scope is documented)
  def g; end
end

Actual cases that needs this compatibility:

optparse.rb

class OptionParser
  # document comment for OptionParser::Switch
  class Switch
    # :nodoc: (Switch should be documented, only code objects inside should be hidden)

    attr_reader :pattern, :conv, :short, :long, :arg, :desc, :block
  end
end

net/http.rb

Combination of :nodoc: :stopdoc: :startdoc: is a frequently used pattern to avoid RDoc's known bug.

module Net   #:nodoc: (Net should be documented. Only this scope should be ignored)
  # :stopdoc:
  class HTTPBadResponse < StandardError; end
  class HTTPHeaderSyntaxError < StandardError; end
  # :startdoc:
  ...
end

mkmf.rb, pp.rb

# mkmf.rb
class String # :nodoc:
  # Wraps a string in escaped quotes if it contains whitespace.
  def quote
    /\s/ =~ self ? "\"#{self}\"" : "#{self}"
  end
end

# pp.rb
class Array # :nodoc:
  def pretty_print(q) # :nodoc:
    q.group(1, '[', ']') { ... }
  end
end

Existing bugs

class A
  class B
  end
  class B::C # :nodoc:
  end
end

class A::B # A::B is wrongly marked as nodoc
  def f; end
end

Make standalone intermediate :nodoc: and :nodoc: all no-op with warning

In the following intermediate :nodoc: usage, the range of :nodoc: is not obvious.
Someone may expect only after :nodoc: is nodoc, some may expect the whole class A ... end range is nodoc.
I think we should prohibit this usage

# Ambiguous prohibited :nodoc:
class A
  def f; end # Should we prevent this to be documented?

  # :nodoc: (prohibit this)

  # comment
  def g; end # This is not documented
end

# Accepted :nodoc:
class A
  # :nodoc: (Only at the beginning of class)
  # Nodoc directives for current class/module are only accepted
  # on the comment lines before any other comments or ruby codes

  #
  def f; end
  def g; end
end

Related to #1578

class A
  def f; end

  # method g is nodoc in #1578 proposal
  # :nodoc:
  def g; end

  # :nodoc: (this is prohibited in this proposal)

  # comment
  def h; end
end

[st0012]

I used some rough greps to get the directive usage numbers in Ruby and Rails:

Directive	Ruby	Rails	Total
:nodoc:	2,030	2,600	4,630
:nodoc: all	21	93	114
:stopdoc:	287	16	303
:startdoc:	107	12	119
:enddoc:	0	12	12
:doc:	4	199	203

(In Ruby, many stopdoc usages just use EOL as the terminator)

So I'd like to propose another option: let's deprecate :stopdoc:/:startdoc:/:enddoc: instead of fixing them. And let's also do your suggestions related to :nodoc: too.

# :nodoc: = lexical scope only                                                                                  
class A # :nodoc:                                                                                               
  def foo; end  # hidden in this file                                                                           
end                                                                                                             
                                                                                                                
class A                                                                                                         
  def bar; end  # DOCUMENTED (different scope)                                                                  
end                                                                                                             
                                                                                                                
# :nodoc: all = global + children                                                                               
class Internal # :nodoc: all                                                                                    
  def foo; end  # hidden everywhere                                                                             
end                                                                                                             
                                                                                                                
class Internal                                                                                                  
  def bar; end  # ALSO hidden (globally marked)                                                                 
end                                                                                                             
                                                                                                                
# :doc: = force override                                                                                        
class Internal # :nodoc: all                                                                                    
  def public_api; end # :doc:  # DOCUMENTED (forced)                                                            
end    

Yes, it'll be a bigger breaking change. But in the long term, I think maintaining this many document behaviour directives is only making things hard to maintain while confusing developers. AFAIK, most languages (Rust, Python, JS) don't offer this many doc directives either.