Basic: extract namespaces (partially derived from #3276)

[masatake]

This pull request is inspired by and derived from #3276 submitted by @Glog78 partially.

This pull request doesn't deal with declarations.
The pull request for dealing declarations comes after merging this pull request.

[codecov]

Codecov Report

Merging #3284 (8001720) into master (39e2e3d) will increase coverage by 0.25%.
The diff coverage is 97.59%.

@@            Coverage Diff             @@
##           master    #3284      +/-   ##
==========================================
+ Coverage   85.21%   85.47%   +0.25%     
==========================================
  Files         210      210              
  Lines       49689    50669     +980     
==========================================
+ Hits        42343    43308     +965     
- Misses       7346     7361      +15     

Impacted Files	Coverage Δ
parsers/openapi.c	96.15% <91.66%> (ø)
parsers/basic.c	98.80% <98.59%> (-0.24%)	⬇️
parsers/python.c	98.53% <0.00%> (+<0.01%)	⬆️
parsers/tex.c	97.60% <0.00%> (+0.13%)	⬆️
main/vstring.c	95.26% <0.00%> (+2.10%)	⬆️
parsers/cxx/cxx_parser.c	88.87% <0.00%> (+3.32%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9369f18...8001720. Read the comment docs.

[Glog78]

@masatake i have put you a little input.bas for testing https://termbin.com/x9wo
basically i don't see currently the extern defined functions in the tag file.

also declare sub or declare functions is a forward declaration so it's very hard to handle. But i would rather prefer the same identifier twice than never.

[masatake]

@masatake i have put you a little input.bas for testing https://termbin.com/x9wo
basically i don't see currently the extern defined functions in the tag file.

Thank you. I needed such an input file for testing.

For https://termbin.com/x9wo, there are two ways to extract declarations:

(1) using reference tags
(2) introducing a kind for declarations

C paresr uses (2). However, it will be better to convert it to (1).
See #1783.

About the Basic parser, we don't have to think about compatibility. So we can choose (1).

How about following tags output for https://termbin.com/x9wo:

curses	input.bas	/^namespace curses 'public use this$/;"	kind:namespace	line:2	language:Basic	roles:def	end:32
win	input.bas	/^		type win as curses.curses_priv._win_st$/;"	kind:type	line:8	language:Basic	scope:namespace:curses	roles:def
_Init	input.bas	/^		declare	function	_Init	alias "initscr" () as win ptr$/;"	kind:function	line:10	language:Basic	scope:namespace:curses	roles:decl	extras:reference
GetMaxX	input.bas	/^		declare	function	GetMaxX alias	"getmaxx" (byval as win ptr) as integer$/;"	kind:function	line:11	language:Basic	scope:namespace:curses	roles:decl	extras:reference
GetMaxY	input.bas	/^		declare	function	GetMaxY alias "getmaxy"	(byval as win ptr) as integer$/;"	kind:function	line:12	language:Basic	scope:namespace:curses	roles:decl	extras:reference
CreateWindow	input.bas	/^		declare	function	CreateWindow alias "newwin" (byval as integer, byval as integer, byval as int/;"	kind:function	line:13	language:Basic	scope:namespace:curses	roles:decl	extras:reference
CreateSubWindow	input.bas	/^		declare	function	CreateSubWindow alias "subwin" (byval as win ptr, byval as integer, byval as /;"	kind:function	line:14	language:Basic	scope:namespace:curses	roles:decl	extras:reference
GetChar	input.bas	/^		declare function	GetChar alias "getch" () as UByte$/;"	kind:function	line:15	language:Basic	scope:namespace:curses	roles:decl	extras:reference
CBreak	input.bas	/^		declare function	CBreak alias "cbreak" () as integer$/;"	kind:function	line:16	language:Basic	scope:namespace:curses	roles:decl	extras:reference
EreaseTerm	input.bas	/^		declare function	EreaseTerm alias "erase" () as integer$/;"	kind:function	line:17	language:Basic	scope:namespace:curses	roles:decl	extras:reference
ClearTerm	input.bas	/^		declare function	ClearTerm alias "clear" () as integer$/;"	kind:function	line:18	language:Basic	scope:namespace:curses	roles:decl	extras:reference
EreaseWindow	input.bas	/^		declare function EreaseWindow alias "werase" (byval as win ptr) as integer$/;"	kind:function	line:19	language:Basic	scope:namespace:curses	roles:decl	extras:reference
ClearWindow	input.bas	/^		declare function ClearWindow alias "wclear" (byval as win ptr) as integer$/;"	kind:function	line:20	language:Basic	scope:namespace:curses	roles:decl	extras:reference
DeleteWindow	input.bas	/^		declare function DeleteWindow alias "delwin" (byval as win ptr) as integer$/;"	kind:function	line:21	language:Basic	scope:namespace:curses	roles:decl	extras:reference
_Exit	input.bas	/^		declare function _Exit alias "endwin" () as integer$/;"	kind:function	line:22	language:Basic	scope:namespace:curses	roles:decl	extras:reference
RefresTerm	input.bas	/^		declare function RefresTerm alias "refresh" () as integer$/;"	kind:function	line:23	language:Basic	scope:namespace:curses	roles:decl	extras:reference
RefreshWindow	input.bas	/^		declare function RefreshWindow alias "wrefresh" (byval as win ptr) as integer$/;"	kind:function	line:24	language:Basic	scope:namespace:curses	roles:decl	extras:reference
PrintTerm	input.bas	/^		declare function PrintTerm  alias "printw" (byval as zstring ptr, ... ) as integer$/;"	kind:function	line:25	language:Basic	scope:namespace:curses	roles:decl	extras:reference
PrintWindow	input.bas	/^		declare function PrintWindow alias "wprintw" (byval as win ptr, byval as zstring ptr, ... ) as/;"	kind:function	line:26	language:Basic	scope:namespace:curses	roles:decl	extras:reference
SetCursorPositionTerm	input.bas	/^		declare function SetCursorPositionTerm alias "move" (byval as integer,byval as integer) as int/;"	kind:function	line:27	language:Basic	scope:namespace:curses	roles:decl	extras:reference
SetCursorPositionWindow	input.bas	/^		declare function SetCursorPositionWindow alias "wmove" (byval as win ptr, byval as integer, by/;"	kind:function	line:28	language:Basic	scope:namespace:curses	roles:decl	extras:reference
input.bas	input.bas	1;"	kind:file	line:1	language:Basic	roles:def	extras:inputFile	end:34	epoch:1643035661

?

[Glog78]

can we strip down the description tags ... so leading spaces/tabs are removed / multiple spaces , tabs are shortened down to 1 space? Does this even make sense ? Also i will check if i can make an example for qbasic

[masatake]

can we strip down the description tags ... so leading spaces/tabs are removed / multiple spaces , tabs are shortened down to 1 space? Does this even make sense ?

What is your concern?
The output of ctags includes many spaces/tabs. I wonder why it is a matter.
These spaces/tabs come from the input.bas (https://termbin.com/x9wo).

Also i will check if i can make an example for basic

Thank you!

[Glog78]

Hier is the promised small example ... the syntax between freebasic and qbasic (qb64) differs quiet alot in a way

' https://wiki.qb64.org/wiki/Variable_Types << this is important
' https://wiki.qb64.org/wiki/C_Libraries
' https://wiki.qb64.org/wiki/DECLARE_DYNAMIC_LIBRARY

https://termbin.com/uxl7

Also the cleaning of the description could make sure that editors who use this field for additional informations when autocomplete don't print all this "wasted data". But i would agree that most likely every editor will reduce it anyway.

[masatake]

It is not wasted data. To use the tags file for the "jumping to definition" feature, the area that you called "description", is referred to by editors as search patterns.
Assuming using tags files only for autocompletion is not a good idea.

Can I merge this pull request?
I don't want to delay merging this pull request till finishing the code for qb64.

Did you write https://termbin.com/x9wo ?
I included it as an input file to my pull request.
I would like to make clear where the test input comes from.

[Glog78]

you can merge this one. Awesome work and thanks alot.
The file is a collection of example code from the qb64 documentation. I put the links inside of the termbin and a little above in my last comment.

[masatake]

The file is a collection of example code from the qb64 documentation. I put the links inside of the termbin and a little above in my last comment.

Are you really talking about https://termbin.com/x9wo?
I assumed https://termbin.com/x9wo is an example of freebasic code. Am I wrong?
fd45032#diff-b7de8933d3e962cbafa71d2901494b5079db63b04bae480eb54db9d1788344ae
In this test input, I added

' Taken from https://termbin.com/x9wo

If you took this example somewhere, I would like to know it URL, and I would like to update the comment.

[masatake]

BTW, you can use -n option to shrink the wanted data. The option suppresses the patterns.

[Glog78]

I assumed https://termbin.com/x9wo is an example of freebasic code. Am I wrong? << oh yes ... this is freebasic code i wrote to play around (testing c library use from freebasic)

qb64 examples -> https://termbin.com/uxl7

[masatake]

I assumed https://termbin.com/x9wo is an example of freebasic code. Am I wrong? << oh yes ... this is freebasic code i wrote to play around (testing c library use from freebasic)

I see. Thank you.

I would like to merge up to the code for namespace.
I would like to take more time to think about how to deal with declarations.

[masatake]

       extern "c"

               type win as curses.curses_priv._win_st

               declare function        _Init   alias "initscr" () as win ptr

In this example, "initscr" is a name of function written in C.
"_Init" is a name for calling "initscr" from Basic.
"_Init" is a newly introduced name for the purpose.

Am I correct?

[masatake]

I found more examples in https://github.com/freebasic/fbc/tree/master/examples .

#inclib "gtk-3"
...
declare sub gtk_init(byval argc as long ptr, byval argv as zstring ptr ptr ptr)
...
declare function gtk_check_version_ alias "gtk_check_version"(byval required_major as guint, byval required_minor as guint, byval required_micro as guint) as const gchar ptr

In this line, gtk_check_version_ is newly introduced name that is for calling gtk_check_version in gtk-3.
So I think gtk_check_version_ is a definition.

expected tags are:

gtk_init	gtk3.bi	/^	declare sub gtk_init(byval argc as long ptr, byval argv as zstring ptr ptr ptr)$/;"	kind:function	line:6799	language:Basic	roles:decl	extras:reference
gtk_check_version_	gtk3.bi	/^declare function gtk_check_version_ alias "gtk_check_version"(byval required_major as guint, byv/;"	kind:function	line:6795	language:Basic	roles:def	alias:gtk_check_version

gtk_check_version is also a candidate for extraction.

As far as I can remember correctly, cython has similar concepts. So kinds, roles, and fields design can be shared between the two parsers.

[Glog78]

You assumed right about the Init and and also gtk_check_version .
What i am not sure about is if gtk_init is a forward declaration to a later defined sub in basic or a forward declaration to a c function.
In case of a forward declaration in basic ... you will have 2 tags

example:

declare sub dosomething

sub dosomething
end sub

A part of the documentation can be found here -> https://documentation.help/FreeBASIC/KeyPgDeclare.html

[masatake]

gtk_check_version_	gtk3.bi	/^declare function gtk_check_version_ alias "gtk_check_version"(byval required_major as guint, byv/;"	kind:function	line:6795	language:Basic	roles:def	alias:gtk_check_version

This should be

gtk_check_version_	gtk3.bi	/^declare function gtk_check_version_ alias "gtk_check_version"(byval required_major as guint, byv/;"	kind:function	line:6795	language:Basic	roles:def	alias::gtk_check_version

If "Lib" is specified in a "Declare" line, we can put the name of the library to alias:HERE:gtk_check_version`.
The information may help a tool to track the definitions. I have to study more but, making the alias field extensible may be a good idea. I must write these design decisions to a man page.

[masatake]

What i am not sure about is if gtk_init is a forward declaration to a later defined sub in basic or a forward declaration to a c function.

YES! I'm thinking about it. I recognized we cannot distinguish the two. Maybe this is the reason why we are taking a lot of time for design. The final version of my design looks good to me. The next step is implementation.

[masatake]

... you will have 2 tags

One is a reference tag and another is a definition tag.
A tool for completion sees two tags having the same name.
The tool may keep the definition tag.

[masatake]

After thinking more, reserving a slot for storing Lib in alias:Lib:gtk_check_version is not a perfect idea.
If a "Lib" includes : character, extracting values from a alias: field becomes harder.
Instead, adding lib: field looks better. We can add lib: field later.

[masatake]

It seems that I misunderstand the meaning of "alias".

declare sub dosomething  alias "_do"

sub dosomething
end sub

It seems that this is a valid code.
I assumed an alias name is for specifying a name defined in a library written in a foreign language.
It is one of usage.
However, an alias can be used for controlling how a function (or something language object) is named outside Basic language.
An alias can be used for exporting a name and importing an external object.

Declaration with an alias may define a new name or declare something. There is no good way to distinguish the two.

[masatake]

I must change the policy. Let's do the same as C parser: introducing "declaration" kind and extracting declarations as definition tags.