main: using regex for choosing a parser for given file name #4270

masatake · 2025-06-08T23:33:56Z

main: using regex for choosing a parser for the given file name

This change extends --map- option to support regular
expression matching with the full file name.

The original --map- option supports glob based matching
and extension comparison with the file basename.
However, two methods are not enough if the file names are too
generic. See #3287 .

The regular expression passed to --map- must be surround
by % character like

--map-RpmMacros='%(.*/)?macros.d/macros.([^/]+)$%'

If you want to match in a case-insensitive way, append `i' after the second % like

--map-RpmMacros='%(.*/)?macros.d/macros.([^/]+)$%i'

If you want to use % as part of an expression, put \ before % for escaping.

TODO:

codecov · 2025-09-27T18:39:53Z

Codecov Report

❌ Patch coverage is 94.01993% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.90%. Comparing base (cea36d3) to head (214afc5).
⚠️ Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
main/parse.c	95.38%	9 Missing ⚠️
main/rexprcode.c	83.78%	6 Missing ⚠️
optlib/rpmMacros.c	0.00%	2 Missing ⚠️
main/options.c	98.24%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4270      +/-   ##
==========================================
+ Coverage   85.89%   85.90%   +0.01%     
==========================================
  Files         250      251       +1     
  Lines       62341    62510     +169     
==========================================
+ Hits        53545    53699     +154     
- Misses       8796     8811      +15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull Request Overview

This PR extends the --map-<LANG> option to support regular expression matching for file names, addressing limitations where glob patterns and extension matching are insufficient for generic file names. The implementation adds a new regex-based mapping type alongside existing extension and pattern mappings.

Key Changes:

Introduced regex pattern support using %regex%[i] syntax for language file mappings
Added new rexprcode module to handle regex compilation and matching
Extended optlib2c to generate C code from regex mapping definitions

Reviewed Changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
main/rexprcode.c	New module implementing regex pattern compilation, matching, and encoding
main/rexprcode_p.h	Public interface for regex code operations
main/parse.c	Core integration of regex matching into language detection logic
main/parse.h	Added rExprSrc structure definition and REXPR_LAST_ENTRY macro
main/parse_p.h	Extended langmapType enum with LMAP_REXPR flag
main/options.c	Command-line option parsing for regex patterns with icase flag support
optlib/rpmMacros.ctags	Example usage replacing commented-out patterns with regex
optlib/rpmMacros.c	Generated C code with regex mapping definitions
misc/optlib2c	Extended Perl script to parse and generate regex mapping code
source.mak	Build system updates for new source files
win32/ctags_vs2013.vcxproj	Visual Studio project file updates
win32/ctags_vs2013.vcxproj.filters	Visual Studio filter file updates
Tmain/list-map-rexprs.d/*	Test cases for new --list-map-rexprs option
Tmain/versioning.d/stdout-expected.txt	Updated test output expectations

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

main/rexprcode.c

main/options.c

main/parse.c

main/options.c

Copilot · 2025-10-19T18:02:21Z

misc/optlib2c

 	       unless ($_[0]->{'langdef'} eq $1);
 	   my $spec = $2;
-	   if ($spec =~ /\((.*)\)/) {
+	   if ($spec =~ /%(.+)%(i)?/) {


The regex pattern %(.+)% is greedy and will match incorrectly if the expression contains '%' characters, even with escaping. For example, %a%b%c% would capture a%b%c instead of a. The pattern should be non-greedy: %(.+?)%(i)?/ or better yet, should properly handle escaped '%' characters in the capture group.

Suggested change

if ($spec =~ /%(.+)%(i)?/) {

if ($spec =~ /%(.+?)%(i)?/) {

main/rexprcode_p.h

Copilot

Pull Request Overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 1 comment.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-19T19:02:34Z

misc/optlib2c

 	       unless ($_[0]->{'langdef'} eq $1);
 	   my $spec = $2;
-	   if ($spec =~ /\((.*)\)/) {
+	   if ($spec =~ /%(.+)%(i)?/) {


The regex should use a non-greedy quantifier .+? instead of .+ to prevent matching across multiple patterns when there are multiple % characters in the input. This could cause incorrect parsing of escaped % characters.

Suggested change

if ($spec =~ /%(.+)%(i)?/) {

if ($spec =~ /%(.+?)%(i)?/) {

Copilot

Pull Request Overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 5 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

misc/optlib2c

main/parse.c

main/options.c

Copilot

Pull Request Overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

main/parse.h

Copilot · 2025-10-19T19:22:39Z

misc/optlib2c

+	   if ($spec =~ /%(.+?)%(i|\{icase\})?/) {
+	       my $rexpr = { expr => $1,
+			     iCase => (defined $2 && ($2 eq 'i' || $2 eq 'icase'))? 1: 0 };


The regex allows {icase} as an alternative to i, but this syntax is not documented in the PR description or help text. Either document this alternative syntax or remove it to avoid confusion.

Suggested change

if ($spec =~ /%(.+?)%(i|\{icase\})?/) {

my $rexpr = { expr => $1,

iCase => (defined $2 && ($2 eq 'i' || $2 eq 'icase'))? 1: 0 };

if ($spec =~ /%(.+?)%(i)?/) {

my $rexpr = { expr => $1,

iCase => (defined $2 && $2 eq 'i')? 1: 0 };

Copilot · 2025-10-19T19:22:40Z

main/options.c

+
+static flagDefinition langmapRexprFlagDef[] = {
+	{ 'i', "icase",  langmap_rexpr_icase_short,  langmap_rexpr_icase_long,
+	  NULL, "applied in a case-insensitive manner"},


The long flag name is 'icase', but the optlib2c script also accepts '{icase}' syntax (line 296 in misc/optlib2c). These should be consistent, or the alternative syntax should be documented.

Suggested change

NULL, "applied in a case-insensitive manner"},

NULL, "applied in a case-insensitive manner (accepts both 'icase' and '{icase}' syntax)"},

Copilot

Pull Request Overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

main/options.c

main/parse.c

Copilot

Pull Request Overview

Copilot reviewed 21 out of 23 changed files in this pull request and generated no new comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot

Pull Request Overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated no new comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot

Pull Request Overview

Copilot reviewed 30 out of 32 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-27T06:03:33Z

man/ctags.1.rst.in

+
+	If you run @CTAGS_NAME_EXECUTABLE@ with ``@CTAGS_NAME_EXECUTABLE@ -R src``,
+	the match is performed with ``src/lib/data.c`` and ``src/lib/logic.c`` If you
+	give ``--langmap='YourParser:%src/lib/.*\.c%'``, @CTAGS_NAME_EXECUTABLE@


Corrected option name from '--langmap' to '--map-YourParser'.

Suggested change

give ``--langmap='YourParser:%src/lib/.*\.c%'``, @CTAGS_NAME_EXECUTABLE@

give ``--map-YourParser='%src/lib/.*\.c%'``, @CTAGS_NAME_EXECUTABLE@

Copilot · 2025-10-27T06:03:34Z

docs/man/ctags.1.rst

+
+	If you run ctags with ``ctags -R src``,
+	the match is performed with ``src/lib/data.c`` and ``src/lib/logic.c`` If you
+	give ``--langmap='YourParser:%src/lib/.*\.c%'``, ctags


Corrected option name from '--langmap' to '--map-YourParser'.

Suggested change

give ``--langmap='YourParser:%src/lib/.*\.c%'``, ctags

give ``--map-YourParser='%src/lib/.*\.c%'``, ctags

Copilot

Pull Request Overview

Copilot reviewed 30 out of 32 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

main/routines.c

main/rexprcode.c

main/parse.c

Copilot

Pull Request Overview

Copilot reviewed 35 out of 37 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull Request Overview

Copilot reviewed 35 out of 37 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull Request Overview

Copilot reviewed 35 out of 37 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: Masatake YAMATO <[email protected]>

…nsExtensionNew Delete baseFilenameSansExtensionNew() from the source tree. Signed-off-by: Masatake YAMATO <[email protected]>

The original code used a boolean value to toggle how filenames were mapped to the parser by glob-like pattern or by extension. To support the third way mapping a file name to a parser, by regular expression pattern, we will use an enum value instead of Boolean. Signed-off-by: Masatake YAMATO <[email protected]>

Signed-off-by: Masatake YAMATO <[email protected]>

This change extends --map-<LANG> option to support regular expression matching with the full file name. The original --map-<LANG> option supports the glob based matching and the extension comparison with the file basename. However, two methods are not enough if the file names are too generic. See universal-ctags#3287 . The regular expression passed to --map-<LANG> must be surrounded by % character like --map-RpmMacros='%(.*/)?macros\.d/macros\.([^/]+)$%' If you want to match in a case-insensitive way, append `i' after the second % like --map-RpmMacros='%(.*/)?macros\.d/macros\.([^/]+)$%i' If you want to use % as part of an expression, put \ before % for escaping. Signed-off-by: Masatake YAMATO <[email protected]>

…options Signed-off-by: Masatake YAMATO <[email protected]>

Signed-off-by: Masatake YAMATO <[email protected]>

Copilot

Pull Request Overview

Copilot reviewed 30 out of 32 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-27T11:13:21Z

man/ctags.1.rst.in

+
+``--list-maps[=(<language>|all)]``
+	Lists the file name patterns, the file extensions, and the relative-path
+	regular extensions which associate a file name with a language for either the


The term 'regular extensions' on line 1302 should be 'regular expressions' to match terminology used elsewhere in the documentation.

Suggested change

regular extensions which associate a file name with a language for either the

regular expressions which associate a file name with a language for either the

Copilot · 2025-10-27T11:13:21Z

docs/man/ctags.1.rst

+
+``--list-maps[=(<language>|all)]``
+	Lists the file name patterns, the file extensions, and the relative-path
+	regular extensions which associate a file name with a language for either the


The term 'regular extensions' on line 1302 should be 'regular expressions' to match terminology used elsewhere in the documentation.

Suggested change

regular extensions which associate a file name with a language for either the

regular expressions which associate a file name with a language for either the

… parser Signed-off-by: Masatake YAMATO <[email protected]>

Copilot

Pull Request Overview

Copilot reviewed 30 out of 32 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

masatake marked this pull request as draft June 8, 2025 23:34

masatake changed the title ~~main: using regex for choosing a paser for given file name~~ [WIP] main: using regex for choosing a paser for given file name Jun 8, 2025

masatake force-pushed the main--rexpr branch 2 times, most recently from a5a3a28 to c50f467 Compare September 27, 2025 18:32

masatake force-pushed the main--rexpr branch from c50f467 to 24ae704 Compare October 14, 2025 20:14

masatake changed the title ~~[WIP] main: using regex for choosing a paser for given file name~~ [WIP] main: using regex for choosing a parser for given file name Oct 17, 2025

masatake force-pushed the main--rexpr branch from 24ae704 to 625a545 Compare October 19, 2025 18:00

masatake requested a review from Copilot October 19, 2025 18:00

Copilot AI reviewed Oct 19, 2025

View reviewed changes

masatake force-pushed the main--rexpr branch from 625a545 to d59092b Compare October 19, 2025 19:01

masatake requested a review from Copilot October 19, 2025 19:01

Copilot AI reviewed Oct 19, 2025

View reviewed changes

masatake force-pushed the main--rexpr branch from d59092b to 7ee055b Compare October 19, 2025 19:11

masatake requested a review from Copilot October 19, 2025 19:11

Copilot AI reviewed Oct 19, 2025

View reviewed changes

misc/optlib2c Show resolved Hide resolved

main/parse.c Outdated Show resolved Hide resolved

main/parse.c Show resolved Hide resolved

main/parse.c Show resolved Hide resolved

main/options.c Outdated Show resolved Hide resolved

masatake force-pushed the main--rexpr branch from 7ee055b to 1c50a08 Compare October 19, 2025 19:21

masatake requested a review from Copilot October 19, 2025 19:21

Copilot AI reviewed Oct 19, 2025

View reviewed changes

masatake force-pushed the main--rexpr branch 2 times, most recently from 231a606 to f865e33 Compare October 20, 2025 02:17

masatake requested a review from Copilot October 20, 2025 02:19

Copilot AI reviewed Oct 20, 2025

View reviewed changes

main/options.c Outdated Show resolved Hide resolved

main/parse.c Outdated Show resolved Hide resolved

masatake force-pushed the main--rexpr branch from f865e33 to d2631e3 Compare October 20, 2025 05:45

masatake requested a review from Copilot October 20, 2025 05:48

Copilot AI reviewed Oct 20, 2025

View reviewed changes

masatake force-pushed the main--rexpr branch from d2631e3 to 4470454 Compare October 20, 2025 06:36

masatake requested a review from Copilot October 20, 2025 06:46

Copilot AI reviewed Oct 20, 2025

View reviewed changes

masatake requested a review from Copilot October 20, 2025 10:07

Copilot AI reviewed Oct 27, 2025

View reviewed changes

masatake force-pushed the main--rexpr branch from 12f3233 to 72669ab Compare October 27, 2025 07:15

masatake requested a review from Copilot October 27, 2025 07:23

Copilot AI reviewed Oct 27, 2025

View reviewed changes

main/routines.c Show resolved Hide resolved

main/rexprcode.c Outdated Show resolved Hide resolved

main/parse.c Outdated Show resolved Hide resolved

masatake force-pushed the main--rexpr branch 2 times, most recently from 5e33670 to 2242572 Compare October 27, 2025 09:49

masatake requested a review from Copilot October 27, 2025 09:51

Copilot AI reviewed Oct 27, 2025

View reviewed changes

masatake requested a review from Copilot October 27, 2025 10:01

Copilot AI reviewed Oct 27, 2025

View reviewed changes

masatake requested a review from Copilot October 27, 2025 10:26

Copilot AI reviewed Oct 27, 2025

View reviewed changes

masatake added 10 commits October 27, 2025 20:10

main: add a new helper function filenameSansExtensionNew

41d137d

Signed-off-by: Masatake YAMATO <[email protected]>

main,cosmetic: clean up whitespace with M-x whitespace-cleanup (Emacs)

68cbbf2

Signed-off-by: Masatake YAMATO <[email protected]>

main,refactor: use filenameSansExtensionNew instead of baseFilenameSa…

37349c4

…nsExtensionNew Delete baseFilenameSansExtensionNew() from the source tree. Signed-off-by: Masatake YAMATO <[email protected]>

main,refactor: don't use else if if-block does return always

01694f4

Signed-off-by: Masatake YAMATO <[email protected]>

Tmain: add a case testing --list-map-rexprs and --map-<LANG>=%REXPR% …

0b6d66f

…options Signed-off-by: Masatake YAMATO <[email protected]>

optlib2c: support --map-<LANG>=%REXPR% option

f1f0602

Signed-off-by: Masatake YAMATO <[email protected]>

rpmMacros: parse macros.d/macros.*

129afac

Signed-off-by: Masatake YAMATO <[email protected]>

docs(web): write about --map-<LANG>=%REXPR% to NEWS file

cd9b32e

Signed-off-by: Masatake YAMATO <[email protected]>

masatake force-pushed the main--rexpr branch from 2242572 to 28c1ac2 Compare October 27, 2025 11:11

masatake requested a review from Copilot October 27, 2025 11:12

Copilot AI reviewed Oct 27, 2025

View reviewed changes

docs(man): write about the way to use regular expressions to choose a…

214afc5

… parser Signed-off-by: Masatake YAMATO <[email protected]>

masatake force-pushed the main--rexpr branch from 28c1ac2 to 214afc5 Compare October 27, 2025 11:15

masatake requested a review from Copilot October 27, 2025 11:15

Copilot AI reviewed Oct 27, 2025

View reviewed changes

masatake merged commit 5a28e00 into universal-ctags:master Oct 27, 2025
83 checks passed

	NULL, "applied in a case-insensitive manner"},
	NULL, "applied in a case-insensitive manner (accepts both 'icase' and '{icase}' syntax)"},

	give ``--langmap='YourParser:%src/lib/.*\.c%'``, @CTAGS_NAME_EXECUTABLE@
	give ``--map-YourParser='%src/lib/.*\.c%'``, @CTAGS_NAME_EXECUTABLE@

	regular extensions which associate a file name with a language for either the
	regular expressions which associate a file name with a language for either the

Uh oh!

main: using regex for choosing a parser for given file name #4270

main: using regex for choosing a parser for given file name #4270

Uh oh!

Conversation

masatake commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Sep 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

masatake commented Jun 8, 2025 •

edited

Loading

codecov bot commented Sep 27, 2025 •

edited

Loading