How to indent text written in a python block based on the location of "<%"

[rdan]

Hello,

I have the following template content:

        some text
        <%
           some python code
        %>
        some text

"some python code" will add some lines between the 2 "some text" and the content of these lines must be indented like both "some text".
So, what I did is to consider the location of "<%" as the reference for all the text written by "some python code".
To achieve that, I created the following function:

   def GetIndent(context):
      return " " * len(context._buffer_stack[-1].getvalue().split("\n")[-1])

which provides the indentation of "<%" not in the template but in the text that has been processed up to the "<%".
Then I can prefix each "context.write()" (except the first one which is already indented correctly) with the indentation string returned by the function "GetIndent()".

The problem of this function is that it uses internal data of mako class.

So I open this discussion in case this could interest enough users of mako to eventually provide this as a feature of mako.

Or maybe there are better ways to achieve this goal.

Thank you
Ronan

[zzzeek]

the idiomatic way to cancel the extra newline caused by the python block is to use a backslash:

t = Template("""
        some text\
        <%
           x = 10
        %>
        some text

""")

but note that will still render trailing whitespace which you have on the line with the <% symbol

to really not have the whitespace render without applying post-processing, you have to not put it in the template in the first place, like this:

t = Template("""
        some text<%
           x = 10
        %>
        some text

""")

Then I can prefix each "context.write()" (except the first one which is already indented correctly) with the indentation string returned by the function "GetIndent()".

I would need to see an example how this function is intended to be used, it sounds like it requires programmatic use of context.write() which is fine to use, but a Mako feature would need to be some syntactical approach that eases the use of <% %>. Perhaps a new symbol like <%= or something like that, indicating that all whitespace on the same lines as the enclosing markers should be quashed.

[zzzeek]

OK, so that's a use case. let's see if we can encapsulate it without exposing internals or needing extra functions.

would this work? I don't know if it's robust and it likely is not. would need some review.

diff --git a/mako/runtime.py b/mako/runtime.py
index 23401b7..db0d4cf 100644
--- a/mako/runtime.py
+++ b/mako/runtime.py
@@ -137,12 +136,20 @@ class Context:

         return self._data.get(key, builtins.__dict__.get(key, default))

-    def write(self, string):
+    def write(self, string, *, indent_current=False):
         """Write a string to this :class:`.Context` object's
         underlying output buffer."""

+        if indent_current:
+            string = self._indent_current(string)
         self._buffer_stack[-1].write(string)

+    def _indent_current(self, string):
+        current_indent, prior_indent = self._buffer_stack[
+            -1
+        ]._get_current_and_prior_indent()
+        return " " * (current_indent - prior_indent) + string
+
     def writer(self):
         """Return the current writer function."""

diff --git a/mako/util.py b/mako/util.py
index 470713a..8e38427 100644
--- a/mako/util.py
+++ b/mako/util.py
@@ -161,6 +161,26 @@ class FastEncodingBuffer:
         else:
             return self.delim.join(self.data)

+    def _get_current_and_prior_indent(self):
+        # get the whole string on the right up until the first newline
+        the_whole_string = ""
+        for elem in reversed(self.data):
+            the_whole_string = elem + the_whole_string
+            if "\n" in self.data:
+                # TODO: what if the current element ends in "\n" and has
+                # no other "\n" in it?
+                break
+
+        current_indent_string = the_whole_string.rstrip("\n")
+        last_segment = current_indent_string.split("\n")[-1]
+        whitespace = re.match(r"^ +", last_segment)
+        current = len(whitespace.group(0)) if whitespace else 0
+
+        last_segment = the_whole_string.split("\n")[-1]
+        whitespace = re.match(r"^ +", last_segment)
+        prior = len(whitespace.group(0)) if whitespace else 0
+
+        return current, prior

 class LRUCache(dict):

then a proof of concept is below, where you pass indent_current=True and it figures it all out:

from mako.template import Template


t = Template(
    r"""

        ENUM
        [
            <%
            for Group in ListOfGroups:
                if ("Parameter" in Group):
                    context.write(f"{Group['Name']}: {Group['Parameter']}\n", indent_current=True)
            %>
        ] GROUP;
"""
)


print(
    t.render(
        ListOfGroups=[
            {"Name": "Group1", "Parameter": "Param1"},
            {"Name": "Group2", "Parameter": "Param2"},
            {"Name": "Group3", "Parameter": "Param3"},
        ]
    )
)

seems to work

        ENUM
        [
            Group1: Param1
            Group2: Param2
            Group3: Param3

        ] GROUP;

[rdan]

Thank you for taking the time to implement a solution.
So the goal: you are right, the need is to indent everything that will be written in the block relative to the indentation of the block itself (so of "<%").
So maybe "indent_current" could be renamed of "indented_relative_to_block", something like that.
Also, while searching for the indentation in the text already processed, not only spaces must be counted but any character must be counted.
For example

   some text <%
                for Group in ListOfGroups:
                   if ("Parameter" in Group):
                      context.write(f"{Group['Name']}: {Group['Parameter']}\n", indent_current=True)
             %>\

should create

   some text Group1: Param1
             Group2: Param2
             Group3: Param3

And if I do

   some text <%
                indent = ""
                for Group in ListOfGroups:
                   if ("Parameter" in Group):
                      context.write(f"{indent}{Group['Name']}: {Group['Parameter']}\n", indent_current=True)
                      indent += "   "
             %>\

it should create

   some text Group1: Param1
                Group2: Param2
                   Group3: Param3

Or if I do

   some text <%
                context.write(f"text 1\n", indent_current=True)
                context.write(f"   text 2\n", indent_current=True)
                context.write(f"text 3\n", indent_current=True)
             %>\

it should create

   some text text 1
                text 2
             text 3

Regarding the implementation of "_get_current_and_prior_indent(self)" (I don't know how mako works internally and what is stored in the different variables (at least not yet), so I may say something totally wrong):

• "if "\n" in self.data": shouldn't it be "if "\n" in elem" ?
• Isn't the code also taking into account any leading space added by a previous call to "context.write(..., indent_current=True)" in the same block ?

In fact, isn't it possible, when the block "<%...%>" starts to be processed to store in a variable the current indentation of the text processed so far (" " * number of trailing characters (whatever they are) until the first "\n" (if "\n" is the first trailing character, then no indentation)) to then automatically prefix the text provided to "context.write()" with the content of this variable if the parameter "indent_current" is set to True ?
It would also have a better performance by doing the analysis only once per block.

[zzzeek]

as far as the various scenarios, I'd want you to write some test cases for that. a good place to test this feature might be in test_runtime.py. for examples of tests that do "here's the template, compare it to this output", see the tests in https://github.com/sqlalchemy/mako/blob/main/test/test_template.py .

As far as the performance, yes, having the buffer track the current indent might make it readily available for this operation. However, tracking this indent would always (unless we did something really dramatic with how templates are compiled to python) need to be looking at strings being output, finding whitespace in them, and doing len() on it, anticipating that someone might come along and use this novel new feature that Mako has never had for 20 years and in all likelihood would be virtually never used outside of a few applications out of millions - that in itself would add a performance penalty to one of the highest velocity tasks Mako has, writing strings to the buffer. So adding latency to the FastEncodingBuffer 100% of the time for the purposes of this small feature would be a non-starter. There are perhaps other ways that either the template could detect the use of this function during compilation and signal to the buffer to track whitespace, or the buffer could track whitespace once the function is called the first time (that in itself though would add an extra boolean test to FastEncodingBuffer), or even an option that has to be passed to the Template; but these are all complicated approaches, and I think searching through the strings is still pretty fast as Python's native functions are quite fast by themselves.

[rdan]

Hello,
I would be happy to do that but I am lost in all the test files.
Do you have any documentation on how to create tests, how to run them... ?

For example, if I modify something in mako, how can I run all the tests to do a non regression test ?

Also, in "test_template.py", I see what seems to be different ways to do tests: _do_memory_test(), do_file_test(), sometimes there is assert() sometimes not, there is also sometimes assert_raises() associated with assert(), sometimes eq() is used also. This is confusing to me.
For sure, I can take time to try to figure out what this does and when to use which method, but it would be a great help if you could provide me some help on that.
Thank you
Ronan

[rdan]

As you can see from my previous message, I am not familiar with the standard ways to implement python libraries.
Chatgpt has been my "friend". So now I know that pytest is used to run the tests. So I will go from there to figure out how to implement the tests. If you have any recommendations, for sure I will take them.
Regarding the process I am using to update mako: I forked the repo, I will create a branch in my repo for each proposed updates and I will do a pull request from this branch on my repo when I think I am ready to show you the update. Is it how I am supposed to do it ?
Thank you
Ronan