Revise Unevaluated[]

[rocky]

Fixes #122

[mmatera]

@rocky, it is good to see you have start to looking at this long-standing issue.
I cannot say how is Unevaluated implemented in WMA, beyound what I can infer
from the behavior.

To see what happend, let's consider some basic examples.
First, let's introduce some definitions:

In[1]:= F[x_Integer, y_Integer, z_Integer]:={x,y,z}
In[2]:= F[x_Sin, y_Integer, z_Integer]:={"sin", x,y,z}
In[3]:= x=4                                                                     
Out[3]= 4

Now, let's compare the output of some evaluations:

In[5]:= F[Sin[Pi],2,x^2]
Out[5]= {0, 2, 16}

In[5]:= F[Unevaluated[Sin[Pi]],2,x^2]
Out[5]= {sin, 0, 2, 16}

In[6]:= F[Unevaluated[Cos[Pi]],2,x^2]
Out[7]= F[Unevaluated[Cos[Pi]], 2, 16]

In the first case, the arguments of F are evaluated, and since Sin[Pi] evaluates to 0, it matches the first rule. In the second case, Unevaluated avoids
the evaluation and, when rules are applied, Unevaluated is removed. Then,
F[Sin[Pi],2, x^2] matches with the second rule, leading to
{"sin", Sin[Pi], 2, 162}. Then, Sin[Pi] is evaluated in a further round, to get {"sin", 0, 2, 162}.

In the third case, F[Cos[Pi], 2, 16] does not match any rule, and we end with
F[Unevaluated[Cos[Pi]], 2, 16], with the other elements evaluated.

I do not think that this behavior can be implemented by associating rules to the Unevaluted symbol, because it happends at the level of the evaluation of the container expression.

Also, the behavior of Unevaluated is not shown at the level of applying rules:

In[8]:= G[Unevaluated[Q[x]],y]/.{G[a_Q, b_v]->{a,b}}
Out[8]= G[Unevaluated[Q[x]],y]

Now, let's consider more sophisticated examples, which happens on expressions involving heads with Flat or Orderless attributes.

In[9]:= Attributes[F]={Orderless}
Out[9]= {Orderless}

In[10]:= F[Sin[x], Cos[x], T[y], Unevaluated[Sin[x]], Zeta[z]]
Out[10]= F[Cos[4], Sin[4], Unevaluated[Sin[x]], T[y], Zeta[z]]

Notice that the expression is ordered neglecting the Unevaluated wrapper, but this just happend
up to the first level:

In[11]:=  F[Sin[x], Cos[x], T[y], Unevaluated[Sin[x]], Unevaluated[Unevaluated[Sin[x]]], Zeta[z]]                                                                             
Out[11]= F[Cos[4], Sin[4], Unevaluated[Sin[x]], T[y], Unevaluated[Unevaluated[Sin[x]]], Zeta[z]]

so this not seems to be implemented at the level of an special sort_key.

Now, let's see what happens with the Flat attribute:

In[12]:= Attributes[F]={Flat}
Out[12]= {Flat}

In[13]:=  F[Unevaluated[F[Sin[x]]], 2, x^2]                                                              
Out[13]= {sin, Sin[4], 2, 16}


In[14]:=  F[Unevaluated[F[Sin[x],2,x^2]], 2, x^2]
Out[14]= F[Unevaluated[Sin[x]], Unevaluated[2], Unevaluated[x^2], 2, 16]

In the first case, looking at the TracePrint output, what happens is that
Unevaluated is removed before flatten the expression:

F[Unevaluated[F[Sin[x]]], 2, x^2]-> F[Unevaluated[F[Sin[x]]], 2, 16] ->
F[F[Sin[x]], 2, 16] -> F[Sin[x], 2, 16] -> {"sin", Sin[x], 2, 16} -> {"sin", Sin[4], 2, 16}

In the second case, Unevaluated is broadcasted to the elements of the inner expression, before flatten, to an expression that can not be evaluated (because has a signature incompatible with the defined rules).

Notice also what happens if the expression could match with the signature:

In[15]:= F[Unevaluated[F[Sin[x],2,x^2]]]
Out[15]= F[Unevaluated[Sin[x]], Unevaluated[2], Unevaluated[x^2]]

In[16]:= F[Unevaluated[F[Sin[x],2,16]]]
Out[16]= {sin, Sin[4], 2, 16}

In the first example, as before, the flat expression does not match, so the elements receives the Unevaluated
wrapper. But in the last one, since the expression matches, the rule is applied and Unevaluated is gone.

Finally, this is what happens if we combine both attributes:

In[17]:= Attributes[F]={Orderless, Flat};

In[18]:= F[Unevaluated[F[Sin[0],2,x^2]],q]
Out[18]=  F[Unevaluated[2], q, Unevaluated[x^2], Unevaluated[Sin[0]]]

The trickiest part of this is is that Unevaluated is removed before sorting and the apply-rulues-step,
and restored afterward. In Mathics, this is done by tagging the elements with the attribute unevaluated.

If the sort step wasn't there, one way to achieve the same would be to store the position of the Unevaluated
elements. But sort mix them. Other possibility would be to work with a more complex sort key function, like

sort(elements, key=lambda x: (x.elements[0].sort_key() if x.hasform("Unevaluated",1) else x.sort_key()))

and then build the list of positions. But I leave the implementation to you.

[rocky]

@rocky, it is good to see you have start to looking at this long-standing issue. I cannot say how is Unevaluated implemented in WMA, beyound what I can infer from the behavior.

To see what happend, let's consider some basic examples. First, let's introduce some definitions:

In[1]:= F[x_Integer, y_Integer, z_Integer]:={x,y,z}
In[2]:= F[x_Sin, y_Integer, z_Integer]:={"sin", x,y,z}
In[3]:= x=4                                                                     
Out[3]= 4

Now, let's compare the output of some evaluations:

....

This is a very detailed examination of Unevaluated, but right now, I am interested in the way it is currently used in programs and the documented behavior. That is, real examples. The basic examples as given do work in this branch. And we could get closer in for those situations that really come up.

Take for example, the sort order being different when patterns contain Unevaluated. Is that important or just the way things turn out? Maybe the order change is done on output formatting?

I'd like to understand, for example, if there is any WMA code that cares about the sort order when Unevaluated is used going into the evaluation phase (as opposed to the Boxing/Format phase.

And if so, why? I am interested in insight into the conditions and purpose a person would use Unevaluated, as opposed to minutiae into how it currently works.

I suspect that some of the behavior is an artifact of the way the code happened to be implemented, rather than there being a need for it by any programmer or WMA program. And if that's the case, the behavior could change because the current undocumented behavior of Unevaluated isn't even important to WMA designers and maintainers.

If that is the case, then it is probably not a good idea to spend time right now matching somewhat arbitrary behavior and complicating code.

What I want to address right now is the intent for the need of this function. Down the line, it is possible or likely that our evaluation algorithm will happen to be more like Mathematica's based on information we glean later.

[mmatera]

Maybe the way to answer that question is to look at packages like Feyncalc, Rubi or even Combinatorica. In the code I have write in WMA for my day to day work, I have never used it. On the other hand, having this in mind could be useful to understand how the evaluation process goes in WMA.

[mmatera]

I asked ChatGPT

Example Context	Source
Flattening suppressed via Unevaluated	“The Flat Attribute, Unevaluated and the Evaluation Process” (Mathematica.SE) (Mathematica Stack Exchange)
Labeling of unevaluated arguments under Flat/Orderless	MathGroup forum archive (forums.wolfram.com)
Explicit statement: Unevaluated stops Flat/Orderless	Verbeia.com guide (verbeia.com)
Order of evaluation: Unevaluated before Flat and Orderless	Wolfram Language Evaluation documentation (reference.wolfram.com)

Example Context	Source
Flattening suppressed via Unevaluated	“The Flat Attribute, Unevaluated and the Evaluation Process” (Mathematica.SE) ([Mathematica Stack Exchange]1)
Labeling of unevaluated arguments under Flat/Orderless	MathGroup forum archive ([forums.wolfram.com]2)
Explicit statement: Unevaluated stops Flat/Orderless	Verbeia.com guide ([verbeia.com]3)
Order of evaluation: Unevaluated before Flat and Orderless	Wolfram Language Evaluation documentation ([reference.wolfram.com]4)

But to find better examples, I would need to do more research...

[rocky]

I asked ChatGPT

...
Example Context Source
Flattening suppressed via Unevaluated “The Flat Attribute, Unevaluated and the Evaluation Process” (Mathematica.SE) ([Mathematica Stack Exchange]1)

This contains a reference to and out-of-print book "Power Programming with the Mathematica Kernel" for Mathematica version 3. I downloaded a PDF of that.

[rocky]

Maybe the way to answer that question is to look at packages like Feyncalc, Rubi or even Combinatorica. In the code I have write in WMA for my day to day work, I have never used it. On the other hand, having this in mind could be useful to understand how the evaluation process goes in WMA.

I've now looked over the ChatGPT references, and have git grep'd WL packages I have.

Most of the discussion is a while ago, from Mathematica version 3.0 to 5.0 days. The McGraw-Hill Wagner book mentions this, and perhaps this sparked an interest. Like you, his interest wasn't driven by any need, but, rather, as a means of understanding how evaluation works.

I suspect there is a bit more work we can do to make the way we do evaluation more in line with what's described there, and in the future, I'll try to align Trace better (with options for extended information) and tweak evaluation so we match WMA behavior more closely.

As for the use that we see in Mathematica packages, it doesn't appear that often. And the uses are not that sophisticated. Especially the ones in Feyncalc, where most of what we encounter is Unevaluated[Sequence[]]

In KnotTheory it appears mostly in a Dbg[] calls like

Dbg[Unevaluated["Submatrix ",ReplaceAll[output,knotComponent->"T"]//MatrixForm]];

The only place it appears in Rubi is:

FixRhsIntRule[u_,x_] :=
  If[MemberQ[{Int, Unintegrable, CannotIntegrate, Subst, Simp}, Head[Unevaluated[u]]],
    u,
  Simp[u,x]]

It does, however, appear more often in WolframEngine System code 13.2 with more sophistication. It appears about 800 times.

[mmatera]

In the line of looking at the way in which WMA evaluated expressions with Unevaluated, I was thinking in this example:

In[1]:= Unprotect[Unevaluated]
Out[1]= {Unevaluated}

In[2]:= HoldPattern[Unevaluated[x_T]]:=1 (*Overwrites the rule for `Unevaluated`*)

In[3]:= Attributes[F]={HoldFirst}                  (*Just to see what happens together with the attributes`*)
Out[3]= {HoldFirst}

In[4]:= F[x_,y_]:={x,y}                                      (*Silly rule for `F`*) 

In[5]:= F[Unevaluated[T[a]],Unevaluated[T[b]]]                     (*The rule for `Unevaluated[T[b]]` is not applyed in any case*)
Out[5]= {T[a], T[b]}

In[6]:= G[Unevaluated[T[a]]]                                                      (*The same happens if `G` does not have a rule*)
Out[6]= G[Unevaluated[T[a]]]

In[7]:= Unevaluated[T[a]]                                                            (*In this case, the rule for Unevaluated is applyied*)
Out[7]= 1

From this, I understand that the approach in this PR is incorrect: if it were right, we could overwrite the rule.

[rocky]

In the line of looking at the way in which WMA evaluated expressions with Unevaluated, I was thinking in this example:

In[1]:= Unprotect[Unevaluated]
Out[1]= {Unevaluated}

In[2]:= HoldPattern[Unevaluated[x_T]]:=1 (*Overwrites the rule for `Unevaluated`*)

In[3]:= Attributes[F]={HoldFirst}                  (*Just to see what happens together with the attributes`*)
Out[3]= {HoldFirst}

In[4]:= F[x_,y_]:={x,y}                                      (*Silly rule for `F`*) 

In[5]:= F[Unevaluated[T[a]],Unevaluated[T[b]]]                     (*The rule for `Unevaluated[T[b]]` is not applyed in any case*)
Out[5]= {T[a], T[b]}

In[6]:= G[Unevaluated[T[a]]]                                                      (*The same happens if `G` does not have a rule*)
Out[6]= G[Unevaluated[T[a]]]

In[7]:= Unevaluated[T[a]]                                                            (*In this case, the rule for Unevaluated is applyied*)
Out[7]= 1

From this, I understand that the approach in this PR is incorrect: if it were right, we could overwrite the rule.

I understood that the PR is incorrect based on the fact that there were tests that were failing, and the descriptions of how Unevaluated is supposed to work are different from what is done in this draft.

I don't think we should try to get this working before release. It is a bit too involved.

Here, though, are things I do understand:

• We shouldn't be putting any properties like "unevaluated" on any Atom for this
• The descriptions indicate changes to the Unevaluated function arguments attributes are made "temporarily" when changes are made. The statement about Evaluation[Unevaluation] suggests that, in this case, no temporary change is made?

From an engineering perspective, ideal would the temporary setting and restoring of HOLD attributes could be done totally inside the Unevaluated method instead of the rewrite_apply_eval_next() which is part of the function application step.

[mmatera]

I understood that this is not something to put into the next release. I just added this comment as information for thinking about the right implementation, which I agree should not involve setting temporary attributes to the expression.