Relax `<select>` parser #10557

josepharhar · 2024-08-13T22:37:01Z

This patch makes the parser allow additional tags in <select> besides <option>, <optgroup>, and <hr>, mostly by removing the "in select" and "in select in table" parser modes.

In order to replicate the behavior where opening a <select> tag within another open <select> tag inserts a </select> close tag, a traversal through the stack of open elements was added which I borrowed from the <button> part of the parser.

This patch also changes the processing model to make <select> look through all its descendants in the DOM tree for <option> elements, rather than just children and optgroup children which conform to the content model. This is needed for compat reasons because there are websites which put other tags in between their <select> and <option>s which would break with this parser change unless we also update this processing model. More context here and here.

Fixes #10310

At least two implementers are interested (and none opposed):
- Chromium
- …
Tests are written and can be reviewed and commented upon at:
- I still need to implement html5lib test changes
Implementation bugs are filed:
- Chromium: Already implemented
- Gecko: …
- WebKit: …
MDN issue is filed: …
The top of this comment includes a clear commit message to use.

(See WHATWG Working Mode: Changes for more details.)

/form-elements.html ( diff )
/index.html ( diff )
/infrastructure.html ( diff )
/parsing.html ( diff )

annevk · 2024-08-14T11:39:54Z

Why does this also make a bunch of changes to the processing model of select elements? If that needs to be part of this change that should really be better motivated in the commit message.

josepharhar · 2024-08-16T00:08:20Z

Why does this also make a bunch of changes to the processing model of select elements? If that needs to be part of this change that should really be better motivated in the commit message.

The changes in the processing model do support the new proposed content model for select, but they also mitigate compat issues for websites which are already putting tags in between <select> and <option> in their HTML.

For example, without these changes to the processing model, the following <select> would not register as having any options at all:

<select>
  <div>
    <option>...</option>
    ...
  </div>
</select>

In my compat analysis, I found a lot of websites which are doing this, so in order to ship the parser changes separately from customizable select in chrome, I plan to ship the parser changes and these processing model changes together, because otherwise there would be too much breakage.

I'm happy to put them in a separate PR if you want, or keep them here and update the commit message (sorry for not putting it in there). Which would you prefer?

annevk · 2024-08-16T06:11:37Z

Thanks! Given that rationale I think it's good to couple the changes, but that should be in the commit message as well.

zcorpan · 2024-08-19T15:59:08Z

This doesn't define optional tags for </option> and </optgroup> correctly.

The definition for "have a particular element in select scope" may be needed for that, but should be changed to be similar to "have a particular element in button scope" (but for select).

In particular, allow these without a parse error:

<select>
<optgroup>
<option>
<optgroup>
</select>

The second <optgroup> should pop the option and optgroup

<select>
<option><p>
<option>
</select>

This should generate implied end tags and pop the option.

<select>
<optgroup>
<hr>
<option>
<hr>
</select>

The hrs should pop the optgroup and option in a select.

See how the parser deals with <ruby><rtc><rt>, I think that can be used as a model for select.

source

josepharhar · 2024-08-19T21:28:14Z

Thanks! Given that rationale I think it's good to couple the changes, but that should be in the commit message as well.

Done.

This doesn't define optional tags for </option> and </optgroup> correctly.

@zcorpan thanks for the feedback! I did some experimenting and added some logic to the start tags for option, optgroup, and hr. What do you think?

zcorpan · 2024-08-22T09:02:46Z

Need to check that a select element is in scope so that parsing of option/optgroup tags outside of select doesn't change. Example <option><p>1</option>2<option>3
Need to add select to the "in scope" list so that <div><select></div><option> doesn't close the select.
Should report parse errors when unexpected elements are popped. Compare with what the spec does for rt/rtc in ruby. Example <select><option><span>1</option>2<option>3

josepharhar · 2024-08-22T23:57:54Z

Need to check that a select element is in scope so that parsing of option/optgroup tags outside of select doesn't change. Example <option><p>1</option>2<option>3

I thought that this is covered by "While the stack of open elements has an option element in select scope". What exactly should I change?

Need to add select to the "in scope" list so that <div><select></div><option> doesn't close the select.

Done

Should report parse errors when unexpected elements are popped. Compare with what the spec does for rt/rtc in ruby. Example <select><option><span>1</option>2<option>3

I'm guessing this is from "If the current node is not now a rtc element or a ruby element, this is a parse error," right?

Should I add "If the current node is not now an option element, this is a parse error" after "While the stack of open elements has an option element in select scope, pop an element from the stack of open elements"?

zcorpan · 2024-08-23T11:58:59Z

The "in select scope" I think should be removed altogether since it assumes the stack will not have other elements when in a select, which is no longer the case. Use the normal "in scope" instead.

High-level of what I think should happen: when parsing option or optgroup start tag: check that select is in scope, check that option or optgroup is in scope, generate implied end tags (except for optgroup when handling <option>), check the current node or the stack of open elements again, report a parse error if appropriate, then pop elements off the stack until the option or optgroup has been popped, then insert the new element.

I can look into this more next week and suggest more specific changes.

josepharhar · 2024-08-28T00:26:31Z

High-level of what I think should happen: when parsing option or optgroup start tag: check that select is in scope, check that option or optgroup is in scope, generate implied end tags (except for optgroup when handling <option>), check the current node or the stack of open elements again, report a parse error if appropriate, then pop elements off the stack until the option or optgroup has been popped, then insert the new element.

Thanks! I gave this a try

josepharhar · 2024-09-04T20:34:56Z

@zcorpan how does the latest text look?

zcorpan

I think the option/optgroup cases are right after these changes.

source

zcorpan · 2024-09-10T09:58:49Z

source

+    <ol>
+     <li><p><span>Generate implied end tags</span> except for <code>option</code> and
+     <code>optgroup</code> elements.</p></li>
+
+     <li><p>If the <span>current node</span> is not now an <code>option</code> or
+     <code>optgroup</code> element, this is a <span>parse error</span>.</p></li>
+
+     <li><p>While the <span>stack of open elements</span> <span data-x="has an element in scope">has
+     an <code>option</code> element in scope</span> or <span data-x="has an element in scope">has an
+     <code>optgroup</code> element in scope</span>, pop an element from the <span>stack of open
+     elements</span>.</p></li>
+    </ol>


This should report a parse error if there's an element between the optgroup and option in the stack:

<select> <optgroup> <div> <option> <optgroup>

This can be done by popping the option element if that is the current node (after generating implied end tags except for option and optgroup), then generate implied end tags except for optgroup elements, parse error if the current node is not an optgroup element.

done. how does it look?

source

josepharhar · 2024-09-10T22:34:51Z

This doesn't define optional tags for </option> and </optgroup> correctly.

I was talking to @mfreed7 about the changes we've made in the PR so far, and I feel like I couldn't provide a good explanation for why we are making the parser not support cases like these:

<hr> inside an <option>
nested <optgroup>s

Is it just compat reasons? Is there a good justification?

This patch makes the parser allow additional tags in <select> besides <option>, <optgroup>, and <hr>, mostly by removing the "in select" and "in select in table" parser modes. In order to replicate the behavior where opening a <select> tag within another open <select> tag inserts a </select> close tag, a traversal through the stack of open elements was added which I borrowed from the <button> part of the parser. This will need test changes to be implemented in html5lib. Fixes whatwg#10310

zcorpan · 2024-09-11T08:53:59Z

It would be a breaking change from what is conforming HTML today, and break compat for sites that omit </option> and </optgroup> tags. It's the same as why we can't allow certain elements in p.

mfreed7 · 2024-09-11T21:19:37Z

It would be a breaking change from what is conforming HTML today

But that's true of the rest of this PR also, correct? E.g. making <input> no longer close the <select>. We are just thinking that the "new" behavior we're headed towards with appearance:base is that roughly "all" content is now valid within <select>. It feels very odd to instead say "all content is valid, except for nested <optgroup> and <hr> within an <option>". Neither of those sound like they come from anything other than a place of web compat grandfathering. So I guess our hope would be that we can try to remove those two restrictions along with the rest, and see what actually falls out as a real web compat issue? Unless there's some deeper semantic problem with nesting optgroups? Or including horizontal lines within an option?

zcorpan · 2024-09-12T07:24:41Z

<input> is not allowed in select today, so it's not a change in behavior for conforming HTML. I think there's an implied contract of not breaking content that follows HTML's authoring requirements.

hr in select is pretty new, so we could probably make it not imply </option> without affecting web compat significantly. A tradeoff is for authors who want to omit optional tags can't omit </option> only when the next child is hr. Is that more annoying than not being able to use hr in option? I suspect no.

I expect we can't allow nesting of optgroup, but I haven't researched it. If that is allowed, it would be reasonable to expect that the native dropdown can support nested optgroups. I don't know if that is possible on the relevant platforms.

zcorpan · 2024-09-12T07:29:33Z

It would be a breaking change from what is conforming HTML today

To clarify, I meant: It would be a breaking change for existing conforming HTML content.

scottaohara · 2024-09-12T16:34:51Z

fwiw, i can't think of a good reason for an author to even need an hr inside of an option.

if someone wants to create a visual separation between the text within an option - then that sounds like something that could/should be accomplished with CSS. The semantics/a11y mappings of a separator dont' really make sense in the context of an option

josepharhar · 2024-09-12T17:51:13Z

I expect we can't allow nesting of optgroup, but I haven't researched it. If that is allowed, it would be reasonable to expect that the native dropdown can support nested optgroups. I don't know if that is possible on the relevant platforms.

Yeah that's fair, UIs won't support nested optgroups, at least not yet. There is a separate issue to track this here: #5789

I suppose that if we want to support nested optgroups then we could revisit this, and disallow nested optgroups in the parser for now.

<input> is not allowed in select today, so it's not a change in behavior for conforming HTML. I think there's an implied contract of not breaking content that follows HTML's authoring requirements.

hr in select is pretty new, so we could probably make it not imply </option> without affecting web compat significantly. A tradeoff is for authors who want to omit optional tags can't omit </option> only when the next child is hr. Is that more annoying than not being able to use hr in option? I suspect no.

fwiw, i can't think of a good reason for an author to even need an hr inside of an option.

if someone wants to create a visual separation between the text within an option - then that sounds like something that could/should be accomplished with CSS. The semantics/a11y mappings of a separator dont' really make sense in the context of an option

We can forbid hr in select in the content model and still allow it to be parsed, just like we are doing with many other tags in this change. I suspect that hr in option would be a significantly less breaking change than other things.

I see the conforming HTML argument though and I don't feel that strongly about this. Mason's point is that we should only create special cases like this if we have a really good reason. If not changing conforming HTML is that important, then I think its probably fine since hr in select is not an important use case.

zcorpan · 2024-09-16T15:24:08Z

hr in select was added just last year, so we shouldn't flip-flop on that. But it could be allowed only as a sibling to an option element in the content model, if we're ok with hr no longer implicitly closing option elements.

zcorpan · 2024-09-16T15:38:18Z

Since the content model in #10586 only allows div and span, maybe there's less need to "aggressively" close elements when seeing option or optgroup. Is only div and span the end goal, or do we eventually want to allow other HTML elements?

If, for web compat, it's ok to only close option if it's the current node when seeing "option" or "optgroup" (maybe also "hr"), that would make the parser change much simpler. It would also allow for a possible future of allowing nested optgroups (would need a div in between).

josepharhar · 2024-09-16T16:50:27Z

hr in select was added just last year, so we shouldn't flip-flop on that. But it could be allowed only as a sibling to an option element in the content model, if we're ok with hr no longer implicitly closing option elements.

The content model PR currently doesn't list hr elements in the content model for options

Is only div and span the end goal, or do we eventually want to allow other HTML elements?

Eventually allowing other HTML elements is ideal as we add support for additional use cases and find ways to make them accessible.

If, for web compat, it's ok to only close option if it's the current node when seeing "option" or "optgroup" (maybe also "hr"), that would make the parser change much simpler. It would also allow for a possible future of allowing nested optgroups (would need a div in between).

As Mason said, ideally we try removing restrictions and see what real issues arise. Would I implement this by removing the "generate implied end tags" line?

zcorpan · 2024-09-16T20:18:37Z

If we envision allowing more HTML elements later, then we should at least generate implied end tags, so that HTML authoring rules are somewhat consistent. For example, it should be possible to omit </p> in select.

josepharhar · 2024-09-17T23:21:00Z

If we envision allowing more HTML elements later, then we should at least generate implied end tags, so that HTML authoring rules are somewhat consistent. For example, it should be possible to omit </p> in select.

Ah I see. So if I leave that in and then remove the "While the stack of open elements has an option element in scope, pop an element" part and the equivalent one for optgroup then we should be good?

zcorpan · 2024-09-18T16:18:48Z

No, still need to pop the option or optgroup if that is the current node. Actually, using the ruby model is maybe best, however the parse error condition needs to be different (since we want to allow <select><div><option><option>, the parent is not a select here).

Something like this:

A start tag whose tag name is "optgroup"

If the stack of open elements has a select element in scope, then generate implied end tags. If the stack of open elements now has an option or optgroup element in scope, then this is a parse error.

Insert an HTML element for the token.

A start tag whose tag name is "option"

If the stack of open elements has a select element in scope, then generate implied end tags, except for optgroup elements. If the stack of open elements now has an option element in scope, then this is a parse error.

Insert an HTML element for the token.

This also removes "Reconstruct the active formatting elements , if any.", so that that happens inside the option element instead when seeing a token that reconstructs active formatting elements.

josepharhar · 2024-09-18T17:00:47Z

Thanks! I implemented your suggestion

josepharhar · 2024-09-18T18:38:43Z

Generating all implied end tags with no exceptions in the optgroup case disallows nested optgroups, but I guess I'm ok with that since we can make additional changes to allow it later.

I'll apply the changes in here so far to the chromium implementation and start working on html5lib tests so we can get this merged.

zcorpan · 2024-09-18T19:33:56Z

Great.

hr no longer closing option and optgroup is a behavior change from what the parser currently does, which I'm somewhat uncomfortable with. It makes currently valid HTML invalid. @hsivonen @annevk WDYT?

annevk · 2024-09-19T10:36:22Z

I'm not a big fan of breaking conforming HTML. Especially as there doesn't appear to be a compelling reason from reading the thread.

I recommend not using "grandfathering" by the way, it has a rather bad history: https://en.wikipedia.org/wiki/Grandfather_clause

josepharhar marked this pull request as ready for review August 13, 2024 22:39

josepharhar mentioned this pull request Aug 13, 2024

HTML parser changes for stylable <select> #10310

Open

annevk requested review from zcorpan and hsivonen August 14, 2024 11:36

annevk added normative change topic: parser labels Aug 14, 2024

dmsnell mentioned this pull request Aug 16, 2024

HTML API: Roadmap WordPress/gutenberg#60397

Open

10 tasks

zcorpan requested changes Aug 19, 2024

View reviewed changes

source Outdated Show resolved Hide resolved

source Outdated Show resolved Hide resolved

source Outdated Show resolved Hide resolved

source Outdated Show resolved Hide resolved

josepharhar force-pushed the selectparser branch from cf37251 to afb732d Compare August 19, 2024 21:28

josepharhar force-pushed the selectparser branch from afb732d to e08df7e Compare August 21, 2024 20:21

josepharhar force-pushed the selectparser branch from ea6d608 to 21bc5e6 Compare August 23, 2024 00:01

josepharhar force-pushed the selectparser branch from 21bc5e6 to a57c84e Compare August 28, 2024 00:26

zcorpan requested changes Sep 5, 2024

View reviewed changes

source Outdated Show resolved Hide resolved

source Outdated Show resolved Hide resolved

josepharhar force-pushed the selectparser branch from a57c84e to a1edb00 Compare September 5, 2024 20:52

zcorpan reviewed Sep 10, 2024

View reviewed changes

josepharhar force-pushed the selectparser branch from a1edb00 to 9507110 Compare September 10, 2024 20:20

josepharhar force-pushed the selectparser branch from 9507110 to 1330ad5 Compare September 10, 2024 22:35

make rules more permissive

b87d93d

josepharhar mentioned this pull request Sep 18, 2024

Customizable <select> element #9799

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Relax `<select>` parser #10557

Relax `<select>` parser #10557

josepharhar commented Aug 13, 2024 •

edited by pr-preview bot

Loading

annevk commented Aug 14, 2024

josepharhar commented Aug 16, 2024

annevk commented Aug 16, 2024

zcorpan commented Aug 19, 2024

josepharhar commented Aug 19, 2024

zcorpan commented Aug 22, 2024

josepharhar commented Aug 22, 2024

zcorpan commented Aug 23, 2024

josepharhar commented Aug 28, 2024

josepharhar commented Sep 4, 2024

zcorpan left a comment

zcorpan Sep 10, 2024

josepharhar Sep 10, 2024

josepharhar commented Sep 10, 2024

zcorpan commented Sep 11, 2024

mfreed7 commented Sep 11, 2024

zcorpan commented Sep 12, 2024

zcorpan commented Sep 12, 2024

scottaohara commented Sep 12, 2024

josepharhar commented Sep 12, 2024

zcorpan commented Sep 16, 2024

zcorpan commented Sep 16, 2024

josepharhar commented Sep 16, 2024

zcorpan commented Sep 16, 2024

josepharhar commented Sep 17, 2024

zcorpan commented Sep 18, 2024 •

edited

Loading

josepharhar commented Sep 18, 2024

josepharhar commented Sep 18, 2024

zcorpan commented Sep 18, 2024

annevk commented Sep 19, 2024

Relax <select> parser #10557

Are you sure you want to change the base?

Relax <select> parser #10557

Conversation

josepharhar commented Aug 13, 2024 • edited by pr-preview bot Loading

annevk commented Aug 14, 2024

josepharhar commented Aug 16, 2024

annevk commented Aug 16, 2024

zcorpan commented Aug 19, 2024

josepharhar commented Aug 19, 2024

zcorpan commented Aug 22, 2024

josepharhar commented Aug 22, 2024

zcorpan commented Aug 23, 2024

josepharhar commented Aug 28, 2024

josepharhar commented Sep 4, 2024

zcorpan left a comment

Choose a reason for hiding this comment

zcorpan Sep 10, 2024

Choose a reason for hiding this comment

josepharhar Sep 10, 2024

Choose a reason for hiding this comment

josepharhar commented Sep 10, 2024

zcorpan commented Sep 11, 2024

mfreed7 commented Sep 11, 2024

zcorpan commented Sep 12, 2024

zcorpan commented Sep 12, 2024

scottaohara commented Sep 12, 2024

josepharhar commented Sep 12, 2024

zcorpan commented Sep 16, 2024

zcorpan commented Sep 16, 2024

josepharhar commented Sep 16, 2024

zcorpan commented Sep 16, 2024

josepharhar commented Sep 17, 2024

zcorpan commented Sep 18, 2024 • edited Loading

josepharhar commented Sep 18, 2024

josepharhar commented Sep 18, 2024

zcorpan commented Sep 18, 2024

annevk commented Sep 19, 2024

Relax `<select>` parser #10557

Relax `<select>` parser #10557

josepharhar commented Aug 13, 2024 •

edited by pr-preview bot

Loading

zcorpan commented Sep 18, 2024 •

edited

Loading