How are "failing" test folders used/managed?

[yihtserns]

Right now tests in failing folders are not executed when running mvn test, so how would you know when they're no longer failing?

Do the maintainers periodically check them? Or you search for the issue number for any failing test when working on the issue? What's the process/workflow involving that folder?

---

For example, some test methods are passing, are the maintainers aware of this but leaving it in failing folder since they're not "officially"/"properly" supported so are considered passing by chance/coincidence/accidentally:

[pjfanning]

Right now tests in failing folders are not executed when running mvn test, so how would you know when they're no longer failing?

Do the maintainers periodically check them? What's the process/workflow involving that folder?

You can run them in an IDE.

[pjfanning]

adding #4645

[pjfanning]

I think #4646 now covers all 4 working tests

Thanks @yihtserns for highlighting this

[yihtserns]

OK while @cowtowncoder clarified that those above shouldn't be done (@pjfanning sorry about that, and thanks for the effort...), I found one that is definitely good to go - this disabled test is passing:

jackson-databind/src/test/java/com/fasterxml/jackson/databind/deser/creators/TestCreators.java

Lines 423 to 434 in aafba87

	// 18-May-2024, tatu: Need to disable for now wrt [databind#4515]:
	// handling seems inconsistent wrt Constructor/Factory precedence,
	// will tackle at a later point -- this is the last JDK8 fail
	@Test
	@Disabled
	public void testConstructorAndFactoryCreator() throws Exception
	{
	CreatorBeanWithBoth bean = MAPPER.readValue
	("{ \"a\" : \"xyz\", \"x\" : 12 }", CreatorBeanWithBoth.class);
	assertEquals(13, bean.x);
	assertEquals("ctor:xyz", bean.a);
	}

[cowtowncoder]

Ahh. Yes, those should definitely be changed, I'll find and change it.

And as to other split cases: I am not dead set against inclusion of some of passing cases. But should be done on case-by-case basis.

[cowtowncoder]

Ok there's just one such test. Thanks @yihtserns !

[cowtowncoder]

Ok, so as for me: I ran these every now and then, to see if some tests were fixed (once every 2 weeks perhaps); plus whenever working in area (or merging PRs in area) where I recall related failing tests.

And yes: if full test classes start passing, they should be moved to their regular place so ../failing/ should only contain test classes with at least 1 failure.
NOTE: it's perfectly fine for some test methods in classes to pass; these are usually to give contrast/context to failures (this case passes while that doesn't). This is not accidental.
But if and how to include passing cases as part of failing tests is judgment call.

[cowtowncoder]

@pjfanning @yihtserns There is a misunderstanding here, indeed -- those passing cases can remain there. The only hard rule is if the whole test class passes (all methods/cases pass), then class should be moved out (and possibly merged).
There is no requirement that none of the cases pass.

[yihtserns]

The reason I'm asking this is because I wanted to come up with a mechanism that I thought would make managing failing test cases more "automated" and needed less "human touch". But from the replies above, seems like the current process/workflow can't be simplistically automated away.

Since I've already spent time on it, I decided to just share what I've created: #4648 - maybe it can be a seed that leads to a more fit-for-purpose idea in the future

Thanks for taking the time to answer my questions @pjfanning @cowtowncoder. 🤝