Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IntelliJ outputs error messages twice #680

Open
Skynet0 opened this issue Apr 19, 2020 · 13 comments
Open

IntelliJ outputs error messages twice #680

Skynet0 opened this issue Apr 19, 2020 · 13 comments
Labels
P3 not scheduled type=enhancement Make an existing feature better

Comments

@Skynet0
Copy link

Skynet0 commented Apr 19, 2020

I am working on IntelliJ 2019.3, with Truth 1.0 and Java 11.0.6, and running tests using Junit 5.6.0.

I'm trying to do test some basic assertions, such as checking the value of an int. However, IntelliJ is printing the messages twice:

@Test
@Tag("D")
public void testDuplicateDemo() {
    int x = 5;
    assertThat(x).isEqualTo(10);
}

Outputs:

Annotation 2020-04-19 010740

This is rather annoying and unexpected behavior. If I test equality of Strings, the same behavior is shown, unless the string has multiple lines, in which case, only expected / but was appears.

Why is this happening? Is there a workaround / fix? Thanks!

@Skynet0
Copy link
Author

Skynet0 commented Apr 19, 2020

Some more tests show me that this is something internal to how IntelliJ reports tests errors (considering that AssertJ behaves the same way). Going to close this and just use JUnit for now, which doesn't double up.

@Skynet0 Skynet0 closed this as completed Apr 19, 2020
@cpovirk
Copy link
Member

cpovirk commented Apr 20, 2020

Thanks. I suspect that the cause is that we throw a ComparisonFailure. So IntelliJ outputs the Truth-generated failure message (the first 2 lines) but then also generates its own message by calling the getters on the ComparisonFailure.

I wouldn't be surprised if IntelliJ does the same for JUnit -- but only when the inputs are strings, since that's the only case in which JUnit creates a ComparisonFailure.

I'm going to reopen this for us to consider doing something better. That might mean talking to the IntelliJ people to ask them for a way to suppress the display of their "Expected" and "Actual" lines, or it might mean something else.

@cpovirk cpovirk reopened this Apr 20, 2020
@cpovirk cpovirk added P3 not scheduled type=enhancement Make an existing feature better labels Apr 20, 2020
@Skynet0
Copy link
Author

Skynet0 commented Apr 20, 2020

For completeness, here's the behavior of Junit5, which does not raise ComparisonFailure:

@Test
@Tag("demo")
public void testDuplicateDemoIntJUnit() {
    int x = 5;
    org.junit.jupiter.api.Assertions.assertEquals(10, x);
}
org.opentest4j.AssertionFailedError: 
Expected :10
Actual   :5
<Click to see difference>
@Test
@Tag("demo")
public void testDuplicateDemoStringJUnit() {
    String x = "foo";
    org.junit.jupiter.api.Assertions.assertEquals("bar", x);
}
org.opentest4j.AssertionFailedError: 
Expected :bar
Actual   :foo
<Click to see difference>

It does not have the double output issue, even on Strings. Hamcrest behaves similarly to JUnit.

@cpovirk
Copy link
Member

cpovirk commented Apr 20, 2020

Thanks! The different behavior might depend not only on the type of exception thrown (ComparisonFailure vs. AssertionError vs. AssertionFailedError) but also on the version of JUnit used to write/run the tests.

It's possible that a straight migration to the opentest4j exceptions would make things worse in some environments. Maybe we should first check with IntelliJ developers (and other tool owners) to ensure that they handle the opentest4j exceptions even under older versions of JUnit.

@Skynet0
Copy link
Author

Skynet0 commented Apr 20, 2020

JUnit 4.12 behavior: no double output, even when a ComparisonFailure is raised:

@Test
public void testJunit4InJUnit4String() {
    System.out.println("JUnit version: " + junit.runner.Version.id());
    String x = "foo";
    assertEquals("bar", x);
}
JUnit version: 4.12

org.junit.ComparisonFailure: 
Expected :bar
Actual   :foo
<Click to see difference>

Truth has the same double output issue in JUnit 4.12.

@cpovirk
Copy link
Member

cpovirk commented Apr 20, 2020

Huh, thanks. I think I also saw something in IntelliJ that looked for messages that match a specific text format. Perhaps it's detecting the JUnit format and dropping the output in that case.

@Skynet0
Copy link
Author

Skynet0 commented Apr 21, 2020

Oh, here's something interesting - AssertJ does not have duplicating behavior in JUnit4, and it raises junit ComparisonFailure for both int and String:

@Test
public void testAssertJInJUnit4Int() {
    int x = 5;
    org.assertj.core.api.Assertions.assertThat(x).isEqualTo(10);
}

@Test
public void testAssertJInJunit4String() {
    String x = "foo";
    org.assertj.core.api.Assertions.assertThat(x).isEqualTo("bar");
}
org.junit.ComparisonFailure: 
Expected :10
Actual   :5
<Click to see difference>

org.junit.ComparisonFailure: 
Expected :"bar"
Actual   :"foo"
<Click to see difference>

(Mentioned above, but AssertJ has duplicate output behavior in JUnit5.)

@cpovirk
Copy link
Member

cpovirk commented Apr 21, 2020

Thanks. I haven't organized the details of this in my head, but it might be that:

  • Under JUnit 5, IntelliJ will duplicate the message unless you use the opentest4j AssertionFailedError.
  • Under JUnit 4, IntelliJ will duplicate the message unless your message matches the default message style of JUnit's ComparisonFailure. (That's my best guess based on this and this.)

@Skynet0
Copy link
Author

Skynet0 commented Apr 21, 2020

Don't think it's quite the first - in the AssertJ issue I linked above, it raises the opentest4j AssertionFailedError, but the output is still duplicated.

@cpovirk
Copy link
Member

cpovirk commented Apr 21, 2020

Oh, thanks. So it does.

The cause there may again be a custom exception message, [edit: which presumably IntelliJ detects under JUnit 5 as it might under JUnit 4]. AssertJ provides a custom message to the AssertionFailedError:
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L118
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L152
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L131
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L191

Contrast to JUnit, which uses the "expected:<...> but was:<...>" format:
https://github.com/junit-team/junit5/blob/d430c63460073b16049d506b05c82751f041d84e/junit-jupiter-api/src/main/java/org/junit/jupiter/api/AssertEquals.java#L177
https://github.com/junit-team/junit5/blob/d430c63460073b16049d506b05c82751f041d84e/junit-jupiter-api/src/main/java/org/junit/jupiter/api/AssertEquals.java#L182
https://github.com/junit-team/junit5/blob/d430c63460073b16049d506b05c82751f041d84e/junit-jupiter-api/src/main/java/org/junit/jupiter/api/AssertionUtils.java#L62
https://github.com/junit-team/junit5/blob/d430c63460073b16049d506b05c82751f041d84e/junit-jupiter-api/src/main/java/org/junit/jupiter/api/AssertionUtils.java#L112
https://github.com/junit-team/junit5/blob/d430c63460073b16049d506b05c82751f041d84e/junit-jupiter-api/src/main/java/org/junit/jupiter/api/AssertionUtils.java#L119

The duplication wouldn't appear under JUnit 4 because AssertJ throws ComparisonFailure without a custom message if the opentest4j types aren't on the classpath:
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L127
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L205

@cpovirk
Copy link
Member

cpovirk commented Apr 21, 2020

There may be drawbacks to this that I'm not thinking of, but....

I would love to see this changed on the IntelliJ side. If the ComparisonFailure or AssertionFailedError has a message, I would like to see IntelliJ skip displaying its own "Expected" and "Actual" (though keep its "<Click to see difference>").

In the case of the "standard JUnit message format," I wouldn't mind if IntelliJ instead skipped displaying the message in favor of displaying only "Expected" + "Actual" + "<Click to see difference>." This doesn't matter to Truth, but it matters to AssertJ under JUnit 4 and to JUnit's own assertions (4 and 5 both, IIUC).

@Skynet0
Copy link
Author

Skynet0 commented Apr 21, 2020

https://youtrack.jetbrains.com/issue/IDEA-238472

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P3 not scheduled type=enhancement Make an existing feature better
Projects
None yet
Development

No branches or pull requests

2 participants