-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Java: Add query to detect special characters in string literals #19875
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+357
−0
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
09a2aee
Java: Add query to detect special characters in string literals
tamasvajk fd8b37c
Exclude Kotlin files
tamasvajk a0c9c98
Adjust references in query doc
tamasvajk 15de398
Adjust query tags
tamasvajk c4def10
Improve query documentation
tamasvajk d16570b
Revert "Adjust query tags"
tamasvajk ccbf705
Adjust query precision
tamasvajk File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
111 changes: 111 additions & 0 deletions
111
...e/SpecialCharactersInLiterals/NonExplicitControlAndWhitespaceCharsInLiterals.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| ## Overview | ||
|
|
||
| This query detects non-explicit control and whitespace characters in Java literals. | ||
| Such characters are often introduced accidentally and can be invisible or hard to recognize, leading to bugs when the actual contents of the string contain control characters. | ||
|
|
||
| ## Recommendation | ||
|
|
||
| To avoid issues, use the encoded versions of control characters (e.g. ASCII `\n`, `\t`, or Unicode `U+000D`, `U+0009`). | ||
| This makes the literals (e.g. string literals) more readable, and also helps to make the surrounding code less error-prone and more maintainable. | ||
|
|
||
| ## Example | ||
|
|
||
| The following examples illustrate good and bad code: | ||
|
|
||
| Bad: | ||
|
|
||
| ```java | ||
| char tabulationChar = ' '; // Non compliant | ||
| String tabulationCharInsideString = "A B"; // Non compliant | ||
| String fooZeroWidthSpacebar = "foobar"; // Non compliant | ||
| ``` | ||
|
|
||
| Good: | ||
|
|
||
| ```java | ||
| char escapedTabulationChar = '\t'; | ||
| String escapedTabulationCharInsideString = "A\tB"; // Compliant | ||
| String fooUnicodeSpacebar = "foo\u0020bar"; // Compliant | ||
| String foo2Spacebar = "foo bar"; // Compliant | ||
| String foo3Spacebar = "foo bar"; // Compliant | ||
| ``` | ||
|
|
||
| ## Implementation notes | ||
|
|
||
| This query detects Java literals that contain reserved control characters and/or non-printable whitespace characters, such as: | ||
|
|
||
| - Decimal and hexidecimal representations of ASCII control characters (code points 0-8, 11, 14-31, and 127). | ||
| - Invisible characters (e.g. zero-width space, zero-width joiner). | ||
| - Unicode C0 control codes, plus the delete character (U+007F), such as: | ||
|
|
||
| | Escaped Unicode | ASCII Decimal | Description | | ||
| | --------------- | ------------- | ------------------------- | | ||
| | `\u0000` | 0 | null character | | ||
| | `\u0001` | 1 | start of heading | | ||
| | `\u0002` | 2 | start of text | | ||
| | `\u0003` | 3 | end of text | | ||
| | `\u0004` | 4 | end of transmission | | ||
| | `\u0005` | 5 | enquiry | | ||
| | `\u0006` | 6 | acknowledge | | ||
| | `\u0007` | 7 | bell | | ||
| | `\u0008` | 8 | backspace | | ||
| | `\u000B` | 11 | vertical tab | | ||
| | `\u000E` | 14 | shift out | | ||
| | `\u000F` | 15 | shift in | | ||
| | `\u0010` | 16 | data link escape | | ||
| | `\u0011` | 17 | device control 1 | | ||
| | `\u0012` | 18 | device control 2 | | ||
| | `\u0013` | 19 | device control 3 | | ||
| | `\u0014` | 20 | device control 4 | | ||
| | `\u0015` | 21 | negative acknowledge | | ||
| | `\u0016` | 22 | synchronous idle | | ||
| | `\u0017` | 23 | end of transmission block | | ||
| | `\u0018` | 24 | cancel | | ||
| | `\u0019` | 25 | end of medium | | ||
| | `\u001A` | 26 | substitute | | ||
| | `\u001B` | 27 | escape | | ||
| | `\u001C` | 28 | file separator | | ||
| | `\u001D` | 29 | group separator | | ||
| | `\u001E` | 30 | record separator | | ||
| | `\u001F` | 31 | unit separator | | ||
| | `\u007F` | 127 | delete | | ||
|
|
||
| - Zero-width Unicode characters (e.g. zero-width space, zero-width joiner), such as: | ||
|
|
||
| | Escaped Unicode | Description | | ||
| | --------------- | ------------------------- | | ||
| | `\u200B` | zero-width space | | ||
| | `\u200C` | zero-width non-joiner | | ||
| | `\u200D` | zero-width joiner | | ||
| | `\u2028` | line separator | | ||
| | `\u2029` | paragraph separator | | ||
| | `\u2060` | word joiner | | ||
| | `\uFEFF` | zero-width no-break space | | ||
|
|
||
| The following list outlines the _**explicit exclusions from query scope**_: | ||
|
|
||
| - any number of simple space characters (`U+0020`, ASCII 32). | ||
| - an escape character sequence (e.g. `\t`), or the Unicode equivalent (e.g. `\u0009`), for printable whitespace characters: | ||
|
|
||
| | Character Sequence | Escaped Unicode | ASCII Decimal | Description | | ||
| | ------------------ | --------------- | ------------- | --------------- | | ||
| | `\t` | \u0009 | 9 | horizontal tab | | ||
| | `\n` | \u000A | 10 | line feed | | ||
| | `\f` | \u000C | 12 | form feed | | ||
| | `\r` | \u000D | 13 | carriage return | | ||
| | | \u0020 | 32 | space | | ||
|
|
||
| - character literals (i.e. single quotes) containing control characters. | ||
| - literals defined within "likely" test methods, such as: | ||
| - JUnit test methods | ||
| - methods annotated with `@Test` | ||
| - methods of a class annotated with `@Test` | ||
| - methods with names containing "test" | ||
|
|
||
| ## References | ||
|
|
||
| - Unicode: [Unicode Control Characters](https://www.unicode.org/charts/PDF/U0000.pdf). | ||
| - Wikipedia: [Unicode C0 control codes](https://en.wikipedia.org/wiki/C0_and_C1_control_codes). | ||
| - Wikipedia: [Unicode characters with property "WSpace=yes" or "White_Space=yes"](https://en.wikipedia.org/wiki/Unicode_character_property#Whitespace). | ||
| - Java API Specification: [Java String Literals](https://docs.oracle.com/javase/tutorial/java/data/characters.html). | ||
| - Java API Specification: [Java Class Charset](https://docs.oracle.com/javase/8/docs/api///?java/nio/charset/Charset.html). |
51 changes: 51 additions & 0 deletions
51
...st Practice/SpecialCharactersInLiterals/NonExplicitControlAndWhitespaceCharsInLiterals.ql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| /** | ||
| * @id java/non-explicit-control-and-whitespace-chars-in-literals | ||
| * @name Non-explicit control and whitespace characters | ||
| * @description Non-explicit control and whitespace characters in literals make code more difficult | ||
| * to read and may lead to incorrect program behavior. | ||
| * @kind problem | ||
| * @precision very-high | ||
| * @problem.severity warning | ||
| * @tags quality | ||
| * correctness | ||
| * maintainability | ||
| * readability | ||
| */ | ||
|
|
||
| import java | ||
|
|
||
| /** | ||
| * A `Literal` that has a Unicode control character within its | ||
| * literal value (as returned by `getLiteral()` member predicate). | ||
| */ | ||
| class ReservedUnicodeInLiteral extends Literal { | ||
| private int indexStart; | ||
|
|
||
| ReservedUnicodeInLiteral() { | ||
| not this instanceof CharacterLiteral and | ||
| exists(int codePoint | | ||
| this.getLiteral().codePointAt(indexStart) = codePoint and | ||
| ( | ||
| // Unicode C0 control characters | ||
| codePoint < 32 and not codePoint in [9, 10, 12, 13] | ||
| or | ||
| codePoint = 127 // delete control character | ||
| or | ||
| codePoint = 8203 // zero-width space | ||
| ) | ||
| ) | ||
| } | ||
|
|
||
| /** Gets the starting index of the Unicode control sequence. */ | ||
| int getIndexStart() { result = indexStart } | ||
| } | ||
|
|
||
| from ReservedUnicodeInLiteral literal, int charIndex, int codePoint | ||
| where | ||
| literal.getIndexStart() = charIndex and | ||
| literal.getLiteral().codePointAt(charIndex) = codePoint and | ||
| not literal.getEnclosingCallable() instanceof LikelyTestMethod and | ||
| not literal.getFile().isKotlinSourceFile() | ||
| select literal, | ||
| "Literal value contains control or non-printable whitespace character(s) starting with Unicode code point " | ||
| + codePoint + " at index " + charIndex + "." | ||
184 changes: 184 additions & 0 deletions
184
java/ql/test/query-tests/NonExplicitControlAndWhitespaceCharsInLiterals/CharTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,184 @@ | ||
| import java.util.List; | ||
| import java.util.ArrayList; | ||
|
|
||
| public class CharTest { | ||
|
|
||
| public static void main(String[] args) { | ||
| CharTest charTest = new CharTest(); | ||
| NonCompliantStringLiterals nonCompliant = charTest.new NonCompliantStringLiterals(); | ||
| CompliantStringLiterals compliant = charTest.new CompliantStringLiterals(); | ||
| CompliantCharLiterals compliantChar = charTest.new CompliantCharLiterals(); | ||
|
|
||
| List<String> nonCompliantStrings = nonCompliant.getNonCompliantStrings(); | ||
| List<String> compliantStrings = compliant.getCompliantStrings(); | ||
| List<Character> compliantChars = compliantChar.getCompliantChars(); | ||
|
|
||
| System.out.println(""); | ||
| System.out.println("Non-compliant strings:"); | ||
| for (String s : nonCompliantStrings) { | ||
| System.out.println(s); | ||
| System.out.println(""); | ||
| } | ||
|
|
||
| System.out.println(""); | ||
| System.out.println("Compliant strings:"); | ||
| for (String s : compliantStrings) { | ||
| System.out.println(s); | ||
| System.out.println(""); | ||
| } | ||
| System.out.println(""); | ||
|
|
||
| System.out.println(""); | ||
| System.out.println("Compliant character literals:"); | ||
| System.out.println(""); | ||
| for (Character c : compliantChars) { | ||
| System.out.println("\\u" + String.format("%04X", (int) c)); | ||
| } | ||
| System.out.println(""); | ||
| } | ||
|
|
||
| class CompliantCharLiterals { | ||
| private List<Character> compliantChars; | ||
|
|
||
| public CompliantCharLiterals() { | ||
| compliantChars = new ArrayList<>(); | ||
| compliantChars.add('A'); // COMPLIANT | ||
| compliantChars.add('a'); // COMPLIANT | ||
| compliantChars.add('\b'); // COMPLIANT | ||
| compliantChars.add('\t'); // COMPLIANT | ||
| compliantChars.add('\n'); // COMPLIANT | ||
| compliantChars.add('\f'); // COMPLIANT | ||
| compliantChars.add('\r'); // COMPLIANT | ||
| compliantChars.add('\u0000'); // COMPLIANT | ||
| compliantChars.add('\u0007'); // COMPLIANT | ||
| compliantChars.add('\u001B'); // COMPLIANT | ||
| compliantChars.add(' '); // COMPLIANT | ||
| compliantChars.add('\u0020'); // COMPLIANT | ||
| compliantChars.add('\u200B'); // COMPLIANT | ||
| compliantChars.add('\u200C'); // COMPLIANT | ||
| compliantChars.add('\u200D'); // COMPLIANT | ||
| compliantChars.add('\u2028'); // COMPLIANT | ||
| compliantChars.add('\u2029'); // COMPLIANT | ||
| compliantChars.add('\u2060'); // COMPLIANT | ||
| compliantChars.add('\uFEFF'); // COMPLIANT | ||
| } | ||
|
|
||
| public List<Character> getCompliantChars() { | ||
| return compliantChars; | ||
| } | ||
| } | ||
|
|
||
| class CompliantStringLiterals { | ||
| private List<String> compliantStrings; | ||
|
|
||
| public CompliantStringLiterals() { | ||
| compliantStrings = new ArrayList<>(); | ||
| compliantStrings.add(""); // COMPLIANT | ||
| compliantStrings.add("X__Y"); // COMPLIANT | ||
| compliantStrings.add("X_ _Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0020_Y"); // COMPLIANT | ||
| compliantStrings.add("X_ _Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0020\u0020_Y"); // COMPLIANT | ||
| compliantStrings.add("X_ _Y"); // COMPLIANT | ||
| compliantStrings.add("X_ _Y"); // COMPLIANT | ||
| compliantStrings.add("X_ _Y"); // COMPLIANT | ||
| compliantStrings.add("X_ _Y"); // COMPLIANT | ||
| compliantStrings.add("X_\b_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0000_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0001_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0002_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0003_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0004_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0005_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0006_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0007_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0008_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0009_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0010_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0011_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0012_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0013_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0014_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0015_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0016_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0017_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0018_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u0019_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u001A_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u001B_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u001C_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u001D_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u001E_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u001F_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u007F_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u200B_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u200C_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u200D_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u2028_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u2029_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\u2060_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\uFEFF_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\uFEFF_Y_\u0020_Z"); // COMPLIANT | ||
| compliantStrings.add("X_\uFEFF_Y_\uFEFF_Z"); // COMPLIANT | ||
| compliantStrings.add("X_\u0020_Y_\uFEFF_Z"); // COMPLIANT | ||
| compliantStrings.add("X_\t_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\t\t_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\b_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\f_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\f_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\n_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\n\t_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\n_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\r_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\r_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\t_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\t_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\u0000_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\u0007_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\u001B_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\u200B_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\u200C_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\u200D_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\u2028_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\u2029_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\u2060_Y"); // COMPLIANT | ||
| compliantStrings.add("X_\\uFEFF_Y"); // COMPLIANT | ||
| compliantStrings.add("lorem ipsum dolor "+"sit amet"); // COMPLIANT | ||
| compliantStrings.add("lorem ipsum dolor " + "sit amet"); // COMPLIANT | ||
| compliantStrings.add("lorem ipsum dolor sit amet, consectetur adipiscing elit, " + // COMPLIANT | ||
| "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."); | ||
| compliantStrings.add("lorem ipsum dolor sit amet, consectetur adipiscing elit, " + // COMPLIANT | ||
| "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad " | ||
| + "minim veniam, quis nostrud exercitation ullamco "+"laboris nisi ut aliquip ex " + | ||
| "ea commodo consequat."); | ||
| compliantStrings.add(""" | ||
| lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. | ||
| Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. | ||
| """); // COMPLIANT | ||
| } | ||
|
|
||
| public List<String> getCompliantStrings() { | ||
| return compliantStrings; | ||
| } | ||
| } | ||
|
|
||
| class NonCompliantStringLiterals { | ||
| private List<String> nonCompliantStrings; | ||
|
|
||
| public NonCompliantStringLiterals() { | ||
| nonCompliantStrings = new ArrayList<>(); | ||
| nonCompliantStrings.add("X__Y"); // NON_COMPLIANT | ||
| nonCompliantStrings.add("X__Y__Z"); // NON_COMPLIANT | ||
| nonCompliantStrings.add("loremipsum dolor sit amet,consectetur adipiscing elit, " + // NON_COMPLIANT | ||
| "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."); | ||
| nonCompliantStrings.add(""" | ||
| loremipsum dolor sit amet,consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. | ||
| Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. | ||
| """); // NON_COMPLIANT | ||
| } | ||
|
|
||
| public List<String> getNonCompliantStrings() { | ||
| return nonCompliantStrings; | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just checking: you believe this matches both the
correctnessandreadabilitycategories?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think
readabilityis definitely okay.correctnessmight/could be challenged. What do you think?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it can belong to both. But between
maintainabilityandreliability, you think it's better to choosemaintainability, becausereadabilityis definitely an issue whereascorrectnessis only occasionally an issue?Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't given it this much thought. I simply thought maintainability goes together with readability. I think we could remove the
correctnesstag, and then it fully matches the guidelines, of having one top level tag and one subcategory from that tag.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Eh, I think it's fine as it is actually.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, I just read the description of this PR and it says that we shouldn't use subcategories from different top-level categories. I will discuss the issue on that PR. Maybe wait until that discussion is complete to see if we should remove
correctness.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You don't need to remove
correctness.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I already removed it. I think it's okay if we keep the tags to a minimum for the time being.