|
| 1 | +# Known Issues and Limitations |
| 2 | + |
| 3 | +## Search Functionality |
| 4 | + |
| 5 | +### 1. NOT Operator Doesn't Exclude Matches |
| 6 | + |
| 7 | +**Status:** Known bug, not yet fixed |
| 8 | + |
| 9 | +**Issue:** Queries like `"foo NOT bar"` should find documents containing "foo" but not "bar". Currently, it returns documents containing both. |
| 10 | + |
| 11 | +**Example:** |
| 12 | +```bash |
| 13 | +km put "foo and bar together" |
| 14 | +km put "only foo here" |
| 15 | +km search "foo NOT bar" |
| 16 | +# Expected: 1 result (only foo here) |
| 17 | +# Actual: 2 results (both documents) |
| 18 | +``` |
| 19 | + |
| 20 | +**Root Cause:** |
| 21 | +- FTS query extraction passes `"NOT (bar)"` to SQLite FTS5 |
| 22 | +- SQLite FTS5's NOT operator support is limited/broken |
| 23 | +- No LINQ post-filtering is applied to exclude NOT terms |
| 24 | +- The architecture assumes FTS handles all logic, but NOT needs LINQ filtering |
| 25 | + |
| 26 | +**Workaround:** None currently. Avoid using NOT operator. |
| 27 | + |
| 28 | +**Fix Required:** |
| 29 | +1. Split query: extract positive terms for FTS, negative terms for filtering |
| 30 | +2. Apply LINQ filter to FTS results using QueryLinqBuilder |
| 31 | +3. Filter out documents matching NOT terms |
| 32 | + |
| 33 | +**Files Affected:** |
| 34 | +- `src/Core/Search/NodeSearchService.cs:190` - ExtractLogical NOT handling |
| 35 | +- Need to add LINQ filtering after line 89 |
| 36 | + |
| 37 | +--- |
| 38 | + |
| 39 | +### 2. Quoted Phrases Don't Escape Operators |
| 40 | + |
| 41 | +**Status:** Known bug, not yet fixed |
| 42 | + |
| 43 | +**Issue:** Cannot search for literal phrases containing reserved words like "AND", "OR", "NOT". |
| 44 | + |
| 45 | +**Example:** |
| 46 | +```bash |
| 47 | +km put "Meeting with Alice AND Bob" |
| 48 | +km search '"Alice AND Bob"' |
| 49 | +# Expected: Find the document |
| 50 | +# Actual: Parser error or incorrect results |
| 51 | +``` |
| 52 | + |
| 53 | +**Root Cause:** |
| 54 | +- Quoted strings should treat content literally |
| 55 | +- Current parser/tokenizer doesn't properly handle operator escaping within quotes |
| 56 | +- May be FTS query generation issue |
| 57 | + |
| 58 | +**Workaround:** Rephrase searches to avoid reserved words. |
| 59 | + |
| 60 | +**Fix Required:** Investigate tokenizer and FTS query extraction for quoted phrases. |
| 61 | + |
| 62 | +--- |
| 63 | + |
| 64 | +### 3. Field Queries with Quoted Values Fail |
| 65 | + |
| 66 | +**Status:** Known bug, not yet fixed |
| 67 | + |
| 68 | +**Issue:** Field-specific queries with quoted values containing special characters fail. |
| 69 | + |
| 70 | +**Example:** |
| 71 | +```bash |
| 72 | +km put "user:password format" |
| 73 | +km search 'content:"user:password"' |
| 74 | +# Expected: Find the document |
| 75 | +# Actual: SQLite error "unknown special query" |
| 76 | +``` |
| 77 | + |
| 78 | +**Root Cause:** |
| 79 | +- Quoted values after field prefix (`content:"..."`) generate invalid FTS queries |
| 80 | +- FTS syntax may not support this pattern |
| 81 | +- Need investigation of FTS query generation |
| 82 | + |
| 83 | +**Workaround:** Search without field prefix or without quotes. |
| 84 | + |
| 85 | +--- |
| 86 | + |
| 87 | +### 4. Reserved Words Cannot Be Searched |
| 88 | + |
| 89 | +**Status:** Known limitation |
| 90 | + |
| 91 | +**Issue:** Cannot search for the literal words "AND", "OR", "NOT" even with quotes. |
| 92 | + |
| 93 | +**Example:** |
| 94 | +```bash |
| 95 | +km put "this is NOT important" |
| 96 | +km search "NOT" |
| 97 | +# Expected: Find the document |
| 98 | +# Actual: Parser error "Unexpected end of query" |
| 99 | +``` |
| 100 | + |
| 101 | +**Root Cause:** |
| 102 | +- Tokenizer treats AND/OR/NOT as reserved keywords (case-insensitive) |
| 103 | +- Even quoted, they're tokenized as operators |
| 104 | +- Parser expects operands after NOT |
| 105 | + |
| 106 | +**Workaround:** None. These words cannot be searched. |
| 107 | + |
| 108 | +**Fix Required:** |
| 109 | +- Tokenizer must recognize quotes and treat content literally |
| 110 | +- Major parser refactoring needed |
| 111 | + |
| 112 | +--- |
| 113 | + |
| 114 | +## Testing Gaps |
| 115 | + |
| 116 | +These bugs were discovered through comprehensive E2E testing. Previous tests only verified: |
| 117 | +- ✅ AST structure correctness |
| 118 | +- ✅ LINQ expression building |
| 119 | +- ✅ Direct FTS calls |
| 120 | + |
| 121 | +But did NOT test: |
| 122 | +- ❌ Full pipeline: Parse → Extract FTS → Search → Filter → Rank |
| 123 | +- ❌ Default settings (MinRelevance=0.3) |
| 124 | +- ❌ Actual result verification |
| 125 | + |
| 126 | +**Lesson:** Exit code testing and structure testing are insufficient. Must test actual behavior with real data. |
| 127 | + |
| 128 | +--- |
| 129 | + |
| 130 | +## Resolved Issues |
| 131 | + |
| 132 | +### BM25 Score Normalization (FIXED) |
| 133 | +- **Issue:** All searches returned 0 results despite FTS finding matches |
| 134 | +- **Cause:** BM25 scores (~0.000001) filtered by MinRelevance=0.3 |
| 135 | +- **Fix:** Exponential normalization maps [-10, 0] → [0.37, 1.0] |
| 136 | +- **Commit:** 4cb283e |
| 137 | + |
| 138 | +### Field-Specific Equal Operator (FIXED) |
| 139 | +- **Issue:** `content:summaries` failed with SQLite error |
| 140 | +- **Cause:** Equal operator didn't extract FTS queries |
| 141 | +- **Fix:** ExtractComparison now handles both Contains and Equal |
| 142 | +- **Commit:** 59bf3f2 |
0 commit comments