Strip Datalab _meta sidecars from extracted_data (#3)

* Strip Datalab _meta sidecars from extracted_data

Datalab's extract API interleaves per-field metadata with the extracted
values inside extraction_schema_json. The provider stripped the documented
{field}_citations and {field}_score sidecars, but balanced extraction mode
attaches a {field}_meta sidecar that was left in place, so every one was
scored as an extra predicted value against ground truth that has no such
key.

Strip *_meta alongside the other sidecar suffixes and skip it in the
citation collector so its inner citations list does not produce spurious
FieldCitation entries. Refresh the Datalab row on the leaderboard: overall
value F1 64.48 -> 85.70.

* Rename extract datalab test to avoid module basename collision

Author: boyang-zhang1

README.md

@@ -32,11 +32,11 @@ Models and prices reflect each provider's official documentation as of July 1, 2
 | 6 | Claude Code (Opus 4.8) | Coding Agents | 87.09 | 90.08 | 79.21 | 88.07 | 16.17¢ |
 | 7 | LlamaExtract Cost-Effective | LlamaExtract | 86.78 | 90.77 | 80.12 | 69.17 | 1.00¢ |
 | 8 | Extend (Max Context) | Specialized APIs | 86.29 | 91.98 | 78.78 | 51.33 | 10.00¢ |
-| 9 | Google Gemini 3.5 Flash | Commercial VLM | 79.84 | 87.87 | 69.76 | 27.90 | 1.00¢ |
-| 10 | Lift Datalab 9B | OSS | 77.31 | 87.17 | 62.59 | 25.26 | — |
-| 11 | OpenAI GPT-5.4 Nano | Commercial VLM | 74.90 | 77.43 | 76.37 | 35.81 | 0.21¢ |
-| 12 | Gemma4 26B | OSS | 66.24 | 80.55 | 40.47 | 12.16 | — |
-| 13 | Datalab (Accurate + Balanced) | Specialized APIs | 64.48 | 62.77 | 73.75 | 40.54 | 3.50¢ |
+| 9 | Datalab (Accurate + Balanced) | Specialized APIs | 85.70 | 89.40 | 85.08 | 42.04 | 3.50¢ |
+| 10 | Google Gemini 3.5 Flash | Commercial VLM | 79.84 | 87.87 | 69.76 | 27.90 | 1.00¢ |
+| 11 | Lift Datalab 9B | OSS | 77.31 | 87.17 | 62.59 | 25.26 | — |
+| 12 | OpenAI GPT-5.4 Nano | Commercial VLM | 74.90 | 77.43 | 76.37 | 35.81 | 0.21¢ |
+| 13 | Gemma4 26B | OSS | 66.24 | 80.55 | 40.47 | 12.16 | — |
 | 14 | NuExtract3 | OSS | 47.93 | 54.36 | 39.34 | 8.95 | — |
 <!-- LEADERBOARD:END -->
 
@@ -54,7 +54,7 @@ Models and prices reflect each provider's official documentation as of July 1, 2
     <tr><td align="right">3</td><td>Reducto Deep Extract</td><td align="right">43.30</td><td align="right"><u>42.84</u></td><td align="right">45.57</td><td align="right">41.13</td><td align="right"><u>71.71</u></td><td align="right"><u>72.60</u></td><td align="right"><u>70.42</u></td><td align="right">67.28</td></tr>
     <tr><td align="right">4</td><td>LlamaExtract Cost-Effective</td><td align="right">40.43</td><td align="right">40.20</td><td align="right">42.30</td><td align="right">36.67</td><td align="right">64.15</td><td align="right">68.90</td><td align="right">53.73</td><td align="right">56.55</td></tr>
     <tr><td align="right">5</td><td>Extend (Max Context)</td><td align="right">25.08</td><td align="right">33.91</td><td align="right">0.20</td><td align="right">0.02</td><td align="right">48.87</td><td align="right">61.71</td><td align="right">27.68</td><td align="right">0.02</td></tr>
-    <tr><td align="right">6</td><td>Datalab (Accurate + Balanced)</td><td align="right">2.02</td><td align="right">2.67</td><td align="right">0.23</td><td align="right">0.00</td><td align="right">48.50</td><td align="right">56.90</td><td align="right">38.56</td><td align="right">0.01</td></tr>
+    <tr><td align="right">6</td><td>Datalab (Accurate + Balanced)</td><td align="right">2.02</td><td align="right">2.67</td><td align="right">0.24</td><td align="right">0.00</td><td align="right">48.50</td><td align="right">56.90</td><td align="right">38.55</td><td align="right">0.01</td></tr>
     <tr><td align="right">—</td><td><em>All 8 other systems</em></td><td align="right">0.00</td><td align="right">0.00</td><td align="right">0.00</td><td align="right">0.00</td><td align="right">0.00</td><td align="right">0.00</td><td align="right">0.00</td><td align="right">0.00</td></tr>
   </tbody>
 </table>

leaderboard.csv

@@ -7,9 +7,9 @@ Qwen3.6 35B,OSS,87.33,93.11,84.85,26.75,,,,,93.22,85.48,50.96,93.27,84.29,25.71,
 Claude Code (Opus 4.8),Coding Agents,87.09,90.08,79.21,88.07,0.1617,0.2144,0.0823,0.0338,90.17,79.80,89.38,90.20,78.94,87.54,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,70.2,157.9,315.5
 LlamaExtract Cost-Effective,LlamaExtract,86.78,90.77,80.12,69.17,0.0100,0.0100,0.0100,0.0100,91.24,85.48,80.73,90.63,79.22,63.41,40.43,40.20,42.30,36.67,64.15,68.90,53.73,56.55,105.7,205.0,510.4
 Extend (Max Context),Specialized APIs,86.29,91.98,78.78,51.33,0.1000,0.1000,0.1000,0.1000,92.05,77.41,51.21,91.94,80.87,51.45,25.08,33.91,0.20,0.02,48.87,61.71,27.68,0.02,81.9,125.7,154.9
+Datalab (Accurate + Balanced),Specialized APIs,85.70,89.40,85.08,42.04,0.0350,0.0350,0.0350,0.0350,89.68,85.52,42.02,89.57,84.70,42.05,2.02,2.67,0.24,0.00,48.50,56.90,38.55,0.01,246.5,378.4,479.7
 Google Gemini 3.5 Flash,Commercial VLM,79.84,87.87,69.76,27.90,0.0100,0.0118,0.0069,0.0024,88.17,75.13,83.72,87.70,69.34,26.49,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,14.0,36.7,22.8
 Lift Datalab 9B,OSS,77.31,87.17,62.59,25.26,,,,,87.43,65.07,37.44,87.17,62.35,24.19,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,70.9,286.6,140.4
 OpenAI GPT-5.4 Nano,Commercial VLM,74.90,77.43,76.37,35.81,0.0021,0.0025,0.0012,0.0005,77.36,80.27,71.17,78.20,75.52,33.72,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,23.5,63.8,184.3
 Gemma4 26B,OSS,66.24,80.55,40.47,12.16,,,,,81.20,41.64,12.49,80.19,40.63,11.88,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,137.0,287.8,561.2
-Datalab (Accurate + Balanced),Specialized APIs,64.48,62.77,73.75,40.54,0.0350,0.0350,0.0350,0.0350,63.06,73.94,40.52,62.90,73.58,40.55,2.02,2.67,0.23,0.00,48.50,56.90,38.56,0.01,246.5,378.4,479.7
 NuExtract3,OSS,47.93,54.36,39.34,8.95,,,,,66.54,61.18,50.09,51.03,37.40,5.83,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,92.2,190.8,317.8

src/extract_bench/inference/providers/extract/datalab.py

@@ -367,10 +367,12 @@ def _adapt_schema_for_datalab(schema: Any) -> Any:
 
 _CITATIONS_SUFFIX = "_citations"
 _SCORE_SUFFIX = "_score"
+_META_SUFFIX = "_meta"
+_SIDECAR_SUFFIXES = (_CITATIONS_SUFFIX, _SCORE_SUFFIX, _META_SUFFIX)
 
 
 def _strip_citation_and_score_keys(node: Any) -> Any:
-    """Return a copy of ``node`` with ``*_citations`` and ``*_score`` keys removed.
+    """Return a copy of ``node`` with ``*_citations``, ``*_score`` and ``*_meta`` keys removed.
 
     Datalab interleaves the citation/score metadata with the actual field
     values inside ``extraction_schema_json``. The bench schema stores those as
@@ -380,7 +382,7 @@ def _strip_citation_and_score_keys(node: Any) -> Any:
     if isinstance(node, Mapping):
         cleaned: dict[str, Any] = {}
         for key, value in node.items():
-            if isinstance(key, str) and (key.endswith(_CITATIONS_SUFFIX) or key.endswith(_SCORE_SUFFIX)):
+            if isinstance(key, str) and key.endswith(_SIDECAR_SUFFIXES):
                 continue
             cleaned[key] = _strip_citation_and_score_keys(value)
         return cleaned
@@ -515,7 +517,7 @@ def _collect_citations(
                         if citation is not None:
                             citations.append(citation)
                 continue
-            if key.endswith(_SCORE_SUFFIX):
+            if key.endswith(_SCORE_SUFFIX) or key.endswith(_META_SUFFIX):
                 continue
             _collect_citations(
                 value,

tests/extract_bench/inference/providers/extract/test_datalab_extract.py

@@ -0,0 +1,61 @@
+from __future__ import annotations
+
+import pytest
+
+pytest.importorskip("pypdf", reason="dev and runners extras required; run: uv sync --extra dev --extra runners")
+pytest.importorskip("datalab_sdk", reason="dev and runners extras required; run: uv sync --extra dev --extra runners")
+
+from extract_bench.inference.providers.extract.datalab import (
+    _build_field_citations,
+    _strip_citation_and_score_keys,
+)
+
+
+def _meta() -> dict:
+    return {
+        "extraction_status": "EXTRACTED",
+        "reasoning": None,
+        "citations": ["/page/0/Table/7"],
+        "verification": {"status": "PASS", "feedback": "ok"},
+    }
+
+
+def test_strip_removes_meta_alongside_citations_and_score() -> None:
+    payload = {
+        "invoice_number": "NF67652",
+        "invoice_number_meta": _meta(),
+        "invoice_number_citations": ["/page/0/Table/7"],
+        "total": 1382.0,
+        "total_meta": _meta(),
+        "total_score": {"score": 5},
+    }
+
+    assert _strip_citation_and_score_keys(payload) == {
+        "invoice_number": "NF67652",
+        "total": 1382.0,
+    }
+
+
+def test_strip_removes_nested_meta_keys() -> None:
+    payload = {
+        "line_items": [
+            {
+                "description": "widget",
+                "description_meta": _meta(),
+                "amount": 10.0,
+                "amount_meta": _meta(),
+            }
+        ],
+        "vendor": {"name": "ACME", "name_meta": _meta()},
+    }
+
+    assert _strip_citation_and_score_keys(payload) == {
+        "line_items": [{"description": "widget", "amount": 10.0}],
+        "vendor": {"name": "ACME"},
+    }
+
+
+def test_meta_sidecar_produces_no_citations() -> None:
+    payload = {"invoice_number": "NF67652", "invoice_number_meta": _meta()}
+
+    assert _build_field_citations(payload, block_lookup={}) == []