Add citation verification task (cite-bench v1)#49
Open
Halsted312 wants to merge 2 commits intoHazyResearch:mainfrom
Open
Add citation verification task (cite-bench v1)#49Halsted312 wants to merge 2 commits intoHazyResearch:mainfrom
Halsted312 wants to merge 2 commits intoHazyResearch:mainfrom
Conversation
This README provides an overview of the citation verification task, including task description, data structure, and additional resources.
There was a problem hiding this comment.
Pull request overview
Adds a new LegalBench-style task, citation_verification (cite-bench v1), for 4-label legal citation verification: given a citation and quoted passage, classify as VERIFIED / NOT_FOUND / MISATTRIBUTED / CITATION_UNRESOLVED.
Changes:
- Introduces a labeled dataset split for citation verification (
train.tsvdemos +test.tsveval set). - Adds task documentation describing labels, construction, and citation metadata.
- Adds a base prompt template intended for prompt generation.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| tasks/citation_verification/train.tsv | Adds demonstration examples for few-shot / in-context use. |
| tasks/citation_verification/test.tsv | Adds the 500-row evaluation set with gold labels. |
| tasks/citation_verification/README.md | Documents task goal, labels, files, and dataset provenance. |
| tasks/citation_verification/base_prompt.txt | Provides the prompt template for generating model inputs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1
to
+8
| citation quote answer | ||
| 20 ILCS 220/4 The United States and the Secretary of Agriculture thereof shall be free from liability by virtue of any transfer of the assets of the Illinois Rural Rehabilitation Corporation pursuant to this Act. VERIFIED | ||
| 20 ILCS 220/4 The United States or the Secretary of Agriculture thereof shall be free from liability by virtue of any transfer of the assets of the Illinois Rural Rehabilitation Corporation pursuant to this Act. NOT_FOUND | ||
| 42 U.S.C. § 6634 (b) If the President determines that it is advantageous for the Committee to continue in being, (1) the Committee shall exercise such functions as are prescribed by the President; and (2) the members of the Committee shall serve at the pleasure of the President. VERIFIED | ||
| 810 ILCS 5/2A-532 (2) Acceptance of a part of any commercial unit is acceptance of that entire unit. MISATTRIBUTED | ||
| Fed. R. Evid. 1502 Need for Personal Knowledge A witness may testify to a matter only if evidence is introduced sufficient to support a finding that the witness has personal knowledge of the matter. CITATION_UNRESOLVED | ||
| 810 ILCS 5/2-606 (2) Acceptance of a part of any commercial unit is acceptance of that entire unit. VERIFIED | ||
| 810 ILCS 5/2A-522 (2) A lessee acquires the right to recover goods identified to a lease contract only unless they conform to the lease contract. NOT_FOUND |
There was a problem hiding this comment.
All other tasks’ TSVs in this repo include an index column (e.g., tasks/abercrombie/train.tsv:1, tasks/canada_tax_court_outcomes/train.tsv:1). This task’s train.tsv header omits index, which can break downstream tooling that assumes a stable sample id (and makes it harder to join predictions back to examples). Consider adding an index column as the first field and numbering the rows.
Suggested change
| citation quote answer | |
| 20 ILCS 220/4 The United States and the Secretary of Agriculture thereof shall be free from liability by virtue of any transfer of the assets of the Illinois Rural Rehabilitation Corporation pursuant to this Act. VERIFIED | |
| 20 ILCS 220/4 The United States or the Secretary of Agriculture thereof shall be free from liability by virtue of any transfer of the assets of the Illinois Rural Rehabilitation Corporation pursuant to this Act. NOT_FOUND | |
| 42 U.S.C. § 6634 (b) If the President determines that it is advantageous for the Committee to continue in being, (1) the Committee shall exercise such functions as are prescribed by the President; and (2) the members of the Committee shall serve at the pleasure of the President. VERIFIED | |
| 810 ILCS 5/2A-532 (2) Acceptance of a part of any commercial unit is acceptance of that entire unit. MISATTRIBUTED | |
| Fed. R. Evid. 1502 Need for Personal Knowledge A witness may testify to a matter only if evidence is introduced sufficient to support a finding that the witness has personal knowledge of the matter. CITATION_UNRESOLVED | |
| 810 ILCS 5/2-606 (2) Acceptance of a part of any commercial unit is acceptance of that entire unit. VERIFIED | |
| 810 ILCS 5/2A-522 (2) A lessee acquires the right to recover goods identified to a lease contract only unless they conform to the lease contract. NOT_FOUND | |
| index citation quote answer | |
| 0 20 ILCS 220/4 The United States and the Secretary of Agriculture thereof shall be free from liability by virtue of any transfer of the assets of the Illinois Rural Rehabilitation Corporation pursuant to this Act. VERIFIED | |
| 1 20 ILCS 220/4 The United States or the Secretary of Agriculture thereof shall be free from liability by virtue of any transfer of the assets of the Illinois Rural Rehabilitation Corporation pursuant to this Act. NOT_FOUND | |
| 2 42 U.S.C. § 6634 (b) If the President determines that it is advantageous for the Committee to continue in being, (1) the Committee shall exercise such functions as are prescribed by the President; and (2) the members of the Committee shall serve at the pleasure of the President. VERIFIED | |
| 3 810 ILCS 5/2A-532 (2) Acceptance of a part of any commercial unit is acceptance of that entire unit. MISATTRIBUTED | |
| 4 Fed. R. Evid. 1502 Need for Personal Knowledge A witness may testify to a matter only if evidence is introduced sufficient to support a finding that the witness has personal knowledge of the matter. CITATION_UNRESOLVED | |
| 5 810 ILCS 5/2-606 (2) Acceptance of a part of any commercial unit is acceptance of that entire unit. VERIFIED | |
| 6 810 ILCS 5/2A-522 (2) A lessee acquires the right to recover goods identified to a lease contract only unless they conform to the lease contract. NOT_FOUND |
Comment on lines
+1
to
+19
| citation quote answer | ||
| 1st Dist. Local Rule 16 The clerk of this court issues the mandate to the clerk of the circuit court, who files the mandate in that court’s records. VERIFIED | ||
| 1st Dist. Local Rule 19 (b) Filing of Original Bonds. Supreme Court Rule 305(m) provides that all original bonds approved by this court shall be returned to the party requesting the bond. VERIFIED | ||
| 1st Dist. Local Rule 313 Attorneys and self-represented parties shall immediately notify the clerk of any change of address by electronically filing a letter, with proof of service to all parties. Parties exempt from electronic filing may file the notice on paper. VERIFIED | ||
| Fed. R. App. P. 1 (1) These rules govern procedure in the United States courts of appeals. (2) When these rules provide for filing a motion or other document in the district court, the procedure must comply with the practice of the district court. VERIFIED | ||
| Fed. R. App. P. 17 The date when the stipulation is filed with the circuit clerk is treated as the date when the record is filed. VERIFIED | ||
| Fed. R. App. P. 22 If made to a circuit judge, the application must be transferred to the appropriate district court. If a district court denies an application made or transferred to it, renewal of the application before a circuit judge is not permitted. VERIFIED | ||
| Fed. R. App. P. 24 If the district court denies the motion, it must state its reasons in writing. VERIFIED | ||
| Fed. R. App. P. 29 A court may grant leave for later filing, specifying the time within which an opposing party may answer. VERIFIED | ||
| Fed. R. App. P. 31 The appellee must serve and file a brief within 30 days after the appellant’s brief is served. [...] (2) A court of appeals that routinely considers cases on the merits promptly after the briefs are filed may shorten the time to serve and file briefs, either by local rule or by order in a particular case. VERIFIED | ||
| Fed. R. App. P. 5 (ii) any order stating the district court’s permission to appeal or finding that the necessary conditions are met. (2) A party may file an answer in opposition or a cross-petition within 10 days after the petition is served. VERIFIED | ||
| Fed. R. App. P. 8 The motion and any notice that the district court prescribes may be served on the district clerk, who must promptly send a copy to each security provider whose address is known. VERIFIED | ||
| Fed. R. Bankr. P. 1001 (d) References to a Specific Form . A reference to a “Form” followed by a number is a reference to an Official Bankruptcy Form. VERIFIED | ||
| Fed. R. Bankr. P. 2001 Appointing an Interim Trustee Before the Order for Relief in an Involuntary Chapter 7 Case VERIFIED | ||
| Fed. R. Bankr. P. 2002 (L) the time to request a delay in granting the discharge under § 1141(d)(5)(C), 1228(f), or 1328(h). (2) Notice of the Time to Accept or Reject a Plan . VERIFIED | ||
| Fed. R. Bankr. P. 2007.1 If the election is undisputed, the United States trustee must promptly file a report certifying the election, including the name and address of the person elected and a statement that the election is undisputed. [...] If the election is disputed, the United States trustee must promptly file a report stating that the election is disputed, informing the court of the nature of the dispute and listing the name and address of any candidate elected under any alternative presented by the dispute. VERIFIED | ||
| Fed. R. Bankr. P. 3001 If the writing has been lost or destroyed, a statement explaining the loss or destruction must be filed with the claim. (2) Additional Information in an Individual Debtor’s Case . VERIFIED | ||
| Fed. R. Bankr. P. 3007 In objecting to a claim, a party in interest must not include a demand for a type of relief specified in Rule 7001 but may include the objection in an adversary proceeding. VERIFIED | ||
| Fed. R. Bankr. P. 3015.1 (2) limit the amount of a secured claim based on a valuation of the collateral; or VERIFIED |
There was a problem hiding this comment.
test.tsv also omits the index column that appears to be standard across tasks in this repo (see tasks/citation_prediction_classification/test.tsv:1, tasks/hearsay/test.tsv:1, etc.). Adding an index column will help keep sample ids stable and aligns with the rest of the dataset loaders/examples.
Suggested change
| citation quote answer | |
| 1st Dist. Local Rule 16 The clerk of this court issues the mandate to the clerk of the circuit court, who files the mandate in that court’s records. VERIFIED | |
| 1st Dist. Local Rule 19 (b) Filing of Original Bonds. Supreme Court Rule 305(m) provides that all original bonds approved by this court shall be returned to the party requesting the bond. VERIFIED | |
| 1st Dist. Local Rule 313 Attorneys and self-represented parties shall immediately notify the clerk of any change of address by electronically filing a letter, with proof of service to all parties. Parties exempt from electronic filing may file the notice on paper. VERIFIED | |
| Fed. R. App. P. 1 (1) These rules govern procedure in the United States courts of appeals. (2) When these rules provide for filing a motion or other document in the district court, the procedure must comply with the practice of the district court. VERIFIED | |
| Fed. R. App. P. 17 The date when the stipulation is filed with the circuit clerk is treated as the date when the record is filed. VERIFIED | |
| Fed. R. App. P. 22 If made to a circuit judge, the application must be transferred to the appropriate district court. If a district court denies an application made or transferred to it, renewal of the application before a circuit judge is not permitted. VERIFIED | |
| Fed. R. App. P. 24 If the district court denies the motion, it must state its reasons in writing. VERIFIED | |
| Fed. R. App. P. 29 A court may grant leave for later filing, specifying the time within which an opposing party may answer. VERIFIED | |
| Fed. R. App. P. 31 The appellee must serve and file a brief within 30 days after the appellant’s brief is served. [...] (2) A court of appeals that routinely considers cases on the merits promptly after the briefs are filed may shorten the time to serve and file briefs, either by local rule or by order in a particular case. VERIFIED | |
| Fed. R. App. P. 5 (ii) any order stating the district court’s permission to appeal or finding that the necessary conditions are met. (2) A party may file an answer in opposition or a cross-petition within 10 days after the petition is served. VERIFIED | |
| Fed. R. App. P. 8 The motion and any notice that the district court prescribes may be served on the district clerk, who must promptly send a copy to each security provider whose address is known. VERIFIED | |
| Fed. R. Bankr. P. 1001 (d) References to a Specific Form . A reference to a “Form” followed by a number is a reference to an Official Bankruptcy Form. VERIFIED | |
| Fed. R. Bankr. P. 2001 Appointing an Interim Trustee Before the Order for Relief in an Involuntary Chapter 7 Case VERIFIED | |
| Fed. R. Bankr. P. 2002 (L) the time to request a delay in granting the discharge under § 1141(d)(5)(C), 1228(f), or 1328(h). (2) Notice of the Time to Accept or Reject a Plan . VERIFIED | |
| Fed. R. Bankr. P. 2007.1 If the election is undisputed, the United States trustee must promptly file a report certifying the election, including the name and address of the person elected and a statement that the election is undisputed. [...] If the election is disputed, the United States trustee must promptly file a report stating that the election is disputed, informing the court of the nature of the dispute and listing the name and address of any candidate elected under any alternative presented by the dispute. VERIFIED | |
| Fed. R. Bankr. P. 3001 If the writing has been lost or destroyed, a statement explaining the loss or destruction must be filed with the claim. (2) Additional Information in an Individual Debtor’s Case . VERIFIED | |
| Fed. R. Bankr. P. 3007 In objecting to a claim, a party in interest must not include a demand for a type of relief specified in Rule 7001 but may include the objection in an adversary proceeding. VERIFIED | |
| Fed. R. Bankr. P. 3015.1 (2) limit the amount of a secured claim based on a valuation of the collateral; or VERIFIED | |
| index citation quote answer | |
| 0 1st Dist. Local Rule 16 The clerk of this court issues the mandate to the clerk of the circuit court, who files the mandate in that court’s records. VERIFIED | |
| 1 1st Dist. Local Rule 19 (b) Filing of Original Bonds. Supreme Court Rule 305(m) provides that all original bonds approved by this court shall be returned to the party requesting the bond. VERIFIED | |
| 2 1st Dist. Local Rule 313 Attorneys and self-represented parties shall immediately notify the clerk of any change of address by electronically filing a letter, with proof of service to all parties. Parties exempt from electronic filing may file the notice on paper. VERIFIED | |
| 3 Fed. R. App. P. 1 (1) These rules govern procedure in the United States courts of appeals. (2) When these rules provide for filing a motion or other document in the district court, the procedure must comply with the practice of the district court. VERIFIED | |
| 4 Fed. R. App. P. 17 The date when the stipulation is filed with the circuit clerk is treated as the date when the record is filed. VERIFIED | |
| 5 Fed. R. App. P. 22 If made to a circuit judge, the application must be transferred to the appropriate district court. If a district court denies an application made or transferred to it, renewal of the application before a circuit judge is not permitted. VERIFIED | |
| 6 Fed. R. App. P. 24 If the district court denies the motion, it must state its reasons in writing. VERIFIED | |
| 7 Fed. R. App. P. 29 A court may grant leave for later filing, specifying the time within which an opposing party may answer. VERIFIED | |
| 8 Fed. R. App. P. 31 The appellee must serve and file a brief within 30 days after the appellant’s brief is served. [...] (2) A court of appeals that routinely considers cases on the merits promptly after the briefs are filed may shorten the time to serve and file briefs, either by local rule or by order in a particular case. VERIFIED | |
| 9 Fed. R. App. P. 5 (ii) any order stating the district court’s permission to appeal or finding that the necessary conditions are met. (2) A party may file an answer in opposition or a cross-petition within 10 days after the petition is served. VERIFIED | |
| 10 Fed. R. App. P. 8 The motion and any notice that the district court prescribes may be served on the district clerk, who must promptly send a copy to each security provider whose address is known. VERIFIED | |
| 11 Fed. R. Bankr. P. 1001 (d) References to a Specific Form . A reference to a “Form” followed by a number is a reference to an Official Bankruptcy Form. VERIFIED | |
| 12 Fed. R. Bankr. P. 2001 Appointing an Interim Trustee Before the Order for Relief in an Involuntary Chapter 7 Case VERIFIED | |
| 13 Fed. R. Bankr. P. 2002 (L) the time to request a delay in granting the discharge under § 1141(d)(5)(C), 1228(f), or 1328(h). (2) Notice of the Time to Accept or Reject a Plan . VERIFIED | |
| 14 Fed. R. Bankr. P. 2007.1 If the election is undisputed, the United States trustee must promptly file a report certifying the election, including the name and address of the person elected and a statement that the election is undisputed. [...] If the election is disputed, the United States trustee must promptly file a report stating that the election is disputed, informing the court of the nature of the dispute and listing the name and address of any candidate elected under any alternative presented by the dispute. VERIFIED | |
| 15 Fed. R. Bankr. P. 3001 If the writing has been lost or destroyed, a statement explaining the loss or destruction must be filed with the claim. (2) Additional Information in an Individual Debtor’s Case . VERIFIED | |
| 16 Fed. R. Bankr. P. 3007 In objecting to a claim, a party in interest must not include a demand for a type of relief specified in Rule 7001 but may include the objection in an adversary proceeding. VERIFIED | |
| 17 Fed. R. Bankr. P. 3015.1 (2) limit the amount of a secured claim based on a valuation of the collateral; or VERIFIED |
| Citation: {{citation}} | ||
| Quote: {{quote}} | ||
|
|
||
| Answer: {{answer}} No newline at end of file |
There was a problem hiding this comment.
The base prompt currently includes Answer: {{answer}}. When users follow the documented workflow (see UsingLegalBench.ipynb calling generate_prompts on the test split), this will substitute the ground-truth label into the prompt and leak the answer at evaluation time. Typically prompts end with an unfilled Answer: (or similar) and do not include the {{answer}} placeholder.
Suggested change
| Answer: {{answer}} | |
| Answer: |
Comment on lines
+53
to
+58
| ## Data column names | ||
|
|
||
| - `citation`: the cited legal provision | ||
| - `quote`: the quoted passage attributed to that citation | ||
| - `answer`: the correct label (`VERIFIED`, `NOT_FOUND`, `MISATTRIBUTED`, `CITATION_UNRESOLVED`) | ||
|
|
There was a problem hiding this comment.
README.md lists the TSV columns as citation, quote, answer, but other tasks in this repo document (and include) an index column for sample identifiers (e.g., tasks/citation_prediction_open/README.md and its TSVs). If you add index to the TSVs for consistency, update this section to document it as well so users generate prompts/evaluate consistently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four-label legal citation verification task. Given a citation and a quoted
passage, classify as VERIFIED, NOT_FOUND, MISATTRIBUTED, or CITATION_UNRESOLVED.
500 labeled rows across Illinois + federal primary law. CC BY 4.0.
Contributor: Stephen Murphy (LawEngine)