Feature request
A structured dataset proposal introducing a Turkish syntactic structure benchmark designed for evaluating LLMs on fine-grained grammatical reasoning in a morphologically rich language.
Current NLP datasets in Hugging Face focus heavily on surface-level semantic tasks or English-centric benchmarks. There is a clear gap in evaluating deep syntactic understanding, especially for languages like Turkish where:
Embedded clause structures are highly productive
Nominalization (-DIK, -MA, -ACAK) plays a central syntactic role
Pro-drop subjects require contextual recovery
Cross-clause dependency relations are essential for meaning
Interrogative constructions involve structural reordering and embedding
This proposal introduces a DIK-style syntactic annotation benchmark dataset, where each sentence is represented in a structured JSON format capturing:
main clause decomposition
embedded clause hierarchy
verb morphology (lemma, tense, mood, polarity)
syntactic roles (subject, object, complement)
clause function mapping
case marking and nominalization tracking
A pilot dataset of 20 fully annotated sentences has already been completed with a consistent schema and manual validation. The design is fully scalable and intended for expansion to 100–300 sentences in Phase 1.
Motivation
I am always frustrated by the lack of evaluation benchmarks that go beyond surface-level semantic correctness and actually test whether LLMs understand deep syntactic structure, especially for underrepresented morphologically rich languages like Turkish.
Existing benchmarks do not adequately evaluate:
embedded clause interpretation
nominalization-driven structure (-DIK, -MA)
pro-drop subject recovery
syntactic role consistency across clauses
dependency tracking in complex sentences
cultural and discourse-level nuances embedded in language use (e.g., pragmatic meaning, register shifts, and culturally grounded expressions in Turkish)
This limits the ability to properly measure LLM performance in languages where meaning is encoded not only in morphology and clause structure, but also in cultural context, discourse conventions, and pragmatic interpretation rather than word order alone.
This dataset aims to fill that gap by providing a structured, machine-evaluable syntactic benchmark suitable for integration into evaluation frameworks such as Hugging Face evaluate and other LLM benchmarking pipelines.****
Your contribution
I am fully prepared to develop this dataset further and contribute a complete pull request, including:
expanding the dataset to 100–300 high-quality annotated sentences
ensuring strict schema consistency across all samples
providing full JSON dataset files compatible with Hugging Face Datasets
creating a reproducible evaluation script for LLM structural scoring
documenting dataset structure and annotation guidelines
If needed, I can also help adapt the dataset into a format compatible with existing evaluation frameworks and contribute ongoing improvements after initial review.
Feature request
A structured dataset proposal introducing a Turkish syntactic structure benchmark designed for evaluating LLMs on fine-grained grammatical reasoning in a morphologically rich language.
Current NLP datasets in Hugging Face focus heavily on surface-level semantic tasks or English-centric benchmarks. There is a clear gap in evaluating deep syntactic understanding, especially for languages like Turkish where:
Embedded clause structures are highly productive
Nominalization (-DIK, -MA, -ACAK) plays a central syntactic role
Pro-drop subjects require contextual recovery
Cross-clause dependency relations are essential for meaning
Interrogative constructions involve structural reordering and embedding
This proposal introduces a DIK-style syntactic annotation benchmark dataset, where each sentence is represented in a structured JSON format capturing:
main clause decomposition
embedded clause hierarchy
verb morphology (lemma, tense, mood, polarity)
syntactic roles (subject, object, complement)
clause function mapping
case marking and nominalization tracking
A pilot dataset of 20 fully annotated sentences has already been completed with a consistent schema and manual validation. The design is fully scalable and intended for expansion to 100–300 sentences in Phase 1.
Motivation
I am always frustrated by the lack of evaluation benchmarks that go beyond surface-level semantic correctness and actually test whether LLMs understand deep syntactic structure, especially for underrepresented morphologically rich languages like Turkish.
Existing benchmarks do not adequately evaluate:
embedded clause interpretation
nominalization-driven structure (-DIK, -MA)
pro-drop subject recovery
syntactic role consistency across clauses
dependency tracking in complex sentences
cultural and discourse-level nuances embedded in language use (e.g., pragmatic meaning, register shifts, and culturally grounded expressions in Turkish)
This limits the ability to properly measure LLM performance in languages where meaning is encoded not only in morphology and clause structure, but also in cultural context, discourse conventions, and pragmatic interpretation rather than word order alone.
This dataset aims to fill that gap by providing a structured, machine-evaluable syntactic benchmark suitable for integration into evaluation frameworks such as Hugging Face evaluate and other LLM benchmarking pipelines.****
Your contribution
I am fully prepared to develop this dataset further and contribute a complete pull request, including:
expanding the dataset to 100–300 high-quality annotated sentences
ensuring strict schema consistency across all samples
providing full JSON dataset files compatible with Hugging Face Datasets
creating a reproducible evaluation script for LLM structural scoring
documenting dataset structure and annotation guidelines
If needed, I can also help adapt the dataset into a format compatible with existing evaluation frameworks and contribute ongoing improvements after initial review.