Discussion: DataFusion Improvement Proposal (DIPs) Process?

[alamb]

Is your feature request related to a problem or challenge?

@viirya says in #16800 (comment):

Sometimes, I feel that some important proposals in DataFusion lack sufficient context, or that the relevant context is scattered across various issues and PR comments. This makes it difficult to fully understand the proposals or to trace their motivations and evaluate their soundness. As a result, we sometimes see large PRs — hundreds or even thousands of lines — that are based on these proposals, making the review process even more challenging. Only the author or those who were involved in the initial discussions seem to be in a position to effectively review them.

For example, Spark has the SPIP (Spark Project Improvement Proposal) mechanism, where contributors submit formal documents for review when proposing significant changes. These documents typically consolidate the technical details, motivation, and background of the proposal into a single place. This approach helps the community better understand and participate in discussions around major changes.

I wonder if it would be beneficial for DataFusion to adopt a similar lightweight proposal process for major design changes — something that allows ideas and context to be collected and reviewed before implementation begins. It could help improve transparency, facilitate broader community involvement, and make the review process more accessible.

If the full SPIP process — including voting and formal approval — feels too heavy or unnecessary for our context, perhaps we could at least establish a lightweight template for major change proposals. This template could include sections for motivation, background, technical details, and other relevant context. Having a consistent format would make it easier for the community to follow and engage with significant design discussions.

My opinions:

1. Finding the outstanding proposals and discussions is difficult. They are all public but there is lots of them going on
2. The context for proposals is often scattered across issues and PRs
3. It is hard to know when "enough" communication has been done for a proposal to move forward and when it needs more work
4. Improving the communication around major changes is becoming more important as the project grows and we have more users and contributors

For example, there are several recent discussions that could benefit from this more formal proposal process, including but not limited to the discussion itself above

• Plan to replace SchemaAdapter with PhysicalExprAdapter #16800 itself from @adriangb
• Support multiple ordered array_agg aggregations #16625 from @findepi
• Document the SQL dialect DataFusion attempts to follow #13704 in general , and Document the SQL dialect DataFusion attempts to follow #13704 (comment) recently with @berkasynnada
• [DISCUSSION] Memory accounting model discussion #16841 from @gabotechs,
• [EPIC] ScalarUDFImpl::equals default implementation is error-prone #16677 with @findepi

Describe the solution you'd like

Some sort of "process" that

1. Makes it easy to find outstanding community improvement proposals
2. Makes it easy to know the steps to create a new improvement proposal
3. Is documented

Describe alternatives you've considered

Here is a strawman (for discussion) proposal:

1. Add a new tag in the DataFusion repo ("DIP - DataFusion Improvement Proposal")
2. Add a new ISSUE_TEMPLATE for proposals issues based on the SPIP one and current DataFusion issue template
3. Add a section to the site documentation describing the process

I personally worry that DataFusion is not at a point I where formal voting / formal approval would add a lot of value, but I do think formalizing the proposal format and making them easier to find would be beneficial.

I propose starting with more formalization around the communication of proposals and we can add more explicitly approval / consensus standards if and when they become necessary.

Additional context

Here is the documentation for the spark process: The https://spark.apache.org/improvement-proposals.html

I looked through the list of SPIPs in Spark and the few I looked at didn't have huge amounts of discussion. They often linked to a google doc with more details.

[Adez017]

Is your feature request related to a problem or challenge?

@viirya says in #16800 (comment):

Sometimes, I feel that some important proposals in DataFusion lack sufficient context, or that the relevant context is scattered across various issues and PR comments. This makes it difficult to fully understand the proposals or to trace their motivations and evaluate their soundness. As a result, we sometimes see large PRs — hundreds or even thousands of lines — that are based on these proposals, making the review process even more challenging. Only the author or those who were involved in the initial discussions seem to be in a position to effectively review them.
For example, Spark has the SPIP (Spark Project Improvement Proposal) mechanism, where contributors submit formal documents for review when proposing significant changes. These documents typically consolidate the technical details, motivation, and background of the proposal into a single place. This approach helps the community better understand and participate in discussions around major changes.
I wonder if it would be beneficial for DataFusion to adopt a similar lightweight proposal process for major design changes — something that allows ideas and context to be collected and reviewed before implementation begins. It could help improve transparency, facilitate broader community involvement, and make the review process more accessible.
If the full SPIP process — including voting and formal approval — feels too heavy or unnecessary for our context, perhaps we could at least establish a lightweight template for major change proposals. This template could include sections for motivation, background, technical details, and other relevant context. Having a consistent format would make it easier for the community to follow and engage with significant design discussions.

My opinions:

1. Finding the outstanding proposals and discussions is difficult. They are all public but there is lots of them going on
2. The context for proposals is often scattered across issues and PRs
3. It is hard to know when "enough" communication has been done for a proposal to move forward and when it needs more work
4. Improving the communication around major changes is becoming more important as the project grows and we have more users and contributors

For example, there are several recent discussions that could benefit from this more formal proposal process, including but not limited to the discussion itself above

• Plan to replace SchemaAdapter with PhysicalExprAdapter #16800 itself from @adriangb
• Support multiple ordered array_agg aggregations #16625 from @findepi
• Document the SQL dialect DataFusion attempts to follow #13704 in general , and Document the SQL dialect DataFusion attempts to follow #13704 (comment) recently with @berkasynnada
• [DISCUSSION] Memory accounting model discussion #16841 from @gabotechs,
• ScalarUDFImpl::equals default implementation is error-prone #16677 with @findepi

Describe the solution you'd like

Some sort of "process" that

1. Makes it easy to find outstanding community improvement proposals
2. Makes it easy to know the steps to create a new improvement proposal
3. Is documented

Describe alternatives you've considered

Here is a strawman (for discussion) proposal:

1. Add a new tag in the DataFusion repo ("DIP - DataFusion Improvement Proposal")
2. Add a new ISSUE_TEMPLATE for proposals issues based on the SPIP one and current DataFusion issue template
3. Add a section to the site documentation describing the process

I personally worry that DataFusion is not at a point I where formal voting / formal approval would add a lot of value, but I do think formalizing the proposal format and making them easier to find would be beneficial.

I propose starting with more formalization around the communication of proposals and we can add more explicitly approval / consensus standards if and when they become necessary.

Additional context

Here is the documentation for the spark process: The https://spark.apache.org/improvement-proposals.html

I looked through the list of SPIPs in Spark and the few I looked at didn't have huge amounts of discussion. They often linked to a google doc with more details.

Great ! @alamb i think surely it would be very helpful for the future context and to keep the proposals on point

[viirya]

I looked through the list of SPIPs in Spark and the few I looked at didn't have huge amounts of discussion. They often linked to a google doc with more details.

Most discussions happened in the google docs, I think. Especially such docs allow to comment/discuss around specified sentences, API proposal, etc.

[phillipleblanc]

I agree that formal voting/approval doesn't make sense yet.

Also having a structured way to propose "larger" changes that incorporates all relevant context for reviewers is definitely a good idea. I think the EPICs that you had are basically this - a focusing a point for a large change that has relevant context + links to the other sub-issues/PRs. (example: #9637)

[xudong963]

Also having a structured way to propose "larger" changes that incorporates all relevant context for reviewers is definitely a good idea

+1, this is a good idea. We all prefer concentrated information rather than searching everywhere for things to help us understand a huge PR.

[alamb]

I will try and summarize the contents of this ticket into a PR to the documentation site later this week and will post an update here

[findepi]

Describe the solution you'd like

Some sort of "process" that

1. Makes it easy to find outstanding community improvement proposals
2. Makes it easy to know the steps to create a new improvement proposal
3. Is documented

Those are noble goals, but i would also add the following

• it's need to be "executable" (each step of the process needs to be feasible)
• it's need to be decisive (reaching a decision)
• it's need to be finite (running to an end in bounded time)

last but not least

• it needs to involve relevant stakeholders (subject matter experts and heavy users of given functional area)
  • of course everyone on the Internet should be able to provide input, but some people are simply more likely to be interested in a topic

Finiteness is expectedly hard to achieve in a distributed community where everyone follows their own timelines, but this should not come at the cost of missing involving relevant people, i.e. people which input can be reasonably expected to be useful.

[alamb]

Those are noble goals, but i would also add the following

• it's need to be "executable" (each step of the process needs to be feasible)
• it's need to be decisive (reaching a decision)
• it's need to be finite (running to an end in bounded time)

When I ran an engineering team at a company, I would 100% agree with all these items and insisted on them myself.

However, after several years in an open source project like Arrow and DataFusion I am not sure they are a great match for this community.

I have seen several projects that start as vague statements of goal and then various people provide parts of the implementation / writeups / tickets, etc and it ends up being completed (utf8view being one, porting all functions to be UDFs is another, the most recent Parquet metadata cache from @nuno-faria is shaping up to be similar as well)

The other question is who decides if a particular proposal meets those criteria 🤔 Maybe if we say they are best practice that would be enough

• it needs to involve relevant stakeholders (subject matter experts and heavy users of given functional area)

  • of course everyone on the Internet should be able to provide input, but some people are simply more likely to be interested in a topic

Finiteness is expectedly hard to achieve in a distributed community where everyone follows their own timelines, but this should not come at the cost of missing involving relevant people, i.e. people which input can be reasonably expected to be useful.

In general I agree -- I think the key challenges are:

1. Identifying who the key people are (some of the core contributors probably know, but figuring this out is a high barrier)
2. Ensuring stakeholders have enough time to provide input if they want, but do not hold up the process indefinitely due to lack of bandwidth

Since all communications are public, I think any stakeholder who is interested in a feature already has the opportunity to discover and participate. However, I think we could do a better job making it easier for relevant stakeholders to find proposals

[findepi]

I have seen several projects that start as vague statements of goal and then various people provide parts of the implementation / writeups / tickets, etc and it ends up being completed (utf8view being one, porting all functions to be UDFs is another, the most recent Parquet metadata cache from @nuno-faria is shaping up to be similar as well)

Those are examples of implementation projects. These can span unforeseeable amount of time, depending on interested of contributors.

I think DIP is a different category. DIP process should run to an end in bounded time, even if this means "timeout => rejection".
There is no point in starting a DIP in 2019 and all of the sudden deciding on it in 2026.

Since all communications are public, I think any stakeholder who is interested in a feature already has the opportunity to discover and participate

That works provided that all potential stakeholder are listening on a communication channel where DIPs are announced.
What would be such a channel?

[comphead]

Copying from ASF DF Slack

Also pretty nice thing is how to define proposals in clean and understandable way which described here https://github.com/kubeflow/community/tree/master/proposals#kep-format-to-propose-and-document-enhancements-to-kubeflow
The example of recent change is https://docs.google.com/document/d/1xj3K6bOT4f0EPiC4zwr2OsbRkROjIA8u_TBRZRVxrHI/edit?tab=t.0#heading=h.bco7r7uaguxy

This approach definitely takes more time for writing up the change, however it would easier for community to digest the change and make fundamental review on design level rather than going through hundreds/thousands lines of code.

Another nice benefit this design would serve as a solid draft for public article

[alamb]

That works provided that all potential stakeholder are listening on a communication channel where DIPs are announced.
What would be such a channel?

I do not know of anything yet except for github alerts, which I agree are hard to navigate in. In my mind, clearly defining the channel where any potential stakeholder should look would be one of the key improvements of a more formal DIP process

[alamb]

I made a small PR to consolidate the documentation we currently have about proposing new features:

• Docs: Consolidate feature proposal content into roadmap #17156

I hope that if that is merged we will then at least have a place to put any new process / writeups