How should pandas-stubs balance typing precision, overload complexity, and runtime behavior for MultiIndex indexing?

[Prem-gosavi45]

Hi pandas maintainers and contributors!

While working on pandas-stubs issue #1018 (MultiIndex column selection with .loc), I encountered an interesting challenge around overload resolution.

A seemingly simple indexing expression can have different return types depending on whether the MultiIndex key is fully specified, partially specified, or contains slices.

For example:

df.loc[:, ("A", "a")]          # Series
df.loc[:, pd.IndexSlice[:, "a"]]  # DataFrame
df.loc[:, pd.IndexSlice["A", :]]  # DataFrame

This raises a broader design question:

How should we model pandas indexing APIs in static typing when the return type depends on the structure and completeness of an index key, especially when Python's type system cannot easily express exclusions between overlapping tuple types?

I'm particularly interested in the following:

1. Should pandas-stubs prioritize precise return-type inference for common indexing patterns, even if that requires more specialized overloads?
2. How should we handle overloads that are semantically distinct at runtime but structurally overlap in Pyright or mypy?
3. Would introducing more specific type aliases for full MultiIndex keys and partial/slice-based keys be a maintainable long-term solution?
4. How do maintainers decide when an overload is worth its complexity versus returning a broader union?
5. Are there plans to establish common typing design principles for complex pandas APIs such as .loc, .iloc, MultiIndex, and GroupBy?

I'd love to understand the maintainers' perspective on the long-term direction of pandas-stubs typing.

Thanks for maintaining this amazing project!

[Gianni300x]

A runtime edge case to include when evaluating an overload for a fully specified key: duplicate column labels can change the result. On pandas 3.0.3 I checked:

import pandas as pd

cols = pd.MultiIndex.from_tuples([("A", "a"), ("A", "a")])
df = pd.DataFrame([[1, 2]], columns=cols)
print(type(df.loc[:, ("A", "a")]).__name__)  # DataFrame

So even a complete tuple alone does not guarantee a Series unless uniqueness is also assumed. The duplicate-label guide discusses how duplicates affect dimensionality. A runtime/type-checker test matrix covering unique and duplicate full keys, partial keys, and slices could make the precision trade-off explicit. I can’t speak for the maintainers’ policy or planned direction.

[Prem-gosavi45]

Thanks, Gianni! That's an important edge case.

You're right that a fully specified MultiIndex key does not necessarily imply a Series, since duplicate column labels can cause .loc to return a DataFrame.

This suggests that key completeness alone is not sufficient to determine the return type. Since column-label uniqueness is generally a runtime property that the type checker cannot guarantee, perhaps a broader return type (Series | DataFrame) is safer for cases where uniqueness cannot be established.

I think a useful next step would be to add a runtime and type-checker test matrix covering:

• Unique full keys
• Duplicate full keys
• Partial keys
• Slice-based keys

I'd be interested to hear whether maintainers prefer this conservative approach or see another maintainable way to model these cases in pandas-stubs.