Ibis for Pandas users

[kesmit13]

You have a section in the documentation for SQL users coming to Ibis, but I think it would be helpful to have one for Pandas users as well. It took me a while to figure out how to do some simpler things in Ibis like using mutate to add a column, the fact that there is no shape or dtypes attributes, just the general idea that there is no way to do an in-place operation (everything returns a new expression). Most are little things, but they aren't always obvious to new users.

[gforsyth]

I think this is a great idea @kesmit13 ! If you have any other things on your list of pain points / things that you expected and can document them here that would be a huge help for us.

[kesmit13]

I can make a list of things. I do think it would be simpler to just add things like dtypes, shape, unique, etc. rather than document why they aren't there. The implementations are quite simple. Is there a philosophical reason for not adding those types of things?

[kesmit13]

It's not necessarily used to perform more operations on (except maybe count), but I sometimes just need to get the distinct values of a column to populate a form, or to do an in operation on the client. The code above isn't quite the same as distinct on a column since calling execute returns a DataFrame rather than a Series. Although, you could add a .a to the end to get that.

[jcmkk3]

You are right. They do not return the same objects. That's why I was wondering how you were using it after. I see how it could be convenient in some contexts, but I agree with @cpcloud that it doesn't compose very in a ColumnExpr. I feel the same way about value_counts and topk. value_counts at least returns a Table instead of the weird TopK object.

[cpcloud]

We are planning to address the topk object problem, but not entirely sure what the best way forward is yet. Topk can be used alone and can also be used as a filter, which effectively IIRC adds a semi-join to the "topk-alone" variety, to show the rows from the original table that match the top K keys.

[cpcloud]

Let's discuss that elsewhere though to avoid hijacking this thread :)

[kesmit13]

I'd be interested in that topic as well. I wrote a version of pandas' describe method that uses existing Ibis operations including topk, but it did complicate things.

[cpcloud]

It seems like we can resolve this discussion since @kesmit13 contributed a wonderful Ibis for Pandas Users notebook!