`convertTo()` requires values for computed properties

[2x2xplz]

When a class contains computed properties, convertTo() still expects values for those properties to be defined:

@DataSchema
data class Foo (val first : Int, val second : Int) {
    val total : Int get() = first.plus(second)
    val highest : Int get() = listOf<Int>(first, second).max()
}

val df : DataFrame<Foo> = dataFrameOf("first", "second") (
    1, 2,
    25, 75,
    500, 499
).convertTo<Foo>()

java.lang.IllegalArgumentException: The following columns were not found in DataFrame: [highest, total], and their type was not nullable. Use fill to initialize these columns

This affects post-processing after .readCSV(), making it tricky to properly create/convert to a DataFrame reflecting known classes. Instead it's much easier to read the CSV externally, convert rows to collection of classes, then use .toDataFrame().

Trying to use fill results in Unresolved Reference and Type Mismatch (it's certainly possible I have the wrong syntax):

val df : DataFrame<Foo> = dataFrameOf("first", "second") (
    1, 2,
    25, 75,
    500, 499
).convertTo<Foo> {
    fill { total }.with { first.plus(second) }
}

[Jolanrensen]

I think this is a bug. total and highest should not be part of the data schema afaik.

@koperagen is that correct?

[koperagen]

Hm, that's interesting. Yes, i think they should not be a part of the data schema.
But what are you trying to achieve? Do I understand it right that you want total and heighest to be present in df after convertTo? I'm not sure if it's possible to do automatically, because we need a way to tell it's a computed property using kotlin reflection.
If i reprhase the task, you want to read csv with some set of initial columns, then add couple of computed columns using type safe syntax and in the end they all should be accessible with extension properties API. Is that so? If yes, I'll think of a ways to do it conveniently

[2x2xplz]

Thanks for the quick replies. No, I do not expect the computed properties to immediately be available as columns.

I think what I was expecting, and this may be more complicated, was for there to be a simple way to create an instance of Foo for each row in the dataframe, from the 2 columns, and then for the resulting instances to be able to produce total or highest.

In other words, because the df is a DataFrame<Foo>, there might be a function df.add("FooColumn") { createInstance() } that builds a column of Foo objects, and those could provide the total or the highest. I understand that some schemas come from interfaces or abstract classes that cannot be instantiated, so maybe this wouldn't work. It also might be really inefficient to try to build an object for every row, so perhaps a bad idea. But in any case the presence of computed properties should not prevent the data frame from being created, like they do currently.

Thanks again. Dataframe is really great, I use it almost every day.

[koperagen]

Ok, i think we can solve the original problem and ignore computed properties in convertTo for data classes. I think in other places we only consider constructor properties to be a part of the data schema, so it would make behavior more consistent. Need to check it.

But actually i just realized that you might be looking for toList or toListOf family of function.
Have a look here https://kotlin.github.io/dataframe/collectionsinterop.html and here https://kotlin.github.io/dataframe/tolist.html

[2x2xplz]

ignore computed properties in convertTo -- awesome!

you might be looking for toList or toListOf -- I'm familiar with these, and they are very useful. But going directly to standard Kotlin collections means you lose the dataframe API and its features. This is going to get a bit long, feel free to set it aside, this is not a super high-priority item:

%use dataframe(0.9.1)
import java.time.LocalDate

@DataSchema
data class Store (
    val storeID: Int,
    val storeRegion: String,
    val storeOpened: LocalDate,
    val revenue: Long,
    val expenses: Long
) {
    val profit: Long get() = revenue.minus(expenses)
}

val df = dataFrameOf("storeID", "storeRegion", "storeOpened", "revenue", "expenses") (
    23, "Chicago", LocalDate.parse("2018-05-21"), 500, 400,
    71, "Chicago", LocalDate.parse("2019-07-31"), 600, 480,
    54, "Los Angeles", LocalDate.parse("2017-03-06"), 350, 300,
    88, "Miami", LocalDate.parse("2018-09-08"), 550, 540,
    15, "Los Angeles", LocalDate.parse("2017-03-06"), 350, 300,
    47, "Miami", LocalDate.parse("2022-11-19"), 60, 140,
    75, "Chicago", LocalDate.parse("2020-08-12"), 450, 500,
    63, "Los Angeles", LocalDate.parse("2020-12-01"), 350, 300
)

df.describe()

gets a df with 5 columns, however profit is unavailable

data class StoreWrapper(val store: Store)

val dfWrapped = dataFrameOf("store") (
    Store(23, "Chicago", LocalDate.parse("2018-05-21"), 500, 400),
    Store(71, "Chicago", LocalDate.parse("2019-07-31"), 600, 480),
    Store(54, "Los Angeles", LocalDate.parse("2017-03-06"), 350, 300),
    Store(88, "Miami", LocalDate.parse("2018-09-08"), 550, 540),
    Store(15, "Los Angeles", LocalDate.parse("2017-03-06"), 350, 300),
    Store(47, "Miami", LocalDate.parse("2022-11-19"), 60, 140),
    Store(75, "Chicago", LocalDate.parse("2020-08-12"), 450, 500),
    Store(63, "Los Angeles", LocalDate.parse("2020-12-01"), 350, 300)
)

dfWrapped.describe()

now just 1 column, but that column contains a full Store object, so profit is available

the initial df can calculate revenue by region:
df.groupBy { storeRegion }.sum { revenue }

but dfWrapped can calculate revenue AND profit by region:

dfWrapped.groupBy { store.map { it.storeRegion } }.aggregate { 
    sum { store.map { it.revenue }} into "revenue" 
    sum { store.map { it.profit }} into "profit" 
}

plus you still can use the rest of the API like drop (for example, drop stores open less than 1 year):

dfWrapped
    .drop { store.storeOpened.isAfter(LocalDate.now().minusYears(1)) }
    // inside groupBy, aggregates .map is required
    .groupBy { store.map { it.storeRegion } }
    .aggregate { 
        sum { store.map { it.revenue }} into "revenue" 
        sum { store.map { it.profit }} into "profit" 
}

and even access calculated properties as top-level (without .map) in drop/filter/etc (for example, drill down on unprofitable stores):

// note that .profit is calculated but still available top-level (no .map needed)
dfWrapped.filter { store.profit < 0 }.add {
    // this would be nice to do easily with something similar to 'explode'
    "id" from store.map { it.storeID }
    "region" from store.map { it.storeRegion }
    "profit" from store.map { it.profit }
}

or something a bit more complex...

dfWrapped.add {
    "region" from store.map { it.storeRegion }
    "year_opened" from store.map { it.storeOpened.year }
}
.groupBy { "year_opened"<Int>() }
    .sortBy { "year_opened"<Int>() }
    .pivot { "region"<String>() } 
        .values { store }
        .flatten()
        // convert single values to a list for consistency
        .convert { colsOf<Store?>() }.with { if (it == null) listOf() else listOf(it) }
        // display table of [storeID : profit] by year opened and region
        .convert { colsOf<List<Store>>() }.with { it.map { s -> Pair(s.storeID, s.profit) } }

so that's a simple use case for not converting directly to a List (yes, lists have .filter and .filterNot = .drop but a lot of the dataframe functions aren't available), and instead, having the objects directly within the df.

Thanks very much for reading. Hope you understand some of the use cases. This is why dataframe is so terrific, once it has the data, dataframe's ability to manipulate it really becomes endless.