Account import & export #2356
Account import & export #2356
Conversation
monorkin
force-pushed
the
import-export
branch
2 times, most recently
from
ffabdbd to
9e37050
Compare
monorkin
force-pushed
the
import-export
branch
from
9e37050 to
fec5819
Compare
| @@ -0,0 +1,48 @@ | |||
| class Account::DataTransfer::AccessRecordSet < Account::DataTransfer::RecordSet | |||
There was a problem hiding this comment.
I quite like the structure of having the RecordSet types, but I notice that most of them are very similar. The bulk of them boil down to "get all the records by account ID, store into a path based on the name". Which, I think, is what we were originally hoping the whole import/export process could be.
I realise that as we get deeper into it, some of the models might deviate from that a bit. But maybe we could try to really push back on the number of deviations where possible, and then have hooks for anywhere we do need to handle something in a different way. That way the bulk of the process could use the same pattern without having to restart parts of it on each model.
So if the places where it strays from convention are few and minor, do you think we could push some of this up into the base class, and then all the "normal" tables could avoid needing to implement most of the methods here?
Or, alternatively, could we avoid deriving individual classes at all, expect in the cases where a table has some special behaviour? So if a model needs a special join, or some pre/post processing we can specify just those parts?
| @@ -0,0 +1,48 @@ | |||
| class Account::DataTransfer::AccessRecordSet < Account::DataTransfer::RecordSet | |||
| ATTRIBUTES = %w[ | |||
There was a problem hiding this comment.
Is the set of attributes in the transfer different than the table's columns? If it's not, could we determine these from the schema automatically?
There was a problem hiding this comment.
Also, what's your thoughts about how to handle situations where data has been migrated and the columns differ in some way? I imagine there would be some cases where this would make in import impossible, but there may be others were it's OK (added nullable column or usable defaults, etc).
| def import_batch(files) | ||
| batch_data = files.map do |file| | ||
| data = load(file) | ||
| data["body"] = convert_gids_to_sgids(data["body"]) |
There was a problem hiding this comment.
If we did try to collapse the record set classes a bit, we could handle this in a sort of before_insert step so that we don't need to supply a full implementation of import_batch to do it.
I also wonder if the export half of this is necessary (converting sgids to gids) or could we implicitly trust the signing on the way in while converting to new sgids? Effectively re-signing from an unknown key to the valid one. Since the transfer is effectively trusting that aspect of the input data anyway, it could be one less step. (Not sure if it makes much difference in practice or not though.)
No description provided.