Reduce bookmark rewind window

[amyfain]

This reduces the bookmark rewind. When it is set at 24 hours and we run the tap in 1 hour intervals, we see the same data pass to us 24 times as it's set right now.

[luandy64]

I am reluctant to make this "default" change. If we need to do this, I think the best course of action is to make this a sort of config change where users can pass in their desired time and the tap adjusts accordingly.

[luandy64]

@cmcarthur Do you know why this was set to 1 day?

[luandy64]

@amyfain Would it be possible to lower the replication frequency- like run the tap on 3 hour intervals- to combat the duplicate data issue?

[amyfain]

@luandy64 while that's possible, we have our replication frequency set to 1 hour because we want the data as quickly as possible. I'm not sure how to make this a config change, but I think that's a great idea!

[luandy64]

@amyfain We can iterate on this idea- and I invite you to check out the Singer Slack if you aren't there already- but I think it could go like:

1. Change get_last_record_value_for_table() to have a third argument that defaults to None
2. In get_last_record_value_for_table() check if the config is present.
3. If so, pass the value into the datetime.timedelta()
4. If not, use date_obj = date_obj - datetime.timedelta(days=1)

There might be a better way of doing this, but I'll sleep on that 👍