Skip to content

[BACK-2638] Move Summary buckets to collection and unify datastructure #763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 46 commits into from
May 12, 2025
Merged

[BACK-2638] Move Summary buckets to collection and unify datastructure #763

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
e2d477d
summary stats rewrite
Roukoswarf Jan 13, 2025
7e11cd5
comment out failing tests for now
Roukoswarf Jan 13, 2025
332c1ef
fullstack tests succeed.
Roukoswarf Jan 15, 2025
56bf475
formatting
Roukoswarf Jan 15, 2025
132dae1
add buckets index
Roukoswarf Jan 16, 2025
b712695
handle a potential nil pointer
Roukoswarf Jan 16, 2025
cb81925
Code review ideas
toddkazakov Jan 23, 2025
5050391
addressing review comments part 1
Roukoswarf Jan 29, 2025
88155fc
review comments part 2
Roukoswarf Feb 3, 2025
1ced6f7
no more "stats"
Roukoswarf Feb 3, 2025
46dd3bf
no more "stats" in continuous
Roukoswarf Feb 4, 2025
1145c66
no more "observations"
Roukoswarf Feb 4, 2025
70b5dd1
more naming consistency
Roukoswarf Feb 4, 2025
d70ddae
review comments part 3
Roukoswarf Feb 6, 2025
fc0166c
make summaryType naming consistent
Roukoswarf Feb 6, 2025
eddd5c6
remove omitempty from ranges, they do nothing. fix bucket initializat…
Roukoswarf Feb 7, 2025
eeac9a7
formatting
Roukoswarf Feb 7, 2025
636c7ab
update unit tests
Roukoswarf Feb 10, 2025
9a2b87a
unit tests work now
Roukoswarf Feb 21, 2025
5313b7d
remove missed TODO
Roukoswarf Feb 21, 2025
2d1a79d
update schema version of summaries
Roukoswarf Feb 24, 2025
24e0424
update some naming to match json as CDC will be complicated by it
Roukoswarf Feb 26, 2025
2f99c8b
integrate the SetOutdated race condition patch, as that branch was ba…
Roukoswarf Feb 26, 2025
6eba16d
replace InsertMany with bulk replace to handle overlapping buckets or…
Roukoswarf Mar 11, 2025
3566b69
dont attempt to write 0 buckets
Roukoswarf Mar 11, 2025
b15ad6b
fix transaction ctx, gt -> gte in some cases
Roukoswarf Mar 14, 2025
50e7f8e
re-add FirstData lower limit
Roukoswarf Mar 14, 2025
03ea2c2
drop buckets on future schema migrations
Roukoswarf Mar 14, 2025
70469b7
add bucket cleanup to nil summary handling just in case
Roukoswarf Mar 14, 2025
69446ad
add custom marshal/unmarshal to cgmperiods/bgmperiods to flatten to m…
Roukoswarf Mar 14, 2025
eb57230
marshal nil correctly
Roukoswarf Mar 14, 2025
422b6ac
formatting
Roukoswarf Mar 14, 2025
1457a1e
add debug to summary status
Roukoswarf Mar 15, 2025
7f427a7
deref status pointer
Roukoswarf Mar 15, 2025
5fd504d
fix editing/reuploading final data point resulting in improper bucket…
Roukoswarf Mar 15, 2025
28471ae
tighten conditions for lower bound of data range query
Roukoswarf Mar 16, 2025
4306e02
move summary package to top level, fix some task bugs/typos
Roukoswarf Mar 17, 2025
7cbb722
Merge branch 'alex/version-updates' into alex/back-2638-buckets-colle…
Roukoswarf Mar 17, 2025
6d067b1
fix one missed types -> types2 rename
Roukoswarf Mar 17, 2025
d50fe94
formatting
Roukoswarf Mar 17, 2025
ce31af6
skip average-related calculations if total glucose is 0 to prevent NaN
Roukoswarf Mar 20, 2025
a7fbeeb
correct range percentage to be of total minutes, not wall minutes
Roukoswarf Apr 1, 2025
ea70d33
replace OutdatedSince nil check in GetMigratableUserIDs, add GetMigra…
Roukoswarf Apr 14, 2025
2fbc8e6
comment update
Roukoswarf Apr 14, 2025
cdec4b1
Modify summary indexes to allow for sorted migratiuon queries
Roukoswarf Apr 16, 2025
d8628bb
fix accidental discarded summary replace err
Roukoswarf Apr 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
64 changes: 20 additions & 44 deletions data/client/client.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,16 @@ import (
"context"
"encoding/json"
"net/http"
"time"

"github.com/tidepool-org/platform/data/summary/types"
"github.com/tidepool-org/platform/log"

"github.com/tidepool-org/platform/data"
"github.com/tidepool-org/platform/errors"
"github.com/tidepool-org/platform/log"
"github.com/tidepool-org/platform/page"
"github.com/tidepool-org/platform/platform"
"github.com/tidepool-org/platform/request"
"github.com/tidepool-org/platform/service"
structureValidator "github.com/tidepool-org/platform/structure/validator"
)

const (
// ExtendedTimeout is used by requests that we expect to take longer than
// usual.
ExtendedTimeout = 15 * time.Minute
"github.com/tidepool-org/platform/summary/types"
)

// TODO: Move interface to data package once upload dependency broken
Expand All @@ -31,18 +23,15 @@ type Client interface {
data.DataSetAccessor

CreateDataSetsData(ctx context.Context, dataSetID string, datumArray []data.Datum) error

DestroyDataForUserByID(ctx context.Context, userID string) error

GetCGMSummary(ctx context.Context, id string) (*types.Summary[*types.CGMStats, types.CGMStats], error)
GetBGMSummary(ctx context.Context, id string) (*types.Summary[*types.BGMStats, types.BGMStats], error)
GetContinuousSummary(ctx context.Context, id string) (*types.Summary[*types.ContinuousStats, types.ContinuousStats], error)
UpdateCGMSummary(ctx context.Context, id string) (*types.Summary[*types.CGMStats, types.CGMStats], error)
UpdateBGMSummary(ctx context.Context, id string) (*types.Summary[*types.BGMStats, types.BGMStats], error)
UpdateContinuousSummary(ctx context.Context, id string) (*types.Summary[*types.ContinuousStats, types.ContinuousStats], error)
GetCGMSummary(ctx context.Context, id string) (*types.Summary[*types.CGMPeriods, *types.GlucoseBucket, types.CGMPeriods, types.GlucoseBucket], error)
GetBGMSummary(ctx context.Context, id string) (*types.Summary[*types.BGMPeriods, *types.GlucoseBucket, types.BGMPeriods, types.GlucoseBucket], error)
GetContinuousSummary(ctx context.Context, id string) (*types.Summary[*types.ContinuousPeriods, *types.ContinuousBucket, types.ContinuousPeriods, types.ContinuousBucket], error)
UpdateCGMSummary(ctx context.Context, id string) (*types.Summary[*types.CGMPeriods, *types.GlucoseBucket, types.CGMPeriods, types.GlucoseBucket], error)
UpdateBGMSummary(ctx context.Context, id string) (*types.Summary[*types.BGMPeriods, *types.GlucoseBucket, types.BGMPeriods, types.GlucoseBucket], error)
UpdateContinuousSummary(ctx context.Context, id string) (*types.Summary[*types.ContinuousPeriods, *types.ContinuousBucket, types.ContinuousPeriods, types.ContinuousBucket], error)
GetOutdatedUserIDs(ctx context.Context, t string, pagination *page.Pagination) (*types.OutdatedSummariesResponse, error)
GetMigratableUserIDs(ctx context.Context, t string, pagination *page.Pagination) ([]string, error)
BackfillSummaries(ctx context.Context, t string) (int, error)
}

type ClientImpl struct {
Expand Down Expand Up @@ -132,7 +121,7 @@ func (c *ClientImpl) GetDataSet(ctx context.Context, id string) (*data.DataSet,
return dataSet, nil
}

func (c *ClientImpl) GetCGMSummary(ctx context.Context, userId string) (*types.Summary[*types.CGMStats, types.CGMStats], error) {
func (c *ClientImpl) GetCGMSummary(ctx context.Context, userId string) (*types.Summary[*types.CGMPeriods, *types.GlucoseBucket, types.CGMPeriods, types.GlucoseBucket], error) {
if ctx == nil {
return nil, errors.New("context is missing")
}
Expand All @@ -141,7 +130,7 @@ func (c *ClientImpl) GetCGMSummary(ctx context.Context, userId string) (*types.S
}

url := c.client.ConstructURL("v1", "summaries", "cgm", userId)
summary := &types.Summary[*types.CGMStats, types.CGMStats]{}
summary := &types.Summary[*types.CGMPeriods, *types.GlucoseBucket, types.CGMPeriods, types.GlucoseBucket]{}
if err := c.client.RequestData(ctx, http.MethodGet, url, nil, nil, summary); err != nil {
if request.IsErrorResourceNotFound(err) {
return nil, nil
Expand All @@ -152,7 +141,7 @@ func (c *ClientImpl) GetCGMSummary(ctx context.Context, userId string) (*types.S
return summary, nil
}

func (c *ClientImpl) GetBGMSummary(ctx context.Context, userId string) (*types.Summary[*types.BGMStats, types.BGMStats], error) {
func (c *ClientImpl) GetBGMSummary(ctx context.Context, userId string) (*types.Summary[*types.BGMPeriods, *types.GlucoseBucket, types.BGMPeriods, types.GlucoseBucket], error) {
if ctx == nil {
return nil, errors.New("context is missing")
}
Expand All @@ -161,7 +150,7 @@ func (c *ClientImpl) GetBGMSummary(ctx context.Context, userId string) (*types.S
}

url := c.client.ConstructURL("v1", "summaries", "bgm", userId)
summary := &types.Summary[*types.BGMStats, types.BGMStats]{}
summary := &types.Summary[*types.BGMPeriods, *types.GlucoseBucket, types.BGMPeriods, types.GlucoseBucket]{}
if err := c.client.RequestData(ctx, http.MethodGet, url, nil, nil, summary); err != nil {
if request.IsErrorResourceNotFound(err) {
return nil, nil
Expand All @@ -172,7 +161,7 @@ func (c *ClientImpl) GetBGMSummary(ctx context.Context, userId string) (*types.S
return summary, nil
}

func (c *ClientImpl) GetContinuousSummary(ctx context.Context, userId string) (*types.Summary[*types.ContinuousStats, types.ContinuousStats], error) {
func (c *ClientImpl) GetContinuousSummary(ctx context.Context, userId string) (*types.Summary[*types.ContinuousPeriods, *types.ContinuousBucket, types.ContinuousPeriods, types.ContinuousBucket], error) {
if ctx == nil {
return nil, errors.New("context is missing")
}
Expand All @@ -181,7 +170,7 @@ func (c *ClientImpl) GetContinuousSummary(ctx context.Context, userId string) (*
}

url := c.client.ConstructURL("v1", "summaries", "continuous", userId)
summary := &types.Summary[*types.ContinuousStats, types.ContinuousStats]{}
summary := &types.Summary[*types.ContinuousPeriods, *types.ContinuousBucket, types.ContinuousPeriods, types.ContinuousBucket]{}
if err := c.client.RequestData(ctx, http.MethodGet, url, nil, nil, summary); err != nil {
if request.IsErrorResourceNotFound(err) {
return nil, nil
Expand All @@ -192,7 +181,7 @@ func (c *ClientImpl) GetContinuousSummary(ctx context.Context, userId string) (*
return summary, nil
}

func (c *ClientImpl) UpdateCGMSummary(ctx context.Context, userId string) (*types.Summary[*types.CGMStats, types.CGMStats], error) {
func (c *ClientImpl) UpdateCGMSummary(ctx context.Context, userId string) (*types.Summary[*types.CGMPeriods, *types.GlucoseBucket, types.CGMPeriods, types.GlucoseBucket], error) {
if ctx == nil {
return nil, errors.New("context is missing")
}
Expand All @@ -201,7 +190,7 @@ func (c *ClientImpl) UpdateCGMSummary(ctx context.Context, userId string) (*type
}

url := c.client.ConstructURL("v1", "summaries", "cgm", userId)
summary := &types.Summary[*types.CGMStats, types.CGMStats]{}
summary := &types.Summary[*types.CGMPeriods, *types.GlucoseBucket, types.CGMPeriods, types.GlucoseBucket]{}
if err := c.client.RequestData(ctx, http.MethodPost, url, nil, nil, summary); err != nil {
if request.IsErrorResourceNotFound(err) {
return nil, nil
Expand All @@ -212,7 +201,7 @@ func (c *ClientImpl) UpdateCGMSummary(ctx context.Context, userId string) (*type
return summary, nil
}

func (c *ClientImpl) UpdateBGMSummary(ctx context.Context, userId string) (*types.Summary[*types.BGMStats, types.BGMStats], error) {
func (c *ClientImpl) UpdateBGMSummary(ctx context.Context, userId string) (*types.Summary[*types.BGMPeriods, *types.GlucoseBucket, types.BGMPeriods, types.GlucoseBucket], error) {
if ctx == nil {
return nil, errors.New("context is missing")
}
Expand All @@ -221,7 +210,7 @@ func (c *ClientImpl) UpdateBGMSummary(ctx context.Context, userId string) (*type
}

url := c.client.ConstructURL("v1", "summaries", "bgm", userId)
summary := &types.Summary[*types.BGMStats, types.BGMStats]{}
summary := &types.Summary[*types.BGMPeriods, *types.GlucoseBucket, types.BGMPeriods, types.GlucoseBucket]{}
if err := c.client.RequestData(ctx, http.MethodPost, url, nil, nil, summary); err != nil {
if request.IsErrorResourceNotFound(err) {
return nil, nil
Expand All @@ -232,7 +221,7 @@ func (c *ClientImpl) UpdateBGMSummary(ctx context.Context, userId string) (*type
return summary, nil
}

func (c *ClientImpl) UpdateContinuousSummary(ctx context.Context, userId string) (*types.Summary[*types.ContinuousStats, types.ContinuousStats], error) {
func (c *ClientImpl) UpdateContinuousSummary(ctx context.Context, userId string) (*types.Summary[*types.ContinuousPeriods, *types.ContinuousBucket, types.ContinuousPeriods, types.ContinuousBucket], error) {
if ctx == nil {
return nil, errors.New("context is missing")
}
Expand All @@ -241,7 +230,7 @@ func (c *ClientImpl) UpdateContinuousSummary(ctx context.Context, userId string)
}

url := c.client.ConstructURL("v1", "summaries", "continuous", userId)
summary := &types.Summary[*types.ContinuousStats, types.ContinuousStats]{}
summary := &types.Summary[*types.ContinuousPeriods, *types.ContinuousBucket, types.ContinuousPeriods, types.ContinuousBucket]{}
if err := c.client.RequestData(ctx, http.MethodPost, url, nil, nil, summary); err != nil {
if request.IsErrorResourceNotFound(err) {
return nil, nil
Expand All @@ -252,19 +241,6 @@ func (c *ClientImpl) UpdateContinuousSummary(ctx context.Context, userId string)
return summary, nil
}

func (c *ClientImpl) BackfillSummaries(ctx context.Context, typ string) (int, error) {
var count int
url := c.client.ConstructURL("v1", "summaries", "backfill", typ)

ctxWithTimeout, cancel := context.WithTimeout(ctx, ExtendedTimeout)
defer cancel()
if err := c.client.RequestData(ctxWithTimeout, http.MethodPost, url, nil, nil, &count); err != nil {
return count, errors.Wrap(err, "backfill request returned an error")
}

return count, nil
}

func (c *ClientImpl) GetOutdatedUserIDs(ctx context.Context, typ string, pagination *page.Pagination) (*types.OutdatedSummariesResponse, error) {
if ctx == nil {
return nil, errors.New("context is missing")
Expand Down
3 changes: 2 additions & 1 deletion data/events/events.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,12 @@ package events
import (
"context"

summaryStore "github.com/tidepool-org/platform/summary/store"

ev "github.com/tidepool-org/go-common/events"

dataSourceStoreStructured "github.com/tidepool-org/platform/data/source/store/structured"
dataStore "github.com/tidepool-org/platform/data/store"
summaryStore "github.com/tidepool-org/platform/data/summary/store"
"github.com/tidepool-org/platform/errors"
"github.com/tidepool-org/platform/log"
)
Expand Down
5 changes: 3 additions & 2 deletions data/service/api/v1/datasets_data_create.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,15 @@ import (
"strconv"
"strings"

"github.com/tidepool-org/platform/summary"
"github.com/tidepool-org/platform/summary/types"

"github.com/ant0ine/go-json-rest/rest"
"github.com/golang-jwt/jwt/v4"

"github.com/tidepool-org/platform/data"
dataNormalizer "github.com/tidepool-org/platform/data/normalizer"
dataService "github.com/tidepool-org/platform/data/service"
"github.com/tidepool-org/platform/data/summary"
"github.com/tidepool-org/platform/data/summary/types"
dataTypesFactory "github.com/tidepool-org/platform/data/types/factory"
"github.com/tidepool-org/platform/log"
"github.com/tidepool-org/platform/permission"
Expand Down
4 changes: 2 additions & 2 deletions data/service/api/v1/datasets_update.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ package v1
import (
"net/http"

"github.com/tidepool-org/platform/data/summary"
"github.com/tidepool-org/platform/summary"
"github.com/tidepool-org/platform/summary/types"

"github.com/tidepool-org/platform/data"
dataService "github.com/tidepool-org/platform/data/service"
"github.com/tidepool-org/platform/data/summary/types"
"github.com/tidepool-org/platform/data/types/upload"
"github.com/tidepool-org/platform/log"
"github.com/tidepool-org/platform/permission"
Expand Down
Loading