Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce OLAP stack for MAU measurement using Kafka and StarRocks #1130

Open
hackerwins opened this issue Jan 31, 2025 · 0 comments
Open

Introduce OLAP stack for MAU measurement using Kafka and StarRocks #1130

hackerwins opened this issue Jan 31, 2025 · 0 comments
Labels
enhancement 🌟 New feature or request

Comments

@hackerwins
Copy link
Member

hackerwins commented Jan 31, 2025

Description:

To achieve the goal of acquiring 5 million MAU for Yorkie Project, it is crucial to implement a mechanism for measuring MAU. I propose to introduce an OLAP stack consisting of Kafka and StarRocks to facilitate this measurement.

Additionally, the current server architecture will need to be augmented with events, stats, messagebroker and warehouse package.

Image
  • events: Handle event generation and processing
  • stats: Handle statistical calculations and aggregations
  • messagebroker: Handle Kafka integration
  • warehouse: Manage StarRocks data storage and queries

Importantly, when a Client is created, it should be able to receive a UserID through metadata. To address security concerns regarding the direct storage of UserIDs in the database, it is recommended that a hashed version of the UserID be stored instead.

const client = new yorkie.Client('https://api.yorkie.dev', {
  apiKey: 'xxxxxxxxxxxxxxxxxxxx', // Identifying Project
  userID: 'hackerwins',           // Identifying User
});

Why:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement 🌟 New feature or request
Projects
Status: Backlog
Development

No branches or pull requests

1 participant