Skip to content

Commit b266ed8

Browse files
leonpolakleonpolak
authored
Initial partial description of system fundamentals (#1)
1 parent acf4a2c commit b266ed8

11 files changed

+140
-0
lines changed

‎README.md

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Specification of Metarhia Distributed Infrastructure
2+
3+
Templates and tasks of the technological stack of an extensible and configurable multifunctional distributed system. Such a system can be useful as a foundation for a wide class of tasks: education, health care, industry, ecommercce, social interactions, etc.
4+
5+
The goal of such a system is to provide out of the box a uniform, simple, efficient and cheap mechanism for systems with unpredictable user growth and associated traffic fluctuations. The maximum versatility of such a system will make it possible to reuse its customizable modules with minimal effort and in minimal time. Ideally, the modules of the system should resemble the elements of the LEGO set, with the described standard connection methods and the variability of the assembly results.
6+
7+
Not everything in this specification is in line with the current state of the art in the technology stack right now. Here
8+
described how it should be. For the latest information, see
9+
See documentation here: https://github.com/metarhia/Docs
10+
11+
Languages: [English](README.md) | [Russian](README.ru.md)
12+
13+
- [Command Center](doc/Management.md)
14+
- [Containers with business logic](doc/Node.md)
15+
- [Layers of information storage and data flow control](doc/Db.md)
16+
- [Internal and external dependencies](doc/Dependency.md)
17+
- [Planning](doc/Task.md)
18+
- [Deployment](doc/Deployment.md)
19+
- [Testing](doc/Testing.md)
20+
- [Security](doc/Security.md)
21+
- [Monitoring and Control](doc/Monitoring.md)

‎README.ru.md

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Спецификация Распределённой Инфраструктуры Metarhia
2+
3+
Шаблоны и задачи технологического стека расширяемой и конфигурируемой мультифункциональной распределённой системы. Подобная система может пригодиться в качестве фундамента для широкого класса задач: образования, здравохранения, индустрии, торговли, социальных взаимодействий и т.д.
4+
5+
Цель такой системы - предоставить из коробки единообразный простой эффективный и дешёвый иеханизм для систем с труднопрогнозируемыми ростом количества пользователей и соответствующими флюктуациями траффика. Максимальная универсальность подобной системы и позволит переиспользовать её настраиваемы модули с минимальной кастомизацией. В идеале, модули системы должны напоминать элементы конструктора ЛЕГО, с описанными стандартными методами подключения и вариативностью результатов сборки.
6+
7+
Не все в этой спецификации соответствует текущему положению дел в технологическом стеке прямо сейчас. Тут описано, как должно быть. Чтобы узнать актуальную информацию, смотрите
8+
документацию тут: https://github.com/metarhia/Docs
9+
10+
Языки: [English](README.md) | [Русский](README.ru.md)
11+
12+
- [Командный центр](doc/management.md)
13+
- [Контейнер с бизнес логикой](doc/node.md)
14+
- [Слои хранения информации и управления потоками данных](doc/db.md)
15+
- [Внутренние и внешние зависимости](doc/Dependency.md)
16+
- [Планирование](doc/Task.md)
17+
- [Развёртывание](doc/Deployment.md)
18+
- [Тестирование](doc/Testing.md)
19+
- [Защита](doc/Security.md)
20+
- [Мониторинг и управление](doc/Monitoring.md)

‎doc/Db.md

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
## Layered Data Storage Module
2+
3+
From hot to cold types of the data
4+
5+
1. Local RAM
6+
2. Redis cache
7+
3. Postgres
8+
4. File storage
9+
5. Analytics
10+
11+
## Local RAM
12+
13+
Should be calculatad upon start and constantly used by all processes running on the same machine for super hot parts of data.
14+
15+
## Redis cache
16+
17+
There are no available alternative to redis for this purpose, considering multiple types of data, speed of data access and manipulation, and large quantity of best practices to use it. Especially, with additional Redis modules - there are claims that those modulas are not free, but I didn't find any obstacles to attach them inside docker container. Tarantool and VoltDB have comparable advantages, but far less information, technical support, and examples of implemetation (including enormous software systems).
18+
19+
## Postgres
20+
21+
There are multiple self claimed alternatives to good old Postgres, including multiple relational, NoSql and column databases. However, right configured Postgres (Read/Write separation) in conjunction with TimescaleDB is enabled to the most wide number of data types for enormous amount of transactions (I was assured by industry top professionals that up to 1M users - there is no worry).
22+
23+
24+
## File storage
25+
26+
Cold data storage using the most effective file management systems, such as ZFS (?)
27+
28+
## Analitycs
29+
30+
Snowflake
31+
Vertica

‎doc/Dependency.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
## Dependencies module
2+
3+
1. CDN
4+
2. EMail
5+
3. Payment

‎doc/Deployment.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
## Deployment module

‎doc/Management.md

+50
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Metarhia Command Panel
2+
3+
## Settings and controls of distributed Metarhia infrastructure
4+
5+
*I suggest all involwed in the development of this model install and review [N|Solid](https://nodesource.com/products/nsolid) panel to review and learn functionality. To prevent reinwentment of a bycicle and to learn good solutions, in case there are amy.*
6+
7+
Control center with GUI, manual controls and scripted configurations input.
8+
9+
A container that consists all required modules for initial setting and configuration of fleet of nodes, DBs and other subsystems. By launch should be able to accept a list of IP addresses/ports, generate private keys for connection to all distributed subsystems, and choose from a list an approximate type and scale of desired system functionality (including general purpose option).
10+
11+
There should be several dublicates of such a center, in case of emergency. This is the most delicate and powerful source of thuth for the entire system, so security measures should be paramaunt.
12+
13+
## Requited Modules:
14+
15+
1. Authentication
16+
2. Transport
17+
3. Self-signed certificates generator
18+
4. Third party services configuration and connectors
19+
5. Load Test design, performance and analisys
20+
6. Configuration data storage
21+
7. Logging
22+
8. Sheduling
23+
9. GUI
24+
25+
## Authentication
26+
27+
The module should provide several types of roles and previlege levels
28+
29+
## Transport module
30+
31+
## Self-signed certificates generator
32+
33+
## Third party services connectors
34+
35+
Cloud hosters, Email services, CDN, payments etc.
36+
This module should calculate in human undestandable figures of cost/per user and approxiamate overoll fees for 3 party services in several scenarios of traffic fluctuations. AWS and several others have done everything to make this unbearable so, we need conremeasures.
37+
38+
## Configuration data storage
39+
40+
## Logging
41+
42+
## Sheduling
43+
44+
## GUI
45+
46+
The gui should provide:
47+
48+
1. Visualisation of the system statistics in real time
49+
2. Graphical planning and management of configurations with auto predictions of outcome
50+
3. Configuration scripts inputs (with syntax checker)

‎doc/Monitoring.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
## Monitoring module

‎doc/Node.md

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
## Node Module
2+
3+
Separate multipurpose customized node with self checking functionality. A basic cell of Metarhia claster. Distributed in docker container with predefined configuration and manageable from command center.
4+
5+
Upon activation receives credentials from command center, connects to particular levels of data storages, and sibling nodes in a claster. Should test and report particular machine characteristics, and launch several processes, according to CPU power, RAM availability and designated purpose in the cluster.
6+

‎doc/Security.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
## Security module

‎doc/Task.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
## Tesk sheduling and performance module

‎doc/Testing.md

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
## Testing module
2+
3+
There is a great functionality of [Artillery](https://artillery.io/) and [CLinic.js](https://clinicjs.org/), including ws testing scenarions and analisys of system bottlneks. We can integrate them for initial testing system, having in mind further developmant of in house analogs.

0 commit comments

Comments
 (0)