Skip to content

Multi instance #670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 49 commits into from
Apr 5, 2024
Merged
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
3815a9f
Checkpoint
lbwexler Jan 27, 2023
f7cd2ff
Merge branch 'develop' into multiInstance
lbwexler Aug 16, 2023
1ef0e19
Checkpoint
lbwexler Aug 23, 2023
2b25940
Checkpoint
lbwexler Aug 24, 2023
394f72a
Checkpoint -- Make Portfolio App Cluster aware
lbwexler Aug 30, 2023
f7141df
Merge branch 'develop' into multiInstance
lbwexler Sep 2, 2023
45b121d
Add changelog
lbwexler Sep 2, 2023
7ceaaae
Fix broken monitors
lbwexler Sep 5, 2023
2b32b8a
+ Use Timer.masterOnly
lbwexler Sep 6, 2023
cbb7c77
Changes from code review
lbwexler Sep 7, 2023
afe160d
Checkpoint
lbwexler Sep 8, 2023
c9fae32
Checkpoint
lbwexler Sep 8, 2023
cdf5228
Checkpoint
lbwexler Sep 8, 2023
dc9a82a
Merge branch 'develop' into multiInstance
lbwexler Sep 11, 2023
d40ddac
Temporarily revert problematic additional "app"
lbwexler Sep 12, 2023
25d0347
Merge branch 'develop' into multiInstance
amcclain Sep 15, 2023
5ffa6c8
Merge branch 'develop' into multiInstance
amcclain Sep 21, 2023
557f799
Update to latest Hoist snapshots
amcclain Sep 21, 2023
e03c3bb
Enable AWS plugin for Toolbox deployment
amcclain Sep 21, 2023
11c41da
Add toolbox-multi to OAuth list
amcclain Sep 21, 2023
9093ba0
checkpoint - java only config
lbwexler Sep 27, 2023
a21e730
checkpoint -- support for isReady
lbwexler Sep 29, 2023
d73f94a
Admin stats for toolbox services
lbwexler Oct 2, 2023
45070f1
Tweak to Timer stats
lbwexler Oct 2, 2023
f0d4738
Tweak to class name and package of ClusterConfig
lbwexler Oct 2, 2023
74d239e
Fix Hazelcast version appropriately with spring BOM
lbwexler Oct 3, 2023
a9f6022
Major improvements to ability of apps and plugins to override all asp…
lbwexler Oct 12, 2023
edd1eaf
Use Shared Object
lbwexler Oct 12, 2023
559e438
Fix bug with missing ClusterConfig
lbwexler Oct 13, 2023
acd8442
Merge branch 'develop' into multiInstance
lbwexler Oct 16, 2023
a028761
Merge branch 'develop' into multiInstance
lbwexler Nov 14, 2023
a2e21a0
Merge yarn
lbwexler Nov 14, 2023
3c548f5
Update Grails version
lbwexler Nov 17, 2023
59599da
Checkpoint
lbwexler Nov 20, 2023
4ea863c
Adapt to recent API changes
lbwexler Nov 21, 2023
bbbf76d
Adapt to latest grails/gradle changes
lbwexler Nov 21, 2023
896d655
Fix to recent grails 6.0.1 regression
lbwexler Nov 21, 2023
f81388b
Fix to recent grails 6.1.0 regression
lbwexler Nov 21, 2023
6180233
syntax cleanup for admin stats
lbwexler Nov 22, 2023
71e9108
Merge branch 'develop' into multiInstance
lbwexler Dec 26, 2023
ce01dd0
Restore Forms based login
lbwexler Dec 27, 2023
8c4551a
Adapt to new simpler cache configuration
lbwexler Dec 27, 2023
7d27395
Use Kryo for Serialization
lbwexler Dec 29, 2023
bf4f01e
Merge branch 'develop' into multiInstance
lbwexler Mar 17, 2024
578fe04
Merge branch 'develop' into multiInstance
lbwexler Mar 18, 2024
368a97a
Upgrade snapshot version for multi-instance
lbwexler Mar 18, 2024
e245da3
Merge branch 'develop' into multiInstance
lbwexler Apr 2, 2024
c8bd8d7
Merge branch 'develop' into multiInstance
lbwexler Apr 4, 2024
9d83569
Remove cruft, update yarn.lock
lbwexler Apr 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -49,6 +49,11 @@ process to always use `Auth0` with `bootstrapAdminUser` for initial login.
* @xh/hoist 59.2.0
* hoist-core 17.4.0

### Libraries
* @xh/hoist 60.0.0
* hoist-core 18.0.0


## v3.3.0 - 2023-09-20

### Libraries
21 changes: 14 additions & 7 deletions build.gradle
Original file line number Diff line number Diff line change
@@ -79,9 +79,18 @@ Map hoistMetaData = [
def allJvmArgs = [
'-Dspring.output.ansi.enabled=always',
'-XX:TieredStopAtLevel=1',
'-Xmx1024m'
'-Xmx1024m',
"--add-modules=java.se",
"--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED",
"--add-opens=java.base/java.lang=ALL-UNNAMED",
"--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
"--add-opens=java.management/sun.management=ALL-UNNAMED",
"--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED",
"--add-opens=java.base/java.util=ALL-UNNAMED"
]



if (parseBoolean(enableHotSwap)) {
def groovyReset = configurations.developmentOnly.resolve().find {it.name.contains("groovyReset")}
allJvmArgs += [
@@ -117,12 +126,10 @@ idea {
//--------------------------------------------------
// Extensions to build.info (Hoist-Core requirement)
//--------------------------------------------------
buildProperties.doLast {
File grailsBuildInfoFile = it.outputs.files.files.find { it.name == 'grails.build.info' }
if (!grailsBuildInfoFile) return

tasks.war.doFirst {
File infoFile = layout.buildDirectory.file('resources/main/META-INF/grails.build.info').get().asFile
Properties properties = new Properties()
grailsBuildInfoFile.withInputStream {properties.load(it)}
infoFile.withInputStream {properties.load(it)}
properties.putAll(hoistMetaData)
grailsBuildInfoFile.withOutputStream {properties.store(it, null)}
infoFile.withOutputStream {properties.store(it, null)}
}
16 changes: 8 additions & 8 deletions client-app/yarn.lock
Original file line number Diff line number Diff line change
@@ -2137,10 +2137,10 @@
eslint-plugin-react-hooks "^4.6"
typescript "5.x"

"@xh/hoist-dev-utils@8.x":
version "8.1.0"
resolved "https://registry.yarnpkg.com/@xh/hoist-dev-utils/-/hoist-dev-utils-8.1.0.tgz#405272d3befa118205ff54766bdc57979c2de418"
integrity sha512-4CjlJ24Ql8CWegsG7E0RvRS3w/iZ86yZgAN7Ocu8aaHYkQEKLfJGWghSuSwEzttg1sFTWFRbvgxWsqa7jd4k6A==
"@xh/hoist-dev-utils@^9.0.0-SNAPSHOT":
version "9.0.0-SNAPSHOT.1711643181370"
resolved "https://registry.yarnpkg.com/@xh/hoist-dev-utils/-/hoist-dev-utils-9.0.0-SNAPSHOT.1711643181370.tgz#282814b4f287d74b6b6dc7401ee114b51d821c80"
integrity sha512-b3BdYVIWnVowe7zetqAKgMf9nKiC/Sv7T7G8lM2/KTq+L5SdCQ1fY1DqhqePDxy9AzlBWSzwmUlA2MJM0hFA2g==
dependencies:
"@babel/core" "^7.24.0"
"@babel/plugin-proposal-decorators" "^7.24.0"
@@ -2179,10 +2179,10 @@
webpack-dev-server "~5.0.3"
webpackbar "~6.0.1"

"@xh/hoist@^63.0.0":
version "63.0.0"
resolved "https://registry.yarnpkg.com/@xh/hoist/-/hoist-63.0.0.tgz#e582d5e90328b8efe4c289f63c14d5a0a74753e8"
integrity sha512-WCmyPnE196qcVNedmkVx9gSWC7k76OtaLP+uPzVC1a7ZqAFPgN2hKNsZ13lwYTu943MWzlveulm06yXvlylWAA==
"@xh/hoist@^64.0.0-SNAPSHOT":
version "64.0.0-SNAPSHOT.1712237357573"
resolved "https://registry.yarnpkg.com/@xh/hoist/-/hoist-64.0.0-SNAPSHOT.1712237357573.tgz#8ec0bcb17c0863c25030bce4b0e63b4ac9ad3431"
integrity sha512-YzPfl5HiOB9wEFcwHOtdS5WmjS1PenkeNNFVtP1YjMq2Sv09lg8IsguW8/lTnCXz+N4YTFaUYih/luqo7SVhpQ==
dependencies:
"@blueprintjs/core" "^5.10.1"
"@blueprintjs/datetime2" "^2.3.3"
9 changes: 5 additions & 4 deletions gradle.properties
Original file line number Diff line number Diff line change
@@ -5,12 +5,13 @@ xhAppVersion=5.0-SNAPSHOT

hoistCoreVersion=20.0-SNAPSHOT

grailsVersion=6.0.0
grailsGradlePluginVersion=6.0.0
grailsHibernatePluginVersion=8.0.0
grailsVersion=6.1.2
grailsGradlePluginVersion=6.1.2
grailsHibernatePluginVersion=8.1.0
groovyVersion=3.0.11
gormVersion=8.0.0
gormVersion=8.1.0
logback.version=1.2.7
hazelcast.version=5.3.6

runHoistInline=false
enableHotSwap=false
43 changes: 0 additions & 43 deletions grails-app/conf/ehcache.xml

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -30,29 +30,25 @@ class PortfolioController extends BaseController {
renderJSON(positionService.ordersForPosition(params.positionId))
}

def rawPositions() {
renderJSON(portfolioService.getData().rawPositions)
}

def orders() {
renderJSON(portfolioService.getData().orders)
renderJSON(portfolioService.getPortfolio().orders)
}

def symbols() {
renderJSON(portfolioService.getData().instruments.keySet())
renderJSON(portfolioService.getPortfolio().instruments.keySet())
}

def instrument() {
renderJSON(portfolioService.getData().instruments[params.id])
renderJSON(portfolioService.getPortfolio().instruments[params.id])
}

// List of MarketPrices for the given instrument identified by its symbol
def prices() {
List<MarketPrice> historicalPrices = portfolioService.getData().historicalPrices[params.id]
List<MarketPrice> historicalPrices = portfolioService.getPortfolio().historicalPrices[params.id]
if (!historicalPrices) {
throw new DataNotAvailableException("No historical prices available for ${params.id}")
}
MarketPrice intradayPrices = portfolioService.getData().intradayPrices[params.id]
MarketPrice intradayPrices = portfolioService.getCurrentPrices()[params.id]
List<MarketPrice> allPrices = intradayPrices ? historicalPrices.dropRight(1)+[intradayPrices] : historicalPrices
renderJSON(allPrices)
}
21 changes: 21 additions & 0 deletions grails-app/init/io/xh/toolbox/ClusterConfig.groovy
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
package io.xh.toolbox

import com.hazelcast.config.Config
import io.xh.hoist.util.Utils
class ClusterConfig extends io.xh.hoist.ClusterConfig {

Config createConfig() {
def ret = super.createConfig()
if (!Utils.isLocalDevelopment) {
def join = ret.networkConfig.join,
interfaces = ret.networkConfig.interfaces

join.multicastConfig.enabled = false
join.awsConfig.enabled = true
interfaces.enabled = true
interfaces.addInterface('172.30.*.*')
}
return ret
}

}
Original file line number Diff line number Diff line change
@@ -116,21 +116,21 @@ class MonitorDefinitionService extends BaseService {
* Check the current number of positions in the Portfolio example
*/
def positionCount(MonitorResult result) {
result.metric = portfolioService.data.rawPositions.size()
result.metric = portfolioService.portfolio.rawPositions.size()
}

/**
* Check the current number of instruments in the Portfolio example
*/
def instrumentCount(MonitorResult result) {
result.metric = portfolioService.data.instruments.size()
result.metric = portfolioService.portfolio.instruments.size()
}

/**
* Check when the most recent prices in the Portfolio example were generated
*/
def pricesAgeMs(MonitorResult result) {
result.metric = currentTimeMillis() - portfolioService.data.timeCreated.time
result.metric = currentTimeMillis() - portfolioService.portfolio.timeCreated.time
}

/**
18 changes: 16 additions & 2 deletions grails-app/services/io/xh/toolbox/SlackAlertService.groovy
Original file line number Diff line number Diff line change
@@ -17,8 +17,17 @@ class SlackAlertService extends BaseService{

ConfigService configService
void init() {
subscribe('xhMonitorStatusReport', this.&formatAndSendMonitorStatusReport)
subscribe('xhClientErrorReceived', this.&formatAndSendClientReport)
subscribeToTopic(
topic: 'xhMonitorStatusReport',
onMessage: this.&formatAndSendMonitorStatusReport,
masterOnly: true

)
subscribeToTopic(
topic: 'xhClientErrorReceived',
onMessage: this.&formatAndSendClientReport,
masterOnly: true
)
}

//------------------------
@@ -89,4 +98,9 @@ Time: ${ce.dateCreated.format('dd-MMM-yyyy HH:mm:ss')}
private Map getConfig(){
return configService.getMap('slackAlertConfig', [enabled: false])
}

Map getAdminStats() {[
config: configForAdminStats('slackAlertConfig')
]}

}
Original file line number Diff line number Diff line change
@@ -70,4 +70,7 @@ class FileManagerService extends BaseService {
return configService.getString('fileManagerStoragePath')
}

Map getAdminStats() {[
config: configForAdminStats('fileManagerStoragePath')
]}
}
32 changes: 21 additions & 11 deletions grails-app/services/io/xh/toolbox/app/GitHubService.groovy
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
package io.xh.toolbox.app

import groovy.util.logging.Slf4j
import com.hazelcast.replicatedmap.ReplicatedMap
import io.xh.hoist.BaseService
import io.xh.hoist.config.ConfigService
import io.xh.hoist.http.JSONClient
@@ -30,20 +30,21 @@ import static io.xh.hoist.util.DateTimeUtils.MINUTES
* environments, where a developer might wish to enable this service to load *some* commits but
* has no reason to load the entire history. (Set to a value of 1 to work in this mode.)
*/
@Slf4j
class GitHubService extends BaseService {

static clearCachesConfigs = ['gitHubRepos', 'gitHubAccessToken', 'gitHubMaxPagesPerLoad']

ConfigService configService
WebSocketService webSocketService
Map<String, CommitHistory> commitsByRepo = new HashMap()

private ReplicatedMap<String, CommitHistory> commitsByRepo = getReplicatedMap('commitsByRepo')

void init() {
if (configService.getString('gitHubAccessToken', 'none') == 'none') {
logWarn('Required "gitHubAccessToken" config not present or set to "none" - no commits will be loaded from GitHub.')
} else {
createTimer(
masterOnly: true,
runFn: this.&loadCommitsForAllRepos,
interval: 'gitHubCommitsRefreshMins',
intervalUnits: MINUTES
@@ -52,14 +53,17 @@ class GitHubService extends BaseService {
}

/** Return the cached history of commits for a single repo, by name. */
CommitHistory getCommitsForRepo(String repoName) {commitsByRepo[repoName]}
CommitHistory getCommitsForRepo(String repoName) {
commitsByRepo[repoName]
}

/**
* Reload commit history for all configured repos from GitHub API.
* @param forceFullLoad - true to drop any already loaded history, false (default) to do an
* incremental load of new commits only.
*/
Map<String, CommitHistory> loadCommitsForAllRepos(Boolean forceFullLoad = false) {
private Map<String, CommitHistory> loadCommitsForAllRepos(Boolean forceFullLoad = false) {

def repos = configService.getList('gitHubRepos', []),
newCommitCount = 0

@@ -82,7 +86,7 @@ class GitHubService extends BaseService {
* Reload commit history for a single repo, by name.
* @return collection of newly loaded commits, if any.
*/
List<Commit> loadCommitsForRepo(String repoName, Boolean forceFullLoad = false) {
private List<Commit> loadCommitsForRepo(String repoName, Boolean forceFullLoad = false) {
def hadError = false,
hasNextPage = true,
cursor = '',
@@ -152,12 +156,11 @@ class GitHubService extends BaseService {
} else {
logDebug('Commit load complete', "${newCommits.size()} new commits")
commitHistory.updateWithNewCommits(newCommits)
this.commitsByRepo.put(repoName, commitHistory)
commitsByRepo.put(repoName, commitHistory)
return newCommits
}
}


//------------------------
// Implementation
//------------------------
@@ -232,9 +235,16 @@ query XHRepoCommits {
}

void clearCaches() {
this._jsonClient = null
this.commitsByRepo = new HashMap()
this.loadCommitsForAllRepos()
_jsonClient = null
if (isMaster) {
commitsByRepo.clear()
loadCommitsForAllRepos()
}
super.clearCaches()
}


Map getAdminStats() { [
config: configForAdminStats('gitHubAccessToken', 'gitHubRepos', 'gitHubMaxPagesPerLoad')
]}
}
Loading