Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix problems with failing prepare-dependencies.sh script #142

Open
michalinacienciala opened this issue May 16, 2023 · 2 comments
Open

Fix problems with failing prepare-dependencies.sh script #142

michalinacienciala opened this issue May 16, 2023 · 2 comments
Assignees

Comments

@michalinacienciala
Copy link
Contributor

Background

We are using keep-network/keep-core dependency in several projects:

The keep-network/keep-core package contains a KEEP-related contract called TokenStaking which is the same name that we use for staking contract for T token. We wanted to deploy both contracts in the same deployments, so we hit a contract name conflict (see issue wighawag/hardhat-deploy#241). That's why we created a prepare-dependencies.sh script that gets executed as a last step of installation of threshold-network/solidity-contracts project and renames the keep-core's TokenStaking artifact to KeepTokenStaking.

The issue(s)

The script is used as a postinstall script and uses the INIT_CWD variable as a parameter:

"postinstall": "./scripts/prepare-dependencies.sh $INIT_CWD",

The INIT_CWD isn't set by us, it's a variable created by Yarn that should hold the full path of the localization where yarn run/yarn install was executed from. Unfortunately, sometimes it does not get configured correctly (it happens seemingly randomly) and causes the failures of the script.

Another issue is that in the script we're copying the files from the following location:

SOURCE_DIR="$ROOT_DIR/node_modules/@keep-network/keep-core/artifacts"

This location sometimes may not be available - this is due to the fact that Yarn flattens the structure of dependencies. For example, when threshold-network/solidity-contracts is a dependency of some project, Yarn may not include @keep-network/keep-core in threshold-network/solidity-contracts's node_modules if @keep-network/keep-core is already included in node_modules of another dependency.

Examples:

https://github.com/keep-network/tbtc-v2/actions/runs/4969553945/jobs/8912960954:

error /home/runner/work/tbtc-v2/tbtc-v2/solidity/node_modules/@threshold-network/solidity-contracts: Command failed.
Exit code: 1
Command: ./scripts/prepare-dependencies.sh $INIT_CWD
Arguments: 
Directory: /home/runner/work/tbtc-v2/tbtc-v2/solidity/node_modules/@threshold-network/solidity-contracts
Output:
Preparing dependencies artifacts
Root directory: /home/runner/work/tbtc-v2/tbtc-v2/solidity
Source directory: /home/runner/work/tbtc-v2/tbtc-v2/solidity/node_modules/@keep-network/keep-core/artifacts
Destination directory: /home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core
mv: cannot stat '/home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core/artifacts/TokenStaking.json': No such file or directory

https://github.com/keep-network/tbtc-v2/actions/runs/4969553945/jobs/8892795133

error /home/runner/work/tbtc-v2/tbtc-v2/solidity/node_modules/@keep-network/ecdsa/node_modules/@keep-network/random-beacon/node_modules/@threshold-network/solidity-contracts: Command failed.
Exit code: 1
Command: ./scripts/prepare-dependencies.sh $INIT_CWD
Arguments: 
Directory: /home/runner/work/tbtc-v2/tbtc-v2/solidity/node_modules/@keep-network/ecdsa/node_modules/@keep-network/random-beacon/node_modules/@threshold-network/solidity-contracts
Output:
Preparing dependencies artifacts
Root directory: /home/runner/work/tbtc-v2/tbtc-v2/solidity
Source directory: /home/runner/work/tbtc-v2/tbtc-v2/solidity/node_modules/@keep-network/keep-core/artifacts
Destination directory: /home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core
cp: cannot create regular file '/home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core/artifacts/TestArrayUtils.json': File exists
cp: cannot create regular file '/home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core/artifacts/TestCurveRewards.json': File exists
cp: cannot create regular file '/home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core/artifacts/TestModUtils.json': File exists
cp: cannot create regular file '/home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core/artifacts/TestSimpleBeneficiary.json': File exists
cp: cannot create regular file '/home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core/artifacts/TestSimpleReceiver.json': File exists
cp: cannot create regular file '/home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core/artifacts/TestSimpleStakerRewards.json': File exists
cp: cannot create regular file '/home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core/artifacts/TestToken.json': File exists
cp: cannot create regular file '/home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core/artifacts/TokenDistributor.json': File exists
cp: cannot create regular file '/home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core/artifacts/TokenGeyser.json': File exists
cp: cannot create regular file '/home/runner/work/tbtc-v2/tbtc-v2/solidity/external/npm/@keep-network/keep-core/artifacts/TokenGrant.json': File exists

https://github.com/keep-network/keep-core/actions/runs/4969527266/jobs/8892744375:

error /home/runner/work/keep-core/keep-core/solidity/ecdsa/node_modules/@keep-network/random-beacon/node_modules/@threshold-network/solidity-contracts: Command failed.
Exit code: 1
Command: ./scripts/prepare-dependencies.sh $INIT_CWD
Arguments: 
Directory: /home/runner/work/keep-core/keep-core/solidity/ecdsa/node_modules/@keep-network/random-beacon/node_modules/@threshold-network/solidity-contracts
Output:
Preparing dependencies artifacts
Root directory: /home/runner/work/keep-core/keep-core/solidity/ecdsa
Source directory: /home/runner/work/keep-core/keep-core/solidity/ecdsa/node_modules/@keep-network/keep-core/artifacts
Destination directory: /home/runner/work/keep-core/keep-core/solidity/ecdsa/external/npm/@keep-network/keep-core
cp: cannot create directory '/home/runner/work/keep-core/keep-core/solidity/ecdsa/external/npm/@keep-network/keep-core/artifacts': File exists

Solutions

There are several approaches we could take to tackle this:

  1. Try to fix each problem directly (change the script to not rely on the INIT_CWD variable, figure out how to work with flattened node_modules structure)
  2. Try to get rid of the problem with the name conflict by publishing new NPM packages with keep-core contracts that would be the exact copies of the current ones that are in use, with the only difference being the name of the TokenStaking contract (it could be renamed to KeepTokenStaking). This way we would no longer have a name conflict and could get rid of the prepare-dependencies.sh script.
  3. Deal with all those dependencies by just hardcoding ABI and addresses. v1 TokenStaking or KeepToken addresses are not going to change. This may be even quicker time-wise than figuring out what is happening with environment variables. And we'll make build times shorter. Double win.
michalinacienciala added a commit that referenced this issue Jun 6, 2023
…ncies-script

Remove prepare dependencies script

Refs #142

This PR removes `prepare-dependencies.sh` script that randomly errors out and gives a lot of headaches to developers. The script was added to mitigate the issue with the same naming of `TokenStaking.sol` contracts that are in `keep-core` and `solidity-contracts` repos. In some projects, both repos were added as dependencies and hardhat had a hard time differentiating the contracts during deployment. Since v1 contracts are already deployed and won't be changed, we can now hardcode the artifacts and get rid of the troubled `prepare-dependencies.sh` script and related code.

The following repos were checked for any potential issues as a consequence of removing this script. I didn't find any.
- threshold-network/token-dashboard
- threshold-network/merkle-distribution
- keep-network/coverage-pools
- keep-network/keep-core (random-beacon and ecdsa)

Depends on #144
@michalinacienciala
Copy link
Contributor Author

The problem got fixed by @dimpar in #143.

I published new NPM -dev packages for the following projects:

@keep-network/random-beacon
@keep-network/ecdsa
@keep-network/coverage-pools

I also removed some no longer needed workarounds in the projects that use @threshold-network/solidity-contracts as a dependency:
threshold-network/token-dashboard#531
keep-network/tbtc-v2#629

We still may hit the issue occasionally when deploying on Mainnet or Goerli (to fix that we would have to redeploy @threshold-network/solidity-contracts NPM packages (and all the downstream projects) for those environments. But I think we can live with this, as we don't deploy on those environments often.

@michalinacienciala
Copy link
Contributor Author

Another PRs related to the removal of prepare-dependencies.sh:
threshold-network/token-dashboard#533
keep-network/keep-core#3618
keep-network/coverage-pools#233
keep-network/tbtc-v2#631

dimpar added a commit to keep-network/tbtc-v2 that referenced this issue Jun 9, 2023
Background:
Previously we've been running postinstall script in a separate step from
the `yarn install` command. We've been doing that as a workaround for a
problem we had with the postinstall script in the
`@threshold-network/solidity-contracts` package (the script in that
package often randomly failed). Running `yarn upgrade` with
`--ignore-scripts` flag prevented the execution of postinstall script in
the main project and its depandencies. And running `yarn run
postinstall` after did execute the postinstall script in the main
project.

The change:
Recently we have refactored the `@threshold-network/solidity-contracts`
project and got rid of the problematic script. We can go back to
executing `yarn install` without the `--ignore-scripts` flag.

Ref:
threshold-network/solidity-contracts#142
threshold-network/solidity-contracts#143
threshold-network/token-dashboard#531
dimpar added a commit to keep-network/keep-core that referenced this issue Jun 14, 2023
We have published new `@threshold-network/solidity-contract` and
`@keep-network/random-beacon` packages. Those packages no longer include
the `prepare-dependencies.sh` script from
`@threshold-network/solidity-contracts` which was causing random
failures during `yarn install`. As in some CI jobs we install the
dependencies based on the lockfile (with the `--frozen-lockfile` flag),
we need to update the dependencies in the lockfile so that the new,
improved packages would be used.

Refs:
threshold-network/solidity-contracts#142
threshold-network/token-dashboard#533
keep-network/coverage-pools#233
keep-network/tbtc-v2#631
dimpar added a commit to keep-network/tbtc-v2 that referenced this issue Jun 14, 2023
We have published new `@keep-network/random-beacon`,
`@keep-network/ecdsa` and `@keep-network/tbtc-v2` packages. Those
packages no longer include the `prepare-dependencies.sh` script from
`@threshold-network/solidity-contracts` which was causing random
failures during `yarn install`. As in some CI jobs we install the
dependencies based on the lockfile (with the `--frozen-lockfile` flag),
we need to update dependencies in the lockfile so that the new, improved
packages would be used.

Refs:
threshold-network/solidity-contracts#142
threshold-network/token-dashboard#533
keep-network/keep-core#3618
keep-network/coverage-pools#233
dimpar added a commit to keep-network/tbtc-v2 that referenced this issue Jun 15, 2023
We have published new `@keep-network/ecdsa` and `@keep-network/tbtc-v2`
packages. Those packages no longer include the `prepare-dependencies.sh`
script from `@threshold-network/solidity-contracts` which was causing
random failures during `yarn install`. As in some CI jobs we install the
dependencies based on the lockfile (with the `--frozen-lockfile` flag),
we need to update dependencies in the lockfile so that the new, improved
packages would be used.

Ref:
threshold-network/solidity-contracts#142
threshold-network/solidity-contracts#143
#631
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants