Skip to content

Commit

Permalink
Vendor OpenSSL on Linux only
Browse files Browse the repository at this point in the history
  • Loading branch information
rgwood committed Dec 14, 2024
1 parent 5d3aeee commit d19e223
Show file tree
Hide file tree
Showing 3 changed files with 24 additions and 28 deletions.
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 5 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
[package]
name = "rezoning-scraper"
version = "3.0.2"
version = "3.0.3"
edition = "2021"

[dependencies]
anyhow = "1.0.94"
base64 = "0.22.1"
chrono = "0.4.39"
clap = { version = "4.5.23", features = ["derive"] }
openssl = { version = "0.10", features = ["vendored"], optional = true}
scraper = "0.22.0"
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
Expand All @@ -17,6 +16,10 @@ reqwest = { version = "0.12.9", features = ["blocking", "json"] }
indicatif = "0.17.8"
colored = "2.1.0"

# Force vendored OpenSSL on Linux to make building for musl easier
[target.'cfg(target_os = "linux")'.dependencies]
openssl = { version = "0.10", features = ["vendored"]}

# build with `cargo build --profile profiling`
# to analyze performance with tooling like linux perf
[profile.profiling]
Expand Down
43 changes: 18 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,41 +10,34 @@ Scrapes the City of Vancouver's website for rezoning and development application

## How to use

Download a binary from [the releases page](https://github.com/rgwood/RezoningScraper/releases) (no external dependencies needed) or build it from source (requires the .NET 8 SDK). Then run it:
Download a binary from [the releases page](https://github.com/rgwood/RezoningScraper/releases) or build it from source ([install Rust](https://rustup.rs/) then run `cargo build --release`). Then run it:

```
❯ .\RezoningScraper.exe --help
RezoningScraper
A tool to detect new+modified postings on Vancouver's shapeyourcity.ca website.
Data is stored in a local SQLite database next to the executable.
Usage:
RezoningScraper [options]
❯ ./rezoning-scraper --help
Usage: rezoning-scraper [OPTIONS]
Options:
--slack-webhook-url <slack-webhook-url> A Slack Incoming Webhook URL. If specified, RezoningScraper will post info about new+modified rezonings to this
address.
--save-to-db Whether to save the API results to database. [default: True]
--version Show version information
-?, -h, --help Show help and usage information
--slack-webhook-url <SLACK_WEBHOOK_URL>
A Slack Incoming Webhook URL. If specified, will post info about new+modified rezonings to this address.
--api-cache
Use cached API responses (up to 1 hour old) when available
--skip-update-db
Skip updating the local database (useful for testing)
-h, --help
Print help
-V, --version
Print version
```



## License

Public domain. Do whatever you like with this code, no attribution needed.

## To Do

- [ ] GitHub Actions for build+test
- [x] `--dry-run` flag that doesn't update the local DB
- [x] Add retries to the token fetching. It times out fairly often
- [x] Some kind of development cache thingy. The API is slowwwww and that makes iterative development painful
- [ ] Discord integration?
- [ ] Twitter integration
- [ ] Display more project fields?
- [ ] [This is rough](https://github.com/rgwood/RezoningScraper/blob/ca38460e6ffbd177ef842b0362ff3449737bf3a5/RezoningScraper/TokenHelper.cs#L54-L60), there's gotta be a better way to query JSON
- [ ] Consider archiving old versions of projects
- [x] Hook up to Sentry for error reporting
- [ ] Script deployment+setup (just rsync and cron lol) to a remote server
- [x] Strip line breaks from project titles - CoV does that sometimes and it breaks the Slack link format.
- [x] Consider excluding `published -> archived` state transitions from Slack. Lots of noise, not particularly useful

- [ ] Hook up to Sentry for error reporting
- [ ] Add option to post to Bluesky

0 comments on commit d19e223

Please sign in to comment.