feat: add bitcoin support (#911)

Author: mduthey

Cargo.lock

@@ -21,6 +21,7 @@ u5c = ["utxorpc", "futures"]
 mithril = ["mithril-client"]
 hydra = ["tungstenite", "tokio-tungstenite", "futures-util", "bytes"]
 eth = ["futures-util"]
+btc = ["bitcoind-async-client", "corepc-types"]
 # elasticsearch = auto feature flag
 # kafka = auto feature flag
 
@@ -83,9 +84,11 @@ tokio-tungstenite = { version = "0.24.0", optional = true }
 futures-util = { version = "0.3", optional = true }
 bytes = { version = "1.7.2", optional = true }
 zmq = { version = "0.10.0", optional = true }
+bitcoind-async-client = { version = "0.6.0", optional = true }
+corepc-types = { version = "0.10.1", optional = true }
 
 # TODO(p): add feature
-alloy = { version = "1.0.30", features = ["full"] }
+alloy = { version = "1.0.38", features = ["full"] }
 
 [dev-dependencies]
 goldenfile = "1.7.3"

Cargo.toml

@@ -39,7 +39,7 @@ The data pipeline is implemented by the [Gasket](https://github.com/construkts/g
 
 ### CLI to Watch Live Transactions
 
-You can run `oura watch <socket>` to print TX data into the terminal from the tip of a local or remote node. It can be useful as a debugging tool for developers or if you're just curious to see what's going on in the network (for example, to see airdrops as they happen or oracles posting new information).
+You can run `oura watch cardano <socket>` to print Cardano transaction data into the terminal from the tip of a local or remote node, or `oura watch bitcoin <rpc_host>` to watch Bitcoin blockchain events. These commands are useful as debugging tools for developers or if you're just curious to see what's going on in the network (for example, to see airdrops as they happen or oracles posting new information).
 
 ### As a Bridge to Other Persistence Mechanisms
 
@@ -80,6 +80,7 @@ Oura is in its essence just a pipeline for processing events. Each stage of the
   - Hydra
   - Mithril
   - Utxorpc
+  - Bitcoin
 - Sinks
   - AWS Lambda
   - AWS S3

README.md

@@ -0,0 +1,3 @@
+label: Advanced Features
+order: 70
+collapsed: true

docs/v3/advanced/_meta.yaml

@@ -0,0 +1,54 @@
+---
+title: Custom networks
+---
+
+Instructions on how to configure Oura for connecting to a custom network (aka: other than mainnet / preprod / preview).
+
+## Context
+
+Oura requires certain information about the chain it is reading from. In a way, this is similar to the json config files required to run the Cardano node. These values are used for procedures such as encoding bech32 addresses, computing wall-clock time for blocks, etc.
+
+Since `mainnet`, `testnet`, `preprod` and `preview` are well-known, heavily used networks, Oura hardcodes these values as part of the binary release so that the user is spared from having to manually specify them. On the other hand, custom networks require the user to configure these values manually for Oura to establish a connection.
+
+## Feature
+
+By adding a `[chain]` section in the daemon configuration file, users can provide the information required by Oura to connect to a custom network.
+
+The `[chain]` section has the following propoerties:
+
+| Name                 | DataType | Description                                                   |
+| :------------------- | :------- | :------------------------------------------------------------ |
+| type                 | string   | types of network (mainnet, testnet, preprod, preview, custom) |
+| magic                | integer  | the network number                                            |
+| byron_epoch_length   | integer  | the length (in seconds) of a Byron epoch in this network      |
+| byron_slot_length    | integer  | the length (in seconds) of a Byron slot in this network       |
+| byron_known_slot     | integer  | the slot of a Byron block known to exist in this network      |
+| byron_known_hash     | string   | the hash of the known Byron block                             |
+| byron_known_time     | integer  | the unix timestamp of the known Byron block                   |
+| shelley_epoch_length | integer  | the length (in seconds) of a Shelley epoch in this network    |
+| shelley_slot_length  | integer  | the length (in seconds) of a Shelley slot in this network     |
+| shelley_known_slot   | integer  | the slot of a Shelley block known to exist in this network    |
+| shelley_known_hash   | String   | the hash of the known Shelley block                           |
+| shelley_known_time   | integer  | the unix timestamp of the known Shelley block                 |
+
+## Examples
+
+### Chain information for mainnet
+
+This example configuration shows the values for mainnet. Since testnet values are hardcoded as part of Oura's release, users are not required to input these exact values anywhere, but it serves as a good example of what the configuration looks like.
+
+```toml
+[chain]
+type = "custom"
+magic = 764824073
+byron_epoch_length  = 432000
+byron_slot_length = 20
+byron_known_slot = 0
+byron_known_time = 1506203091
+byron_known_hash = "f0f7892b5c333cffc4b3c4344de48af4cc63f55e44936196f365a9ef2244134f"
+shelley_epoch_length = 432000
+shelley_slot_length = 1
+shelley_known_slot = 4492800
+shelley_known_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de"
+shelley_known_time = 1596059091
+```

docs/v3/advanced/custom_network.mdx

@@ -0,0 +1,24 @@
+---
+title: Finalize Options
+---
+
+Advanced options for instructing Oura to stop on a chain point. If you'd like it to only sync a specific section of the chain, you can also instruct oura to stop syncing when it reaches a specific block hash by defining a `[finalize]` config.
+
+## Configuration
+
+To modify the default behavior the daemon mode uses, a section named `[finalize]` needs to be added to the `daemon.toml` file.
+
+```toml
+[finalize]
+until_hash = <BlockHash>
+max_block_slot = <SlotNumber>
+```
+
+## Examples
+
+The following example show how to configure Oura to stop sync on Byron era
+
+```toml
+[finalize]
+until_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de"
+```

docs/v3/advanced/finalize_options.mdx

@@ -0,0 +1,44 @@
+---
+title: Intersect Options
+---
+
+Advanced options for instructing Oura from which point in the chain to start reading from.
+
+## Feature
+
+When running in daemon mode, Oura provides 4 different strategies for finding the intersection point within the chain sync process.
+
+- `Tip`: Oura will start reading from the current tip of the chain.
+- `Origin`: Oura will start reading from the beginning of the chain.
+- `Point`: Oura will start reading from a particular point [slot, hash] in the chain.
+- `Breadcrumbs`: Oura will start reading from a some points [[slot, hash],[slot, hash]] in the chain.
+
+The default strategy use by Oura is `Tip`, unless an alternative option is specified via configuration.
+
+You can also define a finalizing point by providing a block hash at which oura will stop reading from the the chain and exit gracefully.
+
+## Configuration
+
+To modify the default behaviour used by the daemon mode, a section named `[intersect]` needs to be added in the `daemon.toml` file.
+
+```toml
+[intersect]
+type = <Type>
+value = <Value>
+```
+
+- `type`: Defines which strategy to use. Valid values are `Origin`, `Tip`, `Point`, `Breadcrumbs`. Default value is `Tip`.
+- `value`: Either a point or an array of points to be used as argument for the selected strategy.
+
+## Examples
+
+The following example show how to configure Oura to use a intersection point.
+
+```toml
+[intersect]
+type = "Point"
+value = [
+    4493860,
+    "ce7f821d2140419fea1a7900cf71b0c0a0e94afbb1f814a6717cff071c3b6afc",
+]
+```

docs/v3/advanced/intersect_options.mdx

@@ -0,0 +1,78 @@
+---
+title: Pipeline Metrics
+---
+
+The _metrics_ features allows operators to track the progress and performance of long-running Oura sessions.
+
+## Context
+
+Some use-cases require Oura to be running either continuosuly or for prolonged periods of time. Keeping a sink updated in real-time requires 24/7 operation. Dumping historical chain-data from origin might take several hours.
+
+These scenarios require an understanding of the internal state of the pipeline to facilitate monitoring and troubleshooting of the system. In a data-processing pipeline such as this, two important aspects need to observable: progress and performance.
+
+## Feature
+
+Oura provides an optional `/metrics` HTTP endpoint that uses Prometheus format to expose real-time opertional metrics of the pipeline. Each stage (source / sink) is responsible for notifying their progress as they process each event. This notifications are then aggregated via counters & gauges and exposed via HTTP using the well-known Prometheus encoding.
+
+The following metrics are available:
+
+- `chain_tip`: the last detected tip of the chain (height)
+- `rollback_count`: number of rollback events occurred
+- `source_current_slot`: last slot processed by the source of the pipeline
+- `source_current_height`: last height (block #) processed by the source of the pipeline
+- `source_event_count`: number of events processed by the source of the pipeline
+- `sink_current_slot`: last slot processed by the sink of the pipeline
+- `sink_event_count`: number of events processed by the sink of the pipeline
+
+## Configuration
+
+The _metrics_ feature is a configurable setting available when running in daemon mode. A top level `[metrics]` section of the daemon toml file controls the feature:
+
+```toml
+# daemon.toml file
+
+[metrics]
+address = "0.0.0.0:9186"
+endpoint = "/metrics"
+```
+
+- `[metrics]` section needs to be present to enable the feature. Absence of the section will not expose any HTTP endpoints.
+- `address`: The address at which the HTTP server will be listening for request. Expected format is `<ip>:<port>`. Use the IP value `0.0.0.0` to allow connections on any of the available IP address of the network interface. Default value is `0.0.0.0:9186`.
+- `endpoint`: The path at which the metrics will be exposed. Default value is `/metrics`.
+
+## Usage
+
+Once enabled, a quick method to check the metrics output is to navigate to the HTTP endpoint using any common browser. A local instance of Oura with metrics enabled on port `9186` can be accessed by opening the URL http://localhost:9186
+
+An output similar to the following should be shown by the browser:
+
+```
+# HELP chain_tip the last detected tip of the chain (height)
+# TYPE chain_tip gauge
+chain_tip 6935733
+# HELP rollback_count number of rollback events occurred
+# TYPE rollback_count counter
+rollback_count 1
+# HELP sink_current_slot last slot processed by the sink of the pipeline
+# TYPE sink_current_slot gauge
+sink_current_slot 2839340
+# HELP sink_event_count number of events processed by the sink of the pipeline
+# TYPE sink_event_count counter
+sink_event_count 2277714
+# HELP source_current_height last height (block #) processed by the source of the pipeline
+# TYPE source_current_height gauge
+source_current_height 2837810
+# HELP source_current_slot last slot processed by the source of the pipeline
+# TYPE source_current_slot gauge
+source_current_slot 2839340
+# HELP source_event_count number of events processed by the source of the pipeline
+# TYPE source_event_count counter
+source_event_count 2277715
+```
+
+Regardless of the above mechanism, the inteded approach for tracking Oura's metrics is to use a monitoring infrastructure compatible with Prometheus format. Setting up and managing the monitoring stack is outside the scope of Oura. If you don't have any infrastructure in place, we recommend checking out some of the more commons stacks:
+
+- Prometheus Server + Grafana
+- Metricbeat + Elasticsearch + Kibana
+- Telegraf + InfluxDB + Chronograf
+

docs/v3/advanced/pipeline_metrics.mdx

@@ -0,0 +1,23 @@
+---
+title: Retry Policy
+---
+
+Advanced options for instructing Oura how to deal with failed attempts.
+
+## Configuration
+
+To modify the default behavior, a section named `[retries]` needs to be added to the `daemon.toml` file.
+
+```toml
+[retries]
+max_retries = 3
+backoff_unit_sec = 10
+backoff_factor = 3
+max_backoff_sec = 10
+dismissible = true
+```
+
+- `max_retries`: the max number of retries before failing the whole pipeline.
+- `backoff_unit_sec`: the delay expressed in seconds between each retry.
+- `backoff_factor`: the amount to increase the backoff delay after each attempt.
+- `max_backoff_sec`: the longest possible delay in seconds.

docs/v3/advanced/retry_policy.mdx

@@ -0,0 +1,35 @@
+---
+title: Stateful Cursor
+---
+
+The _cursor_ feature provides a mechanism to persist the "position" of the processing pipeline to make it resilient to restarts.
+
+## Context
+
+When running in daemon mode, a restart of the process will make Oura start crawling from the initially configured point. This default behavior can be problematic depending on the use-case of the operator.
+
+An scenario where continuous "coverage" of all the processed blocks is important (eg: building a stateful view of the chain), it's prohibitive to have "gaps" while procesing. If Oura is configure to start from the "tip" of the chain, a restart of the process might miss some blocks during the bootstrap procedure.
+
+A valid workaround to the above problem is to configure Oura to start from a fixed point in the chain. If a restart occurs, the pipeline will re-process the blocks, ensuring that each block was processed at least once. When working with sinks that implement idempotency when processing an event, receiving data from the same block multiple times should not impose a problem.
+
+Although valid, the workaround described above is very inefficient. If the fixed point at which the pipeline starts is too far behind, catching up could take several hours, wasting time and resource.
+
+## Feature
+
+Oura implements an optional stateful cursor that receives notifications from the sink stage of the pipeline to continuously track the current position of the chain. At certain checkpoints (every 10 secs by default), the position is persisted onto the file system at a configurable location.
+
+Assuming that a restart occurs and the cursor feature is enabled, the process will attempt to locate and load the persisted value and instruct the source stage to begin reading chain data from the last known position. 
+
+## Configuration
+
+The _cursor_ feature is a configurable setting available when running in daemon mode. A top level `[cursor]` section of the daemon toml file controls the feature:
+
+```toml
+[cursor]
+type = "File"
+path = "/var/oura/cursor"
+```
+
+- `[cursor]`: The presence of this section in the toml file indicates Oura to enable the _cursor_ feature.
+- `type`: The type of persistence backend to use for storing the state of the cursor. The only available option at the moment is `File`, which stores the cursor in the file system.
+- `path`: The location of the cursor file within the file system. Default value is `var/oura/cursor`.