Skip to content

Rework generation of the VID with the Entity wrapper #5691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 129 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
129 commits
Select commit Hold shift + click to select a range
3fe298e
graph: Add support for subgraph datasource in manifest
incrypto32 Jul 11, 2024
a45daf3
graph: wrap TriggerFilter with TriggerFilterWrapper
incrypto32 Jul 12, 2024
af3468c
graph,chain: add build_subgraph_block_stream method
incrypto32 Jul 12, 2024
d59c14b
graph,core,chain: use TriggerFilterWrapper in PollingBlockStream
incrypto32 Jul 15, 2024
edc806c
graph: created TriggersAdapterWrapper
incrypto32 Jul 15, 2024
eb41c6b
graph,core,chain: Add a wrapper enum for Triggers to handle subgraph …
incrypto32 Jul 16, 2024
3710cf8
graph, chain: Build subgraph trigger filters in build_filter
incrypto32 Jul 16, 2024
579d886
graph,core: Add subgraph_hosts to RuntimeHostBuilder
incrypto32 Jul 17, 2024
cb4d082
chain, core, graph: Add source_subgraph_stores to indexing inputs
incrypto32 Jul 18, 2024
ff572cf
graph, chain: minor refactoring and formatting
incrypto32 Jul 18, 2024
35b2a54
graph, chain: fix typo in TriggersAdapterWrapper
incrypto32 Jul 18, 2024
fe5d5ba
chain, graph: use TriggerFilterWrapper in scan_triggers
incrypto32 Jul 18, 2024
4c05119
chain/ethereum: implement load_blocks_by_numbers for EthereumAdapter
incrypto32 Jul 24, 2024
d20bafd
graph: refactor BlockWithTriggers impl
incrypto32 Jul 24, 2024
3ab4941
graph, core: Add a new SubgraphFilter struct
incrypto32 Jul 24, 2024
aec82dc
chain/ethereum: Mock implementation of subgraph_triggers for ethereum
incrypto32 Jul 24, 2024
ae7559d
chain,graph : use `chain_head_ptr` method from adapter
incrypto32 Jul 24, 2024
686e7e1
chain, core, graph : use TriggersAdapterWrapper at top level
incrypto32 Jul 25, 2024
3aba98f
chain, graph : move subgraph trigger scanning back to TriggersAdapter…
incrypto32 Jul 25, 2024
9069486
graph: use ReadStore instead of WriteStore for source subgraph stores
incrypto32 Nov 12, 2024
c7ca25a
graph: Better comments
incrypto32 Nov 12, 2024
1dab182
first attempt at reading entities for mutable ones
zorancv Jul 17, 2024
f931d37
remove the key as a parameter
zorancv Jul 18, 2024
a6e4b23
fix regex
zorancv Jul 24, 2024
b7bc168
proper return type
zorancv Jul 30, 2024
5903ee9
support immutable entities
zorancv Aug 2, 2024
57e1463
add test for immutable entities
zorancv Aug 5, 2024
553fb82
store: fix block range query bug when getting entity triggers
incrypto32 Jul 31, 2024
4fc6b3c
fix test
zorancv Aug 5, 2024
441300a
refactor
zorancv Aug 8, 2024
239c6cb
change the range type
zorancv Aug 12, 2024
48accf7
test multiple entities
zorancv Aug 15, 2024
9acd51e
address review
zorancv Nov 13, 2024
e25e9bf
remove get_range from the Queue and Writer
zorancv Nov 15, 2024
0d7e4d5
remove get_range() from WritableStore
zorancv Nov 28, 2024
38c4286
replace ReadSTore with SourceableStore
zorancv Dec 3, 2024
f57b3ba
add block tracking
zorancv Dec 3, 2024
77850be
compiler fix
zorancv Dec 4, 2024
8878caf
cleanup WritableStore
zorancv Dec 5, 2024
70ca609
fetch block from DB
zorancv Dec 6, 2024
0a4a2a6
address PR comments
zorancv Dec 9, 2024
c6e4b36
tests: Add integration test for subgraph datasources
incrypto32 Jul 19, 2024
3ac0c1a
tests: change entity name for integration test
incrypto32 Jul 24, 2024
331ab3f
read entities from database
zorancv Jul 25, 2024
08f17c0
clean up
zorancv Jul 29, 2024
142ce94
simplify
zorancv Jul 29, 2024
8f51340
remove mocks
zorancv Jul 30, 2024
94cbf59
fixes
zorancv Jul 30, 2024
f17fdec
fixes
zorancv Aug 13, 2024
2e96b73
fix
zorancv Dec 5, 2024
3814175
hash removal fix
zorancv Dec 9, 2024
882a111
graph: subgraph_triggers to take block_number hashset
incrypto32 Jul 31, 2024
fc239b8
graph: refactor TriggersAdapterWrapper.scan_triggers
incrypto32 Jul 31, 2024
27d3c1f
graph: fix manifest parsing of source subgraphs
incrypto32 Aug 1, 2024
72a16ca
tests: fix runner tests
incrypto32 Aug 1, 2024
2deaec4
graph: TriggersAdapterWrapper.chain_head_ptr to consider head of sou…
incrypto32 Aug 1, 2024
48b7ffe
chain, core, graph : Refactor subgraph trigger creation
incrypto32 Aug 12, 2024
c4ea03f
graph: refactor TriggersAdapterWrapper.triggers_in_block to not rely …
incrypto32 Aug 12, 2024
58b9231
graph: fix `get_entities_for_range` bug which didn't return all entit…
incrypto32 Aug 16, 2024
60e9d47
tests for multiple entities
incrypto32 Aug 16, 2024
3fb0e72
graph: fix subgraph filter mismatch bug
incrypto32 Aug 22, 2024
c47236c
graph: Avoid unwraps in `blocks_with_subgraph_triggers`
incrypto32 Nov 14, 2024
a60c653
fix futures
zorancv Dec 6, 2024
37bf3cd
rebase fixes for vec
zorancv Dec 9, 2024
5eb3c9a
request all entity types in a single SQL query
zorancv Aug 21, 2024
7561307
add types to the enities
zorancv Aug 23, 2024
b69e99a
query for both lower and upper range
zorancv Aug 28, 2024
b675b16
deduce the types of operation create, modify or delete
zorancv Sep 2, 2024
fae61ee
add vid to the result struct in order to sort it afterwards
zorancv Sep 3, 2024
d0d1bb0
better test
zorancv Sep 4, 2024
5309219
Add casuality region to the SQL querry
zorancv Sep 6, 2024
fbc5313
fix the empty select bug
zorancv Nov 4, 2024
30a5e59
fixes
zorancv Dec 5, 2024
ebfe059
fix return
zorancv Dec 6, 2024
501c8ba
address review
zorancv Dec 9, 2024
4136889
substitute if with match
zorancv Dec 9, 2024
940f4f6
better comment
zorancv Dec 10, 2024
ff19081
graph, runtime: Support subgraph entity operation detection in compos…
incrypto32 Sep 9, 2024
a6eb97e
tests: udpate subgraph composition integration test cases to handle e…
incrypto32 Sep 9, 2024
8933a8f
chain/ethereum, graph, store: use block cache to load blocks by numbers
incrypto32 Aug 19, 2024
33e5784
chain/ethereum: Better logs when block is loaded from RPC
incrypto32 Aug 29, 2024
f83723b
graph, node: Allow rpc clients when using firehose
incrypto32 Sep 2, 2024
82c1fbb
store: improve comments in `ChainStore.blocks()`
incrypto32 Nov 19, 2024
74bb0a9
chain/ethereum: remove rpc client from firehose chain clientff
incrypto32 Nov 25, 2024
2063909
graph, chain/ethereum: use block ptrs when fetching blocks for compos…
incrypto32 Sep 2, 2024
e6d304f
graph: implement deserialize for BlockPtrExt and add tests
incrypto32 Sep 5, 2024
b00658e
chain/ethereum: Add block_ptr_batch_size env variable
incrypto32 Sep 5, 2024
d1109ac
chain/ethereum: bump block range size for composed subgraphs
incrypto32 Sep 5, 2024
ae66751
chain, graph: Fix block ptr ext parsing bug
incrypto32 Oct 7, 2024
3bd36f6
chain, graph: rename BlockPtrExt to ExtendedBlockPtr
incrypto32 Nov 19, 2024
1511cfa
chain,graph: use firehose get_block to get block ptrs for subgraph tr…
incrypto32 Nov 13, 2024
196f2a5
graph, chain: make TriggersAdapterTrait methods work with firehose en…
incrypto32 Nov 15, 2024
8b86d0b
chain/ethereum: Refactor load_block_ptrs_by_numbers
incrypto32 Nov 18, 2024
7858651
graph: implement custom deserializer for BlockTime
incrypto32 Nov 18, 2024
9943058
graph: mock chain store
incrypto32 Nov 22, 2024
a1ed8b2
chain/ethereum: refactor load_block_ptrs_by_numbers and add unit tests
incrypto32 Nov 22, 2024
e0a65dd
graph: enable ethereum host fns for subgraph datasource
incrypto32 Oct 31, 2024
8b12d10
graph, chain/ethereum: Move declared call related structs to graph crate
incrypto32 Dec 3, 2024
9ead743
graph, chain/ethreum: add FindMappingABI trait and use it to refactor…
incrypto32 Dec 3, 2024
2f52600
graph: Refactor CallArg to include subgraph datasource specific args
incrypto32 Dec 3, 2024
00c78d9
graph, chain/ethereum: Move DeclaredCalls struct to graph crate
incrypto32 Dec 3, 2024
6b2d28a
graph: Refactor DeclareCall for readability
incrypto32 Dec 3, 2024
fe391b4
graph: Add methods to handle declared calls in subgraph datasource
incrypto32 Dec 3, 2024
fc7ab82
graph, runtime: Add new MappingEntityTrigger struct that takes declar…
incrypto32 Dec 3, 2024
b93c3ae
chain/ethereum: Handle subgraph datasource triggers also in `DecoderH…
incrypto32 Dec 3, 2024
56f041a
store/test-store: Add a test for declared eth calls parsing in manifest
incrypto32 Dec 3, 2024
632ef9f
renames
zorancv Dec 10, 2024
7cc896b
add vid
zorancv Oct 4, 2024
02e5fb4
fixes
zorancv Oct 6, 2024
8cac2e7
more test fixes
zorancv Oct 8, 2024
66743bf
fix entity_cache tests
zorancv Oct 8, 2024
74086b4
fix graphql_query tests
zorancv Oct 9, 2024
b871eb5
fix postgress_writable tests
zorancv Oct 10, 2024
213924c
fix more tests
zorancv Oct 10, 2024
cdcc929
fix relational tests
zorancv Oct 11, 2024
3ce43c5
fix graft test
zorancv Oct 11, 2024
a537a23
try to fix graft prune test
zorancv Oct 14, 2024
4c959f6
fix runner test data_source_revert
zorancv Oct 22, 2024
6fc2362
try to remove optional vid fetch
zorancv Oct 25, 2024
dd40887
fix
zorancv Oct 28, 2024
78872dc
add wrapping
zorancv Oct 31, 2024
e7d2c86
cleanup
zorancv Nov 6, 2024
4023edf
cleanup
zorancv Nov 18, 2024
b8a47e2
cleanup
zorancv Nov 18, 2024
c9df3db
correct vid
zorancv Nov 19, 2024
7cee434
cleanup
zorancv Nov 19, 2024
0189010
cleanup
zorancv Nov 19, 2024
270356d
bump the spec version
zorancv Nov 22, 2024
5910d01
simplify
zorancv Dec 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 22 additions & 4 deletions chain/arweave/src/chain.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ use graph::blockchain::client::ChainClient;
use graph::blockchain::firehose_block_ingestor::FirehoseBlockIngestor;
use graph::blockchain::{
BasicBlockchainBuilder, Block, BlockIngestor, BlockchainBuilder, BlockchainKind,
EmptyNodeCapabilities, NoopDecoderHook, NoopRuntimeAdapter,
EmptyNodeCapabilities, NoopDecoderHook, NoopRuntimeAdapter, TriggerFilterWrapper,
};
use graph::cheap_clone::CheapClone;
use graph::components::adapter::ChainId;
use graph::components::store::DeploymentCursorTracker;
use graph::components::store::{DeploymentCursorTracker, SourceableStore};
use graph::data::subgraph::UnifiedMappingApiVersion;
use graph::env::EnvVars;
use graph::firehose::FirehoseEndpoint;
Expand All @@ -27,11 +27,13 @@ use graph::{
prelude::{async_trait, o, BlockNumber, ChainStore, Error, Logger, LoggerFactory},
};
use prost::Message;
use std::collections::HashSet;
use std::sync::Arc;

use crate::adapter::TriggerFilter;
use crate::data_source::{DataSourceTemplate, UnresolvedDataSourceTemplate};
use crate::trigger::{self, ArweaveTrigger};
use crate::Block as ArweaveBlock;
use crate::{
codec,
data_source::{DataSource, UnresolvedDataSource},
Expand Down Expand Up @@ -119,7 +121,8 @@ impl Blockchain for Chain {
deployment: DeploymentLocator,
store: impl DeploymentCursorTracker,
start_blocks: Vec<BlockNumber>,
filter: Arc<Self::TriggerFilter>,
_source_subgraph_stores: Vec<Arc<dyn SourceableStore>>,
filter: Arc<TriggerFilterWrapper<Self>>,
unified_api_version: UnifiedMappingApiVersion,
) -> Result<Box<dyn BlockStream<Self>>, Error> {
let adapter = self
Expand All @@ -135,7 +138,10 @@ impl Blockchain for Chain {
.subgraph_logger(&deployment)
.new(o!("component" => "FirehoseBlockStream"));

let firehose_mapper = Arc::new(FirehoseMapper { adapter, filter });
let firehose_mapper = Arc::new(FirehoseMapper {
adapter,
filter: filter.chain_filter.clone(),
});

Ok(Box::new(FirehoseBlockStream::new(
deployment.hash,
Expand Down Expand Up @@ -199,6 +205,10 @@ impl TriggersAdapterTrait<Chain> for TriggersAdapter {
panic!("Should never be called since not used by FirehoseBlockStream")
}

async fn chain_head_ptr(&self) -> Result<Option<BlockPtr>, Error> {
unimplemented!()
}

async fn triggers_in_block(
&self,
logger: &Logger,
Expand Down Expand Up @@ -258,6 +268,14 @@ impl TriggersAdapterTrait<Chain> for TriggersAdapter {
number: block.number.saturating_sub(1),
}))
}

async fn load_block_ptrs_by_numbers(
&self,
_logger: Logger,
_block_numbers: HashSet<BlockNumber>,
) -> Result<Vec<ArweaveBlock>, Error> {
todo!()
}
}

pub struct FirehoseMapper {
Expand Down
39 changes: 30 additions & 9 deletions chain/cosmos/src/chain.rs
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
use graph::blockchain::firehose_block_ingestor::FirehoseBlockIngestor;
use graph::blockchain::{BlockIngestor, NoopDecoderHook};
use graph::blockchain::{BlockIngestor, NoopDecoderHook, TriggerFilterWrapper};
use graph::components::adapter::ChainId;
use graph::env::EnvVars;
use graph::prelude::MetricsRegistry;
use graph::substreams::Clock;
use std::collections::HashSet;
use std::convert::TryFrom;
use std::sync::Arc;

use graph::blockchain::block_stream::{BlockStreamError, BlockStreamMapper, FirehoseCursor};
use graph::blockchain::client::ChainClient;
use graph::blockchain::{BasicBlockchainBuilder, BlockchainBuilder, NoopRuntimeAdapter};
use graph::cheap_clone::CheapClone;
use graph::components::store::DeploymentCursorTracker;
use graph::components::store::{DeploymentCursorTracker, SourceableStore};
use graph::data::subgraph::UnifiedMappingApiVersion;
use graph::{
blockchain::{
Expand All @@ -33,7 +34,7 @@ use crate::data_source::{
DataSource, DataSourceTemplate, EventOrigin, UnresolvedDataSource, UnresolvedDataSourceTemplate,
};
use crate::trigger::CosmosTrigger;
use crate::{codec, TriggerFilter};
use crate::{codec, Block, TriggerFilter};

pub struct Chain {
logger_factory: LoggerFactory,
Expand Down Expand Up @@ -113,7 +114,8 @@ impl Blockchain for Chain {
deployment: DeploymentLocator,
store: impl DeploymentCursorTracker,
start_blocks: Vec<BlockNumber>,
filter: Arc<Self::TriggerFilter>,
_source_subgraph_stores: Vec<Arc<dyn SourceableStore>>,
filter: Arc<TriggerFilterWrapper<Self>>,
unified_api_version: UnifiedMappingApiVersion,
) -> Result<Box<dyn BlockStream<Self>>, Error> {
let adapter = self
Expand All @@ -129,7 +131,10 @@ impl Blockchain for Chain {
.subgraph_logger(&deployment)
.new(o!("component" => "FirehoseBlockStream"));

let firehose_mapper = Arc::new(FirehoseMapper { adapter, filter });
let firehose_mapper = Arc::new(FirehoseMapper {
adapter,
filter: filter.chain_filter.clone(),
});

Ok(Box::new(FirehoseBlockStream::new(
deployment.hash,
Expand Down Expand Up @@ -192,6 +197,16 @@ impl TriggersAdapterTrait<Chain> for TriggersAdapter {
) -> Result<Option<codec::Block>, Error> {
panic!("Should never be called since not used by FirehoseBlockStream")
}
async fn load_block_ptrs_by_numbers(
&self,
_logger: Logger,
_block_numbers: HashSet<BlockNumber>,
) -> Result<Vec<Block>, Error> {
todo!()
}
async fn chain_head_ptr(&self) -> Result<Option<BlockPtr>, Error> {
unimplemented!()
}

async fn scan_triggers(
&self,
Expand Down Expand Up @@ -467,9 +482,12 @@ impl FirehoseMapperTrait<Chain> for FirehoseMapper {

#[cfg(test)]
mod test {
use graph::prelude::{
slog::{o, Discard, Logger},
tokio,
use graph::{
blockchain::Trigger,
prelude::{
slog::{o, Discard, Logger},
tokio,
},
};

use super::*;
Expand Down Expand Up @@ -600,7 +618,10 @@ mod test {
// they may not be in the same order
for trigger in expected_triggers {
assert!(
triggers.trigger_data.contains(&trigger),
triggers.trigger_data.iter().any(|t| match t {
Trigger::Chain(t) => t == &trigger,
_ => false,
}),
"Expected trigger list to contain {:?}, but it only contains: {:?}",
trigger,
triggers.trigger_data
Expand Down
13 changes: 2 additions & 11 deletions chain/ethereum/src/adapter.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
use anyhow::Error;
use ethabi::{Error as ABIError, Function, ParamType, Token};
use ethabi::{Error as ABIError, ParamType, Token};
use graph::blockchain::ChainIdentifier;
use graph::components::subgraph::MappingError;
use graph::data::store::ethereum::call;
use graph::data_source::common::ContractCall;
use graph::firehose::CallToFilter;
use graph::firehose::CombinedFilter;
use graph::firehose::LogFilter;
Expand Down Expand Up @@ -93,16 +94,6 @@ impl EventSignatureWithTopics {
}
}

#[derive(Clone, Debug)]
pub struct ContractCall {
pub contract_name: String,
pub address: Address,
pub block_ptr: BlockPtr,
pub function: Function,
pub args: Vec<Token>,
pub gas: Option<u32>,
}

#[derive(Error, Debug)]
pub enum EthereumRpcError {
#[error("call error: {0}")]
Expand Down
Loading
Loading