Near Cache ignores entries read from back-up partitions

[mnovoseltsev]

We have a simple 3 nodes cluster and we wanted to move read heavy data to ignite and use near cache to optimize cache read operations. After PoC with near caches we noticed that requested item not always saved in near cache even though near cache limit is not yet reached. I have run a simple scenario:
Test scenario:

1. Simple 3 nodes ignite cluster on bare metal hosts with mostly default configs:
   • Ignite version 2.16 & 2.17
   • Static discovery SPI of three IPs & ports
   • Static communication SPI
   • Discovery and Cache events are enabled
   • Custom storage path for persistent regions
   • two regions defined: default one is in memory:
     • in memory with initial and max size set to 356/1024 MB and RANDOM_2_LRU eviction
     • persistent one with initial and max size set to 64/512MB
2. Having near cache created with limit of 100 items and FIFO expiry
3. Request 100 unique items from cache (items saved in cache from different client)
4. Check near cache state with peaking it explicitly and simple get operation

Expected behavior:

• Every time item requested by key and absent from local near cache, it gets added to it. So that next request should always read it from local instance (considering it is not evicted in the meantime)
• Every update for locally stored item also reflected in local near cache

I found the following behavior:

• Having partitioned cache with zero back-ups - everything works as expected
• Having partitioned cache with two copies - only around 30% of requested items saved in cache. It is a complete random when item might be added to local cache, it can be after 1 or after 10 requests
  • when after couple of rounds of request of the same set of keys cache finally gets full, switch to another set of keys causes eviction of items but new keys again not always added, so local cache shrinks
• Having replicated cache has same results on the same 3 server nodes set-up
• Having partitioned cache with back-ups with readFromBackup flag set to false - works as expected
• Having replicated cache with copies with readFromBackup flag set to false - properly saves to near cache but never reads from it when using plain get method. But it works fine when using explicit peak into near cache
• In each scenario near cache properly saves items that are created from the given client
• All scenarios work fine when all 3 servers and 1 client nodes are collocates on single host
• All scenarios have the same results for in-memory or persistent caches

So my assumption is that near cache for whatever reason not saving entries read from back-up partitions.
I have reviewed ignite documentation and haven't found any specific config required for Near Cache to work. I wonder if you can help me to identify what is wrong.

Sample cluster server config:

        IgniteConfiguration config = new IgniteConfiguration();

        TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder()
                .setAddresses(List.of("<host1.ip>:22222", "host2.ip:22222", "host3.ip:22222"));

        config.setDiscoverySpi(new TcpDiscoverySpi()
                .setIpFinder(ipFinder)
                .setLocalPort(22222)
                .setLocalAddress(ipAddress)
                .setLocalPortRange(0)
        );

        config.setCommunicationSpi(new TcpCommunicationSpi()
                .setLocalPortRange(0)
                .setLocalAddress(ipAddress)
        );

        config.getDataStorageConfiguration()
                .setDefaultDataRegionConfiguration(
                        new DataRegionConfiguration()
                                .setInitialSize(356 * 1024 * 1024)
                                .setMaxSize(1024 * 1024 * 1024)
                                .setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU)
                ).setDataRegionConfigurations(
                        new DataRegionConfiguration()
                                .setName("persistence")
                                .setInitialSize(64* 1024 * 1024)
                                .setMaxSize(512* 1024 * 1024)
                                .setPersistenceEnabled(true)
                );

Sample cache config

        NearCacheConfiguration<Integer, String> nearCacheConfiguration = new NearCacheConfiguration<>();
        nearCacheConfiguration.setNearEvictionPolicyFactory(new FifoEvictionPolicyFactory<>(100));

        CacheConfiguration<Integer, String> cacheConfiguration = new CacheConfiguration<>(cacheName);
        cacheConfiguration
                .setCacheMode(CacheMode.PARTITIONED)
                .setBackups(2)
                .setDataRegionName("persistence")
                .setNearConfiguration(nearCacheConfiguration);

Depends if cache is already present or a not we use ignite.createCache or ignite.createNearCache