reap_conjs failing with WRONGTYPE Operation against a key holding the wrong kind of value

[gmuj]

Calling reap_conjs will fail because of wrong type in the redis keys conj:*.

It looks like the values stored in those keys are strings

In [14]: redis_client.type(conj_key)
Out[14]: b'string'

In [15]: redis_client.get(conj_key)
Out[15]: b'0.01411662170257244'

The code inside reap_conjs is using total = redis_client.scard(conj_key) and by checking the redis doc this is expecting a set - redis scard

Is there a workaround for this?

Traceback error:

In [1]: from cacheops import reap_conjs
   ...: 

In [2]: reap_conjs(
   ...:     chunk_size=1000,
   ...:     min_conj_set_size=500,
   ...: )
---------------------------------------------------------------------------
ResponseError                             Traceback (most recent call last)
Cell In[2], line 1
----> 1 reap_conjs(
      2     chunk_size=1000,
      3     min_conj_set_size=500,
      4 )

File /.venv/lib/python3.11/site-packages/cacheops/reaper.py:34, in reap_conjs(chunk_size, min_conj_set_size, using, dry_run)
     32 prefix = get_prefix(dbs=[using])
     33 for conj_key in redis_client.scan_iter(prefix + 'conj:*', count=chunk_size):
---> 34     total = redis_client.scard(conj_key)
     35     if total < min_conj_set_size:
     36         continue

File /.venv/lib/python3.11/site-packages/redis/commands/core.py:3309, in SetCommands.scard(self, name)
   3303 def scard(self, name: str) -> Union[Awaitable[int], int]:
   3304     """
   3305     Return the number of elements in set ``name``
   3306 
   3307     For more information see https://redis.io/commands/scard
   3308     """
-> 3309     return self.execute_command("SCARD", name)

File /.venv/lib/python3.11/site-packages/elasticapm/instrumentation/packages/base.py:208, in AbstractInstrumentedModule.call_if_sampling(self, module, method, wrapped, instance, args, kwargs)
    206 transaction = execution_context.get_transaction()
    207 if not transaction:
--> 208     return wrapped(*args, **kwargs)
    209 elif not transaction.is_sampled or transaction.pause_sampling:
    210     args, kwargs = self.mutate_unsampled_call_args(module, method, wrapped, instance, args, kwargs, transaction)

File /.venv/lib/python3.11/site-packages/sentry_sdk/integrations/redis/_sync_common.py:103, in patch_redis_client.<locals>.sentry_patched_execute_command(self, name, *args, **kwargs)
    100 set_db_data_fn(db_span, self)
    101 _set_client_data(db_span, is_cluster, name, *args)
--> 103 value = old_execute_command(self, name, *args, **kwargs)
    105 db_span.__exit__(None, None, None)
    107 if cache_span:

File /.venv/lib/python3.11/site-packages/redis/client.py:1269, in Redis.execute_command(self, *args, **options)
   1266 conn = self.connection or pool.get_connection(command_name, **options)
   1268 try:
-> 1269     return conn.retry.call_with_retry(
   1270         lambda: self._send_command_parse_response(
   1271             conn, command_name, *args, **options
   1272         ),
   1273         lambda error: self._disconnect_raise(conn, error),
   1274     )
   1275 finally:
   1276     if not self.connection:

File /.venv/lib/python3.11/site-packages/redis/retry.py:46, in Retry.call_with_retry(self, do, fail)
     44 while True:
     45     try:
---> 46         return do()
     47     except self._supported_errors as error:
     48         failures += 1

File /.venv/lib/python3.11/site-packages/redis/client.py:1270, in Redis.execute_command.<locals>.<lambda>()
   1266 conn = self.connection or pool.get_connection(command_name, **options)
   1268 try:
   1269     return conn.retry.call_with_retry(
-> 1270         lambda: self._send_command_parse_response(
   1271             conn, command_name, *args, **options
   1272         ),
   1273         lambda error: self._disconnect_raise(conn, error),
   1274     )
   1275 finally:
   1276     if not self.connection:

File /.venv/lib/python3.11/site-packages/redis/client.py:1246, in Redis._send_command_parse_response(self, conn, command_name, *args, **options)
   1242 """
   1243 Send a command and parse the response
   1244 """
   1245 conn.send_command(*args)
-> 1246 return self.parse_response(conn, command_name, **options)

File /.venv/lib/python3.11/site-packages/redis/client.py:1286, in Redis.parse_response(self, connection, command_name, **options)
   1284         options.pop(NEVER_DECODE)
   1285     else:
-> 1286         response = connection.read_response()
   1287 except ResponseError:
   1288     if EMPTY_RESPONSE in options:

File /.venv/lib/python3.11/site-packages/redis/connection.py:905, in AbstractConnection.read_response(self, disable_decoding, disconnect_on_error)
    902     self.next_health_check = time() + self.health_check_interval
    904 if isinstance(response, ResponseError):
--> 905     raise response
    906 return response

ResponseError: WRONGTYPE Operation against a key holding the wrong kind of value

[Suor]

Conj keys are sets in default mode, and under particular circumstances these sets can grow big slowing down subsequent invalidation and taking memory. reap_conjs is designed to remove stale keys from those sets.

In insideout mode conj keys are simple random floats like you have. In this mode reap_conjs command is not needed and won't work.

If you have memory issues with insideout mode you should set up redis maxmemory setting to evict some cache keys automatically.