Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update packaging + CI, switch aioredis -> redis #13

Merged
merged 3 commits into from
Jun 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 0 additions & 69 deletions .github/workflows/build_deploy.yml

This file was deleted.

72 changes: 23 additions & 49 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,63 +1,37 @@
name: CI
on: [push, pull_request]

on:
push:
pull_request:

jobs:
black:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.9"
- run: pip install riot==0.19.0
- run: riot -v run -s black -- --check .
mypy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.9"
- run: pip install riot==0.19.0
- run: riot -v run mypy
flake8:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.9"
- run: pip install riot==0.19.0
- run: riot -v run flake8
- run: pip install uv
- run: uv venv
- run: uv pip install --requirement pyproject.toml --all-extras
- run: .venv/bin/ruff format --check .
- run: .venv/bin/ruff check .
- run: .venv/bin/mypy asyncio_connection_pool

test:
runs-on: ubuntu-latest
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: install riot
run: pip install riot==0.19.0
- name: run tests
run: riot -v run --python=${{ matrix.python-version }} test -- --cov=asyncio_connection_pool --cov-branch --cov-config=.coveragerc
- name: install coverage
run: pip install coverage
- name: upload coverage
uses: coverallsapp/github-action@v2
with:
parallel: true
flag-name: run ${{ join(matrix.*, ' - ') }}
finish-coveralls:
needs: test
if: ${{ always() }}
runs-on: ubuntu-latest
steps:
- name: Coveralls Finished
uses: coverallsapp/github-action@v2
with:
parallel-finished: true
allow-empty: true
- run: pip install uv
- run: uv venv
- run: uv pip install --editable ".[dev,datadog,aioredis]"
- run: .venv/bin/pytest --cov=asyncio_connection_pool --cov-branch --cov-config=.coveragerc
- run: .venv/bin/codecov
if: matrix.python-version == '3.11'
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
25 changes: 25 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: Release

on:
release:
types:
- created

jobs:
build:
runs-on: ubuntu-latest
permissions:
# IMPORTANT: this permission is mandatory for trusted publishing
id-token: write
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install uv
- run: uv venv
- run: uv pip install --requirement pyproject.toml
- run: uv pip install setuptools setuptools-scm wheel build
- run: .venv/bin/python -m build --no-isolation
- name: Publish package distributions to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
27 changes: 10 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
# asyncio-connection-pool

[![GitHub Workflow Status (main)](https://img.shields.io/github/workflow/status/fellowinsights/asyncio-connection-pool/CI/main?style=flat)][main CI]
[![GitHub Workflow Status (main)](https://img.shields.io/github/actions/workflow/status/fellowapp/asyncio-connection-pool/ci.yml?branch=main&style=flat)][main CI]
[![PyPI](https://img.shields.io/pypi/v/asyncio-connection-pool?style=flat)][package]
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/asyncio-connection-pool?style=flat)][package]
[![codecov](https://codecov.io/gh/fellowapp/asyncio-connection-pool/graph/badge.svg?token=F3D4D9EG6M)](https://codecov.io/gh/fellowapp/asyncio-connection-pool)
[![License](https://img.shields.io/pypi/l/prosemirror.svg?style=flat)](https://github.com/fellowapp/asyncio-connection-pool/blob/master/LICENSE.md)
[![Fellow Careers](https://img.shields.io/badge/fellow.app-hiring-576cf7.svg?style=flat)](https://fellow.app/careers/)

[main CI]: https://github.com/fellowinsights/asyncio-connection-pool/actions?query=workflow%3ACI+branch%3Amain
[main CI]: https://github.com/fellowapp/asyncio-connection-pool/actions?query=workflow%3ACI+branch%3Amain
[package]: https://pypi.org/project/asyncio-connection-pool/

This is a generic, high-throughput, optionally-burstable pool for asyncio.
Expand All @@ -21,10 +23,11 @@ Some cool features:
- The contents of the pool can be anything; just implement a
`ConnectionStrategy`.

[^1]: Theoretically, there is an implicit "lock" that is held while an asyncio
task is executing. No other task can execute until the current task
yields (since it's cooperative multitasking), so any operations during
that time are atomic.
[^1]:
Theoretically, there is an implicit "lock" that is held while an asyncio
task is executing. No other task can execute until the current task
yields (since it's cooperative multitasking), so any operations during
that time are atomic.

## Why?

Expand All @@ -36,10 +39,8 @@ We also thought it would be nice if we didn't need to keep many connections
open when they weren't needed, but still have the ability to make more when
they are required.


## API


### `asyncio_connection_pool.ConnectionPool`

This is the implementation of the pool. It is generic over a type of
Expand All @@ -57,7 +58,6 @@ pool = ConnectionPool(strategy=my_strategy, max_size=15)
The constructor can optionally be passed an integer as `burst_limit`. This
allows the pool to open more connections than `max_size` temporarily.


#### `@asynccontextmanager async def get_connection(self) -> AsyncIterator[Conn]`

This method is the only way to get a connection from the pool. It is expected
Expand All @@ -77,13 +77,11 @@ are available, the caller will yield to the event loop.

When the block is exited, the connection will be returned to the pool.


### `asyncio_connection_pool.ConnectionStrategy`

This is an abstract class that defines the interface of the object passed as
`strategy`. A subclass _must_ implement the following methods:


#### `async def create_connection(self) -> Awaitable[Conn]`

This method is called to create a new connection to the resource. This happens
Expand All @@ -96,7 +94,6 @@ the pool, and in most cases will be stored in the pool to be re-used later.
If this method raises an exception, it will bubble up to the frame where
`ConnectionPool.get_connection()` was called.


#### `def connection_is_closed(self, conn: Conn) -> bool`

This method is called to check if a connection is no longer able to be used.
Expand All @@ -111,7 +108,6 @@ exception is suppressed unless it is not a `BaseException`, like
`asyncio.CancelledError`. It is the responsibility of the `ConnectionStrategy`
implementation to avoid leaking a connection in this case.


#### `async def close_connection(self, conn: Conn)`

This method is called to close a connection. This occurs when the pool has
Expand All @@ -122,8 +118,7 @@ If this method raises an exception, the connection is assumed to be closed and
the exception bubbles to the caller of `ConnectionPool.get_connection().__aexit__`
(usually an `async with` block).


## Integrations with 3rd-party libraries
## Integrations with 3rd-party libraries

This package includes support for [`ddtrace`][ddtrace]/[`datadog`][datadog] and
for [`aioredis`][aioredis] (<2.0.0).
Expand All @@ -142,14 +137,12 @@ arguments of the base class, supports:
- Optional `extra_tags` argument: Additional tags to provide to all metrics
(strings in a `"key:value"` format)


### `asyncio_connection_pool.contrib.aioredis.RedisConnectionStrategy`

This class implements the `ConnectionStrategy` abstract methods, using
`aioredis.Redis` objects as connections. The constructor takes arbitrary
arguments and forwards them to `aioredis.create_redis`.


## How is this safe without locks?

I encourage you to read the [source](https://github.com/fellowinsights/asyncio-connection-pool/blob/master/asyncio_connection_pool/__init__.py)
Expand Down
20 changes: 9 additions & 11 deletions asyncio_connection_pool/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from abc import ABC, abstractmethod
import asyncio
import inspect
from abc import ABC, abstractmethod
from contextlib import asynccontextmanager
from typing import AsyncIterator, Awaitable, Generic, Optional, TypeVar

Expand All @@ -10,16 +10,13 @@

class ConnectionStrategy(ABC, Generic[Conn]):
@abstractmethod
async def make_connection(self) -> Conn:
...
async def make_connection(self) -> Conn: ...

@abstractmethod
def connection_is_closed(self, conn: Conn) -> bool:
...
def connection_is_closed(self, conn: Conn) -> bool: ...

@abstractmethod
async def close_connection(self, conn: Conn) -> None:
...
async def close_connection(self, conn: Conn) -> None: ...


async def _close_connection_compat(
Expand Down Expand Up @@ -60,18 +57,19 @@ def __init__(
*,
strategy: ConnectionStrategy[Conn],
max_size: int,
burst_limit: Optional[int] = None
burst_limit: Optional[int] = None,
) -> None:
self._loop = asyncio.get_event_loop()
self.strategy = strategy
self.max_size = max_size
self.burst_limit = burst_limit
if burst_limit is not None and burst_limit < max_size:
raise ValueError("burst_limit must be greater than or equal to max_size")
msg = "burst_limit must be greater than or equal to max_size"
raise ValueError(msg)
self.in_use = 0
self.currently_allocating = 0
self.currently_deallocating = 0
self.available: "asyncio.Queue[Conn]" = asyncio.Queue(maxsize=self.max_size)
self.available: asyncio.Queue[Conn] = asyncio.Queue(maxsize=self.max_size)

@property
def _total(self) -> int:
Expand Down Expand Up @@ -110,7 +108,7 @@ def _get_conn(self) -> Awaitable[Conn]:
# Incidentally, awaiting a done Future doesn't involve yielding to
# the event loop; it's more like getting the next value from a
# generator.
fut: "asyncio.Future[Conn]" = self._loop.create_future()
fut: asyncio.Future[Conn] = self._loop.create_future()
fut.set_result(self.available.get_nowait())
self.in_use += 1
return fut
Expand Down
20 changes: 0 additions & 20 deletions asyncio_connection_pool/contrib/aioredis.py

This file was deleted.

Loading
Loading