Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
6c05f89
Merge pull request #34 from thewebscraping/develop
thewebscraping May 8, 2025
e29c78d
v1.1.3
markdownpipe May 8, 2025
a00bcb6
[Bug]: Cannot unpickle CookieJar #35
markdownpipe May 18, 2025
6375fbf
Merge pull request #36 from thewebscraping/bugs/cannot-unpickle-cooki…
thewebscraping May 18, 2025
952808a
Add response and request attributes to HTTPError
ytinyui May 25, 2025
7fb2f1d
Add super().__init__() to HTTPError
ytinyui May 27, 2025
c126ba1
Merge pull request #37 from ytinyui/main
thewebscraping May 27, 2025
7964828
v1.1.5
thewebscraping May 27, 2025
4da8f3f
feat: rotator supported
markdownpipe Oct 10, 2025
e1f1dac
chore: pytest
markdownpipe Oct 10, 2025
aae2b74
chore: new tls-client version
markdownpipe Oct 10, 2025
89fe99e
chore: remove dependencies
markdownpipe Oct 10, 2025
7857e57
v1.1.6
markdownpipe Oct 14, 2025
d8d98c5
Merge pull request #41 from thewebscraping/feat/rotator
thewebscraping Oct 14, 2025
0f95a19
fix: optimize logging and fix cookie domain
markdownpipe Nov 23, 2025
e262d0d
fix: improve logging and cookie handling
markdownpipe Nov 23, 2025
b9e1b3f
chore: update changelog for 1.1.7 and bump version
markdownpipe Nov 23, 2025
b7d006f
Merge pull request #48 from thewebscraping/feat/optimize-logging-cookies
thewebscraping Nov 23, 2025
4ab6b7e
chore: bump version to 1.1.8
markdownpipe Nov 23, 2025
0a4f836
Merge pull request #49 from thewebscraping/feat/optimize-logging-cookies
thewebscraping Nov 23, 2025
3138407
chore: bump version to 1.1.8
markdownpipe Nov 23, 2025
24a99ad
fix(response): normalize charset=utf-8,gbk to utf-8
markdownpipe Dec 27, 2025
ee15974
Merge pull request #50 from thewebscraping/bugs/charset-utf-8-fallback
thewebscraping Dec 27, 2025
ead9aa6
Bump version to 1.1.9
markdownpipe Dec 27, 2025
8cf1831
feat: improve IPv6 support and add comprehensive tests for URL/Proxy
markdownpipe Jan 20, 2026
2bd2988
Merge pull request #52 from thewebscraping/bugs/ipv6-netloc
thewebscraping Jan 20, 2026
6b68027
Bump version to 1.2.0
markdownpipe Jan 20, 2026
82f54c3
chore: upgrade tls-client to v1.13.1 and refactor library management
markdownpipe Jan 21, 2026
f026321
Merge pull request #54 from thewebscraping/feat/upgrade-tls-client-v1…
thewebscraping Jan 21, 2026
29153f2
chore: bump version to 1.2.1
markdownpipe Jan 21, 2026
4b2280e
refactor: type hardening and python 3.9 compatibility
markdownpipe Jan 21, 2026
c33d566
refactor: migrate to src layout, hatch build system and wrap-tls-requ…
markdownpipe Jan 21, 2026
82d457d
refactor: rename tls_identifier to client_identifier and tls_debug to…
markdownpipe Jan 21, 2026
605efc3
docs: finalize changelog and ci modernization for v1.2.2
markdownpipe Jan 21, 2026
3157c19
fix: add missing dev dependencies for testing
markdownpipe Jan 21, 2026
37c8e62
fix: make TLS library download more robust against 404s and naming va…
markdownpipe Jan 21, 2026
7762e46
Merge pull request #55 from thewebscraping/feat/mypy-uv-support
thewebscraping Jan 21, 2026
cff7900
fix: normalize machine architecture casing and improve Windows fallba…
markdownpipe Jan 21, 2026
4dd5bc1
fix: add mkautodoc to dev dependencies for documentation build
markdownpipe Jan 21, 2026
104fbc5
fix: update documentation trigger and permissions
markdownpipe Jan 21, 2026
e80972f
docs: convert pycon to python blocks
markdownpipe Jan 21, 2026
851066f
docs/tests: increase coverage to 93%, hit 100% for client.py/tls.py
markdownpipe Jan 22, 2026
cfd7854
fix: lazy asyncio lock for python 3.9 and headers KeyError
markdownpipe Jan 24, 2026
b4b34db
fix: add IPv6 auto-repair and improve Python 3.9 compatibility
markdownpipe Jan 24, 2026
08463e2
fix: python 3.9 compatibility and robust IPv6 URL handling
markdownpipe Jan 24, 2026
52d4b4e
Merge pull request #56 from thewebscraping/bugs/shared-extra-configs
thewebscraping Jan 24, 2026
c987fe1
chore: bump version to 1.2.3
markdownpipe Jan 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
github: [thewebscraping]
94 changes: 36 additions & 58 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,69 +3,47 @@ name: CI
on:
push:
branches: ['master', 'main']

pull_request:
branches: ['master', 'main', 'dev', 'develop']
branches: ['master', 'main', 'dev', 'develop', 'feat/*']

jobs:
build-on-ubuntu:
runs-on: ubuntu-latest
tests:
name: Tests on ${{ matrix.os }} / ${{ matrix.python-version }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
max-parallel: 3
matrix:
python-version: ['3.9', '3.10', '3.11']
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
exclude:
- os: windows-latest
python-version: '3.10'
- os: windows-latest
python-version: '3.11'
- os: windows-latest
python-version: '3.12'
- os: macos-latest
python-version: '3.10'
- os: macos-latest
python-version: '3.11'
- os: macos-latest
python-version: '3.12'

- name: Install Dependencies, Lint and Tests
run: |
make init-actions

build-on-mac:
runs-on: macos-latest
strategy:
fail-fast: false
max-parallel: 1
matrix:
python-version: ['3.9']
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: Install Dependencies, Lint and Tests
run: |
make init-actions

- name: Pytest
run: |
make pytest

build-on-windows:
runs-on: windows-latest
strategy:
fail-fast: false
max-parallel: 1
matrix:
python-version: ['3.9']
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: Install Dependencies, Lint
run: |
make init-actions

- name: Pytest
run: |
make pytest
- uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
python-version: ${{ matrix.python-version }}

- name: Run linting
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.9'
run: |
uv run ruff check .
uv run mypy src/tls_requests

- name: Run tests
run: |
uv run pytest
53 changes: 24 additions & 29 deletions .github/workflows/documentation.yml
Original file line number Diff line number Diff line change
@@ -1,35 +1,30 @@
name: Build Documentation
on:
release:
types: [created]
push:
branches:
- main
tags:
- 'v*'
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
strategy:
max-parallel: 1
matrix:
python-version: ['3.9']
permissions:
contents: write
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install Dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
- name: Configure Git Credentials
run: |
git config user.name github-actions[bot]
git config user.email 41898282+github-actions[bot]@users.noreply.github.com
- uses: actions/cache@v4
with:
key: mkdocs-material-${{ env.cache_id }}
path: .cache
restore-keys: |
mkdocs-material-
- name: Publish Documentation
run: |
echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
mkdocs gh-deploy --force
- uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
python-version: '3.11'

- name: Configure Git Credentials
run: |
git config user.name github-actions[bot]
git config user.email 41898282+github-actions[bot]@users.noreply.github.com

- name: Publish Documentation
run: |
uv run mkdocs gh-deploy --force
41 changes: 23 additions & 18 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -1,26 +1,31 @@
name: Upload Python Package
name: Publish to PyPI

on:
release:
types: [created]
push:
tags:
- 'v*'

jobs:
deploy:
name: Publish PyPI
build-n-publish:
name: Build and publish to PyPI
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/p/wrapper-tls-requests
permissions:
id-token: write # Required for Trusted Publishing
contents: read

steps:
- uses: actions/checkout@main
- name: Set up Python 3.9
uses: actions/setup-python@v4
- uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
python-version: '3.9'
enable-cache: true

- name: Build package
run: |
python -m pip install -U pip build setuptools twine
- name: Publish
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: |
python setup.py sdist bdist_wheel
twine upload --skip-existing dist/*
run: uv build

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -160,4 +160,8 @@ cython_debug/
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
.idea/
tls_requests/bin/*xgo*
src/tls_requests/bin/*.dll
src/tls_requests/bin/*.so
src/tls_requests/bin/*.dylib
src/tls_requests/bin/*.json
!src/tls_requests/bin/__init__.py
69 changes: 49 additions & 20 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,33 +1,62 @@
exclude: '^docs.sh/|scripts/'
default_stages: [pre-commit]

default_language_version:
python: python3.10
# Pre-commit hooks configuration
# See https://pre-commit.com for more information

repos:
# General file checks
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
rev: v6.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
args: [--maxkb=1500]
- id: check-json
- id: check-toml
- id: check-xml
- id: check-yaml
- id: debug-statements
- id: check-builtin-literals
- id: check-merge-conflict
- id: check-case-conflict
- id: check-docstring-first
- id: detect-private-key
- id: mixed-line-ending

# run the isort.
- repo: https://github.com/PyCQA/isort
rev: 5.13.2
# Ruff - Fast Python linter and formatter
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.14.13
hooks:
- id: isort
# Run the linter
- id: ruff
args: [--extend-select, I, --fix, --exit-non-zero-on-fix]
# Run the formatter
- id: ruff-format
args: [--exit-non-zero-on-fix]

# run the flake8.
- repo: https://github.com/PyCQA/flake8
rev: 7.0.0
# Type checking with mypy
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.15.0
hooks:
- id: flake8
- id: mypy
additional_dependencies:
- types-setuptools
- idna
- orjson
- charset-normalizer
- chardet
args: [--ignore-missing-imports, --config-file=pyproject.toml]

# Markdown linting (disabled due to duplicate heading issues)
# - repo: https://github.com/igorshubovych/markdownlint-cli
# rev: v0.46.0
# hooks:
# - id: markdownlint
# args: [--fix]

# CI configuration
ci:
autofix_commit_msg: |
[pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci
autofix_prs: true
autoupdate_branch: ''
autoupdate_commit_msg: '[pre-commit.ci] pre-commit autoupdate'
autoupdate_schedule: weekly
skip: []
submodules: false
52 changes: 51 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,55 @@
Release History
===============
================

1.2.3 (2026-01-24)
------------------
**Improvements:**

- **Enhanced IPv6 Support**: Implemented automatic repair for naked IPv6 addresses (e.g., `2001:db8::1` -> `[2001:db8::1]`) and strict bracket validation.
- **Robust URL Handling**: Added default `http://` scheme for schemeless URLs and improved component-based building logic to prevent crashes.
- **Type Safety**: Resolved Mypy strict type inconsistencies and added comprehensive test coverage for IPv6 edge cases.

**Bugfixes:**

- **Python 3.9 Compatibility**: Fixed `RuntimeError: There is no current event loop` during rotator initialization by implementing lazy asyncio lock loading.
- **Redirect Logic**: Fixed `Headers.__getitem__` to correctly raise `KeyError` for missing keys, ensuring compatibility with standard redirect handling.

1.2.2 (2026-01-21)
------------------
**Improvements:**

- **Standardized API Parameters**: Renamed `tls_identifier` to `client_identifier` and `tls_debug` to `debug` for better consistency and professional API surface.
- **Src Layout Migration**: Refactored the project structure to use a modern `src/` layout, improving package isolation and build reliability.
- **Type Safety & Testing**: Integrated `mypy` for strict type checking and migrated to `uv run pytest` for comprehensive test verification (80+ tests passing).
- **Backward Compatibility**: Implemented a centralized fallback and logging warning system for legacy parameters (planned removal in v1.3.0).
- **Modernized CI/CD**: Fully migrated from `Makefile` to `uv`. Optimized GitHub Actions matrix to support Python 3.9 through 3.13 across Linux, macOS, and Windows.
- **Enhanced Platform Support**: Improved detection and loading logic for ARM64/aarch64 architectures (Apple Silicon, AWS Graviton).
- **Project Sustainability**: Added GitHub Sponsors support to the repository metadata and documentation.

1.1.7 (2025-11-23)
------------------
**Improvements:**

- Optimized logging. ([#46](https://github.com/thewebscraping/tls-requests/issues/46))
- Fixed cookie response handling. ([#47](https://github.com/thewebscraping/tls-requests/issues/47))

1.1.6 (2025-10-14)
------------------
**Enhancements:**
This pull request introduces two major enhancements that significantly improve the library's anti‑detection capabilities and overall robustness:

**A Smart Rotator System**
- Automatically rotates proxies, headers, and TLS identifiers to mimic authentic traffic.
- Introduced three new rotator classes: `ProxyRotator`, `HeaderRotator`, and `TLSIdentifierRotator`.
- Client and AsyncClient now enable header and TLS identifier rotation by default, using built‑in realistic templates.
- Unified parameters accept a single value, a list, or a pre‑configured Rotator instance.
- Proxy feedback loop (`mark_result`/`amark_result`) optimizes weighted rotation strategy.

**Robust Library Management**
- Dependency‑free, self‑managing mechanism for the core `tls-client` C library lifecycle.
- Removed `requests` and `tqdm`; now uses built‑in `urllib` and [json](cci:1://file:///Users/twofarm/Desktop/works/tls_requests/tls_requests/models/response.py:204:4-205:43).
- TLSLibrary is version‑aware, automatically downloading the correct version from GitHub when needed.
- Automatic cleanup of old library files after successful updates.

1.0.7 (2024-12-14)
-------------------
Expand Down
4 changes: 0 additions & 4 deletions MANIFEST.in

This file was deleted.

Loading
Loading