-brown?logo=linux&style=for-the-badge){kind=icon}
-blue?logo=apple&style=for-the-badge){kind=icon}
- Installation
- Running from source
- General Usage
- Supported sources
- Rejected sources
- Supported output formats
This application uses Calibre to convert ebooks.
Install it from https://calibre-ebook.com/download
Without it, you will only get output in epub, text, and web formats.
For macOS, you need to manually add the path to Calibre.
Before starting lncrawl, use the command:
$ export PATH="$PATH:/Applications/calibre.app/Contents/MacOS"If you used a folder other than Applications during installation, replace /Applications/ with your path to Calibre.
⏬ Windows: lncrawl.exe
In Windows 8, 10 or later versions, it might say that
lncrawl.exeis not safe to download or execute. You should bypass/ignore this security check to execute this program.
⏬ Linux: lncrawl
It is recommended to install via pip if you are on Linux
⏬ MacOS: lncrawl
It is recommended to install via pip if you are on Mac
⏬ To get older versions visit the Releases page
📦 A python package named lightnovel-crawler is available at pypi.
Make sure you have installed Python v3.8 or higher and have pip enabled. Visit these links to install python with pip in Windows, Linux, and MacOS. Feel free to ask on the discussions page if you are stuck.
To install this app via pip:
$ pip install -U lightnovel-crawlerIn some cases you have to use python3 -m pip or pip3 or python -m pip.
Next, open your terminal and run:
$ lncrawl -h
If that does not work, you have not configured PATH variables properly. You can access it anyway using:
$ python3 -m lncrawl -h
The master branch contains the latest stable code. If you can not wait for it to be released in the PyPi, you can get it like this:
$ pip install -U git+https://github.com/lncrawl/lightnovel-crawler.git#egg=lightnovel-crawler
The dev branch contains cutting-edge, sometimes unstable changes. To install it:
$ pip install -U https://github.com/lncrawl/lightnovel-crawler/tarball/refs/heads/dev#egg=lightnovel-crawler
Docker is a convenient way to run it anywhere.
- Pull docker image:
$ docker pull ghcr.io/lncrawl/lightnovel-crawler
$ docker tag ghcr.io/lncrawl/lightnovel-crawler lncrawl
- Run using docker:
$ mkdir -p ~/.lncrawl
$ docker run -v ~/.lncrawl:/data --rm -it lncrawl
You can setup alias to the above command in your terminal's profile. e.g.:
alias docker-lncrawl=docker run -v ~/.lncrawl:/data --rm -it -p 8080:8080 lncrawl
- Run server with docker:
$ docker-lncrawl server
- First clone the repository:
$ git clone https://github.com/lncrawl/lightnovel-crawler- Install uv then inside the project folder run:
$ uv sync --extra dev- Run the program (use python v3.8 or higher):
$ uv run python -m lncrawlOr use the Makefile: make install then make start (or make watch for auto-reload).
- Build Docker from source (builds base image then app image):
$ make docker-build
# or: docker build -t lncrawl . # requires existing base image- Run server with Docker Compose:
$ make docker-up
# or: docker compose -f scripts/local-compose.yml up -d$ lncrawl -h
Usage: lncrawl [OPTIONS] COMMAND [ARGS]...
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --verbose -l Log levels: -l = warn, -ll = info, -lll │
│ = debug │
│ --config -c PATH Config file │
│ --install-completion Install completion for the current │
│ shell. │
│ --show-completion Show completion for the current shell, │
│ to copy it or customize the │
│ installation. │
│ --help -h Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ version Show current version. │
│ config View and modify configuration settings. │
│ sources Manage sources. │
│ crawl Crawl from novel page URL. │
│ search Search for novels by query string. │
│ server Run web server. │
╰──────────────────────────────────────────────────────────────────────────────╯
Run lncrawl server to start the server and visit http://localhost:8080.
$ lncrawl -ll server -p 8080
The default login for the web is:
username: admin
password: admin
For terminal usage, you can check help on each subcommands by -h:
$ lncrawl crawl -h
Usage: lncrawl crawl [OPTIONS] [URL]
Crawl from novel page URL.
╭─ Arguments ───────────────────────────────────────────────────────────────────────────────────────╮
│ url [URL] Novel details page URL. │
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────╮
│ --noin Disable interactive mode │
│ --all Download all chapters │
│ --first N [x>=1] Download first few chapters │
│ --last N [x>=1] Download latest few chapters │
│ --format -f [json|epub|txt|pdf|mobi|docx|rtf|fb2|a Output formats │
│ zw3|lit|lrf|pdb|rb|tcr] │
│ --help -h Show this message and exit. │
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
Visit the discussions page for more information. You can also post your query there too.
Interested in contributing? Reference the
- DeepWiki - AI generated project documentation and resources
- CONTRIBUTING.md - Development setup and contribution guidelines
- .github/docs/ARCHITECTURE.md - Setting up CI on forks
- .github/docs/CREATING_CRAWLERS.md - How to add new source crawlers
- .github/docs/DOCKER.md - Docker usage and deployment
- .github/FORKING.md - Setting up CI on forks
git clone https://github.com/lncrawl/lightnovel-crawler.git
cd lightnovel-crawler
make setup && make install
make startLightnovel Crawler Build System
Setup & Install:
make setup Sync submodules, install uv if needed
make install Install Python dependencies (uv sync)
Development:
make start Start backend server
make watch Start backend server with auto-reload
make lint Run Python linter (flake8)
Build:
make build Full build (version, install, wheel, exe)
make build-wheel Build Python wheel only
make build-exe Build PyInstaller executable
Dependencies (uv):
make add-dep PKG Add runtime dependency
make add-dev PKG Add dev dependency
make rm-dep PKG Remove runtime dependency
make rm-dev PKG Remove dev dependency
Docker:
make docker-base Build base image (Calibre + deps)
make docker-build Build base then application image
make docker-up Start stack (compose)
make docker-down Stop stack
make docker-logs Stream container logs
Other:
make version Show current version
make clean Remove .venv, build, dist
make pull Git pull + submodule update
Recommended: Copy sources/_examples/_01_general_soup.py into the right sources/{lang}/ folder, then implement the four required methods (parse_title, parse_cover, parse_chapter_list, select_chapter_body). See .github/docs/CREATING_CRAWLERS.md for the full guide.
You can also use the CLI to generate a crawler with ChatGPT:
$ lncrawl sources createRequest new one by creating a new issue.
We are supporting 460 sources and 390 crawlers.
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍 | https://arnovel.me/ | 23 |   |
| 🔍 | https://kolnovel.com/ | 1 | |
| https://rewayat.club/ | 10 | |
| Source URL | Version | Contributors | |
|---|---|---|---|
| https://cclawtranslations.home.blog/ | 71 |  |
|
| https://domentranslations.wordpress.com/ | 71 | |
|
| 🔍🔑 | https://novelasligeras.net/ | 3 |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍🖼️ | https://anime-sama.fr/ | 4 |  |
| 🔍 | https://chireads.com/ | 10 | |
| 🔍 | https://lightnovelfr.com/ | 9 |  |
| 🤖🔍 | https://lnmtlfr.com/ | 9 | |
| 🤖🔍 | https://massnovel.fr/ | 1 | |
| https://noveldeglace.com/ | 3 | |
|
| https://xiaowaz.fr/ | 4 | |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🤖🔍 | https://ncode.syosetu.com/ | 19 |  |
| 🤖🔍 | https://syosetu.org/ | 2 |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍 | https://blnovels.net/ | 6 | |
| 🔍 | https://centralnovel.com/ | 13 | |
| 🔍 | https://illusia.com.br/ | 1 | |
| 🔍 | https://phoenixnovels.com.br/ | 1 |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍 | https://fenrirscans.com/ | 1 |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🤖🔍 | https://docln.net/ | 11 | |
| 🤖🔍 | https://ln.hako.vn/ | 11 | |
| 🤖🔍 | https://truyenfull.vn/ | 5 | |
| 🤖🔍 | https://truyentr.info/ | 5 | |
- JSON
- EPUB
- TEXT
- WEB
- DOCX
- MOBI
- RTF
- TXT
- AZW3
- FB2
- LIT
- LRF
- OEB
- PDB
- RB
- SNB
- TCR