pywtk

Easy to use REPL wordlists manager with "lazy" merger for low-memory machines written in Python.

Main feature: ability to merge large files fast and without a RAM overflow

Packages

• dask
• riposte
• xxhash
• numpy
• pandas

Installing

github clone https://github.com/magicnum/pywtk
cd pywtk
pip install -r requirements.txt
python3 pywtk.py

Commands

Basic REPL commands:

• load [paths]
• merge [item, outfilename]
• makedb [db_path]
• show [item]
• chdir [path]
• exit

---

load: load files paths into a wordlists list or files content into db dictionary. Choose of item based on file extension (json for db, any other for wordlists).

pywtk:~$ load abc.txt

load supports wildcard symbol, so load hashes0.txt hashes1.txt is equal to load hashes*.txt.

---

merge: merges files content and delete duplicates using dask parallel computation.

pywtk:~$ merge wordlists merged.dic

---

makedb: make json structure with hashes and full paths of wordlists.

pywtk:~$ load old_200*.txt
...
pywtk:~$ makedb old.json

---

show: output current state of loaded item.

pywtk:~$ show all

---

chdir: setup current working folder.

pywtk:~$ chdir /home/user/wordlists/

Current items are:

• wordlists
• dbs
• all

CLI

Thanks to riposte you can use it as CLI too:

python pywtk.py -c "load passwords201*.dic; merge wordlists merged.dic"

TODO

• regex, minimal and maximum length of string filters