Robofinder is a powerful Python script designed to search for and retrieve historical robots.txt files from Archive.org for any given website
Installation
Usage
Installation
Code:
cd robofinder
pip install -r requirements.txt
Usage
Code:
python3 robofinder.py -u https://example.com