2.4 KiB
spiderss - a plaintext RSS crawler
spiderss is a plaintext RSS crawler, based on feedparser, python-readability and html2text Articles are parsed as Markdown files from the original article web page and stored on the filesystem.
Features
- Categories
- Delete articles after a few days
- Distinguish new from read articles
- Store loved articles forever
- OPML import
Installation
NixOS
Just call nix-shell in the project directory.
Legacy OS
Install the requirements with pip install -r requirements.txt.
Usage
./spiderss.py [-h] [-V] [-v] [-c CONFIG]
optional arguments:
-h, --help show this help message and exit
-V, --version show version and exit
-v, --verbose verbose output
-c CONFIG, --config CONFIG
config file (default: ./config.toml)
Config
The config file is written in TOML and has the following variables:
base_dir: The base directory where your articles are stored.
max_age: The amount of days, your articles are kept on the filesystem. Articles in the loved folder are skipped.
feed: Is a feed element. It has the following attributes:
category: Category of the feed.
name: Name of the feed.
url: URL of the feed.
Why?
Because plaintext is God.
How can I read the articles?
Use your favourite Markdown viewer, or just the console. spiderss integrates nice with the ranger filemanager to browse categories.
How does it work?
Edit the config.toml file to your liking and run the script.
The script creates a folder structure the following way:
base_directory
| - category
| - feedname
| - new
| - read
| - another feedname
| - new
| - read
| - another category
| - a third feedname
| - new
| - read
| - loved
Every feed gets a new and a read subfolder. Article files are stored in the new folder. Move them to the read folder if you're done reading them. You can do this easily e.g. by setting a keybinding in ranger.
A special folder, called loved, is created on startup in the base_directory. It is intended for articles, you want to keep. Articles inside are never deleted, even if they expire the max_age.