spiderss - a plaintext RSS crawler

spiderss is a plaintext RSS crawler, based on feedparser, python-readability and html2text. Actually, it's just a python script.

Read the news you want, the way you want it. Without advertisements, clickbait and trackers. Drop unresponsive web interfaces and stop accepting cookies, because plaintext is God.

Articles are parsed as Markdown files from the original article web page and stored on the filesystem.

Note: This script is under development and far from being complete. Until now it works for the most feeds I read. Use at your own risk!

Features

• Store articles in categories
• Delete articles after a few days
• Distinguish new from read articles
• Store loved articles forever
• OPML import

Installation

Until now there is no install method, just ways to call the script.

NixOS

Call nix-shell in the project directory. This will drop you into a python environment with all necessary requirements.

Legacy OS

Install the requirements with pip install -r requirements.txt.

Android

Use Nix-on-Droid and call nix-shell.

Usage

./spiderss.py [-h] [-V] [-v] [-c CONFIG]

optional arguments:
  -h, --help            show this help message and exit
  -V, --version         show version and exit
  -v, --verbose         verbose output
  -c CONFIG, --config CONFIG
                        config file (default: ./config.toml)

Config

The config file is written in TOML. Edit it to your liking before calling the script.

# This defines the base directory for the feeds. Please use an absolute path.
base_directory = '/home/<user>/rss'

# Articles older than max_age will be deleted and not be added
max_age = 30

# Feeds
# The category can be empty (''). The feed fill then be stored in the base_directory.
# The category can also be a path, which will result in subdirectories (e.g. 'technology/hardware').
# The name can also be empty (''). feeds with the same category will then be stored in the same directory.

[[feed]]
category = 'News'
name = 'Newssite'
url = 'https://example.org/feed'

[[feed]]
category = 'News'
name = 'Newssite 2'
url = 'https://example.org/feed'

OPML import

Use the opml2spiderss.py script in the script/ folder. It prints the TOML format of the feeds to stdout. You can append the feeds to your config e.g. the following way:

./opml2spiderss.py <your_feeds>.opml >> <your_config>.toml

Keep articles up to date

Just create a cron job or a systemd.service, which calls the script every now and then.

How to read articles

Use your favourite Markdown viewer, or just the console. spiderss integrates nice with the ranger filemanager which eases navigating through complex folder structures.

The folder structure

The script creates a folder structure the following way:

base_directory
| - category
    | - feedname
        | - new
        | - read
    | - another feedname
        | - new
        | - read
| - another category
    | - a third feedname
        | - new
        | - read
| - loved

Every feed gets a new and a read subfolder. Article files are stored in the new folder. Move them to the read folder if you're done reading them. You can do this easily e.g. by setting a keybinding in ranger.

A special folder, called loved, is created on startup in the base_directory. It is intended for articles, you want to keep. Articles inside are never deleted, even if they expire the max_age.

Articles are stored in files with the name YYYYMMDDHHmm_article_name.md. Thus articles are sorted by publishing date automatically.

Read articles on multiple devices

Just synchronize the base_dir with Syncthing or put it in your Nextcloud