A Python script to request data in a specific interval.

You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Go to file

trivernis 8269c78f9a Add initial working program - Add miner for main execution - Add Readme with install, usage etc. - Add LICENSE as GPL-3.0		4 years ago
lib	Add initial working program	4 years ago
.gitignore	Add initial working program	4 years ago
LICENSE	Add initial working program	4 years ago
Pipfile	Add initial working program	4 years ago
Pipfile.lock	Add initial working program	4 years ago
README.md	Add initial working program	4 years ago
miner.py	Add initial working program	4 years ago

README.md

data-miner

A tool to periodically collect data from api endpoints and store them inside a folder. Please ensure that you are allowed to use the api this way before using this tool!

Prerequisites

You need to have python 3.8 or higher and pipenv installed.

Install

pipenv install

Usage

usage: miner.py [-h] [-t] -o OUTPUT_DIR [-i INTERVAL] [-m METHOD] [-p PAYLOAD_FILE] url

Periodically mine data

positional arguments:
  url                   the data endpoint url

optional arguments:
  -h, --help            show this help message and exit
  -t, --tor             If tor should be used for requests
  -o OUTPUT_DIR, --output-dir OUTPUT_DIR
                        The output directory for the data
  -i INTERVAL, --interval INTERVAL
                        The interval in which the data is requested
  -m METHOD, --method METHOD
                        The HTTP method that is used
  -p PAYLOAD_FILE, --payload-file PAYLOAD_FILE
                        The file containing the requests payload.

License

GPL-3.0