Finish documentation (more or less)
This commit is contained in:
parent
fff8e6e4cc
commit
b23bf037e0
17
README.rst
17
README.rst
|
|
@ -2,4 +2,19 @@
|
|||
Website stats collector demo
|
||||
============================
|
||||
|
||||
...
|
||||
CHWEB is a website checking tool.
|
||||
|
||||
ATM in its very early stages meant to demo `aiven <https://aiven.io>`_'s
|
||||
platform, using their `kafka <https://aiven.io/kafka>`_ and `postgresql
|
||||
<https://aiven.io/postgresql>`_ services.
|
||||
|
||||
|
||||
Install and run the application with::
|
||||
|
||||
pip install https://github.com/vladan/aiven-status-checker/releases/...
|
||||
chweb_collect -c config.yaml &
|
||||
chweb_consume -c config.yaml &
|
||||
|
||||
or run in docker with::
|
||||
|
||||
docker-compose up
|
||||
|
|
|
|||
|
|
@ -0,0 +1,65 @@
|
|||
Overall architecture
|
||||
====================
|
||||
|
||||
A simple representation of the architectore would be the producer -> kafka ->
|
||||
consumer -> postgres, e.g.::
|
||||
|
||||
|
||||
+---------------+
|
||||
+--------------+ | aiven |
|
||||
| checker |+--------> kafka |
|
||||
+--------------+ +-------+-------+
|
||||
|
|
||||
|
|
||||
|
|
||||
+-------v-------+
|
||||
| consumer |
|
||||
+-------+-------+
|
||||
|
|
||||
|
|
||||
|
|
||||
+-------v-------+
|
||||
| aiven |
|
||||
| PostgreSQL |
|
||||
+---------------+
|
||||
|
||||
|
||||
|
||||
|
||||
A more detailed graph includes the inner workings of the services. The site
|
||||
checker module creates an asyncio task running periodically for every site in
|
||||
the config, feeding an asyncio [mpsc] Queue from which the consumer is
|
||||
forwarding messages in the kafka topic specified in the configuration::
|
||||
|
||||
|
||||
+ periodical asyncio tasks ----+
|
||||
| |
|
||||
| |
|
||||
| +-------------+ |
|
||||
| | example.com +--------+---------+
|
||||
| +-------------+ | +----v----+ +------------+
|
||||
| +-----------+ | | asyncio | | |
|
||||
| | aiven.com +--------+----> Queue +-------> producer |
|
||||
| +-----------+ | +----^----+ | |
|
||||
| +------------+ | | +-----+------+
|
||||
| | google.com +--------+---------+ |
|
||||
| +------------+ | |
|
||||
| | |
|
||||
| | |
|
||||
+------------------------------+ |
|
||||
|
|
||||
+--------v---------+
|
||||
| |
|
||||
| aiven kafka |
|
||||
| |
|
||||
+--------+---------+
|
||||
|
|
||||
+----------------+ |
|
||||
| | |
|
||||
| aiven | +------v------+
|
||||
| PostgreSQL <-----------+ consumer |
|
||||
| | +-------------+
|
||||
+----------------+
|
||||
|
||||
|
||||
|
||||
|
|
@ -12,6 +12,9 @@
|
|||
#
|
||||
import os
|
||||
import sys
|
||||
import sphinx_typlog_theme
|
||||
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.abspath('.'),
|
||||
"..", "..", "src"))
|
||||
|
||||
|
|
@ -45,7 +48,8 @@ exclude_patterns = []
|
|||
# The theme to use for HTML and HTML Help pages. See the documentation for
|
||||
# a list of builtin themes.
|
||||
#
|
||||
html_theme = 'alabaster'
|
||||
html_theme_path = [sphinx_typlog_theme.get_path()]
|
||||
html_theme = 'sphinx_typlog_theme'
|
||||
|
||||
# Add any paths that contain custom static files (such as style sheets) here,
|
||||
# relative to this directory. They are copied after the builtin static files,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,63 @@
|
|||
Running the services
|
||||
====================
|
||||
|
||||
When you install the package you get two commad-line applications:
|
||||
``chweb_collect`` and ``chweb_consume``.
|
||||
|
||||
Both of them need a configuration file to start, e.g.::
|
||||
|
||||
chweb_consume --help
|
||||
usage: chweb_consume [-h] [-c CONFIG]
|
||||
|
||||
Website availibility checker.
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
-c CONFIG, --config CONFIG
|
||||
The yaml config file. Defaults to /etc/checker.yaml
|
||||
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
Both cli scripts use the same config file to simplify deployment. You only set
|
||||
it up once and supply it to the scripts on their locations.
|
||||
|
||||
The config file is somewhat straightforward, as it can be seen in this example::
|
||||
|
||||
kafka:
|
||||
servers:
|
||||
- "localhost:9992"
|
||||
topic: "sample"
|
||||
postgres:
|
||||
dbhost: "localhost"
|
||||
dbport: 5432
|
||||
dbname: "chweb"
|
||||
dbuser: "vladan"
|
||||
dbpass: ""
|
||||
sites:
|
||||
- url: "https://dsadakjhkjsahkjh.com"
|
||||
regex: "domain"
|
||||
check_interval: 5
|
||||
- url: "https://example.com"
|
||||
regex: "aaaaaaaaaaaaa"
|
||||
check_interval: 8
|
||||
- url: "https://example.com/404"
|
||||
check_interval: 13
|
||||
logging:
|
||||
version: 1
|
||||
formatters:
|
||||
standard:
|
||||
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||
error:
|
||||
format: "%(levelname)s <PID %(process)d:%(processName)s> %(name)s.%(funcName)s(): %(message)s"
|
||||
handlers:
|
||||
console:
|
||||
class: logging.StreamHandler
|
||||
level: DEBUG
|
||||
formatter: standard
|
||||
stream: ext://sys.stdout
|
||||
root:
|
||||
level: DEBUG
|
||||
handlers: [console]
|
||||
propogate: yes
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
Environment config
|
||||
==================
|
||||
|
||||
The kafka and postgres settings can be overriden via the environment.
|
||||
These are the env variables accepted by the applications with example values::
|
||||
|
||||
KAFKA_SERVERS=kafkaserver.one.com:9092,kfk02.server.two.com
|
||||
KAFKA_TOPIC=status_checker
|
||||
|
||||
POSTGRES_DB=chweb
|
||||
POSTGRES_HOST=localhost
|
||||
POSTGRES_PORT=5432
|
||||
POSTGRES_USER=vladan
|
||||
POSTGRES_PASS=secret
|
||||
|
|
@ -3,17 +3,33 @@
|
|||
You can adapt this file completely to your liking, but it should at least
|
||||
contain the root `toctree` directive.
|
||||
|
||||
Welcome to chweb's documentation!
|
||||
=================================
|
||||
chweb - Check websites state
|
||||
============================
|
||||
|
||||
CHWEB is a website checking tool.
|
||||
|
||||
ATM in its very early stages meant to demo
|
||||
`aiven <https://aiven.io>`_'s platform, using their `kafka
|
||||
<https://aiven.io/kafka>`_ and `postgresql <https://aiven.io/postgresql>`_
|
||||
services.
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
* Easy and intuitive YAML configuration.
|
||||
* Command line executables for the checker and the consumer/psql writer:
|
||||
``chweb_collect`` and ``chweb_consume``.
|
||||
* Environment configuration.
|
||||
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:caption: Contents:
|
||||
|
||||
configuration
|
||||
environment
|
||||
architecture
|
||||
apidocs
|
||||
|
||||
|
||||
|
||||
Indices and tables
|
||||
==================
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue