Finish documentation (more or less)

README.rst

@@ -2,4 +2,19 @@
 Website stats collector demo
 ============================
 
-...
+CHWEB is a website checking tool.
+
+ATM in its very early stages meant to demo `aiven <https://aiven.io>`_'s
+platform, using their `kafka <https://aiven.io/kafka>`_ and `postgresql
+<https://aiven.io/postgresql>`_ services.
+
+
+Install and run the application with::
+
+    pip install https://github.com/vladan/aiven-status-checker/releases/...
+    chweb_collect -c config.yaml &
+    chweb_consume -c config.yaml &
+
+or run in docker with::
+
+    docker-compose up

docs/source/architecture.rst

@@ -0,0 +1,65 @@
+Overall architecture
+====================
+
+A simple representation of the architectore would be the producer -> kafka ->
+consumer -> postgres, e.g.::
+
+
+                              +---------------+
+     +--------------+         |     aiven     |
+     |   checker    |+-------->     kafka     |
+     +--------------+         +-------+-------+
+                                      |
+                                      |
+                                      |
+                              +-------v-------+
+                              |   consumer    |
+                              +-------+-------+
+                                      |
+                                      |
+                                      |
+                              +-------v-------+
+                              |    aiven      |
+                              |  PostgreSQL   |
+                              +---------------+
+
+
+
+
+A more detailed graph includes the inner workings of the services. The site
+checker module creates an asyncio task running periodically for every site in
+the config, feeding an asyncio [mpsc] Queue from which the consumer is
+forwarding messages in the kafka topic specified in the configuration::
+
+
+     + periodical asyncio tasks ----+
+     |                              |
+     |                              |
+     |       +-------------+        |
+     |       | example.com +--------+---------+
+     |       +-------------+        |    +----v----+       +------------+
+     |         +-----------+        |    | asyncio |       |            |
+     |         | aiven.com +--------+---->  Queue  +------->  producer  |
+     |         +-----------+        |    +----^----+       |            |
+     |        +------------+        |         |            +-----+------+
+     |        | google.com +--------+---------+                  |
+     |        +------------+        |                            |
+     |                              |                            |
+     |                              |                            |
+     +------------------------------+                            |
+                                                                 |
+                                                        +--------v---------+
+                                                        |                  |
+                                                        |   aiven kafka    |
+                                                        |                  |
+                                                        +--------+---------+
+                                                                 |
+                             +----------------+                  |
+                             |                |                  |
+                             |     aiven      |           +------v------+
+                             |   PostgreSQL   <-----------+  consumer   |
+                             |                |           +-------------+
+                             +----------------+
+
+
+

docs/source/conf.py

@@ -12,6 +12,9 @@
 #
 import os
 import sys
+import sphinx_typlog_theme
+
+
 sys.path.insert(0, os.path.join(os.path.abspath('.'),
                                 "..", "..", "src"))
 
@@ -45,7 +48,8 @@ exclude_patterns = []
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'alabaster'
+html_theme_path = [sphinx_typlog_theme.get_path()]
+html_theme = 'sphinx_typlog_theme'
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,

docs/source/configuration.rst

@@ -0,0 +1,63 @@
+Running the services
+====================
+
+When you install the package you get two commad-line applications:
+``chweb_collect`` and ``chweb_consume``.
+
+Both of them need a configuration file to start, e.g.::
+
+    chweb_consume --help
+    usage: chweb_consume [-h] [-c CONFIG]
+
+    Website availibility checker.
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      -c CONFIG, --config CONFIG
+                            The yaml config file. Defaults to /etc/checker.yaml
+
+
+Configuration
+-------------
+
+Both cli scripts use the same config file to simplify deployment. You only set
+it up once and supply it to the scripts on their locations.
+
+The config file is somewhat straightforward, as it can be seen in this example::
+
+    kafka:
+      servers:
+      - "localhost:9992"
+      topic: "sample"
+    postgres:
+      dbhost: "localhost"
+      dbport: 5432
+      dbname: "chweb"
+      dbuser: "vladan"
+      dbpass: ""
+    sites:
+    - url: "https://dsadakjhkjsahkjh.com"
+      regex: "domain"
+      check_interval: 5
+    - url: "https://example.com"
+      regex: "aaaaaaaaaaaaa"
+      check_interval: 8
+    - url: "https://example.com/404"
+      check_interval: 13
+    logging:
+      version: 1
+      formatters:
+        standard:
+          format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+        error:
+          format: "%(levelname)s <PID %(process)d:%(processName)s> %(name)s.%(funcName)s(): %(message)s"
+      handlers:
+        console:
+          class: logging.StreamHandler
+          level: DEBUG
+          formatter: standard
+          stream: ext://sys.stdout
+      root:
+        level: DEBUG
+        handlers: [console]
+        propogate: yes

docs/source/environment.rst

@@ -0,0 +1,14 @@
+Environment config
+==================
+
+The kafka and postgres settings can be overriden via the environment.
+These are the env variables accepted by the applications with example values::
+
+    KAFKA_SERVERS=kafkaserver.one.com:9092,kfk02.server.two.com
+    KAFKA_TOPIC=status_checker
+
+    POSTGRES_DB=chweb
+    POSTGRES_HOST=localhost
+    POSTGRES_PORT=5432
+    POSTGRES_USER=vladan
+    POSTGRES_PASS=secret

docs/source/index.rst

@@ -3,17 +3,33 @@
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
-Welcome to chweb's documentation!
+chweb - Check websites state
-=================================
+============================
+
+CHWEB is a website checking tool.
+
+ATM in its very early stages meant to demo
+`aiven <https://aiven.io>`_'s platform, using their `kafka
+<https://aiven.io/kafka>`_ and `postgresql <https://aiven.io/postgresql>`_
+services.
+
+Features
+--------
+
+* Easy and intuitive YAML configuration.
+* Command line executables for the checker and the consumer/psql writer:
+  ``chweb_collect`` and ``chweb_consume``.
+* Environment configuration.
+
 
 .. toctree::
    :maxdepth: 1
-   :caption: Contents:
 
+   configuration
+   environment
+   architecture
    apidocs
 
-
-
 Indices and tables
 ==================
 

tox.ini

@@ -26,7 +26,9 @@ commands =
 
 [testenv:docs]
 changedir = docs
-deps = sphinx
+deps =
+    sphinx
+    sphinx-typlog-theme
 commands =
     sphinx-build -W -b html -E ./source/ ./build/