diff --git a/README.rst b/README.rst index 6c112f5..6aebc36 100644 --- a/README.rst +++ b/README.rst @@ -2,4 +2,19 @@ Website stats collector demo ============================ -... +CHWEB is a website checking tool. + +ATM in its very early stages meant to demo `aiven `_'s +platform, using their `kafka `_ and `postgresql +`_ services. + + +Install and run the application with:: + + pip install https://github.com/vladan/aiven-status-checker/releases/... + chweb_collect -c config.yaml & + chweb_consume -c config.yaml & + +or run in docker with:: + + docker-compose up diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst new file mode 100644 index 0000000..53aa268 --- /dev/null +++ b/docs/source/architecture.rst @@ -0,0 +1,65 @@ +Overall architecture +==================== + +A simple representation of the architectore would be the producer -> kafka -> +consumer -> postgres, e.g.:: + + + +---------------+ + +--------------+ | aiven | + | checker |+--------> kafka | + +--------------+ +-------+-------+ + | + | + | + +-------v-------+ + | consumer | + +-------+-------+ + | + | + | + +-------v-------+ + | aiven | + | PostgreSQL | + +---------------+ + + + + +A more detailed graph includes the inner workings of the services. The site +checker module creates an asyncio task running periodically for every site in +the config, feeding an asyncio [mpsc] Queue from which the consumer is +forwarding messages in the kafka topic specified in the configuration:: + + + + periodical asyncio tasks ----+ + | | + | | + | +-------------+ | + | | example.com +--------+---------+ + | +-------------+ | +----v----+ +------------+ + | +-----------+ | | asyncio | | | + | | aiven.com +--------+----> Queue +-------> producer | + | +-----------+ | +----^----+ | | + | +------------+ | | +-----+------+ + | | google.com +--------+---------+ | + | +------------+ | | + | | | + | | | + +------------------------------+ | + | + +--------v---------+ + | | + | aiven kafka | + | | + +--------+---------+ + | + +----------------+ | + | | | + | aiven | +------v------+ + | PostgreSQL <-----------+ consumer | + | | +-------------+ + +----------------+ + + + diff --git a/docs/source/conf.py b/docs/source/conf.py index 247c95b..4166f67 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -12,6 +12,9 @@ # import os import sys +import sphinx_typlog_theme + + sys.path.insert(0, os.path.join(os.path.abspath('.'), "..", "..", "src")) @@ -45,7 +48,8 @@ exclude_patterns = [] # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = 'alabaster' +html_theme_path = [sphinx_typlog_theme.get_path()] +html_theme = 'sphinx_typlog_theme' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst new file mode 100644 index 0000000..e7a6db1 --- /dev/null +++ b/docs/source/configuration.rst @@ -0,0 +1,63 @@ +Running the services +==================== + +When you install the package you get two commad-line applications: +``chweb_collect`` and ``chweb_consume``. + +Both of them need a configuration file to start, e.g.:: + + chweb_consume --help + usage: chweb_consume [-h] [-c CONFIG] + + Website availibility checker. + + optional arguments: + -h, --help show this help message and exit + -c CONFIG, --config CONFIG + The yaml config file. Defaults to /etc/checker.yaml + + +Configuration +------------- + +Both cli scripts use the same config file to simplify deployment. You only set +it up once and supply it to the scripts on their locations. + +The config file is somewhat straightforward, as it can be seen in this example:: + + kafka: + servers: + - "localhost:9992" + topic: "sample" + postgres: + dbhost: "localhost" + dbport: 5432 + dbname: "chweb" + dbuser: "vladan" + dbpass: "" + sites: + - url: "https://dsadakjhkjsahkjh.com" + regex: "domain" + check_interval: 5 + - url: "https://example.com" + regex: "aaaaaaaaaaaaa" + check_interval: 8 + - url: "https://example.com/404" + check_interval: 13 + logging: + version: 1 + formatters: + standard: + format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" + error: + format: "%(levelname)s %(name)s.%(funcName)s(): %(message)s" + handlers: + console: + class: logging.StreamHandler + level: DEBUG + formatter: standard + stream: ext://sys.stdout + root: + level: DEBUG + handlers: [console] + propogate: yes diff --git a/docs/source/environment.rst b/docs/source/environment.rst new file mode 100644 index 0000000..e8a5df2 --- /dev/null +++ b/docs/source/environment.rst @@ -0,0 +1,14 @@ +Environment config +================== + +The kafka and postgres settings can be overriden via the environment. +These are the env variables accepted by the applications with example values:: + + KAFKA_SERVERS=kafkaserver.one.com:9092,kfk02.server.two.com + KAFKA_TOPIC=status_checker + + POSTGRES_DB=chweb + POSTGRES_HOST=localhost + POSTGRES_PORT=5432 + POSTGRES_USER=vladan + POSTGRES_PASS=secret diff --git a/docs/source/index.rst b/docs/source/index.rst index 4d3c31e..ce0fe2b 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,17 +3,33 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Welcome to chweb's documentation! -================================= +chweb - Check websites state +============================ + +CHWEB is a website checking tool. + +ATM in its very early stages meant to demo +`aiven `_'s platform, using their `kafka +`_ and `postgresql `_ +services. + +Features +-------- + +* Easy and intuitive YAML configuration. +* Command line executables for the checker and the consumer/psql writer: + ``chweb_collect`` and ``chweb_consume``. +* Environment configuration. + .. toctree:: :maxdepth: 1 - :caption: Contents: + configuration + environment + architecture apidocs - - Indices and tables ================== diff --git a/tox.ini b/tox.ini index a4bcfcc..c740a5d 100644 --- a/tox.ini +++ b/tox.ini @@ -26,7 +26,9 @@ commands = [testenv:docs] changedir = docs -deps = sphinx +deps = + sphinx + sphinx-typlog-theme commands = sphinx-build -W -b html -E ./source/ ./build/