[WIP] Add/exporters

.circleci/config.yml

@@ -36,7 +36,7 @@ install_python_3: &install_python_3
        fi
 
 install_python_2: &install_python_2
-  name: install Python 3.5 dependencies
+  name: install Python 2 dependencies
   command: | 
       ls $HOME
       if [ ! -d "/home/circleci/conda" ]; then

CHANGELOG.md

@@ -13,6 +13,7 @@ Critical items to know are:
  - changed behaviour
 
 ## [master](https://github.com/vsoch/watchme/tree/master)
+ - Addition of exporters (0.0.17)
  - Adding option for regular expression for URL wachers, user agent header (0.0.16)
  - requests is missing from install dependencies (0.0.15)
  - small bug fixes (0.0.14)

docs/_data/links.yml

@@ -12,5 +12,7 @@ navigation:
   url: watchers/
 - name: Examples
   url: examples/
+- name: Exporters
+  url: exporters/
 - name: Contributing
   url: contributing/

docs/_docs/exporters/index.md

@@ -0,0 +1,115 @@
+---
+title: WatchMe Exporters
+category: Exporters
+permalink: /exporters/index.html
+order: 1
+---
+
+Once you've followed the [getting started]({{ site.baseurl }}/getting-started/)
+guide to [install]({{ site.baseurl }}/install/) watchme and have created your first 
+set of watchers, you might want to do more! Specifically, you can add 
+extra exporters to push your data to custom locations. The following exporters
+are available:
+
+ - [pushgateway]({{ site.baseurl }}/exporters/pushgateway/) to push data to a Prometheus gateway.
+
+And instructions for using a general exporter, either in sync with running a task
+or separately, are provided below.
+
+
+## List Exporters
+
+To see exporters available, you can list them:
+
+```bash
+$ watchme list --exporters
+watchme: exporters
+pushgateway
+```
+
+An exporter is like a task in that it can be activated
+(or not), and you can define more than one for any particular watcher.
+
+## Add an Exporter
+
+The most basic thing to do is to add an exporter to a watcher. You should
+read the documentation for your exporter of interest (linked above) to
+know the required parameters for the exporter (it varies). 
+Adding an exporter is similar to adding a task! The command asks you to
+name the exporter, and the watcher to which you are adding it:
+
+```bash
+$ watchme add-exporter <watcher> <exporter>
+```
+
+Most likely the exporter requires parameters, so you add these in the same way
+you added parameters to a task. Notice that "type" is essential to designate the
+type that you want (the list at the top of the page):
+
+```bash
+$ watchme add-exporter <watcher> exporter-<name> param@value type@<exporter_type>
+```
+
+If you already have tasks defined that you want the exporter added to, just add them to the command:
+
+```bash
+$ watchme add-exporter <watcher> exporter-<name> param@value type@<exporter_type> <tasks>
+```
+
+If you've already added the exporter to the watcher and want to add it to a task:
+
+```bash
+$ watchme add-exporter <watcher> <exporter> <task>
+```
+
+Watchme will know that you aren't adding a new exporter (but instead using
+an existing one) because you haven't provided a `type@<exporter_type>`
+variable. If you want to re-create an exporter that already exists, you
+can issue the same command but you'll need `--force`.
+
+```bash
+$ watchme add-exporter <watcher> exporter-<name> param@value type@<exporter_type> --force
+``` 
+
+## Remove an Exporter
+
+If you change your mind, you can remove an exporter from a watcher entirely:
+
+```bash
+$ watchme remove <watcher> <exporter>
+```
+
+You can also remove it from a specific task (but not from the watcher entirely)
+
+```bash
+$ watchme remove <watcher> <exporter> <task>
+```
+
+
+## Run the Exporter
+
+By default, an exporter will run with a task. This means that you should
+run the task to run the exporter. If the watcher is active, and the task
+is active, and the exporter is defined for the task, it will run.
+
+```bash
+$ watchme activate <watcher>
+$ watchme run <watcher> <task>
+```
+
+If an exporter doesn't run successfully, it will be disabled for the task.
+This is in case you forget about it.
+
+
+## Push Manually
+
+Optionally, you can choose to push data to an exporter manually.
+This will ignore if the exporter is active, and only check that it's 
+defined for the task and is valid.
+
+```bash
+$ watchme push <watcher> <exporter> <task> <filename>
+```
+
+If you want to customize how the export works (e.g., what commits)
+then you can use the python functions directly.

docs/_docs/exporters/pushgateway/pushgateway.md

@@ -0,0 +1,243 @@
+---
+title: Pushgateway
+category: Exporters
+permalink: /exporters/pushgateway/
+order: 2
+---
+
+The [pushgateway](https://github.com/prometheus/pushgateway) exporter
+will allow you to export data for a watcher to a Prometheus gateway.
+To install the exporter, you will need to install its dependencies:
+
+```bash
+pip install watchme[exporter-pushgateway]
+```
+
+The dependencies include the [prometheus client](https://github.com/prometheus/client_python).
+
+<br>
+
+## Quick Start
+
+Setup a gateway
+
+```bash
+$ docker run -d -p 9091:9091 prom/pushgateway
+```
+
+Create and add the exporter to an existing task. The only required parameter is the
+url for the gateway (above we use http://localhost:9091).
+
+```bash
+$ watchme add-exporter <watcher> exporter-pushgateway url@http://localhost:9091 type@pushgateway task-<task>
+```
+
+<br>
+
+## Detailed Start
+
+### 1. Setup a Gateway
+
+To set up a gateway, you can find full instructions (section "Run It") 
+at [https://github.com/prometheus/pushgateway](https://github.com/prometheus/pushgateway).
+A brief summary using Docker is provided here. First, bring up the gateway:
+
+```bash
+$ docker run -d -p 9091:9091 prom/pushgateway
+```
+
+The image will pull and run, exposing the gateway on port 9091. First use
+`docker ps` to ensure that it's running without issue:
+
+```bash
+$ docker ps
+CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS              PORTS                    NAMES
+fd6e3d3fb8db        prom/pushgateway    "/bin/pushgateway"   7 seconds ago       Up 6 seconds        0.0.0.0:9091->9091/tcp   gifted_merkle
+```
+
+Then, open your browser to [http://localhost:9091/](http://localhost:9091/). You should
+see an empty portal.
+
+![pushgateway.png]({{ site.url }}{{ site.baseurl }}/exporters/pushgateway/pushgateway.png)
+
+It's a fairly boring interface, but that's okay - we will fix it up when we push data to it!
+
+
+### 2. Create a Watcher
+
+Next, let's create a watcher that will collect the data that we want to push to the
+exporter. Imagine that we are interested in monitoring the temperate for a region.
+Let's call the watcher "weather-watcher"
+
+```bash
+$ watchme create temp-watcher
+Adding watcher /home/vanessa/.watchme/temp-watcher...
+Generating watcher config /home/vanessa/.watchme/temp-watcher/watchme.cfg
+```
+
+This is an empty watcher in that there aren't any tasks to run. 
+
+### 3. Create a Task
+Let's create a task to get the temperature for Luxembourg.
+
+```bash
+$ watchme add-task temp-watcher task-luxembourg url@https://www.accuweather.com/en/lu/luxembourg/228714/weather-forecast/228714 [email protected] get_text@true func@get_url_selection type@urls regex@[0-9]+
+
+[task-luxembourg]
+url  = https://www.accuweather.com/en/lu/luxembourg/228714/weather-forecast/228714
+selection  = .local-temp
+get_text  = true
+func  = get_url_selection
+regex  = [0-9]+
+active  = true
+type  = urls
+```
+
+Above, we create a task called "task-luxembourg" that will go to the URL with the weather,
+select the class ".local-temp" on the page, get the text for the class, and filter to the regular expression
+for one or more numbers (`[0-9]+`). If you are interested in these details, see the
+[urls watcher tasks]({{ site.url }}/watchers/urls/).
+
+#### Test the task
+
+We haven't scheduled it to run, but we might try testing it out to make sure
+that it runs as we expect.
+
+```bash
+$ watchme run temp-watcher task-luxembourg --test
+Found 1 contender tasks.
+[1/1] |===================================| 100.0% 
+{
+    "task-luxembourg": [
+        "54"
+    ]
+}
+```
+
+The `--test` flag will ensure that the data is not saved and written to git.
+The above worked great, because we see that the temperature is 54.
+
+### 4. Add an Exporter
+
+Let's now add an exporter. An exporter is like a task in that it can be activated
+(or not), and you can define more than one for any particular watcher.
+If you want to see the exporters available:
+
+```bash
+$ watchme list --exporters
+watchme: exporters
+pushgateway
+```
+
+Adding an exporter is similar to adding a task! The command asks you to
+name the exporter, and the watcher to which you are adding it:
+
+```bash
+$ watchme add-exporter <watcher> <exporter>
+```
+
+And you can follow these labels by any parameters required for the exporter.
+For the pushgateway exporter, along with specifying the type, 
+you are only required one parameter (url) that must start with http.
+
+
+```bash
+$ watchme add-exporter temp-watcher exporter-pushgateway url@http://localhost:9091 type@pushgateway 
+```
+
+Along with seeing it on the screen, you can inspect the watcher to see that
+the exporter is added:
+
+```bash
+$ watchme inspect temp-watcher
+[watcher]
+active  = false
+type  = urls
+
+[task-luxembourg]
+url  = https://www.accuweather.com/en/lu/luxembourg/228714/weather-forecast/228714
+selection  = .local-temp
+get_text  = true
+func  = get_url_selection
+regex  = [0-9]+
+active  = true
+type  = urls
+
+[exporter-pushgateway]
+url  = http://localhost:9091
+active  = true
+type  = pushgateway
+```
+
+But notice how the exporter isn't added to the task? We can add it manually:
+
+```bash
+# watchme add-exporter <watcher> <exporter> <task>
+$ watchme add-exporter temp-watcher exporter-pushgateway task-luxembourg
+```
+
+Or you could have added it when you first created the exporter. Let's show you
+the command to remove the exporter:
+
+```bash
+$ watchme remove temp-watcher exporter-pushgateway
+Removing exporter-pushgateway from task-luxembourg
+exporter-pushgateway removed successfully.
+```
+
+And then add it again, not only to the watcher but also to the task, with one command!
+
+```bash
+$ watchme add-exporter temp-watcher exporter-pushgateway url@http://localhost:9091 type@pushgateway task-luxembourg
+[exporter-pushgateway]
+url  = http://localhost:9091
+active  = true
+type  = pushgateway
+```
+
+You can then inspect the watcher to see the full addition (note that "exporter-pushgateway" is added
+to exporters for task-luxembourg:
+
+```bash
+vanessa@vanessa-ThinkPad-T460s:~/Documents/Dropbox/Code/Python/watchme$ watchme inspect temp-watcher[watcher]
+active  = false
+type  = urls
+
+[task-luxembourg]
+url  = https://www.accuweather.com/en/lu/luxembourg/228714/weather-forecast/228714
+selection  = .local-temp
+get_text  = true
+func  = get_url_selection
+regex  = [0-9]+
+active  = true
+type  = urls
+exporters  = exporter-pushgateway
+
+[exporter-pushgateway]
+url  = http://localhost:9091
+active  = true
+type  = pushgateway
+```
+
+### 5. Run the Exporter
+
+We can see that pushgateway is active, so when we run the task, it should push 
+data there. To do it fully, we need to activate the watcher:
+
+```bash
+$ watchme activate temp-watcher
+[watcher|temp-watcher] active: true
+```
+
+And then run the task.
+
+```bash
+$ watchme run temp-watcher task-luxembourg
+Exporting list to exporter-pushgateway
+```
+
+If all goes well, you can return to the web interface and see your data added!
+
+![pushed.png]({{ site.url }}{{ site.baseurl }}/exporters/pushgateway/pushed.png)
+
+Great job! You can learn more about the use cases for [push gateway here](https://prometheus.io/docs/practices/pushing/).