uptime_monitor (Hercules)

Uptime_monitor is a ragios plugin that uses a real web browser to perform actions on a website to ensure that features of the site are still working correctly. It can check elements of a webpage to ensure they still exist and it can also perform actions like a website login to ensure that the process still works correctly. When uptime_monitor detects a problem with the website, Uptime_monitor can take a screenshot of the web page it sees, and it can also record a video of the actions it performed on the website. The video feature is not yet implemented.

Requirements

Ruby: At least Ruby 2.4.1 or higher is recommended

Ragios

Installation:

Add the uptime_monitor gem to your ragios Gemfile

 gem "uptime_monitor"

Run bundle install from the ragios root directory

 bundle install

Restart ragios

To run Ragios with uptime_monitor and all its dependencies already setup and configured in Docker Compose, See details here: Using Maestro. This is the easiest way to get Ragios and Uptime_monitor up and running.

Usage:

A quick example, to monitor the title tag of a web page to ensure that it hasn't changed. Using Ragios ruby client

monitor = {
  monitor: "My Blog title tag",
  url: "http://obi-akubue.org",
  every: "5m",
  contact: "[email protected]",
  via: "ses",
  plugin: "uptime_monitor",
  exists?: 'title.with_text("Obi Akubue")',
  browser: "firefox"
}
ragios.create(monitor)

The above example will create a ragios monitor that will, every 5 minutes, use firefox to visit the website url http://obi-akubue.org, and verify that the home page title tag matches the text "Obi Akubue". The validations performed are defined in the exists? key/value pair, in this statement.

exists?: 'title.with_text("Obi Akubue")'

When the title tag on the web page doesn't match the text "Obi Akubue", a failure notification will be sent out to the provided contact "[email protected]".

Using the plugin

To use the uptime monitor plugin add the key/value pair to the monitor

plugin: "uptime_monitor"

Browsers

A browser is specified, by adding a browser key/value pair to the monitor

browser: "firefox"

Supported browsers include Firefox, Chrome, Safari and Phantomjs. Other browsers can be specified as

browser: "chrome"
browser: "safari"
browser: "phantomjs"

uptime_monitor uses Watir, to easily access the different browsers see the development section, running uptime_monitor with Selenium Grid and Docker Compose.

Validations

To verify that a html element exists on the web page, a validation needs to be added to the monitor. Validations are specified with the exists? key/value pair which takes a string of html elements as it's value. It verifies that the html elements in the array exists on the current web page.

exists?: "h1 div"

The above example will verify that a h1 and a div exists on the page.

HTML Elements

The simplest way to specify a html element is using the element name.

exists?: "h1 div a img span"

HTML elements can also be specified with their attributes in a ruby-like syntax.

exists?: 'div.where(class: "box_content")'

The above will verify that a div with class attribute equals to "box_content" exists on the page.

Other examples:

img.where(src: "https://fc03.deviantart.net/fs14/f/2007/047/f/2/Street_Addiction_by_gizmodus.jpg")

Specifies an img tag with src="https://fc03.deviantart.net/fs14/f/2007/047/f/2/Street_Addiction_by_gizmodus.jpg".

div.where(id:"test", class: "test-section")

Specifes a div with id="test" and class="test-section".

Multiple validations

Multiple validations can be added to a monitor, see below:

validations = <<-eos
  div.where(class: "box_content")
  img.where(src: "https://fc03.deviantart.net/fs14/f/2007/047/f/2/Street_Addiction_by_gizmodus.jpg")
  div.where(id:"test", class: "test-section")
eos

monitor = {
  monitor: "My Blog title tag",
  url: "http://obi-akubue.org",
  every: "5m",
  contact: "[email protected]",
  via: "ses",
  plugin: "uptime_monitor",
  exists?: validations,
  browser: "firefox"
}

ragios.create(monitor)

In the above example the string variable validations contains validations that need to be performed when the monitor runs. During execution, the monitor will visit the website and verify that all the html elements specified in the validations exists on the page, if any of them doesn't exist, a notification of the failure will be sent out.

Standard attributes

Only standard attributes for an element can be included in the above format div.where(id: "test"), for example a div has the following standard attributes id, class, lang, dir, title, align, onclick, ondblclick, onmousedown, onmouseup, onmouseover, onmousemove, onmouseout, onkeypress, onkeydown, onkeyup. So div.where(class: "anything") will work.

But custom or data attributes cannot be included in the same format, for example

<div data-brand="toyota">

The following

div.where(data-brand: "toyota")

will give an error because "data-brand" is not a standard attibute for div, to specify elements by data or custom attributes use css selectors, see below.

Using CSS Selectors

HTML elements can also be specified with css selectors.

element.where(css: "#rss-link")

This specifies an element with id="rss-link".

To specify an element by data attributes

element.where(css: "[data-brand='toyota']")

Helpers for HTML elements

Helpers are available to make some elements easier to reason about:

Links

An anchor tag could be specified with a link helper, this makes it more readable and easier to reason about.

Using the anchor tag

a.where(text: "Click Here")

Buttons

button.where(id: "searchsubmit")

Text Fields

text_field.where(id: "search")

Checkboxes

checkbox.where(value: "Butter")

Radio Buttons

radio.where(name: "group1", value: "Milk")

Drop Down menus

<select name="mydropdown">
<option value="Milk">Fresh Milk</option>
<option value="Cheese">Old Cheese</option>
<option value="Bread">Hot Bread</option>
</select>

Helper

select_list.where(name: "mydropdown")

Or HTML select tag

select.where(name: "mydropdown")

Options of the drop-down menu can be specified using option

option.where(value: "Milk")

Text Validations

A text validation is used to verify that the text content of a html element hasn't changed. For example,

exists?: 'title.with_text("Welcome to my site")'

The above example first verifies that a title tag exists on the page, then it verifies that title tag text is equal to "Welcome to my site". The above example is a text validation. Text validations can also verify that the html element's text includes the provided string:

exists?: 'title.includes_text("Welcome")'

The above example verifies that the title tag's text includes the string "Welcome". Another example, to verify that a div with class="box_content" includes the string "SouthMunn is a Website"

exists?: 'div.where(class: "box_content").includes_text("SouthMunn is a Website")'

Text validations can be used on html elements that can contain text like title, div, span, h1, h2 etc.

Actions

Validations can also include actions. The actions are performed on the html element after it is verfied that the element exists. Example to set a text field's value

exists?: 'text_field.where(id: "username").set("admin")'

The above example is an action that will set the text field's value to the string "admin".

Actions on html elements

Common actions performed on elements are set, select and click.

For example to set value for a textfield or textarea.

text_field.where(name: "q").set("ruby")
text_area.where(name: "longtext").set("In a world...")

Select an option from a drop down menu

<select name="mydropdown">
<option value="Milk">Fresh Milk</option>
<option value="Cheese">Old Cheese</option>
<option value="Bread">Hot Bread</option>
</select>

select_list.where(name: "mydropdown").select("Old Cheese")

Click a radio button, checkbox, link or button

radio.where(name: "group1", value: "Milk").click
checkbox.where(name: "checkbox").click
link.where(text: "Click Here").click
button.where(id: "submit").click

Waiting

For webpages that use a lot of AJAX, it's possible to wait until an element exists, by using the wait_for keyword. This keyword takes an element as value. It is a special type of validation, it will wait for 30 seconds for the provided element to exist, if the element doesn't exist in 30 seconds the validation fails.

wait_for div.where(id: "open-section")

The above example will wait 30 seconds until the div exists, if it doesn't exist after 30 seconds the validation will fail.

Multiple validations and actions

validations = <<-eos
  text_field.where(id: "username").set("admin")
  text_field.where(id: "password").set("pass")
  button.click
  title.includes_text("Dashboard")
eos

With multiple validations like the example above, the monitor will run the validations, line by line, from top to bottom. When it meets an action it will apply it on the element in the validation. The monitor fails its test if any of the validation fails. So for the monitor to pass, all validations must pass.

When actions like clicking a link, changes the current page, the following validations will be performed on the new page.

A combination of multiple validations and actions form the basis for performing transactions.

Performing Transactions

Transactions are achieved by a combination of multiple validations and actions.

Example, to monitor the keyword search feature on my blog, notice the validations in the exists? key's value:

steps = <<-eos
  title.with_text("Obi Akubue")
  text_field.where(id: "s").set("ruby")
  button.where(id: "searchsubmit").click
  title.includes_text("ruby").includes_text("Search Results")
  h2.where(class: "pagetitle").includes_text("Search results for")
eos

monitor = {
  monitor: "My Blog: keyword search",
  url: "http://obi-akubue.org",
  every: "1h",
  contact: "[email protected]",
  via: "ses",
  plugin: "uptime_monitor",
  exists?: steps,
  browser: "firefox"
}
ragios.create(monitor)

In the above example, the monitor will visit "http://obi-akubue.org" every hour, and perform a search for keyword 'ruby', then confirm that the search works by checking that the title and h2 tag of the search results page contains the expected text.

Another example, to search my friend's ecommerce site http://akross.net, for a citizen brand wristwatch, add the watch to cart, and go to the checkout page.

This transaction verifies the following about the site:

Product search is working
Add to cart works
Checkout page is reachable
All three above works together as a sequence

add_items_to_cart = <<-eos
  title.with_text("All Watches Shop, Authentic Watches at Akross")
  text_field.where(name: "filter_name").set("citizen")
  div.where(class: "button-search").click
  title.with_text("search")
  link.where(text: "search")
  button.where(value: "Add to Cart").click
  link.where(text: "Checkout").click
  title.with_text("Checkout")
eos

monitor = {
  monitor: "Akross.net: Add citizen watch to cart and checkout",
  url: "http://akross.net",
  every: "1h",
  contact: "[email protected]",
  via: "ses",
  plugin: "uptime_monitor",
  exists?: add_items_to_cart,
  browser: "firefox"
}


ragios.create(monitor)

Another example, to monitor the login process of the website http://southmunn.com

login_process = <<-eos
  title.with_text("Website Uptime Monitoring | SouthMunn.com")
  link.where(text: "login").click
  title.with_text("Sign in - Website Uptime Monitoring | SouthMunn.com")
  text_field.where(id: "username").set("admin")
  text_field.where(id: "password").set("pass")
  button.click
  title.with_text("Dashboard - Website Uptime Monitoring | SouthMunn.com")
eos

monitor = {
  monitor: "My Website login processs",
  url: "https:/southmunn.com",
  every: "1h",
  contact: "[email protected]",
  via: "email_notifier",
  plugin: "uptime_monitor",
  exists?: login_process,
  browser: "firefox"
}

ragios.create(monitor)

Using Uptime Monitor without Ragios

Running Uptime Monitor outside Ragios with Docker Compose

First clone the uptime_monitor Repo on github.

git clone git@github.com:obi-a/uptime_monitor.git

Change to the uptime_monitor directory:

cd uptime_monitor

Build the docker containers:

docker-compose build

This builds the uptime_monitor container

Load the uptime_monitor into PRY console in a container:

docker-compose run  --rm uptime_monitor

This will give you access to the entire uptime_monitor and all its objects loaded into PRY console. It will also run Selenium Grid and firefox in separate docker containers already connected to the uptime_monitor.

Testing the validations outside Ragios

Sometimes it's useful to run validations outside Ragios to verify that the validations are syntactically correct and don't raise any exceptions. This is best done by running the uptime_monitor plugin as a Plain Old Ruby Object.

First load uptime_monitor into PRY console:

docker-compose run uptime_monitor

From the console you can load the uptime_monitor directly:

monitor = {
  url: "http://obi-akubue.org",
  browser: "firefox",
  exists?: "title div"
}

u = Ragios::Plugins::UptimeMonitor.new
u.init(monitor)
u.test_command?
# => true
u.test_result
#=> {
#     :results =>
#       [
#         ["title", "exists_as_expected"],
#         ["div", "exists_as_expected"]
#       ]
#   }

# test result for a failed test during downtime
monitor = {
  url: "http://obi-akubue.org",
  browser: "firefox",
  exists?: 'title.with_text("something")'
}

u.init(monitor)
u.test_command?
# => false
u.test_result
# => {
#     :results =>
#       [
#         ["title, with text \"something\"", "does_not_exist_as_expected"]
#       ]
#   }

In the above example the test_command? method runs the validations and returns true when all validations passes, returns false when any of the validation fails. test_result is a hash that contains the result of the tests ran by test_command?.

Testing individual validations

It can be very useful to test validations/actions individually before adding them to Ragios. This can be done by running plugin's browser directly.

url= "http://obi-akubue.org"
browser_name = "firefox"
browser = Hercules::Maestro::Browser.new(url, browser_name)

browser.exists? 'title.includes_text("ruby")'

browser.exists? 'text_field.where(id: "s").set("ruby")'

browser.exists? 'checkbox.where(name: "checkbox").click'

browser.close

The above example will launch firebox and open the provided url. The exists? method takes a single validation/action as parameter and performs the validation on the current page, it returns true if the validation passes and returns false if the validation fails. In the first validation

browser.exists? 'title.includes_text("ruby")'

it checks if the title tag on the current webpage includes the text 'ruby'.

Screenshots

The uptime_monitor can be configured to take a screenshot of the webpage when a test fails. This screenshot is uploaded to Amazon s3 and its url is included in the test_result. So the website admin can see what the site looks like when transaction failed.

This feature is disable by default, to enable it set following environment variable.

RAGIOS_HERCULES_ENABLE_SCREENSHOTS=true

Also set environment variables for the Amazon s3 account you want to use for storing the screenshots, add the AWS Access Key ID and AWS SECRET ACCESS KEY for the AWS account with S3 access.

S3_AWS_ACCESS_KEY_ID=xxxxxxx
S3_AWS_SECRET_ACCESS_KEY=xxxxxx
RAGIOS_HERCULES_S3_DIR=xxxxxx

The above env vars are for the Amazon AWS access key, AWS secret key and s3 directory/bucket you want to use for storage. First create this s3 bucket manually.

With the screenshots feature enabled, the results of a failed test will include a screenshot of the webpage when the test failed. See an example below:

monitor = {
  url: "http://obi-akubue.org",
  browser: "firefox",
  exists?: 'title.with_text("dont_exist")'
}

u = Ragios::Plugins::UptimeMonitor.new
u.init(monitor)
u.test_command?
#=>false
u.test_result
#=>  {
#      :results=>
#        [
#          ["title, with text \"dont_exist\"", "does_not_exist_as_expected"]
#        ],
#      :screenshot=>
#        "https://screenshot-ragios.s3.amazonaws.com/uploads/screenshot1428783237.png"
#    }

Notice that test_result includes a url to the screenshot of the webpage when the test failed. This test result is also included in the notifications sent to site admin by Ragios when a test fails. So this way the admin can see exactly what webpage looked like when the transaction failed.

Disable screenshots on individual monitors

To disable screenshots on a particular monitor in Ragios add the key/value pair disable_screenshots: true to the monitor. For example:

monitor = {
  url: "http://obi-akubue.org",
  browser: "firefox",
  exists?: "title",
  disable_screenshots: true
}

ragios.create(monitor)

This will diable screenshots only for this monitor, no screenshots will be taken when its test fails.

To run all the unit tests

docker-compose run  --rm unit_tests

License:

MIT License.