OLAT

Test Documentation

MELS - Multimedia & E-Learning Services, University of Zuerich, Switzerland

Table of Contents

Preface
1. Getting Started
1.1. Selenium IDE with Firefox - Install and Intro
Installation
Configure the Olat Map file
Recording the first test case
1.2. Selenium/Olat Map-file aka Locator Abstraction Layer
Introduction
Olat Map file
1.3. Selenium IDE Test Recording Cookbook
Language Dependency
Handling of Login
Locator encoding
Transition between Selenium IDE Test Case and 'Automated Nightly Selenium Test Case'
1.4. Selenium RC (Remote-Control / Server) with Eclipse
Install Selenium RC
Run a Selenium test from Eclipse
Modifying test for nightly
2. An overview of Testing in Olat
2.1. This document
2.2. Test Concept
Overall Test Goal
Test Matrix
2.3. Infrastructure and Frameworks
Selenium
Codepoints and Breakpoints
Setup Helper Scripts/Framework
TestNG
2.4. Deployment
2.5. Outstanding: Next steps and ideas
Distributed Deployment (DTF?)
Codepoints extension: Time Measurements between Codepoints
Test data concept: DBUnit / Reset-DB - Concept
JUnitReport to include relevant parts of olat.log from all involved nodes
3. Test Patterns
3.1. Distributed, clustered breakpointing
3.2. Contension simulation using random delays, combined with looping/stress-testing
3.3. Test by Asserts
Measure critical location times and apply weak-asserts
Managed Block: force DB access to be only allowed within managed block - plus optimize single-node case without plock
4. Selenium
4.1. Quick introduction to Selenium
Selenium IDE
Selenium RC
Selenium Grid
5. Codepoints and Breakpoints
5.1. Quick introduction to codepoints
6. Setup Helper Scripts/Framework
6.1. Quick introduction to Setup Helper Scripts/Framework
7. TestNG
7.1. Quick introduction to TestNG
7.2. Tips and Tricks with TestNG
8. Deployment Description
8.1. Nightly / Idmels03-05

List of Tables

2.1. Test Matrix

Preface

This documentation is intended for OLAT developers. It explains the test concept and framework used for testing the OLAT system both in standalone and clustered mode.

Chapter 1. Getting Started

1.1. Selenium IDE with Firefox - Install and Intro

Here's a quick guide on how to get started with Selenium IDE

Installation

  • Get Selenium IDE from here

  • Install Selenium IDE as plugin in Firefox

  • Restart Firefox

  • Start Selenium IDE: under Tools there's now a new entry called Selenium IDE: opens a new window

As a result you should get an empty Selenium IDE window. Notice that there is a list of test cases (with one entry 'Untitled') in this case) to the left of the table. You can make it appear by moving the resize indicator in the left.

Here's what it should look like

An empty Selenium IDE

Configure the Olat Map file

We'll go into details about how the Olat Map file works later. Here's how to configure it in Selenium IDE

  • Go to Options>Options...

    Setting for the Olat Map file in Selenium IDE

  • Configure the user-extensions.js file with one of the following paths:

    • Copy the nightly generated user-extensions.js from http://nightly.olat.uzh.ch/nightly/raw/doc/user-extensions.js to a local file and configure that local file. Make sure to repeat this often to catch changes in the olat-ui-map!!!

    • if you're doing modifications on the map file (see later), build it yourself and configure it, e.g.: C:\eclipse\workspace\seleniumtests\build\classes\user-extensions.js

  • Enable 'Remmber base URL'

  • Enable 'Record assertTitle automatically'

  • Enable 'Record absolute URL'

  • Enable 'Enable UI-Element' (important!)

  • Restart Selenium IDE (not the whole browser, just Selenium IDE. Activates the UI-Elements)

Recording the first test case

This section walks through recording the first test case with Selenium IDE

  • Make sure the Record button in the right end of the toolbar is switched on

  • Open https://nightly.olat.unizh.ch/nightly/dmz/

  • Enter 'administrator' and the corresponding password and click 'Login'

  • This should result in something like this:

    The first test case in Selenium IDE

  • To rerun the test, click on one of the green Play buttons

A few things Gotchas here

  • Currently, Selenium IDE sometimes has refresh problems and you don't see the recorded steps until you move the mouse into the command table

  • Selenium sometimes has troubles catching keyboard input when Firefox does form-autocompletion.

  • Sometimes Selenium records 'click' instead of 'clickAndWait' and rerunning fails because of this.

  • Selenium-Recording is not always 100% automated - in fact there is quite a bit of manual effort involved and you can do it all manually in Selenium IDE as well. For example: you can easily type 'clickAndWait' in the command field and enter the target link in the target field yourself - instead of relying on the record mechanism

  • You can set Breakpoints in Selenium IDE by right clicking on any line and selecting 'Toggle Breakpoint' [B]

  • You can run single Selenium commands by double-clicking on a line

  • Given that you are on the right page, you can let Selenium visualize a UI-Element by clicking on the 'Find' button. It will let the element blink green for a while

Make sure to play with the basics of Selenium such as Running a test, Breakpointing, Stepping, Finding a UI-Element before going forward

1.2. Selenium/Olat Map-file aka Locator Abstraction Layer

Introduction

The start of a normal Selenium test case in OLAT could look something like this (for details refer to the Selenium in-depth chapter):

  • open http://localhost:8080/nightly/dmz

  • type <username-locator> administrator

  • type <password-locator> the_password

  • clickAndWait <login-locator>

The tricky part lies in the locators: these are generally speaking XPaths into the XHtml DOM tree.

In order to achieve the goal of having tests that survive between releases, and at the same time allow structural, graphical and internal changes that effect only the DOM tree and not complete workflows, there must be a concept of having locator URLs independent from the current XPath - i.e. we need to have a Locator Abstraction Layer

Luckily Selenium already comes with a concept called 'UI-Element'. If you have Selenium IDE installed, check here for the documentation

'UI-Element' allows defining customized locator. This means that we can define an alias locator, use that in the test and hide the implementation in a OLAT-version-specific Olat Map file. This allows tests to remain stable from release to release depending on the stability of workflows! If workflows change from release to release, then obviously tests that tested based on the old workflow need to be adjusted

Olat Map file

The Olat Map file contains a hierarchically structured list of UI-Element locators mapped to XPaths

These UI-Element locators are then used by Selenium tests and are the only locators that must be used within an Olat Selenium test (enforced by the base testclass BaseSeleneseTestCase)

The Olat map file is stored here:

$CVS/olat3ext/tests/seleniumtests/ui-map/olat-ui-map.xml

Here is an excerpt from the olat-ui-map.xml file:

		 	
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE ui-map PUBLIC "olat/ui-map" "olat-ui-map.dtd">

<ui-map>
	<pageset name="dmz" description="DMZ links">
		<xpath-ui-element name="username" xpath="//input[@id='lf_login']" description="username input field"/>
		<xpath-ui-element name="password" xpath="//input[@name='lf_pass']" description="password input field"/>
		<xpath-ui-element name="login" xpath="//input[@value='Login']" description="login button"/>
	</pageset>
	<pageset name="home" description="The home page">
		<pageset name="menu" description="the left hand menu">
			<xpath-ui-element name="settings" xpath="//div[contains(@class, 'o_home')]//div[@class='b_tree']//a[text()='Settings']" description="left navigation link 'Settings'"/>
		</pageset>
	</pageset>
</ui-map>
		 	

The above map translates into the following UI-Element locators:

ui=dmz::username()
ui=dmz::password()
ui=dmz::login()
ui=home::menu_settings()

Here is a description of the steps involved in converting the olat-ui-map.xml file into user-extensions.js

  • /olat3ext/tests/seleniumtests/build.xml triggers the whole process in target 'gen_olat-ui-map.js'

  • /olat3ext/tests/seleniumtests/ui-map/olat-ui-map.xml is the input - the actual olat ui map file

  • /olat3ext/tests/seleniumtests/ui-map/gen_olat-ui-map.xsl converts the olat-ui-map.xml into /olat3ext/tests/seleniumtests/build/classes/olat-ui-map.js

  • the three files ui-element.js, olat-ui-map.js and rollups.js are then concatenated into /olat3ext/tests/seleniumtests/build/classes/user-extensions.js which is the final user-extensions file that Selenium needs to be configured with

1.3. Selenium IDE Test Recording Cookbook

This section is about how to record selenium tests for Olat and how they get integrated into the nightly test infrastructure

Language Dependency

Virtually all links in Olat are language dependent - therefore the Olat Map file is also language dependent. This means that the test will only succeed, if the current language is correctly set to English. For test recording this is best configured before start. For the nightly test framework, this is handled by the BaseSeleneseTestCase (the base test class)

Therefore: always switch to English before you start testing

Handling of Login

Logging into Olat is going to be the first task that every test needs to execute. At the same time it is somewhat complicated since we have to make sure, that we're running in English mode.

To simplify all this, there is framework code that handles the login:

BaseSeleneseTestCase.olatLogin(selenium,url,username,password,language)

Note that by default we should be using 'English' as the test language since the olat-ui-map file only works with English

What this means for test recording in Selenium IDE is, that the input for the automated Selenium test classes only need the part after the login

Locator encoding

As described in the Olat Map file section earlier, tests are only maintainable if they don't use XPaths/Id-Links directly, but use the Olat Map file with their UI-Element locator notation (ui=dmz::username()). When recording tests Selenium IDE most of the times finds corresponding mapped locators and automagically records Ui-Element locators instead of XPaths or IDs. If that's the case then everything is fine. You will surely come across non-encoded locators though - or special cases which are UI-Element locators that are parametrized. In these cases you must manually fix the locator

Here's a list of known parametrized UI-Element locators:

  • ui=learningResources::content_clickCourseEntry(nameOfCourse=Demo course wiki)
  • ui=tabs::closeCourse(nameOfCourse=Demo course wiki)
  • ui=course::menu_link(link=Wiki sandbox)

Therefore: always use locators that start with ui= and add it to the olat-ui-map.js if the link is not encoded yet

Transition between Selenium IDE Test Case and 'Automated Nightly Selenium Test Case'

A Selenium test recorded in Selenium IDE can be stored in a Selenium-own format (HTML) or converted into any programing language.

Supported conversion formats in Selenium IDE

The process of converting a Selenium IDE test into a nighlty olat selenium test is as follows:

  • Record and store the test in Selenium IDE's own HTML format

    The test example

  • Convert the test into Java

    The test example converted into Java. The green box marks the relevant part

  • Extract the relevant part (marked green above) and put it in a Junit class:

    package org.olat.test.tutorial.singlenode.junit;
    
    import org.olat.test.util.selenium.BaseSeleneseTestCase;
    
    public class TutorialTestTest extends BaseSeleneseTestCase {
    	public void setUp() throws Exception {
        	selenium = connect("idmels04.unizh.ch", "*chrome", "https://nightly.olat.uzh.ch/nightly/dmz/");
    	}
    
    	public void testNew() throws Exception {
    		olatLogin(selenium, "https://nightly.olat.uzh.ch/nightly/dmz/", "administrator", "******", "English");
    		
    		assertEquals("OLAT - Home", selenium.getTitle());
    		selenium.click("ui=tabs::learningResources()");
    		selenium.waitForPageToLoad("30000");
    		selenium.click("ui=learningResources::menu_courses()");
    		selenium.waitForPageToLoad("30000");
    		selenium.click("ui=learningResources::content_clickCourseEntry(nameOfCourse=Demo course wiki)");
    		selenium.waitForPageToLoad("30000");
    		selenium.click("ui=learningResources::content_showContent()");
    		selenium.waitForPageToLoad("30000");
    		selenium.click("ui=tabs::closeCourse(nameOfCourse=Demo course wiki)");
    		selenium.waitForPageToLoad("30000");
    		assertEquals("OLAT - Learning resources", selenium.getTitle());	
    	}
    }
    					

1.4. Selenium RC (Remote-Control / Server) with Eclipse

Selenium IDE is the tool for recording and playing back directly in the browser, while Selenium RC (Remote-Control) is for testing remotely on the one hand and automated browser-launch on the other hand.

Install Selenium RC

Download Selenium RC here

But note: Version 1.0 beta 1 does NOT work with Firefox 3 - therefore download the nightly version or ask the developers for the right version since nightly is very unstable at times

Start Selenium RC as follows:

  • Open a command shell

  • Change into selenium-remote-control../selenium-server-.. directory

  • Make sure you have formerly compiled the user-extensions.js file

  • Then start Selenium RC with this command:

    java -cp ./selenium-server.jar org.openqa.selenium.server.SeleniumServer -userExtensions \eclipse\workspace\seleniumtests\build\classes\user-extensions.js
    	  				

    And you are ready to go!

Run a Selenium test from Eclipse

Use the tutorial examples in $CVS/olat3ext/tests/seleniumtests/src/org/olat/test/tutorial as starting points.

The main idea is to have your JUnit test class extend BaseSeleneseTestCase and call the two main features:

  • connect(seleniumHostname, browserId, baseTestUrl);

  • olatLogin(seleniumInstance, baseTestUrl, username, password, language);

Modifying test for nightly

After a test was successfully made run locally, it needs some tweaking to run nightly as well

First of all, the selenium host is no longer localhost but idmels04.uzh.ch instead. (at least for the moment)

Second, the baseTestUrl might have to be adjusted, e.g. test against nightly.uzh.ch

Finally any other setup which was dependent on having an olat instance or cluster running locally or anywhere else needs to be migrated to nightly - that would include using the Setup Framework as described later in this document

Chapter 2. An overview of Testing in Olat

2.1. This document

2.2. Test Concept

Overall Test Goal

The declared goal of testing in Olat is to achieve a high degree of automatism

Any guarantee of high code quality that can be achieved in an automated way - for example in a nightly test - is very valuable. Having automated tests allows to both keep test costs low and achieve an improvement of quality over release-cycles rather than a decrease due to higher complexities etc.

This automatism is not feasible on exactly all levels of testing, but the goal is to automate where it makes sense from a effort/gain point of view

Test Matrix

Table 2.1. Test Matrix

 Functional TestingLoad/Stress TestingPerformance/Scalability Testing
Class
  • JUnit

  • TestNG

  • Codepoints

  • JUnit

  • TestNG

 
Module
  • JUnit

  • TestNG

  • Codepoints

  • JUnit

  • TestNG

  • Codepoints

 
Node
  • JUnit

  • TestNG

  • Codepoints

  • Selenium

  • JMeter

  • Webdriver/Selenium

  • JMeter

  • Webdriver/Selenium

Cluster
  • JUnit

  • TestNG

  • Codepoints

  • Selenium

  • JMeter

  • Webdriver/Selenium

  • JMeter

  • Webdriver/Selenium


2.3. Infrastructure and Frameworks

Selenium

Codepoints and Breakpoints

Setup Helper Scripts/Framework

TestNG

2.4. Deployment

2.5. Outstanding: Next steps and ideas

Distributed Deployment (DTF?)

Codepoints extension: Time Measurements between Codepoints

Test data concept: DBUnit / Reset-DB - Concept

JUnitReport to include relevant parts of olat.log from all involved nodes

Chapter 3. Test Patterns

3.1. Distributed, clustered breakpointing

3.2. Contension simulation using random delays, combined with looping/stress-testing

3.3. Test by Asserts

Measure critical location times and apply weak-asserts

Managed Block: force DB access to be only allowed within managed block - plus optimize single-node case without plock

Chapter 4. Selenium

4.1. Quick introduction to Selenium

Selenium IDE

Selenium RC

Selenium Grid

Chapter 5. Codepoints and Breakpoints

5.1. Quick introduction to codepoints

Chapter 6. Setup Helper Scripts/Framework

6.1. Quick introduction to Setup Helper Scripts/Framework

Chapter 7. TestNG

7.1. Quick introduction to TestNG

7.2. Tips and Tricks with TestNG

Chapter 8. Deployment Description

8.1. Nightly / Idmels03-05