Rx - RIExerciser - Home Networking Testing Tool

Rx Project Management:

Current Project Overall Status:

Summary of Rx Project:

Phase 1 is now complete.  Project is entering intergation phase. Next set of milestones has not yet been established.  For current issues against the Rx project, see OCORI issues with component "Rx".

Here is a summary of OCORI issues that is plaguing the beanshell scripts:

OCORI-4319: level 2 scripts can not be run back-to-back in the same level 3 script.

 Agenda for Monday, May 7th

Review of action items & status from last week

1. See if Mark D issues are related to fresh checkout - Lori A - DONE
2. Review HN OAD to make sure we're only including public APIs which could be cause of class not found exception - Lori A - DONE
   1. Work with Mark D to figure out build issue - Neha
3. Restore Lab B to do nightly run - Scott A - IN-PROGRESS 
4. Update wiki with naming conventions - Neha - DONE
5. Investigate if test exists for background recording - Cherie - ?

Agenda Items:

1. Discuss current script writing process.  Are there areas that could be improved?
   1. Question what should level 1 scripts be doing?  
   2. Question why can't we call OAD driver functions directly from level 2
   3. Strong typing, undeclared variables - Steve A - to investigate if there is a way to enforce
   4. Method names within scripts should be unique
   5. Level 3 = entire spreadsheet, Level 2 = each test w/i spreadsheet, Level 1 = (wrapper - / each step in spreadsheet)
      1. Level 1 tests need rework - remove the level 1 tests that are tested elsewhere - Neha 
      2. Revisit the reason why level 2 can't call OAD functions directly - Steve H to follow up with Marcin 
   6. Different test harness because:
      1. automate result checking - similar to JUNIT results 
         1. Create OCORI to improve status reporting - adding option to report current execution location - Lori A
      2. automate report generation
         1. More info is included via the link
      3. collect stats on time spent, report data - Scott A
   7. Level 1 - is it not just a wrapper or
2. Structure of OAD Rx scripts should look like at different levels 
3. Are the tests spreadsheets impeding us from making RiExerciser Menu changes?
   1. We don't think this should be the case.  Spreadsheet should be worded so that they are not dependent of specific key presses.
4. Method naming convention - needs to be unique, prefixed with containing script abbreviation?
   1. Add new convention about prefixing script name to method names - Lori 
5. Are there scripts for HN segmented recording and live streaming playback? 
   1. Where should tests/requirements be documented?  Spreadsheet?  Wiki?  OCORI? - 
      1. Use spreadsheet, include link on wiki to spreadsheet - Neha  
6. Bug scrub

Past Meeting Agenda & Minutes - Meeting Notes

Rx Project Overview:

The RxProject is an effort to take the current DvrExerciser OCAP Xlet and refactor it into a frame work which provides the following:

1. A library of functionality, OcapAppDriver, which aggregates OCAP API calls into functional building blocks provided by RI Stack & PC Platform. 
2. A scripting interface, RiScriptlet, which allows automation of testing of these functional building blocks.
3. A "Guide" Type OCAP Xlet which is referred to as the RiExerciser which supports functional/application level testing and development through a GUI.

Rx Project Purpose:

QUESTION: We already have a bunch of existing test frameworks (CTP, Swak, DvrTest, etc.), why are we inventing another one?

ANSWER:  

1. DvrExerciser is our current "guide type application" and widely used by RI developer.  It is also used to support DLNA CTT and UPnP CTT tests.  As part of the rewrite, we needed ability to verify its functionality which will be done through this framework.  What better way to test RiExerciser? 
2. RiExerciser which is an ad hoc tool which is currently used by RI developers.  Developers put the effort in to adding additional functionality to DvrExerciser so features can be tested at a functional level.  Why not leverage the results of developer testing?  This framework allows for automation without much additional effort.  An example is the need for automated integration type tests for HN Streaming and Resource Contention Handling.
3. RxProject will be part of the open source so outside contributors will be able to write bean shell scripts to illustrate issues and also demonstrate fixes (can't do this with the CTP framework).
4. Currently have a multiple of Xlets in the QA directory.  Each are required to implement Xlet and there is much redundant code.  If we used the Rx framework instead, could either utilize common utilities already found in OcapAppDriver or add to the common functionality (Rx framework is better than writing individual xlet and duplicating existing code).
5. For HN, we need a framework where tests can be run to verify robustness in a network environment with many devices. HN CTP tests assume an isolated network.

QUESTION: Because of the limitations placed on the OcapAppDriver interface, no callbacks, only simple type calling parameters, minimal thread support, it will never meet the needs of RI developers for testing, can it?  
ANSWER: 

1. If the framework needs additional concurrency support, we could leverage some existing code such as the Concurrency Utils Scott D recommended, DvrTest or Swak framework classes, etc.  
2. The purpose of this framework is to meet the majority of our integration type test needs.  We will adhere to our existing rules and keep the framework simple to meet the needs of the majority of our tests.  Test cases which are so complex that can't be performed in this framework, will need to be performed elsewhere.

Rx Project Phases:

The Rx Project has been divided into the following phases along with proposed dates for completion:

Phase 0: Minimal Functionality required to demonstrate the POC UC (Completion Date: Monday, Dec. 5, 2011):

* Proof Of Concept Use Case (POC UC):

1. Tune
2. Make a 30 sec recording
3. Publish recording to CDS
4. Playback the remote recording

Phase 1: Implement all current functionality that is in DvrExerciser (Completion Date: Monday, Feb. 13, 2012)

Rx Project Requirements:

High Level Requirements and Requested Features

RxProject Follow On Work & Suggestions

Rx Design & User Documentation:

"How It Works" Design Documentation

"How to Add A New Function" Design Documentation

"How the GUI Works" User Documentation

"Regression Testing" Users Guide

How to use the RiScriptlet to run a script

Rx Script Guidelines

Rx Project Testing:

Nightly Testing Strategy

UI Testing Strategy

Rx Project Related Documentation:

Power Point presentation describing initial architecture proposal

Power Point presentation of Existing set of features of DVR Exerciser

"Existing Features of DVRExerciser"