Rx - RIExerciser - Home Networking Testing Tool

Rx Project Management:

Current Project Overall Status:

Finish UI implementation for Phase 1 and work on RiScriptlet scripts, focusing on HN streaming tests, completion date is Monday, Feb. 13th, 2012

Agenda and Minutes for Monday, Jan. 23th

Review of action items & status from last week

Create OCORI to track live streaming on same machine issue – Parthi - DONE
Running RxScripts as part of HN Nightly test in HN Lab - Steve A
Remove duplicate RCH tests in spreadsheets – Parthi - DONE
Update spreadsheets to specify acceptable ranges – Parthi - DONE
Using UPnPDiag APIs in tests to determine live point – Parthi
Using telnet interface to segment recordings and TSBs – Cherie
Copyright headers on scripts – Parthi - DONE

Current status of RiExerciser? How should we track what work remains? Should we do a comparison against current DvrExerciser functionality and write up Rx OCORIs to track what remains?
- Nic will enter Rx issues to identify work that still needs to be done to bring RiExerciser up to the functional level of DvrExerciser
- OcapAppDriverTest rx script and Cherie's spreadsheet. Process for adding new functions to OcapAppDriver
  - the group agreed that any new functions added to OcapAppDriver need to have rx test script coverage
  - as a bonus, need to convert original tests from Steve M that are currently in the OcapAppDriverTest stubs
  - also need to update the spreadsheet which Cherie created and checked in - ri\RI_Stack\apps\qa\org\cablelabs\lib\rxdocs\ OcapAppDriverScriptInventory.xlsx
Recording Duration Issue OCORI-4010 – Is this impacting our tests? Can we work around it?
- - - Currently working around it
Media Timing intermittent failures – Are we waiting for events and should we be more precise in our expected range calculations? Or are there other factors that need to be addressed?
- Make modifications to current rx scripts similar to SimpleTrick40 to be more precise regarding timing
  - - - Automating DLNA CTT with Rx

*# Did not discuss, waiting for Scott A return
Clarification regarding filing Rx related bugs:
- If the bug is AGAINST any of the Rx code, select the Jira Component to be "Rx".
- If the bug is EXPOSED by Rx, select the Jira Tag to be "Rx".

Decided that any Rx bug entered should include the Rx tag to aid in filtering of bugs

Using UPnPIncomingMessageHandler to determine live point in rx script –
- Parthi & Lori had follow on discussions after meeting. They decided to add a Telnet I/F to RI to enable "HEAD Response Capture" which would send all HEAD Response as a string through the interface to assist in live streaming testing. Lori will investigate required RI support, Parthi will investigate required script support.
OcapAppDriver Issue – playbackStop() and deleteHnPlayback()
- Discuss player start/stop/create and delete functions, Nicolas will follow up with Steve M
Progress on automation of running of rx scripts
- - All functionality is in place, need Scott A to integrate with HN Lab run
Determination of pass or fail in rx script – if we want to come up with a set of hn streaming "smoke" tests to run when modifying hn code, how would a developer go about this? Just run a level 3 script? How does one determine pass or fail? Is this reported by RiScriptlet?
- Level 3 scripts should be setting rxReturn (Boolean) and rxReturnString (string). rxReturn should be TRUE to indicate pass or FALSE to indicate test failure. The rxReturnString is "free form" and can be a concatenation of all sub test results.
- Discussed how best to support running a test 10 times. It was decided that a level 3 script with a for loop is the best approach and that the rxReturn code would be used to determine PASS or FAIL.
- RI developers should use telnet interface when running rx scripts which are part of the set of "smoke tests" since interface will report PASS or FAIL to interface.

ACTION ITEMS:

Integrate running rx scripts into HN Lab – Scott A
Enter Rx issues to identify work that still needs to be done to bring RiExerciser up to the functional level of DvrExerciser – Nicolas
Determine if current DvrExerciser issues are still present in RiExerciser – Parthi
Add RI support for capturing HEAD responses through telnet interface – Lori
Add rx script support for extracting info from HEAD responses received through telnet interface - Parthi
Decide what changes are required in regard to player stop, start, create and delete methods in OcapAppDriver – Nicolas/Steve M
Modify existing level 3 scripts to set rxReturn and rxReturnString - Parthi
Using telnet interface to segment recordings and TSBs – Cherie

Past Meeting Agenda & Minutes - Meeting Notes

Rx Project Overview:

The RxProject is an effort to take the current DvrExerciser OCAP Xlet and refactor it into a frame work which provides the following:

A library of functionality which aggregates OCAP API calls into functional building blocks provided by RI Stack & PC Platform.
A scripting interface which allows automation of testing of these functional building blocks.
A "Guide" Type OCAP Xlet which is referred to as the RxExerciser which supports functional/application level testing and development through a GUI.

Rx Project Purpose:

QUESTION: We already have a bunch of existing test frameworks (CTP, Swak, DvrTest, etc.), why are we inventing another one?

ANSWER:

DvrExerciser is our current "guide type application" and widely used by RI developer. It is also used to support DLNA CTT and UPnP CTT tests. As part of the rewrite, we needed ability to verify its functionality which will be done through this framework. What better way to test RiExerciser?
RiExerciser which is an ad hoc tool which is currently used by RI developers. Developers put the effort in to adding additional functionality to DvrExerciser so features can be tested at a functional level. Why not leverage the results of developer testing? This framework allows for automation without much additional effort. An example is the need for automated integration type tests for HN Streaming and Resource Contention Handling.
RxProject will be part of the open source so outside contributors will be able to write bean shell scripts to illustrate issues and also demonstrate fixes (can't do this with the CTP framework).
Currently have a multiple of Xlets in the QA directory. Each are required to implement Xlet and there is much redundant code. If we used the Rx framework instead, could either utilize common utilities already found in OcapAppDriver or add to the common functionality (Rx framework is better than writing individual xlet and duplicating existing code).
For HN, we need a framework where tests can be run to verify robustness in a network environment with many devices. HN CTP tests assume an isolated network.

QUESTION: Because of the limitations placed on the OcapAppDriver interface, no callbacks, only simple type calling parameters, minimal thread support, it will never meet the needs of RI developers for testing, can it?
ANSWER:

If the framework needs additional concurrency support, we could leverage some existing code such as the Concurrency Utils Scott D recommended, DvrTest or Swak framework classes, etc.
The purpose of this framework is to meet the majority of our integration type test needs. We will adhere to our existing rules and keep the framework simple to meet the needs of the majority of our tests. Test cases which are so complex that can't be performed in this framework, will need to be performed elsewhere.

Rx Project Phases:

The Rx Project has been divided into the following phases along with proposed dates for completion:

Phase 0: Minimal Functionality required to demonstrate the POC UC (Completion Date: Monday, Dec. 5, 2011):