Rx - RIExerciser - Home Networking Testing Tool

Rx Project Overview:

The RxProject is an effort to take the current DvrExerciser OCAP Xlet and refactor it into a frame work which provides the following:

A library of functionality which aggregates OCAP API calls into functional building blocks provided by RI Stack & PC Platform.
A scripting interface which allows automation of testing of these functional building blocks.
A "Guide" Type OCAP Xlet which is referred to as the RxExerciser which supports functional/application level testing and development through a GUI.

Rx Project Purpose:

QUESTION: We already have a bunch of existing test frameworks (CTP, Swak, DvrTest, etc.), why are we inventing another one?

ANSWER:

DvrExerciser is our current "guide type application" and widely used by RI developer. It is also used to support DLNA CTT and UPnP CTT tests. As part of the rewrite, we needed ability to verify its functionality which will be done through this framework. What better way to test RiExerciser?
RiExerciser which is an ad hoc tool which is currently used by RI developers. Developers put the effort in to adding additional functionality to DvrExerciser so features can be tested at a functional level. Why not leverage the results of developer testing? This framework allows for automation without much additional effort. An example is the need for automated integration type tests for HN Streaming and Resource Contention Handling.
RxProject will be part of the open source so outside contributors will be able to write bean shell scripts to illustrate issues and also demonstrate fixes (can't do this with the CTP framework).
Currently have a multiple of Xlets in the QA directory. Each are required to implement Xlet and there is much redundant code. If we used the Rx framework instead, could either utilize common utilities already found in OcapAppDriver or add to the common functionality (Rx framework is better than writing individual xlet and duplicating existing code).
For HN, we need a framework where tests can be run to verify robustness in a network environment with many devices. HN CTP tests assume an isolated network.

QUESTION: Because of the limitations placed on the OcapAppDriver interface, no callbacks, only simple type calling parameters, minimal thread support, it will never meet the needs of RI developers for testing, can it?
ANSWER:

If the framework needs additional concurrency support, we could leverage some existing code such as the Concurrency Utils Scott D recommended, DvrTest or Swak framework classes, etc.
The purpose of this framework is to meet the majority of our integration type test needs. We will adhere to our existing rules and keep the framework simple to meet the needs of the majority of our tests. Test cases which are so complex that can't be performed in this framework, will need to be performed elsewhere.

Rx Project Management:

Current Project Overall Status:

Finish UI implementation for Phase 1 and work on RiScriptlet scripts, focusing on HN streaming tests, completion date is Monday, Feb. 13th, 2012

Meeting Agenda & Minutes for Monday, Dec. 12th

Progress on the smoke test rework – launching, monitoring and reporting of bean shell scripts framework – Parthi & Steve A
1. Parthi completed rework
2. Steve A has telnet method call in place, has a couple questions for Marcin
3. Steve A to explore using java stack trace to support logging of call chain
Current status of RiExerciser - Nicolas
1. Nicolas has addressed issues identified last week, continuing with implementation
Current status of automating HN Streaming Tests – Parthi
1. Parthi will continue to work on this, get Cherie up to speed this week
2. Talked about various apis needed to support tests such as getMediaTime(), setting title, etc.
java.security.AccessControlException: access denied issue with RiScriptlet
1. Steve A & Nicolas will review Xlet methods - constructor, initXlet(), startXlet(), determine cause of this exception
LOGGING- The Logging EC will be implemented soon..I noticed RIScriptlet uses log4j (which is available because it's in the apps classpath)...to get ready for it, I'd suggest changing Logger in the apps/qa folder to have a getLogger static method that just calls getInstance..then any app
1. App classes except xlet: - Declare loggers using a NON-STATIC declaration: Logger log = Logger.getLogger(Classname.class);
2. Xlet: Don't define ANY classes, including app classes, as static Define the logger but don't initialize it In initXlet:First initialize the logging framework: propertyconfigurator.configure(blah);- AND THEN define the logger:log = Logger.getLogger(Classname.class);
3. Steve M will follow up with Scott D on suggestions
Other Xlet advice:
1. Don't reuse existing util classes like OcapTuner (it's used by other xlets and doesn't leverage log4j) Don't run long-running tasks on the UI thread
2. Then there's lazy initialization of singletons: don't do it...either avoid singletons and pass the helper in, make all the methods static, or declare and initialize your private instance at the same time.
Ocap Tuner Reuse
1. Discuss this in 2 PM meeting today
Hidden Content Testing
1. Scott A gave overview, will setup another meeting to discuss in more detail
Dogfooding - using our own software - RiScriptlet scripts to test software changes

Current Action Items

Follow up on logging change suggestions - Steve M
Complete telnet i/f shutdown - Steve A
Investigate using java stack trace to support calling chain logging - Steve A
Setup crucible reviews for code check-ins for RxProject - Lori

Rx Project Weekly Monday Meeting Agenda & Notes