How We Test The OCAP Reference Implementation

This wiki page will introduce you to the tools and tests we use to assess the quality of CableLab's OCAP reference implementation. You will appreciate our answers to the question, “How is the Reference Implementation tested?”, if you first understand our testing mission, some history of testing the RI code, and our software development processes.

Three distinct missions are met through different strategies. Each mission is supported through its own set of tools, tests, and software development processes. Our missions are:

  • Demonstrate compliance with the OCAP specification
  • Explore and demonstrate the RI as a usable product for the cable industry
  • Deliver high quality software in a timely manner

The OCAP RI is descended from work pioneered by the ODL, a consortium of cable operators. At its peak the ODL employed a large team of professional software testers who wrote tens of thousands of lines of Java code to demonstrate OCAP functionality. The test team architected and coded special tools to automate testing. Their work is available as part of the CableLabs RI open source project.

Professional quality assurance drives our software development and release processes. We use a hybrid agile/scrum/waterfall/whatever-is-needed methodology. Our basic philosophies are:

  • Test early and often. While we have processes to prevent bugs, we also try to discover and contain bugs as quickly as we can.
  • Developers are responsible for unit tests while QA is responsible for integration/system testing, and spec conformance testing.
  • Automate, automate, automate - wherever pragmatic. Our first choice for test automation tools are popular, free, open source tools.
  • Formal processes are in place for configuration managment, issue management, code reviews and periodic releases.
Table of Contents

Demonstrate Compliance with the OCAP Specification

Our first mission is to create a reference implementation that is compliant with the OCAP specification. If the spec says such-and-such is the case, then the RI had better do so, or otherwise note the feature is not implemented. In addition, operating software is a concrete testimonial to the quality of the spec itself. When we write code conforming to the spec we demonstrate that implementation can be done. Another benefit of spec conformance is some assurance that applications will behave the same independent of the underlying OCAP implementation which runs them.

CTP Tests

Suites of several thousand conformance tests are automatically run and manually analyzed each week by RI QA. Tests are run on the same hardware as is used to test conformance of commercial set top boxes. A sophisticated spreadsheet archives the results for easy analysis of the history of a particular test. Each week new tests are added, some tests are updated and occasionally a test is retired.
Each test sets up and automatically evaluates a single assertion about a single requirement from the OCAP spec. Logs from tests with unexpected results are manually analyzed and issues filed.
A sample of the CTP tests are also part of the RI smoke tests. Our policy is to run the smoke tests for each code change.

DLNA CTT 1.5 Tests

The RI QA team executes version 2.0.0.06 of the DLNA CTT tests against our implementation of Home Networking. 

UPnP CTT  Tests

The RI QA team executes version 2.0.45.7501 of the UPnP CTT tests.

LPTT Tests

The RI QA team executes version 2.0.10 of the LPTT tests.

Comcast TDK tests

The Comcast TDK tests are a collection of approximately four hundred unit level tests of OCAP functionality. An automation harness runs the tests, analyses the results and produces a final report. These tests are run under special licensing agreement with Comcast.

Oracle Java javax.tv tests

A TCK is a suite of tests that determines whether the OCAP RI complies with the Java technology specifications. TCKs are part of the Java Community process. Of special interest to the OCAP RI are the tests in the javax.tv suite. These tests are run under special licensing agreement with Oracle.

Findbugs static analyzer

Periodically the static analyzer Findbugs is run against RI code to look for bug patterns.

Explore and Demonstrate the RI as a Usable Product for the Cable Industry

Our second testing mission is to demonstrate the RI is a usable product. You can't specify everything. Much is implied about how the various components may be fitted together to create compelling cable applications. The RI integration tests go beyond the OCAP specification to explore how different modules work together.
Using integration tests inherited from the ODL we have collected a a large number of fairly sophisticated applications. Reading the code of the test applications is a great way to gain a black box understanding of the OCAP specification. The tests are clear, easy to understand java XLet applications.

Exerciser Applications

These applications allow you to interactively invoke OCAP APIs. One application is the framework for running many sequences of tests. Examples are:

  • TuneTest (manually, or randomly tune)
  • Dvr Test Runner (hundreds of focused DVR tests)
  • RiExerciser (record, invoke trick modes, and special Home Networking tests)

MSO Guide Tests

Tests with MSO guides simulate real-world market conditions. In one sense, these guides represent a set of acceptance tests from our customers. Throughout the OCAP project daily smoke tests have been run against the currently available guides.  Guides tested include the Comcast Aspen and Barcelona guides, the Cox Trio guide and the TWC ODN. The smoke tests are a subset of the tests used by the MSOs. All of the tests for a guide were run periodically, typically before a release of the RI.

Integration Tests

These are the tests that cover the major scenarios and major functionality of OCAP. Whereas the CTP tests are focused on atomic functionality, these tests demonstrate how the modules work together.
Some tests are automated using a framework name AutoXlet. Remote control key presses are specified in an .xml file. The tests use assert, similar to the ones in JUnit, to test for correct behavior.
Each test has:

  • Readme (describes how to run the test)
  • Hostapp.properties (needed to run all the applications used by the test)
  • AutoXlet files (If the test is automated)

Documentation on AutoXlet may be found at:

$OCAPROOT/apps/qa/Autoxlet

The code and associated files may be found at:

$OCAPROOT/apps/qa/org/cablelabs/xlet

Integration tests for Home Networking functionality

Manual tests of important scenarios in home networking may be found in spreadsheets located in:

$OCAPROOT/apps/qa/hn/HNIntegrationTests

The areas covered are:

  • Connection Stalling
  • Content Transformation
  • DIAGE
  • DTCP Initialization
  • NetAuthorization Handler 2
  • Platform Performance Tuning
  • Interlaced Input
  • LPE
  • VPOP Clarifications
  • Resource Contention Handler
  • Service Resolution Handler
  • Streaming and Recording
  • View Primary Output

These tests have been typically run on a weekly or per-release basis.

Rx - RI Exerciser and RiScriplet

Rx is comprised of the two test tools - RiExerciser and Ri Scriplet.  RiExerciser, developed as a replacement for DVR Exerciser, is used for testing DVR recording, playback and trickplay, HN remote recording, playback and trickplay, VPOP and Content Transformation.   Additional functionality includes tools for HN testing and diagnostics as well as tools for creating encrypted recordings.

RiScriptlet executes Bean shell scripts to run tests without any manual intervention.  These scripts tie long sequences of tasks into integration tests. These scenarios demonstrate popular home networking features. Internally RiExerciser and RiScriplet call a shared set of methods [see OcapAppDriverCore, OcapAppDriverDVR, OcapAppDriverHN, and OcapAppDriverHNDVR].

The xlets RiScriptlet and RiExerciser may be found in the usual directory for test application xlets:

$OCAPROOT/apps/qa/org/cablelabs/xlet

For more background please consult the Rx wiki, located here.

Tests for Supported Extensions

The RI my be built in numerous configurations of four extensions: Home Networking, DVR, Front Panel and Device Settings. See the WIki page, Extension-Based Build Configuration.

For each of the sixteen combinations of RI configurations the signature tests are run to demonstrate that all and only the appropriate public APIs are included in a certain configuration. Six configurations are supported (see the wiki page) and for those configurations additional CTP tests and Integration tests are executed.

Deliver High Quality Software in a Timely Manner

Our third mission is deliver a stable, high quality product in a timely manner. This is more of a quality assurance mission than a testing one. Much of what we do is better understood as strategies for quality assurance than as testing.
Our process is documented in several pages on this wiki.
The tools we use are:

  • SVN for configuration management
  • Cruise Control for continuous integration/build/test
  • JIRA for issue management
  • TestLink manages test cases and test results
  • Chainsaw analyzes logs
  • Emma for coverage analysis
  • FindBugs and PC-Lint for static analysis
  • No labels