Automating test labs has been a hot trend for service providers, network equipment manufacturers and services providers over the last several years, as these labs respond to demands for increased testing and faster cycle times, yet have fewer resources. Test lab automation has produced dramatic productivity gains, but benefits have been vendor specific and often times can't be applied to all of the test tools in a lab. One key industry objective is to have an interoperable solution.
With that in mind, the Network Test Automation Forum (NTAF), was formed in 2010 and has just this month released its first interoperability specifications that are the basic framework for the future of interoperability for lab test equipment. The Tool Registration, Discovery and Activation Specification describes a way in which tools can use a common language to register themselves so that other tools can discover and communicate with them without the need for time-consuming custom integrations. The Tool Automation Services Specification enables an application or tool to be controlled and/or monitored by another tool via the same common language.
With equipment incorporating the NTAF specifications, customers can now assemble complete test automation solutions using pluggable tool components, ensuring that the best tool is used for a specific job, be it commercial, proprietary or open-source. Since all tools are self-registering and self-describing, everything is built to be transparent and simple.
The goal of the NTAF is to facilitate and promote the interoperability of commercial testing tools and test infrastructure for the data communications and telecommunications industry. It brings together commercial testing vendors, test equipment vendors and other industry experts to create interoperable testing solutions for service providers, network equipment manufacturers and other enterprise organizations with large network deployments.
The NTAF technical committee met for the first time in April 2010. Members began by defining the initial NTAF abstract reference model, which was defined as having the following capabilities:
* Send/receive NTAF schema messages
* Support the standard NTAF request/response model
* Provide support for standardized error handling
* May emit asynchronous notifications (events)
XMPP Base Protocol
After exploring a range of alternatives, the technical committee decided to adopt the Extensible Messaging and Presence Protocol (XMPP) as the core protocol for communication among NTAF entities. Because XMPP uses an open systems approach of development and application, by which anyone may implement an XMPP service and interoperate with other organizations' implementations, it was found to be well matched to the NTAF requirements and goals for tool interoperability. An entity that declares an NTAF interface is known as a "provider," while the entity that consumes or uses this interface is known as a "requestor." Figure 1 illustrates the basic NTAF XMPP model.
XMPP is widely used today and mature in its feature content and testing. It defines a standard for entities to discover and communicate with one another via a network of servers. The communication is from client to server, server to server and server to client. It addresses all of the questions important in NTAF interoperability, including discovery, privacy, authentication, authorization, encryption and security. The communication itself is all done as a sequence of textual XML messages passed over a socket. Certain special messages are used for requesting information (such as presence).
Figure 2 provides a view of the different interfaces to the reference NTAF entity.
Network Test Automation Interoperability Specification
This first document developed by the committee provides an overview of NTAF, the reference model, common definitions and foundation requirements for interoperability among compliant tools. The goal of NTAF interoperability is to enable all the different elements in a network to communicate in a common language so that tools from multiple vendors can ask questions and get answers from one another according to NTAF specifications.
Specification TS-001: Tool Registration, Discovery and Activation
Traditionally, XMPP has been used for human-to-human exchanges. For users to establish communication with one another, they find out about each other through various mechanisms that may not involve XMPP. In the world of machine-to-machine or human-to-machine scenarios, this ad hoc approach to discovery is problematic. Provisioning each tool's roster with a list of all other relevant tools is time-consuming and error-prone. Instead, one wants "plug-and-play" among tools where each tool automatically discovers all other tools available in its environment and can take advantage of them without requiring manual provisioning.
When a tool is to be made available to others via NTAF, it is registered using XMPP+s publish-subscribe service (XEP-0060). For each tool or set of tools, a leaf node underneath a single root collection node is created. The publish-subscribe service JID and the collection node on that server are application-specific. For NTAF-compliant tools, the default address to use is the XMPP service address (e.g., "mycorp.com") prefixed by "pubsub." and the default collection node to use is "ntaf.tools". Tools may be provisioned to use a different pubsub address and/or node name.
A key part of the registration entry for a tool is the location element. This is useful for identifying where a registered tool is located, as well as for inventory and asset management purposes. It is also important for interoperability reasons in some cases.
One particular case is when humans are involved in the use of the tools. In that scenario, the requesting tool (a test authoring tool, for example) is going to initiate an interactive session on behalf of a user with another tool of a certain type. In this case, the choice of which instance of a given type of tool to use is not arbitrary. The user would like to be able to see the user interfaces for the two tools on the same display. For that reason, it is important for one tool to be able to determine whether it is co-resident with another tool. To be co-resident does not only mean that the other tool is installed on the same PC, but also that it is available within the same user's workspace so that it can share the display with the first tool.
Figure 3. XML Schema for Specification 1: Tool Registration, Discovery and Activation Specification
Request specification at: http://www.ntaforum.org/resources/request.html
Specification TS-002: Tool Automation Services
In the world of network test automation, it is common for a large set of both specialized and general-purpose tools to be assembled to create a complete solution. Traditionally, the implementation of such a solution usually involves a large amount of custom programming and is challenging because the tools involved are constantly evolving. Bilateral integrations between these various tools are prohibitively costly in terms of both time and labor. So the leaders in this market are seeking a better approach.
The Tool Automation Services Specification enables an application or tool, with or without its own specialized man-machine user interface, to expose an "automation harness" that allows that tool to be controlled and/or monitored by another tool via XMPP packet exchanges.
At the heart of this specification is the notion of a "harness." This is a generic mechanism over XMPP whereby tools can advertise certain domain-specific command sets, providing all of the details about how these commands can be issued within the context of a session. The harness is fully self-describing, with an understanding that this is not just advertising an API that could be used by programmers, but may also be used to provide a user interface and therefore needs information in the description suitable for humans to use in participating in decisions about how commands will be invoked.
A provider of a given harness is allowed to declare what modes it supports for sessions using that harness. These session modes are the most important element of the specification. The provider of a harness is required to support at least one mode for its sessions, but it is free to support as many modes as it is able. Not all providers of the same harness may support the same modes -- and that is why the supported modes are not specified in the harness declaration itself, but, rather, in the list-harnesses response.
Invisible and Automated Mode
In this mode, the session provider is essentially just providing a traditional application programming interface (API) for the session. There is no human involved and no user interface on the provider side. The client requests a "session" and then invokes a set of requests for actions on the session. Essentially this is like any traditional mechanism for one computer process to talk to another computer process -- making requests to perform actions. In this mode, it works very similarly to, say, web services, or RPC mechanisms such as DCOM or RMI.
Visible and Interactive Mode
In this mode, there is typically a human involved on the provider side. One tool asks another tool to open a session and (depending on the declaration of the "open" action in the harness declaration) may describe certain properties about the behavior of that session. But unlike mode 1 above, the intention of this session is for a human user to interact with it from that point. And, very importantly, in this mode, the provider agrees to send notifications to the originator for each action performed interactively (by the human user).
Visible and Automated Mode
This mode requires the provider to present a user interface suitable for humans to view during the lifetime of that session, and yet the user interface is typically read-only -- because the actions performed on that session will still be driven by the originator as a sequence of actions requested using the harness protocol. As these actions are performed, the user interface presents the appropriate information to the human user -- much as if that user had performed those same (or equivalent) actions.
NTAF implementers are strongly encouraged to support as many modes in their tools as possible. Note that another possibility is to create companion tools where one tool is primarily designed for interactive use (and only exposes mode 2), while its companion is optimized for automation (and supports only mode 1).
Basic Harness Declarations
Regardless of the session mode, the notion is that a session is made up of a series of actions being performed, each of which produces a response. In addition, certain events may occur during the lifetime of the session. Each of these, a request, a response, and an event, are communicated between requester and provider using XML elements conforming to the schemas defined by this extension. But more than that, the contents of these elements are further constrained by the contract for these as described in the response to a query-harness inquiry.
In simple cases, a request for an action may include zero, one, or many parameters. These parameters are declared in the harness declaration for that action so that the requester knows what parameters are allowed (and/or required) and the meaning of each of these.
In an actual request, each parameter is passed using its name and a value. The declaration of the parameter helps to clarify its purpose and how it may be used.
Each parameter must have a unique name within its container. In addition, the parameter declaration must also provide a "label" and "tooltip" that may help a human to understand the purpose of that parameter. That is all that is mandatory when declaring a parameter for an action. But the declaration may also provide many other elements to further document and/or constrain the way in which that parameter is used in an actual request.
Just as a simple request may take one or several parameters, each response may return zero, one, or several "items" of information in return. The response can also carry more richly structured data, but for the simple cases, you can think of a response as consisting of a set of name-value pairs. But, just like for the request, there is a question as to how the requester should interpret this information returned for a given requested action. Hence, the harness declaration includes a "response" declaration for each action. If the action will return nothing other than a result (pass-fail) for the action, then the response declaration can be omitted. But if it is going to return any information, then it must also document what information that might be. One can think of an "item" as being the response's counterpart to a request's "parameter."
At any time during any type of session, the provider may choose to notify the requester of notable events as long as they are declared in the harness declaration returned by the query harness. A simple event can just identify the harness and the name of the event. A more complicated event may carry additional information. If an event carries additional information, this must be declared as part of the harness. An event declaration follows exactly the same model as a response. So any structure of information that can be expressed in a response, can equally be expressed in an event.
Implementers of all NTAF-enabled tools should consider reliability carefully. XMPP is carried over TCP and therefore information flows are reliable. However, a problem can occur if the provider or requester involved in a harness session goes offline while the session is under way, or worse, when a long-running request is being processed by the provider. Since the TCP sockets are between the XMPP clients and their server (rather than end-to-end between clients) there is no guarantee that a packet sent by a requester to a provider will, in fact, arrive at that provider, or vice versa.
The solution to this problem comes from using XMPP presence information. Before a requester opens a session to a provider, it should subscribe to presence information for that provider. And before the provider accepts the requested session, it should subscribe to presence information for that requester. If either end learns that the other has gone offline, then it can close its session accordingly.
It may be common to encounter hundreds or more of the same kind of tool in an environment. These identical tools will respond identically to queries made to them. If a client requests harness information about the same harness from every new tool that it encounters, then it is possible that the network will be burdened with a lot of useless traffic. Moreover, the time to get this response and to process it is likely to reduce the performance of these client tools.
Therefore, it is recommended that client tools maintain a registry or cache of harness information where the key is based on the name of each item. In that way, when a tool discovers a new provider, it need only fetch harness information for those that it does not already recognize.
Security, Authorization, and Authentication
This specification does not address the question of how a provider should determine authorization for certain requests that are made using this protocol. It is up to the implementer of each harness to decide how and when to authorize various actions. Typically, XMPP itself provides the means to identify the requester (using its JID) and a provider can assume that the XMPP service has authenticated that entity.
Request specification at: http://www.ntaforum.org/resources/request.html
This article has provided an overview of the first technical specifications developed by the Network Test Automation Forum to automate multi-vendor network test automation in the lab. To summarize, the Tool Registration, Discovery and Activation Specification enables tools to use a common language to register themselves so that other tools can discover and communicate with them without the need for time-consuming custom integrations, and the Tool Automation Services Specification enables an application or tool to be controlled and/or monitored by another tool via the same common language. The detailed reports on both these specifications and schema for use in developing NTAF-compliant tools are available on the NTAF website at: http://www.ntaforum.org/resources/request.html.
Membership in NTAF is rapidly expanding and includes a mix of companies from across the communications and networking industries, including BreakingPoint Systems, Brocade, BT, Cisco, Empirix, Ericsson, EXFO, Ixia, JDSU, Linkbit, MRV, Polystar, Shenick, Spirent Communications, Tata Communications, Tech Mahindra and Verizon. Users, vendors, and other telecommunications and data communications industry experts interested in joining the effort to create interoperable testing solutions should consider joining NTAF. Benefits of membership include the ability to influence specification creation, access to draft versions of specifications, the opportunity to be the first to know new network test automation market directions, services and technology trends, the ability to participate in standardization of network test automation, putting your stamp on this industry - and the opportunity to network with your partners, suppliers, customers and competitors in an informal, non-competitive environment.
More information is available at http://www.ntaforum.org.
About the Author
Ameya BarvÄ is chairman of the marketing committee for the Network Test Automation Forum and product marketing manager at Spirent Communications, where he is responsible for establishing and executing on Spirent's global marketing strategy for its portfolio of automation products and solutions. He has over 15 years of experience in the software and technology industries, holding various titles in the fields of software engineering, product management, product marketing and business development.
Ameya has an MBA from the University of Wollongong in Australia, a master's degree in computer information systems from Southern New Hampshire University and a bachelor's degree in management from the University of Bombay (now Mumbai).