Editor's Note: To view a PDF version of this article, click here
Provider-provisioned virtual private networks (PP-VPNs) are a relatively new technology that combines elements of network and customer premises equipment (CPE) VPNs to provide a secure networking solution operating at the Internet Protocol (IP) layer and above. By doing so, PP-VPNs offload complexity from the enterprise, which is no longer concerned with underlying Layer 2 technologies, and move management from the CPE to the service-provider network. The results are a simplified VPN for the enterprise, along with cost savings and much-needed sources of new revenue streams for service providers and carriers.
Based on an Internet Engineering Task Force (IETF) draft known as rfc2547bis (Figure 1), PP-VPNs will not displace existing service overnight, but can coexist with it and offer a smooth migration path as new access technologies, such as Ethernet, emerge. However, the industry must carefully consider the challenges of scalability, security and management, along with the integration of complex technologies, if it is to deliver cost-effective solutions for service providers to implement.
Learn the PP-VPN lingo
Before launching into a discussion of PP-VPNs it's necessary to review some specific terminology inherent in their operation (Figure 2).
A customer edge (CE) device connects users at customer sites to a provider edge (PE) device. CEs may be routers or switches that let users communicate with other VPN sites. A PE is a router or switch that interacts with a CE and provider backbone devices (P devices). PEs also operate as label edge routers (LERs) in a multiprotocol label switching (MPLS) network. The P device operates within a provider network that interconnects PE devices, but with no direct attachment to CEs. P devices are also MPLS label switch routers. Finally, a service-provider (SP) network is defined as a network of interconnected PE and P devices administered by a single provider or operator.
VPN solutions must address the challenges of provisioning, scalability, performance, security, availability, interoperability and management. To be viable, PP-VPNs must have no CPE impact and minimal network upgrades. This places the burden of the solution on the PE.
The PP-VPN challenges
PP-VPNs use IP technology for all elements of the solution. This introduces a number of system design challenges. As with any VPN solution, scalability is crucial. Traditional network- and CPE-based solutions achieve this by requiring the enterprise to provide substantial connectivity and routing support. For CPE tunneling solutions, enterprise tunnel termination is key. Network-based solutions combine network-provided connections, such as frame relay/asynchronous transfer mode (ATM) virtual circuits with an enterprise routing overlay.
A PP-VPN solution, and thus a PE device, must be scalable, supporting tens of thousands of VPNs-with up to 50,000 or more interfaces per VPN. It must also scale to support a wide range and number of routes per VPN. All these must be rapidly and easily configurable.
With both the customer IP VPN and the service-provider network using IP addressing, some issues arise. Generally, customer site addresses are configured using private IP addresses. Many enterprises use that same private IP address space. This means Layer 3 PP-VPN solutions must support overlapping customer IP address spaces, isolating them from one another and from the service provider's IP-addressing scheme. IP addresses must be unique within the set of sites reachable from the VPNs of which a particular site is a member.
For security, PP-VPN solutions should provide protection through any means other than defined policies. This type of security provides equivalent protection to traditional network-based VPNs, but a PP-VPN solution does not preclude additional encryption using, for example, Internet Protocol Secure (IPsec).
Rfc2547bis PP-VPNs require extensions to the BGP4 protocol along with MPLS. These are used to create two-level label stacks for transferring data between VPN locations. The implementation of these features and the architectural decisions greatly determine how PE devices meet the above challenges.
PE device requirements
Traditional service-provider routers are part of a single routing domain, though they may have complex interactions with other routing domains. The main functional blocks involve Interior Gateway Protocols (IGPs), such as open shortest path first (OSPF) and intermediate system to intermediate system (ISIS); Exterior Gateway Protocols, such as BGP4; the forwarding information base (FIB); and associated IP forwarding hardware. Also, complex policy configurations dictate how one routing domain learns information about another. Increasingly, MPLS support is a requirement.
While PE routers are members of the service-provider network, they are also members of the customer network. They must maintain routing information for each VPN and keep this segregated from other VPNs and from the service provider's network. They also must be able to create connections among the members of a VPN. The functions outlined by rfc2547bis to accomplish this are:
- Configuration of VPNs.
- Route distribution from and to connected CEs.
- Route distribution to other PEs in the service-provider backbone.
- Label switch path (LSP) establishment between PE routers.
- Data transfer among customer sites in a VPN.
Using the premise that PP-VPNs must not affect core service-provider routers and must be transparent to CE routers, multiprotocol enhancements to BGP4 are required to distribute VPN routes between PEs. MPLS is also required to create tunnels between PEs, using as a minimum Label Distribution Protocol (LDP) in downstream unsolicited (DU) mode. This basic mechanism provides a best-effort capability that can be supplemented with constrained routing using LDP (CR-LDP) or the Resource Reservation Protocol with traffic-engineering extensions (RSVP-TE) to create traffic-engineered tunnels supporting quality of service (QoS).
PE functional blocks
Both the general and specific functional requirements for devices supporting PP-VPNs lead to a number of ideal architectural requirements. Figure 3 shows one implementation that has been drawn to logically map to the LER PE2 in Fig. 2.
Click here for Figure 3
Here, the FIB is the distillation of best routes from all the routing protocols operating within a device. Since the PE is a traditional router in the sense that it forms part of the service-provider network, it has an FIB component. However, interfaces on the PE are also part of the customer networks and must contain information on how to communicate with other members of the VPN. This is accomplished through separate routing tables known as VPN routing and forwarding tables (VRFs). The FIB maintains all the VRFs for the VPNs.
The VPN configuration server facilitates configuration of VRFs, interface association with VRFs and the policies associated with VRFs.
A two-level label stack is used to forward VPN information between two CE devices of a VPN. This involves coordinating the label information in the FIB, carried via BGP4, which relates to each specific VPN and the tunnel-level label used between PE routers. This is handled by a VPN tunnel manager (VTM), which provides label information to the FIB.
To provide scalability and some level of security, multiple routing instances must be supported, since not only the service-provider interfaces but also the PE-CE connections could be running OSPF, RIP and/or BGP4.
In conjunction with the defined protocol features, each of these blocks performs specific functionality, providing a complete system implementation.
VPN configuration is critical for basic connectivity, but also to enable service providers to create the appropriate customer network operation. Since each CE device has only one connection to the service-provider network (PE), VPN connectivity becomes the responsibility of the PE configuration and operation.
For example, CEs may be connected via a full mesh. Alternatively, a star configuration could be used, in which remote-site traffic always moves through a central location. With traditional network VPNs this is handled via private line or frame relay private-virtual-circuit topology selection. PP-VPNs accomplish this by using the multiprotocol extensions and extended-communities features of BGP4.
The VRF is the repository of routing updates received from customer sites. They can cater to multiple VPNs based on policy configuration. Information propagation from VRFs to various PE components, the FIB, BGP and other routing protocolsboth local and remotedepends on how the VPN server is configured. Several fields are used to control the information flow, the route distinguisher (RD), the import/export route targets and the VPN label information.
The RD is a unique 8-byte value associated with each VRF. This value is appended to routes before they are added into the VRF, ensuring that the same IPv4 address used in two different VPNs generates two distinct routes.
VPN route distribution
Each VPN is identified by one or more route targets. When a route is received from a connected CE, it is advertised to other PE routers, and configured export route targets are appended. When routes are received from other PEs, configured import route targets are compared with the received route targets. If any match, the route is installed into the VRF.
For example, if the set of import and export route filters are the same at each VRF of a VPN, then a full mesh of routes is created. Star configurations are created by enabling the central site to import routes from all the remote locations, with the remote locations configured to import the single route from the central location only. This prevents remote locations from directly talking to one another.
In addition to routing information, VRFs store label information, which PEs use when advertising routes to other PEs. These VPN-level labels identify the VPNs within a PE, but are not used within the core of the network (Fig. 2 again).
There are two types of VPN labels, depending on configuration and use of BGP. In all cases, labels are managed from a central location-the VTM. In the simplest case, a label is advertised for an interface associated with VRF. The FIB requests a label for the interface from the VTM. This label is used by the VTM to program the MPLS forwarder for transferring data on the interface if a data packet is received with that label. The FIB also propagates the interface-to-label mapping to BGP. When a CE connected on this interface advertises routes, the PE associates the routes with the interface label before advertising these routes to connected CEs.
More complex situations occur when BGP aggregates routes from a number of VRFs into a single advertisement. A simple example would be several CEs that are subnets advertising into VRFs in a PE. Rather than advertise a label per subnet, BGP can summarize this information as an aggregate label to connected PEs. However, this creates more work in the data plane since the VPN label no longer uniquely identifies the VRF, and additional lookups are required to determine the destination.
The PE and CE can establish routing sessions through OSPF, Exterior BGP, RIP or by statically injecting routes into the associated VRF. If the PE is an OSPF peer of CEs that are in distinct VPNs, the PE must run multiple instances of OSPF. If the PE-CE routing session is OSPF, RIP or static, routes received from the VPN sites are redistributed into BGP. The FIB stores the advertised VPN site routes in the corresponding VRFs and the BGP converts CE routes into VPN IPv4 routes by attaching RDs and associated label information, and storing the routes in the BGP local routing-information base (RIB) for the VPN address family.
All the PE routers in the SP network are connected via full-mesh I-BGP or route reflectors. To exchange VPN routing information, MP-BGP is enhanced to support an additional address family (address family identifier [AFI] = 1; subsequent AFI [SAFI] = 128).
BGP runs a decision process on a per-VRF basis and then passes the best routes through export policies configured for the VRF. It then advertises the route to its PE peers. Note that if the route's export policies match the import policies of another VRF on the same PE router, the route is advertised to the CE router associated with that local VRF.
When the remote PE receives the route, it passes through the import policies for each VRF. If the route matches the import policies, it is injected into the VPN local RIB, after replacing the RD with the one associated with that VRF.
BGP runs the decision process on that VRF, and if the route is found to be a best route, it is advertised to connected CEs as an IPv4 route. BGP also propagates the route to the FIB so the VRF forwarding tables are updated. If the CE session associated with that VRF is OSPF, routes are redistributed to OSPF, which advertises the routes to its neighbors. The key part of this mechanism is that BGP distributes routes downstream-that is, from the egress PE to the ingress PE, in the opposite direction of the data flow.
The FIB queries the VTM for LSP and interface information to the BGP next hop that's specified in the route received from the remote PE. It then updates the VRF table. In distributed operation, VRF information is copied from the controller card to the line card. When a data packet arrives from the CE, a VRF lookup occurs directly on the line card and, after applying the label stack, it is given to the MPLS forwarder.
Rfc2547bis uses MPLS tunnels between PE routers to provide scalability for the service-provider network. The specification minimally requires LDP DU to create best-effort tunnels. In practice, RSVP-TE is frequently used to establish QoS-based tunnels to meet service-level agreement requirements.
However, LDP DU and RSVP-TE operate differently, complicating the overall design. DU is an egress-initiated label-mapping mechanism. The PE routers distribute labels to their downstream peers, eventually reaching the ingress PE router. The label negotiated at the ingress is supplied to the VTM, which must form a mapping between the MPLS tunnel and VRFs that use the tunnel.
The MPLS tunnel is identified by the forwarding equivalency class, which is the router identification, or IP address, of the egress PE. This is also generally the loopback address.
The issue is in associating VRFs with MPLS tunnels. The key is to ensure that when the BGP advertises VPN information, the next-hop address becomes the router ID of the destination PE. Since the egress router ID is available from the BGP advertisement and from the forwarding equivalency class distributed via LDP DU, the VTM can make the association between VRFs and the MPLS tunnel. Once this link is made, the MPLS tunnel and BGP VPN labels are stored in the VRF; now the MPLS forwarder can be programmed with the label stack.
Similar programming occurs at the egress point to direct traffic to the CE interface. Data plane operations depend on whether penultimate hop popping (PHP) is used (as shown in Fig. 2), which saves additional lookups at the egress PE.
RSVP-TE is ingress-initiated since it has the capability to establish QoS. Additionally, a tunnel can be established per VPN connection, if required. The destination egress PE is determined from a route table lookup. A constrained shortest-path-first engine calculation can then generate a traffic-engineered path. The same BGP mechanism that serves to associate the VPN and the MPLS tunnel is used to create the label stack programming in the MPLS forwarder.
In rfc2547bis-based VPNs, data traffic is forwarded using the LSPs established between a PE router that learns routes and a PE router that advertises routes. VPN data traffic is forwarded through pre-established LSPs in the SP backbone. As shown in Fig. 2, the data transfer between CE2 and CE1 is as follows:
- CE2 forwards IPv4 packets addressed to CE1 to PE2.
- PE2 looks up the packet's destination in the green VRF and applies the VPN and MPLS label stack.
- PE2 forwards the MPLS packet to P1.
- P2 swaps the outer MPLS label and forwards the packet to P1.
- P1 uses PHP to pop the MPLS label, leaving the VPN label on top, and forwards the packet to PE1.
- PE1 pops out the VPN label and forwards the packet to CE1.
In cases with an aggregate VPN label, PE1 consults the green VRF and determines CE1's interface, on which it forwards the packet. This mechanism ensures network scalability since the P routers in the core of the network remain unaware that they are transporting traffic from multiple VPNs.
Scalability and security
The scalability in the rfc2547 framework results from no single box needing to know about all the VPNs in the SP network. Knowledge of a particular VPN is confined to the PE routers that attach to sites in that VPN, and to the BGP route reflectors, which receive routing information from these PEs.
The IGP running in the service-provider backbone must carry a host route for every LSP egress node (PE) within the routing domain. Thus the number of routes that can be carried by IGPs limits the number of PEs.
Another constraint is the number of routing instances that the PE can support, assuming, for example, that the PE-CE routing protocol is OSPF.
Security in rfc2547bis-based VPNs is divided between the control and data planes. To achieve control plane security, neither BGP nor LDP sessions should be established with untrusted peers. Transmission Control Protocol MD5 authentication should be used with both the protocols. IGPs running with SP backbone should also be secured.
Data plane security provided by this architecture is virtually identical to that provided by frame relay and ATM VPNs. If the device under service-provider control is properly configured, data will neither enter nor leave the VPN unless it is authorized. This is achieved by checking the packet's top label. If it has a label value that the receiving system has distributed to that neighbor, or if the packet's top label has a label value which the receiving system has distributed to a system beyond the neighbor, the packet is acceptable.
Rfc2547bis implementations are highly complex, involving interaction among routing protocols, their extensions and MPLS technologies. The scalability aspects almost dictate a distributed-processing approach and the ability to create multiple instances of routing protocols, from a control plane perspective. Ultimately, the disparities in the information must be resolved by programming one set of hardware, which must be scalable and flexible. These factors create most of the design complexity and must be considered at inception.
Although the focus has been on creating connections between CE members of a VPN, CEs may also be sending traffic to destinations outside the VPN. Therefore, the solution must enable packets arriving from a CE device to be forwarded using either MPLS, a traditional IP routing scheme, or to have no route.
From a broader perspective, the mechanisms described create unidirectional paths. A complete solution must enable bidirectional paths, adding to the network management load. If a service-provider network is growing rapidly, this in itself requires scalability and security.
Other network factors require consideration-for example, rerouting around network failures and the asynchronous interaction of the service-provider IGP, MPLS and BGP VPN advertisements.
Finally, service providers are demanding "five nines" reliability in their IP networks. High availability and fast-recovery options for complex implementations such as PP-VPNs must be considered early in the design cycle. This is further complicated by the current standards for BGP and LDP DU, which do not effectively support this capability-though restart standards are emerging.
The IETF is continuing to refine service requirements for PP-VPNs at both Layer 3 and Layer 2. Rfc2547bis and related standards discuss the use and implementation of multicarrier PP-VPNs and the concept of the carrier's carrier. Although conceptually most of the standards exist, implementation is still focused on enterprise applications within a single carrier.
There is a lot of interest in extending the rfc2547bis concepts to Layer 2-based VPN technologies such as virtual private LAN segment (VPLS) and optical VPNs.
A VPLS service allows the connection of multiple sites in a single bridged domain over a provider-managed IP or MPLS networks. All customer sites in the VPLS appear to be in the same LAN regardless of their location.
The basic unit of an optical VPN service is an optical or time-division multiplexed connection between a pair of CEs. Conceptually, this may allow customers to support their own on-demand bandwidth within the VPN.
- "Cut to the Core of Optimal MPLS Router Design," www.commsdesign.com/story/OEG20020701S0021.
- "Engineering Traffic in MPLS Networks," www.commsdesign.com/story/OEG20011121S0066.
- "BGP/MPLS VPNs," Eric C. Rosen (draft-ietf-ppvpn-rfc2547bis-01.txt).
- "A framework for Layer 3 provider-provisioned virtual private networks" (draft-ietf-ppvpn- framework-05.txt).
- "Applicability statement for VPNs based on RFC2547bis" (draft-ietf-ppvpn-as2547-00.txt).
About the Authors
E.S.N. Murthy (firstname.lastname@example.org) is a project manager at NetPlane Systems. He holds a master's degree in digital systems from Osmania University in India.
Bill Herbst (email@example.com) is a principal software engineer at NetPlane Systems. He holds a bachelor's degree in chemistry from Clark University (Worcester, Mass.).