[ICN-655] Multi tenant capable SDEWAN Hub in K8s clusters with single public IP address Created: 19/Jul/22  Updated: 25/Jul/22

Status: To Do
Project: Integrated Cloud Native NFV
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Epic Priority: Medium
Reporter: Srinivasa Addepalli Assignee: Huifeng Le
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Epic Name: Multi-tenant capable SDEWAN Hub in K8s cluster

 Description   

Current solution for multiple customers:

Say there are X number of customers

Say there are Y number of Hub locations

To address Multi tenancy,  SDEWAN Hub expects X * Y number of public IP addresses.

Each site of a customer (uCPE) would use nearest Hub for making the tunnel.

Problem statement:

  •  Public IP addresses are expensive and not available.

Enhancement request:

  • Ensure that only Y number of public IPs are used.  That is number of IP addresses needed are based on number of PoP locations.
  • All customers' sites that are near one hub will use the same public IP address as Hub Gateway.

In my view, following is needed:

  • SDEWAN VPN Concentrator POD shall use host network in the K8s (root network namespace) or its own network namespace like any other POD. But there is no tenant specific SDEWAN POD. It is global across all tenants.  Of course, scale-out of VPN Concentrator for load sharing though. 
  • Whenever VTI interface is created, based on the ID of the client, it shall assign that VTI interface to packet processing POD of the tenant.  
  • Packet processing POD is the one that does rest of the packet processing such as firewall/NAT/IPSEC to other Hubs. 

It means that there shall be a controller which listens on new tunnel establishments   (IPSEC tunnel is established by BO router or or by the VPN client in PCs).  It needs to assign the new VTI interface created to tenant specific packet processing POD's namespace.

Do let me know if that can be made possible?

 Comments   
Comment by Srinivasa Addepalli [ 25/Jul/22 ]

Hi hle2,

As I come to think about this more, I am wondering we could have a challenge if the SDEWAN VPN Concentrator is running in one physical node in the K8s cluster and the packet processing POD running on some other physical node. In that case, it is not possible to shift the VTI tunnel from one network namespace to another.

Now, I am wondering whether the following solution would be good.

  • VPN Concentrator PODs and packet processing PODs are connected with, say some tunnel such as GRE tunnel.
  • VPN Concentrator for all the traffic for a given tenant sends the traffic GRE tunnels associated with the tenants. That is, if there are say 2 packet processing PODs for a given tenant (due to scale-out), then there would be 2 GRE tunnels. VPN Concentrator will choose a GRE tunnel for a 5 tuple session (kind of load balancing).

This requires setting up IPTables rules and IP route rules with corresponding routing database to pass the packets among the VPN tunnels and GRE tunnels.

What do you think?

If that seems okay, I would imagine a controller within K8s clusters to do following

  • Knows about all packet processing PODs and SDEWAN PODs.
  • For every packet processing POD that comes up or goes down:
    • Knows about tenant (from the namespace).
    • Knows about all SDEWAN PODs that are running (Due to scale-out, there could be multiple instances)
    • Create GRE tunnels on both sides.
    • Setup route policies and route tables such as way that anything that is coming out of GRE tunnel go to VPN tunnel and vice versa.
    • Setup IPTables rules such a way that all the traffic for a given 5-tuple session go to the same tunnel.
    • Every time new SDEWAN POD comes up (due to scale-out)
    • Repeat the above process.
Comment by Srinivasa Addepalli [ 25/Jul/22 ]

Different tenants will certainly will need to have different tunnels.

In case of one tenant, there could be multiple Branch offices and multiple WFH users connecting to the same Hub. In that case, there could be multiple tunnels - At least one from each branch office and at least one from each WFH user.   In summary,  there are multiple tunnels.  You can assume that a given tunnel is not shared by multiple tenants. 

 

Identification of tenant :  It shall be based on Initiator ID (https://datatracker.ietf.org/doc/html/rfc4306#section-3.5).  Since, Client IP can't be used as Client IP gateway can be dynamic IP address.  Tenant determination from tunnel establishment, in my mind can be

  • Based on email address (getting domain name from the email address)
  • Domain name itself
Comment by Huifeng Le [ 25/Jul/22 ]

saddepalli Some opens:
1. Different tenants will share same tunnel or different tunnels (with different client ip/oip + same hub ip), as VTI interface (also the related iptable rules) is created in strongswan's updown script, maybe not able to create multiple VTI interface in same tunnel.
2. How to identify client id? based on different client ip?

Generated at Sat Feb 10 06:01:44 UTC 2024 using Jira 9.4.5#940005-sha1:e3094934eac4fd8653cf39da58f39364fb9cc7c1.