diff --git a/README.rst b/README.rst index 9551c41b..9172335c 100644 --- a/README.rst +++ b/README.rst @@ -12,4 +12,6 @@ The OVN BGP Agent allows to expose VMs/Containers through BGP on OVN Features -------- -* TODO +* Expose VMs with FIPs or on Provider Networks through BGP on OVN + environments. + diff --git a/doc/images/evpn_traffic_flow.png b/doc/images/evpn_traffic_flow.png new file mode 100644 index 00000000..0dc83afb Binary files /dev/null and b/doc/images/evpn_traffic_flow.png differ diff --git a/doc/images/networking-bgpvpn_integration.png b/doc/images/networking-bgpvpn_integration.png new file mode 100644 index 00000000..fdd21f96 Binary files /dev/null and b/doc/images/networking-bgpvpn_integration.png differ diff --git a/doc/source/contributor/bgp_mode_design.rst b/doc/source/contributor/bgp_mode_design.rst new file mode 100644 index 00000000..99e98bfd --- /dev/null +++ b/doc/source/contributor/bgp_mode_design.rst @@ -0,0 +1,43 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + + Convention for heading levels in Neutron devref: + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + (Avoid deeper levels because they do not render well.) + +=================================== +Design of OVN Agent with BGP Driver +=================================== + +Purpose +------- + +Overview +-------- + +With the increment of virtualized/containerized workloads it is becoming more +and more common to use pure layer-3 Spine and Leaf network deployments at +datacenters. There are several benefits of this, such as reduced complexity at +scale, reduced failures domains, limiting broadcast traffic, among others. + +Proposed Solution +----------------- + +OVN SB DB Events +~~~~~~~~~~~~~~~~ + +Driver Logic +~~~~~~~~~~~~ + +Traffic flow +~~~~~~~~~~~~ + +Agent deployment +~~~~~~~~~~~~~~~~ \ No newline at end of file diff --git a/doc/source/contributor/evpn_mode_design.rst b/doc/source/contributor/evpn_mode_design.rst new file mode 100644 index 00000000..72d000e6 --- /dev/null +++ b/doc/source/contributor/evpn_mode_design.rst @@ -0,0 +1,468 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + + Convention for heading levels in Neutron devref: + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + (Avoid deeper levels because they do not render well.) + +==================================== +Design of OVN Agent with EVPN Driver +==================================== + +Purpose +------- + +The purpose of this document is to present the design decision behind +the EVPN Driver for the Networking BGP OVN agent. + +The main purpose of adding support for EVPN is to be able to provide +multitenancy aspects by using BGP in conjunction with EVPN/VXLAN. It allows +tenants to have connectivity between VMs running in different clouds, +with overlapping subnet CIDRs among tenants. + + +Overview +-------- + +The networking bgp ovn agent is a Python based daemon that runs on each node +(e.g., OpenStack controllers and/or compute nodes). It connects to the OVN +Southbound DataBase (OVN SB DB) to detect the specific events it needs to +react to, and then leverages FRR to expose the routes towards the VMs, and +kernel networking capabilities to redirect the traffic once on the nodes to +the OVN overlay. + +This simple design allows the agent to implement different drivers, depending +on what OVN SB DB events are being watched (watchers examples at +``networking_bgp_onn/drivers/openstack/watchers/``), and what actions are +triggered in reaction to them (drivers examples at +``networking_bgp_ovn/drivers/openstack/XXXX_driver.py``, implementing the +``networking_bgp_von/drivers/driver_api.py``). + +A new driver implements the support for EVPN capabilities with multitenancy +(overlapping CIDRs), by leveraging VRFs and EVPN Type-5 Routes. The API used +is the ``networking_bgpvpn`` upstream project, and a new watcher is created to +react to the information being added by it into the OVN SB DB (using the +``external-ids`` field). + + +Proposed Solution +----------------- + +To support EVPN the functionality of the ``networking-bgp-ovn`` agent needs +to be extended with a new driver that performs the extra steps +required for the EVPN configuration and steering the traffic to/from the node +from/to the OVN overlay. The only configuration needed is to enable the +specific driver on the ``bgp-agent.conf`` file. + +This new driver will also require a new watcher to react to the EVPN-related +events. In this case, the EVPN events will be triggered by the addition of +EVPN/VNI information into the relevant OVN ``logical_switch_ports`` at the +OVN SB DB. + +This information is added into OVN DBs by the ``networking-bgpvpn`` projects. +The admin and user API to leverage the EVPN functionality is provided by +extending the ``networking-bgpvpn`` upstream project with a new service plugin +for ML2/OVN. This plugin will annotate the needed information regarding VNI +ids into the OVN DBs by using the ``external-ids`` field. + + +BGPVPN API +~~~~~~~~~~ + +To allow users to expose their tenant networks through EVPN, without worring +about overlapping CIDRs from other tenants, the ``networking-bgpvpn`` +upstream project is leveraged as the API. It fits nicely as it has: + +- An Admin API to define the BGPVPN properties, such as the VNI or the BGP AS + to be used, and to associate it to a given tenant. + +- A Tenant API to allow users to associate the BGPVPN to a router or to a + network. + +This provides an API that allows users to expose their tenant networks, and +admins to provide the needed EVPN/VNI information. Then, we need to enhance +``networking-bgpvpn`` with ML2/OVN support so that the provided information +is stored on the OVN SB DB and consumed by the new driver (when the +watcher detects it). + +The overall arquitecture and integration between the ``networking-bgpvpn`` +and the ``networking-bgp-ovn`` agent are shown in the next figure: + +.. image:: ../../images/networking-bgpvpn_integration.png + :alt: integration components + :align: center + :width: 100% + +There are 3 main components: + +- ``BGPVPN API``: This is the component that enables the association of RT/VNIs + to tenant network/routers. It creates a couple of extra DBs on Neutron to + keep the information. This is the component we leverage, restricting some + of the APIs. + +- ``OVN Service Plugin Driver``: (for ml2/ovs, the equivalent is the bagpipe + driver) This is the component in charge of triggering the extra actions to + notify the backend driver about the changes needed (RPCs for the ml2/ovs + bagpipe driver). In our case it is a simple driver that just integrates with + OVN (OVN NB DB) to ensure the information gets propagated to the + corresponding OVN resource in the OVN Southbound database — by adding the + information into the external_ids field. The Neutron ML2/OVN driver already + copies the external_ids information of the ports from the + ``Logical_Switch_Port`` table at the OVN NB DB into the ``Port_Binding`` + table at the OVN SB DB. Thus the new OVN service plugin driver only needs + to annotate the relevant ports at the ``Logical_Switch_Port`` table with + the required EVPN information (BGP AS number and VNI number) on the + ``external_ids`` field. Then, it gets automatically translated into the + OVN SB DB at the ``Port_Binding`` table, ``external_ids`` field, and + the OVN BGP Agent can react to it. + +- ``Backend driver``, i.e., the networking-bgp-ovn with the EVPN driver: + (for ml2/ovs, the equivalent is the bagpipe-bgp project) + This is the backend driver running on the nodes, in charge of configuring + the networking layer based on the needs. In this case, the agent continues + to consume information from the OVN SB DB (reading the extra information + at external_ids, instead of relying on RPC as in the bagpipe-bgp case), and + adds the needed kernel routing and FRR configuration, as well as OVS flows + to steer the traffic to/from OVN overlay. + + +As regards to the API actions implemented, the user can: + +- Associate the BGPVPN to a network: + The OVN service plugin driver annotates the information into the + ``external_ids`` field of the ``Logical_Switch_Port`` associated to the + network router interface port (ovn patch port). Additionally, the router + where the network is connected also gets the ``Logical_Switch_Port`` + associated to the router gateway port annotated (ovn patch port). + +- Associate the BGPVPN to a router: + The OVN service plugin driver performs the same actions as before, but + annotating all the router interface ports connected to the router (i.e., + all the subnets attached to the router). + + +OVN SB DB Events +~~~~~~~~~~~~~~~~ + +The networking-bgp-ovn watcher that the EVPN driver uses need to detect the +relevant events on the OVN SB DB to call the driver functions to configure +EVPN. +When the VNI information is added/updated/delete to either a router gateway +port (patch port on the Port_Binding table) or a router interface port (also +a patch port on the Port_Binding table), it is clear that some actions need +to be trigger. +However there are other events that should be processed such as: + +- VM creation on a exposed network/router + +- Router exposed being attached/detached from the provider network + +- Subnet exposed being attached/detached from the router + + +The EVPN watcher detects OVN SB DB events of ``RowEvent`` type at the +``Port_Binding`` table. It creates a new event class named +``PortBindingChassisEvent``, that all the rest extend. +The EVPN watcher reacts to the same type of events as the BGP watcher, but +with some differences. Also, it does not react to FIPs related events as +EVPN is only used for tenant networks. + +The specific defined events to react to are: + +- ``PortBindingChassisCreatedEvent`` (set gateway port for router): + Detects when a port of type ``chassisredirect`` gets attached to the OVN + chassis where the agent is running. This is the case for neutron gateway + router ports (CR-LRPs). It calls ``expose_ip`` driver method to decide if + it needs to expose it through EVPN (in case it has related EVPN info + annotated). + +- ``PortBindingChassisDeletedEvent`` (unset gateway port for router): + Detects when a port of type ``chassisredirect`` gets detached from the OVN + chassis where teh agent is running. This is the case for neutron gateway + router ports (CR-LRPs). It calls ``withdraw_ip`` driver method to decide if + it needs to withdraw the exposed EVPN route (in case it had EVPN info + annotated). + +- ``SubnetRouterAttachedEvent`` (add BGPVPN to router/network or attach + subnet to router): Detects when a port of type ``patch`` gets + created/updated with EVPN information (VNI and BGP_AS). These type of + ports can be of 2 types: + + 1) related to the router gateway port and therefore calling the + ``expose_ip`` method, as in the ``PortBindingChassisCreateEvent``. The + different is that in ``PortBindingChassisCreateEvent`` event the port was + being created as a result of attaching the router to the provider network, + while in the ``SubnetRouterAttachedEvent`` event the port was already there + but information related to EVPN was added, i.e., the router was exposed by + associating it a BGPVPN. + + 2) related to the router interface port and therefore calling the + ``expose_subnet`` method. This method will check if the associated gateway + port is on the local chassis (where the agent runs) to proceed with the + configuration steps to redirect the traffic to/from OVN overlay. + +- ``SubnetRouterDetachedEvent`` (remove BGPVPN from router/network or detach + subnet from router): Detects when a port of type ``patch`` gets either + updated (removal of EVPN information) or directly deleted. The same 2 type + of ports as in the previous event can be found, and the method + ``withdraw_ip`` or ``withdraw_subnet`` are called for router gateway and + router interface ports, respectively. + +- ``TenantPortCreatedEvent`` (VM created): + Detects when a port of type ``""`` or ``virtual`` gets updated (chassis + added). It calls the method ``expose_remote_ip``. The method checks if + the port is not on a provider network and the chassis where the agent is + running has the gateway port for the router the VM is connected to. + +- ``TenantPortDeletedEvent`` (VM deleted): + Detects when a port of type ``""`` or ``virtual`` gets updated (chassis + deleted) or deleted. It calls the method ``withdraw_remote_ip``. The method + checks if the port is not on a provider network and the chassis where the + agent is running has the gateway port for the router the VM is connected to. + + +Driver Logic +~~~~~~~~~~~~ + +The EVPN driver is in charge of the networking configuration ensuring that +VMs on tenant networks can be reached through EVPN (N/S traffic). To acomplish +this, it needs to ensure that: + +- VM IPs can be advertized in a node where the traffic could be injected into + OVN overlay, in this case the node where the router gateway port is + scheduled (see limitations subsection). + +- Once the traffic reaches the specific node, the traffic is redirected to the + OVN overlay. + +To do that it needs to: + +1. Create the EVPN related devices when a router gets attached to the provider + network and/or gets a BGPVPN assigned to it. + + - Create the VRF device, using the VNI number as the routing table number + associated to it, as well as for the name suffix: vrf-1001 for vni 1001 + + .. code-block:: ini + + ip link add vrf-1001 type vrf table 1001 + + - Create the VXLAN device, using the VNI number as the vxlan id, as well as + for the name suffix: vxlan-1001 + + .. code-block:: ini + + ip link add vxlan-1001 type vxlan id 1001 dstport 4789 local LOOPBACK_IP nolearning + + - Create the Bridge device, where the vxlan device is connected, and + associate it to the created vrf, also using the VNI number as name suffix: + br-1001 + + .. code-block:: ini + + ip link add name br-1001 type bridge stp_state 0 + ip link set br-1001 master vrf-1001 + ip link set vxlan-1001 master br-1001 + + - Create a dummy device, where the IPs to be exposed will be added. It is + associated to the created vrf, and also using the VNI number as name + suffix: lo-1001 + + .. code-block:: ini + + ip link add name lo-1001 type dummy + ip link set lo-1001 master vrf-1001 + + .. note:: + + The VRF is not associated to an OpenStack tenant but to a router + gateway ports, meaning that if a tenant has several Neutron routers + connected to the provider network, it will have a different VRFs, one + associated with each one of them. + +2. Reconfigure local FRR instance (``frr.conf``) to ensure the new VRF is + exposed. To do that it uses ``vtysh shell``. It connects to the existing + FRR socket (--vty_socket option) and executes the next commands, passing + them through a file (-c FILE_NAME option): + + .. code-block:: ini + + ADD_VRF_TEMPLATE = ''' + vrf {{ vrf_name }} + vni {{ vni }} + exit-vrf + + router bgp {{ bgp_as }} vrf {{ vrf_name }} + address-family ipv4 unicast + redistribute connected + exit-address-family + address-family ipv6 unicast + redistribute connected + exit-address-family + address-family l2vpn evpn + advertise ipv4 unicast + advertise ipv6 unicast + exit-address-family + + ''' + +3. Connect EVPN to OVN overlay so that traffic can be redirected from the node + to the OVN virtual networking. It needs to: + + - Attach the VRF device to the OVS provider bridge (e.g., br-ex) + + .. code-block:: ini + + ovs-vsctl add-port br-ex vrf-1001 + + - Add route on the VRF routing table for both the router gateway port IP + and the subnet CIDR so that the traffic is redirected to the OVS provider + bridge (e.g., br-ex) + + .. code-block:: ini + + $ ip route show vrf vrf-1001 + 10.0.0.0/26 via 172.24.4.146 dev br-ex + 172.24.4.146 dev br-ex scope link + +4. Add needed OVS flows into the OVS provider bridge (e.g., br-ex) to redirect + the traffic back from OVN to the proper VRF, based on the subnet CIDR and + the router gateway port MAC address. + + .. code-block:: ini + + $ ovs-ofctl add-flow br-ex cookie=0x3e7,priority=1000,ip,in_port=1,dl_src:ROUTER_GATEWAY_PORT_MAC,nw_src=SUBNET_CIDR, actions=mod_dl_dst:BR_EX_MAC,output=VRF_PORT + +5. Add IPs to expose to VRF associated dummy device. This interface is only + used for the purpose of exposing the IPs, but not meant to receive the + traffic. Thus, the local route being automatically added pointing to the + dummy interface on the VRF for that (VM) IP is removed so that the traffic + can get redirected properly to the OVN overlay. + + .. code-block:: ini + + $ ip addr add 10.0.0.5/32 dev lo-1001 + $ ip route show vrf table 1001 | grep local + 10.0.0.5 dev lo-1001 + $ ip route delete local 10.0.0.5 dev 1001 table 1001 + + +Driver API +++++++++++ + +The EVPN driver needs to implement the ``driver_api.py`` interface. +It implements the next functions: + +- ``expose_ip``: Creates all the VRF/VXLAN configuration (devices and its + connection to the OVN overlay) as well as the VRF configuration at FRR + (steps 1 to 3). It also checks if there are subnets and VMs connected to + the ovn gateway router port that must be exposed through EVPN (steps 4-5). + +- ``withdraw_ip``: removes the above configuration (devices and FRR + configuration). + +- ``expose_subnet``: add kernel and ovs networking configuration to ensure + traffic can go from the node to the OVN overlay, and viceversa, for IPs + within the subnet CIDR and on the right VRF -- step 4. + +- ``withdraw_subnet``: removes the above kernel and ovs networking + configuration. + +- ``expose_remote_ip``: EVPN expose VM tenant network IPs through the chassis + hosting the ovn gateway port for the router where the VM is connected. + It ensures traffic destinated to the VM IP arrives to this node (step 5). + The previous steps ensure the traffic is redirected to the OVN overlay + once on the node. + +- ``withdraw_remote_ip``: EVPN withdraw VM tenant network IPs through the + chassis hosting the ovn gateway port for the router where the VM is + connected. It ensures traffic destinated to the VM IP stops arriving to + this node. + + +Traffic flow +~~~~~~~~~~~~ + +The next figure shows the N/S traffic flow through the VRF to the VM, +including information regarding the OVS flows on the provider bridge (br-ex), +and the routes on the VRF routing table. + +.. image:: ../../images/evpn_traffic_flow.png + :alt: integration components + :align: center + :width: 100% + + +The IPs of both the router gateway port (cr-lrp, 172.24.1.20), as well as the +IP of the VM itself (20.0.0.241/32) gets added to the dummy device (lo-101) +associated to the vrf (vrf-101) which was used for defining the BGPVPN +(vni 101). That together with the other devices created on the VRF (vxlan-101 +and br-101), and with the FRR reconfiguration ensure the IPs get exposed in +the right EVPN. This allows the traffic to reach the node with the router +gateway port (cr-lrp on ovn). + +However this is not enough as the traffic needs to be redirected to the OVN +Overlay. To do that the VRF is added to the br-ex OVS provider bridge (br-ex), +and two routes are added to the VRF routing table to redirect the traffic +going to the network (20.0.0.0/24) through the CR-LRP port to the br-ex OVS +bridge. +That injects the traffic properly into the OVN overlay, which will redirect +it through the geneve tunnel (by the br-int ovs flows) to the compute node +hosting the VM. The reply from the VM will come back through the same tunnel. +However an extra OVS flow needs to be added to the OVS provider bridge (br-ex) +to ensure the traffic is redirected back to the VRF (vrf-101) if the traffic +is coming from the exposed network (20.0.0.0/24) -- instead of using the +default routing table (action=NORMAL). To that end, the next rule is added: + +.. code-block:: ini + + cookie=0x3e6, duration=4.141s, table=0, n_packets=0, n_bytes=0, priority=1000,ip,in_port="patch-provnet-c",dl_src=fa:16:3e:b7:cc:47,nw_src=20.0.0.0/24 actions=mod_dl_dst:1e:8b:ac:5d:98:4a,output:"vrf-101" + +It matches the traffic coming from the router gateway port (cr-lrp port) from +br-int (in_port="patch-provnet-c"), with the MAC address of the router gateway +port (dl_src=fa:16:3e:b7:cc:47) and from the exposed network (nw_src=20.0.0.0/24). +For that case it changes the MAC by the br-ex device one +(mod_dl_dst:1e:8b:ac:5d:98:4a), and redirect the traffic to the vrf device +(output:"vrf-101"). + + +Agent deployment +~~~~~~~~~~~~~~~~ + +The EVPN mode exposes the VMs on tenant networks (on their respective +EVPN/VXLAN). At OpenStack, with OVN networking, the N/S traffic to the +tenant VMs (without FIPs) needs to go through the networking nodes, more +specifically the one hosting the chassisredirect ovn port (cr-lrp), connecting +the provider network to the OVN virtual router. As a result, there is no need +to deploy the agent in all the nodes. Only the nodes that are able to host +router gateway ports (cr-lrps), i.e., the ones tagged with the +``enable-chassis-gw``. Hence, the VM IPs are advertised through BGP/EVPN in +one of those nodes, and from there it follows the normal path to the OpenStack +compute node where the VM is allocated — the Geneve tunnel. + + +Limitations +----------- + +The following limitations apply: + +- Network traffic is steer by kernel routing (VRF, VXLAN, Bridges), therefore + DPDK, where the kernel space is skipped, is not supported + +- Network traffic is steer by kernel routing (VRF, VXLAN, Bridges), therefore + SRIOV, where the hypervisor is skipped, is not supported. + +- In OpenStack with OVN networking the N/S traffic to the tenant VMs (without + FIPs) needs to go through the networking nodes (the ones hosting the Neutron + Router Gateway Ports, i.e., the chassisredirect cr-lrp ports). Therefore, the + entry point into the OVN overlay need to be one of those networking nodes, + and consequently the VMs are exposed through them. From those nodes the + traffic will follow the normal tunneled path (Geneve tunnel) to the OpenStack + compute node where the VM is allocated. diff --git a/doc/source/contributor/index.rst b/doc/source/contributor/index.rst new file mode 100644 index 00000000..865b13ed --- /dev/null +++ b/doc/source/contributor/index.rst @@ -0,0 +1,10 @@ +=========================== + Contributor Documentation +=========================== + + .. toctree:: + :maxdepth: 2 + + bgp_mode_design + evpn_mode_design + diff --git a/doc/source/index.rst b/doc/source/index.rst index f3616222..c03ce5aa 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -13,6 +13,7 @@ Contents: :maxdepth: 2 readme + contributor/index Indices and tables ================== diff --git a/ovn_bgp_agent/agent.py b/ovn_bgp_agent/agent.py new file mode 100644 index 00000000..4a72281a --- /dev/null +++ b/ovn_bgp_agent/agent.py @@ -0,0 +1,77 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import functools +import sys + +from oslo_config import cfg +from oslo_log import log as logging +from oslo_service import periodic_task +from oslo_service import service + +from ovn_bgp_agent import config +from ovn_bgp_agent.drivers import driver_api + + +CONF = cfg.CONF +LOG = logging.getLogger(__name__) + + +class BGPAgentMeta(type(service.Service), + type(periodic_task.PeriodicTasks)): + pass + + +class BGPAgent(service.Service, periodic_task.PeriodicTasks, + metaclass=BGPAgentMeta): + """BGP OVN Agent.""" + + def __init__(self): + super(BGPAgent, self).__init__() + periodic_task.PeriodicTasks.__init__(self, CONF) + + self.agent_driver = driver_api.AgentDriverBase.get_instance( + CONF.driver) + + def start(self): + LOG.info("Service '%s' starting", self.__class__.__name__) + super(BGPAgent, self).start() + self.agent_driver.start() + + LOG.info("Service '%s' started", self.__class__.__name__) + f = functools.partial(self.run_periodic_tasks, None) + self.tg.add_timer(1, f) + + @periodic_task.periodic_task(spacing=CONF.reconcile_interval, + run_immediately=True) + def sync(self, context): + LOG.info("Running reconciliation loop to ensure routes/rules are " + "in place.") + self.agent_driver.sync() + + def wait(self): + super(BGPAgent, self).wait() + LOG.info("Service '%s' stopped", self.__class__.__name__) + + def stop(self, graceful=False): + LOG.info("Service '%s' stopping", self.__class__.__name__) + super(BGPAgent, self).stop(graceful) + + +def start(): + config.init(sys.argv[1:]) + config.setup_logging() + + bgp_agent_launcher = service.launch(config.CONF, BGPAgent()) + bgp_agent_launcher.wait() diff --git a/ovn_bgp_agent/cmd/__init__.py b/ovn_bgp_agent/cmd/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ovn_bgp_agent/cmd/agent.py b/ovn_bgp_agent/cmd/agent.py new file mode 100644 index 00000000..1c779a11 --- /dev/null +++ b/ovn_bgp_agent/cmd/agent.py @@ -0,0 +1,20 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from ovn_bgp_agent import agent + +start = agent.start + +if __name__ == '__main__': + start() diff --git a/ovn_bgp_agent/config.py b/ovn_bgp_agent/config.py new file mode 100644 index 00000000..963774ed --- /dev/null +++ b/ovn_bgp_agent/config.py @@ -0,0 +1,65 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from oslo_config import cfg +from oslo_log import log as logging + +LOG = logging.getLogger(__name__) + +agent_opts = [ + cfg.IntOpt('reconcile_interval', + help='Time between re-sync actions.', + default=120), + cfg.BoolOpt('expose_tenant_networks', + help='Expose VM IPs on tenant networks', + default=False), + cfg.StrOpt('driver', + help='Driver to be used', + default='osp_ovn_bgp_driver'), + cfg.StrOpt('ovn_sb_private_key', + default='/etc/pki/tls/private/ovn_controller.key', + help='The PEM file with private key for SSL connection to ' + 'OVN-SB-DB'), + cfg.StrOpt('ovn_sb_certificate', + default='/etc/pki/tls/certs/ovn_controller.crt', + help='The PEM file with certificate that certifies the ' + 'private key specified in ovn_sb_private_key'), + cfg.StrOpt('ovn_sb_ca_cert', + default='/etc/ipa/ca.crt', + help='The PEM file with CA certificate that OVN should use to' + ' verify certificates presented to it by SSL peers'), + cfg.StrOpt('bgp_AS', + default='64999', + help='AS number to be used by the Agent when running in BGP ' + 'mode and configuring the VRF route leaking.'), + cfg.StrOpt('bgp_router_id', + default=None, + help='Router ID to be used by the Agent when running in BGP ' + 'mode and configuring the VRF route leaking.'), +] + +CONF = cfg.CONF +CONF.register_opts(agent_opts) + +logging.register_options(CONF) + + +def init(args, **kwargs): + CONF(args=args, project='bgp-agent', **kwargs) + + +def setup_logging(): + logging.setup(CONF, 'bgp-agent') + logging.set_defaults(default_log_levels=logging.get_default_log_levels()) + LOG.info("Logging enabled!") diff --git a/ovn_bgp_agent/constants.py b/ovn_bgp_agent/constants.py new file mode 100644 index 00000000..2047b00b --- /dev/null +++ b/ovn_bgp_agent/constants.py @@ -0,0 +1,41 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +OVN_VIF_PORT_TYPES = ("", "chassisredirect", "virtual") + +OVN_VIRTUAL_VIF_PORT_TYPE = "virtual" +OVN_VM_VIF_PORT_TYPE = "" +OVN_PATCH_VIF_PORT_TYPE = "patch" +OVN_CHASSISREDIRECT_VIF_PORT_TYPE = "chassisredirect" +OVN_LOCALNET_VIF_PORT_TYPE = "localnet" + +OVN_BGP_NIC = "ovn" +OVN_BGP_VRF = "ovn-bgp-vrf" +OVN_BGP_VRF_TABLE = 10 +OVS_CONNECTION_STRING = "unix:/var/run/openvswitch/db.sock" +OVS_RULE_COOKIE = "999" +OVS_VRF_RULE_COOKIE = "998" + +FRR_SOCKET_PATH = "/run/frr/" + +IP_VERSION_6 = 6 +IP_VERSION_4 = 4 + +BGP_MODE = 'BGP' + +OVN_INTEGRATION_BRIDGE = 'br-int' + + +LINK_UP = "up" +LINK_DOWN = "down" \ No newline at end of file diff --git a/ovn_bgp_agent/drivers/__init__.py b/ovn_bgp_agent/drivers/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ovn_bgp_agent/drivers/driver_api.py b/ovn_bgp_agent/drivers/driver_api.py new file mode 100644 index 00000000..5a657a0a --- /dev/null +++ b/ovn_bgp_agent/drivers/driver_api.py @@ -0,0 +1,56 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import abc +from stevedore import driver as stevedore_driver + + +class AgentDriverBase(object, metaclass=abc.ABCMeta): + """Base class for agent drivers. + + """ + + @classmethod + def get_instance(cls, specific_driver): + agent_driver = stevedore_driver.DriverManager( + namespace='ovn_bgp_agent.drivers', + name=specific_driver, + invoke_on_load=True + ).driver + + return agent_driver + + @abc.abstractmethod + def expose_ip(self, ip_address): + raise NotImplementedError() + + @abc.abstractmethod + def withdraw_ip(self, ip_address): + raise NotImplementedError() + + @abc.abstractmethod + def expose_remote_ip(self, ip_address): + raise NotImplementedError() + + @abc.abstractmethod + def withdraw_remote_ip(self, ip_address): + raise NotImplementedError() + + @abc.abstractmethod + def expose_subnet(self, subnet): + raise NotImplementedError() + + @abc.abstractmethod + def withdraw_subnet(self, subnet): + raise NotImplementedError() diff --git a/ovn_bgp_agent/drivers/openstack/__init__.py b/ovn_bgp_agent/drivers/openstack/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ovn_bgp_agent/drivers/openstack/ovn_bgp_driver.py b/ovn_bgp_agent/drivers/openstack/ovn_bgp_driver.py new file mode 100644 index 00000000..1e46a6bf --- /dev/null +++ b/ovn_bgp_agent/drivers/openstack/ovn_bgp_driver.py @@ -0,0 +1,702 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import collections +import ipaddress +import pyroute2 + +from oslo_concurrency import lockutils +from oslo_config import cfg +from oslo_log import log as logging + +from ovn_bgp_agent import constants +from ovn_bgp_agent.drivers import driver_api +from ovn_bgp_agent.drivers.openstack.utils import frr +from ovn_bgp_agent.drivers.openstack.utils import ovn +from ovn_bgp_agent.drivers.openstack.utils import ovs +from ovn_bgp_agent.drivers.openstack.watchers import bgp_watcher as watcher +from ovn_bgp_agent.utils import linux_net + + +CONF = cfg.CONF +LOG = logging.getLogger(__name__) +# LOG.setLevel(logging.DEBUG) +# logging.basicConfig(level=logging.DEBUG) + +OVN_TABLES = ("Port_Binding", "Chassis", "Datapath_Binding", "Chassis_Private") + + +class OSPOVNBGPDriver(driver_api.AgentDriverBase): + + def __init__(self): + self._expose_tenant_networks = CONF.expose_tenant_networks + self.ovn_routing_tables = {} # {'br-ex': 200} + self.ovn_bridge_mappings = {} # {'public': 'br-ex'} + self.ovn_local_cr_lrps = {} + self.ovn_local_lrps = set([]) + # {'br-ex': [route1, route2]} + self.ovn_routing_tables_routes = collections.defaultdict() + + self.ovs_idl = ovs.OvsIdl() + self.ovs_idl.start(constants.OVS_CONNECTION_STRING) + self.chassis = self.ovs_idl.get_own_chassis_name() + self.ovn_remote = self.ovs_idl.get_ovn_remote() + LOG.debug("Loaded chassis {}.".format(self.chassis)) + + events = () + for event in self._get_events(): + event_class = getattr(watcher, event) + events += (event_class(self),) + + self._sb_idl = ovn.OvnSbIdl( + self.ovn_remote, + chassis=self.chassis, + tables=OVN_TABLES, + events=events) + + def start(self): + # Ensure FRR is configure to leak the routes + # NOTE: If we want to recheck this every X time, we should move it + # inside the sync function instead + frr.vrf_leak(constants.OVN_BGP_VRF, CONF.bgp_AS, CONF.bgp_router_id) + + # start the subscriptions to the OSP events. This ensures the watcher + # calls the relevant driver methods upon registered events + self.sb_idl = self._sb_idl.start() + + def _get_events(self): + events = set(["PortBindingChassisCreatedEvent", + "PortBindingChassisDeletedEvent", + "FIPSetEvent", + "FIPUnsetEvent", + "ChassisCreateEvent"]) + if self._expose_tenant_networks: + events.update(["SubnetRouterAttachedEvent", + "SubnetRouterDetachedEvent", + "TenantPortCreatedEvent", + "TenantPortDeletedEvent"]) + return events + + @lockutils.synchronized('bgp') + def sync(self): + self.ovn_local_cr_lrps = {} + self.ovn_local_lrps = set([]) + self.ovn_routing_tables_routes = collections.defaultdict() + + LOG.debug("Ensuring VRF configuration for advertising routes") + # Create VRF + linux_net.ensure_vrf(constants.OVN_BGP_VRF, + constants.OVN_BGP_VRF_TABLE) + # Create OVN dummy device + linux_net.ensure_ovn_device(constants.OVN_BGP_NIC, + constants.OVN_BGP_VRF) + + LOG.debug("Configuring br-ex default rule and routing tables for " + "each provider network") + flows_info = {} + # 1) Get bridge mappings: xxxx:br-ex,yyyy:br-ex2 + bridge_mappings = self.ovs_idl.get_ovn_bridge_mappings() + # 2) Get macs for bridge mappings + extra_routes = {} + with pyroute2.NDB() as ndb: + for bridge_mapping in bridge_mappings: + network = bridge_mapping.split(":")[0] + bridge = bridge_mapping.split(":")[1] + self.ovn_bridge_mappings[network] = bridge + if not extra_routes.get(bridge): + extra_routes[bridge] = ( + linux_net.ensure_routing_table_for_bridge( + self.ovn_routing_tables, bridge)) + vlan_tag = self.sb_idl.get_network_vlan_tag_by_network_name( + network) + if vlan_tag: + vlan_tag = vlan_tag[0] + linux_net.ensure_vlan_device_for_network(bridge, + vlan_tag) + + if flows_info.get(bridge): + continue + flows_info[bridge] = { + 'mac': ndb.interfaces[bridge]['address'], + 'in_port': set([])} + # 3) Get in_port for bridge mappings (br-ex, br-ex2) + ovs.get_ovs_flows_info(bridge, flows_info, + constants.OVS_RULE_COOKIE) + # 4) Add/Remove flows for each bridge mappings + ovs.remove_extra_ovs_flows(flows_info, constants.OVS_RULE_COOKIE) + + LOG.debug("Syncing current routes.") + exposed_ips = linux_net.get_exposed_ips(constants.OVN_BGP_NIC) + # get the rules pointing to ovn bridges + ovn_ip_rules = linux_net.get_ovn_ip_rules( + self.ovn_routing_tables.values()) + + # add missing routes/ips for fips/provider VMs + ports = self.sb_idl.get_ports_on_chassis(self.chassis) + for port in ports: + self._ensure_port_exposed(port, exposed_ips, ovn_ip_rules) + + # add missing route/ips for tenant network VMs + if self._expose_tenant_networks: + for cr_lrp_info in self.ovn_local_cr_lrps.values(): + lrp_ports = self.sb_idl.get_lrp_ports_for_router( + cr_lrp_info['router_datapath']) + for lrp in lrp_ports: + if lrp.chassis: + continue + self._ensure_network_exposed( + lrp, cr_lrp_info, exposed_ips, ovn_ip_rules) + + # remove extra routes/ips + # remove all the leftovers on the list of current ips on dev OVN + linux_net.delete_exposed_ips(exposed_ips, constants.OVN_BGP_NIC) + # remove all the leftovers on the list of current ip rules for ovn + # bridges + linux_net.delete_ip_rules(ovn_ip_rules) + + # remove all the extra rules not needed + linux_net.delete_bridge_ip_routes(self.ovn_routing_tables, + self.ovn_routing_tables_routes, + extra_routes) + + def _ensure_port_exposed(self, port, exposed_ips, ovn_ip_rules): + if port.type not in constants.OVN_VIF_PORT_TYPES: + return + if (len(port.mac[0].split(' ')) != 2 and + len(port.mac[0].split(' ')) != 3): + return + port_ips = [port.mac[0].split(' ')[1]] + if len(port.mac[0].split(' ')) == 3: + port_ips.append(port.mac[0].split(' ')[2]) + + fip = self._expose_ip(port_ips, port) + if fip: + if fip in exposed_ips: + exposed_ips.remove(fip) + fip_dst = "{}/32".format(fip) + if fip_dst in ovn_ip_rules.keys(): + del ovn_ip_rules[fip_dst] + + for port_ip in port_ips: + ip_address = port_ip.split("/")[0] + ip_version = linux_net.get_ip_version(port_ip) + if ip_version == constants.IP_VERSION_6: + ip_dst = "{}/128".format(ip_address) + else: + ip_dst = "{}/32".format(ip_address) + if ip_address in exposed_ips: + # remove each ip to add from the list of current ips on dev OVN + exposed_ips.remove(ip_address) + if ip_dst in ovn_ip_rules.keys(): + del ovn_ip_rules[ip_dst] + + def _ensure_network_exposed(self, router_port, gateway, exposed_ips=[], + ovn_ip_rules={}): + gateway_ips = [ip.split('/')[0] for ip in gateway['ips']] + try: + router_port_ip = router_port.mac[0].split(' ')[1] + except IndexError: + return + router_ip = router_port_ip.split('/')[0] + if router_ip in gateway_ips: + return + self.ovn_local_lrps.add(router_port.logical_port) + rule_bridge, vlan_tag = self._get_bridge_for_datapath( + gateway['provider_datapath']) + + linux_net.add_ip_rule(router_port_ip, + self.ovn_routing_tables[rule_bridge], + rule_bridge) + if router_port_ip in ovn_ip_rules.keys(): + del ovn_ip_rules[router_port_ip] + + router_port_ip_version = linux_net.get_ip_version(router_port_ip) + for gateway_ip in gateway_ips: + if linux_net.get_ip_version(gateway_ip) == router_port_ip_version: + linux_net.add_ip_route( + self.ovn_routing_tables_routes, + router_ip, + self.ovn_routing_tables[rule_bridge], + rule_bridge, + vlan=vlan_tag, + mask=router_port_ip.split("/")[1], + via=gateway_ip) + break + + network_port_datapath = self.sb_idl.get_port_datapath( + router_port.options['peer']) + if network_port_datapath: + ports = self.sb_idl.get_ports_on_datapath( + network_port_datapath) + for port in ports: + if ((port.type != constants.OVN_VM_VIF_PORT_TYPE and port.type != constants.OVN_VIRTUAL_VIF_PORT_TYPE) or + (port.type == constants.OVN_VM_VIF_PORT_TYPE and not port.chassis)): + continue + try: + port_ips = [port.mac[0].split(' ')[1]] + except IndexError: + continue + if len(port.mac[0].split(' ')) == 3: + port_ips.append(port.mac[0].split(' ')[2]) + + for port_ip in port_ips: + # Only adding the port ips that match the lrp + # IP version + port_ip_version = linux_net.get_ip_version(port_ip) + if port_ip_version == router_port_ip_version: + linux_net.add_ips_to_dev( + constants.OVN_BGP_NIC, [port_ip]) + if port_ip in exposed_ips: + exposed_ips.remove(port_ip) + if router_port_ip_version == constants.IP_VERSION_6: + ip_dst = "{}/128".format(port_ip) + else: + ip_dst = "{}/32".format(port_ip) + + if ip_dst in ovn_ip_rules.keys(): + del ovn_ip_rules[ip_dst] + + def _remove_network_exposed(self, router_port, gateway): + gateway_ips = [ip.split('/')[0] for ip in gateway['ips']] + try: + router_port_ip = router_port.mac[0].split(' ')[1] + except IndexError: + return + router_ip = router_port_ip.split('/')[0] + if router_ip in gateway_ips: + return + + if router_port.logical_port in self.ovn_local_lrps: + self.ovn_local_lrps.remove(router_port.logical_port) + rule_bridge, vlan_tag = self._get_bridge_for_datapath( + gateway['provider_datapath']) + + linux_net.del_ip_rule(router_port_ip, + self.ovn_routing_tables[rule_bridge], + rule_bridge) + + router_port_ip_version = linux_net.get_ip_version(router_port_ip) + for gateway_ip in gateway_ips: + if linux_net.get_ip_version(gateway_ip) == router_port_ip_version: + linux_net.del_ip_route( + self.ovn_routing_tables_routes, + router_ip, + self.ovn_routing_tables[rule_bridge], + rule_bridge, + vlan=vlan_tag, + mask=router_port_ip.split("/")[1], + via=gateway_ip) + if (linux_net.get_ip_version(gateway_ip) == + constants.IP_VERSION_6): + net = ipaddress.IPv6Network(router_port_ip, strict=False) + else: + net = ipaddress.IPv4Network(router_port_ip, strict=False) + break + # Check if there are VMs on the network + # and if so withdraw the routes + vms_on_net = linux_net.get_exposed_ips_on_network( + constants.OVN_BGP_NIC, net) + linux_net.delete_exposed_ips(vms_on_net, constants.OVN_BGP_NIC) + + def _get_bridge_for_datapath(self, datapath): + network_name, network_tag = self.sb_idl.get_network_name_and_tag( + datapath, self.ovn_bridge_mappings.keys()) + if network_name: + if network_tag: + return self.ovn_bridge_mappings[network_name], network_tag[0] + return self.ovn_bridge_mappings[network_name], None + return None, None + + @lockutils.synchronized('bgp') + def expose_ip(self, ips, row, associated_port=None): + '''Advertice BGP route by adding IP to device. + + This methods ensures BGP advertises the IP of the VM in the provider + network, or the FIP associated to a VM in a tenant networks. + + It relies on Zebra, which creates and advertises a route when an IP + is added to a local interface. + + This method assumes a device named self.ovn_decice exists (inside a + VRF), and adds the IP of either: + - VM IP on the provider network, + - VM FIP, or + - CR-LRP OVN port + ''' + self._expose_ip(ips, row, associated_port) + + def _expose_ip(self, ips, row, associated_port=None): + # VM on provider Network + if ((row.type == constants.OVN_VM_VIF_PORT_TYPE + or row.type == constants.OVN_VIRTUAL_VIF_PORT_TYPE) and + self.sb_idl.is_provider_network(row.datapath)): + LOG.info("Add BGP route for logical port with ip %s", ips) + linux_net.add_ips_to_dev(constants.OVN_BGP_NIC, ips) + + rule_bridge, vlan_tag = self._get_bridge_for_datapath(row.datapath) + for ip in ips: + linux_net.add_ip_rule(ip, + self.ovn_routing_tables[rule_bridge], + rule_bridge) + linux_net.add_ip_route( + self.ovn_routing_tables_routes, ip, + self.ovn_routing_tables[rule_bridge], rule_bridge, + vlan=vlan_tag) + + # VM with FIP + elif (row.type == constants.OVN_VM_VIF_PORT_TYPE + or row.type == constants.OVN_VIRTUAL_VIF_PORT_TYPE): + # FIPs are only supported with IPv4 + fip_address, fip_datapath = self.sb_idl.get_fip_associated( + row.logical_port) + if fip_address: + LOG.info("Add BGP route for FIP with ip %s", fip_address) + linux_net.add_ips_to_dev(constants.OVN_BGP_NIC, + [fip_address]) + + rule_bridge, vlan_tag = self._get_bridge_for_datapath( + fip_datapath) + linux_net.add_ip_rule(fip_address, + self.ovn_routing_tables[rule_bridge], + rule_bridge) + linux_net.add_ip_route( + self.ovn_routing_tables_routes, fip_address, + self.ovn_routing_tables[rule_bridge], rule_bridge, + vlan=vlan_tag) + return fip_address + else: + ovs.ensure_default_ovs_flows(self.ovn_bridge_mappings.values(), + constants.OVS_RULE_COOKIE) + + # FIP association to VM + elif row.type == constants.OVN_PATCH_VIF_PORT_TYPE: + if (associated_port and self.sb_idl.is_port_on_chassis( + associated_port, self.chassis)): + LOG.info("Add BGP route for FIP with ip %s", ips) + linux_net.add_ips_to_dev(constants.OVN_BGP_NIC, ips) + + rule_bridge, vlan_tag = self._get_bridge_for_datapath( + row.datapath) + for ip in ips: + linux_net.add_ip_rule(ip, + self.ovn_routing_tables[rule_bridge], + rule_bridge) + linux_net.add_ip_route( + self.ovn_routing_tables_routes, ip, + self.ovn_routing_tables[rule_bridge], rule_bridge, + vlan=vlan_tag) + + # CR-LRP Port + elif (row.type == constants.OVN_CHASSISREDIRECT_VIF_PORT_TYPE and + row.logical_port.startswith('cr-')): + _, cr_lrp_datapath = self.sb_idl.get_fip_associated( + row.logical_port) + if cr_lrp_datapath: + LOG.info("Add BGP route for CR-LRP Port %s", ips) + # Keeping information about the associated network for + # tenant network advertisement + self.ovn_local_cr_lrps[row.logical_port] = { + 'router_datapath': row.datapath, + 'provider_datapath': cr_lrp_datapath, + 'ips': ips + } + ips_without_mask = [ip.split("/")[0] for ip in ips] + linux_net.add_ips_to_dev(constants.OVN_BGP_NIC, + ips_without_mask) + + rule_bridge, vlan_tag = self._get_bridge_for_datapath( + cr_lrp_datapath) + + for ip in ips: + ip_without_mask = ip.split("/")[0] + linux_net.add_ip_rule( + ip_without_mask, self.ovn_routing_tables[rule_bridge], + rule_bridge, lladdr=row.mac[0].split(' ')[0]) + linux_net.add_ip_route( + self.ovn_routing_tables_routes, ip_without_mask, + self.ovn_routing_tables[rule_bridge], rule_bridge, + vlan=vlan_tag) + # add proxy ndp config for ipv6 + if (linux_net.get_ip_version(ip_without_mask) == + constants.IP_VERSION_6): + linux_net.add_ndp_proxy(ip, rule_bridge, vlan_tag) + + # Check if there are networks attached to the router, + # and if so, add the needed routes/rules + if not self._expose_tenant_networks: + return + lrp_ports = self.sb_idl.get_lrp_ports_for_router( + row.datapath) + for lrp in lrp_ports: + if lrp.chassis: + continue + self._ensure_network_exposed( + lrp, self.ovn_local_cr_lrps[row.logical_port]) + + @lockutils.synchronized('bgp') + def withdraw_ip(self, ips, row, associated_port=None): + '''Withdraw BGP route by removing IP from device. + + This methods ensures BGP withdraw an advertised IP of a VM, either + in the provider network, or the FIP associated to a VM in a tenant + networks. + + It relies on Zebra, which withdraws the advertisement as soon as the + IP is deleted from the local interface. + + This method assumes a device named self.ovn_decice exists (inside a + VRF), and removes the IP of either: + - VM IP on the provider network, + - VM FIP, or + - CR-LRP OVN port + ''' + # VM on provider Network + if ((row.type == constants.OVN_VM_VIF_PORT_TYPE + or row.type == constants.OVN_VIRTUAL_VIF_PORT_TYPE) and + self.sb_idl.is_provider_network(row.datapath)): + LOG.info("Delete BGP route for logical port with ip %s", ips) + linux_net.del_ips_from_dev(constants.OVN_BGP_NIC, ips) + + rule_bridge, vlan_tag = self._get_bridge_for_datapath(row.datapath) + for ip in ips: + linux_net.del_ip_rule(ip, + self.ovn_routing_tables[rule_bridge], + rule_bridge) + linux_net.del_ip_route( + self.ovn_routing_tables_routes, ip, + self.ovn_routing_tables[rule_bridge], rule_bridge, + vlan=vlan_tag) + + # VM with FIP + elif (row.type == constants.OVN_VM_VIF_PORT_TYPE + or row.type == constants.OVN_VIRTUAL_VIF_PORT_TYPE): + # FIPs are only supported with IPv4 + fip_address, fip_datapath = self.sb_idl.get_fip_associated( + row.logical_port) + if fip_address: + LOG.info("Delete BGP route for FIP with ip %s", fip_address) + linux_net.del_ips_from_dev(constants.OVN_BGP_NIC, + [fip_address]) + + rule_bridge, vlan_tag = self._get_bridge_for_datapath( + fip_datapath) + linux_net.del_ip_rule(fip_address, + self.ovn_routing_tables[rule_bridge], + rule_bridge) + linux_net.del_ip_route( + self.ovn_routing_tables_routes, fip_address, + self.ovn_routing_tables[rule_bridge], rule_bridge, + vlan=vlan_tag) + + # FIP association to VM + elif row.type == constants.OVN_PATCH_VIF_PORT_TYPE: + if (associated_port and ( + self.sb_idl.is_port_on_chassis( + associated_port, self.chassis) or + self.sb_idl.is_port_deleted(associated_port))): + LOG.info("Delete BGP route for FIP with ip %s", ips) + linux_net.del_ips_from_dev(constants.OVN_BGP_NIC, ips) + + rule_bridge, vlan_tag = self._get_bridge_for_datapath( + row.datapath) + for ip in ips: + linux_net.del_ip_rule(ip, + self.ovn_routing_tables[rule_bridge], + rule_bridge) + linux_net.del_ip_route( + self.ovn_routing_tables_routes, ip, + self.ovn_routing_tables[rule_bridge], rule_bridge, + vlan=vlan_tag) + + # CR-LRP Port + elif (row.type == constants.OVN_CHASSISREDIRECT_VIF_PORT_TYPE and + row.logical_port.startswith('cr-')): + cr_lrp_datapath = self.ovn_local_cr_lrps.get( + row.logical_port, {}).get('provider_datapath') + if cr_lrp_datapath: + LOG.info("Delete BGP route for CR-LRP Port %s", ips) + # Removing information about the associated network for + # tenant network advertisement + ips_without_mask = [ip.split("/")[0] for ip in ips] + linux_net.del_ips_from_dev(constants.OVN_BGP_NIC, + ips_without_mask) + + rule_bridge, vlan_tag = self._get_bridge_for_datapath( + cr_lrp_datapath) + + for ip in ips_without_mask: + if linux_net.get_ip_version(ip) == constants.IP_VERSION_6: + cr_lrp_ip = '{}/128'.format(ip) + else: + cr_lrp_ip = '{}/32'.format(ip) + linux_net.del_ip_rule( + cr_lrp_ip, self.ovn_routing_tables[rule_bridge], + rule_bridge, lladdr=row.mac[0].split(' ')[0]) + linux_net.del_ip_route( + self.ovn_routing_tables_routes, ip, + self.ovn_routing_tables[rule_bridge], rule_bridge, + vlan=vlan_tag) + # del proxy ndp config for ipv6 + if linux_net.get_ip_version(ip) == constants.IP_VERSION_6: + cr_lrps_on_same_provider = [ + p for p in self.ovn_local_cr_lrps.values() + if p['provider_datapath'] == cr_lrp_datapath] + if (len(cr_lrps_on_same_provider) > 1): + linux_net.del_ndp_proxy(ip, rule_bridge, vlan_tag) + + # Check if there are networks attached to the router, + # and if so, delete the needed routes/rules + lrp_ports = self.sb_idl.get_lrp_ports_for_router( + row.datapath) + for lrp in lrp_ports: + if lrp.chassis: + continue + local_cr_lrp_info = self.ovn_local_cr_lrps.get( + row.logical_port) + if local_cr_lrp_info: + self._remove_network_exposed(lrp, local_cr_lrp_info) + try: + del self.ovn_local_cr_lrps[row.logical_port] + except KeyError: + LOG.debug("Gateway port %s already cleanup from the " + "agent", row.logical_port) + + @lockutils.synchronized('bgp') + def expose_remote_ip(self, ips, row): + if (self.sb_idl.is_provider_network(row.datapath) or + not self._expose_tenant_networks): + return + port_lrp = self.sb_idl.get_lrp_port_for_datapath(row.datapath) + if port_lrp in self.ovn_local_lrps: + LOG.info("Add BGP route for tenant IP %s on chassis %s", + ips, self.chassis) + linux_net.add_ips_to_dev(constants.OVN_BGP_NIC, ips) + + @lockutils.synchronized('bgp') + def withdraw_remote_ip(self, ips, row): + if (self.sb_idl.is_provider_network(row.datapath) or + not self._expose_tenant_networks): + return + port_lrp = self.sb_idl.get_lrp_port_for_datapath(row.datapath) + if port_lrp in self.ovn_local_lrps: + LOG.info("Delete BGP route for tenant IP %s on chassis %s", + ips, self.chassis) + linux_net.del_ips_from_dev(constants.OVN_BGP_NIC, ips) + + @lockutils.synchronized('bgp') + def expose_subnet(self, ip, row): + if not self._expose_tenant_networks: + return + cr_lrp = self.sb_idl.is_router_gateway_on_chassis(row.datapath, + self.chassis) + if cr_lrp: + LOG.info("Add IP Rules for network %s on chassis %s", + ip, self.chassis) + self.ovn_local_lrps.add(row.logical_port) + cr_lrp_info = self.ovn_local_cr_lrps.get(cr_lrp, {}) + cr_lrp_datapath = cr_lrp_info.get('provider_datapath') + if cr_lrp_datapath: + cr_lrp_ips = [ip_address.split('/')[0] + for ip_address in cr_lrp_info.get('ips', [])] + rule_bridge, vlan_tag = self._get_bridge_for_datapath( + cr_lrp_datapath) + linux_net.add_ip_rule(ip, + self.ovn_routing_tables[rule_bridge], + rule_bridge) + + ip_version = linux_net.get_ip_version(ip) + for cr_lrp_ip in cr_lrp_ips: + if linux_net.get_ip_version(cr_lrp_ip) == ip_version: + linux_net.add_ip_route( + self.ovn_routing_tables_routes, + ip.split("/")[0], + self.ovn_routing_tables[rule_bridge], + rule_bridge, + vlan=vlan_tag, + mask=ip.split("/")[1], + via=cr_lrp_ip) + break + + # Check if there are VMs on the network + # and if so expose the route + network_port_datapath = self.sb_idl.get_port_datapath( + row.options['peer']) + if network_port_datapath: + ports = self.sb_idl.get_ports_on_datapath( + network_port_datapath) + for port in ports: + if port.type != constants.OVN_VM_VIF_PORT_TYPE and port.type != constants.OVN_VIRTUAL_VIF_PORT_TYPE: + continue + try: + port_ips = [port.mac[0].split(' ')[1]] + except IndexError: + continue + if len(port.mac[0].split(' ')) == 3: + port_ips.append(port.mac[0].split(' ')[2]) + + for port_ip in port_ips: + # Only adding the port ips that match the lrp + # IP version + port_ip_version = linux_net.get_ip_version(port_ip) + if port_ip_version == ip_version: + linux_net.add_ips_to_dev( + constants.OVN_BGP_NIC, [port_ip]) + + @lockutils.synchronized('bgp') + def withdraw_subnet(self, ip, row): + if not self._expose_tenant_networks: + return + cr_lrp = self.sb_idl.is_router_gateway_on_chassis(row.datapath, + self.chassis) + if cr_lrp: + LOG.info("Delete IP Rules for network %s on chassis %s", + ip, self.chassis) + if row.logical_port in self.ovn_local_lrps: + self.ovn_local_lrps.remove(row.logical_port) + cr_lrp_info = self.ovn_local_cr_lrps.get(cr_lrp, {}) + cr_lrp_datapath = cr_lrp_info.get('provider_datapath') + + if cr_lrp_datapath: + cr_lrp_ips = [ip_address.split('/')[0] + for ip_address in cr_lrp_info.get('ips', [])] + rule_bridge, vlan_tag = self._get_bridge_for_datapath( + cr_lrp_datapath) + linux_net.del_ip_rule(ip, + self.ovn_routing_tables[rule_bridge], + rule_bridge) + + ip_version = linux_net.get_ip_version(ip) + for cr_lrp_ip in cr_lrp_ips: + if linux_net.get_ip_version(cr_lrp_ip) == ip_version: + linux_net.del_ip_route( + self.ovn_routing_tables_routes, + ip.split("/")[0], + self.ovn_routing_tables[rule_bridge], + rule_bridge, + vlan=vlan_tag, + mask=ip.split("/")[1], + via=cr_lrp_ip) + if (linux_net.get_ip_version(cr_lrp_ip) == + constants.IP_VERSION_6): + net = ipaddress.IPv6Network(ip, strict=False) + else: + net = ipaddress.IPv4Network(ip, strict=False) + break + + # Check if there are VMs on the network + # and if so withdraw the routes + vms_on_net = linux_net.get_exposed_ips_on_network( + constants.OVN_BGP_NIC, net) + linux_net.delete_exposed_ips(vms_on_net, + constants.OVN_BGP_NIC) diff --git a/ovn_bgp_agent/drivers/openstack/utils/__init__.py b/ovn_bgp_agent/drivers/openstack/utils/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ovn_bgp_agent/drivers/openstack/utils/frr.py b/ovn_bgp_agent/drivers/openstack/utils/frr.py new file mode 100644 index 00000000..0aba17c2 --- /dev/null +++ b/ovn_bgp_agent/drivers/openstack/utils/frr.py @@ -0,0 +1,144 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import json + +from jinja2 import Template +from oslo_concurrency import processutils +from oslo_log import log as logging + +from ovn_bgp_agent import constants + +LOG = logging.getLogger(__name__) + +ADD_VRF_TEMPLATE = ''' +vrf {{ vrf_name }} + vni {{ vni }} +exit-vrf + +router bgp {{ bgp_as }} vrf {{ vrf_name }} + address-family ipv4 unicast + redistribute connected + exit-address-family + address-family ipv6 unicast + redistribute connected + exit-address-family + address-family l2vpn evpn + advertise ipv4 unicast + advertise ipv6 unicast + exit-address-family + +''' + +DEL_VRF_TEMPLATE = ''' +no vrf {{ vrf_name }} +no router bgp {{ bgp_as }} vrf {{ vrf_name }} + +''' + +LEAK_VRF_TEMPLATE = ''' +router bgp {{ bgp_as }} + address-family ipv4 unicast + import vrf {{ vrf_name }} + exit-address-family + + address-family ipv6 unicast + import vrf {{ vrf_name }} + exit-address-family + +router bgp {{ bgp_as }} vrf {{ vrf_name }} + bgp router-id {{ bgp_router_id }} + address-family ipv4 unicast + redistribute connected + exit-address-family + + address-family ipv6 unicast + redistribute connected + exit-address-family + +''' + + +def _run_vtysh_config(frr_config_file): + vtysh_command = "copy {} running-config".format(frr_config_file) + full_args = ['/usr/bin/vtysh', '--vty_socket', constants.FRR_SOCKET_PATH, + '-c', vtysh_command] + try: + return processutils.execute(*full_args, run_as_root=True) + except Exception as e: + print("Unable to execute vtysh with {}. Exception: {}".format( + full_args, e)) + raise + + +def _run_vtysh_command(command): + full_args = ['/usr/bin/vtysh', '--vty_socket', constants.FRR_SOCKET_PATH, + '-c', command] + try: + return processutils.execute(*full_args, run_as_root=True)[0] + except Exception as e: + print("Unable to execute vtysh with {}. Exception: {}".format( + full_args, e)) + raise + + +def _get_router_id(bgp_as): + output = _run_vtysh_command(command='show ip bgp summary json') + return json.loads(output).get('ipv4Unicast', {}).get('routerId') + + +def vrf_leak(vrf, bgp_as, bgp_router_id=None): + LOG.info("Add VRF leak for VRF {} on router bgp {}".format(vrf, bgp_as)) + if not bgp_router_id: + bgp_router_id = _get_router_id(bgp_as) + if not bgp_router_id: + LOG.error("Unknown router-id, needed for route leaking") + return + + vrf_template = Template(LEAK_VRF_TEMPLATE) + vrf_config = vrf_template.render(vrf_name=vrf, bgp_as=bgp_as, + bgp_router_id=bgp_router_id) + frr_config_file = "frr-config-vrf-leak-{}".format(vrf) + with open(frr_config_file, 'w') as vrf_config_file: + vrf_config_file.write(vrf_config) + + _run_vtysh_config(frr_config_file) + + +def vrf_reconfigure(evpn_info, action): + LOG.info("FRR reconfiguration (action = {}) for evpn: {}".format( + action, evpn_info)) + frr_config_file = None + if action == "add-vrf": + vrf_template = Template(ADD_VRF_TEMPLATE) + vrf_config = vrf_template.render( + vrf_name="{}{}".format(constants.OVN_EVPN_VRF_PREFIX, + evpn_info['vni']), + bgp_as=evpn_info['bgp_as'], + vni=evpn_info['vni']) + frr_config_file = "frr-config-add-vrf-{}".format(evpn_info['vni']) + elif action == "del-vrf": + vrf_template = Template(DEL_VRF_TEMPLATE) + vrf_config = vrf_template.render( + vrf_name="{}{}".format(constants.OVN_EVPN_VRF_PREFIX, + evpn_info['vni']), + bgp_as=evpn_info['bgp_as']) + frr_config_file = "frr-config-del-vrf-{}".format(evpn_info['vni']) + else: + LOG.error("Unknown FRR reconfiguration action: %s", action) + return + with open(frr_config_file, 'w') as vrf_config_file: + vrf_config_file.write(vrf_config) + + _run_vtysh_config(frr_config_file) diff --git a/ovn_bgp_agent/drivers/openstack/utils/ovn.py b/ovn_bgp_agent/drivers/openstack/utils/ovn.py new file mode 100644 index 00000000..b697ab93 --- /dev/null +++ b/ovn_bgp_agent/drivers/openstack/utils/ovn.py @@ -0,0 +1,249 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from oslo_config import cfg + +from ovs.stream import Stream +from ovsdbapp.backend import ovs_idl +from ovsdbapp.backend.ovs_idl import connection +from ovsdbapp.backend.ovs_idl import idlutils +from ovsdbapp import event +from ovsdbapp.schema.ovn_southbound import impl_idl as sb_impl_idl + +from ovn_bgp_agent import constants + +CONF = cfg.CONF + + +class OvnIdl(connection.OvsdbIdl): + def __init__(self, driver, remote, schema): + super(OvnIdl, self).__init__(remote, schema) + self.driver = driver + self.notify_handler = OvnDbNotifyHandler(driver) + self.event_lock_name = "neutron_ovn_event_lock" + + def notify(self, event, row, updates=None): + if self.is_lock_contended: + return + self.notify_handler.notify(event, row, updates) + + +class OvnDbNotifyHandler(event.RowEventHandler): + def __init__(self, driver): + super(OvnDbNotifyHandler, self).__init__() + self.driver = driver + + +class OvnSbIdl(OvnIdl): + SCHEMA = 'OVN_Southbound' + + def __init__(self, connection_string, chassis=None, events=None, + tables=None): + if connection_string.startswith("ssl"): + self._check_and_set_ssl_files(self.SCHEMA) + helper = self._get_ovsdb_helper(connection_string) + self._events = events + if tables is None: + tables = ('Chassis', 'Encap', 'Port_Binding', 'Datapath_Binding', + 'SB_Global') + for table in tables: + helper.register_table(table) + super(OvnSbIdl, self).__init__( + None, connection_string, helper) + if chassis: + table = ('Chassis_Private' if 'Chassis_Private' in tables + else 'Chassis') + self.tables[table].condition = [['name', '==', chassis]] + + def _get_ovsdb_helper(self, connection_string): + return idlutils.get_schema_helper(connection_string, self.SCHEMA) + + def _check_and_set_ssl_files(self, schema_name): + priv_key_file = CONF.ovn_sb_private_key + cert_file = CONF.ovn_sb_certificate + ca_cert_file = CONF.ovn_sb_ca_cert + + if priv_key_file: + Stream.ssl_set_private_key_file(priv_key_file) + + if cert_file: + Stream.ssl_set_certificate_file(cert_file) + + if ca_cert_file: + Stream.ssl_set_ca_cert_file(ca_cert_file) + + def start(self): + conn = connection.Connection( + self, timeout=180) + ovsdbSbConn = OvsdbSbOvnIdl(conn) + if self._events: + self.notify_handler.watch_events(self._events) + return ovsdbSbConn + + +class Backend(ovs_idl.Backend): + lookup_table = {} + ovsdb_connection = None + + def __init__(self, connection): + self.ovsdb_connection = connection + super(Backend, self).__init__(connection) + + @property + def idl(self): + return self.ovsdb_connection.idl + + @property + def tables(self): + return self.idl.tables + + +class OvsdbSbOvnIdl(sb_impl_idl.OvnSbApiIdlImpl, Backend): + def __init__(self, connection): + super(OvsdbSbOvnIdl, self).__init__(connection) + self.idl._session.reconnect.set_probe_interval(60000) + + def _get_port_by_name(self, port): + cmd = self.db_find_rows('Port_Binding', ('logical_port', '=', port)) + port_info = cmd.execute(check_error=True) + if port_info: + return port_info[0] + return [] + + def _get_ports_by_datapath(self, datapath, port_type=None): + if port_type: + cmd = self.db_find_rows('Port_Binding', + ('datapath', '=', datapath), + ('type', '=', port_type)) + else: + cmd = self.db_find_rows('Port_Binding', + ('datapath', '=', datapath)) + return cmd.execute(check_error=True) + + def is_provider_network(self, datapath): + cmd = self.db_find_rows('Port_Binding', ('datapath', '=', datapath), + ('type', '=', 'localnet')) + return next(iter(cmd.execute(check_error=True)), None) + + def get_fip_associated(self, port): + cmd = self.db_find_rows('Port_Binding', ('type', '=', 'patch')) + for row in cmd.execute(check_error=True): + for fip in row.nat_addresses: + if port in fip: + return fip.split(" ")[1], row.datapath + return None, None + + def is_port_on_chassis(self, port_name, chassis): + port_info = self._get_port_by_name(port_name) + try: + if (port_info and port_info.type == constants.OVN_VM_VIF_PORT_TYPE and + port_info.chassis[0].name == chassis): + return True + except IndexError: + pass + return False + + def is_port_deleted(self, port_name): + port_info = self._get_port_by_name(port_name) + if port_info: + return False + return True + + def get_ports_on_chassis(self, chassis): + rows = self.db_list_rows('Port_Binding').execute(check_error=True) + return [r for r in rows if r.chassis and r.chassis[0].name == chassis] + + def get_network_name_and_tag(self, datapath, bridge_mappings): + for row in self._get_ports_by_datapath( + datapath, constants.OVN_LOCALNET_VIF_PORT_TYPE): + if (row.options and + row.options.get('network_name') in bridge_mappings): + return row.options.get('network_name'), row.tag + return None, None + + def get_network_vlan_tag_by_network_name(self, network_name): + cmd = self.db_find_rows('Port_Binding', ('type', '=', + constants.OVN_LOCALNET_VIF_PORT_TYPE)) + for row in cmd.execute(check_error=True): + if (row.options and + row.options.get('network_name') == network_name): + return row.tag + return None + + def is_router_gateway_on_chassis(self, datapath, chassis): + port_info = self._get_ports_by_datapath( + datapath, constants.OVN_CHASSISREDIRECT_VIF_PORT_TYPE) + try: + if port_info and port_info[0].chassis[0].name == chassis: + return port_info[0].logical_port + except IndexError: + pass + return None + + def get_lrp_port_for_datapath(self, datapath): + for row in self._get_ports_by_datapath( + datapath, constants.OVN_PATCH_VIF_PORT_TYPE): + if row.options: + return row.options['peer'] + return None + + def get_lrp_ports_for_router(self, datapath): + return self._get_ports_by_datapath( + datapath, constants.OVN_PATCH_VIF_PORT_TYPE) + + def get_port_datapath(self, port_name): + port_info = self._get_port_by_name(port_name) + if port_info: + return port_info.datapath + return None + + def get_ports_on_datapath(self, datapath): + return self._get_ports_by_datapath(datapath) + + def get_evpn_info_from_crlrp_port_name(self, port_name): + router_gateway_port_name = port_name.split('cr-lrp-')[1] + return self.get_evpn_info_from_port_name(router_gateway_port_name) + + def get_evpn_info_from_lrp_port_name(self, port_name): + router_interface_port_name = port_name.split('lrp-')[1] + return self.get_evpn_info_from_port_name(router_interface_port_name) + + def get_ip_from_port_peer(self, port): + peer_name = port.options['peer'] + peer_port = self._get_port_by_name(peer_name) + return peer_port.mac[0].split(' ')[1] + + def get_evpn_info_from_port(self, port): + return self.get_evpn_info(port) + + def get_evpn_info_from_port_name(self, port_name): + port = self._get_port_by_name(port_name) + return self.get_evpn_info(port) + + def get_evpn_info(self, port): + try: + evpn_info = { + 'vni': int(port.external_ids[ + constants.OVN_EVPN_VNI_EXT_ID_KEY]), + 'bgp_as': int(port.external_ids[ + constants.OVN_EVPN_AS_EXT_ID_KEY])} + except KeyError: + return {} + return evpn_info + + def get_port_if_local_chassis(self, port_name, chassis): + port = self._get_port_by_name(port_name) + if port.chassis[0].name == chassis: + return port + return None diff --git a/ovn_bgp_agent/drivers/openstack/utils/ovs.py b/ovn_bgp_agent/drivers/openstack/utils/ovs.py new file mode 100644 index 00000000..73128547 --- /dev/null +++ b/ovn_bgp_agent/drivers/openstack/utils/ovs.py @@ -0,0 +1,319 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import pyroute2 + +from oslo_concurrency import processutils +from ovs.db import idl + +from ovn_bgp_agent import constants +from ovn_bgp_agent.utils import linux_net + +from ovsdbapp.backend.ovs_idl import connection +from ovsdbapp.backend.ovs_idl import idlutils +from ovsdbapp.schema.open_vswitch import impl_idl as idl_ovs + + +def ovs_cmd(command, args, timeout=None): + full_args = [command] + if timeout is not None: + full_args += ['--timeout=%s' % timeout] + full_args += args + try: + return processutils.execute(*full_args, run_as_root=True) + except Exception as e: + print("Unable to execute {} {}. Exception: {}".format( + command, full_args, e)) + raise + + +def get_ovs_flows_info(bridge, flows_info, cookie): + ovs_ports = ovs_cmd('ovs-vsctl', + ['list-ports', bridge])[0].rstrip() + if not ovs_ports: + flow = ("cookie={}/-1").format(cookie) + ovs_cmd('ovs-ofctl', ['del-flows', bridge, flow]) + return + for ovs_port in ovs_ports.split("\n"): + ovs_ofport = ovs_cmd( + 'ovs-vsctl', + ['get', 'Interface', ovs_port, 'ofport'])[0].rstrip() + flows_info[bridge]['in_port'].add(ovs_ofport) + + +def remove_extra_ovs_flows(flows_info, cookie): + for bridge, info in flows_info.items(): + for in_port in info.get('in_port'): + flow = ("cookie={},priority=900,ip,in_port={}," + "actions=mod_dl_dst:{},NORMAL".format( + cookie, in_port, info['mac'])) + flow_v6 = ("cookie={},priority=900,ipv6,in_port={}," + "actions=mod_dl_dst:{},NORMAL".format( + cookie, in_port, info['mac'])) + ovs_cmd('ovs-ofctl', ['add-flow', bridge, flow]) + ovs_cmd('ovs-ofctl', ['add-flow', bridge, flow_v6]) + + cookie_id = ("cookie={}/-1").format(cookie) + current_flows = ovs_cmd( + 'ovs-ofctl', ['dump-flows', bridge, cookie_id] + )[0].split('\n')[1:-1] + for flow in current_flows: + agent_flow = False + for port in info.get('in_port'): + in_port = 'in_port={}'.format(port) + if in_port in flow: + agent_flow = True + break + if agent_flow: + continue + in_port = flow.split("in_port=")[1].split(" ")[0] + del_flow = ('{},in_port={}').format(cookie_id, in_port) + ovs_cmd('ovs-ofctl', ['del-flows', bridge, del_flow]) + + +def ensure_evpn_ovs_flow(bridge, cookie, mac, port, net, strip_vlan=False): + ovs_port = None + ovs_ports = ovs_cmd('ovs-vsctl', ['list-ports', bridge])[0].rstrip() + for p in ovs_ports.split('\n'): + if p.startswith('patch-provnet-'): + ovs_port = p + if not ovs_port: + return + ovs_ofport = ovs_cmd( + 'ovs-vsctl', ['get', 'Interface', ovs_port, 'ofport'] + )[0].rstrip() + vrf_ofport = ovs_cmd( + 'ovs-vsctl', ['get', 'Interface', port, 'ofport'] + )[0].rstrip() + + ip_version = linux_net.get_ip_version(net) + if ip_version == constants.IP_VERSION_6: + with pyroute2.NDB() as ndb: + if strip_vlan: + flow = ( + "cookie={},priority=1000,ipv6,in_port={},dl_src:{}," + "ipv6_src={} actions=mod_dl_dst:{},strip_vlan," + "output={}".format( + cookie, ovs_ofport, mac, net, + ndb.interfaces[bridge]['address'], vrf_ofport)) + else: + flow = ( + "cookie={},priority=1000,ipv6,in_port={},dl_src:{}," + "ipv6_src={} actions=mod_dl_dst:{},output={}".format( + cookie, ovs_ofport, mac, net, + ndb.interfaces[bridge]['address'], vrf_ofport)) + else: + with pyroute2.NDB() as ndb: + if strip_vlan: + flow = ( + "cookie={},priority=1000,ip,in_port={},dl_src:{},nw_src={}" + "actions=mod_dl_dst:{},strip_vlan,output={}".format( + cookie, ovs_ofport, mac, net, + ndb.interfaces[bridge]['address'], vrf_ofport)) + else: + flow = ( + "cookie={},priority=1000,ip,in_port={},dl_src:{},nw_src={}" + "actions=mod_dl_dst:{},output={}".format( + cookie, ovs_ofport, mac, net, + ndb.interfaces[bridge]['address'], vrf_ofport)) + ovs_cmd('ovs-ofctl', ['add-flow', bridge, flow]) + + +def remove_evpn_router_ovs_flows(bridge, cookie, mac): + cookie_id = ("cookie={}/-1").format(cookie) + + ovs_port = None + ovs_ports = ovs_cmd('ovs-vsctl', ['list-ports', bridge])[0].rstrip() + for p in ovs_ports.split('\n'): + if p.startswith('patch-provnet-'): + ovs_port = p + if not ovs_port: + return + ovs_ofport = ovs_cmd( + 'ovs-vsctl', ['get', 'Interface', ovs_port, 'ofport'] + )[0].rstrip() + + flow = ("{},ip,in_port={},dl_src:{}".format( + cookie_id, ovs_ofport, mac)) + ovs_cmd('ovs-ofctl', ['del-flows', bridge, flow]) + + flow_v6 = ("{},ipv6,in_port={},dl_src:{}".format(cookie_id, ovs_ofport, + mac)) + ovs_cmd('ovs-ofctl', ['del-flows', bridge, flow_v6]) + + +def remove_evpn_network_ovs_flow(bridge, cookie, mac, net): + cookie_id = ("cookie={}/-1").format(cookie) + + ovs_port = None + ovs_ports = ovs_cmd('ovs-vsctl', ['list-ports', bridge])[0].rstrip() + for p in ovs_ports.split('\n'): + if p.startswith('patch-provnet-'): + ovs_port = p + if not ovs_port: + return + ovs_ofport = ovs_cmd( + 'ovs-vsctl', ['get', 'Interface', ovs_port, 'ofport'] + )[0].rstrip() + + ip_version = linux_net.get_ip_version(net) + if ip_version == constants.IP_VERSION_6: + flow = ("{},ipv6,in_port={},dl_src:{},ipv6_src={}".format( + cookie_id, ovs_ofport, mac, net)) + else: + flow = ("{},ip,in_port={},dl_src:{},nw_src={}".format( + cookie_id, ovs_ofport, mac, net)) + ovs_cmd('ovs-ofctl', ['del-flows', bridge, flow]) + + +def ensure_default_ovs_flows(ovn_bridge_mappings, cookie): + cookie_id = ("cookie={}/-1").format(cookie) + for bridge in ovn_bridge_mappings: + ovs_port = ovs_cmd('ovs-vsctl', ['list-ports', bridge])[0].rstrip() + if not ovs_port: + continue + ovs_ofport = ovs_cmd( + 'ovs-vsctl', ['get', 'Interface', ovs_port, 'ofport'] + )[0].rstrip() + flow_filter = ('{},in_port={}').format(cookie_id, ovs_ofport) + current_flows = ovs_cmd( + 'ovs-ofctl', ['dump-flows', bridge, flow_filter] + )[0].split('\n')[1:-1] + if len(current_flows) == 1: + # assume the rule is the right one as it has the right cookie + # and in_port + continue + + with pyroute2.NDB() as ndb: + flow = ("cookie={},priority=900,ip,in_port={}," + "actions=mod_dl_dst:{},NORMAL".format( + cookie, ovs_ofport, + ndb.interfaces[bridge]['address'])) + flow_v6 = ("cookie={},priority=900,ipv6,in_port={}," + "actions=mod_dl_dst:{},NORMAL".format( + cookie, ovs_ofport, + ndb.interfaces[bridge]['address'])) + ovs_cmd('ovs-ofctl', ['add-flow', bridge, flow]) + ovs_cmd('ovs-ofctl', ['add-flow', bridge, flow_v6]) + + # Remove unneeded flows + port = 'in_port={}'.format(ovs_ofport) + current_flows = ovs_cmd( + 'ovs-ofctl', ['dump-flows', bridge, cookie_id] + )[0].split('\n')[1:-1] + for flow in current_flows: + if not flow or port in flow: + continue + in_port = flow.split("in_port=")[1].split(" ")[0] + del_flow = ('{},in_port={}').format(cookie_id, in_port) + ovs_cmd('ovs-ofctl', ['del-flows', bridge, del_flow]) + + +def add_device_to_ovs_bridge(device, bridge, vlan_tag=None): + if vlan_tag: + tag = "tag={}".format(vlan_tag) + ovs_cmd('ovs-vsctl', ['--may-exist', 'add-port', bridge, device, tag]) + else: + ovs_cmd('ovs-vsctl', ['--may-exist', 'add-port', bridge, device]) + + +def del_device_from_ovs_bridge(device, bridge=None): + if bridge: + ovs_cmd('ovs-vsctl', ['--if-exists', 'del-port', bridge, device]) + else: + ovs_cmd('ovs-vsctl', ['--if-exists', 'del-port', device]) + + +def get_bridge_flows_by_cookie(bridge, cookie): + cookie_id = ("cookie={}/-1").format(cookie) + return ovs_cmd('ovs-ofctl', + ['dump-flows', bridge, cookie_id])[0].split('\n')[1:-1] + + +def get_device_port_at_ovs(device): + return ovs_cmd( + 'ovs-vsctl', ['get', 'Interface', device, 'ofport'])[0].rstrip() + + +def del_flow(flow, bridge, cookie): + cookie_id = ("cookie={}/-1").format(cookie) + f = '{},priority{}'.format( + cookie_id, flow.split(' actions')[0].split(' priority')[1]) + ovs_cmd('ovs-ofctl', ['--strict', 'del-flows', bridge, f]) + + +def get_flow_info(flow): + # example: + # cookie=0x3e7, duration=85.005s, table=0, n_packets=0, + # n_bytes=0, idle_age=65534, priority=1000,ip,in_port=1 + # nw_src=20.0.0.0/24 actions=mod_dl_dst:1a:bd:c3:dc:6a:4c, + # output:5 + flow_mac = flow_port = flow_nw_src = flow_ipv6_src = None + try: + flow_mac = flow.split('dl_src=')[1].split(',')[0] + flow_port = flow.split('output:')[1].split(',')[0] + except (IndexError, TypeError): + pass + flow_nw = flow.split('nw_src=') + if len(flow_nw) == 2: + flow_nw_src = flow_nw[1].split(' ')[0] + flow_ipv6 = flow.split('ipv6_src=') + if len(flow_ipv6) == 2: + flow_ipv6_src = flow_ipv6[1].split(' ')[0] + + return {'mac': flow_mac, 'port': flow_port, 'nw_src': flow_nw_src, + 'ipv6_src': flow_ipv6_src} + + +class OvsIdl(object): + def start(self, connection_string): + helper = idlutils.get_schema_helper(connection_string, + 'Open_vSwitch') + tables = ('Open_vSwitch', 'Bridge', 'Port', 'Interface') + for table in tables: + helper.register_table(table) + ovs_idl = idl.Idl(connection_string, helper) + ovs_idl._session.reconnect.set_probe_interval(60000) + conn = connection.Connection( + ovs_idl, timeout=180) + self.idl_ovs = idl_ovs.OvsdbIdl(conn) + + def get_own_chassis_name(self): + """Return the external_ids:system-id value of the Open_vSwitch table. + + As long as ovn-controller is running on this node, the key is + guaranteed to exist and will include the chassis name. + """ + ext_ids = self.idl_ovs.db_get( + 'Open_vSwitch', '.', 'external_ids').execute() + return ext_ids['system-id'] + + def get_ovn_remote(self): + """Return the external_ids:ovn-remote value of the Open_vSwitch table. + + """ + ext_ids = self.idl_ovs.db_get( + 'Open_vSwitch', '.', 'external_ids').execute() + return ext_ids['ovn-remote'] + + def get_ovn_bridge_mappings(self): + """Return the external_ids:ovn-bridge-mappings value of the Open_vSwitch table. + + """ + ext_ids = self.idl_ovs.db_get( + 'Open_vSwitch', '.', 'external_ids').execute() + try: + return ext_ids['ovn-bridge-mappings'].split(",") + except KeyError: + return [] diff --git a/ovn_bgp_agent/drivers/openstack/watchers/__init__.py b/ovn_bgp_agent/drivers/openstack/watchers/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ovn_bgp_agent/drivers/openstack/watchers/bgp_watcher.py b/ovn_bgp_agent/drivers/openstack/watchers/bgp_watcher.py new file mode 100644 index 00000000..836b244e --- /dev/null +++ b/ovn_bgp_agent/drivers/openstack/watchers/bgp_watcher.py @@ -0,0 +1,270 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from ovn_bgp_agent import constants + +from ovsdbapp.backend.ovs_idl import event as row_event + +from oslo_concurrency import lockutils + +_SYNC_STATE_LOCK = lockutils.ReaderWriterLock() + + +class PortBindingChassisEvent(row_event.RowEvent): + def __init__(self, bgp_agent, events): + self.agent = bgp_agent + table = 'Port_Binding' + super(PortBindingChassisEvent, self).__init__( + events, table, None) + self.event_name = self.__class__.__name__ + + +class PortBindingChassisCreatedEvent(PortBindingChassisEvent): + def __init__(self, bgp_agent): + events = (self.ROW_UPDATE,) + super(PortBindingChassisCreatedEvent, self).__init__( + bgp_agent, events) + + def match_fn(self, event, row, old): + try: + # single and dual-stack format + if (len(row.mac[0].split(' ')) != 2 and + len(row.mac[0].split(' ')) != 3): + return False + return (row.chassis[0].name == self.agent.chassis and + not old.chassis) + except (IndexError, AttributeError): + return False + + def run(self, event, row, old): + if row.type not in constants.OVN_VIF_PORT_TYPES: + return + with _SYNC_STATE_LOCK.read_lock(): + ips = [row.mac[0].split(' ')[1]] + # for dual-stack + if len(row.mac[0].split(' ')) == 3: + ips.append(row.mac[0].split(' ')[2]) + self.agent.expose_ip(ips, row) + + +class PortBindingChassisDeletedEvent(PortBindingChassisEvent): + def __init__(self, bgp_agent): + events = (self.ROW_UPDATE, self.ROW_DELETE,) + super(PortBindingChassisDeletedEvent, self).__init__( + bgp_agent, events) + + def match_fn(self, event, row, old): + try: + # single and dual-stack format + if (len(row.mac[0].split(' ')) != 2 and + len(row.mac[0].split(' ')) != 3): + return False + if event == self.ROW_UPDATE: + return (old.chassis[0].name == self.agent.chassis and + not row.chassis) + else: + if row.chassis[0].name == self.agent.chassis: + return True + except (IndexError, AttributeError): + return False + + def run(self, event, row, old): + if row.type not in constants.OVN_VIF_PORT_TYPES: + return + with _SYNC_STATE_LOCK.read_lock(): + ips = [row.mac[0].split(' ')[1]] + # for dual-stack + if len(row.mac[0].split(' ')) == 3: + ips.append(row.mac[0].split(' ')[2]) + self.agent.withdraw_ip(ips, row) + + +class FIPSetEvent(PortBindingChassisEvent): + def __init__(self, bgp_agent): + events = (self.ROW_UPDATE,) + super(FIPSetEvent, self).__init__( + bgp_agent, events) + + def match_fn(self, event, row, old): + try: + return (not row.chassis and + row.nat_addresses != old.nat_addresses and + not row.logical_port.startswith('lrp-')) + except (AttributeError): + return False + + def run(self, event, row, old): + if row.type != 'patch': + return + with _SYNC_STATE_LOCK.read_lock(): + for nat in row.nat_addresses: + if nat not in old.nat_addresses: + ip = nat.split(" ")[1] + port = nat.split(" ")[2].split("\"")[1] + self.agent.expose_ip([ip], row, associated_port=port) + + +class FIPUnsetEvent(PortBindingChassisEvent): + def __init__(self, bgp_agent): + events = (self.ROW_UPDATE,) + super(FIPUnsetEvent, self).__init__( + bgp_agent, events) + + def match_fn(self, event, row, old): + try: + return (not row.chassis and + row.nat_addresses != old.nat_addresses and + not row.logical_port.startswith('lrp-')) + except (AttributeError): + return False + + def run(self, event, row, old): + if row.type != 'patch': + return + with _SYNC_STATE_LOCK.read_lock(): + for nat in old.nat_addresses: + if nat not in row.nat_addresses: + ip = nat.split(" ")[1] + port = nat.split(" ")[2].split("\"")[1] + self.agent.withdraw_ip([ip], row, associated_port=port) + + +class SubnetRouterAttachedEvent(PortBindingChassisEvent): + def __init__(self, bgp_agent): + events = (self.ROW_CREATE,) + super(SubnetRouterAttachedEvent, self).__init__( + bgp_agent, events) + + def match_fn(self, event, row, old): + try: + # single and dual-stack format + if (len(row.mac[0].split(' ')) != 2 and + len(row.mac[0].split(' ')) != 3): + return False + return (not row.chassis and row.logical_port.startswith('lrp-')) + except (IndexError, AttributeError): + return False + + def run(self, event, row, old): + if row.type != 'patch': + return + with _SYNC_STATE_LOCK.read_lock(): + ip_address = row.mac[0].split(' ')[1] + self.agent.expose_subnet(ip_address, row) + + +class SubnetRouterDetachedEvent(PortBindingChassisEvent): + def __init__(self, bgp_agent): + events = (self.ROW_DELETE,) + super(SubnetRouterDetachedEvent, self).__init__( + bgp_agent, events) + + def match_fn(self, event, row, old): + try: + # single and dual-stack format + if (len(row.mac[0].split(' ')) != 2 and + len(row.mac[0].split(' ')) != 3): + return False + return (not row.chassis and row.logical_port.startswith('lrp-')) + except (IndexError, AttributeError): + return False + + def run(self, event, row, old): + if row.type != 'patch': + return + with _SYNC_STATE_LOCK.read_lock(): + ip_address = row.mac[0].split(' ')[1] + self.agent.withdraw_subnet(ip_address, row) + + +class TenantPortCreatedEvent(PortBindingChassisEvent): + def __init__(self, bgp_agent): + events = (self.ROW_UPDATE,) + super(TenantPortCreatedEvent, self).__init__( + bgp_agent, events) + + def match_fn(self, event, row, old): + try: + # single and dual-stack format + if (len(row.mac[0].split(' ')) != 2 and + len(row.mac[0].split(' ')) != 3): + return False + return (not old.chassis and + self.agent.ovn_local_lrps != []) + except (IndexError, AttributeError): + return False + + def run(self, event, row, old): + if row.type != constants.OVN_VM_VIF_PORT_TYPE and row.type != constants.OVN_VIRTUAL_VIF_PORT_TYPE: + return + with _SYNC_STATE_LOCK.read_lock(): + ips = [row.mac[0].split(' ')[1]] + # for dual-stack + if len(row.mac[0].split(' ')) == 3: + ips.append(row.mac[0].split(' ')[2]) + self.agent.expose_remote_ip(ips, row) + + +class TenantPortDeletedEvent(PortBindingChassisEvent): + def __init__(self, bgp_agent): + events = (self.ROW_DELETE,) + super(TenantPortDeletedEvent, self).__init__( + bgp_agent, events) + + def match_fn(self, event, row, old): + try: + # single and dual-stack format + if (len(row.mac[0].split(' ')) != 2 and + len(row.mac[0].split(' ')) != 3): + return False + return (self.agent.ovn_local_lrps != []) + except (IndexError, AttributeError): + return False + + def run(self, event, row, old): + if row.type != constants.OVN_VM_VIF_PORT_TYPE and row.type != constants.OVN_VIRTUAL_VIF_PORT_TYPE: + return + with _SYNC_STATE_LOCK.read_lock(): + ips = [row.mac[0].split(' ')[1]] + # for dual-stack + if len(row.mac[0].split(' ')) == 3: + ips.append(row.mac[0].split(' ')[2]) + self.agent.withdraw_remote_ip(ips, row) + + +class ChassisCreateEventBase(row_event.RowEvent): + table = None + + def __init__(self, bgp_agent): + self.agent = bgp_agent + self.first_time = True + events = (self.ROW_CREATE,) + super(ChassisCreateEventBase, self).__init__( + events, self.table, (('name', '=', self.agent.chassis),)) + self.event_name = self.__class__.__name__ + + def run(self, event, row, old): + if self.first_time: + self.first_time = False + else: + print("Connection to OVSDB established, doing a full sync") + self.agent.sync() + + +class ChassisCreateEvent(ChassisCreateEventBase): + table = 'Chassis' + + +class ChassisPrivateCreateEvent(ChassisCreateEventBase): + table = 'Chassis_Private' diff --git a/ovn_bgp_agent/tests/unit/__init__.py b/ovn_bgp_agent/tests/unit/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ovn_bgp_agent/tests/unit/cmd/__init__.py b/ovn_bgp_agent/tests/unit/cmd/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ovn_bgp_agent/tests/unit/cmd/test_agent.py b/ovn_bgp_agent/tests/unit/cmd/test_agent.py new file mode 100644 index 00000000..e7acf5b7 --- /dev/null +++ b/ovn_bgp_agent/tests/unit/cmd/test_agent.py @@ -0,0 +1,27 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from unittest import mock + +from ovn_bgp_agent.tests import base as test_base + + +class TestAgentCmd(test_base.TestCase): + @mock.patch('ovn_bgp_agent.agent.start') + def test_start(self, m_start): + from ovn_bgp_agent.cmd import agent # To make it import a mock. + agent.start() + + m_start.assert_called() diff --git a/ovn_bgp_agent/tests/unit/test_agent.py b/ovn_bgp_agent/tests/unit/test_agent.py new file mode 100644 index 00000000..55c4e81c --- /dev/null +++ b/ovn_bgp_agent/tests/unit/test_agent.py @@ -0,0 +1,38 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from unittest import mock + +from ovn_bgp_agent import agent +from ovn_bgp_agent.tests import base as test_base + + +class TestAgent(test_base.TestCase): + + @mock.patch('oslo_service.service.launch') + @mock.patch('ovn_bgp_agent.config.init') + @mock.patch('ovn_bgp_agent.config.setup_logging') + @mock.patch('ovn_bgp_agent.agent.BGPAgent') + def test_start(self, m_agent, m_setup_logging, + m_config_init, m_oslo_launch): + m_launcher = mock.Mock() + m_oslo_launch.return_value = m_launcher + + agent.start() + + m_config_init.assert_called() + m_setup_logging.assert_called() + m_agent.assert_called() + m_oslo_launch.assert_called() + m_launcher.wait.assert_called() diff --git a/ovn_bgp_agent/utils/__init__.py b/ovn_bgp_agent/utils/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ovn_bgp_agent/utils/linux_net.py b/ovn_bgp_agent/utils/linux_net.py new file mode 100644 index 00000000..bed841cc --- /dev/null +++ b/ovn_bgp_agent/utils/linux_net.py @@ -0,0 +1,684 @@ +# Copyright 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import ipaddress +import pyroute2 +import random +import re +import sys + +from pyroute2.netlink.rtnl import ndmsg +from socket import AF_INET +from socket import AF_INET6 + +from oslo_concurrency import processutils +from oslo_log import log as logging + +from ovn_bgp_agent import constants + +LOG = logging.getLogger(__name__) + + +def get_ip_version(ip): + return ipaddress.ip_address(ip.split('/')[0]).version + + +def get_interfaces(filter_out=[]): + with pyroute2.NDB() as ndb: + return [iface.ifname for iface in ndb.interfaces + if iface.ifname not in filter_out] + + +def get_interface_index(nic): + with pyroute2.NDB() as ndb: + return ndb.interfaces[nic]['index'] + + +def ensure_vrf(vrf_name, vrf_table): + with pyroute2.NDB() as ndb: + try: + with ndb.interfaces[vrf_name] as vrf: + if vrf['state'] != constants.LINK_UP: + vrf['state'] = constants.LINK_UP + except KeyError: + ndb.interfaces.create( + kind="vrf", ifname=vrf_name, vrf_table=int(vrf_table)).set( + 'state', constants.LINK_UP).commit() + + +def ensure_bridge(bridge_name): + with pyroute2.NDB() as ndb: + try: + with ndb.interfaces[bridge_name] as bridge: + if bridge['state'] != constants.LINK_UP: + bridge['state'] = constants.LINK_UP + except KeyError: + ndb.interfaces.create( + kind="bridge", ifname=bridge_name, br_stp_state=0).set( + 'state', constants.LINK_UP).commit() + + +def ensure_vxlan(vxlan_name, vni, lo_ip): + with pyroute2.NDB() as ndb: + try: + with ndb.interfaces[vxlan_name] as vxlan: + if vxlan['state'] != constants.LINK_UP: + vxlan['state'] = constants.LINK_UP + except KeyError: + # FIXME: Perhaps we need to set neigh_suppress on + ndb.interfaces.create( + kind="vxlan", ifname=vxlan_name, vxlan_id=int(vni), + vxlan_port=4789, vxlan_local=lo_ip, vxlan_learning=False).set( + 'state', constants.LINK_UP).commit() + + +def set_master_for_device(device, master): + with pyroute2.NDB() as ndb: + # Check if already associated to the master, and associate it if not + if (ndb.interfaces[device].get('master') != + ndb.interfaces[master]['index']): + with ndb.interfaces[device] as iface: + iface.set('master', ndb.interfaces[master]['index']) + + +def ensure_dummy_device(device): + with pyroute2.NDB() as ndb: + try: + with ndb.interfaces[device] as iface: + if iface['state'] != constants.LINK_UP: + iface['state'] = constants.LINK_UP + except KeyError: + ndb.interfaces.create( + kind="dummy", ifname=device).set('state', + constants.LINK_UP).commit() + + +def ensure_ovn_device(ovn_ifname, vrf_name): + ensure_dummy_device(ovn_ifname) + set_master_for_device(ovn_ifname, vrf_name) + + +def delete_device(device): + try: + with pyroute2.NDB() as ndb: + ndb.interfaces[device].remove().commit() + except KeyError: + LOG.debug("Interfaces %s already deleted.", device) + + +def ensure_routing_table_for_bridge(ovn_routing_tables, bridge): + # check a routing table with the bridge name exists on + # /etc/iproute2/rt_tables + regex = r'^[0-9]*[\s]*{}$'.format(bridge) + matching_table = [line.replace('\t', ' ') + for line in open('/etc/iproute2/rt_tables') + if re.findall(regex, line)] + if matching_table: + table_info = matching_table[0].strip().split() + ovn_routing_tables[table_info[1]] = int(table_info[0]) + LOG.debug("Found routing table for %s with: %s", bridge, + table_info) + # if not configured, add random number for the table + else: + LOG.debug("Routing table for bridge %s not configured " + "at /etc/iproute2/rt_tables", bridge) + regex = r'^[0-9]+[\s]*' + existing_routes = [int(line.replace('\t', ' ').split(' ')[0]) + for line in open('/etc/iproute2/rt_tables') + if re.findall(regex, line)] + # pick a number between 1 and 252 + try: + table_number = random.choice(list( + set([x for x in range(1, 253)]).difference( + set(existing_routes)))) + except IndexError: + LOG.error("No more routing tables available for bridge %s " + "at /etc/iproute2/rt_tables", bridge) + sys.exit() + + with open('/etc/iproute2/rt_tables', 'a') as rt_tables: + rt_tables.write('{} {}\n'.format(table_number, bridge)) + + ovn_routing_tables[bridge] = int(table_number) + LOG.debug("Added routing table for %s with number: %s", bridge, + table_number) + + # add default route on that table if it does not exist + extra_routes = [] + + with pyroute2.NDB() as ndb: + table_route_dsts = set([r.dst for r in ndb.routes.summary().filter( + table=ovn_routing_tables[bridge])]) + if not table_route_dsts: + ndb.routes.create(dst='default', + oif=ndb.interfaces[bridge]['index'], + table=ovn_routing_tables[bridge], + scope=253, + proto=3).commit() + ndb.routes.create(dst='default', + oif=ndb.interfaces[bridge]['index'], + table=ovn_routing_tables[bridge], + family=AF_INET6, + proto=3).commit() + else: + route_missing = True + route6_missing = True + for dst in table_route_dsts: + if not dst: # default route + try: + route = ndb.routes[ + {'table': ovn_routing_tables[bridge], + 'dst': '', + 'family': AF_INET}] + if (bridge == + ndb.interfaces[{'index': route['oif']}][ + 'ifname']): + route_missing = False + else: + extra_routes.append(route) + except KeyError: + pass # no ipv4 default rule + try: + route_6 = ndb.routes[ + {'table': ovn_routing_tables[bridge], + 'dst': '', + 'family': AF_INET6}] + if (bridge == + ndb.interfaces[{'index': route_6['oif']}][ + 'ifname']): + route6_missing = False + else: + extra_routes.append(route_6) + except KeyError: + pass # no ipv6 default rule + else: + extra_routes.append( + ndb.routes[{'table': ovn_routing_tables[bridge], + 'dst': dst}] + ) + + if route_missing: + ndb.routes.create(dst='default', + oif=ndb.interfaces[bridge]['index'], + table=ovn_routing_tables[bridge], + scope=253, + proto=3).commit() + if route6_missing: + ndb.routes.create(dst='default', + oif=ndb.interfaces[bridge]['index'], + table=ovn_routing_tables[bridge], + family=AF_INET6, + proto=3).commit() + return extra_routes + + +def ensure_vlan_device_for_network(bridge, vlan_tag): + vlan_device_name = '{}.{}'.format(bridge, vlan_tag) + + with pyroute2.NDB() as ndb: + try: + with ndb.interfaces[vlan_device_name] as iface: + if iface['state'] != constants.LINK_UP: + iface['state'] = constants.LINK_UP + except KeyError: + ndb.interfaces.create( + kind="vlan", ifname=vlan_device_name, vlan_id=vlan_tag, + link=ndb.interfaces[bridge]['index']).set('state', + constants.LINK_UP).commit() + + ipv4_flag = "net.ipv4.conf.{}/{}.proxy_arp".format(bridge, vlan_tag) + _set_kernel_flag(ipv4_flag, 1) + ipv6_flag = "net.ipv6.conf.{}/{}.proxy_ndp".format(bridge, vlan_tag) + _set_kernel_flag(ipv6_flag, 1) + + +def delete_vlan_device_for_network(bridge, vlan_tag): + vlan_device_name = '{}.{}'.format(bridge, vlan_tag) + delete_device(vlan_device_name) + + +def _set_kernel_flag(flag, value): + command = ["sysctl", "-w", "{}={}".format(flag, value)] + try: + return processutils.execute(*command, run_as_root=True) + except Exception as e: + LOG.error("Unable to execute %s. Exception: %s", command, e) + raise + + +def get_exposed_ips(nic): + exposed_ips = [] + with pyroute2.NDB() as ndb: + exposed_ips = [ip.address + for ip in ndb.interfaces[nic].ipaddr.summary() + if ip.prefixlen == 32 or ip.prefixlen == 128] + return exposed_ips + + +def get_nic_ip(nic, ip_version): + prefix = 32 + if ip_version == constants.IP_VERSION_6: + prefix = 128 + exposed_ips = [] + with pyroute2.NDB() as ndb: + exposed_ips = [ip.address + for ip in ndb.interfaces[nic].ipaddr.summary().filter( + prefixlen=prefix)] + return exposed_ips + + +def get_exposed_ips_on_network(nic, network): + exposed_ips = [] + with pyroute2.NDB() as ndb: + exposed_ips = [ip.address + for ip in ndb.interfaces[nic].ipaddr.summary() + if ((ip.prefixlen == 32 or ip.prefixlen == 128) and + ipaddress.ip_address(ip.address) in network)] + return exposed_ips + + +def get_ovn_ip_rules(routing_table): + # get the rules pointing to ovn bridges + ovn_ip_rules = {} + with pyroute2.NDB() as ndb: + rules_info = [(rule.table, + "{}/{}".format(rule.dst, rule.dst_len), + rule.family) for rule in ndb.rules.dump() + if rule.table in routing_table] + for table, dst, family in rules_info: + ovn_ip_rules[dst] = {'table': table, 'family': family} + return ovn_ip_rules + + +def delete_exposed_ips(ips, nic): + with pyroute2.NDB() as ndb: + for ip in ips: + address = '{}/32'.format(ip) + if get_ip_version(ip) == constants.IP_VERSION_6: + address = '{}/128'.format(ip) + try: + ndb.interfaces[nic].ipaddr[address].remove().commit() + except KeyError: + LOG.debug("IP address {} already removed from nic {}.".format( + ip, nic)) + + +def delete_ip_rules(ip_rules): + with pyroute2.NDB() as ndb: + for rule_ip, rule_info in ip_rules.items(): + rule = {'dst': rule_ip.split("/")[0], + 'dst_len': rule_ip.split("/")[1], + 'table': rule_info['table'], + 'family': rule_info['family']} + try: + with ndb.rules[rule] as r: + r.remove() + except KeyError: + LOG.debug("Rule {} already deleted".format(rule)) + except pyroute2.netlink.exceptions.NetlinkError: + # FIXME: There is a issue with NDB and ip rules deletion: + # https://github.com/svinota/pyroute2/issues/771 + LOG.debug("This should not happen, skipping: NetlinkError " + "deleting rule %s", rule) + + +def delete_bridge_ip_routes(routing_tables, routing_tables_routes, + extra_routes): + with pyroute2.NDB() as ndb: + for bridge, routes_info in routing_tables_routes.items(): + if not extra_routes[bridge]: + continue + for route_info in routes_info: + oif = ndb.interfaces[bridge]['index'] + if route_info['vlan']: + vlan_device_name = '{}.{}'.format(bridge, + route_info['vlan']) + oif = ndb.interfaces[vlan_device_name]['index'] + if 'gateway' in route_info['route'].keys(): # subnet route + possible_matchings = [ + r for r in extra_routes[bridge] + if (r['dst'] == route_info['route']['dst'] and + r['dst_len'] == route_info['route']['dst_len'] and + r['gateway'] == route_info['route']['gateway'])] + else: # cr-lrp + possible_matchings = [ + r for r in extra_routes[bridge] + if (r['dst'] == route_info['route']['dst'] and + r['dst_len'] == route_info['route']['dst_len'] and + r['oif'] == oif)] + for r in possible_matchings: + extra_routes[bridge].remove(r) + + for bridge, routes in extra_routes.items(): + for route in routes: + r_info = {'dst': route['dst'], + 'dst_len': route['dst_len'], + 'family': route['family'], + 'oif': route['oif'], + 'gateway': route['gateway'], + 'table': routing_tables[bridge]} + try: + with ndb.routes[r_info] as r: + r.remove() + except KeyError: + LOG.debug("Route already deleted: {}".format(route)) + + +def delete_routes_from_table(table): + with pyroute2.NDB() as ndb: + # FIXME: problem in pyroute2 removing routes with local (254) scope + table_routes = [r for r in ndb.routes.dump().filter(table=table) + if r.scope != 254 and r.proto != 186] + for route in table_routes: + try: + with ndb.routes[route] as r: + r.remove() + except KeyError: + LOG.debug("Route already deleted: %s", route) + + +def get_routes_on_tables(table_ids): + with pyroute2.NDB() as ndb: + # NOTE: skip bgp routes (proto 186) + return [r for r in ndb.routes.dump() + if r.table in table_ids and r.dst != '' and r.proto != 186] + + +def delete_ip_routes(routes): + with pyroute2.NDB() as ndb: + for route in routes: + r_info = {'dst': route['dst'], + 'dst_len': route['dst_len'], + 'family': route['family'], + 'oif': route['oif'], + 'gateway': route['gateway'], + 'table': route['table']} + try: + with ndb.routes[r_info] as r: + r.remove() + except KeyError: + LOG.debug("Route already deleted: %s", route) + + +def add_ndp_proxy(ip, dev, vlan=None): + # FIXME(ltomasbo): This should use pyroute instead but I didn't find + # out how + net_ip = str(ipaddress.IPv6Network(ip, strict=False).network_address) + dev_name = dev + if vlan: + dev_name = "{}.{}".format(dev, vlan) + command = ["ip", "-6", "nei", "add", "proxy", net_ip, "dev", dev_name] + try: + return processutils.execute(*command, run_as_root=True) + except Exception as e: + LOG.error("Unable to execute %s. Exception: %s", command, e) + raise + + +def del_ndp_proxy(ip, dev, vlan=None): + # FIXME(ltomasbo): This should use pyroute instead but I didn't find + # out how + net_ip = str(ipaddress.IPv6Network(ip, strict=False).network_address) + dev_name = dev + if vlan: + dev_name = "{}.{}".format(dev, vlan) + command = ["ip", "-6", "nei", "del", "proxy", net_ip, "dev", dev_name] + try: + return processutils.execute(*command, run_as_root=True) + except Exception as e: + if "No such file or directory" in e.stderr: + # Already deleted + return + LOG.error("Unable to execute %s. Exception: %s", command, e) + raise + + +def add_ips_to_dev(nic, ips, clear_local_route_at_table=False): + with pyroute2.NDB() as ndb: + try: + with ndb.interfaces[nic] as iface: + for ip in ips: + address = '{}/32'.format(ip) + if get_ip_version(ip) == constants.IP_VERSION_6: + address = '{}/128'.format(ip) + iface.add_ip(address) + except KeyError: + # NDB raises KeyError: 'object exists' + # if the ip is already added + pass + + if clear_local_route_at_table: + with pyroute2.NDB() as ndb: + for ip in ips: + route = {'table': clear_local_route_at_table, + 'proto': 2, + 'scope': 254, + 'dst': ip} + try: + with ndb.routes[route] as r: + r.remove() + except (KeyError, ValueError): + LOG.debug("Local route already deleted: %s", route) + + +def del_ips_from_dev(nic, ips): + with pyroute2.NDB() as ndb: + with ndb.interfaces[nic] as iface: + for ip in ips: + address = '{}/32'.format(ip) + if get_ip_version(ip) == constants.IP_VERSION_6: + address = '{}/128'.format(ip) + iface.del_ip(address) + + +def add_ip_rule(ip, table, dev=None, lladdr=None): + ip_version = get_ip_version(ip) + ip_info = ip.split("/") + + if len(ip_info) == 1: + rule = {'dst': ip_info[0], 'table': table, 'dst_len': 32} + if ip_version == constants.IP_VERSION_6: + rule['dst_len'] = 128 + rule['family'] = AF_INET6 + elif len(ip_info) == 2: + rule = {'dst': ip_info[0], 'table': table, 'dst_len': int(ip_info[1])} + if ip_version == constants.IP_VERSION_6: + rule['family'] = AF_INET6 + else: + LOG.error("Invalid ip: %s", ip) + return + + with pyroute2.NDB() as ndb: + try: + ndb.rules[rule] + except KeyError: + LOG.debug("Creating ip rule with: %s", rule) + ndb.rules.create(rule).commit() + + # FIXME: There is no support for creating neighbours in NDB + # So we are using iproute here + if lladdr: + ip_version = get_ip_version(ip) + with pyroute2.IPRoute() as iproute: + # This is doing something like: + # sudo ip nei replace 172.24.4.69 + # lladdr fa:16:3e:d3:5d:7b dev br-ex nud permanent + network_bridge_if = iproute.link_lookup(ifname=dev)[0] + if ip_version == constants.IP_VERSION_6: + iproute.neigh('set', + dst=ip, + lladdr=lladdr, + family=AF_INET6, + ifindex=network_bridge_if, + state=ndmsg.states['permanent']) + else: + iproute.neigh('set', + dst=ip, + lladdr=lladdr, + ifindex=network_bridge_if, + state=ndmsg.states['permanent']) + + +def del_ip_rule(ip, table, dev=None, lladdr=None): + ip_version = get_ip_version(ip) + ip_info = ip.split("/") + + if len(ip_info) == 1: + rule = {'dst': ip_info[0], 'table': table, 'dst_len': 32} + if ip_version == constants.IP_VERSION_6: + rule['dst_len'] = 128 + rule['family'] = AF_INET6 + elif len(ip_info) == 2: + rule = {'dst': ip_info[0], 'table': table, 'dst_len': int(ip_info[1])} + if ip_version == constants.IP_VERSION_6: + rule['family'] = AF_INET6 + else: + LOG.error("Invalid ip: {}".format(ip)) + return + with pyroute2.NDB() as ndb: + try: + ndb.rules[rule].remove().commit() + LOG.debug("Deleting ip rule with: %s", rule) + except KeyError: + LOG.debug("Rule already deleted: %s", rule) + + # FIXME: There is no support for deleting neighbours in NDB + # So we are using iproute here + if lladdr: + ip_version = get_ip_version(ip) + with pyroute2.IPRoute() as iproute: + # This is doing something like: + # sudo ip nei del 172.24.4.69 + # lladdr fa:16:3e:d3:5d:7b dev br-ex nud permanent + network_bridge_if = iproute.link_lookup( + ifname=dev)[0] + if ip_version == constants.IP_VERSION_6: + iproute.neigh('del', + dst=ip.split("/")[0], + lladdr=lladdr, + family=AF_INET6, + ifindex=network_bridge_if, + state=ndmsg.states['permanent']) + else: + iproute.neigh('del', + dst=ip.split("/")[0], + lladdr=lladdr, + ifindex=network_bridge_if, + state=ndmsg.states['permanent']) + + +def add_unreachable_route(vrf_name): + # FIXME: This should use pyroute instead but I didn't find + # out how + for ip_version in [-4, -6]: + command = ["ip", ip_version, "route", "add", "vrf", vrf_name, + "unreachable", "default", "metric", "4278198272"] + try: + return processutils.execute(*command, run_as_root=True) + except Exception as e: + if "RTNETLINK answers: File exists" in e.stderr: + continue + LOG.error("Unable to execute %s. Exception: %s", command, e) + raise + + +def add_ip_route(ovn_routing_tables_routes, ip_address, route_table, dev, + vlan=None, mask=None, via=None): + net_ip = ip_address + if not mask: # default /32 or /128 + if get_ip_version(ip_address) == constants.IP_VERSION_6: + mask = 128 + else: + mask = 32 + else: + ip = '{}/{}'.format(ip_address, mask) + if get_ip_version(ip_address) == constants.IP_VERSION_6: + net_ip = '{}'.format(ipaddress.IPv6Network( + ip, strict=False).network_address) + else: + net_ip = '{}'.format(ipaddress.IPv4Network( + ip, strict=False).network_address) + + with pyroute2.NDB() as ndb: + if vlan: + oif_name = '{}.{}'.format(dev, vlan) + oif = ndb.interfaces[oif_name]['index'] + else: + oif = ndb.interfaces[dev]['index'] + + route = {'dst': net_ip, 'dst_len': int(mask), 'oif': oif, + 'table': int(route_table), 'proto': 3} + if via: + route['gateway'] = via + route['scope'] = 0 + else: + route['scope'] = 253 + if get_ip_version(net_ip) == constants.IP_VERSION_6: + route['family'] = AF_INET6 + del route['scope'] + + with pyroute2.NDB() as ndb: + try: + with ndb.routes[route] as r: + LOG.debug("Route already existing: %s", r) + except KeyError: + ndb.routes.create(route).commit() + LOG.debug("Route created at table %s: %s", route_table, route) + route_info = {'vlan': vlan, 'route': route} + ovn_routing_tables_routes.setdefault(dev, []).append(route_info) + + +def del_ip_route(ovn_routing_tables_routes, ip_address, route_table, dev, + vlan=None, mask=None, via=None): + net_ip = ip_address + if not mask: # default /32 or /128 + if get_ip_version(ip_address) == constants.IP_VERSION_6: + mask = 128 + else: + mask = 32 + else: + ip = '{}/{}'.format(ip_address, mask) + if get_ip_version(ip_address) == constants.IP_VERSION_6: + net_ip = '{}'.format(ipaddress.IPv6Network( + ip, strict=False).network_address) + else: + net_ip = '{}'.format(ipaddress.IPv4Network( + ip, strict=False).network_address) + + with pyroute2.NDB() as ndb: + if vlan: + oif_name = '{}.{}'.format(dev, vlan) + oif = ndb.interfaces[oif_name]['index'] + else: + oif = ndb.interfaces[dev]['index'] + + route = {'dst': net_ip, 'dst_len': int(mask), 'oif': oif, + 'table': int(route_table), 'proto': 3} + if via: + route['gateway'] = via + route['scope'] = 0 + else: + route['scope'] = 253 + if get_ip_version(net_ip) == constants.IP_VERSION_6: + route['family'] = AF_INET6 + + with pyroute2.NDB() as ndb: + try: + with ndb.routes[route] as r: + r.remove() + LOG.debug("Route deleted at table %s: %s", route_table, route) + route_info = {'vlan': vlan, 'route': route} + ovn_routing_tables_routes[dev].remove(route_info) + except (KeyError, ValueError): + LOG.debug("Route already deleted: %s", route) diff --git a/requirements.txt b/requirements.txt index 1d18dd3d..b6ab2212 100644 --- a/requirements.txt +++ b/requirements.txt @@ -3,3 +3,14 @@ # process, which may cause wedges in the gate later. pbr>=2.0 # Apache-2.0 + +Jinja2>=2.10 # BSD License (3 clause) +oslo.concurrency>=3.26.0 # Apache-2.0 +oslo.config>=6.1.0 # Apache-2.0 +oslo.log>=3.36.0 # Apache-2.0 +oslo.service>=1.40.2 # Apache-2.0 +ovs>=2.8.0 # Apache-2.0 +ovsdbapp>=1.4.0 # Apache-2.0 +pyroute2>=0.6.4;sys_platform!='win32' # Apache-2.0 (+ dual licensed GPL2) +stevedore>=1.20.0 # Apache-2.0 + diff --git a/setup.cfg b/setup.cfg index 25bab596..c21a2cef 100644 --- a/setup.cfg +++ b/setup.cfg @@ -23,3 +23,10 @@ classifier = [files] packages = ovn_bgp_agent + +[entry_points] +console_scripts = + bgp-agent = ovn_bgp_agent.cmd.agent:start + +ovn_bgp_agent.drivers = + osp_ovn_bgp_driver = ovn_bgp_agent.drivers.openstack.ovn_bgp_driver:OSPOVNBGPDriver diff --git a/tox.ini b/tox.ini index 634f1ae8..9406555e 100644 --- a/tox.ini +++ b/tox.ini @@ -53,6 +53,6 @@ commands = oslo_debug_helper {posargs} # E123, E125 skipped as they are invalid PEP-8. show-source = True -ignore = E123,E125 +ignore = E123,E125,W504 builtins = _ exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build