Tungsten Fabric入门宝典丨首次启动和运行指南



  • 作者:Tatsuya Naganawa 译者:TF编译组

    Tungsten Fabric入门宝典系列文章,来自技术大牛倾囊相授的实践经验,由TF中文社区为您编译呈现,旨在帮助新手深入理解TF的运行、安装、集成、调试等全流程。如果您有相关经验或疑问,欢迎与我们互动,并与社区极客们进一步交流。更多TF技术文章,请点击TF中文社区公众号底部按钮>学习>文章合集。

    7f4cb3d0-d84f-44cc-84f7-7e120f791971-image.png

    两年来,我在Tungsten Fabric的旅程中学到了不少知识,接下来,我会用几篇文章来简要介绍一下。

    1为什么要用Tungsten Fabric?

    首先,让我们来看一个重要的问题,SDN/Neutron/CNI都有很多不错的实现案例,为什么还要再尝试另一个呢?据我所知,Tungsten Fabric具有两个关键的差异化特征,这使它变得如此与众不同。

    I. 与ASIC的互操作性

    尽管有很多技术使Linux软件成为生产路由器/交换机的理想选择,但ASIC仍然是这一行业的重要组成部分。为了与它们进行互操作,SDN平台需要使用路由协议,例如BGP或OVSDB。

    许多服务提供商和云服务商使用VRF来终结和分离每个客户的网络连接,这使得路由器和SDN之间的连接变得很复杂。

    • 通常,它们之间可以使用VLAN,但是SDN平台上的终结点可能会成为瓶颈

    • 此外,每个SDN终结点(类似于OpenStack中的网络节点)需要为每个客户进行单独的配置,这使配置更加复杂

    借助MP-BGP这个已经成熟实现的协议,Tungsten Fabric解决了上述问题,该协议允许路由器上的每个VRF将报文直接发送到vRouter,这些vRouter服务于每个客户的应用程序。此功能允许基于控制平面(而不是数据平面),对具有每个客户独立网络的计算节点进行水平扩展,并使其更加直观。

    II. 可扩展性

    由于报文是从路由器直接发送到vRouter的,因此不需要网络节点,这使得Tungsten Fabric在数据平面上具有更大的可扩展性。

    另外,从控制平面的角度来看,Tungsten Fabric具有一个名为“路由目标过滤器(route target filtering)”(https://tools.ietf.org/html/rfc4684)的有意思的功能。

    • 此功能在MP-BGP中很常见,其他路由器也具有这个功能
    • 这个功能意味着,如果vRouter没有该路由目标(route target)的前缀,那么控制平面接收到它时会丢弃前缀

    在云服务中,客户只使用云服务商的数据中心的有限功能,并且不同客户使用不同的路由目标,因此vRouter和控制器不需要知道所有前缀。路由目标过滤功能使这种行为成为可能,并大大减少了每个vRouter以及每个控制器(如果在它们之间使用RR的话)需要考虑的前缀数量,这使得该控制平面更具可扩展性。

    对于私有云或托管云来说,将其与安全策略、网络策略/逻辑路由器(与AWS中的VPC peerling或transit-gateway类似)等其它功能结合在一起,这将是VPC基础架构(类似于AWS/Azure/GCP VPC/vnet)的一个不错的选择,并且使其成为一个非常值得一试的有趣平台。

    2Tungsten Fabric,启动并运行

    在第一次尝试使用Tungsten Fabric时,即使你已经熟悉其他CNI的部署,还是建议使用ansible-deployer(https://github.com/Juniper/contrail-ansible-deployer),因为Tungsten Fabric使用了很多种没有集成在Vanilla Linux中的工具。因此,建议首先尝试可以良好运行的设置,以查看新功能,然后再集成其它系统。

    不幸的是,许多Tungsten Fabric的repos都与rawhide类似,并且在某些情况下已经失去了依赖性。

    因此,我选择了一种组合,是我认为通常都能正常工作并且足够稳定,可以用来尝试大多数功能。

    要尝试此操作,你需要两台服务器,一台用于K8s主服务器,另一台用于K8s节点。K8s主服务器需要至少有2个vCPU、8GB内存和8GB磁盘空间。K8s节点服务器需要1个vCPU、4GB内存和8GB磁盘空间。

    • 我个人通常在AWS的ap-northeast-1区域中使用ami-3185744e(CentOS7.5, login-id: centos),t2.large的规格
    • 由于在我看来,与Tungsten Fabric进行OpenStack和vCenter集成比使用Kubernetes进行集成要复杂得多,因此即使你不需要容器支持,我还是建议首先尝试此设置
    • 安装过程,需要有互联网连接
    ##all the commands are typed at k8s master node
    sudo yum -y remove PyYAML python-requests
    sudo yum -y install git
    sudo easy_install pip
    sudo pip install PyYAML requests ansible==2.7.15
    ssh-keygen
    cd .ssh/
    cat id_rsa.pub >> authorized_keys
    ssh-copy-id root@(k8s node's ip) ## or manually register id_rsa.pub to authorized_keys
    cd
    git clone -b R5.1 http://github.com/Juniper/contrail-ansible-deployer
    cd contrail-ansible-deployer
    vi config/instances.yaml
    (replace contents with this)
    provider_config:
      bms:
       ssh_user: root
       ssh_public_key: /root/.ssh/id_rsa.pub
       ssh_private_key: /root/.ssh/id_rsa
       domainsuffix: local
       ntpserver: 0.centos.pool.ntp.org
    instances:
      bms1:
       provider: bms
       roles:
          config_database:
          config:
          control:
          analytics:
          analytics_database:
          webui:
          k8s_master:
          kubemanager:
       ip: 172.31.14.47 ## k8s master's ip
      bms2:
       provider: bms
       roles:
         vrouter:
         k8s_node:
       ip: 172.31.41.236 ## k8s node's ip
    contrail_configuration:
      CONTRAIL_CONTAINER_TAG: r5.1
      KUBERNETES_CLUSTER_PROJECT: {}
      JVM_EXTRA_OPTS: "-Xms128m -Xmx1g"
    global_configuration:
      CONTAINER_REGISTRY: tungstenfabric
    
    ansible-playbook -e orchestrator=kubernetes -i inventory/ playbooks/configure_instances.yml
     - it takes about 10 minutes
    ansible-playbook -e orchestrator=kubernetes -i inventory/ playbooks/install_k8s.yml
     - it takes about 5 minutes
    ansible-playbook -e orchestrator=kubernetes -i inventory/ playbooks/install_contrail.yml
     - it takes about 20 minutes
    

    需要注意的一点是,使用受支持的内核版本是一项相当严格的要求,因为Tungsten Fabric将自己的内核模块(vrouter.ko)用于其数据平面。我尝试了CentOS7.5、7.6,Ubuntu Xenial,并发现它也运行良好(对于Ubuntu Bionic,需要进行一些修改),但如果是第一次尝试,我建议使用特定的AMI ID,因为由于不能正常工作而进行调试和排错不是一件容易的事。

    如果所有“剧本”(playbooks)都运行良好,可以先输入内容,

    contrail-status
    

    它负责检查一切是否正常。

    [root@ip-172-31-14-47 contrail-ansible-deployer]# contrail-status 
    Pod              Service         Original Name                          State    Status             
                     redis           contrail-external-redis                running  Up 5 minutes       
    analytics        alarm-gen       contrail-analytics-alarm-gen           running  Up 2 minutes       
    analytics        api             contrail-analytics-api                 running  Up 2 minutes       
    analytics        collector       contrail-analytics-collector           running  Up 2 minutes       
    analytics        nodemgr         contrail-nodemgr                       running  Up 2 minutes       
    analytics        query-engine    contrail-analytics-query-engine        running  Up 2 minutes       
    analytics        snmp-collector  contrail-analytics-snmp-collector      running  Up 2 minutes       
    analytics        topology        contrail-analytics-topology            running  Up 2 minutes       
    config           api             contrail-controller-config-api         running  Up 4 minutes       
    config           device-manager  contrail-controller-config-devicemgr   running  Up 3 minutes       
    config           nodemgr         contrail-nodemgr                       running  Up 4 minutes       
    config           schema          contrail-controller-config-schema      running  Up 4 minutes       
    config           svc-monitor     contrail-controller-config-svcmonitor  running  Up 4 minutes       
    config-database  cassandra       contrail-external-cassandra            running  Up 4 minutes       
    config-database  nodemgr         contrail-nodemgr                       running  Up 4 minutes       
    config-database  rabbitmq        contrail-external-rabbitmq             running  Up 4 minutes       
    config-database  zookeeper       contrail-external-zookeeper            running  Up 4 minutes       
    control          control         contrail-controller-control-control    running  Up 3 minutes       
    control          dns             contrail-controller-control-dns        running  Up 3 minutes       
    control          named           contrail-controller-control-named      running  Up 3 minutes       
    control          nodemgr         contrail-nodemgr                       running  Up 3 minutes       
    database         cassandra       contrail-external-cassandra            running  Up 2 minutes       
    database         kafka           contrail-external-kafka                running  Up 2 minutes       
    database         nodemgr         contrail-nodemgr                       running  Up 2 minutes       
    database         zookeeper       contrail-external-zookeeper            running  Up 2 minutes       
    kubernetes       kube-manager    contrail-kubernetes-kube-manager       running  Up About a minute  
    webui            job             contrail-controller-webui-job          running  Up 3 minutes       
    webui            web             contrail-controller-webui-web          running  Up 3 minutes       
    
    WARNING: container with original name 'contrail-external-redis' have Pod or Service empty. Pod: '' / Service: 'redis'. Please pass NODE_TYPE with pod name to container's env
    
    == Contrail control ==
    control: active
    nodemgr: active
    named: active
    dns: active
    
    == Contrail config-database ==
    nodemgr: initializing (Disk for DB is too low. )
    zookeeper: active
    rabbitmq: active
    cassandra: active
    
    == Contrail kubernetes ==
    kube-manager: active
    
    == Contrail database ==
    kafka: active
    nodemgr: initializing (Disk for DB is too low. )
    zookeeper: active
    cassandra: active
    
    == Contrail analytics ==
    snmp-collector: active
    query-engine: active
    api: active
    alarm-gen: active
    nodemgr: active
    collector: active
    topology: active
    
    == Contrail webui ==
    web: active
    job: active
    
    == Contrail config ==
    svc-monitor: active
    nodemgr: active
    device-manager: active
    api: active
    schema: active
    
    [root@ip-172-31-14-47 contrail-ansible-deployer]# 
    
    [root@ip-172-31-41-236 ~]# contrail-status 
    Pod      Service  Original Name           State    Status         
    vrouter  agent    contrail-vrouter-agent  running  Up 52 seconds  
    vrouter  nodemgr  contrail-nodemgr        running  Up 52 seconds  
    
    vrouter kernel module is PRESENT
    == Contrail vrouter ==
    nodemgr: active
    agent: active
    
    [root@ip-172-31-41-236 ~]#
    

    这应该表明大多数组件都处于“active”状态,除了以下的:

    nodemgr: initializing (Disk for DB is too low.)
    

    这是可以在demo设置中安全地忽略的问题。

    注意:这基本上表明使用率超过50%,对于Cassandra来说这是一个很重要的问题。

    如果一切正常,则可以尝试使用此命令查看Tungsten Fabric路由表的状态。

    pip install lxml prettytable
    git clone https://github.com/vcheny/contrail-introspect-cli.git
    ##or curl -O https://raw.githubusercontent.com/vcheny/contrail-introspect-cli/master/ist.py
    ./contrail-introspect-cli/ist.py ctr status
    ./contrail-introspect-cli/ist.py ctr nei ## similar to 'show bgp summary'
    ./contrail-introspect-cli/ist.py ctr route summary ## similar to 'show route summary'
    ./contrail-introspect-cli/ist.py ctr route tables ## show routing-tables
    ./contrail-introspect-cli/ist.py ctr route show ## similar to 'show route'
    
    
    [root@ip-172-31-14-47 contrail-ansible-deployer]# ./contrail-introspect-cli/ist.py ctr status
    module_id: contrail-control
    state: Functional
    description
    +-----------+-----------+---------------------+--------+----------------------------------+
    | type      | name      | server_addrs        | status | description                      |
    +-----------+-----------+---------------------+--------+----------------------------------+
    | Collector | n/a       |   172.31.14.47:8086 | Up     | Established                      |
    | Database  | Cassandra |   172.31.14.47:9041 | Up     | Established Cassandra connection |
    | Database  | RabbitMQ  |   172.31.14.47:5673 | Up     | RabbitMQ connection established  |
    +-----------+-----------+---------------------+--------+----------------------------------+
    
    [root@ip-172-31-14-47 contrail-ansible-deployer]# ./contrail-introspect-cli/ist.py ctr nei
    +--------------------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
    | peer                                 | peer_address  | peer_asn | encoding | peer_type | state       | send_state | flap_count | flap_time |
    +--------------------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
    | ip-172-31-41-236.ap-                 | 172.31.41.236 | 0        | XMPP     | internal  | Established | in sync    | 0          | n/a       |
    | northeast-1.compute.internal         |               |          |          |           |             |            |            |           |
    +--------------------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
    
    [root@ip-172-31-14-47 contrail-ansible-deployer]# ./contrail-introspect-cli/ist.py ctr route summary
    +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
    | name                                               | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
    +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
    | default-domain:default-                            | 0        | 0     | 0             | 0               | 0                |
    | project:__link_local__:__link_local__.inet.0       |          |       |               |                 |                  |
    | default-domain:default-project:default-virtual-    | 0        | 0     | 0             | 0               | 0                |
    | network:default-virtual-network.inet.0             |          |       |               |                 |                  |
    | inet.0                                             | 0        | 0     | 0             | 0               | 0                |
    | default-domain:default-project:ip-fabric:ip-       | 3        | 3     | 1             | 2               | 0                |
    | fabric.inet.0                                      |          |       |               |                 |                  |
    | default-domain:k8s-default:k8s-default-pod-network | 3        | 3     | 1             | 2               | 0                |
    | :k8s-default-pod-network.inet.0                    |          |       |               |                 |                  |
    | default-domain:k8s-default:k8s-default-service-    | 3        | 3     | 1             | 2               | 0                |
    | network:k8s-default-service-network.inet.0         |          |       |               |                 |                  |
    +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
    
    [root@ip-172-31-14-47 contrail-ansible-deployer]# ./contrail-introspect-cli/ist.py ctr route tables
    name: default-domain:default-project:__link_local__:__link_local__.inet.0
    name: default-domain:default-project:default-virtual-network:default-virtual-network.inet.0
    name: inet.0
    name: default-domain:default-project:ip-fabric:ip-fabric.inet.0
    name: default-domain:k8s-default:k8s-default-pod-network:k8s-default-pod-network.inet.0
    name: default-domain:k8s-default:k8s-default-service-network:k8s-default-service-network.inet.0
    
    [root@ip-172-31-14-47 contrail-ansible-deployer]# ./contrail-introspect-cli/ist.py ctr route show
    
    bgp.ermvpn.0: 6 destinations, 6 routes (0 primary, 6 secondary, 0 infeasible)
    
    1-172.31.41.236:1-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:02:26.545449, last_modified: 2019-Apr-13 01:41:18.023211
        [Local|None] age: 0:02:26.548569, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    1-172.31.41.236:2-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:01:09.096721, last_modified: 2019-Apr-13 01:42:35.471939
        [Local|None] age: 0:01:09.100272, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    1-172.31.41.236:3-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:00:41.812247, last_modified: 2019-Apr-13 01:43:02.756413
        [Local|None] age: 0:00:41.816037, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    2-172.31.41.236:1-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:02:26.544851, last_modified: 2019-Apr-13 01:41:18.023809
        [Local|None] age: 0:02:26.548875, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    2-172.31.41.236:2-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:01:09.096567, last_modified: 2019-Apr-13 01:42:35.472093
        [Local|None] age: 0:01:09.100828, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    2-172.31.41.236:3-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:00:41.812032, last_modified: 2019-Apr-13 01:43:02.756628
        [Local|None] age: 0:00:41.816542, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    bgp.evpn.0: 3 destinations, 3 routes (0 primary, 3 secondary, 0 infeasible)
    
    2-172.31.41.236:1-0-0e:92:cc:bd:aa:08,0.0.0.0, age: 0:02:26.545224, last_modified: 2019-Apr-13 01:41:18.023436
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.550028, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'mpls-o-gre', 'udp'], label: 20, AS path: None
    
    2-172.31.41.236:1-0-0e:92:cc:bd:aa:08,172.31.41.236, age: 0:02:26.545271, last_modified: 2019-Apr-13 01:41:18.023389
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.550313, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'mpls-o-gre', 'udp'], label: 20, AS path: None
    
    3-172.31.41.236:1-2-172.31.41.236, age: 0:02:26.545365, last_modified: 2019-Apr-13 01:41:18.023295
        [Local|None] age: 0:02:26.550656, localpref: 100, nh: 172.31.41.236, encap: ['vxlan'], label: 2, AS path: None
    
    bgp.l3vpn.0: 3 destinations, 3 routes (0 primary, 3 secondary, 0 infeasible)
    
    172.31.41.236:1:172.31.41.236/32, age: 0:02:26.545019, last_modified: 2019-Apr-13 01:41:18.023641
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.550608, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp', 'native'], label: 16, AS path: None
    
    172.31.41.236:2:10.47.255.252/32, age: 0:00:41.733374, last_modified: 2019-Apr-13 01:43:02.835286
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:00:41.739187, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 25, AS path: None
    
    172.31.41.236:3:10.96.0.10/32, age: 0:00:41.732905, last_modified: 2019-Apr-13 01:43:02.835755
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:00:41.738945, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 25, AS path: None
    
    bgp.rtarget.0: 7 destinations, 7 routes (7 primary, 0 secondary, 0 infeasible)
    
    64512:target:64512:8000001, age: 0:02:26.592101, last_modified: 2019-Apr-13 01:41:17.976559
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.598445, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    64512:target:64512:8000002, age: 0:02:26.592073, last_modified: 2019-Apr-13 01:41:17.976587
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.598626, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    64512:target:64512:8000003, age: 0:02:26.592051, last_modified: 2019-Apr-13 01:41:17.976609
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.598800, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    64512:target:172.31.14.47:0, age: 0:05:09.194543, last_modified: 2019-Apr-13 01:38:35.374117
        [Local|None] age: 0:05:09.201488, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    64512:target:172.31.14.47:1, age: 0:02:26.592028, last_modified: 2019-Apr-13 01:41:17.976632
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.599168, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    64512:target:172.31.14.47:4, age: 0:01:09.099898, last_modified: 2019-Apr-13 01:42:35.468762
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:01:09.107253, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    64512:target:172.31.14.47:5, age: 0:00:41.824049, last_modified: 2019-Apr-13 01:43:02.744611
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:00:41.831612, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    default-domain:default-project:ip-fabric:ip-fabric.ermvpn.0: 3 destinations, 3 routes (3 primary, 0 secondary, 0 infeasible)
    
    0-172.31.41.236:1-0.0.0.0,255.255.255.255,0.0.0.0, age: 0:02:26.544896, last_modified: 2019-Apr-13 01:41:18.023764
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.552710, localpref: 100, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 0, AS path: None
    
    1-0:0-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:02:26.545544, last_modified: 2019-Apr-13 01:41:18.023116
        [Local|None] age: 0:02:26.553571, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    2-0:0-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:02:26.544992, last_modified: 2019-Apr-13 01:41:18.023668
        [Local|None] age: 0:02:26.553215, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    default-domain:default-project:ip-fabric:ip-fabric.evpn.0: 4 destinations, 4 routes (4 primary, 0 secondary, 0 infeasible)
    
    2-0:0-0-0e:92:cc:bd:aa:08,0.0.0.0, age: 0:02:26.545298, last_modified: 2019-Apr-13 01:41:18.023362
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.553810, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'mpls-o-gre', 'udp'], label: 20, AS path: None
    
    2-0:0-0-0e:92:cc:bd:aa:08,172.31.41.236, age: 0:02:26.545318, last_modified: 2019-Apr-13 01:41:18.023342
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.554076, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'mpls-o-gre', 'udp'], label: 20, AS path: None
    
    2-172.31.41.236:1-2-ff:ff:ff:ff:ff:ff,0.0.0.0, age: 0:02:26.545486, last_modified: 2019-Apr-13 01:41:18.023174
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.554476, localpref: 100, nh: 172.31.41.236, encap: ['vxlan'], label: 2, AS path: None
    
    3-172.31.41.236:1-2-172.31.41.236, age: 0:02:26.545411, last_modified: 2019-Apr-13 01:41:18.023249
        [Local|None] age: 0:02:26.554614, localpref: 100, nh: 172.31.41.236, encap: ['vxlan'], label: 2, AS path: None
    
    default-domain:default-project:ip-fabric:ip-fabric.inet.0: 3 destinations, 3 routes (1 primary, 2 secondary, 0 infeasible)
    
    10.47.255.252/32, age: 0:00:41.733312, last_modified: 2019-Apr-13 01:43:02.835348
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:00:41.742801, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 25, AS path: None
    
    10.96.0.10/32, age: 0:00:41.732847, last_modified: 2019-Apr-13 01:43:02.835813
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:00:41.742561, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 25, AS path: None
    
    172.31.41.236/32, age: 0:02:26.545051, last_modified: 2019-Apr-13 01:41:18.023609
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.554985, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp', 'native'], label: 16, AS path: None
    
    default-domain:k8s-default:k8s-default-pod-network:k8s-default-pod-network.ermvpn.0: 3 destinations, 3 routes (3 primary, 0 secondary, 0 infeasible)
    
    0-172.31.41.236:2-0.0.0.0,255.255.255.255,0.0.0.0, age: 0:01:09.096823, last_modified: 2019-Apr-13 01:42:35.471837
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:01:09.107020, localpref: 100, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 0, AS path: None
    
    1-0:0-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:01:09.096765, last_modified: 2019-Apr-13 01:42:35.471895
        [Local|None] age: 0:01:09.107383, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    2-0:0-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:01:09.096621, last_modified: 2019-Apr-13 01:42:35.472039
        [Local|None] age: 0:01:09.107473, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    default-domain:k8s-default:k8s-default-pod-network:k8s-default-pod-network.inet.0: 3 destinations, 3 routes (1 primary, 2 secondary, 0 infeasible)
    
    10.47.255.252/32, age: 0:00:41.733411, last_modified: 2019-Apr-13 01:43:02.835249
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:00:41.744526, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 25, AS path: None
    
    10.96.0.10/32, age: 0:00:41.732872, last_modified: 2019-Apr-13 01:43:02.835788
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:00:41.744256, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 25, AS path: None
    
    172.31.41.236/32, age: 0:02:26.544986, last_modified: 2019-Apr-13 01:41:18.023674
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.556602, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp', 'native'], label: 16, AS path: None
    
    default-domain:k8s-default:k8s-default-service-network:k8s-default-service-network.ermvpn.0: 3 destinations, 3 routes (3 primary, 0 secondary, 0 infeasible)
    
    0-172.31.41.236:3-0.0.0.0,255.255.255.255,0.0.0.0, age: 0:00:41.812457, last_modified: 2019-Apr-13 01:43:02.756203
        [XMPP|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:00:41.824352, localpref: 100, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 0, AS path: None
    
    1-0:0-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:00:41.812393, last_modified: 2019-Apr-13 01:43:02.756267
        [Local|None] age: 0:00:41.824504, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    2-0:0-172.31.14.47,255.255.255.255,0.0.0.0, age: 0:00:41.812099, last_modified: 2019-Apr-13 01:43:02.756561
        [Local|None] age: 0:00:41.824428, localpref: 100, nh: 172.31.14.47, encap: [], label: 0, AS path: None
    
    default-domain:k8s-default:k8s-default-service-network:k8s-default-service-network.inet.0: 3 destinations, 3 routes (1 primary, 2 secondary, 0 infeasible)
    
    10.47.255.252/32, age: 0:00:41.733337, last_modified: 2019-Apr-13 01:43:02.835323
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:00:41.745932, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 25, AS path: None
    
    10.96.0.10/32, age: 0:00:41.732935, last_modified: 2019-Apr-13 01:43:02.835725
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:00:41.745758, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 25, AS path: None
    
    172.31.41.236/32, age: 0:02:26.544959, last_modified: 2019-Apr-13 01:41:18.023701
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:02:26.558031, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp', 'native'], label: 16, AS path: None
    [root@ip-172-31-14-47 contrail-ansible-deployer]#
    

    如果显示类似信息,则说明一切正常,你可以基于K8s yaml创建容器了。

    vi first-containers.yaml
    apiVersion: apps/v1beta1
    kind: Deployment
    metadata:
      name: cirros-deployment
      labels:
        app: cirros-deployment
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: cirros-deployment
      template:
        metadata:
          labels:
            app: cirros-deployment
        spec:
          containers:
          - name: cirros
            image: cirros
            ports:
            - containerPort: 22
    
    kubectl create -f first-containers.yaml
    kubectl get pod -o wide ## check pod name and ip
    kubectl exec -it cirros-deployment-xxxx sh
    ping (another pod's ip)
    
    
    [root@ip-172-31-14-47 ~]# kubectl create -f first-containers.yaml
    deployment "cirros-deployment" created
    [root@ip-172-31-14-47 ~]# 
    [root@ip-172-31-14-47 ~]# kubectl get pod -o wide
    NAME                                 READY     STATUS    RESTARTS   AGE       IP              NODE
    cirros-deployment-54b65ccf48-cr9dd   1/1       Running   0          34s       10.47.255.250   ip-172-31-41-236.ap-northeast-1.compute.internal
    cirros-deployment-54b65ccf48-z9dds   1/1       Running   0          34s       10.47.255.251   ip-172-31-41-236.ap-northeast-1.compute.internal
    [root@ip-172-31-14-47 ~]#
    
    [root@ip-172-31-14-47 ~]# kubectl exec -it cirros-deployment-54b65ccf48-cr9dd sh
    / # 
    / # 
    / # ping 10.47.255.251
    PING 10.47.255.251 (10.47.255.251): 56 data bytes
    64 bytes from 10.47.255.251: seq=0 ttl=63 time=0.572 ms
    64 bytes from 10.47.255.251: seq=1 ttl=63 time=0.086 ms
    ^C
    --- 10.47.255.251 ping statistics ---
    2 packets transmitted, 2 packets received, 0% packet loss
    round-trip min/avg/max = 0.086/0.329/0.572 ms
    / #
    

    太棒了!这是通过Tungsten Fabric vRouter传输的第一个报文。

    如果运行有问题,也请不要介意。Tungten Fabric的Slack站点可以为你提供帮助。在以下的地址提交你的日志,并尝试寻求帮助来解决该问题。地址:https://tungstenfabric.slack.com
    (编者按:国内用户请关注微信TF中文社区,中文官网:https://tungstenfabric.org.cn/ )

    再次输入“ist.py ctr route show”,你将看到k8s-pod-network包含了来自两个pod以及每个pod的下一跳,与K8s节点的IP相同。

    ./contrail-introspect-cli/ist.py ctr route show (pod ip) ## similar to 'show route (some ip)'
    
    
    [root@ip-172-31-14-47 contrail-ansible-deployer]# ./contrail-introspect-cli/ist.py ctr route show 10.47.255.250
    
    default-domain:default-project:ip-fabric:ip-fabric.inet.0: 5 destinations, 5 routes (1 primary, 4 secondary, 0 infeasible)
    
    10.47.255.250/32, age: 0:03:10.553628, last_modified: 2019-Apr-13 01:46:13.217388
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:03:10.556716, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 37, AS path: None
    
    default-domain:k8s-default:k8s-default-pod-network:k8s-default-pod-network.inet.0: 5 destinations, 5 routes (3 primary, 2 secondary, 0 infeasible)
    
    10.47.255.250/32, age: 0:03:10.553734, last_modified: 2019-Apr-13 01:46:13.217282
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:03:10.557251, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 37, AS path: None
    
    default-domain:k8s-default:k8s-default-service-network:k8s-default-service-network.inet.0: 5 destinations, 5 routes (1 primary, 4 secondary, 0 infeasible)
    
    10.47.255.250/32, age: 0:03:10.553654, last_modified: 2019-Apr-13 01:46:13.217362
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:03:10.557453, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 37, AS path: None
    [root@ip-172-31-14-47 contrail-ansible-deployer]#
    

    请注意,ip-fabric VN和k8s-default-service-network也具有该前缀,因为k8s-pod-network的路由已“泄漏”到那些网络。想要具有某个特定路由表的路由,可以使用-t选项。

    [root@ip-172-31-14-47 contrail-ansible-deployer]# ./contrail-introspect-cli/ist.py ctr route show -t default-domain:k8s-default:k8s-default-pod-network:k8s-default-pod-network.inet.0 10.47.255.251
    
    default-domain:k8s-default:k8s-default-pod-network:k8s-default-pod-network.inet.0: 5 destinations, 5 routes (3 primary, 2 secondary, 0 infeasible)
    
    10.47.255.251/32, age: 0:05:44.533377, last_modified: 2019-Apr-13 01:46:09.193202
        [XMPP (interface)|ip-172-31-41-236.ap-northeast-1.compute.internal] age: 0:05:44.536291, localpref: 200, nh: 172.31.41.236, encap: ['gre', 'udp'], label: 32, AS path: None
    [root@ip-172-31-14-47 contrail-ansible-deployer]#
    

    附录:外部访问

    有人说Tungsten Fabric总是需要好的路由器才能实现外部访问,我认为这里有些误解。

    实际上这是不对的,因为从v4.1开始,Tungsten Fabric开始支持一项称为无网关(gatewayless)的功能,该功能允许容器直接与外界通信(对于带有Calico的类似用例也很有用)。

    要启用此功能,你可以登录Tungsten Fabric WebUI(https://(k8s masters's ip):8143,admin:contrail123)并进入Configure > Networks > k8s-default-pod-network,以切换Advanced Options > IP Fabric Forwarding。

    • 你还需要在该VN和default-domain:default-project:ip-fabric之间设置网络策略,因为如果没有此策略,RPF check将丢弃该报文

    如果ping一下从容器到K8s master的IP,你会发现K8s master从容器接收到一个报文,并向K8s master添加静态路由,ping的运行效果良好。

    • 请注意,如果你使用的是AWS,则需要进行K8s节点的接口设置(EC2 > Network Interfaces > Change Source/Dest Check > Disabled)

    因此,它允许对基于外部访问的网络节点进行类似的设置,该访问基于路由器上的静态路由。

    你还可以选择将IPv4 BGP与无网关(gatewayless)结合使用,因为它可以动态更新每个容器的下一跳,并将报文直接发送到正确的vRouter,从而消除了瓶颈。

    注意:此虚拟网络(virtual-network)也可以用作浮动IP(floating-ip)的来源。

    1.此虚拟网络上设置“Advanced Options”>“External”( 浮动IP池将使用“default”作为名称来创建)

    2.从Kubernetes或OpenStack分配浮动IP(floating-ip)

    • 对于Kubernetes,它将是外部IP的来源,并且需要将此参数指定给kube-manager: KUBERNETES_PUBLIC_FIP_POOL。例如:KUBERNETES_PUBLIC_FIP_POOL={'domain': 'default-domain', 'project': 'default', 'network': 'public-network1', 'name': 'default' }
    • 对于OpenStack,horizon或cli可用于为虚拟机分配浮动IP

    3.你也可以从Tungsten Fabric WebUI直接将浮动IP分配给特定端口(Configure > Ports > edit > floating-ip)

    3接下来的行动

    这可能是你第一次接触Tungsten Fabric,关于看完这篇文章后要怎么做,这里有一些建议。其实,有很多事情可以做,例如高可用性、监视,与其它编排器或路由器/交换机的集成等。

    网络上有很多资源,但是你需要做好选择,我将首先推荐Contrail软件包和教育材料中的一些资源,即便你只会使用开源版本。

    Tungsten Fabric是一个功能强大的平台,具有很多功能,例如安全策略、分析、l3dsr负载平衡器、服务链、BGPaaS等,其中许多功能对于解决现实世界中的问题都是非常棒的。这些链接将包含很多内容以及到其他资源的链接。

    编者按:这里有几个沟通的渠道,如果需要帮助,请尝试使用它们:
    官网社区:https://tungsten.io/community/
    中文官网:https://tungstenfabric.org.cn/
    微信公众号:TF中文社区

    Tungsten Fabric 架构解析系列文章——

    第一篇:TF主要特点和用例
    第二篇:TF怎么运作
    第三篇:详解vRouter体系结构
    第四篇:TF的服务链
    第五篇:vRouter的部署选项
    第六篇:TF如何收集、分析、部署?
    第七篇:TF如何编排
    第八篇:TF支持API一览
    第九篇:TF如何连接到物理网络?
    第十篇:TF基于应用程序的安全策略


Log in to reply