Continuing the story

My initial blogs showed examples of using APIC-EM REST API to upload configuration files and create rules for PnP devices.  More recently, I showed some of the common deployment models for switches, taking into account, VLANs, management VLANs, trunks, EtherChannels etc.

This blog covers an advanced use case.  Etherchannel + trunk + Non Vlan 1 for management (NV1) + Non vlan1 native vlan.

 

For all of these examples I am using a 3650 switch running 16.3.1 code, but you could use versions of 3.6.5 and 3.7.4 (For other platforms such 2960x please see release notes for details).

 

Make sure you do not hit any keys on the console while the switch is booting, as this can interrupt the PnP process.

 

Setup DHCP server

 

  The first thing we need is a mechanism for the switch to discover the controller.  In our examples we are going to use DHCP, but you could also use DNS etc. as covered in earlier blogs.  Here is a sample configuration for an IOS switch. The controller IP address is 10.10.10.140.  Note also the use of the "5A1D" in the option 43 string.  The "D" displays debug messages for PnP on the console of the PnP switch.


ip dhcp pool ZTD-switches

network 10.10.14.0 255.255.255.0

default-router 10.10.14.1

option 43 ascii "5A1D;B2;K4;I10.10.10.140;J80"

  remember

 

 

EtherChannel + NV1 for management + NV1 Native

The switches are going to be connected by two links bound together in an ether-channel.

 

nv.png

 

The upstream switch needs to have an ether channel configured.  To avoid issues when the PnP switch first comes up, the "no port-channel standalone-disable" command is required.  If this is left out the channel will be disabled as it has not been configured on the PnP switch at boot up.

 

In addition the native VLAN has been set to 999.

 

The "pnp startup-vlan 14" command is required to create a new management VLAN on the PnP switch. By default VLAN 1 would be used.

 

pnp startup-vlan 14

 

interface Port-channel1

switchport trunk native vlan 999

switchport mode dynamic desirable

no port-channel standalone-disable

 

interface GigabitEthernet1/0/5

description PNP switch 3650->g1/0/1

switchport mode dynamic desirable

switchport trunk native vlan 999

channel-protocol lacp

channel-group 1 mode passive

 

interface GigabitEthernet1/0/6

description 2nd link to 3650 etherchannel test

switchport trunk native vlan 999

switchport mode dynamic desirable

channel-protocol lacp

channel-group 1 mode passive

 

 

The configuration of the PnP switch includes the EtherChannel.  It also has native VLAN of 999.

NOTE: remember to define VLAN 999 otherwise you will have issues after you deploy the configuration.

 

hostname 3650-dhcp

enable password xxx

!

username xxx password 0 xxx

!

ip http server

ip http secure-server

snmp-server community xxx RO

 

interface Port-channel1

switchport mode dynamic desirable

switchport trunk native vlan 999

no port-channel standalone-disable

!

int range g1/0/1,g1/0/3

switchport mode dynamic desirable

switchport trunk native vlan 999

channel-protocol lacp

channel-group 1 mode active

 

vlan 999

!

!

!

!

line con 0

line vty 0 4

login local

transport input ssh telnet

line vty 5 15

login local

 

 

Once the switch has booted, you will see the following log messages (remember to use the "5A1D" string as mentioned earlier to get the debugs).  Both VLAN 1 and 999 are initially blocked, but VLAN 14 is created as the management VLAN. VLAN 14 is used to communicate with APIC-EM, and you can see the switch successfully connects to the controller.

 

You will also see warning messages about the Native VLAN mismatch, and you can ignore them for now. Once the configuration is downloaded to the PnP switch, these will go away.

 


*Oct 10 22:16:27.480: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
*Oct 10 22:16:27.816: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to up
*Oct 10 22:16:28.414: %SYS-6-BOOTTIME: Time taken to reboot after reload =  339 seconds
*Oct 10 22:16:31.489: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Oct 10 22:16:31.825: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up
*Oct 10 22:16:38.810: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 999 on GigabitEthernet1/0/1 VLAN1.
*Oct 10 22:16:38.810: %SPANTREE-2-BLOCK_PVID_PEER: Blocking GigabitEthernet1/0/1 on VLAN0999. Inconsistent peer vlan.
*Oct 10 22:16:38.811: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/1 on VLAN0001. Inconsistent local vlan.
*Oct 10 22:16:38.811: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 999 on GigabitEthernet1/0/3 VLAN1.
*Oct 10 22:16:38.811: %SPANTREE-2-BLOCK_PVID_PEER: Blocking GigabitEthernet1/0/3 on VLAN0999. Inconsistent peer vlan.
*Oct 10 22:16:38.812: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/3 on VLAN0001. Inconsistent local vlan.
*Oct 10 22:16:40.072: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
*Oct 10 22:16:47.124: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/3 (1), with 3850-core GigabitEthernet1/0/6 (999).
*Oct 10 22:16:47.254: %SYS-5-CONFIG_I: Configured from console by tty100
*Oct 10 22:16:47.291: %SYS-5-CONFIG_I: Configured from console by tty100
*Oct 10 22:16:50.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan14, changed state to up
*Oct 10 22:16:56.126: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/1 (1), with 3850-core GigabitEthernet1/0/5 (999).
*Oct 10 22:17:05.127: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/1 (1), with 3850-core GigabitEthernet1/0/5 (999).
*Oct 10 22:17:14.127: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/3 (1), with 3850-core GigabitEthernet1/0/6 (999).
*Oct 10 22:17:21.376: %PNPA-DHCP Op-43 Msg: Process state = READY
*Oct 10 22:17:21.376: %PNPA-DHCP Op-43 Msg: OK to process message
*Oct 10 22:17:21.377: XML-UPDOWN: PNPA_DHCP_OP43 XML Interface(102) UP. PID=279
*Oct 10 22:17:21.377: %PNPA-DHCP Op-43 Msg: _pdoon.1.ntf.don=279
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdoop.1.org=[A1D;B2;K4;I10.10.10.140;J80]
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdgfa.1.inp=[B2;K4;I10.10.10.140;J80]
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdgfa.1.B2.s12=[ ipv4 ]
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdgfa.1.K4.htp=[ transport http ]
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdgfa.1.Ix.srv.ip.rm=[ 10.10.10.140 ]
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdgfa.1.Jx.srv.rt.rm=[ port 80 ]
*Oct 10 22:17:21.379: %PNPA-DHCP Op-43 Msg: _pdoop.1.ztp=[pnp-zero-touch] host=[] ipad=[10.10.10.140] port=80
*Oct 10 22:17:21.379: %PNPA-DHCP Op-43 Msg: _pors.done=1
*Oct 10 22:17:21.379: %PNPA-DHCP Op-43 Msg: _pdokp.1.kil=[PNPA_DHCP_OP43] pid=279 idn=[Vlan14]
*Oct 10 22:17:21.379: XML-UPDOWN: Vlan14 XML Interface(102) SHUTDOWN(101). PID=279
*Oct 10 22:17:21.493: %DHCP-6-ADDRESS_ASSIGN: Interface Vlan14 assigned DHCP address 10.10.14.3, mask 255.255.255.0, hostname 

*Oct 10 22:17:23.129: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/1 (1), with 3850-core GigabitEthernet1/0/5 (999).
*Oct 10 22:17:32.130: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/3 (1), with 3850-core GigabitEthernet1/0/6 (999).% Generating 2048 bit RSA keys, keys will be non-exportable... got vend id vend spec. info ret: succeed
*Oct 10 22:17:39.848: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://10.10.10.140:80/pnp/HELLO
*Oct 10 22:17:39.858: %PNP-6-HTTP_CONNECTED: PnP Discovery connected to PnP server http://10.10.10.140:80/pnp/HELLO
*Oct 10 22:17:41.142: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/1 (1), with 3850-core GigabitEthernet1/0/5 (999).
[OK] (elapsed time was 6 seconds)
protocol on Interface Vlan14, changed state to up



 

 

Looking at the initial configuration created as a result of the "pnp startup-vlan 14" command, both active interfaces have been placed in VLAN 14. Interface VLAN 14 was created and set to use DHCP.

 

 

interface GigabitEthernet1/0/1

switchport access vlan 14

macro description CISCO_SMI_EVENT

!

interface GigabitEthernet1/0/3

switchport access vlan 14

macro description CISCO_SMI_EVENT

!

interface Vlan14

ip address dhcp

 

 

Both interfaces are in trunk mode.

 

 

Switch#show int g1/0/1 trunk

 

Port Mode             Encapsulation  Status Native vlan

Gi1/0/1 auto             802.1q         trunking      1

 

Port Vlans allowed on trunk

Gi1/0/1 1-4094

 

Port Vlans allowed and active in management domain

Gi1/0/1 1,14,999

 

Port Vlans in spanning tree forwarding state and not pruned

Gi1/0/1     14S

Switch#show int g1/0/3 trunk

 

Port        Mode             Encapsulation  Status Native vlan

Gi1/0/3     auto             802.1q         trunking      1

 

Port        Vlans allowed on trunk

Gi1/0/3     1-4094

 

Port        Vlans allowed and active in management domain

Gi1/0/3     1,14,999

 

Port        Vlans in spanning tree forwarding state and not pruned

Gi1/0/3     none

 

At this point, I now claim the device and push the configuration to it.  I could have also used a pre defined rule, but I wanted to show the intermediate steps.

 

Looking at debugs, you can see both interfaces are up, and then the port channel comes up, after the configuration has been downloaded to the PnP switch.  Again, VLAN 14 is used for the management VLAN.

 

 


Oct 10 23:12:59.673: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
Oct 10 23:12:59.689: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to down
Oct 10 23:13:00.619: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
Oct 10 23:13:00.674: %LINK-3-UPDOWN: Interface Vlan14, changed state to down
Oct 10 23:13:01.578: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Oct 10 23:13:01.620: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up
Oct 10 23:13:01.728: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up
Oct 10 23:13:03.267: %LINK-3-UPDOWN: Interface Vlan14, changed state to up



 

 

We can see the status of the ether-channel.  Both ports are active and a part of the ether-channel

 

 

3650-dhcp#show etherchannel 1 port-channel

         Port-channels in the group:

         ---------------------------

 

Port-channel: Po1    (Primary Aggregator)

 

------------

 

Age of the Port-channel   = 0d:00h:17m:14s

Logical slot/port   = 12/1          Number of ports = 2

HotStandBy port = null

Port state          = Port-channel Ag-Inuse

Protocol            = LACP

Port security       = Disabled

Standalone          = Enabled (independent mode)

 

Ports in the Port-channel:

 

Index Load   Port     EC state        No of bits

------+------+------+------------------+-----------

  0     00 Gi1/0/1  Active             0

  0 00     Gi1/0/3  Active             0

 

 

This also shows VLAN999 is now the native VLAN.

 

 

3650-dhcp#show int port-channel 1 switchport

Name: Po1

Switchport: Enabled

Administrative Mode: dynamic desirable

Operational Mode: trunk

Administrative Trunking Encapsulation: dot1q

Operational Trunking Encapsulation: dot1q

Negotiation of Trunking: On

Access Mode VLAN: 14 (VLAN0014)

Trunking Native Mode VLAN: 999 (VLAN0999)

Administrative Native VLAN tagging: enabled

Voice VLAN: none

Administrative private-vlan host-association: none

Administrative private-vlan mapping: none

Administrative private-vlan trunk native VLAN: none

Administrative private-vlan trunk Native VLAN tagging: enabled

Administrative private-vlan trunk encapsulation: dot1q

Administrative private-vlan trunk normal VLANs: none

Administrative private-vlan trunk associations: none

Administrative private-vlan trunk mappings: none

Operational private-vlan: none

Trunking VLANs Enabled: ALL

Pruning VLANs Enabled: 2-1001

 

Protected: false

Unknown unicast blocked: disabled

Unknown multicast blocked: disabled

Appliance trust: none

 

 

What Next?

This blog covered an advanced deployment models for network plug and play.  Other blogs in the series have covered simpler deployment models and the API and how to automate the creation, upload of configuration files as well as the automation of rules.  In future I will cover switch stacking.

 

In the meantime, if you would like to learn more about this, you could come hang out with us in The Cisco Devnet DNA Community. We’ll have a continuous stream of blogs like this and you can ask questions and we’ll get you answers.  In addition, we have a Github repository where you can get examples related to PnP.

 

Thanks for reading,

  @adamradford123