1 2 First Previous 21 Replies Latest reply on Dec 9, 2016 3:51 AM by aradford

    Design Recommendation for APIC-EM Network

    cdeluna

      Hi,

       

      We are working on building out a proof of concept for zero touch provisioning and would like to focus on APIC-EM.

       

      We have a temp server that we are using for a single APIC-EM VM.  Its an ESXi 5.5 host. Along with that we have a MS DHCP server configured to provide option 43.  We are running 1.3.1.9 as a standalone server.

       

      Eth0 of the host goes to an external network with services (NTP etc.), Eth1 goes to the private "Provisioning Network".  The APIC-EM VM is similarly configured so that it is accessible for scripting etc on its first interface and has access to the private network on the second interface.

       

      Everything seems to be working as expected except that we can't seem to successfully use the PnP application.  I've tried disabling the "external interface" so that only the private network is available thinking the network interfaces were more 'HA' /NIC Teaming but that did not make any difference.

       

      We've tried a variety of devices which all meet the minimum requirements for hardware and software

      2901 ISR (Gen2)

      2960S Switch

      3650 Switch

       

      We've tried projects as well as just seeing they will be "discovered" without success.

       

      The devices do get an IP address and they start the AutoInstall process and the APIC-EM never recognizes them, pre-provisioined or not.

       

      Here is the log from the 2960S

       

      ```

      *Mar  1 00:02:26.098: %SYS-5-RESTART: System restarted --

      Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(58)SE2, RELEASE SOFTWARE (fc1)

      Technical Support: http://www.cisco.com/techsupport

      Copyright (c) 1986-2011 by Cisco Systems, Inc.

      Compiled Thu 21-Jul-11 02:22 by prod_rel_team

      *Mar  1 00:02:27.634: %LINK-3-UPDOWN: Interface FastEthernet0, changed state to down

      to up

      *Mar  1 00:02:29.107: %USB_CONSOLE-6-MEDIA_RJ45: Console media-type is RJ45.

      *Mar  1 00:02:30.444: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up

      *Mar  1 00:02:58.446: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up

      *Mar  1 00:03:09.215: AUTOINSTALL: Vlan1 is assigned 192.0.2.102 got vend id vend spec. info ret: succeed

      *Mar  1 00:03:19.224: AUTOINSTALL: Obtain siaddr 192.0.2.100 (as config server) <--.100 is the DHCP server

      %Error opening tftp://192.0.2.100/network-confg (Timed out)

      %Error opening tftp://192.0.2.100/cisconet.cfg (Timed out)

      %Error opening tftp://192.0.2.100/router-confg (Timed out)

      %Error opening tftp://192.0.2.100/ciscortr.cfg (Timed out)

      %Error opening tftp://192.0.2.100/network-confg (Timed out)

      %Error opening tftp://192.0.2.100/cisconet.cfg (Timed out)

      %Error opening tftp://192.0.2.100/router-confg (Timed out)

      ```

       

      Any suggestions on where to look for issues would be very welcome!

       

      Thanks!

        • 1. Re: Design Recommendation for APIC-EM Network
          ngoldwat

          Hi

           

          First off what application are you using? PnP? IWAN?

          From what you posted it looks to me like post provisioning, the device does not have a route back to 192.0.2.100. Is that APIC-EM or a different tftp server?  If different, did you verify that tftp is running and that no ACL or firewall rules drop the traffic?

           

          Thanks

          • 2. Re: Design Recommendation for APIC-EM Network
            cdeluna

            Hi Nicholas,

             

            Sorry...should have mentioned...PnP.

            • 3. Re: Design Recommendation for APIC-EM Network
              ngoldwat

              I have taking ownership of the TAC case you have opened.

              • 4. Re: Design Recommendation for APIC-EM Network
                aradford

                Hi Claudia (and anyone else reading this thread),

                this is the scenario i use every day in my lab.

                 

                Some troubleshooting tips (I will probably blogs these if there is interest).

                1) Connect to the PnP switch and determine the following:

                • the switch has a valid IP address
                • the switch can ping the controller (by the appropriate IP address -- internal vs external)
                • that the PnP profile has been established on the switch (which discovery mechanism are you using) - "show run | inc pnp" should show a "pnp profile pnp-zero-touch"
                • If there is an issue with discovery, you can create the pap-profile manually to test the rest of the process

                 

                pnp profile manual-test

                transport http ipv4 x.x.x.x port 80

                 

                you can then use the "debug pnp all" command to get great insight into what is going on

                 

                Adam

                • 5. Re: Design Recommendation for APIC-EM Network
                  cdeluna

                  Adam!  The man himself...I've been listening to you alot lately! (Youtube and CiscoLive)

                  thanks for chiming in!

                  Nick and I just took a quick look.

                  On the 2960S switch, even though it running the minimum code it does not look like pnp is on there, so that may explain that.

                  On the ISR Gen2 2901, also running > recommended ios, it has ccp ans so Nick mentioned that that will cause it to fail.  I'm deleting and retrying.

                  On the 3650s I may have had an incorrect SN so I'm about to verify and try that again.

                   

                  I'll be using these commands shortly so thank you and Nick and I will be looking at this later on tonight.

                   

                  Switch Ports Model              SW Version            SW Image               

                  ------ ----- -----              ----------            ----------             

                  *    1 52    WS-C2960S-48LPS-L  12.2(58)SE2           C2960S-UNIVERSALK9-M

                   

                  BTW:

                   

                  • the switch has a valid IP address - Yes it does so DHCP is working TBD on the Opt 43
                  • the switch can ping the controller (by the appropriate IP address -- internal vs external)  It can on the internal

                   

                  • I will check these in a few:
                  • that the PnP profile has been established on the switch (which discovery mechanism are you using) - "show run | inc pnp" should show a "pnp profile pnp-zero-touch"
                  • If there is an issue with discovery, you can create the pap-profile manually to test the rest of the process
                  • 6. Re: Design Recommendation for APIC-EM Network
                    cdeluna

                    OK..interesting:

                     

                    On the Gen2 ISR 2901:

                     

                    There is no pnp bootstrap and its trying to use DNS to go to the cloud server I believe....?

                     

                    *Dec  8 22:10:29.439: %SYS-5-RESTART: System restarted --

                    Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.4(3)M1, RELEASE SOFTWARE (fc1)

                    Technical Support: http://www.cisco.com/techsupport

                    Copyright (c) 1986-2014 by Cisco Systems, Inc.

                    Compiled Sat 25-Oct-14 03:34 by prod_rel_team

                    *Dec  8 22:10:30.123: %SYS-6-BOOTTIME: Time taken to reboot after reload =  670 seconds

                    *Dec  8 22:10:30.735: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

                    *Dec  8 22:10:30.735: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

                    *Dec  8 22:10:30.735: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

                    *Dec  8 22:10:30.735: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

                    %Error opening tftp://192.0.2.100/router-confg (Timed out)

                    *Dec  8 22:10:49.551: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://devicehelper.cisco.com/pnp/HELLO

                    Router>en

                    Router#sh run | inc pnp

                    Router#

                    %Error opening tftp://192.0.2.100/ciscortr.cfg (Timed out)

                    *Dec  8 22:11:27.551: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://devicehelper.cisco.com/pnp/HELLO

                    %Error opening tftp://255.255.255.255/network-confg (Timed out)

                    • 7. Re: Design Recommendation for APIC-EM Network
                      aradford

                      Thanks Claudia :-),

                       

                      glad it is under control.  Good catch on three common issues. CCP is also a common gotcha.

                       

                      Let us know if there is anything else we can help with?

                       

                      Not sure if you have seen my blogs, but sometimes that can also help.  There are quite a few on different pnp deployment models

                       

                      My Blog Index

                       

                      Adam

                      1 of 1 people found this helpful
                      • 8. Re: Design Recommendation for APIC-EM Network
                        aradford

                        There is a cloud fallback option too.  That is in controlled availability today.

                         

                        What discovery mechanism are you attempting to use for your testing?  I just read your earlier post.... looks like option 43.

                         

                        That should create the pnp profile.

                         

                        "show  pnp trace" is another useful command too.

                         

                        Adam

                        1 of 1 people found this helpful
                        • 9. Re: Design Recommendation for APIC-EM Network
                          cchitnis

                          PnP protocol will be using "time-pnp.cisco.com" and/or "pool.ntp.org"  for time sync.   Some lab networks block public NTP access, for such DNS mappings to local NTP server is needed.

                          Does this apply to your case CLaudia? If yes, can you do the needful in your setup?

                          1 of 1 people found this helpful
                          • 10. Re: Design Recommendation for APIC-EM Network
                            cdeluna

                            We are going for zero touch.  I have 150 3650s that are "fresh out of the box" and we are using them to as an APIC-EM proof of concept for new staging options.   So we are using DHCP option 43 to point the devices on the private network to the APIC-EM private network interface.

                             

                            On NTP, so yes, the provisioning network is totally isolated.  I am including the local NTP server in the DHCP offer. Will that make a difference?

                            • 11. Re: Design Recommendation for APIC-EM Network
                              cdeluna

                              Thank you for the additional information, Chakrapani.  Yes the network is totally isolated.  If sending the NTP server in the DHCP offer does not work what would you suggest in the isolated lab.  dummy DNS A records?

                              • 12. Re: Design Recommendation for APIC-EM Network
                                cchitnis

                                Yes, a dummy record of the following sort in your DNS server:

                                ip host time-pnp.cisco.com <IP of your choice>

                                Similar entry for pool.ntp.org I believe.

                                This way, you can resolve name queries for time-pnp.cisco.com and/or pool.ntp.org with an IP of your choice

                                1 of 1 people found this helpful
                                • 13. Re: Design Recommendation for APIC-EM Network
                                  cdeluna

                                  OK..so on the 2901 ISR, if i use the manual bootstrap that Adam suggested it pops right up now to figure out how to get zero touch going!

                                   

                                  Can someone confirm the DHCP string to use (we are using Microsoft 2012 Server)?

                                  • 14. Re: Design Recommendation for APIC-EM Network
                                    aradford

                                    Hi Claudia,

                                     

                                    option 43 needs to be

                                    an ascii string - "5A1N;B2;K4;Iw.x.y.z.;J80"


                                    where "w.x.y.z" is the IP address of your controller.


                                    Adam


                                    NOTE:  Initial post had a typo with the "I" for the IP address

                                    1 2 First Previous