After a few days playing with the IOx platform on a Connected Grid Router (CGR), I decided to write an informal post with my experience and lessons learned during this initiative. The main objective here is to describe the overview of the platform and also provide no-brainier quick start configuration to have everything up and running.
Before we start, be aware that IOx is evolving very fast and this post could be outdated at the time you are reading. At the time I'm writing, the latest software version public available for CGR1240 is 15.4(3)M2 (cgr1000-universalk9-bundle.SPA.154-3.M2.bin). Everything described here is taking this scenario as baseline.
Fog Computing and IOx Platform
The huge increasing amount of data generated due growth of Internet of Everything (IoE) is becoming a challenge for the applications and infrastructure, that need to scale accordingly in order to be able to process them. To be successful we need to take computational resources to the edge of the network, near where the end-users/devices are connected, providing a way to compute and aggregate information before send it back to the Cloud. This concept is called Fog Computing, and it is the motivation behind the Cisco IOx platform.
The IOx platform disaggregates the IOS network operating system from the device's hardware through a specialized hypervisor technology. This hypervisor is capable of hosting the IOS and also a Linux virtual machine called Guest OS (GOS), which shares hardware resources. The communication between GOS and IOS, or external network, uses GigabitEthernet 0/1 interface of IOS as gateway. The Linux running on Guest-OS is based on Yocto Project, an embedded Linux distribution.
The amount of resources available for the GOS depend on which platform you are running IOx. On CGR1240 device, the GOS virtual machine specification have 192MB of RAM and 1GB of HDD, sharing the Intel(R) Atom(TM) CPU E640 processor. In order to determine which platforms are currently supporting IOx, and what are the GOS capabilities on them, please check the platform documentation.
Before You Start
If your CGR is running CG-OS software, such as "cgr1000-uk9-final.5.2.1.CG4.3", you will need to migrate it to IOS (IOx-based) software. This process is hard-working but not that difficult. Our recommendation is to contact the Cisco Technical Assistance Center (TAC) and ask for support of a specialized engineer. However, if you feel brave and understand the risks check this:
If your CGR is already running IOS (IOX-based), such as "cgr1000-universalk9-bundle.SPA.154-3.M1.bin", just make sure to upgrade your IOS to the newest version in order to get new features and bug fixes. To accomplish that, just follow:
1. Download proper software version from Cisco.com and put it on a FTP/TFTP server reachable from your CGR:
2. CGR's flash partition:
CGR1240-IOX# copy tftp://192.168.1.100/cgr1000-universalk9-bundle.SPA.154-3.M2.bin flash:
3. After copy it, install using the bundle command. This will install all 3 components of IOx (hypervisor, IOS, Guest OS):
CGR1240-IOX# bundle install flash:cgr1000-universalk9-bundle.SPA.154-3.M2.bin
4. Save the config and reload the device.
CGR1240-IOX# write CGR1240-IOX# reload
PS: As said in step 3, the binary image contains all the 3 components images bundled together. In case you need access to individual components (hypervisor, IOS or GOS) images, you can use GNU Tar command to unpack it from Linux or Mac OS. These files are handy in case you need to recover from ROMmon or change component images individually.
MACBOOK:~ davigar$ tar xvf cgr1000-universalk9-bundle.SPA.154-3.M2.bin
Validate IOx Installation
In order to validate the proper IOx environment, we can issue two commands to confirm all components are installed properly. The first validates the IOx hypervisor version currently installed:
CGR1240-IOX# show platform hypervisor version: 1.1.0
The second validates the IOx GOS image version currently installed and the status. Don't worry to see running even though you didn't configure anything yet, but IOx starts GOS as soon as it boots. The second command is:
CGR1240-IOX# show platform guest-os Guest OS status: Installation: Cisco-GOS,version-1.26 State: RUNNING
Configure IOS to provide Networking for GOS
There are multiple ways to configure IOx in terms of networking between IOS and GOS, however I will describe one based on IPv4 with DHCP and NAT. On this scenario, the GOS will receive an IPv4 network configuration, including address, DNS and gateway, through DHCP server configured on IOS.
Lets start configuring interface GigabitEthernet0/1 with a private IP address from 192.168.1.0/24 segment, which will be the default gateway for the GOS. Even though we are not using IPv6, we need to enable it on the interface because this is used to management services. Also, we will configure it as a NAT Inside (private) interface. After get in configuration mode, the configuration should be done like:
interface GigabitEthernet0/1 description Internal Gateway for Guest-OS ip address 192.168.1.1 255.255.255.0 ip nat inside ip virtual-reassembly in duplex auto speed auto ipv6 enable no shutdown
The interface which is used by CGR as exit to the external network (WAN, for example) need to have the NAT Outside (public) configured in order to have NAT working properly. In my case I'm using GigabitEthernet2/1 as WAN, so the configuration should be:
interface GigabitEthernet2/1 description WAN Interface (Lab MGMT) no switchport ip address 172.16.1.201 255.255.255.0 ip nat outside ip virtual-reassembly in duplex auto speed auto no shutdown
The DHCP configuration is simple. In my case the GigabitEthernet0/1 has IP 192.168.1.1 from 192.168.1.0/24 segment, which will be the default gateway of GOS. Also, my DNS servers are 172.16.1.50 172.16.1.51. I configured DHCP to remember the bindings and lease time for 5 days, but this is not required. The configuration should looks like:
ip dhcp pool guest-os network 192.168.1.0 255.255.255.0 default-router 192.168.1.1 domain-name coibr.cisco.com dns-server 172.16.1.50 172.16.1.51 lease 5 remember
The last piece of configuration is the NAT. In my case I got an IP from my GigabitEthernet2/0 interface's segment, 172.16.1.0/24, and created a static NAT between GOS private IP address (192.168.1.2) and an public IP address (172.16.1.202). The configuration is:
ip nat inside source static 192.168.1.2 172.16.1.202
After that you just need to exit configuration mode and restart GOS. The boot can take around 5 minutes. The command is:
CGR1240-IOX# guest-os 1 restart Restarting Guest OS ......... Done!
Initial Configuration of GOS
After configure the IOx networking, we need to validate if the environment is UP and RUNNING before configure it. The most important command that I used was:
CGR1240-IOX# show iox host list detail IOX Server is running. Process ID: 301 Count of hosts registered: 1 Host registered: =============== IOX Server Address: FE80::D68C:B5FF:FEA2:F049; Port: 22222 Link Local Address of Host: FE80::D68C:B5FF:FEA2:F04C IPV4 Address of Host: 192.168.1.2 IPV6 Address of Host: fe80::d68c:b5ff:fea2:f04c Client Version: 0.1 Session ID: 1 OS Nodename: CGR1240-IOX-GOS-1 Host Hardware Vendor: Cisco Systems, Inc. Host Hardware Version: 1.0 Host Card Type: not implemented Host OS Version: 1.26 OS status: RUNNING Interface Hardware Vendor: None Interface Hardware Version: None Interface Card Type: None Applications Registered: ======================= Count of applications registered by this host: 0
On the output above you can see many information about the GOS, including its link-local IPv6 internal management address and also its IPv4 private address, received from IOS. The hostname is dynamically assigned, adding the prefix "-GOS-1" to the CGR hostname.
We are able to execute commands from IOS into the GOS image, and this is required for initial GOS setup. The first configuration is redefining the administrative account (root) password. In the command below I'm defining root's password as "cisco123":
CGR1240-IOX#iox host exec "resetpw cisco123" CGR1240-IOX-GOS-1 Sending command to IOX Client and waiting up to 300 seconds for response... Password reset successfully.
The second and last configuration is enabling the SSH service inside GOS. For that, we need to issue the command below:
CGR1240-IOX#iox host exec enablessh CGR1240-IOX-GOS-1 Sending command to IOX Client and waiting up to 300 seconds for response... ssh enabled successfully.
After that, you should be able to log in by SSH, using private IP address of GOS from IOS, or using NATed public address from outside networks connected to CGR.
GOS Connectivity Troubleshooting
If you get in trouble during the processes described previously, there are a few commands that may help you understand what is going on. The first one is to determine which IP address the GOS received from DHCP, which should be the first one:
CGR1240-IOX#show ip dhcp binding Bindings from all pools not associated with VRF: IP address Client-ID/ Lease expiration Type Hardware address/ User name 192.168.1.2 01d4.8cb5.a2f0.4c Mar 08 2015 05:34 PM Automatic
The second validation is to ping and arp between IOS to GOS private address. This will validate internal communication, which should be simple back-to-back scheme. The commands:
CGR1240-IOX#ping 192.168.1.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.1.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/8 ms CGR1240-IOX#show ip arp 192.168.1.2 Protocol Address Age (min) Hardware Addr Type Interface Internet 192.168.1.2 0 d48c.b5a2.f04c ARPA GigabitEthernet0/1
The third validation is to verify if NAT is working properly. For that, test a few connections (ICMP, SSH) from networks outside CGR and verify if the translations are done. The command:
CGR1240-IOX#show ip nat translations Pro Inside global Inside local Outside local Outside global tcp 172.16.1.202:22 192.168.1.2:22 10.134.15.122:56242 10.134.15.122:56242 udp 172.16.1.202:123 192.168.1.2:123 22.214.171.124:123 126.96.36.199:123 udp 172.16.1.202:51294 192.168.1.2:51294 10.97.20.54:53 10.97.20.54:53 icmp 172.16.1.202:53590 192.168.1.2:53590 10.134.15.122:53590 10.134.15.122:53590 --- 172.16.1.202 192.168.1.2 --- ---
Now it's time to play and having fun! Please understand that this is just a very simple use case, where you uses GOS as a Linux virtual machine inside CGR. However you could also develop an application and pack it into a bundle and install on GOS through IOS, but this is a theme for another talk... ;-)
root@CGR1240-IOX-GOS-1:~# uname -a Linux CGR1240-IOX-GOS-1 3.8.11-yocto-standard #1 SMP PREEMPT Thu Jun 5 22:46:43 PDT 2014 i686 root@CGR1240-IOX-GOS-1:~# ip neighbor fe80::d68c:b5ff:fea2:f049 dev eth0 lladdr d4:8c:b5:a2:f0:49 REACHABLE 192.168.1.1 dev eth0 lladdr d4:8c:b5:a2:f0:49 REACHABLE root@CGR1240-IOX-GOS-1:~# ip address show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether d4:8c:b5:a2:f0:4c brd ff:ff:ff:ff:ff:ff inet 192.168.1.2/24 scope global eth0 inet6 fe80::d68c:b5ff:fea2:f04c/64 scope link valid_lft forever preferred_lft forever 3: sit0: <NOARP> mtu 1480 qdisc noop state DOWN link/sit 0.0.0.0 brd 0.0.0.0 root@CGR1240-IOX-GOS-1:~# ip route default via 192.168.1.1 dev eth0 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.2 root@CGR1240-IOX-GOS-1:~# python Python 2.7.3 (default, Jun 5 2014, 22:58:09) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> print("I'm alive inside CGR!") I'm alive inside CGR!
- Cisco 1240 Connected Grid Router
- Research at Cisco:Fog Computing, Ecosystem, Architecture and Applications
- Cisco IOx: An Application Enablement Framework for the Internet of Things
- Cisco IOx: Making Fog Real for IoT
- Yocto Project - Embedded Linux Distribution