GRE Tunnels Unleashed - Making , Breaking & Troubleshooting

Generic Routing Encapsulation (GRE) is a tunneling protocol 
that can encapsulate a wide variety of network layer protocol 
packet types inside an IP tunnels, creating a virtual point-to
-point link to various brands of routers at remote points over 
an Internet Protocol (IP) internetwork. 

It always kind of amazed me but in the beginning of my career in network I had lot of problems in understanding and configuring GRE tunnels. Maybe the reason is that this topic is not very well documented on Cisco DOC cd - Atleast that's what I think.

The problem for me was never to configure the tunnel itself but rather understanding how to choose tunnel source vs tunnel destination, how GRE tunnels works from technology point of view, their benefits, their drawbacks, understanding GRE keep-alive mechanism & what things can potentially cause problems with tunnels etc.

Some Tiny details before getting deeper:

GRE refers to IP protocol number type 47

RFC - 1702

Types - P2P, P2M (eg. - DMVPN - More Of a Security Topic)

So let's begin with it and let me explain each of these things one by one.

1.) Why we need GRE tunnels in First place - 

GRE tunnels are helpful for a network administrator in many ways actually. For example if you are running a routing protocol or may be static routing with your ISP. But you don't want ISP to see your internal network. So in that case with the help of Static Routing or IGP (that ISP is running too with you) you will establish basic connectivity between your branches with help of ISP. Now later you can create tunnels between all branches with help of basic connectivity you had established earlier and then you can run routing protocol over the tunnel & can exchange routes. Now technically once GRE tunnel is established between your branch offices then ISP can't see what data is traveling inside the tunnel....or in other words..your routes & data.

Another situation where we can probably use GRE tunnel is to pass OSPF LSAs through Stub/Totally Stub Area. For example once Area is configured as stub, by default it will not allow you to create virtual link over it or to pass LSAs which it is supposed to block as Stub area but some how you need do these things....I mean you get the idea...there are many other twists in Real world and CCIE lab exam where you can think of using tunnels...so basically a bandage which you can use in many situations. There are many more reasons but these are most common one

2.) Requirements to Setup the tunnel:

The requirement to setup tunnel is very simple. All you need is the reach-ability between the two end points of the tunnel. Usually you will see people using loopback interface as tunnel source address but technically you can choose any interface as source at your side but destination will always be some ip address of other side and can't be interface...that actually makes sense too as Local Router don't know about what interfaces we have on the other side. 

But one thing you should keep in mind while choosing any interface as source address is that by default it will pick the primary IP of the interface, so in case you want to choose secondary IP then mention Secondary IP address specifically as source ip address.

3.) Drawbacks Of GRE tunnels :

There is only one major drawback of GRE tunnel and that is - it adds two extra headers to our original IP packet. So those guys who are concerned with Bandwidth may have some issues. 

However if you change the tunnel mode to "IPIP" then you can reduce some overhead as "IPIP" tunnel adds 1 header instead of 2 compare to "GRE". So that makes it more efficient. 

Here is how process occurs for GRE

You can also use other method to prevent bandwidth like traditional TCP compression techniques or modern MQC based techniques.

Another drawback is that GRE tunnels are not very scalable solution. The default BW for GRE tunnels is just 9 Kbps with MTU size 1514 byes, so your routing protocol may not follow the path you desire to, so watch out for such things in that case.

Enough theory I guess :-)

For rest of theoretical details I'll post some URL for further reference.

Now lets create a GRE tunnel.

Here is the quick topology Diagram:-> Click on Image for better view

Step 1:

Establish the basic connectivity between tunnel source and destination points. In our case we will use R1's Lo0 interface as tunnel source and R3's interface Lo0 (It's IP Address) as our tunnel destination point.

To establish the reach-ability between these two end point we will be running OSPF with our ISP Router(R2).

So lets first establish basic connectivity between end point:

################

***R1***

!
en
!
conf t
!
no ip domain-lo
!
ho R1
!
int lo0
ip add 10.0.0.1 255.255.255.255
exit
!
int lo1
ip add 100.100.100.100 255.255.255.0
exit
!
int s0/0
ip add 12.12.12.1 255.255.255.0
no sh
exit
!
router ospf 1
router-id 1.1.1.1
net 10.0.0.1 0.0.0.0 a 0
net 12.12.12.1 0.0.0.0 a 0
exit
!

##########################
 

***R2***

!
en
!
conf t
!
no ip domain-lo
!
ho R2
!
int s0/0
ip add 12.12.12.2 255.255.255.0
ip ospf 1 a 0
no sh
exit
!
int s0/1
ip add 23.23.23.2 255.255.255.0
ip ospf 1 a 0
no sh
exit
!

##########################

***R3***

!
en
!
conf t
!
no ip domain-lo
!
int lo0
ip add 20.0.0.1 255.255.255.255
exit
!
int lo1
ip add 200.200.200.200 255.255.255.0
exit
!
int s0/0
ip add 23.23.23.3 255.255.255.0
no sh
exit
!
router ospf 1
router-id 3.3.3.3
net 20.0.0.1 0.0.0.0 a 0
net 23.23.23.3 0.0.0.0 a 0
exit
!

----

So by this time our OSPF should be working fine and R1/R3's Interface Lo0 should have reach-ability to each other.

Step 2:

In second step we will create a virtual tunnel interface and will assign it following things:

1. IP address

2. Tunnel Source 

3. Tunnel Desitnation

Here is the quick config now:

***R1***

!
int tu 0
ip address 13.13.13.1 255.255.255.0
tunnel source Loopback0
tunnel destination 13.13.13.3
tunnel destination 20.0.0.1
keepalive 1 3 -> Optional
exit
!

***R3***

!
int tu 0
ip address 13.13.13.3 255.255.255.0
keepalive 1 3 -> Optional
tunnel source Loopback0
tunnel destination 10.0.0.1
exit
!

So by now your tunnel should have come up :-)

Step 3:

 

Now once tunnel is up and running we can can RUN our Organizational IGP over the tunnel to establish complete connectivity among all locations.

In our case Lo1 interface over R1/R3 represents their LAN segment and IGP we are running inside organization is EIGRP AS 100.

Lets configure EIGRP for Interface Lo1 on both routers along with enabling it over Tunnel interface.

***R1***

!
router eigrp 100
no au
net 100.100.100.100 0.0.0.0
net 13.13.13.1 0.0.0.0
exit
!

***R3***

!
router eigrp 100
no au
net 200.200.200.200 0.0.0.0
net 13.13.13.3 0.0.0.0
exit
!

################

Lets verify the routing tables as see if it's working as expected:

R1(config-router)#do sh ip ro
Gateway of last resort is not set

D    200.200.200.0/24 [90/297372416] via 13.13.13.3, 00:01:38, Tunnel0
     100.0.0.0/24 is subnetted, 1 subnets
C       100.100.100.0 is directly connected, Loopback1
     20.0.0.0/32 is subnetted, 1 subnets
O       20.0.0.1 [110/129] via 12.12.12.2, 00:10:24, Serial0/0
     23.0.0.0/24 is subnetted, 1 subnets
O       23.23.23.0 [110/128] via 12.12.12.2, 00:10:24, Serial0/0
     10.0.0.0/32 is subnetted, 1 subnets
C       10.0.0.1 is directly connected, Loopback0
     12.0.0.0/24 is subnetted, 1 subnets
C       12.12.12.0 is directly connected, Serial0/0
     13.0.0.0/24 is subnetted, 1 subnets
C       13.13.13.0 is directly connected, Tunnel0

***********************************

R2(config-router)#do sh ip ro -> ISP Router Dont's have info about Client's EIGRP subnets
Gateway of last resort is not set

     20.0.0.0/32 is subnetted, 1 subnets
O       20.0.0.1 [110/65] via 23.23.23.3, 00:11:33, Serial0/1
     23.0.0.0/24 is subnetted, 1 subnets
C       23.23.23.0 is directly connected, Serial0/1
     10.0.0.0/32 is subnetted, 1 subnets
O       10.0.0.1 [110/65] via 12.12.12.1, 00:12:01, Serial0/0
     12.0.0.0/24 is subnetted, 1 subnets
C       12.12.12.0 is directly connected, Serial0/0


************************************

R3(config-router)#do sh ip ro
Gateway of last resort is not set

C    200.200.200.0/24 is directly connected, Loopback1
     100.0.0.0/24 is subnetted, 1 subnets
D       100.100.100.0 [90/297372416] via 13.13.13.1, 00:03:33, Tunnel0
     20.0.0.0/32 is subnetted, 1 subnets
C       20.0.0.1 is directly connected, Loopback0
     23.0.0.0/24 is subnetted, 1 subnets
C       23.23.23.0 is directly connected, Serial0/0
     10.0.0.0/32 is subnetted, 1 subnets
O       10.0.0.1 [110/129] via 23.23.23.2, 00:12:13, Serial0/0
     12.0.0.0/24 is subnetted, 1 subnets
O       12.12.12.0 [110/128] via 23.23.23.2, 00:12:13, Serial0/0
     13.0.0.0/24 is subnetted, 1 subnets
C       13.13.13.0 is directly connected, Tunnel0

So congratulations on building GRE P2P tunnel successfully :-)

#####################################

Now One Important thing you should always remember while building GRE tunnels is that - Tunnel Source and Destination should always be learned outside the tunnel and not from inside the tunnel....confused ?

Let me show it to you guys.

Right now Tunnel Source and Destinations (Which are Lo0 of R1/R3) are learned via ospf and inside the tunnel we are running EIGRP. But say if I enable EIGRP on Lo0 over R1 & R2 then by means of Basic routing fundamental, EIGRP will start advertising the Lo0 interfaces on both routers and soon router will update it's routing table and will install EIGRP routes for Lo0s. Because EIGRP has less AD compare to OSPF. Now we are learning Tunnel Source & Destination From Tunnel itself which leads us to chicken & egg problem. :-)

so lets first introduce this problem and see how we can fix it.

#####################################################

R1
 

!
router eigrp 100
net 10.0.0.1 0.0.0.0
exit
!

#####################################################

R3

!
router eigrp 100
net 20.0.0.1 0.0.0.0
exit
!

As soon as we do this, we will start seeing following error messages on screen:

#####################################################

R3(config-router)#
*Mar  1 00:19:47.735: %TUN-5-RECURDOWN: Tunnel0 temporarily disabled due to recursive
 routing
*Mar  1 00:19:48.735: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, change
d state to down
*Mar  1 00:19:48.875: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 100: Neighbor 13.13.13.1 (Tunnel
0) is down: interface down!

####################################################

soon after:

*Mar  1 00:20:49.747: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, change
d state to up
*Mar  1 00:20:54.131: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 100: Neighbor 13.13.13.1 (Tunnel
0) is up: new adjacency
*Mar  1 00:20:55.747: %TUN-5-RECURDOWN: Tunnel0 temporarily disabled due to recursive
 routing
*Mar  1 00:20:56.747: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, change
d state to down
*Mar  1 00:20:56.851: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 100: Neighbor 13.13.13.1 (Tunnel
0) is down: interface down
######################################################

So basically what you will notice is tunnel is coming up and going down , up and down.....

Also you will notice that it's telling you about route recursion issue. Earlier route for tunnel source and destination was re-cursing to OSPF next hop but now it sees as EIGRP next hop pointing to tunnel itself...so it recognizes Chicken and Egg problem and bounce the tunnel assuming that there is a routing issue and bouncing tunnel may solve it :-)

To solve this problem there are many ways actually.

e.g. - Roll back to previous config, Filter tunnel source & destination IPs from the protocol running over tunnel using AD or ACL or Prefix-List etc.

Here is how I fixed it by filtering tunnel Source & Destination IPs using a distribute list:

R1

!
access-list 100 deny ip ho 13.13.13.3 20.0.0.0 0.255.255.255 log
access-l 100 permit ip any any log
!
router eigrp 100
distribute-list 100 in Tunnel0
exit
!

######################################################

R3

!
access-list 100 deny ip ho 13.13.13.1 10.0.0.0 0.255.255.255 log
access-l 100 permit ip any any log
!
router eigrp 100
distribute-list 100 in Tunnel0
exit
!

And sure enough this will solve the problem:

Just as a side note in case you are new with distribute list - In the ACL, the first portion defined after host keyword is next hop IP address from which we are assuming to receive update and second portion is actual network which we need to filter. In my example I filter the entire network though of that range but can go as much specific as you want.

More on GRE -

http://www.cisco.com/en/US/docs/ios/interface/configuration/guide/ir_impl_tun_ps6441_TSD_Products_Configuration_Guide_Chapter.html

http://en.wikipedia.org/wiki/Generic_Routing_Encapsulation

http://deepakarora1984.blogspot.com/search/label/GRE

http://packetlife.net/blog/2008/jul/11/visualizing-tunnels/

http://packetlife.net/blog/2012/feb/27/gre-vs-ipip-tunneling/

http://www.mplsvpn.info/2009/02/gre-keepalives.html

http://www.mplsvpn.info/search/label/GRE%20Tunnel

http://blog.ipexpert.com/2010/05/21/exploring-the-frame-relay-cloud/

http://cciethebeginning.wordpress.com/2008/06/29/gre-generic-routing-encapsulation-point-to-point-multipoint-gre/

http://cciethebeginning.wordpress.com/2008/07/03/manual-ipv6-gre-tunnel-over-ipv4/

http://cciethebeginning.wordpress.com/2010/07/26/load-balancing-gre-tunneled-multicast-traffic-3/

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_tech_note09186a0080093f1f.shtml

http://blog.ine.com/2012/08/17/otv-decoded-a-fancy-gre-tunnel/ 

HTH...

Deepak Arora