Friday, August 24, 2018

Git Talk 01 - Merge vs rebase

Git is a very useful tool for version control (source control) especially for software developing projects. It helps me a log when I writing the Ansible playbooks for the network automation projects. In this "git talk" series, I would like to introduce some of the interesting topics I have during the studying of Git.

In this 1st post of this series, let's talk about the difference between "git merge" and "git rebase". Generally speaking, both "git merge" and "git rebase" can combine the change from one branch into another. 

"git merge"
When using "git merge xxx" (xxx is the branch which you would like to combine from), git will merge the diff from "xxx" branch to the local branch. If conflict happens, it has to be fixed manually. 

 I will use the following graph as example. In this example, we have two branches, master and feature. These two branches both have commit A and B in the history. Then master has commit C & D. Feature has commit E & F.
 When we merge feature branch to master, the following change will happen. There is a new commit "G" appears in branch master and it combines commit D from master and commit F in feature.
Therefore, we can see the commit history of master branch has not been changed by the "git merge". Just a new commit is added with the combined changes.


"git rebase"

When using "git rebase feature", git will take the commit history of the "feature" branch which is A, B, E, F. Then add its own C and D on top. However, C and D are not the original ones as the new commit C will have the original C plus F. The same, the new commit D will contain the original D plus F. So the master branch commit history will be A, B, E, F, C, D. Although the C and D have different content, their commit time doesn't change. 


Saturday, August 11, 2018

Ansible REST API - Interacting with Cisco FirePower Management Center (FMC) - 08 - Get deployable devices and deploy policy

This post belongs to my "Ansible REST API - Interacting with Cisco FMC" series. The following is the table of content of this series:
  1. Introduction and Ansible playbook download
  2. Script flow charts
  3. Introduction of REST API and Cisco FMC API Explorer
  4. Script prerequisites
  5. Request Access Token
  6. Get policy content, modify content and "PUT' in FMC - Part 1
  7. Get policy content, modify content and "PUT' in FMC - Part 2 
  8. Get deployable devices and deploy policy
In the previous post, we have successfully upload the modified policy rule to FMC. Now we need to deploy the new policy to the edge device(s). The following are the steps:

a. get the array of the devices which do not have updated configuration (deployable devices)
b. obtain the "version" of the configuartion
c. obtain the "Device ID" of the deployable devices
d. lookup the deployment JSON template and modify its "version" and "device ID" keys
e. make a "POST" call to FMC to deploy the update configuration to the edge device(s)

First, we make a "GET" call to fetch the deployable device information from FMC.
 
  - name: Get_Deployment_Device_Info

The return of this "GET" call is information about the devices which have not applied the updated configuration. The same as other "GET" returns, it contains a "items" array in a key which named "json".

Please be noted that all of the following tasks will only be executed when the "config_has_been_changed" is equal to "true".

location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: Get_Deployment_Device_Info
    uri:
      url: "https://{{FMC_IP}}/api/fmc_config/v1/domain/{{domain}}/deployment/deployabledevices?expanded=true"
      method: GET
      headers:
        "x-auth-access-token": "{{acc_token}}"
        Connection: keep-alive
      validate_certs: no
      status_code: 200
      body_format: json
    register: DeployableDevice
    when: config_has_been_changed


- name: get device array

WE need to abstract the "items" array from the "DeployableDevice.json" key value:


 location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: get device array
    set_fact:
      device_array: "{{DeployableDevice.json | json_query('items[]')}}"
    when: config_has_been_changed


We utilise the "json_query" here to achieve this task. 

- name: get version number

In the "items" array, we will find one more multiple items. Each item contains the information for a deployable device. What we need here is the "version" key value and the "device ID" key value.

The "version" number of all the deployable devices are the same in FMC. Every time, even you just make a single change on a single device policy, the version for all the deployable devices will be updated. And this "version" number is calculated by timestamp. 

Therefore, we just need to pull the "version" number from the 1st item in this array for our task.

 location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: get version number
    set_fact:
      deployment_version: "{{device_array[0].version}}"
    when: config_has_been_changed


  - name: get the deployment device list

Here, we will create a new array which contains the deployable device ID(s).Here, we are getting all the deployable device IDs. Not only the one we just modified in the previous post.

First, we create a new empty array call "deployment_device_list". Then we add every "item.device.id" to this array as new item.

Here we also add "no_log" to this task. As the output log for this task is too much. I just don't want this log to be shown in the terminal.

 location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: get the deployment device list  # here we get all the deployable devices, NOT the policy we just changed
    set_fact:
      deployment_device_list: "{{deployment_device_list | default([])}} + ['{{item.device.id}}']"
    with_items: "{{device_array}}"
    no_log: true # don't display output for this session, since too much!
    when: config_has_been_changed


- name: get JSON POST file

In order to make the "POST" call, we need the "POST" call template. We can make this template based on the example provided by FMC API explorer. The template is shown as below:

location: /etc/ansible/roles/FMC-disable-policyrule/tasks/deployment_POST.json

{
  "type": "DeploymentRequest",
  "version": "",
  "forceDeploy": true,
  "ignoreWarning": true,
  "deviceList": []
}


As we can see, this "POST" template is very simple. Its "type" is "DeploymentRequest". Its "version" has a empty string. And the "deviceList" has a empty array.

The following is the way to use "lookup" method to pull the JSON file template.

 location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: get JSON POST file
    set_fact:
      deployment_POST: "{{lookup('file','/etc/ansible/roles/FMC-disable-policyrule/tasks/deployment_POST.json')}}"
    when: config_has_been_changed


  - name: modify the version

Since the template has a empty string as "version" value, we will need to update it with our obtained "version".

location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: modify the version
    set_fact:
      deployment_POST: "{{deployment_POST | combine({'version': deployment_version})}}"
    when: config_has_been_changed


As mentioned in the previous post, if the key exists in the JSON, "combine" filter will only update the key value. So in this case, the "version" key will be updated with our new version number.

- name: modify the deployment device list

The same, we update the "deviceList" array with our deployable device ID array:

location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: modify the deployment device list
    set_fact:
      deployment_POST: "{{deployment_POST | combine({'deviceList': deployment_device_list})}}"
    when: config_has_been_changed


- name: Deployment POST

Finally, we have our "POST" JSON ready. Then we need to make a "POST" call to the FMC and raise a device deployment request:

location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: Deployment POST
    uri:
      url: "https://{{FMC_IP}}/api/fmc_config/v1/domain/{{domain}}/deployment/deploymentrequests"
      method: POST
      headers:
        "x-auth-access-token": "{{acc_token}}"
      validate_certs: no
      status_code: 202
      body_format: json
      body: "{{deployment_POST}}"
    when: config_has_been_changed


Just pay attention to the "status_code". For this "POST" call, the return code is "202".

After making this "POST" call, our changes will be deployed to the edge firewalls. And our goal has been achieved.

Conclusion:

The purpose of writing this post series is not only to demonstrate the way how to work with Cisco FMC with Ansible. The most important thing is that we can learn about how to use ansible modules, filters and jinja2 templates to add or modify key/values in JSON and make different API calls. These are the basic skills for REST API interaction.


As more and more network devices have REST API support, if we fully master the way to interact with REST API, network automation will become a common task in our daily work. 

Reference:

Friday, August 10, 2018

Ansible REST API - Interacting with Cisco FirePower Management Center (FMC) - 07 - Get policy content, modify content and "PUT' in FMC - Part 2

This post belongs to my "Ansible REST API - Interacting with Cisco FMC" series. The following is the table of content of this series:
  1. Introduction and Ansible playbook download
  2. Script flow charts
  3. Introduction of REST API and Cisco FMC API Explorer
  4. Script prerequisites
  5. Request Access Token
  6. Get policy content, modify content and "PUT' in FMC - Part 1
  7. Get policy content, modify content and "PUT' in FMC - Part 2 
  8. Get deployable devices and deploy policy 
In the previous post, we have successfully managed to pull the content of the "URL filter" policy rule. Now we will need modify this rule and there are three steps for doing it:

a. remove the "link" and "metadata" keys from the obtain rule in JSON format
b. change the "enabled" key value to "false"
c. initial a "PUT" call to apply the policy rule update to FMC

FMC REST API can only support "PUT" calls. "PATCH" is not supported. So we need to apply the whole JSON data for the policy rule even we just need to modify a single key value of it. If you are not familiar with difference between "PUT" and "PATCH", please refer to my POST "REST API 101".

The format of this "policy rule PUT" can be checked via the FMC API Explorer. From the API Explorer examples, we find that in the PUT call, we should remove the "link" and "metadata" key from the obtained JSON of policy rule. However, there is not a filter which can remove keys from a JSON directly. So we have to build a new dictionary and add the required keys in. 

Before writing any scripts, we need to understand know about the way to modify JSON data. JSON data is actually the Dictionary in Python. And in Ansible, we will use the "combine" filter to add or modify the key values of a dictionary. You can go to this link to get more details about the "combine filter". The "combine" filter is very smart. It will check the original dictionary to see if the "key" already exists. If not, it will add a new key and new value. If yes, then it will modify the existing key value with the new one.

- name: modify the put json

location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: modify the put json
    set_fact:
      policy_rule_put: "{{policy_rule_put|default({}) | combine({item.key: item.value})}}"
    with_dict: "{{policy_rule_content}}"
    when: (item.key != "links") and (item.key != "metadata")


A couple of tasks are achieved in the above code:

  1. We set a new variable "poicy_rule_put" which is a dictionary. And make its initial value as empty.
  2. Then we call a "with_dict" loop to go through every key in the obtained policy rule ("policy_rule_content"). If the key name is not equal to "links" and "metadata", we will add the key and its value to the new dictionary "policy_rule_put".
After that, we should have the right format to make the "PUT" call. However, we have not modify the "enabled" key value yet. 

- name: modify the "enable"

location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml


  - name: modify the "enable"
    set_fact:
      policy_rule_put: "{{policy_rule_put | combine({'enabled': false})}}"
      config_has_been_changed: true
    when: policy_rule_put.enabled


In here, we use the "combine" filter again to update the "enabled" key with a new value "false" as the key "enabled" is already exist in the dictionary. 

You can also notice that I change the value of "config_has_been_changed" variable to "true" here. This "This "config_has_been_changed" was set to "false" in the var file (/etc/ansible/roles/FMC-enable-policyrule/vars/main.yml). The system will decide whether it should apply the config change to FMC based on this value.

- name: PUT_Access_Policy_roles

location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml


  - name: PUT_Access_Policy_roles
    uri:
      url: "https://{{FMC_IP}}/api/fmc_config/v1/domain/{{domain}}/policy/accesspolicies/{{policy_id}}/accessrules/{{policy_rule_put.id}}"
      method: PUT
      headers:
        "x-auth-access-token": "{{acc_token}}"
      validate_certs: no
      status_code: 200
      body_format: json
      body: "{{policy_rule_put}}"
    when: config_has_been_changed


This is the "PUT" call which modifies the "URL Filter" policy rule in FMC. Again, we put the "acc_token" in the "headers" field. And the "policy_rule_put" in the body field. The success return code is "200". And this task will only be executed when "config_has_been_changed" is "true".

Conclusion:

Now, we have successfully modified the FMC policy rule. But if you are familiar with Cisco FMC, you will know the actual firewall (SFR or FTD) configuration has not yet been updated at this stage. We will need to deploy the updated configuration to the edge devices. I am going to talk about how to deploy the config to the particular edge device with Ansible in the next POST which is also the last one in this series. 

Reference:

Tuesday, August 7, 2018

Ansible REST API - Interacting with Cisco FirePower Management Center (FMC) - 06 - Get policy content, modify content and "PUT' in FMC - Part 1

This post belongs to my "Ansible REST API - Interacting with Cisco FMC" series. The following is the table of content of this series:
  1. Introduction and Ansible playbook download
  2. Script flow charts
  3. Introduction of REST API and Cisco FMC API Explorer
  4. Script prerequisites
  5. Request Access Token
  6. Get policy content, modify content and "PUT' in FMC - Part 1
  7. Get policy content, modify content and "PUT' in FMC - Part 2 
  8. Get deployable devices and deploy policy
In the last post in this series, I introduced the way to obtain the Access Token. Now we are ready to do some "real work" with the FMC REST APIs. In this post, we will modify the value of the JSON key of a policy rule. There are a couple of key steps to achieve this task:


a. Locate the policy ID which named "FTD-1-Access-Policy-SYD".
b. Locate the policy rule ID name "Application Filter" under the above policy.
c. Modify the value of the "enable" key in the above policy rule.


Click to Expand
As we know, every operation in FMC API is based on the object ID which is a system generated string. We cannot make changes by reference the configured name directly. So in our Ansible playbook, we need to search for the ID for Policy Name "FTD-1-Access-Policy-SYD". In the following, I will describe each name task in details:

- name: Get_Access_Policy

The first task is to obtain the whole JSON file of the "Access Policies". 

The "url" for this operation can be find in the FMC API Explorer. If you are not familiar with it, please go back to my previous post Introduction of REST API and Cisco FMC API Explorer. The {{FMC_IP}} & {{domain}} are vars in "/etc/ansible/roles/FMC-enable-policyrule/vars/main.yml".

Also, in this task, we put the obtained "acc_token" in the headers field for authentication purpose. The return code for this "GET" call is "200".

location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: Get_Access_Policy
    uri:
      url: "https://{{FMC_IP}}/api/fmc_config/v1/domain/{{domain}}/policy/accesspolicies"
      method: GET
      headers:
        "x-auth-access-token": "{{acc_token}}"
        Connection: keep-alive
      validate_certs: no
      status_code: 200
      body_format: json
    register: accesspolicies


After having the whole JSON file, we need to check what's the structure of this file. Under the "json", there is a "items" array. And the "accessPolicy" details including the "id" are in this array. 

    "device_list": {
        "accept_ranges": "bytes",
        "cache_control": "no-cache, no-store, must-revalidate, max-age=0",
        "changed": false,
        "connection": "close",
        "content_type": "application/json",
        "cookies": {},
        "date": "Sun, 24 Jun 2018 00:54:34 GMT",
        "failed": false,
        "json": {
            "items": [
                {
                    "accessPolicy": {
                        "id": "000c2911-ed30-0ed3-0000-008589936316",
                        "name": "FTD-1-Access-Policy-SYD",
                        "type": "AccessPolicy"

                    },
        ...

 - name: get policy_json

In this step, we will pull the "json" key value which includes the "items" array:


location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: get policy_json
    set_fact:
      policy_json: "{{accesspolicies.json}}"


- name: get policy array

Here we build a new array variable to hold the "items" array:


location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: get policy array
    set_fact:
      policy_array: "{{policy_json | json_query('items[]')}}"


In the above task, we use the "json_query" method. If you want to know more about this method, please refer to the detailed description of "json_query" in Ansible documentation.

 - name: get policy_id

In the above step, we got the access policy array. In this array, we may have multiple access policy items. The following shows the data structure of this array:
Click to Expand
So we need to search the ID which belongs to the policy named "FTD-1-Access-Policy-SYD". I achieve this searching operation by using the Jinja2 template.


location: /etc/ansible/roles/FMC-enable-policyrule/tasks/get_policy_id.j2

{%- for item in policy_array -%}
  {%- if item.name == policy_name -%}
    {{ item.id }}
  {%- endif -%}
{%- endfor -%}


This jinja2 code is very straight forward. Basically it use a "for loop" to go through all the items in this array. If the "item.name" is equal to the "policy_name" variable defined in the VAR file ("/etc/ansible/roles/FMC-enable-policyrule/vars/main.yml"), it will return all the "item.id" value. 

location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: get policy_id
    set_fact:
      policy_id: "{{lookup('template','/etc/ansible/roles/FMC-disable-policyrule/tasks/get_policy_id.j2') | trim}}"  #"trim" will remove the "\n" retrun from the j2 file


Here we use the Ansible "lookup template" to call the Jinja2 file. The "trim" keyword is used to get rid of the "\n" returned from Jinja2's return. 

The method mentioned above is very useful for fetching the useful information from the JSON configuration file of a network device. For example, a network device has multiple interfaces. Each interface has it's own configuration such as IP address, duplex/speed etc.  All the network interface configurations will be put in the same array. By using the jinja2 template, we can go through all the config items in this array and get the interface config based on the particular interface name or description.

 - name: Get_Access_Policy_rules

Now we have the policy ID for "FTD-1-Access-Policy-SYD". We can pull the policy rules in this policy and find out the specific rule we want to modify.


location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: Get_Access_Policy_rules
    uri:
      url: "https://{{FMC_IP}}/api/fmc_config/v1/domain/{{domain}}/policy/accesspolicies/{{policy_id}}/accessrules?expanded=true"
      method: GET
      headers:
        "x-auth-access-token": "{{acc_token}}"
      validate_certs: no
      status_code: 200
      body_format: json
    register: accesspolicierules


I won't talk much about this as it's very similar to "- name: Get_Access_Policy" above.

  - name: get policy_rule_json

Similar to  "- name: get policy_json"

location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: get policy_rule_json
    set_fact:
      policy__rule_json: "{{accesspolicierules.json}}"


- name: get policy rule array

Similar to "- name: get policy array"


location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: get policy rule array
    set_fact:
      policy_rule_array: "{{policy__rule_json | json_query('items[]')}}"


 - name: get policy_rule_content

Similar to " - name: get policy_id". However, in the Jinja2 file, we return the whole content of the item instead of only the "id".


location: /etc/ansible/roles/FMC-enable-policyrule/tasks/get_policy_rule_id.j2

{%- for item in policy_rule_array -%}
  {%- if item.name == policy_rule_name -%}
    {{ item }}
  {%- endif -%}
{%- endfor -%}


The "policy_rule_name" we are going to search is "URL Filter"


location: /etc/ansible/roles/FMC-enable-policyrule/tasks/main.yml

  - name: get policy_rule_content
    set_fact:
      policy_rule_content: "{{lookup('template','/etc/ansible/roles/FMC-disable-policyrule/tasks/get_policy_rule_id.j2')}}"  #"trim" will remove the "\n" retrun from the j2 file


The return will be something like this:

    "policy_rule_content": {
        "action": "BLOCK_RESET",
        "enabled": true,
        "id": "000C2911-ED30-0ed3-0000-000268436480",
 ...
        },
        "name": "URL Filter",
        "sendEventsToFMC": true,
...

The "enabled": true above is the key value we want to modify. 

We have the "URL Filter" policy rule content now. In next step, I will modify this policy content to be the right format for a "PUT" call to modify the policy rule and this will be described in "Get policy content, modify content and "PUT' in FMC - Part 2".






NSX Load Balancer "Application Rules" Examples:

Load Balancing is one of the features provided by the NSX Edge Services Gateway (ESG). It can provide L7 Load Balancing by utilizing the HA...