How to Integrate External Data with the Ansible Lookup

Published:23 February 2022 - 6 min. read

Azure Cloud Labs: these FREE, on‑demand Azure Cloud Labs will get you into a real‑world environment and account, walking you through step‑by‑step how to best protect, secure, and recover Azure data.

While running Ansible playbooks, have you ever wondered how to retrieve data from outside sources, such as files, databases, key/value stores, APIs, and other services? If yes, then exploring Ansible lookup will be worth your while.

In this tutorial, you’re going to learn everything about Ansible lookups and how to work with them to fetch external data.

Read on and start integrating data!

Prerequisites

This tutorial comprises step-by-step instructions. If you’d like to follow along, be sure you have the following in place:

  • A remote Linux computer to test the tomcat installation – This tutorial uses Ubuntu 20.04.3 LTS as the remote node.
  • An inventory file and one or more hosts configured to run Ansible commands and playbooks. The remote Linux computer is called myserver, and this tutorial uses an inventory group called web.
  • NGINX installed on Ansible controller host.
  • Python v3.6 or later installed both on your Ansible controller host and the remote node machine – This tutorial uses Python v3.9 on an Ubuntu machine.

Searching for a File using Ansible lookup

You typically don’t look up a file and expect that all goes well. What if Ansible can’t find the file, or it doesn’t exist? Use Ansible lookup in Ansible debug module to manage how errors should treat Ansible tasks.

The Ansible debug module prints the statements when you execute an Ansible playbook. In a nutshell, the module allows you to debug variables or expressions without necessarily stopping the playbook.

1. SSH into your Ansible controller host with a user you typically use to manage Ansible.

2. Next, create a directory called ansible_lookup_playbook_demo in your home directory. This directory will contain the playbook you’ll invoke later in the section.

mkdir ~/ansible_lookup_playbook_demo
cd ~/ansible_lookup_playbook_demo

3. Create a file called main.yml in the ~/ansible_become_playbook_demo directory and copy/paste the following YAML playbook contents.

---
- name: Ansible Lookup functionality with Ansible debug module demo
  hosts: web
# Executing the tasks with remote_user as ubuntu
  remote_user: ubuntu
  tasks:
    - name: Task ignores the error if shanky.txt does not exist
      ansible.builtin.debug:
         msg: "{{ lookup('file', '/shanky.txt', errors='ignore') }}"

    - name: Task gives warning but continues even if shanky.txt does not exist.
      ansible.builtin.debug:
        msg: "{{ lookup('file', '/shanky.txt', errors='warn') }}"

    - name: Task fails if file doesnt exists
      ansible.builtin.debug:
        msg: "{{ lookup('file', '/shanky.txt', errors='strict') }}"
  1. Finally, run the below command to invoke the playbook (main.yml). Ansible then executes the tasks in the playbook.

Validating the Ansible playbook using the --check flag with the ansible-playbook command is a good practice before actually executing the playbook. The --check flag tells Ansible to perform a simulation without running the playbook.

ansible-playbook main.yml 

Below, you can see that all the TASK has an OK status, which indicates the task was not required to execute. Each task did not execute as Ansible couldn’t find the shanky.txt file, and no change was required.

Executing the Ansible Playbook (main.yml)
Executing the Ansible Playbook (main.yml)

Fetching Environment Variables

Earlier in the previous section, you learned how to work with files using Ansible lookup. But at times, you need to read the value of environment variables which are further used in querying URLs or posting as the header or the data in the URL.

Create a file called main2.yml in the *~/*ansible_become_playbook_demo directory and paste in the following YAML playbook contents.

The below Ansible playbook executes two tasks:

  • The first task checks for the HOME environment variable on your ubuntu machine and prints the path of that variable.
  • Similarly, the second task looks for the USR environment variable but displays the no-user-found default value as the variable doesn’t exist.
---
- name: Ansible Lookup with Ansible env module demo
  hosts: web
# Executing the tasks with remote_user as ubuntu
  remote_user: ubuntu
  tasks:
    - name: Checking for enviornment variable HOME
      ansible.builtin.debug:
        msg:  "'{{ lookup('env', 'HOME') }}' is the HOME environment variable."

    - name: Checking for enviornment variable USR
      ansible.builtin.debug:
# Ansible task checks for USR variable, if not found takes the default value no-user-found
        msg: "'{{ lookup('env', 'USR') | default('no-user-found', True) }}' is the user."

Once the playbook execution completes, you’ll see that the first task displays the environment path. But in the second task, Ansible can’t find the USR user, so Ansible displays the no-user-found message.

Executing an Ansible Playbook to Fetch Environment Variables
Executing an Ansible Playbook to Fetch Environment Variables

Reading Website Contents

You previously learned to retrieve data from environment variables stored on your Ubuntu machine using Ansible lookup with Ansible’s built-in debug module.

But did you know Ansible lookup works well with multiple other Ansible modules? Some of these Ansible modules are ansible.builtin.config, ansible.builtin.template, ansible.builtin.uri, and so on.

To find out all the modules that work with Ansible lookup, run the ansible-doc -l -t lookup command.

For this demo, you’ll work on one of the most widely used Ansible modules, the Ansible URL module (ansible.builtin.uri), and learn how it integrates with Ansible lookup. Ansible URL module returns the content of a requested URL to be used as data in an Ansible playbook.

Create a file called main3.yml in the *~/*ansible_become_playbook_demo directory and paste in the following YAML playbook contents.

The ansible-playbook below contains tasks that send requests to a local HTML page and a website to retrieve data.

---
- name: Ansible Lookup functionality with Ansible URL module demo
# Executing the tasks on the inventory group (web)
  hosts: web
# Executing the tasks with remote_user as ubuntu
  remote_user: ubuntu
  tasks:
    - name: Task sends the request to index.html page on nginx installed locally to retrive the data
      ansible.builtin.debug:
        msg: "{{ lookup('url', '<http://localhost:80/index.html>')}}"
    - name: Task sends the request to adamtheautomator website and retrive the data
      ansible.builtin.debug:
        msg: "{{ lookup('url', '<https://adamtheautomator.com/resources/>', username='U', password='p') }}"

Now, run the below command to execute the Ansible playbook (main3.yml), retrieving data from a local page and a live website.

ansible-playbook main3.yml

After successful playbook execution, you’ll see both tasks retrieved and printed data (source code) of both the local page and the adamtheautomator.com website.

Requesting and Retrieving Data from Both Local and Live Website
Requesting and Retrieving Data from Both Local and Live Website

Returning Key/Value Pairs with Ansible Dictionary and lookup

At times you need to have some environment variables declared in dictionaries format that is key: value format. Reading those values could be tricky, but the combination of Ansible lookup and Ansible dictionary works the magic. Let’s see it in action.

Replace the content of the main.yaml playbook you created in the “Searching for a File using the Ansible lookup” section with the content below.

In the below Ansible Playbook, Ansible lookup fetches the dictionary in one go; then Ansible dictionary returns a list with each item Ansible lookup requested.

# Declaring the variable users that further contains two users (alistek and abertram)
vars:
  users:
    alistek:
# users contain two keys: name and telephone
# Values of the key name are: Adamlistek and AdamBertram
# Values of the key telephone are 1234567890 and 1122334455.
      name: Adamlistek
      telephone: 1234567890
    abertram:
      name: AdamBertram
      telephone: 1122334455
# Ansible task that uses a lookup to display the user's information from the dictionary
tasks:
  - name: Print phone number of employees
    debug:
      msg: "User {{ item.key }} is {{ item.value.name }} ({{ item.value.telephone }})"
    loop: "{{ lookup('dict', users) }}"

Now, run the ansible-playbook command below to execute the main.yml playbook.

ansible-playbook main.yml

As you see below, the Ansible task prints the dictionary items with their values retrieved from the users variable.

Returning Key/Value Pairs in Dictionary Form
Returning Key/Value Pairs in Dictionary Form

Retrieving DNS records with Ansible dig

Previously you learned how the Ansible lookup plugin interlinks with various Ansible modules such as Ansible Dictionary, Ansible debugs, Ansible templates, etc. But there are dozens of other Ansible modules that work well with the Ansible Lookup such as dig plugin. Let’s quickly check out how to work with Ansible dig.

To find the list of all the Ansible lookup Plugins that are locally installed on your Ansible controller run the ansible-doc -t lookup -l command.

Again, replace the content of the main.yaml playbook you created in the previous section with the content below. In the below Ansible Playbook, Ansible dig retrieves the DNS records for google.com that Ansible lookup requested.

---
- name: Ansible dig functionality with Ansible Lookup
  hosts: localhost
# Executing the tasks on the local machine
  tasks:
    - name: Retriving DNS records for google.com
      debug:
        msg: "{{ lookup ('dig', 'google.com')}}"

After you execute ansible playbook as you can the DNS record of google.com is displayed on the output.

DNS record of google.com using Ansible Dig and Lookup
DNS record of google.com using Ansible Dig and Lookup

Conclusion

In this tutorial, you’ve taken advantage of the Ansible lookup parameter to read external data from various external sources, such as websites environment variables with a single command.

Now that you have sound knowledge of the Ansible lookup, which data do you plan to retrieve next on your machine with the Ansible playbook?

Hate ads? Want to support the writer? Get many of our tutorials packaged as an ATA Guidebook.

Explore ATA Guidebooks

Looks like you're offline!