o

�N�gH�@s�dZd
dl
Z
d
dlZd
dlZd
dlZd
dlZd
dlZd
dlZd
dl	Zd
dl
Zd
dlZd
dlZe�
e�
Zdd�Zdd�Z					ddd	�
Z								dd
d�
Zdd
�ZdS)ao
The ``file_tree`` external pillar allows values from all files in a directory
tree to be imported as Pillar data.

.. note::

    This is an external pillar and is subject to the :ref:`rules and
    constraints <external-pillars>` governing external pillars.

.. versionadded:: 2015.5.0

In this pillar, data is organized by either Minion ID or Nodegroup name.  To
setup pillar data for a specific Minion, place it in
``<root_dir>/hosts/<minion_id>``.  To setup pillar data for an entire
Nodegroup, place it in ``<root_dir>/nodegroups/<node_group>`` where
``<node_group>`` is the Nodegroup's name.

Example ``file_tree`` Pillar
============================

Master Configuration
--------------------

.. code-block:: yaml

    ext_pillar:
      - file_tree:
          root_dir: /srv/ext_pillar
          follow_dir_links: False
          keep_newline: True

The ``root_dir`` parameter is required and points to the directory where files
for each host are stored. The ``follow_dir_links`` parameter is optional and
defaults to False. If ``follow_dir_links`` is set to True, this external pillar
will follow symbolic links to other directories.

.. warning::
    Be careful when using ``follow_dir_links``, as a recursive symlink chain
    will result in unexpected results.

.. versionchanged:: 2018.3.0
    If ``root_dir`` is a relative path, it will be treated as relative to the
    :conf_master:`pillar_roots` of the environment specified by
    :conf_minion:`pillarenv`. If an environment specifies multiple
    roots, this module will search for files relative to all of them, in order,
    merging the results.

If ``keep_newline`` is set to ``True``, then the pillar values for files ending
in newlines will keep that newline. The default behavior is to remove the
end-of-file newline. ``keep_newline`` should be turned on if the pillar data is
intended to be used to deploy a file using ``contents_pillar`` with a
:py:func:`file.managed <salt.states.file.managed>` state.

.. versionchanged:: 2015.8.4
    The ``raw_data`` parameter has been renamed to ``keep_newline``. In earlier
    releases, ``raw_data`` must be used. Also, this parameter can now be a list
    of globs, allowing for more granular control over which pillar values keep
    their end-of-file newline. The globs match paths relative to the
    directories named for minion IDs and nodegroups underneath the ``root_dir``
    (see the layout examples in the below sections).

    .. code-block:: yaml

        ext_pillar:
          - file_tree:
              root_dir: /path/to/root/directory
              keep_newline:
                - files/testdir/*

.. note::
    In earlier releases, this documentation incorrectly stated that binary
    files would not affected by the ``keep_newline`` configuration.  However,
    this module does not actually distinguish between binary and text files.

.. versionchanged:: 2017.7.0
    Templating/rendering has been added. You can now specify a default render
    pipeline and a black- and whitelist of (dis)allowed renderers.

    ``template`` must be set to ``True`` for templating to happen.

    .. code-block:: yaml

        ext_pillar:
          - file_tree:
            root_dir: /path/to/root/directory
            render_default: jinja|yaml
            renderer_blacklist:
              - gpg
            renderer_whitelist:
              - jinja
              - yaml
            template: True

Assigning Pillar Data to Individual Hosts
-----------------------------------------

To configure pillar data for each host, this external pillar will recursively
iterate over ``root_dir``/hosts/``id`` (where ``id`` is a minion ID), and
compile pillar data with each subdirectory as a dictionary key and each file
as a value.

For example, the following ``root_dir`` tree:

.. code-block:: text

    ./hosts/
    ./hosts/test-host/
    ./hosts/test-host/files/
    ./hosts/test-host/files/testdir/
    ./hosts/test-host/files/testdir/file1.txt
    ./hosts/test-host/files/testdir/file2.txt
    ./hosts/test-host/files/another-testdir/
    ./hosts/test-host/files/another-testdir/symlink-to-file1.txt

will result in the following pillar tree for minion with ID ``test-host``:

.. code-block:: text

    test-host:
        ----------
        apache:
            ----------
            config.d:
                ----------
                00_important.conf:
                    <important_config important_setting="yes" />
                20_bob_extra.conf:
                    <bob_specific_cfg has_freeze_ray="yes" />
        corporate_app:
            ----------
            settings:
                ----------
                common_settings:
                    // This is the main settings file for the corporate
                    // internal web app
                    main_setting: probably
                bob_settings:
                    role: bob

.. note::

    The leaf data in the example shown is the contents of the pillar files.
�Nc

Cst�
d
|j|j�
dS)z.
    Log salt.utils.path.os_walk() error.
    z%s: %sN)�log�error�filename�strerror)
�err�r�I/opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/file_tree.py�_on_walk_error�sr	c	Csnt|t
�r|Stj�||
�}|D]#}zt�||�rW
d
SWqty4


t�|t|�
�r2Y
d
SYqwdS)a`

    Return a boolean stating whether or not a file's trailing newline should be
    removed. To figure this out, first check if keep_newline is a boolean and
    if so, return its opposite. Otherwise, iterate over keep_newline and check
    if any of the patterns match the file path. If a match is found, return
    False, otherwise return True.
    FT)�
isinstance�bool�os�path�join�fnmatch�	TypeError�str)�prefix�	file_name�keep_newline�	full_path�patternrrr�_check_newline�s






�

��rFcCs�
i}tj
�tt�}tj�|�
}	tjjj	|d
t
|
d�D]�\}
}}|}
tj�|
�
}tj�||	�}||	krSg}|}|rHtj�|�
\}}|�
d|�
|s8|rS|
|�d�
}
|sJ|D]}i|
|<qU|D]�}tj�|
|�}tj�|�
stt�d|�
q^d}zAtjj�|d��/}|�td�
}|r�||7}|�td�
}|s�|�d�
r�t|||�r�|d	d
�}Wd	�
n1s�w



Y
Wnty�
}
zt�d||j�
WYd	}~q^d	}~w
w|}|d
ur�tjjtjj�|�
||||d�}tjj�|�
r�|� �|
|<q^||
|<q^q|S)
z*
    Construct pillar from file tree.
    T)�topdown�onerror�followlinksr
z!file_tree: %s: not a regular file��rbZfile_buffer_size�

N���zfile_tree: Error reading %s: %s)�template�	renderers�defaultZ	blacklistZ	whitelist)!�salt�loaderZrender�__opts__Z__salt__rr
�normpath�utilsZos_walkr	�relpath�split�insert�popr�isfilerr�filesZfopen�read�endswithr�OSErrorrrZcompile_template_strZstringutilsZ
to_unicodeZstringioZis_readable�getvalue)Ztop_dir�follow_dir_linksr�render_default�renderer_blacklist�renderer_whitelistr�pillarr Znorm_top_dirZdir_pathZ	dir_namesZ
file_namesZpillar_nodeZ
norm_dir_pathr�
path_parts�head�tailZdir_namerZ	file_path�contentsZfhr�buf�exc�datarrr�_construct_pillar�sv


�







�
�











�
����	
��






�

�"r=c
Cs�~
|s
t�
d
�

iStj�|�
s]td}
|
durt�
d�

iSt�d|
�
td�|
d�}|dur9t�
d|
�
iSg}|D]}
tj�tj�	|
|��
}||v
sT||dkrY|�
|�

q=|}n|g
}i}|D]}t|||||||||	�	}tj
jj||d	d
�}qd|S)al
    Compile pillar data from the given ``root_dir`` specific to Nodegroup names
    and Minion IDs.

    If a Minion's ID is not found at ``<root_dir>/host/<minion_id>`` or if it
    is not included in any Nodegroups named at
    ``<root_dir>/nodegroups/<node_group>``, no pillar data provided by this
    pillar module will be available for that Minion.

    .. versionchanged:: 2017.7.0
        Templating/rendering has been added. You can now specify a default
        render pipeline and a black- and whitelist of (dis)allowed renderers.

        ``template`` must be set to ``True`` for templating to happen.

        .. code-block:: yaml

            ext_pillar:
              - file_tree:
                root_dir: /path/to/root/directory
                render_default: jinja|yaml
                renderer_blacklist:
                  - gpg
                renderer_whitelist:
                  - jinja
                  - yaml
                template: True

    :param minion_id:
        The ID of the Minion whose pillar data is to be collected

    :param pillar:
        Unused by the ``file_tree`` pillar module

    :param root_dir:
        Filesystem directory used as the root for pillar data (e.g.
        ``/srv/ext_pillar``)

        .. versionchanged:: 2018.3.0
            If ``root_dir`` is a relative path, it will be treated as relative to the
            :conf_master:`pillar_roots` of the environment specified by
            :conf_minion:`pillarenv`. If an environment specifies multiple
            roots, this module will search for files relative to all of them, in order,
            merging the results.

    :param follow_dir_links:
        Follow symbolic links to directories while collecting pillar files.
        Defaults to ``False``.

        .. warning::

            Care should be exercised when enabling this option as it will
            follow links that point outside of ``root_dir``.

        .. warning::

            Symbolic links that lead to infinite recursion are not filtered.

    :param debug:
        Enable debug information at log level ``debug``.  Defaults to
        ``False``.  This option may be useful to help debug errors when setting
        up the ``file_tree`` pillar module.

    :param keep_newline:
        Preserve the end-of-file newline in files.  Defaults to ``False``.
        This option may either be a boolean or a list of file globs (as defined
        by the `Python fnmatch package
        <https://docs.python.org/library/fnmatch.html>`_) for which end-of-file
        newlines are to be kept.

        ``keep_newline`` should be turned on if the pillar data is intended to
        be used to deploy a file using ``contents_pillar`` with a
        :py:func:`file.managed <salt.states.file.managed>` state.

        .. versionchanged:: 2015.8.4
            The ``raw_data`` parameter has been renamed to ``keep_newline``. In
            earlier releases, ``raw_data`` must be used. Also, this parameter
            can now be a list of globs, allowing for more granular control over
            which pillar values keep their end-of-file newline. The globs match
            paths relative to the directories named for Minion IDs and
            Nodegroup namess underneath the ``root_dir``.

            .. code-block:: yaml

                ext_pillar:
                  - file_tree:
                      root_dir: /srv/ext_pillar
                      keep_newline:
                        - apache/config.d/*
                        - corporate_app/settings/*

        .. note::
            In earlier releases, this documentation incorrectly stated that
            binary files would not affected by the ``keep_newline``.  However,
            this module does not actually distinguish between binary and text
            files.


    :param render_default:
        Override Salt's :conf_master:`default global renderer <renderer>` for
        the ``file_tree`` pillar.

        .. code-block:: yaml

            render_default: jinja

    :param renderer_blacklist:
        Disallow renderers for pillar files.

        .. code-block:: yaml

            renderer_blacklist:
              - json

    :param renderer_whitelist:
        Allow renderers for pillar files.

        .. code-block:: yaml

            renderer_whitelist:
              - yaml
              - jinja

    :param template:
        Enable templating of pillar files.  Defaults to ``False``.
    z file_tree: no root_dir specified�	pillarenvNz8file_tree: root_dir is relative but pillarenv is not setzfile_tree: pillarenv = %sZpillar_rootszSfile_tree: root_dir is relative but no pillar_roots are specified  for pillarenv %sr�recurse�
Zstrategy)rrrr
�isabsr$�debug�getr%r�append�_ext_pillarr"r&�
dictupdate�merge)�	minion_idr5�root_dirr1rBrr2r3r4rr>Z	env_rootsZenv_dirsZenv_rootZenv_dir�dirsZ
result_pillar�rootZ
dir_pillarrrr�
ext_pillar
sV











�



�











�
�rLc	Cs�
t�
d
|
�
tj�|
�
st�d|
�
iSt|ttf�s#t�d|
�
iSi}	tj�	|
d�}
tj�
|
�
r�tt�
dd��
dkr�td}t�|
�
}t|�
dkr�|D]>}
tj�|
�
r�|
|vr�tjj�t�
}|�||
d�}|d}||vr�tj�	|
t|
�
�}tjjj|	t|||||||�d	d
�}	qMn|dur�t�
d|�
n	|dur�t�
d
�

tj�	|
d|�}tj�
|�
s�|dur�t�
d||�
|	Stj�|�
s�t�d|�
|	St|||||||�}tjjj|	|d	d
�S)zO
    Compile pillar data for a single root_dir for the specified minion ID
    zfile_tree: reading %sz;file_tree: root_dir %s does not exist or is not a directoryzsfile_tree: keep_newline must be either True/False or a list of file globs. Skipping this ext_pillar for root_dir %sZ
nodegroupsrr
Zcompound�minionsr?r@TzEfile_tree: no nodegroups found in file tree directory %s, skipping...z6file_tree: no nodegroups found in master configuration�hostszGfile_tree: no pillar data for minion %s found in file tree directory %sz,file_tree: %s exists, but is not a directory)rrBrr
�isdirrr
r�listr�exists�lenr$rC�listdirr"r&rMZ	CkMinionsZ
check_minionsrrFrGr=)rHrIr1rBrr2r3r4rZ
ngroup_pillarZnodegroups_dirZmaster_ngroupsZext_pillar_dirsZ	nodegroupZ	ckminionsZ_res�matchZ
ngroup_dirZhost_dirZhost_pillarrrrrE�
s�

�

�
 









�











�	���

��






�








�	rE)FNNNF)NFFFNNNF)�__doc__r�loggingrZsalt.loaderr"Z
salt.templateZsalt.utils.dictupdateZsalt.utils.filesZsalt.utils.minionsZsalt.utils.pathZsalt.utils.stringioZsalt.utils.stringutils�	getLogger�__name__rr	rr=rLrErrrr�<module>
sB














�M







�?