o

�N�g�S�@s�dZd
dl
Z
d
dlZd
dlZd
dlZd
dlZd
dlZd
dlZd
dl	ZdZ
dZdZdZ
zd
dlZdZWney=


dZYn
wdZed	kr�d
dlZe��Zejd
dd�
ejd
dddd�
ejddddd�
ejdddd�
ejdddd�
ejddddd�
ejd d!dd"d�
ejd#d$dd%d�
ejd&d'd(d�
ejd)d*d+d�
ejd,d-d.d�
e��ZejZejr�ejZdZejs�zd
dlZe� d/�
ZWney�


e�!d0�
ZYnwe�!d0�
Ze�"�Z#e#�$e�

e#�%e�

e�&d1�
Ze�$e�

e�'e#�

ne�&e�
Zd2d3i
d4dd5�Z(d6d7�Z)d8d9�Z*d^d:d;�
Z+d<d=�Z,ed	k�r�ej-�.ej/�
�
s.e�0d>ej/�
e�1d?�

ej2j3�4ej/�
�Z5e(�6ej2j7�8e5�
�

Wd�
n	1�
sLw



Y
d
Z9d@dA�e(d;D�
D]Z:e:dBk�
re
ne9d?7Z9�
q\iZ;dCe(v�
rve(dCZ;ej<�
r�e;�6ej2j7�8ej<�
�

iZ=dDe(v�
r�e(dDZ=ej>�
r�e=�6ej2j7�8ej>�
�

ej,�
r�de(dE<ej?�r+d
dl@Z@d
dlZejAZAejBZBeAdu�
r�eCdF�
ZAeBdu�
r�e@�@�ZBe�DdG�

eAeBdHdI�ZEejFejGdJeEdKdL�ZHeHjI�
s�eJdMeHjK���
�
eH�L�ZMeMdNd
dOZNe�DdP�

eNdQdR�ZOejPejGdSejQeOdKdT�ZHeH�L��PdNig
�d
ZRejQeRv
�r&eJdU�
�
eRejQZ;dZSdZTdVe(d;e9dBv�rDe(d;e9dBdVZSdWe(d;e9dBv�rYe(d;e9dBdWZTe+ejQe=e(d;e9dBdXe(d;e9dBdYeSeT�ZRe(dE�r�e,eRe(d;e9dBdX�
ej2j7jUjVZWz`dZd[�ej2j7jU_Vd\d]�ZXej�s�z"d
dlYZYd
dlZZYd
dl[ZYe\eY�]ej2j7�^eR�
eYj_�`�eYja�b���

Wne�y�


eXeR�

YnweXeR�

WeWej2j7jU_VdSWeWej2j7jU_VdSWeWej2j7jU_VdSeWej2j7jU_VwdS)_a�
Pepa
====

Configuration templating for SaltStack using Hierarchical substitution and Jinja.

Configuring Pepa
================

.. code-block:: yaml

    extension_modules: /srv/salt/ext

    ext_pillar:
      - pepa:
          resource: host                # Name of resource directory and sub-key in pillars
          sequence:                     # Sequence used for hierarchical substitution
            - hostname:                 # Name of key
                name: input             # Alias used for template directory
                base_only: True         # Only use templates from Base environment, i.e. no staging
            - default:
            - environment:
            - location..region:
                name: region
            - location..country:
                name: country
            - location..datacenter:
                name: datacenter
            - roles:
            - osfinger:
                name: os
            - hostname:
                name: override
                base_only: True
          subkey: True                  # Create a sub-key in pillars, named after the resource in this case [host]
          subkey_only: True             # Only create a sub-key, and leave the top level untouched

    pepa_roots:                         # Base directory for each environment
      base: /srv/pepa/base              # Path for base environment
      dev: /srv/pepa/base               # Associate dev with base
      qa: /srv/pepa/qa
      prod: /srv/pepa/prod

    # Use a different delimiter for nested dictionaries, defaults to '..' since some keys may use '.' in the name
    #pepa_delimiter: ..

    # Supply Grains for Pepa, this should **ONLY** be used for testing or validation
    #pepa_grains:
    #  environment: dev

    # Supply Pillar for Pepa, this should **ONLY** be used for testing or validation
    #pepa_pillars:
    #  saltversion: 0.17.4

    # Enable debug for Pepa, and keep Salt on warning
    #log_level: debug

    #log_granular_levels:
    #  salt: warning
    #  salt.loaded.ext.pillar.pepa: debug

Pepa can also be used in Master-less SaltStack setup.

Command line
============

.. code-block:: bash

    usage: pepa.py [-h] [-c CONFIG] [-d] [-g GRAINS] [-p PILLAR] [-n] [-v]
                   hostname

    positional arguments:
      hostname              Hostname

    optional arguments:
      -h, --help            show this help message and exit
      -c CONFIG, --config CONFIG
                            Configuration file
      -d, --debug           Print debug info
      -g GRAINS, --grains GRAINS
                            Input Grains as YAML
      -p PILLAR, --pillar PILLAR
                            Input Pillar as YAML
      -n, --no-color        No color output
      -v, --validate        Validate output

Templates
=========

Templates is configuration for a host or software, that can use information from Grains or Pillars. These can then be used for hierarchically substitution.

**Example File:** host/input/test_example_com.yaml

.. code-block:: yaml

    location..region: emea
    location..country: nl
    location..datacenter: foobar
    environment: dev
    roles:
      - salt.master
    network..gateway: 10.0.0.254
    network..interfaces..eth0..hwaddr: 00:20:26:a1:12:12
    network..interfaces..eth0..dhcp: False
    network..interfaces..eth0..ipv4: 10.0.0.3
    network..interfaces..eth0..netmask: 255.255.255.0
    network..interfaces..eth0..fqdn: {{ hostname }}
    cobbler..profile: fedora-19-x86_64

As you see in this example you can use Jinja directly inside the template.

**Example File:** host/region/amer.yaml

.. code-block:: yaml

    network..dns..servers:
      - 10.0.0.1
      - 10.0.0.2
    time..ntp..servers:
      - ntp1.amer.example.com
      - ntp2.amer.example.com
      - ntp3.amer.example.com
    time..timezone: America/Chihuahua
    yum..mirror: yum.amer.example.com

Each template is named after the value of the key using lowercase and all extended characters are replaced with underscore.

**Example:**

osfinger: Fedora-19

**Would become:**

fedora_19.yaml

Nested dictionaries
===================

In order to create nested dictionaries as output you can use double dot **".."** as a delimiter. You can change this using "pepa_delimiter" we choose double dot since single dot is already used by key names in some modules, and using ":" requires quoting in the YAML.

**Example:**

.. code-block:: yaml

    network..dns..servers:
      - 10.0.0.1
      - 10.0.0.2
    network..dns..options:
      - timeout:2
      - attempts:1
      - ndots:1
    network..dns..search:
      - example.com

**Would become:**

.. code-block:: yaml

    network:
      dns:
        servers:
          - 10.0.0.1
          - 10.0.0.2
        options:
          - timeout:2
          - attempts:1
          - ndots:1
        search:
          - example.com

Operators
=========

Operators can be used to merge/unset a list/hash or set the key as immutable, so it can't be changed.

=========== ================================================
Operator    Description
=========== ================================================
merge()     Merge list or hash
unset()     Unset key
immutable() Set the key as immutable, so it can't be changed
imerge()    Set immutable and merge
iunset()    Set immutable and unset
=========== ================================================

**Example:**

.. code-block:: yaml

    network..dns..search..merge():
      - foobar.com
      - dummy.nl
    owner..immutable(): Operations
    host..printers..unset():

Validation
==========

Since it's very hard to test Jinja as is, the best approach is to run all the permutations of input and validate the output, i.e. Unit Testing.

To facilitate this in Pepa we use YAML, Jinja and Cerberus <https://github.com/nicolaiarocci/cerberus>.

Schema
======

So this is a validation schema for network configuration, as you see it can be customized with Jinja just as Pepa templates.

This was designed to be run as a build job in Jenkins or similar tool. You can provide Grains/Pillar input using either the config file or command line arguments.

**File Example: host/validation/network.yaml**

.. code-block:: jinja

    network..dns..search:
      type: list
      allowed:
        - example.com

    network..dns..options:
      type: list
      allowed: ['timeout:2', 'attempts:1', 'ndots:1']

    network..dns..servers:
      type: list
      schema:
        regex: ^([0-9]{1,3}\.){3}[0-9]{1,3}$

    network..gateway:
      type: string
      regex: ^([0-9]{1,3}\.){3}[0-9]{1,3}$

    {% if network.interfaces is defined %}
    {% for interface in network.interfaces %}

    network..interfaces..{{ interface }}..dhcp:
      type: boolean

    network..interfaces..{{ interface }}..fqdn:
      type: string
      regex: ^([a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?\.)+[a-zA-Z]{2,6}$

    network..interfaces..{{ interface }}..hwaddr:
      type: string
      regex: ^([0-9a-f]{1,2}\:){5}[0-9a-f]{1,2}$

    network..interfaces..{{ interface }}..ipv4:
      type: string
      regex: ^([0-9]{1,3}\.){3}[0-9]{1,3}$

    network..interfaces..{{ interface }}..netmask:
      type: string
      regex: ^([0-9]{1,3}\.){3}[0-9]{1,3}$

    {% endfor %}
    {% endif %}

Links
=====

For more examples and information see <https://github.com/mickep76/pepa>.
�Nz/Michael Persson <michael.ake.persson@gmail.com>z"Copyright (c) 2013 Michael PerssonzApache License, Version 2.0z0.6.6TF�__main__�hostnameZHostname)
�helpz-cz--configz/etc/salt/masterzConfiguration file)�defaultrz-dz--debug�
store_truezPrint debug info)�actionrz-gz--grainszInput Grains as YAMLz-pz--pillarzInput Pillar as YAMLz-nz
--no-colorzNo color outputz-vz
--validatezValidate outputz-qz--query-apiz#Query Saltstack REST API for Grainsz--urlzhttps://salt:8000zURL for SaltStack REST APIz-uz
--usernamezUsername for SaltStack REST APIz-Pz
--passwordzPassword for SaltStack REST APIzI[%(log_color)s%(levelname)-8s%(reset)s] %(log_color)s%(message)s%(reset)sz[%(levelname)-8s] %(message)sZpythonConfig�basez	/srv/saltz..)�
pepa_roots�pepa_delimiter�
pepa_validatec
Cstsd
SdS)z6
    Only return if all the modules are available
    FT)
�HAS_REQUESTS�r
r
�D/opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/pepa.py�__virtual__d
s
rc
Csbi}
|��D](\}}|
}|�
td
�
}t|d�D]\}}|t|�
kr'|||<q|�|i�}qq|
S)z#
    Convert key/value to tree
    r
�
)�items�split�__opts__�	enumerate�len�
setdefault)�data�treeZflatkey�value�
t�keys�
i�keyr
r
r�key_value_to_treen
s






�rcCs&td
}i}d|d<||d<d|
vr|
d|d<ndt
vr$t
d|d<nd|d<|}g|d<i}	dd�|D�
D�]/\}
}|
|v
rGt�d	|
�
q7d
}t|t�rWd|vrW|d}n|
}d
}
|rpd|vrp|drptj�|d||�}
ntj�||d||�}
g}t||
t	�r�||
}n||
s�t�d
|
�
q7||
g
}|D�
]�}d
}d
}tj�|
t
�dd|���d�}tj�
|�
�
r;t�d|�
tjj�|�
�}t�|���
}Wd
�
n1s�w



Y
|d�|�

zt|�
}t
��|d<|
��|d<|�|�
}tjj�|�
}Wn>tj�
y
}
z
t�d||�
WYd
}~n(d
}~w
tjjj�
y:
}
z
t�d||�
WYd
}~nd
}~w
wt�d|�
q�|d
u
�rf|D�
]}|�tdd�}d
}d
}t |�
dk�
rt|�!d�
dk�
rt|d�"td�
}|d}||	v�
r�t�d|�
�
qI||	v�
r�t�d|�
�
qI|dk�
s�|dk�r|dk�
r�t�#d|||�
n
t�#d|||�
d |	|<||v
�
r�t�d!|�
�
qIt||t$||�
��
s�t�d"|�
�
qIt||t��
r�||�%||�

�
qIt||t	��
r�||�&||�

�
qIt�d#|�
�
qI|d$k�s
|d%k�r-|d$k�rt�#d&|�
n
t�#d'|�
d |	|<||v�r+||=�
qI|d(k�rGt�#d)|||�
d |	|<||||<�
qI|d
u
�rUt�d*||�
�
qIt�#d+|||�
||||<�
qIq�q7t|�
}i}|�rx|��||<n|�r�|}|��||<n|}td,�r�|��|d-<|S).z!
    Evaluate Pepa templates
    r	rrZenvironmentrZpepa_templatesc
Ssg|]
}
tt
|
���
�
�qSr
)�next�iterr)�.0�
sr
r
r�
<listcomp>�
szext_pillar.<locals>.<listcomp>zCategory is not defined: %sN�nameZ	base_onlyzCategory has no value set: %sz\W�
_z.yamlzLoading template: %s�grains�pillarz%Failed to parse JINJA template: %s
%sz'Failed to parse YAML in template: %s
%szTemplate doesn't exist: %sr
rz()r
z,Key %s is immutable, changes are not allowedzmerge()zimerge()zMerge key %s: %sz"Set immutable and merge key %s: %sTz!Cant't merge key %s doesn't existz&Can't merge different types for key %sz3Unsupported type need to be list or dict for key %szunset()ziunset()zUnset key %szSet immutable and unset key %szimmutable()z'Set immutable and substitute key %s: %sz(Unsupported operator %s, skipping key %szSubstitute key %s: %sr�	pepa_keys)'r�
__grains__�log�warning�
isinstance�dict�os�path�join�list�re�sub�lower�isfile�info�salt�utils�files�fopen�jinja2�Template�read�appendr�copy�render�yaml�	safe_loadZUndefinedError�errorZ	YAMLError�rsplitr�rfind�rstrip�debug�type�update�extend)Z	minion_idr'�resource�sequence�subkey�subkey_only�roots�inp�outputZ	immutableZcategr6�aliasZtempldir�entries�entryZ
results_jinja�results�fn�fhr�templater�errrZskeyZrkey�operatorrZpillar_datar
r
r�
ext_pillar~
s�
























 




�







�

��















�




�










�






�





�




�
��T









r[c	Cs>
zd
dl}Wnt
y


t�d�

YdSwtd}tj�|d|
d�}i}g}t�|d�
D]H}t�	d|�
t
jj�
|�
�}t�|���
}	Wd�
n1sPw



Y
|}
t��|
d	<t��|
d
<t
jj�|	�|
�
�
}|�|�

|�|�

q/|��}|�|d|�s�|j��D]\}
}t�d|
|�
q�||d
<||d<dS)z!
    Validate Pepa templates
    r
Nz3You need module cerberus in order to use validationr	r�validatez/*.yamlzLoading schema: %sr&r'r(z Validation failed for key %s: %sZpepa_schema_keys�pepa_schemas)�cerberus�ImportErrorr*�criticalrr.r/r0�globr6r7r8r9r:r;r<r=r)r?�
__pillar__rArBr@rIr>Z	Validatorr\�errorsrr+)rQrKr^rOZvaldirZall_schemasr]rVrWrXrZschema�valZekeyrCr
r
rr\s6




�




�









r\z$Configuration file doesn't exist: %src
Cs g|]}
tt
t|
���
�
�
�qSr
)rr r1r)r!�
er
r
rr#As r#ZpepaZpepa_grainsZpepa_pillarrz
Username: zAuthenticate REST APIZpam)�username�passwordZeauthz/login�x)
�timeoutz.Failed to authenticate to SaltStack REST API: �return�tokenzRequest Grains from REST APIzapplication/json)zX-Auth-TokenZAcceptz	/minions/)�headersriz,Failed to get Grains from SaltStack REST APIrMrNrKrLc
Csd
S)NTr
)�
x�
yr
r
r�<lambda>�sroc

Cstt
jjj|d
dd��

dS)N�F)�indentZdefault_flow_style)�printr7r8rA�	safe_dump)
�resultr
r
r�
_print_result�s
ru)FF)c�__doc__ra�loggingr.r2�sysr;Zsalt.utils.filesr7Zsalt.utils.yaml�
__author__�
__copyright__Z__license__�__version__Zrequestsrr_r*�__name__�argparse�ArgumentParser�parser�add_argument�
parse_args�args�WARNINGZ	LOG_LEVELrG�DEBUG�	formatterZno_colorZcolorlogZColoredFormatter�	Formatter�
StreamHandler�stream�setLevel�setFormatter�	getLogger�
addHandlerrrrr[r\r/r5Zconfigr`�exitr8r9r:Zfh_rIrArB�locr$r)r&rbr'Z	query_apiZgetpassrfrg�inputr6ZauthZpost�url�request�ok�RuntimeError�text�json�responserkrl�getrrtZ	ex_subkeyZex_subkey_onlyZSafeOrderedDumperZignore_aliasesZorig_ignoreruZpygmentsZpygments.formattersZpygments.lexersrrZ	highlightrsZlexersZ	YamlLexerZ
formattersZTerminalFormatterr
r
r
r�<module>
sR













�




�








�
�







�
�











�


%



�





























�





�













�
	









��
�
���