HEX
Server: Apache
System: Linux server2.voipitup.com.au 4.18.0-553.109.1.lve.el8.x86_64 #1 SMP Thu Mar 5 20:23:46 UTC 2026 x86_64
User: posscale (1027)
PHP: 8.2.30
Disabled: exec,passthru,shell_exec,system
Upload Files
File: //opt/saltstack/salt/lib/python3.10/site-packages/salt/__pycache__/state.cpython-310.pyc
o

�N�g�@�@s�dZddlZddlZddlZddlZddlZddlZddlZddlZddl	Z	ddl
Z
ddlZddlZddl
Z
ddlZddlZddlZddlZddlZddlmZddlZddlZddlZddlZddlZddlZddlZddlZddlmm Z ddl!Zddl"Zddl#Zddl$Zddl%Zddl&mm'Z'ddl(m)Z)m*Z*m+Z+ddl,m-Z.ddl,m/Z0ddl1m2Z2m3Z3ddl4m5Z5m6Z6e�7e8�Z9e:gd��Z;e:gd	��Z<e:gd
��Z=e;�>e<��>e=�Z?dd�Z@d
d�ZAdd�ZBdd�ZCdd�ZDdd�ZEdd�ZFdd�ZGd:dd�ZHdd�ZId d!�ZJd"d#�ZKd$d%�ZLd&d'�ZMGd(d)�d)eN�ZOGd*d+�d+�ZPGd,d-�d-�ZQGd.d/�d/�ZRGd0d1�d1�ZSGd2d3�d3eS�ZTGd4d5�d5eQ�ZUGd6d7�d7eT�ZVGd8d9�d9�ZWdS);a�
The State Compiler is used to execute states in Salt. A state is unlike
an execution module in that instead of just executing a command, it
ensures that a certain state is present on the system.

The data sent to the state calls is as follows:
    { 'state': '<state module name>',
      'fun': '<state function name>',
      'name': '<the name argument passed to all states>'
      'argn': '<arbitrary argument, can have many of these>'
      }
�N)�CommandExecutionError�SaltRenderError�SaltReqTimeoutError)�deserialize)�	serialize)�compile_template�compile_template_str)�DefaultOrderedDict�HashableOrderedDict)
�	onchanges�
onchanges_any�onfail�
onfail_any�
onfail_all�onfail_stop�prereq�prerequired�watch�	watch_any�require�require_any�listen)�onchanges_in�	onfail_in�	prereq_in�watch_in�
require_in�	listen_in)'�fun�state�	check_cmd�cmd_opts_exclude�failhard�onlyif�unless�creates�retry�order�parallelrrr�reload_modules�
reload_grains�
reload_pillar�runas�runas_password�
fire_event�saltenv�umask�use�use_in�__env__�__sls__�__id__�__orchestration_jid__Z
__pub_userZ	__pub_argZ	__pub_jidZ	__pub_funZ	__pub_tgtZ	__pub_retZ	__pub_pidZ__pub_tgt_type�
__prereq__�__prerequired__�	__umask__cCs |�d�\}}}}||||d�S)zN
    Take a low tag and split it back into the low dict that it came from
    z_|-)rr5�namer)�split)�tagr�id_r:r�r>�>/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py�
split_low_tag�sr@cCs
d�|�S)zJ
    Generate the running dict tag string from the low data structure
    z/{0[state]}_|-{0[__id__]}_|-{0[name]}_|-{0[fun]}��format��lowr>r>r?�_gen_tag��
rEcCstjj�|�S)z*
    Make tag name safe for filenames
    )�salt�utils�filesZsafe_filename_leaf)r<r>r>r?�
_clean_tag�srJcCs"d|��d|��ddd�}t|�S)N�listen_ZListen_Error)r:r5rr�rE)r:r=rDr>r>r?�_l_tag�s�rMcCs\tj��}|tj��tj��}tj��}|����}||}|jd|jd}||fS)zk
    Generate a NULL duration for when states do not run
    but we want the results to be consistent.
    �@B�@�@)�datetime�utcnow�now�time�	isoformat�seconds�microseconds)�utc_start_time�local_start_time�utc_finish_time�
start_time�delta�durationr>r>r?�_calculate_fake_duration�s
�
r]cCs(tj�|d�}tj�|�st�|�|S)zg
    Return the directory that accumulator data is stored in, creating it if it
    doesn't exist.
    Zaccumulator)�os�path�join�isdir�makedirs)�cachedir�fn_r>r>r?�get_accumulator_dir�s
recCs.tt|��}d|vr|�d�d||iS|S)z.
    Trim any function off of a requisite
    �.r)�next�iterr;)�req�reqfirstr>r>r?�trim_req�srkcCsdt�}||vr	|S|||vr|S|||D]}t|t�sqt|�dkr&q|�tt|���q|S)zA
    Return a set of the arguments passed to the named state
    �)�set�
isinstance�dict�len�addrgrh)r=r�high�args�itemr>r>r?�
state_args�s
ruFc	Cs�g}|dur	d}n||�|i�v}||vr |r |�||f�|S|dkr@|��D]\}}|d|kr=|�|tt|��f�q(|S|D]8}|||vrzt|||t�rz|||D] }t|t�saqYt|�dkrhqY|tt|��|kry|�||f�qYqB|S)z�
    Scan high data for the id referencing the given name and return a list of (IDs, state) tuples that match

    Note: if `state` is sls, then we are looking for all IDs that match the given SLS
    FT�slsr4rl)	�get�append�itemsrgrhrn�listrorp)	r:rrr�strict�ext_idZcheck2�nidrt�argr>r>r?�	find_name�s2���
�rc	Cs|g}|��D]5\}}z|d}Wnty$|dkr"t�d||�Yqw||kr;|D]}|�d�r3q+|�||f�q+q|S)zT
    Scan for all ids in the given sls and return them in a dict; {name: state}
    r4�__exclude__z2Invalid non-dict item '%s' in high data. Value: %r�__)ry�	TypeError�log�error�
startswithrx)rvrr�retr}rtZsls_tgt�st_r>r>r?�find_sls_idss$��
�r�cCsLd}t|t�r�d|vr�|d}|s |dr|d}n_d�|�}nYt|t�ryd|vr7t|dt�r7d�|d�}tdd�|��D��rytd	d�|��D��ryd
}|D]'}||d}|sa|dvrad
}||d}|so|dvrod
}|d�|||�7}qQ|s�t|d�}|ddus�|ddur�t�|�dSt�|�dSdSt�t|��dS)z-
    Format the state into a log message
    ��changes�commentzNo changes made for {0[name]}�diffzFile changed:
{}cSsg|]}t|t��qSr>)rnro��.0�xr>r>r?�
<listcomp>*szformat_log.<locals>.<listcomp>cSsg|]
}d|vod|v�qS)�old�newr>r�r>r>r?r�+�zMade the following changes:
r��FNZabsentr�z'{}' changed from '{}' to '{}'
�resultTN)	rnrorB�str�all�valuesr��infor�)r��msgZchg�pkgr�r�r>r>r?�
format_logs@


��#r�cCst|||||�}|��S)zQ
    Compile the master side low state data, and build the hidden state file
    )�MasterHighState�compile_highstate)�master_opts�minion_opts�grainsr=r/r�r>r>r?�master_compileDsr�cCs$zt|�WdStyYdSw)NFT)�hashr�)�objr>r>r?�
ishashableLs
��r�cCs0|dr|dd}n|dd}|didd�S)zo
    Returns a mocked return dict with information about the run, without
    executing the state function
    rsr�kwargsr:zNot called, mockedT�r:r�r�r�r>)�cdatar:r>r>r?�mock_retTs�r�c@seZdZdZdS)�
StateErrorz!
    Custom exception class.
    N)�__name__�
__module__�__qualname__�__doc__r>r>r>r?r�gsr�c@sHeZdZdZdd�Zdd�Zdd�Zdd	�Zd
d�Zdd
�Z	dd�Z
dS)�CompilerzB
    Class used to compile and manage the High Data structure
    cCs||_||_dS�N)�opts�rend)�selfr��	renderersr>r>r?�__init__rs
zCompiler.__init__cKs>t||j|jd|jd|jdfi|��}|s|S|�|�S�z2
        Enforce the states in a template
        �renderer�renderer_blacklist�renderer_whitelist)rr�r��	pad_funcs)r��templater�rrr>r>r?�render_templatevs��
zCompiler.render_templatecCsP|D]�}t||t�s?t||t�r?d||vr>||�d�}t|�dkr2d�|dt|���|d<|d|dgi||<qqt�}t||�D]\}|�d�rPqHt|||t	�sZqHd|vr�|�d�}t|�dkrvd�|dt|���|d<|d|vr}qH||�
|�|||d<|||d�|d�|�|d�qH|�|�qHq|S)zI
        Turns dot delimited function refs into function strings
        rf�rlr�_)
rnror�r;rpr`rm�sortedr�rz�poprxrq)r�rrr:�comps�skeys�keyr>r>r?r��s<�


�zCompiler.pad_funcscCs�g}t|t�s|�d�t�}|�s�|��D�]�\}}z	|�d�r#WqWnttfy/Ynwt|t�sD|�d�	||dt
|�j��t|t�sUd�	||�}|�|�q|D�]i}|�d�r`qWt||t�ss|�d�	||d��qWd}d|vr}|d	7}||D]�}	t|	t�r�|d	7}d
|	�
�vr�|�d|	�d|�d
|d�d��q�t|	t��rntt|	��}
|
dv�rnt|	|
t�s�|�d�	|
||d��n�d|i||<|	|
D]�}t|t�r�d|i}t|t�s�|�d�	||d��q�tt|��}||}
d|v�r|�d�	|||d|d|�d����t|
��s%|�d�	t|
�|d��q�||||
<|
|v�rY|||
v�rY||
||k�rY||
d|||
k�rY|�d�	|d||
��q�tt|	��d	k�rn|�d�	||d��q�|�s�|dk�s||dk�r}qW|�d|�d|d�d��qW|d	k�r�d|v�r�|jdd	d�d	gng}|�dd �||D��|�d!|�d|d�d"�d#�|��qWq|S)$�T
        Verify that the high data is viable and follows the data structure
        �,High data is not a dictionary and is invalidr�zTID '{}' in SLS '{}' is not formed as a string, but is a {}. It may need to be quotedr4�2The type {} in {} is not formatted as a dictionary�.State '{}' in SLS '{}' is not formed as a listrrfrl� �The function "�" in state "�
" in SLS "�p" has whitespace, a function with whitespace is not supported, perhaps this is an argument that is missing a ":"�rrrr�GThe {} statement in state '{}' in SLS '{}' needs to be formed as a listr�id�KRequisite declaration {} in SLS {} is not formed as a single key dictionary�qInvalid requisite type '{}' in state '{}', in SLS '{}'. Requisite types must not contain dots, did you mean '{}'?Nz"Illegal requisite "{}", is SLS {}
�9A recursive requisite was found, SLS "{}" ID "{}" ID "{}"�CMultiple dictionaries defined in argument of state '{}' in SLS '{}'rr�No function declared in state '�
' in SLS '�'��maxsplitcs��|]
}t|t�r|VqdSr��rnr��r�r~r>r>r?�	<genexpr>Vs�
�
�z'Compiler.verify_high.<locals>.<genexpr>�&Too many functions declared in state '�''. Please choose one of the following: �, )rnrorxr
ryr��AttributeErrorr�r�rB�typer�rz�striprgrh�findr�rpr;�extendr`�r�rr�errors�reqsr:�body�errrrr~�argfirstri�req_key�req_valZfunsr>r>r?�verify_high�s


��
��
�


��
����

��	

��
��
��


���������

��
�������
zCompiler.verify_highcCsd}|D] }d|vr$t|dt�sq|d}||dkr$|dkr$|d}q|D]\}d|vr2||d<q't|dttf�sW|ddkrH|d|d<n|ddkrSd|d<n||d<d|vrh|d|�d�d	|d<|ddkrx|d|d|d<tjj�|d
�|d
<q'|jdd�d
�|S)�x
        Sort the chunk list verifying that the chunks follow the order
        specified in the order options.
        rlr'r�d�lastrN�first�
name_order���@r:cS�|dd�|�fS�Nr'z{0[state]}{0[name]}{0[fun]}rA��chunkr>r>r?�<lambda>���z'Compiler.order_chunks.<locals>.<lambda>�r�)	rn�int�floatr�rGrH�data�decode�sort�r��chunksZcapr�Zchunk_orderr>r>r?�order_chunks`s8�
�zCompiler.order_chunkscCs�g}|��D]�\}}|�d�rq|��D]�\}}t�}g}|�d�r#q||d�}	d|vr2|d|	d<d|vr<|d|	d<||	d<|D]3}
t|
t�rO|�|
�qBt|
t�ru|
��D]\}}|dkro|D]}
|
|vrm|�|
�qbqX|	�|
�qXqB|r�d}|D]=}t	�
|	�}t|t�r�tt|�
���}||d<tt|j||��n||d<||d	<|d}|D]}||d
<|�|�q�q|qt	�
|	�}|D]}||d
<|�|�q�qq|�|�}|S)��
        "Compile" the high data as it is retrieved from the CLI or YAML into
        the individual state executor structures
        r��rr:r4r3r5�namesrlr:r�r)ryr�rmrnr�rqrorx�update�copy�deepcopyrgrh�keysrz�mapr)r�rrrr:r�r�run�funcsrr�r~r��val�_namer��entry�live�low_namerr>r>r?�compile_high_data�sj






��

��
��
,zCompiler.compile_high_datac
Cs�d|vr|St�}t�}|�d�}|D]7}t|t�r|�|�t|t�rJt|�dkr+qtt|�	���}|dkr?|�|d�q|dkrJ|�|d�q|ri|�
�D]\}}|�d�r[qQ|�dd�|vrh|�|�qQ|D]}	|	|vrv|�|	�qk|S�zi
        Read in the __exclude__ list and remove all excluded objects from the
        high data
        r�rlrvr�r�r4r�)
rmr�rnr�rqrorprgrhr	ryr�rw)
r�rr�ex_sls�ex_id�exclude�excr�r:r�r=r>r>r?�
apply_exclude�s:



�

�
�zCompiler.apply_excludeN)r�r�r�r�r�r�r�r�rrrr>r>r>r?r�ms1*'8r�c@s�eZdZdZ										dgdd�Zdd�Zd	d
�Zdd�Zd
d�Zdd�Z	dd�Z
dd�Zdd�Zdd�Z
dd�Zdd�Zdd�Zdd �Zdhd!d"�Zd#d$�Zd%d&�Zd'd(�Zd)d*�Zd+d,�Zd-d.�Zdid/d0�Zdjd1d2�Zd3d4�Zd5d6�Zed7d8��Zd9d:�Ze j!j"j#�$d;d<�dkd>d?��Z%d@dA�Z&dBdC�Z'dDdE�Z(dFdG�Z)dHdI�Z*dJdK�Z+dLdM�Z,djdNdO�Z-djdPdQ�Z.dldSdT�Z/dUdV�Z0dWdX�Z1didYdZ�Z2d[d\�Z3d]d^�Z4d_d`�Z5dadb�Z6dcdd�Z7dedf�Z8dS)m�Statez+
    Class used to execute salt states
    NF�statescCs�|dur|}|durtjj�|�}|||||||||	|d�
|_||_d|vr.tj�|�|d<||_|
r:|
|_	d|_
ntj�|j�|_	d|_
||_
||_|durez|��}Wntydt|���}Ynw||_t�d�|	rx|jsx|	|jd<n+|��|jd<|jr�tjj�|jd|j|j�dd	�|j�d
d�|j�dd��|jd<t�d
�|dur�i|_n||_|j	|jd<|��t�|_t�|_i|_d|_||_||_ tt!|��|_"i|_#||_$d|_%dS)a
        When instantiating an object of this class, do not pass
        ``_invocation_id``. It is an internal field for tracking
        parallel executions where no jid is available (Salt-SSH) and
        only exposed as an init argument to work on spawning platforms.
        N)
r��pillar_override�jid�
pillar_enc�proxy�context�mocked�loader�initial_pillar�_invocation_idr�TFz#Gathering pillar data for state run�pillarZpillar_source_merging_strategyZsmartr�ZyamlZpillar_merge_listsz,Finished gathering pillar data for state run�
fileclientr)&rGrHrZgen_jid�_init_kwargs�
states_loaderr!r�r��file_client�preserve_file_clientr%�get_file_clientr�_pillar_override�lowerr�r��_pillar_encr��debug�_gather_pillarZ
dictupdate�mergerw�	state_con�load_modulesrm�active�mod_init�pre�_State__run_num�
invocation_idr��instance_id�inject_globalsr �global_state_conditions)r�r�rrrrrr r!r"r(r#r>r>r?r��sz��


�

zState.__init__c	s�g}d}|didd�}t�jt�s�jdd�pi�_�j��D]$\}}|d||fvrDt|t�r3|g}t|t�rD|��fdd�|D��q |rMt|�sM|}|S)	a+
        Return ``None`` if global state conditions are met. Otherwise, pass a
        return dictionary which effectively creates a no-op outcome.

        This operation is "explicit allow", in that ANY state and condition
        combination which matches will allow the state to be run.
        Nz9Failed to meet global state conditions. State not called.r��
config.optionr:�*c3s�|]
}�jd|�VqdS)zmatch.compoundN)�	functions)r�Z	condition�r�r>r?r�^s
��
�z7State._match_global_state_conditions.<locals>.<genexpr>)	rnr:ror=ryr�rzr��any)	r��fullrr:�matchesr�Zret_dictZstate_matchZ
conditionsr>r>r?�_match_global_state_conditionsCs,��

��z$State._match_global_state_conditionsc
Cs*|jrv|jr9ztjjj|j|jdt|dd�|j|jdd�|_Wnty8}zt	�
d|�WYd}~nd}~wwt|jt�rhzt
j|jt
jd�|_Wntyg}zt	�
d�t	�|�WYd}~nd}~wwt|jt�svt	�
d	�d|_tjj|j|jd
|jd|jd|j|j�d
�d�}|��S)zK
        Whenever a state run starts, gather the pillar data fresh
        Tr�NZdecrypt_pillar_renderers)Ztranslate_newlinesr�r�Z
valid_rendz%Failed to decrypt pillar override: %s)�Loaderz"Failed to load CLI pillar overridez.Pillar override was not passed as a dictionaryr�r�r/�	pillarenv)rrD)r+r-rGrHZcryptZdecrypt�getattrr��	Exceptionr�r�rnr��
yamlloader�loadZSaltYamlSafeLoader�	exceptionror$Z
get_pillarrwZcompile_pillar)r�rr$r>r>r?r/hsJ
����
��

�zState._gather_pillarcCs�z|jd�|d|d�Wn
tyYdSwd�|d�}|d|jvrC||jjvrE|j||�}|s9dS|j�|d�dSdSdS)z�
        Check the module initialization function, if this is the first run
        of a state package that has a mod_init function, then execute the
        mod_init function in the state module.
        �{}.{}rrNz{}.mod_init)rrB�KeyErrorr4�_dictrq)r�rDZminitZmretr>r>r?�	_mod_init�s ���zState._mod_initcCs�i}|D]9}|d|dkr=|�d�r=|d|dkr=tjttg�D]}||vr<||vr6||�||�q"||||<q"q|�|�|S)z*
        Aggregate the requisites
        r�__agg__r:)rw�	frozenset�union�STATE_REQUISITE_KEYWORDS�STATE_REQUISITE_IN_KEYWORDSr�r)r�rDr�
requisitesr�rir>r>r?�_aggregate_requisites�s ���
zState._aggregate_requisitescCs|jdd�}d|vr|d}|dur|dg}nt|t�s |S|d|vr�|�d�s�d�|d�}d|jvr<i|jd<||jdvrL|jd|sL|S||jvr�d|jd|<zd|d<|j||||�}|�||�}W|Sty�t	�
d	|d�Y|Swd
|jd|<|S)zQ
        Execute the aggregation systems to runtime modify the low chunk
        r;�state_aggregateZ	aggregateTrrNz{}.mod_aggregateZloader_cachez(Failed to execute aggregate for state %sF)r=rnrzrwrBr1rrTr�r�r�)r�rD�runningrZagg_optZagg_funr>r>r?�_mod_aggregate�s8


�
��zState._mod_aggregatec	Cs�dgd�}dD]	}|�|�||<qi}d}d|vr+t|dt�s&|dg}n|d}ng}|D]
}||vr<|�|�||<q/d|vrLd|vrL|d|d<nd|jdvr]|jd�d�|d<d|vr�|�||�}|d	|d	<|d
�|d
�d|vr�|d|d<d|vr�|�||�}|d	p�|d	|d	<|d
�|d
�d|vr�|dp�|d|d<d
|vr�|�|�}|d	p�|d	|d	<|d
�|d
�d|vr�|dp�|d|d<|S)zR
        Check that unless doesn't return 0, and that onlyif returns a 0.
        F)r�r��r4r5r:)�cwd�rootr,�envZprepend_pathr0�timeoutZsuccess_retcodesr!�shellr�r#r�r��
skip_watchr$r%)rwrnrzr��_run_check_onlyifrx�_run_check_unless�_run_check_creates)	r��low_datar�r��cmd_optsZPOSSIBLE_CMD_ARGSr!Zrun_cmd_argZ_retr>r>r?�
_run_check�sJ


�
zState._run_checkcCsP|�d�}d|vr|�d�ng}||d�}|�|�|j||di|d��S)z0Format slot args and run unless/onlyif function.rrs)rsr�r�)r��format_slotsr=)r�rrrsr�r>r>r?�_run_check_function#s



zState._run_check_functionc
	s�ddi�dD]	}|�|��|<qt|dt�s|dg}n|d}�fdd�}|D]�}t|t�r[z|jd|fddd	�|��}WntyKd
}Ynwt�d|�||�sZ�Sq)t|t�r�d|vrvd
|���d<t�	�d��S|�
dd�}|�|�}	|r�tj
j�|	|�}	|j�dd�r�||jd�s��Sq)|	s���dddd���S��ddd��q)��ddd���S�S)z�
        Make sure that all commands return True for the state to run. If any
        command returns False (non 0), the state will not run
        r�FrXr#csF|dkr�ddur��dddd��dS|dkr!��ddd��dS)	Nrr�F�onlyif condition is falseT�r�r^r��onlyif condition is true�r�r�)r��cmd�r�r>r?�
_check_cmd:s��z+State._run_check_onlyif.<locals>._check_cmd�cmd.retcodeT�Zignore_retcode�python_shellr��Last command return code: %srzno `fun` argument in onlyif: r��
get_returnN�retcoderrgrhrirjz(onlyif execution failed, bad type passed)rwrnrzr�r=rr�r.ro�warningr�rfrGrHr��traverse_dict_and_listr1r)
r�rbrcr�Zlow_data_onlyifrnrrlrsr�r>rmr?r_+sj
��
���

�����zState._run_check_onlyifc
	s�ddi�dD]	}|�|��|<qt|dt�s|dg}n|d}�fdd�}|D]�}t|t�r[z|jd|fddd	�|��}t�d
|�WntyQd}Ynw||�rZ�Sq)t|t�r�d|vrvd
|���d<t�	�d��S|�
dd�}|�|�}	|r�tj
j�|	|�}	|j�dd�r�||jd�r��Sq)|	r���dddd��q)��ddd���S��ddd��q)�S)zq
        Check if any of the commands return False (non 0). If any are False the
        state will run.
        r�FrXr$cs>|dkr��dddd��dS��dd���ddd��dS)	Nr�unless condition is trueTrhFr^�unless condition is falserj)rr�rkrmr>r?rn�s��z+State._run_check_unless.<locals>._check_cmdroTrprrrrzno `fun` argument in unless: r�rsNrtrwrhrxrjz*unless condition is false, bad type passed)rwrnrzr�r=r�r.rrorur�rfrGrHr�rvr1r)
r�rbrcr�Zlow_data_unlessrnrrlrsr�r>rmr?r`xsl
�����

������zState._run_check_unlesscCs�ddi}dD]	}|�|�||<qi}d|jdvr#|jd�d�|d<|dD]9}|jd|fddd	�|��}t�d
|�|dkrP|ddurP|�ddd
��q'|dkr`|�ddd
��|Sq'|S)zD
        Alter the way a successful state run is determined
        r�FrXr]r�r roTrprrrz(check_cmd determined the state succeededrjz%check_cmd determined the state failed)rwr�r=r�r.r)r�rbr�r�rcrrlr>r>r?�_run_check_cmd�s<��������zState._run_check_cmdcCs�ddi}dD]	}|�|�||<qt|dt�r2tj�|d�r2d�|d�|d<d|d<d|d<|St|dt�rRtd	d
�|dD��rRd|d<d|d<d|d<|Sd|d<d|d<|S)
z/
        Check that listed files exist
        r�FrXr%z	{} existsr�Tr^cSsg|]}tj�|��qSr>)r^r_�exists)r�r_r>r>r?r���z,State._run_check_creates.<locals>.<listcomp>zAll files in creates existzCreates files not found)	rwrnr�r^r_rzrBrzr�)r�rbr�r�r>r>r?ra�s$���zState._run_check_createscCs
d|_dS)z-
        Rest the run_num value to 0
        rN)r6r>r>r>r?�
reset_run_numrFzState.reset_run_numc
CsZ|jdkrtj�|j|ji�|_dStjj|j|j|j|j|j	|j
tj�|j
�d�|_dS)zU
        Read the state loader value and loadup the correct states subsystem
        �thorium)rrr(N)r'rGr!r}r�r=rrH�serializersr1rr%�ContextlessFileClientr(r>r>r>r?�_load_statess


��zState._load_statesc	
CsBt�d�tjj|j|jd�|_tjj|j|j|j|j	tj
�|j�d�|_t
|t�r�|�dd�r�t
|dt�rA|d|dig}nt
|dt�rM|d}ni}|D].}|D])}tj�|j|||j�}|r~|D]}d�|||�d�d	��}|||j|<qgqUqQtj�|j�|_|��tjj|j|j|j|j	|j|jd
�|_d	S)�1
        Load the modules into the state
        �(Loading fresh modules for state activity)r()rHrr(�providerFrz{}{}rfN)rrr(r)r�r�rGr!rHr�r(Zminion_modsr1rr%rr=rnrorwr�rzZraw_modrB�rindexr~r��renderrr�)	r�r�rZ	providersr��modr�funcZf_keyr>r>r?r2sJ
�

����zState.load_modulescCs�t�d�|jd�d�dkr2zt�t�Wnty$t�d�Ynt	y1t�d�Ynw|�
�|j�dd�sL|j�dd	�rN|jd
�dSdSdS)z)
        Refresh all the modules
        zRefreshing modules...r�r^ZMacOSzBError encountered during module reload. Modules were not reloaded.�localF�multiprocessingTzsaltutil.refresh_modulesN)r�r.r�rw�	importlib�reload�site�RuntimeErrorr�r�r2r=r>r>r>r?�module_refreshBs"
����zState.module_refreshcCs,d}|�dd�rt�d�tj�|j�|jd<d}|�dd�r-t�d�|��|jd<d}|d	s=|�d
d�r;|��dS|�dd�sE|rK|��dS|d
dkr�|ddkrf|d�	d�rd|��dSdS|ddkrr|��dS|ddkr�d|dvr�|��dSdSdS|d
dvr�|��dSdS)a�
        Check to see if the modules for this state instance need to be updated,
        only update if the state is a file or a package and if it changed
        something. If the file function is managed check to see if the file is a
        possible module type, e.g. a python, pyx, or .so. Always refresh if the
        function is recurse, since that can lay down anything.
        Fr*zRefreshing grains...r�Tr+zRefreshing pillar...r$r�Zforce_reload_modulesNr)r�filerZmanagedr:)z.pyz.pyxz.pyoz.pycz.soZrecurse�symlink�bin)r�ZportsZpip)
rwr�r.rGr!r�r�r/r��endswith)r�r�r�Z_reload_modulesr>r>r?�
check_refreshYs>

���zState.check_refreshc
	Cs�g}d|vr|�d�d|vr|�d�d|vr|�d�|drDt|dt�sD|�d�|dd|vr:d	�|d�nd
t|d�j��d|vrl|�d�}ztt|�d�|d
<Wntt	fyk|�d|���Ynw|rp|S|dd|d}||j
vr�d|vr�|�d�||d��|j
�|�}|r�|�d|���nJ|�d|�d��n@tj
j�|j
|�}d}d}t|jt�r�t|j�}t|jt�r�t|j�}t||�D]}	|j|	|vr�|�d�|j|	|��q�d
}
d|vr�d}
d|v�rd�|d�|j
v�rd|v�r|d�|�d��n|�d�|d<d}
nd}
|
�rY||
D]4}tt|��}|d|k�rWt�|d||��sIt�|d||��rW|�d�|d||d���q$|S)zR
        Verify the data, return an error statement if something is wrong
        rzMissing "state" datarzMissing "fun" datar:zMissing "name" dataz0ID '{}' {}is not formed as a string, but is a {}r4zin SLS '{}' r�r0�r9zInvalid umask: rfz$State '{}' was not found in SLS '{}'zReason: zSpecified state 'z' was not foundrz!Missing parameter {} for state {}rr�{}.mod_watchr5z<Recursive require detected in SLS {} for require {} in ID {})rxrnr�rBr�r�r�r�r��
ValueErrorrZmissing_fun_stringrGrHrsZget_function_argspecrzrp�defaults�tuple�ranger�rgrh�fnmatch)
r�r�r�r0r@�reasonZaspecZarglenZdeflen�indZreqdecrirjr>r>r?�verify_data�s�


��
�
��


���

����zState.verify_datacCs�g}t|t�s|�d�t�}|��D�]�\}}z	|�d�r WqWn	ty*Ynwt|t�s?|�d�||dt	|�j
��t|t�sPd�||�}|�|�q|D�]�}|�d�r[qR||durn|�d�||d|��qRt||t�s�|�d�||d��qRd	}d
|vr�|d7}||D�]}	t|	t�r�|d7}d|	��vr�|�d
|	�d|�d|d�d��q�t|	t��r�t
t|	��}
|
dkr�t|	|
t�s�|�d�||d��|
dv�r�t|	|
t�s�|�d�|
||d��n�t|d�||<|	|
D]�}t|t��rd|i}t|t��s|�d�||d��q�t
t|��}||}
d
|v�r:|�d�|||d|d|�d
����t|
��sH|�d�|
��q�||||
<|
|v�r||||
v�r|||
||k�r|||
d|||
k�r||�d�|d||
��q�tt|	��dk�r�|�d�||d��q�|�s�|dk�s�|dk�r�qR|�d|�d |d�d!��qR|dk�r�d
|v�r�|jd
dd"�dgng}|�d#d$�||D��|�d%|�d |d�d&�d'�|��qRq|S)(r�r�r�zUID '{}' in SLS '{}' is not formed as a string, but is a {}. It may need to be quoted.r4r�Nz�ID '{}' in SLS '{}' contains a short declaration ({}) with a trailing colon. When not passing any arguments to a state, the colon must be omitted.r�rrfrlr�r�r�r�r�rzKThe 'names' argument in state '{}' in SLS '{}' needs to be formed as a listr�r�)rr�r�r�z2Illegal requisite "{}", please check your syntax.
rr�r�rrr�r�r�r�csr�r�r�r�r>r>r?r�z��z$State.verify_high.<locals>.<genexpr>r�r�r�)rnrorxr
ryr�r�r�rBr�r�rzr�rgrhr�r�rpr;r�r`r�r>r>r?r��s(


��
��
�



����
�����
���
��
�


���������
"������zState.verify_highcCs"g}|D]
}|�|�|��q|S)zD
        Verify the chunks in a list of low data structures
        )r�r�)r�rr�r�r>r>r?�
verify_chunks�szState.verify_chunkscCsd}|D] }d|vr$t|dt�sq|d}||dkr$|dkr$|d}q|D]Q}d|vr2||d<q't|dttf�sW|ddkrH|d|d<n|ddkrSd|d<n||d<d|vrh|d|�d�d	|d<|ddkrx|d|d|d<q'|jd
d�d�|S)
r�rlr'rr�r�rNr�r�r�cSr�r�rAr�r>r>r?r��r�z$State.order_chunks.<locals>.<lambda>r�)rnr�r�r�r�rr>r>r?r�s8�
��zState.order_chunkscCs�g}|��D]�\}}|�d�rq|��D]�\}}t�}g}	|�d�r#q||d�}
|dur0||
d<d|vr:|d|
d<d|vrD|d|
d<||
d<|D]E}t|t�rW|�|�qJt|t�r�|��D].\}}
|dkrw|
D]}||	vru|	�|�qjq`|d	kr|q`|d
kr�t|
t�s�||
|<q`|
|
|<q`qJ|	r�d}|	D]=}t�	|
�}t|t�r�t
t|����}||d
<t
t|j||��n||d
<||d<|d7}|D]}||d
<|�|�q�q�qt�	|
�}|D]}||d
<|�|�q�qq|�|�}|S)rr�rNr6r4r3r5rrr:rlr�r)ryr�rmrnr�rqrorxrrrgrhr	rzr
rr)r�rr�orchestration_jidrr:r�rrrrr�r~r�r
rr�rrrrr>r>r?r�sv






��

�

��
��
3zState.compile_high_datac
Cs�g}d|vr
||fS|�d�}|D�]B}|��D�]:\}}tdd�|D��}||vs0|||vrWt||||d�}	t|	�dkrQ|�d�||�dd�|�d	d���q|	d
d
}g}
|�dg�D]}t|t	�rl|
�|�q_|�d�rx|
�|d�q_|�d	�|
vr�t
�d
||�dd�|�d	d�|�dd�|�d	d��q|��D]�\}}
|�d�r�q�|||vr�|
|||<q�|
D]�}d}t
|||�D]�\}}t|t	�r�t|t	�r�|||�|�|||�||�d}q�t|t��rDt|t��rDtt|��}|tt||||��k�r(|tv�r|||||�||�n|||||<d}|dk�rDtt||||��dk�rD|||||<q�|�sQ|||�|�q�q�qq||fS)zM
        Pull the extend data and add it to the respective high data
        �
__extend__css�|]
}|�d�s|VqdS)r�N�r�r�r>r>r?r��r�z)State.reconcile_extend.<locals>.<genexpr>�r{rlz�Cannot extend ID '{0}' in '{1}:{2}'. It is not part of the high state.
This is likely due to a missing include statement or an incorrectly typed ID.
Ensure that a state with an ID of '{0}' is available
in environment '{1}' and to SLS '{2}'r3�baser4rr�rvz=Cannot extend ID '%s' in '%s:%s' because '%s:%s' is excluded.r�FTr:r)r�ryrgrrprxrBrwrnr�r�r.r��	enumerate�insertrorhrQr�)r�rrr{r��extZ	ext_chunkr:r�Z
state_typeZidsZsls_excludesrrrr~rZhindr
r�r>r>r?�reconcile_extend�s�



��

�



�

�
�����OzState.reconcile_extendcCs
d|vr|St�}t�}|�d�}|D]7}t|t�r|�|�t|t�rJt|�dkr+qtt|�	���}|dkr?|�|d�q|dkrJ|�|d�q|ru|�
�D]#\}}|�d�r[qQ|�dd�}	|	sdqQ|D]
}
t
�
|	|
�rs|�|�qfqQ|D]}||vr�|�|�qw|Sr)rmr�rnr�rqrorprgrhr	ryr�rwr�)r�rrrrrrr�r:r�rvZex_r=r>r>r?rJsD



�

��
�zState.apply_excludec$Cs�hd�}|�hd��}i}g}|j�dg�}t|t�s|g}|��D�]|\}}t|t�s-q"|��D�]l\}	}
|	�d�r<q1|
D�]^}t|t��r�t|�dkrNq>t	t
|��}||vrYq>||vrdt�d|�q>|�
d�d}
||}t|t��r|��D]�\}}d	}||vr�t�||<d
|vr�|�d�|
|||d|d
|�d
����|�
d
�d}|||vr�g|||<|d||d<|d||d<tt|||��D]!}t	t
||||��|
kr�|||||
�|	|i�d}q�|r�qy|||�|
|	|igi�qyt|t��r�g}|D�]�}t|t��sl||v�r*dd�||D�}|d|i}nBd	}t
|�D]6}dd�t
||�D�D]'}t
|||�D]}t|t��rad|v�ra|d|k�ra||i}d}�qG�q=�q0|�sl�qt|�dk�ru�qt	t
|��}||}|dk�r�t||�}n|�||f�d
|v�r�|�d�|
|||d|d
|�d
����|�
d
�d}|D�]�}|\}}|dk�r�||v�r�t�||<|	||v�r�g|||	<|||	�d||igi�|dk�r!t|||dd�}|D]*\}}||v�rt�||<|||v�rg|||<|||�d|	|igi��q�q�|dk�r�t|||dd�}|D]u\}}|�s9�q0t|||�}||v�rIt�||<|||v�rVg|||<|�|�}|||	D]B}t|t��sk�qat|�dk�rt�qat	t
|��|v�r�qat	t
|����dk�r��qat	t
|����dk�r��qa|||�|��qa�q0�q�|dk�r/t|||dd�}|D]u\}}|�s��q�t||	|�} ||v�r�t�||<|	||v�r�g|||	<|�| �}|||D]B}t|t��s�q�t|�dk�r��q�t	t
|��|v�r�q�t	t
|����dk�r�q�t	t
|����dk�r �q�|||	�|��q�q��q�d	}||v�r;t�||<|||v�rHg|||<|d||d<|d||d<tt|||��D]#}t	t
||||��|
k�r�|||||
�|	|i�d}�qb|�r��q�|||�|
|	|igi��q��qq>q1q"g|d<|��D]\}}!|d�||!i��q�|j|dd�\}"}#|�|#�|"|fS)zG
        Extend the data reference with requisite_in arguments
        >rr1rrr2rrr>rr
rrr�disabled_requisitesr�rl�-The %s requisite has been disabled, Ignoring.r�rFrfzeInvalid requisite in {}: {} for {}, in SLS '{}'. Requisites must not contain dots, did you mean '{}'?r4Nr3TcS�g|]	}|�d�s|�qS�r�r�r�r>r>r?r�����z&State.requisite_in.<locals>.<listcomp>cSr�r�r�)r�Z_st8r>r>r?r��r�r:rvrrrr�r2rr1r�)rPr�rwrnrzryror�rprgrhr�rur;r
rxrBr�r�r�rrur	r�r�)$r�rrZreq_inZ
req_in_allr�r��
disabled_reqsr=r�rrr~r�Zrkeyry�_stater:�foundr�ZhingesZ	_ind_high�_idZst8�jZpstateZpname�tupZext_idsr|Z
_req_stateZext_argsZignore_argsZloc_argsr
Zreq_in_high�
req_in_errorsr>r>r?�requisite_inrs�
�



�
����

�

��
���


��



�
�

�
�


�
�


�


��� ����Y
zState.requisite_inc
Cs�|dur|d#i|��}||j_tj��}|�|�t|�}z|j|d|di|d��}	Wn)tyX}
ztjd|
t	j
d�t��}d|id|��d	�}	WYd}
~
nd}
~
wwtj��}tj��tj�
�}
||
}||
}|����|	d
<||}|jd|jd}||	d
<d|	d<d|v�rnd}|�|d�|d<|jjd�sR|dd|k�rQ|dd|	dkr�n�|dd}|dddkr�|t�d|dd�}t�d|�t�|�|j|d|di|d��}tj��}tj��}||}|jd|jd}||d
<|	}|}	|	d�sd}nd�||	d|	d�}d�|d|g�|	d<|	d
|d
|d|	d
<|dk�rE|d
|	d
<|d}|dd|ks�nd�|	d�s[dnt|	d�d jd#i|d��g�|	d<tj�|jd!|j�}tj�|tjj �!|��}tj�"|��s�zt�#|�Wn
t$�y�Ynwtjj%�&|d"��}|�'t(|	��Wd�dS1�s�wYdS)$zZ
        The target function to call that will create the parallel thread/process
        Nr@rsr��'An exception occurred in this state: %s�Zexc_info_on_loglevelF�%An exception occurred in this state: �r�r:r�r�rZrNrOr\TZ__parallel__r&rl�test�attempts�untilr��interval�splayr�QState result does not match retry until value, state will be re-run in %s secondsr�r��GAttempt {}: Returned a result of "{}", with the following comment: "{}"�
���  ��The state would be retried every {interval} seconds (with a splay of up to {splay} seconds) a maximum of {attempts} times or until a result of {until} is returnedrczwb+r>))rr9rPrQrerErFr�r.�logging�DEBUG�	traceback�
format_excrRrSrTrUrV�verify_retry_datar��random�randintr��sleeprBr`r�r^r_r7rGrH�	hashutils�sha1_digestrarb�OSErrorrI�fopen�write�msgpack_serialize)�cls�instance�init_kwargsr:r�rDr9rWr<r�r�trbrY�timezone_delta�local_finish_timerXr[r\�retriesr��	retry_ret�orig_retZ_commentZtrootZtfile�fp_r>r>r?�_call_parallel_targetjs�

$����

�
��


���
��-����
�$�zState._call_parallel_targetc
Cs\|�d�pdgdp|d�d�}|s|�d|�d��}tjj��r%d}n|}d}tjjj|j||j||||fd|�d�d	�}z|�	�Wn^t
y�}zRd
t|�vrT�i}|jd��D]\}	}
zt
�|
�Wn	t
yqYq]w|
||	<q]|j��}||d<tjjj|j||||||fd|�d�d	�}|�	�WYd}~nd}~ww|did|d
�}|S)zG
        Call the state defined in the given cdata in parallel
        rsNrr�r:r5zParallelState(�))�targetrsr:z
cannot picklerzStarted in a separate process)r:r�r�r��proc)rwrGrH�platformZspawning_platform�processZProcessr�r&�startr�r�ry�pickle�dumpsr)
r�r�rDr9r:r�r�r�Z
clean_context�varr
r�r�r>r>r?�
call_parallel�sT"	
��


����zState.call_parallelZ
content_checkZunifyrlc
s�tj��}|tj��tj��}t�dt�dt�r!�d��n�d|���	��|�
��}|rcd�didd�}|D]
}	|d|	�d�7<q=|j|d<|jd	7_t|�|�
�|�|Sd�did
�}��dd�|jd<�d
dkr�d�vr��d|jd<n	��dd�|jd<��d�s�t�d�d
�dt�dt�r��d��n�d�d�vr�|���d���}
tjjj|j|
�d|
itd�}t���|r�t�|�ni|j|r�t�|�ni|j�dd�d�}d�vr�t�d�|d<|j�r|�|j���d��r|jjd}
d|jjd<�zv�zd}t�fdd �|D���r7d!���|jv�r7|�|����|j�d"d��scd#�v�rMt�d#�|d<nt|d$�d%d�t��rct|d$d%�|d<d|v�rld&|d<d'�v�rw�d'|d'<d(|v�s�|d(du�r�||j_|j �r�t!|�}n]|�"|d�d
�d�}|�r�t�d)�d�nD��d��s���d*��r�|�#|�|�}n0|�$|�tjj%�&��d+���|j|d|d,i|d$��}Wd�n	1�s�wYi|j_d-�v�r%d.���}d���}t'�(|j|�}||jv�r|�|�)���nd-|j*v�r%|�|�)���WnIt+�yo}z<tj,d/|t-j.d0�t/�0�}|�d,��pDdgd1�pN|d$�d�}|�s[��d��d2��}d|id3|��d�}WYd}~nd}~wwW��d��r}|
|jjd<|j�1dd�|j�1dd�n��d��r�|
|jjd<|j�1dd�|j�1dd�wt|t2��s�|Sd4|v�r�|�3d4g��4|d4�d�v�r�|����d��r�d�d<|S��d5�|d5<|j|d<|jd	7_t|�|�
�|�tj��}tj��tj��}||}||}|���	�|d6<||}|j5d7|j6d8}||d9<�d2|d2<t�d:t�dt��r9�d��n�d|���	�|�d;�v�r�d*�v�r�|�7�d;��d;<|jjd�s߈d;d<|d(k�r݈d;d=|k�r݈d;d>}�d;d?d1k�r�|t8�9d1�d;d?�}t�d@|�|j:dA|�|j;�|||d	dB�}|}|}d�<dC�||d(|d�|d�s�dn|dg�|d<|d9|d9|dD|d9<|d	k�r�|d6|d6<|SdE�<|d�s�dnt|d�dFjdGi�d;��g�|d<|S)Hzk
        Call a state directly with the low data structure, verify data
        before processing.
        zRunning state [%s] at time %sr:Fr�r�r�r��__run_num__rl)r�r:r�r,Nrrl�passwordr-r7zExecuting state %s.%s for [%s]rr�z{0[state]}.{0[fun]}r@)Zinitial_retZexpected_extra_kws�user�UNKNOWN)Z__low__Z__running__Z__instance_id__Z__lowstate__Z__user__r3r�T)r$r#r%c3s�|]}|�vVqdSr�r>)r�rirCr>r?r�p	��zState.call.<locals>.<genexpr>z{0[state]}.mod_run_checkZlock_saltenvr/r�r[r�r6r�z>Failed to meet global state conditions. State '%s' not called.r(r9rsr z{0[state]}.mod_run_check_cmdr�r�rr5r��warningsr4rZrNrOr\z3Completed state [%s] at time %s (duration_in_ms=%s)r&r�r�r�r�r�z
test.sleep)r�r�r�r�r�r>)=rPrQrRr�r�rnr�r�rSrTr�r6r�r�rwr1r2rBrGrHrsZformat_callr�STATE_INTERNAL_KEYWORDS�immutabletypesZfreezer8r�r9rr?rdr r�rBr�rerI�	set_umask�inspectZ	signatureryZ
parametersrFr.r�r�r�r�r�ro�
setdefaultr�rUrVr�r�r�r=�callr`)r�rDrrVr�rWrXr�r�r�Zstate_func_namer�r9r�Zreq_listZstate_check_cmdZ
state_funcZstate_func_sigrr�r:rYr�r�r[r\r�r�r�r>rCr?r�	s�
� 
�
�

 �

��	�



��
���


��&�����



"
������
�
�����	z
State.callc
Cs�t�d|�|�dd�}t|�dkrt�d|�|S|ddkr-t�d|�t�d�|Stjj�|d�\}}}|r@||j	vrMt�d|�t�d	�|St�d
|||�|j	||i|��}|d�d�d}|d�dd�dd�}t�d
|�t�d|�d}	z
||�
d�dd�}	Wn	ty�Ynw|	r�|	�dd�d��}	t�d||	�tjj
j||	ddd�}|r�t|t�r�d�|���}t�d|�||7}|St�d�|S)NzEvaluating slot: %s�:r��zMalformed slot: %srlrGzkOnly execution modules are currently supported in slots. This means slot should start with "__slot__:salt:"z[Execution module should be specified in a function call format: test.arg('arg', kw='kwarg')zCalling slot: %s(%s, %s)�~rz
slot_text: %szappend_data: %sr�rfzSearching slot result %s for %s)�default�	delimiterr�zappending to slot result: %sz1Ignoring slot append, slot result is not a string)r�r.r;rprurGrHrsZparse_functionr=r�r�r�r�rvrnr�r`r�)
r�Zslot�fmtrrsr�Zslot_returnZ	slot_textZappend_dataZ
return_getr>r>r?Z__eval_slot
sX����

�zState.__eval_slotcCs�d}dt|d�fd|d��ff}|D]�\}}|D]�\}}tjjj|dd�}t|t�rW|��D]%\}}	z|	�|�rKt	�
d|	�|�|	�||||<Wq0tyUYq0wqt|t
�r�t|�D]V\}
}t	�
d|�t|t�r�|��D]'\}}	z|	�|�r�t	�
d|	�|�|	�||||
|<Wqsty�Yqswt|t�r�|�|�r�t	�
d	|�|�|�||||
<q`qt|t�r�|�|�r�t	�
d
|�|�|�|||<qqqdS)a!
        Read in the arguments from the low level slot syntax to make a last
        minute runtime call to gather relevant data for the specific routine

        Will parse strings, first level of dictionary values, and strings and
        first level dict values inside of lists
        z	__slot__:rsr�T)ZkeepzSlot processsing dict value %szSlot processing list value: %sz%Slot processsing nested dict value %sz!Slot processsing nested string %szSlot processsing %sN)r�ryrGrHr�r�rnror�r�Ztrace�_State__eval_slotr�rzr�)r�r�Z	SLOT_TEXT�ctxZatypeZavaluesr�r~r��value�idxZ	listvaluer>r>r?reS
s`	 

���


����

�����zState.format_slotscCs�ddddd�}ttttd�}i}t|t�rH|��D],\}}||vr?t|||�r/||||<qt�d|||�||||<q||||<q|St�d�|}|S)	z3
        verifies the specified retry data
        Tr�r�)r�r�r�r�)r�r�r�r�zFAn invalid value was passed for the retry %s, using default value '%s'zdState is set to retry, but a valid dict for retry configuration was not found.  Using retry defaults)�boolr�rnroryr�ru)r�Z
retry_dataZretry_defaultsZ
expected_dataZvalidated_retry_dataZexpected_keyZ
value_typer>r>r?r��
s>��
�
���zState.verify_retry_datacCsTi}d|jdvrQ|dd�D]A}d�|d|d�}|jddD]-}t�||�rOd�||�}t|�}id||j|d	d
�||<|jd7_|�|�nq"qi}|D]7}d|vrd|�d�|St|�}	|	|vr�|�|�}
|
d
krwn|�|||�}|�	||�r�|St
�|_qU	|�|�r�nt
�d�q�tt|���t|����}|S)zU
        Iterate over a list of chunks and call them, checking for requires.
        Zstate_runs_disabledr�NrJrrz\The state function "{0}" is currently disabled by "{1}", to re-enable, run state.enable {1}.Fr4)r�r�r�r�r4rl�__FAILHARD__�killT�{�G�z�?)r�rBr�rEr6�remover��check_pause�
call_chunk�check_failhardrmr3�reconcile_procsrSr�rorzry)r�r�disabledrDZstate_�patr�Z_tagrVr<�actionr�r>r>r?�call_chunks�
sZ��
�
��




�zState.call_chunkscCsZt|�}|j�dd�r
dS|�d|jd�r+||vr+||ddur$dS||dSdS)zK
        Check if the low data chunk should send a failhard signal
        r�Fr"r�N)rEr�rw)r�rDrVr<r>r>r?r�
szState.check_failhardc
Cs�|jsdStj�|jdd|j�}t��}tj�|�r�z�	d}tjj	�
|d���}zt|���}Wn*tjj
jjyZ|dkrIYWd�WdS|d7}t�d�YWd�qw|d	}d
}||vrh|}nd|vrnd}|r�d||vr�t��}	|	|||dkr�	Wd�Wd
Sd||vr�	Wd�WdSn
	Wd�Wd
St�d�Wd�n1s�wYqty�}
z
t�d|�WYd}
~
d
Sd}
~
wwd
S)z@
        Check to see if this low chunk has been paused
        Nrc�state_pauseTr�rb�
rlr5r��__all__r\rrz4Failed to read in pause data for file located at: %s)rr^r_r`r�rS�isfilerGrHrIr��msgpack_deserialize�readZmsgpack�
exceptionsZUnpackValueErrorr�rFr�r�)r�rD�
pause_pathr��triesr�Zpdatr=r�rRrr>r>r?r�
sb�
���������zState.check_pausec	Cst�}|D]z}||�d�}|r|��sztj�|jd|jtj	j
�|��}tj�|�s6dd||did�}z tj	j
�|d��}t|���}Wd�n1sPwYWntyjdd	||did�}Ynw||�|�||�d�q|�d�qd|vS)
zG
        Check the running dict for processes and resolve them
        r�rcFz!Parallel process failed to returnr:)r�r�r:r�r
NzParallel cache failure)rmrw�is_aliver^r_r`r�r7rGrHr�r�rrIr�rrr�rr�rq)r�rVZretsetr<r�Z	ret_cacher�r�r>r>r?r%sB�
���

��
�zState.reconcile_procsc
Csh|j�dg�}t|t�s|g}d}d|vr8d�|d�|jvr6d|vr.|d�|�d��n
|�d�|d<nd}d|vr_d�|d�|jvr]d	|vrU|d	�|�d��n
|�d�|d	<nd}d|vred}d	|vrkd}d
|vrqd}d|vrwd}d|vr}d}d
|vr�d}d|vr�d}d|vr�d}d|vr�d}|s�dS|�|�ggggggggggd�
}|r�g|d
<|D]�}||v�ro||du�ro||vr�t	�
d|�q�||D]�}	t|	t�r�d|	i}	t|	�}	d}
t
t|	��}|	|}|dur�q�|D]t}
|dk�rt�|
d|��rd}
||�|
�q�z2t|t��r;t�|
d|��s%t�|
d|��r:|dk�s1|
d|k�r:d}
||�|
�nt�Wq�t�yU}z
td�||
d���d}~wt�yftd�||
d���w|
�sndSq�q�t�}|��D�]S\}}t�}|�d��r�|�d
��s�|j}n|}i}|D]}
t|
�}|�|�}|�r�|||<�q�|}	|�|��r�nt�d��q�|D]�}
t|
�}||v�r�|�d��q�|�d��r�||ddu�r�|�d��q�n||ddu�r�|�d��q�|�d��r||d �s|�d�n|�d!��q�|�d��r#||d �r#|�d"��q�|�d��r=||ddu�r=|�d
��s=|�d#�|�d��rY||ddu�rY|�d
��sW|�d$��q�||�d%d��rg|�d&��q�|�d'��st|dk�r�d&|v�s~d"|v�r�d|v�r�|�d�d!|v�r�d|v�r�|�d�d|v�r�|�d�d|v�r�d&|v�r�|�d�|�d(��r�d|v�r�d&|v�r�|�d&�|�|��qwd|v�r�d}||fSd|v�r�d}||fSd$|v�r�d#|v�r�d&}||fSd$}||fSd|v�rd!|v�rd}||fSd|v�rd!|v�rd}||fSd"|v�r#d"}||fSd|v�r.d}||fSd&}||fS))z`
        Look into the running data to check the status of all requisite
        states
        r�Frr�rrTrrrrr
rrrr)�metr>)
rrrrrr
rrrrNr�r�rvr4r:r5zBCould not locate requisite of [{}] present in state with name [{}])�unmetr>rrr��failr�Zonchangesmet�changeZpremetr5�
__state_ran__rZ_anyZ_all)r�rwrnrzrBrr�r�rr�rur�rkrgrhr�rxrKrr�rmryr�r5rErSr�rqr�rr)r�rDrVrr5r�Zpresentr�Zr_staterir�r�r�r�rZ	fun_statsZ	req_stats�run_dictZfiltered_run_dictr<Zrun_dict_chunk�statusr>r>r?�check_requisiteJs�

��

����������
�
�


�




�
�














�
�
��
�
�
��zState.check_requisitecs��j�d�s{�j�dd�s|r}�j�d�sd�fdd�	}n�jd}d	|i}|dur>tjj��j�jd
t|d�gd�}n/t	|t�rUtjj��j�jd
t|�gd�}ntjj��jd
�jd
t|d�gd�}||d<d�ji}||||d�dSdSdS)a
        Fire an event on the master bus

        If `fire_event` is set to True an event will be sent with the
        chunk name in the tag and the chunk result in the event data.

        If `fire_event` is set to a string such as `mystate/is/finished`,
        an event will be sent with the string added to the tag and the chunk
        result in the event data.

        If the `state_events` is set to True in the config, then after the
        chunk is evaluated an event will be set up to the master with the
        results.
        r��state_eventsT�
master_uriNcsNtjjj�j�jddd��}|�||�Wd�dS1s wYdS)NZsock_dirF)r)rGrH�eventZget_master_eventr�r.)r�r<�preloadZ_evtr>r>r?�ev_func)s�"�zState.event.<locals>.ev_funczevent.fire_masterr�r�r:Zstate_result�progr�Zjobrpr)r"r�)
r�rwr=rGrHr!Ztagifyrr�rn)r�Z	chunk_ret�lengthr.r#r�r<r"r>r>r?r!s6��
�
��
�zState.eventrc"
Cs8	|�|||�}|�|�t|�}|�d�s|j�|�gd�}|�d�s5|�d�|j|||dd�\}}n	|�|||�\}}|dk�ri}	g}|D]�}
g|	|
<|
|vrTqI||
D]|}t|t	�rcd|i}t
|�}d}tt|��}
||
}|D]U}|d	ur|qu|
d
kr�t
�
|d|�r�|
dkr�d|d<|�|�d}qut
�
|d
|�s�t
�
|d|�r�|
dks�|d|
kr�|
dkr�d|d<n|
dkr�d|d<|�|�d}qu|s�|	|
�|�qXqI|	d�s	|	d�s	|	d�s	|	d�s	|	d�s	|	d�s	|	d�s	|	d�s	|	d�s	|	�d��rd}|	��D]+\}
}|�s�q|d�d|
�7}|D]}tt|��}
||
}|d�d|
|�7}�q"�q|�d��rE|j}n|}t�\}}id||||jd�||<dD]}|�|�|||<�qZ|jd 7_|j||t|�|�d!�d"�|S|D]�}t|�}||v�r||jv�r�|�d��r�||jv�r�d|d<|�|||�|j|<|S|S||v�r�t�d#�idd#|jd$�||<dD]}|�|�|||<�q�|jd 7_|j||t|�|�d!�d"�|S|�|||�}|�||��rd|d%<|S�q�|�d��r@|�|||�\}}|�|||�|j|<|j|d&�s?|d'k�r?ddi|j|d&<d	|j|d(<n0|d 7}|d)k�rht�d#�idd#|jd$�||<dD]}|�|�|||<�qZn|�||||�}|�||��r}d|d%<|S�n�|d*k�r�|�d��r�|�|||�|j|<�n�|�|||�||<�n�|d+k�rS||jv�r�|j|�d&��r�i|j|d&<|j|||<|j||d,<dD]}|�|�|||<�q�nqt�}|��D]5}|D]/}t|�}|j�||�|��}|d	u�r��q�|d(du�rd-j|d|dd.�}|�|��q�q�d/�d0�d1d2�|D���}t�\}}id||||jd�||<dD]}|�|�|||<�q7|||j|<|jd 7_n�|d'k�r�|�d��s�|�|||�}|d&�s�|�d3d��s�|��}|d4|d5<d6|d4<||d7<|�|||�}|||<n�|d8k�r�t�\}}id||d9|jd�}dD]
}|�|�||<�q�|||<||j|<|jd 7_nv|dk�r�t�\}}id||d:d|jd;�||<dD]}|�|�|||<�q�|jd 7_nH|dk�rt�\}}id||d<d|jd;�||<dD]}|�|�|||<�q|jd 7_n|�d��r.|�|||�|j|<n	|�|||�||<||v�r�|j||t|�|�d!�d"�||�d=d>�D]F} t�\}}|jd 7_t| d?�}!| d?d
| d&| d(| �d@|�| �dA|�| �dBdC�d|jdD�||!<dD]}|�|�||!|<�q��qS|S)Ezd
        Check if a chunk has any requires, execute the requires and then
        the chunk
        r)	rrrrrr
rrrr7T)r5rr�FNrvr4rr:r5rr8rrr
rrrrrz)The following requisites were not found:
z{}{}:
z                   z	{}{}: {}
z                       )r�r�r\rZr�r�rXrlr.)r.zRecursive requisite found)r�r�r�r�rr�rr��rrr�z{sls}.{_id})rvr�z One or more requisite failed: {}r�css�|]}t|�VqdSr�)r�)r��ir>r>r?r�
r�z#State.call_chunk.<locals>.<genexpr>r^r�sfun�	mod_watchZ__reqs__r5zNo changes detectedz3State was not run because onfail req did not change)r�r�r\rZr�rr�z<State was not run because none of the onchanges reqs changedZ
sub_state_runr>rDr\rZr�r�)r:r�r�r\rZr�rr�)rWrMrErwr3rqrxrrnr�rkrgrhr�ryrBr5r]r6r!rpr�r�r�rrrmr�r`rr�)"r�rDrVr�depthr<rSrr�ZlostZ	requisiterir�r�r�r�r�ZlreqsZlreqrrZr\r�ZctagZfailed_requisitesZreq_lowsZreq_lowZreq_tagZreq_retZ_cmtr�Zpre_retZsub_state_dataZsub_tagr>r>r?rFs






�

�����������	�
��

�



����


���

�
�
���

�


�



�	


�






�
�zState.call_chunkc
Cs�g}|D]}d|vr|�|�qg}i}|D]}|��}|d|d<d|d<d�|d�|d<|�|�q|�|�}|�|�|D]}	|j||	d<|jd7_qA|�|�|S)	zY
        Find all of the beacon routines and call the associated mod_beacon runs
        Zbeaconrr(Z
mod_beaconz	beacon_{}r5r�rl)rxrrBrrr6)
r�rrVZbeaconsr�Zmod_beaconsr�rDr�r�r>r>r?�call_beaconsq
s(
�


zState.call_beaconscs�g}i}|D]J}|||d|d|df<d|vr+|�|d|d|df|di�d|vrP|dD]}|��D]\�}|��|df|d|digi�q9q3qg}i}	|D]�}
|
��D]�\�}|D]�}t|t�s�d}|D]}|d|ks||d|kr�|d|di}d}qn|s�qc|��D]�\��t��fd	d
�|D��s�t���d����d��d
���did�i}
|	�|
�q���fdd�|��D�}|D]}}||vr�q�||d�rEt�fdd
�|D���st�d�d�d��d�d�d��d�d�did�i}
|	�|
�qȇfdd�|��D�}|D]/}|��}|d|d<d|d<d�|d�|d<t	D]
}||v�r<|�
|��q0|�|��qq�q�qcq]qW|�|�}|�|�|	D]}|j|	|d<|jd7_�qV|�|	�|S)zX
        Find all of the listen routines and call the associated mod_watch runs
        rr5r:rr�lookupFTc3s$�|]
}�|dko
�|vVqdS)rNr>�r��cref��lkey�lvalr>r?r��
s�"z$State.call_listen.<locals>.<genexpr>z&Referenced state {}: {} does not existrKr�)r�r:r�r�cs,g|]\}}�|dkr�|vrt|��qS�rrL�r�r.r�r/r>r?r��
s
�z%State.call_listen.<locals>.<listcomp>r�c3s,�|]}�d|dko�d|vVqdS)rrlNr>r-r�r>r?r��
s
��
�rrlzlisten_{}:{}cs0g|]\}}�d|dkr�d|vr|�qS�rrlr>r3r�r>r?r��
s
�rr(r)zlistener_{}r�)
rxryrnror?rMrBrrrQr�rr6)r�rrVZ	listenersZcrefsr�Zl_inr
Zmod_watchersr�Zl_dictZ	listen_tor�ZrerrorZto_tagsZto_tagZ
new_chunksrDrir�r�r>)r�r0r1r?�call_listen�
s�����
����
�
�����

�

������
I

zState.call_listenc
s�g}��|�\}}|�|�|���|��|r|S��|�\}}|�|���|�}|r/|S��||�}|r9|S��|�}��||�}��||�}�fdd�}|��j	dur}t
j��j
dd�j	�}	t
j�|	�r}zt
�|	�W|Sty|Y|Sw|S)zI
        Process a high data call and ensure the defined states.
        csXtj�t�jd��j�}zt�|�t�d|�WdSt	y+t�d|�YdSw)Nrcz Deleted accumulator data file %sz*File %s does not exist, no need to cleanup)
r^r_r`rer�r8rr�r.r�)Zaccum_data_pathr>r>r?�_cleanup_accumulator_data
s�
�z2State.call_high.<locals>._cleanup_accumulator_dataNrcr)r�r�r�r�rrrr5r+rr^r_r`r�rrr�)
r�rrr�r��
ext_errorsr�rr�r6rr>r>r?�	call_high�
s<





��zState.call_highc
	Cs�g}|s||fSt|t�s|�d|�d��||fSd}|D]}||vr3|�d�||��||fSq|D]�}t||t�sit||t�rid||vr_||�d�}|d|dgi||<q6|�d�||��q6t�}t||�D]e}	|	�d	�rzqr|||	dur�|�d
�|||	��qrt|||	t	�s�qrd|	vr�|	�d�}|d|vr�|�d�||��qr||�
|	�|||d<|||d�|d�|�|d�qr|�|	�qrq6||fS)Nz	Template �  does not render to a dictionary)�includerZextendszMThe '{}' declaration found on '{}' is invalid when rendering single templatesrfrrlz(ID {} in template {} is not a dictionaryr�z�ID '{}' in template {} contains a short declaration ({}) with a trailing colon. When not passing any arguments to a state, the colon must be omitted.zNID '{}' in template '{}' contains multiple state declarations of the same type)rnrorxrBr�r;rmr�r�rzr�rq)
r�rrr�r�Z
invalid_itemsrtr:r�r�r�r>r>r?r�#sp
�����

�

��$zState.render_templatecCsNt||j|jd|jd|jd�}|s|S|�||�\}}|r"|S|�|�Sr�)rr�r�r�r8�r�r�rrr�r>r>r?�
call_templateo��
zState.call_templatecCsNt||j|jd|jd|jd�}|s|S|�|d�\}}|r"|S|�|�S)zQ
        Enforce the states in a template, pass the template as a string
        r�r�r�z<template-str>)rr�r�r�r8r;r>r>r?�call_template_str�r=zState.call_template_strcC�|js
|j��dSdSr�)r)r(�closer>r>r>r?�destroy���z
State.destroycC�|Sr�r>r>r>r>r?�	__enter__��zState.__enter__cG�|��dSr��rA�r�r�r>r>r?�__exit__��zState.__exit__)
NNNNNFrNNN�NNr��F)NNrlr2)9r�r�r�r�r�rBr/rMrTrWrdrfr_r`ryrar|r�r2r�r�r�r�r�rrr�rr��classmethodr�r�rGrHZ
decoratorsrZ
OutputUnifierr�r�rer�rrrrrr!rr+r5r8r�r<r>rArDrIr>r>r>r?r�s�
�Z%-!>MP 
))X)	
&
?Z(y
q9;9*3
/
%
L
1-
g2Lrc@s8eZdZdZdd�Zdd�Zdd�Zdd	�Zd
d�ZdS)
�LazyAvailStatesz�
    The LazyAvailStates lazily loads the list of states of available
    environments.

    This is particularly usefull when top_file_merging_strategy=same and there
    are many environments.
    cCs||_ddi|_d|_dS)Nr�F)�_hs�_avail�_filled)r�Zhsr>r>r?r��s

zLazyAvailStates.__init__cCs8|jrdS|j��D]}||jvrd|j|<q
d|_dS�NT)rQrO�	_get_envsrP�r�r/r>r>r?�_fill�s

�
zLazyAvailStates._fillcCs|dkrdS|��||jvS)Nr�T)rUrPrTr>r>r?�__contains__�s
zLazyAvailStates.__contains__cCsF|dkr|��||jvs|j|dur|jj�|�|j|<|j|S)Nr�)rUrPrO�clientZlist_statesrTr>r>r?�__getitem__�s

zLazyAvailStates.__getitem__cCs.|��g}|jD]}|�|||f�q	|Sr�)rUrPrx)r�r�r/r>r>r?ry�s

zLazyAvailStates.itemsN)	r�r�r�r�r�rUrVrXryr>r>r>r?rN�srNc@s$eZdZdZdd�Zdd�Zdd�Zdd	�ZdCdd�Zd
d�Z	dd�Z
dd�Zdd�Zdd�Z
dCdd�Zdd�Zdd�Zdd�ZdDd d!�Zd"d#�Zd$d%�Zd&d'�Zd(d)�ZdCd*d+�Zd,d-�Zd.d/�ZdEd0d1�Zd2d3�Z	
	
	4		
	
dFd5d6�ZdCd7d8�ZdCd9d:�Zd;d<�Zd=d>�Z d?d@�Z!dAdB�Z"d
S)G�
BaseHighStatea
    The BaseHighState is an abstract base class that is the foundation of
    running a highstate, extend it and add a self.state object of type State.

    When extending this class, please note that ``self.client`` and
    ``self.matcher`` should be instantiated and handled.
    cCs(|�|�|_d|_|��|_t�|_dS)Ni')�_BaseHighState__gen_optsr��iorder�_BaseHighState__gather_avail�availr
�building_highstate)r�r�r>r>r?r��s
zBaseHighState.__init__cCst|�S)zO
        Lazily gather the lists of available sls data from the master
        )rNr>r>r>r?Z__gather_avail�szBaseHighState.__gather_availcCs�d|vr
|dr
|S|j��}t|t�s3d|d<d|d<tjj�d�|d<i|d<d	tj	gi|d
<|S|d|d<|�
dd�|d<|d�d�rO|d|d<n"|d�d�rftjj�|dd
d��|d<ntjj�|d�|d<|�
dd�|d<|�
di�|d<|�
d|d�|d<|d
|d
<|�
d|�
d��|d<|�
d|�
dg��|d<|�
d|�
d��|d<|�
d�p�|�
d�p�d|d<|�
d�p�|�
d�p�d|d<|�
di�|d<|�
di�|d<|�
dd�|d<|�
dd�|d<|S)z�
        The options used by the High State object are derived from options
        on the minion and the master, or just the minion if the high state
        call is entirely local.
        Zlocal_statez
jinja|yamlr�Fr"ztop.sls�	state_top�
nodegroupsr��
file_rootszsalt://�/rlN�state_top_saltenv�state_auto_order�top_file_merging_strategy�	env_order�default_toprrUZ	jinja_envZ
jinja_sls_envZjinja_lstrip_blocksZjinja_trim_blocks)rWr�rnrorGrH�url�create�syspathsZBASE_FILE_ROOTS_DIRrwr�)r�r�Zmoptsr>r>r?Z
__gen_opts�sL	

� �
���zBaseHighState.__gen_optscs�dg�d|jvr���fdd�t|jd�D��|j�dg�}t���fdd�|D�}|j���|r>�r>�fdd�|D�S|rB|S���fdd��D���S)	zM
        Pull the file server environments out of the master options
        r�rac�g|]}|�vr|�qSr>r>r���envsr>r?r�r{z+BaseHighState._get_envs.<locals>.<listcomp>rfcs"g|]
}|�vs��|�s|�qSr>)rq�r�r[)�membersr>r?r�scsg|]}|�vr|�qSr>r>rn)�client_envsr>r?r�!r{crkr>r>rnrlr>r?r�&r{)r�r�rzrwrmrWrm)r�rfr>)rprmror?rSs
"
�
zBaseHighState._get_envsNcCstt�}tt�}tt�}d}|jd}|dkr%|jds%|jds%td��|jdrg|j�|jd|jd�}|r^d}t||jj|jjd	|jjd
|jjd|jd|d�g||jd<nzig||jd<nqd}|j�	d
d�}|r{t
|t�s{t|�}|r�|gn|��D]>}	|j�|jd|	�}|r�|d}||	�
t||jj|jjd	|jjd
|jjd|	|d��q�||	�
i�t�d|	�q�|dkr�|dkr�|j�	dd�s�t�d|�|dkr�t�dt|jjd��|��D]!\}	}
|
D]}d|vr�q�|dD]	}||	�
|�q�|�d�q�q�|�r�g}
|��D]Y\}	}|
�
|	�|�s%�q|D]G}t�|j|	|�D]:}|||	v�r=�q2||	�
t|j�||	��	dd�|jj|jjd	|jjd
|jjd|	|d��||	�
|��q2�q'�q|
D]
}	|	|v�r|�|	��qs|�s|S)z&
        Gather the top files
        rreZsamer/rgzXtop_file_merging_strategy set to 'same', but no default_top configuration option was setr_rlr�r�r�)r/rrcFz#No contents loaded for saltenv '%s'r0rfNatop_file_merging_strategy is set to '%s' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.z�No contents found in top file. If this is not expected, verify that the 'file_roots' specified in 'etc/master' are accessible. The 'file_roots' configuration is: %srar:�dest�r)r	rzr�rrWZ
cache_filerrr�rwrnr�rSrxr�r.ru�reprryr�r��filterr]�	get_state)r�r�topsr:Zdoner��merging_strategy�contentsrcr/�ctops�ctoprvZpopsr�	sls_matchr>r>r?�get_tops)s�

�
�


���


����
��



����

��zBaseHighState.get_topsc	Cs�|jd}z#d|��}t||�}t|d�s$d|�d�}t�|�t|��W||�Sttfy?t�d|�|j}Y||�Sw)z-
        Cleanly merge the top files
        reZ_merge_tops_�__call__r�z' is not callablez?Invalid top_file_merging_strategy '%s', falling back to 'merge')	r�rE�hasattrr�r�r�r�ru�_merge_tops_merge)r�rvrwZ
merge_attrZ
merge_funcr�r>r>r?�
merge_tops�s$




�
���zBaseHighState.merge_topsc
Cstt�}|�dtt��}|D],}|��D]%\}}|dkrqz|D]}||||||<q Wqty9td��wq|��D]K\}}	|	D]D}|��D]=\}}|dkrTqK||kr`t�d||�qK||vrlt�d||�qKz|D]}||||||<qoWqKty�td��wqEq?|S)a�
        The default merging strategy. The base env is authoritative, so it is
        checked first, followed by the remaining environments. In top files
        from environments other than "base", only the section matching the
        environment from the top file will be considered, and it too will be
        ignored if that environment was defined in the "base" top file.
        r�r:�,Unable to render top file. No targets found.z�Section for saltenv '%s' in the '%s' saltenv's top file will be ignored, as the top_file_merging_strategy is set to 'merge' and the saltenvs do not matchz�Section for saltenv '%s' in the '%s' saltenv's top file will be ignored, as this saltenv was already defined in the 'base' top file)r	r
r�ryr�rr�r.)
r�rv�topZ	base_topsrzr/�targets�tgt�cenvryr>r>r?r�s^�����������zBaseHighState._merge_tops_mergecCsBtt�}|��D]�\}}tdd�|D��rc|jd}|�|g�}tdd�|D��r1t�d||�q|D].}|��D] \}}	||krBq9t�d||�|	D]}
|||
|||
<qKnt�d||�q3q|D]8}|��D]1\}}	|dkrtqk||kr�t�d	||�qkz|	D]}
|||
|||
<q�Wqkt	y�t
d
��wqeq|S)z�
        For each saltenv, only consider the top file from that saltenv. All
        sections matching a given saltenv, which appear in a different
        saltenv's top file, will be ignored.
        cS�g|]}|ik�qSr>r>r�r>r>r?r���z2BaseHighState._merge_tops_same.<locals>.<listcomp>rgcSr�r>r>r�r>r>r?r��r�zmThe '%s' saltenv has no top file, and the fallback saltenv specified by default_top (%s) also has no top filezDThe '%s' saltenv has no top file, using the default_top saltenv (%s)zlThe '%s' saltenv has no top file, and no matches were found in the top file for the default_top saltenv (%s)r:z�Section for saltenv '%s' in the '%s' saltenv's top file will be ignored, as the top_file_merging_strategy is set to 'same' and the saltenvs do not matchr�)r	r
ryr�r�rwr�r�r.r�r)r�rvr�r�ryrgZ
fallback_topsrzr/r�r�r>r>r?�_merge_tops_same�sl
����������zBaseHighState._merge_tops_samecsdd�}tt�}|��D]r}|D]m}|��D]f\}}|dkrqzR|D]M}|||vr5||||||<q"||||�\}	}
||||�\}}g�|pN|	}
|
durX��|
���|
����fdd�|D���|||<q"Wqty|td��wqq|S)z>
        Merge the top files into a single dictionary
        cSs<d}g}|D]}t|t�r|}t|t�r|�|�q||fSr�)rnror�rx)r��
match_typerrtr>r>r?�	_read_tgt?s


�z6BaseHighState._merge_tops_merge_all.<locals>._read_tgtr:Ncrkr>r>r��Zmergedr>r?r�[r{z7BaseHighState._merge_tops_merge_all.<locals>.<listcomp>r�)r	r
r�ryrxr�r�r)r�rvr�r�ryrzr/r�r�Zm_type1Z	m_states1Zm_type2Z	m_states2r�r>r�r?�_merge_tops_merge_all:s<


�
����z#BaseHighState._merge_tops_merge_allc
Cs�g}t|t�s|�d�|S|��D]j\}}|dkrqt|t�s(|�d�|��|dkr1|�d�t|t�s>|�d�|��|��D]9}t|t�sO|�d�qB|D])}t|t�rk|��D]
}|si|�d�||��q\qQt|t�rz|sz|�d	�|��qQqBq|S)
z:
        Verify the contents of the top file data
        z!Top data was not formed as a dictr:z4Environment {} in top file is not formed as a stringr�z#Empty saltenv statement in top filez?The top file matches for saltenv {} are not formatted as a dictz;Malformed topfile (state declarations not formed as a list)z<Improperly formatted top file matcher in saltenv {}: {} filez*Environment {} contains an empty sls index)rnrorxryr�rBr�rz)r�rvr�r/rAZslsmodsZslsmodr
r>r>r?�verify_topscs^


��

�
�
���
�����zBaseHighState.verify_topsc
CsRz|j|d�}Wnty#}zt�d|j�iWYd}~Sd}~ww|�|�S)zA
        Returns the high data derived from the top file
        rrzUnable to render top file: %sN)r|rr�r�r�)r�rrvr�r>r>r?�get_top�s��
zBaseHighState.get_topcs�tt��|��D],\�}�jdr��jdkrq|��D]\}}����fdd���||�jd�qq���}|D](����g�}�j�d�rP|��|}n|�|�}��fdd�|D���<q;�S)z�
        Search through the top high data for matches and return the states
        that this minion needs to execute.

        Returns:
        {'saltenv': ['state1', 'state2', ...]}
        r/c	s�t|t�r|g}�jd|||�rc��vrg��<|D]I}d|vr5|�d�}|��D]
\}}�|||�q*t|t�rB���|�qt|t�rb|��\}}|�jvrSq|�vr[g�|<�|�|�qdSdS)Nzconfirm_top.confirm_topZ	subfilter)	rnr��matchersr�ryrxro�popitemr])	Z_match�_dataZ_optsrtZ_tmpdata�matchr��env_key�inc_sls)�_filter_matchesrAr/r�r>r?r��s,




��z2BaseHighState.top_matches.<locals>._filter_matchesr`Zmaster_tops_firstcrkr>r>r�)r�r>r?r��r{z-BaseHighState.top_matches.<locals>.<listcomp>)r	r
ryr��_master_topsrw)r�r�r�r�r�Zext_matchesZtop_file_matches�secondr>)r�r�rAr/r�r?�top_matches�s&
�zBaseHighState.top_matchescCs
|j��S)z�
        Get results from the master_tops system. Override this function if the
        execution of the master_tops needs customization.
        )rWZmaster_topsr>r>r>r?r��s
zBaseHighState._master_topscCsb|jdsdS|jjdt|�dd�}|dr*tj�|j�|jd<|j��|jjd<|j��dS)zi
        If autoload_dynamic_modules is True then automatically load the
        dynamic modules
        Zautoload_dynamic_modulesNzsaltutil.sync_allF)Zrefreshr�r$)	r�rr=rzrGr!r�r/r�)r�rAZsyncdr>r>r?�load_dynamic�s
zBaseHighState.load_dynamicFcsg}|s�j�||�}|�dd�}	n|}	tj�|	�s#|�d|�d��d}
|	s1|�d�||��n�zt|	�j	j
�j	jd�j	jd�j	jd	||||d
�	}
WnTtyu}zd|�d|�d
|��}t
�|�|�|�WYd}~n2d}~wty�}z"d|�d|��}t
j|tjd�|�|�dt�����WYd}~nd}~wwz|�|�d|���Wn	ty�Ynw|
�r|t|
t�s�|�d|�d��|
|fSg}
d|
vr�t|
dt�s�d�|�}|�|�n|
�d�}
��|
|||���|
|||���|
|||�|
D�]W�d}t�t��r���\}�n|}|�jv�r9d�jv�r9d�|�||�}t
�|�|�|��q�� d��r�t!�"d��}|�rO|�#�\}}
nd��||�}t
�|�|�|��qt$|�}|�%d�}|�dd��&d��rz|�d�|t$|�k�r�d ��||�}t
�|�|�|��qd�'|d|�|
g��||k�r�|du�r�g}||v�s�t(�)�j|���r�|g}n
g}n
��fd!d"�|D�}t$|�d#k�s�||v�r*t(�)�j|���p�g}|D]?}t$|�d#k�r�|d$n|}|�d|��}||v�r&�j*|||||d%�\}}|�r��+|
||�|
�,|�|�r&|�-|��q�qd}|�sAd&�|�||k�r=d'�'|�n|�}nt$|�d#k�rRd(�|�d'�'|��}t
�|�|�|��qz
��.|
�W|
|fSt/�y{t
�d)|�Y|
|fSwi}
|
|fS)*zL
        Render a state file and retrieve all of the include states
        rqFzSpecified SLS z% on local filesystem cannot be found.NzeSpecified SLS {} in saltenv {} is not available on the salt master or through a configured fileserverr�r�r�)Zrendered_slsrzRendering SLS 'r�z
' failed: zRendering SLS z failed, render error: r�r��SLS r9r:z5Include Declaration in SLS {} is not formed as a listZ_xenvr3zDNonexistent saltenv '{}' found in include of '{}' within SLS '{}:{}'rfz^(\.+)(.*)$z:Badly formatted include {} found in include in SLS '{}:{}'�sourcer�z	/init.sls�initzTAttempted relative include of '{}' within SLS '{}:{}' goes beyond top level package cs"g|]
}t��j|��r|�qSr>)r�rtr])r�Zaenv�r�r�r>r?r�os��z.BaseHighState.render_state.<locals>.<listcomp>rlrrrz\Unknown include: Specified SLS {}: {} is not available on the salt master in saltenv(s): {} r�zjAmbiguous include: Specified SLS {}: {} is available on the salt master in multiple available saltenvs: {}z/Could not render SLS %s. Syntax error detected.)0rWrurwr^r_rrxrBrrr�r�rr��criticalrFr�r�r�r�rqr�rnrorzr��_handle_extend�_handle_exclude�_handle_state_declsr�r]r�r��rer��groupsrpr;r�r`r�rt�render_state�merge_included_statesrr��_handle_iorderr�)r�rvr/�modsrAr�rr�Z
state_datardrrr�r:r�Zxenv_keyr�r�ZlevelsZlevel_countZp_compsZ
resolved_envsZsls_targetsZ
sls_target�r_envZmod_tgtZnstater>r�r?r��sF
��



�
��$���
���

��


�



��




����

�

��

�����
��zBaseHighState.render_statecCs�|jdrp|D]h}||D]a}t|t�sq
t||t�sq
t|||t�s'q
d}|�d�r/q
|||D]}t|t�rNt|�dkrNtt|�	���dkrNd}q5|snt|||t�s[q
|||�
d|ji�|jd7_q
q|S)z:
        Take a state and apply the iorder system
        rdFr�rr'Trl)r�rnr�rorzr�rprgrhr	rxr[)r�rr:Zs_decr�r~r>r>r?r��s4



���zBaseHighState._handle_iorderc	Csz|D]�}t||t�sF|dkrq|dkrqt||t�r9d||vr9||�d�}d|d||d|dgi||<q|�d|�d	|�d
��qt�}t||�D]R}|�d�rWqOt|||t�saqOd|vr�|�d�}|d|vrz|�d�||��qO||�	|�|||d<|||d�|d�|�
|d�qO|�
|�qOd||vr�|||d<d||vr�|||d<qd
S)z=
        Add sls and saltenv components to the state
        r�r�rfr4r3rrlzID z in SLS z is not a dictionaryr�zIID '{}' in SLS '{}' contains multiple state declarations of the same typeN)rnror�r;rxrmrzr�rBr�rq)	r�rrvr/r�r:r�r�r�r>r>r?r��sR�

���z!BaseHighState._handle_state_declsc	Cs d|vr�|�d�}t|t�s|�d|�d��dS|D]g}t||t�s.|�d�||��qd||vr:|||d<d||vrF|||d<t||�D]5}|�d�rTqLt|||t�s^qLd	|vr�|�d	�}||�|�|||d
<|||d
�|d�qLq|�dg��|�dSdS)
z`
        Take the extend dec out of state and apply to the highstate global
        dec
        r�zExtension value in SLS 'z' is not a dictionaryNz3Extension name '{}' in SLS '{}' is not a dictionaryr4r3r�rfrrlr�)	r�rnrorxrBrzr�r;r�)	r�rrvr/r�r�r:r�r�r>r>r?r�s<

��

��	�zBaseHighState._handle_extendcCsJd|vr#|�d�}t|t�sd�|�}|�|�|�dg��|�dSdS)zh
        Take the exclude dec out of the state and apply it to the highstate
        global dec
        rz5Exclude Declaration in SLS {} is not formed as a listr�N)r�rnrzrBrxr�r�)r�rrvr/r�rr�r>r>r?r�0s

�
�zBaseHighState._handle_excludec
Cs4|j}g}t�}g}|��D]�\}}|D]{}	||jvr%t�|j||	�}nd|jvr4t�|jd|	�}n|�d�|��|sA|	g}|D]K}
|�d|
��}||vrQqC|j|
||||d�\}}
|rf|�	|||
�t
|
dd��D]\}}d|vr�d|	�d�}||vr�d	�|	|�|
|<qn|�|
�qCqq|�|�||fS)
zp
        Gather the state files and render them into a single unified salt
        high data structure.
        r3z7No matching salt environment for environment '{}' foundr�rrNzis not availabler�z in saltenvz*No matching sls found for '{}' in env '{}')
r^rmryr]r�rtrxrBr�r�r�r��clean_duplicate_extends)r�rAr�	highstateZ
all_errorsr�Z
statefilesr/rr{rvr�rr�r'r�Zthis_slsr>r>r?�render_highstate>sN

�

������
&zBaseHighState.render_highstatecCsZd|vr+g}dd�|dD�D]}|D]}||vr|�|�qqdd�|D�|d<dSdS)Nr�css�|]}|��VqdSr�)ry)r�r�r>r>r?r�tr�z8BaseHighState.clean_duplicate_extends.<locals>.<genexpr>cSsg|]
}|d|di�qSr4r>)r��tr>r>r?r�xr�z9BaseHighState.clean_duplicate_extends.<locals>.<listcomp>)rx)r�r�Zhighextryrtr>r>r?r�qs
���z%BaseHighState.clean_duplicate_extendscCs�d|vr|�dg��|�d��d|vr |�dg��|�d��|D]*}||vrL||||krL|�d�|||d||d||d||d��q"z|�|�WdStyg|�d|���YdSw)Nr�r�z�Detected conflicting IDs, SLS IDs need to be globally unique.
    The conflicting ID is '{}' and is found in SLS '{}:{}' and SLS '{}:{}'r3r4z*Error when rendering state with contents: )r�r�r�rxrBrr�)r�r�rr�r=r>r>r?r�zs.



����z#BaseHighState.merge_included_statescCs |rdSd|jjdvrdSdS)z�
        Check the pillar for errors, refuse to run the state if there are
        errors in the pillar and return the pillar errors
        T�_errorsr$F)rr�)r��forcer>r>r?�
_check_pillar�s
zBaseHighState._check_pillarcCsj|s|Si}t|t�s|�d�}|D] }||D]}||vr1||vr&||ng||<||�|�qq|S)zx
        Reads over the matches and returns a matches dict with just the ones
        that are in the whitelist
        �,)rnrzr;rx)r�rA�	whitelistZret_matchesr[rvr>r>r?�matches_whitelist�s

��zBaseHighState.matches_whitelistr�c
Cs�d}|dddidd�i}tj�|jd|�d��}	|rFtj�|	�rFtjj�|	d	��}
tj	�
|
�}|j�||�Wd
�S1sAwYg}z|�
�}
Wn:tyu}zd||d<||dt|j�7<|WYd
}~Sd
}~wty�t��}|�|�|YSw||�|
�7}|�|
�}|s�d
}|||d<|S|�||�}|�|�|�|�s�|dg7}||jjdd7}n'|�|�\}}|r�t|t�r�|�d�}d|vr�|d�|�n||d<||7}|r�|S|s�|Stjj�d��]zDtjj� ��r|jj!ddd|	gddd�tjj�|	d��}
z	tj	�"||
�Wn
t#�y*YnwWd
�n	1�s6wYWnt$�yLt%�d|	�YnwWd
�n	1�sXwY|j�||�S)zP
        Run the sequence to execute the salt highstate for this minion
        zno_|-states_|-states_|-NoneFzNo states found for this minionz	No Statesr)r�r�r:r�r�rcz.cache.pr
NzUnable to render top file: r�zQNo Top file or master_tops data matches found. Please see master log for details.z4Pillar failed to render with the following messages:r$r�r�r��?zcmd.runZattribz-R�quiet)rqZoutput_loglevelzw+bz2Unable to write to "state.highstate" cache file %s)&r^r_r`r�rrGrHrIr��payloadrHrr8r�rr�r�rFr�r�rxr�r�r�r�r�r�rnr;r�r�r�Z
is_windowsr=�dumpr�r�r�)r�r�cacheZ
cache_namer�r�r�Ztag_namer�Zcfnr�rrr�r�r�rAr�r�r>r>r?�call_highstate�s�
��	 ��
�
�





��������zBaseHighState.call_highstatecCsNg}|j|d�}||�|�7}|�|�}|j||d�\}}||7}|r%|S|S)z9
        Return just the highstate or the errors
        rr)r�r�r�r�)r�rr�r�rArrr�r>r>r?r�
s
zBaseHighState.compile_highstatec	Cs�|j|d�}|�|�}|j||d�\}}|j�|�\}}||7}||j�|�7}|j�|�\}}||7}|j�|�}|r>|S|j�|�}|S)z�
        Compile the highstate but don't run it, return the low chunks to
        see exactly what the highstate will execute
        rr)	r�r�r�rr�r�r�rr)	r�rr�rArrr�r7r�rr>r>r?�compile_low_chunkss
z BaseHighState.compile_low_chunksc
Cs�g}|��}||�|�7}|r|S|�|�}i}|j��D]F\}}ggdddd�}|�|g�}|D]-}	|dd7<|	|vrO|dd7<|d�|	�q1|dd7<|d�|	�q1|||<q|S)	z^
        Return all used and unused states for the minion based on the top match data
        r)�used�unused�	count_all�
count_used�count_unusedr�rlr�r�r�r�)r�r�r�r]ryrwrx)
r�r�r�rAZstate_usager/rZ	env_usageZenv_matchesrr>r>r?�compile_state_usage5s0
�
z!BaseHighState.compile_state_usagecCst�r�)�NotImplementedErrorr>r>r>r?rA[rEzBaseHighState.destroycCrCr�r>r>r>r>r?rD^rEzBaseHighState.__enter__cGrFr�rGrHr>r>r?rIarJzBaseHighState.__exit__r�r�rL)NNr�FNN)#r�r�r�r�r�r\rZrSr|r�rr�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�rArDrIr>r>r>r?rY�sP4
{:H)
34

K#9!
3	

�
Z
&rYc@steZdZdZgZ									ddd�Zdd�Zed	d
��Zedd��Z	ed
d��Z
dd�Zdd�Zdd�Z
dS)�	HighStatez�
    Generate and execute the salt "High State". The High State is the
    compound state derived from a group of template files stored on the
    salt master or in the local cache.
    NFrcCs�||_|
r|
|_d|_ntj�|j�|_d|_t�||�t|j||||||||	|jd�
|_	tj
�|j�|_||_i|_
g|_dS)NTF)rrr r!r"r()r�rW�preserve_clientrGr%r*rYr�rrr!r�rZ_pydsl_all_declsZ_pydsl_render_stack)r�r�rrrrrr r!r"r(r>r>r?r�os.
�
zHighState.__init__cCs|j�|�dSr�)�stackrxr>r>r>r?�push_active��zHighState.push_activecCs
g|_dSr�)r��r�r>r>r?�clear_active�s
zHighState.clear_activecCs|j��dSr�)r�r�r�r>r>r?�
pop_active�szHighState.pop_activecCs"z|jdWStyYdSw)N���)r��
IndexErrorr�r>r>r?�
get_active�s
�zHighState.get_activecCr?r�)r�rWrAr>r>r>r?rA�rBzHighState.destroycCrCr�r>r>r>r>r?rD�rEzHighState.__enter__cGrFr�rGrHr>r>r?rI�rJzHighState.__exit__)	NNNNNFrNN)r�r�r�r�r�r�r�rMr�r�r�rArDrIr>r>r>r?r�es.
�*


r�c@s"eZdZdZdd�Zddd�ZdS)�MasterStatez9
    Create a State object for master side compiling
    cCst�||�dSr�)rr�)r�r�Zminionr>r>r?r��r�zMasterState.__init__NcCs�t�d�tj�|j|jd�|_tj�|j�|_tj�	|j�|_	tj�
|j|j|j|j	�|_
tjj|j|j|j
|jd�|_
dS)r�r�r�)rrN)r�r�rGrWZFunctionWrapperr�r=r!rHr~rr�r1r�)r�r�rr>r>r?r2�s
��zMasterState.load_modulesrK)r�r�r�r�r�r2r>r>r>r?r��sr�c@seZdZdZddd�ZdS)r�z7
    Execute highstate compilation from the master
    NcCsVt�|�}d|d<|d|d<|d|d<|d|d<||d<||d<t�||�dS)	Nr�r(Zmaster_rootsrar�r_r�r�)rrr�r�)r�r�r�r�r=r/r�r>r>r?r��s
zMasterHighState.__init__r�)r�r�r�r�r�r>r>r>r?r��sr�c@s0eZdZdZdd�Zdd�Zdd�Zdd	�Zd
S)�RemoteHighStatez3
    Manage gathering the data from the master
    cCs.||_||_tjjj�|jd�|_d|_dS)Nr F)r�r�rG�channelrWZ
ReqChannel�factory�_closing)r�r�r�r>r>r?r��s
zRemoteHighState.__init__cCs<|j|jdd�}z
|jj|ddd�WStyiYSw)z7
        Return the state data from the master
        Z
_master_state)r�r�rlr�i@)rr\)r�r�r��sendr)r�rHr>r>r?�compile_master�s�zRemoteHighState.compile_mastercCs|jrdSd|_|j��dSrR)r�r�r@r>r>r>r?rAszRemoteHighState.destroycCrFr�rGr>r>r>r?�__del__rJzRemoteHighState.__del__N)r�r�r�r�r�r�rAr�r>r>r>r?r��s
r�rL)Xr�rrPr�r�r�r�r^r�r�r�r�rSr�Zsalt.channel.clientrGZsalt.fileclientZsalt.loaderZsalt.minionZsalt.pillarZ
salt.syspathsrjZsalt.utils.argsZsalt.utils.cryptZsalt.utils.dataZsalt.utils.decorators.stateZsalt.utils.dictupdateZsalt.utils.eventZsalt.utils.filesZsalt.utils.hashutilsZsalt.utils.immutabletypesrHr�Zsalt.utils.jidZsalt.utils.msgpackZsalt.utils.platformZsalt.utils.processZsalt.utils.urlZsalt.utils.yamlloaderrGZsalt.exceptionsrrrZsalt.serializers.msgpackrrrr�Z
salt.templaterrZsalt.utils.odictr	r
�	getLoggerr�r�rOrQrRZSTATE_RUNTIME_KEYWORDSrPr�r@rErJrMr]rerkrurr�r�r�r�r�rFr�r�rrNrYr�r�r�r�r>r>r>r?�<module>s�

���,��	


"-yQ+%V