HEX

File: //opt/saltstack/salt/lib/python3.10/site-packages/salt/__pycache__/minion.cpython-310.pyc
o

�N�g�}�@s�dZddlZddlZddlZddlZddlZddlZddlZddlZddl	Z	ddl
Z
ddlZddlZddl
Z
ddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Zddl!Zddl"Zddl#Zddl$Zddl%Zddl&Zddl'Zddl(Zddl)Zddl*Zddl+Zddl,Zddl-Zddl.Zddl/Zddl0Zddl1Zddl2Zddl3Zddl4Zddl5Zddl6Zddl7m8Z8ddl9m:Z:ddl;m<Z<ddl=m>Z>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFddlGmHZHddlImJZJdd	lKmLZLdd
l*mMZMddl0mNZNddlOmPZPdd
l2mQZQmRZRmSZSddl6mTZTmUZUzddlVZVdZWWneX�yLdZWYnwzddlYZYdZZWneX�yadZZYnwzddl[ZdZ\WneX�yvdZ\Ynwe�]e^�Z_d5dd�Z`dd�Zadd�Zbd6dd�Zcdd�Zdd7dd�Zedd�ZfGdd �d �ZgGd!d"�d"eg�ZhGd#d$�d$�ZiGd%d&�d&eg�ZjGd'd(�d(eg�ZkGd)d*�d*ek�ZlGd+d,�d,eg�ZmGd-d.�d.ej�Znd/d0�ZoGd1d2�d2ek�ZpGd3d4�d4eh�ZqdS)8z
Routines to set up a minion
�N)�	ipaddress)�DEFAULT_MINION_OPTS)�DEFAULT_TARGET_DELIM)	�CommandExecutionError�CommandNotFoundError�SaltClientError�SaltDaemonNotRunning�
SaltException�SaltInvocationError�SaltMasterUnresolvableError�SaltReqTimeoutError�SaltSystemExit)�SLS_ENCODING)�RequestContext)�enable_sigusr1_handler)�tagify)�parse_host_port)�OrderedDict)�ProcessManager�SignalHandlingProcess�default_signals)�ZMQ_VERSION_INFO�zmqTFcCs`i}d}|�dd�dkr|�dd�sd}ddl}|dur�z|d	d
kr%t�|jj�|d	t|d�d|d�|d
<Wn�ty�|�dd�}|dr�	|durZ|dkrVt�|d8}t	�
d|d	|d�t�|d�z|jj�|d	t|d�d|d�|d
<Wn
ty�YnwqLn|r�d|d
<n�Yn6ty�d}|�d	|�}|d
kr�|}|�d�dkr�d�
|�}nd�
|�}t	�
|�td|d��wd|d
<d
|vr�d
|vr�|d
|d
kr�t	�d|d
|d
�|d�rNt	�d|d�|jj��}	t	�d�t	�|	�|d|	v�rE|	|dd�r<|d�s#|	|ddn|	|dd }
|
dd!|d"<t	�d#|d"�n3t	�d$|d�n*t	�d%|d�n!|d&�ro|jj�|d&t|d'�d|d�|d"<t	�d#|d"�|d'�r�t|d'�|d'<t	�d(|d'�|d)�r�t|d)�|d)<t	�d*|d)�d+j
|d
|dd,�|d-<t	�d.|d-�|S)/z7
    Resolves the master_ip and master_uri options
    T�file_client�remote�local�use_master_when_localFrN�master��master_portZipv6Z	master_ip�retry_dns_count�	retry_dns�zIMaster hostname: '%s' not found or not responsive. Retrying in %s secondsz	127.0.0.1zunknown address�__role�syndiczyMaster address: '{}' could not be resolved. Invalid or unresolveable address. Set 'syndic_master' value in minion config.�rMaster address: '{}' could not be resolved. Invalid or unresolveable address. Set 'master' value in minion config.�*��code�msgz'Master ip address changed from %s to %sZsource_interface_namez$Custom source interface required: %sz6The following interfaces are available on this Minion:ZupZinetZinet6�addressZ	source_ipzUsing %s as source IP addresszPThe interface %s is down so it cannot be used as source to connect to the Masterz&%s is not a valid interface. Ignoring.�source_addressZsource_ret_portz*Using %d as source port for the ret serverZsource_publish_portz*Using %d as source port for the master pubztcp://{ip}:{port})�ip�port�
master_urizMaster URI: %s)�get�salt.utils.networkr
�utils�networkZ	dns_check�intrr�log�error�time�sleep�format�warning�trace�
interfaces�debug)�opts�fallback�retZ	check_dns�saltr Zunknown_strr�errr;�addrs�rC�?/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py�resolve_dnsts�������
����
��


����
�

�
�rEc
Cszi}|ddkrt�|d�|d<|Sz
t|d�\}}Wnty-}zt|��d}~wwd|i}|r;|�d|i�|S)a
    parse host:port values from opts['master'] and return valid:
        master: ip address or hostname as a string
        master_port: (optional) master returner port as integer

    e.g.:
      - master: 'localhost:1234' -> {'master': 'localhost', 'master_port': 1234}
      - master: '127.0.0.1:1234' -> {'master': '127.0.0.1', 'master_port' :1234}
      - master: '[::1]:1234' -> {'master': '::1', 'master_port': 1234}
      - master: 'fe80::a00:27ff:fedc:ba98' -> {'master': 'fe80::a00:27ff:fedc:ba98'}
    Zmaster_uri_formatZip_onlyrNr)r�
ip_addressr�
ValueErrorr�update)r=r?�hostr-�excrCrCrD�prep_ip_port�s
���rKcKs�tj�|d�}|�dd�}|duri}nd|i}tj�|�s'tj|fi|��t�|�}|rGt�|j�}||dkrGt�	||j|A|dB�t
td�rr|�dd�}|�dd�}|j|ksb|j|krrdd	�||fD�rrt�
|||�|S)
a�
    Given the cache directory, return the directory that process data is
    stored in, creating it if it doesn't exist.
    The following optional Keyword Arguments are handled:

    mode: which is anything os.makedir would accept as mode.

    uid: the uid to set, if not set, or it is None or -1 no changes are
         made. Same applies if the directory is already owned by this
         uid. Must be int. Works only on unix/unix like systems.

    gid: the gid to set, if not set, or it is None or -1 no changes are
         made. Same applies if the directory is already owned by this
         gid. Must be int. Works only on unix/unix like systems.
    �proc�modeN�chown�uid����gidcSsg|]}|dkr|�qS)rPrC)�.0�irCrCrD�
<listcomp>9sz get_proc_dir.<locals>.<listcomp>)�os�path�join�pop�isdir�makedirs�stat�S_IMODE�st_mode�chmod�hasattr�st_uid�st_gidrN)�cachedir�kwargs�fn_rMZd_statZ	mode_partrOrQrCrCrD�get_proc_dirs(

�recCsNtjj�|�}g}i}g}|D]p}t|t�rD|�dd�durD|��D] \}	}
|	dkr+q"|js3|	|jvr8|
||	<q"|�	|	�d|
���q"qtjjj
|gdd�d}|rz|js`tt|�
���|jvrf|�|�q|��D]\}	}
|�	|	�d|
���qjq|�	|�q|r�|s�tjj�|�|jr�t|t�r�|��D]\}	}
|
|d|	��<q�||fS)z{
    Detect the args and kwargs that need to be passed to a function call, and
    check them against what was passed.
    Z	__kwarg__FT�=)Z	conditionr"Z__pub_)r@r1�argsZget_function_argspec�
isinstance�dictr/�items�keywords�appendZparse_input�next�iter�keysrH�invalid_kwargs)�funcrg�dataZignore_invalidZargspec�_args�_kwargsrp�arg�key�valZstring_kwargrCrCrD�load_args_and_kwargsBs:
��rxcCs�d|vrm|d}|�d�\}}z%tj�|||�}|st�||�|d<t|dtftf�s/t�d|d<Wn0tyKt	�
d|�t�tj
jj�Yntydt	�
d|d|�t�tj
jj�Ynwt	�d|�dSdS)	zd
    Evaluate master function if master type is 'func'
    and save it result in opts['master']
    Z__master_func_evaluatedr�.TzFailed to load module %sz#%s returned from %s is not a stringz Evaluated master from module: %sN)�splitr@�loaderZraw_mod�KeyErrorrh�str�list�	TypeErrorr4r5�sys�exit�defaults�	exitcodes�
EX_GENERIC�info)r=Zmod_fun�mod�funZ
master_modrCrCrD�eval_master_func{s(��r�cCs>ddddd�}|dkr|dur|�|��d|��S|�|d�S)	z[
    Centralized master event function which will return event type based on event_map
    Z__master_connectedZ__master_disconnectedZ__master_failbackZ__master_alive)�	connected�disconnected�failback�aliver�N�_�r/)�typerZ	event_maprCrCrD�master_event�s�r�cCsdtjvrdSdS)z:
    Return the proper service name based on platform
    ZbsdZsalt_minionzsalt-minion)r��platformrCrCrCrD�service_name�sr�c@sXeZdZdd�Zddd�Zedd��Zd	d
�Zej	j
jjdd
d��Z
dd�Zdd�ZdS)�
MinionBasecCs||_|�dd�|_dS)N�beacons_leaderT)r=r/r���selfr=rCrCrD�__init__�szMinionBase.__init__FNcCs4|duri}|r&tjj|j|jd|jd|jd|j�d�d���|jd<tjj|j|d�|_tjj|j|j|d	�|_	tj�
|j�|_
tjj|j|j	|d
�|_tjj|j|j	|jd�|_i|_
tjj|j|j	|j|j
|d�|_tjj|j|j	|d
�|_tj�|j�|_|j|j	d
<tjj|j|j	|j|d�|_dS)��
        Tell the minion to reload the execution modules

        CLI Example:

        .. code-block:: bash

            salt '*' sys.reload_modules
        N�grains�id�saltenv�	pillarenv�r��pillar)�context)r1r�)�	functionsr�)r��	returners)r�r1�serializersr��sys.reload_modules�r��proxyr�)r@r��
get_pillarr=r/�compile_pillarr{r1�minion_modsr�r�r�r��function_errors�states�render�rend�matchers�gen_modules�	executors)r��initial_loadr�rCrCrDr��sN

��
�
���
��zMinionBase.gen_modulesc
Cs�z(t|d�r|j��nt�d�WdS|jj|kr&|jj}t�d�W|SW|StyA}z
t�d|�WYd}~|Sd}~ww)N�schedulezAMinion scheduler not initialized. Scheduled jobs will not be run.z3Overriding loop_interval because of scheduled jobs.z&Exception %s occurred in scheduled job)r_r��evalr4r5�
loop_intervalr<�	Exception)�minionr�rJrCrCrD�process_schedule�s$
�����zMinionBase.process_schedulecCs<d|vr|dd|jddd�}|r|j�||jd�SgS)z|
        Evaluate all of the configured beacons, grab the config again in case
        the pillar or grains changed
        zconfig.merge�beaconsT)Z	omit_optsr�)r=r��process)r�r�Zb_confrCrCrD�process_beacons�s�
�zMinionBase.process_beacons�<Tc#s���ddkrt�d�d|_tjjj�d��|���ddk�rj�ddk�rj�dd	kr5t	���n5�dd
v�rZt
�dt�r�t�d�d��dd
kr�t
�d�}|dkr��ddd�}t��d�|}z+�d��d|�d<�fdd��dD��dd�<��d<t�d�dd�Wntttfy�t�d�Ynwt�d��dr�t�d��d�d<d�d<�drшddkrшd�d<nrt
�dt�r�d�vr�dg�d<n_�ddkr�t�d�d�nP|�r2|�r�d�d<nCt�d �d��d�d!v�r+�d}	�fd"d��d!D��d<�d�|	�n�d�d<nt�d#t�d��t�tjjj��d$�rY�dd%k�rYt�d&�d�d$<nt�d'�d�t�tjjj�||d(�}
t|d)d��r{|j|
d)<��d*d�}d}t
�dt��rd}
d}g�d+<t � �d��d!<�d�rʈdd%k�rÈd�rÈd!dd�}t!�"|�|�d!dd�<nt!�"�d!��d!D];}|�d<��#t$����dd%k�r�z�d+�t%�d�d,�W�q�t&�y�Y�q�w�d+�t%��d,��qΈd+�sd-}t�|�t&|��d}	|dk�r-tjjj�'�d/�V|d7}|dk�r>t�(d0||�nt�(d1|��d!D]�}|�d<��#t$����dd%k�rrz
��#t%�d��Wnt&�yqY�qHw��#t%���d�v�r�t � �d!��d<�|_)tj*j+j,j-�fi|
��}z
|�.�Vd.}
Wn5t&�y�}z(|}|j/�0d2��r�t�d3�d|�nt�d4�d�|�1�d}WYd}~�qHd}~ww|
�s�||k�r�d|_t � |j)d!�|j)d<t�d5�|�r�|�1�|�n|j2�3d6�|_4d.|_tjjj��d|f���q�d�rt�d7�d}	|dk�r0tjjj�'�d/�V|d7}|dk�rAt�(d0||�nt�(d1|���#t$�����#t%���zi|j)d8d9k�r�d.|j)d:<tj5j6D].}|d;k�rst7�ss�qg||j)d8<tj*j+j,j-|j)fi|
��}|�.�V|j2j8�s��qg|j)d:=ntj*j+j,j-|j)fi|
��}|�.�V|j2�3d6�|_4d.|_tjjj��d|f��t&�y�|�r�|�1�||k�r�d|_�Ynw�q )<a
        Evaluates and returns a tuple of the current master address and the pub_channel.

        In standard mode, just creates a pub_channel with the given master address.

        With master_type=func evaluates the current master address from the given
        module and then creates a pub_channel.

        With master_type=failover takes the list of masters and loops through them.
        The first one that allows the minion to create a pub_channel is then
        returned. If this function is called outside the minions initialization
        phase (for example from the minions main event-loop when a master connection
        loss was detected), 'failed' should be set to True. The current
        (possibly failed) master will then be removed from the list of masters.
        �master_type�disablez-Master is set to disable, skipping connectionF�NNr}r#r$rq��failover�distributedrz*Got list of available master addresses: %sr�r"Nr�rcsg|]
}|�dkr|�qS)rrC)rR�m)�preferred_mastersrCrDrT9s
�z*MinionBase.eval_master.<locals>.<listcomp>z"Distributed to the master at '%s'.z*Failed to distribute to a specific master.z3master_type = distributed needs more than 1 master.Zmaster_shufflez�Use of 'master_shuffle' detected. 'master_shuffle' is deprecated in favor of 'random_master'. Please update your minion config file.Z
random_masterZ
auth_tries�master_failback�master_failback_interval�master_alive_interval�master_listz$Syndic setting master_syndic to '%s'zBMoving possibly failed master %s to the end of the list of mastersZ
local_masterscsg|]
}�d|kr|�qS)rrC�rR�x�r=rCrDrTtszMmaster_type set to 'failover' but 'master' is not of type list but of type %sr!r�z�'master_type' set to 'failover' but 'retry_dns' is not 0. Setting 'retry_dns' to 0 to failover to the next master on DNS errors.z/Invalid keyword '%s' for variable 'master_type')�timeout�safe�io_loopZmaster_triesZmaster_uri_listr.zNo master could be resolvedT�acceptance_wait_timez&Connecting to master. Attempt %s of %sz4Connecting to master. Attempt %s (infinite attempts)zCould not accessz\Failed to initiate connection with Master %s: check ownership/permissions. Error message: %sz;Master %s could not be reached, trying next master (if any)zQNo master could be reached or all masters denied the minion's connection attempt.ssaltzGrandom_master is True but there is only one master specified. Ignoring.�	transportZdetectZdetect_modeZzeromq)9r4r9r�r@�ext�tornado�gen�Return�_discover_mastersr�rhr~r��len�binascii�crc32r|�AttributeErrorrr}rlr5r�r�r�r�r�r��critical�getattrr�r/�copy�random�shufflerHrKrErr7r<r=�channel�clientZAsyncPubChannel�factory�connect�strerror�
startswith�close�authZ	gen_token�tokr�Z
TRANSPORTSrZ
authenticated)r�r=r�r��failedr�Z
master_lenZsecondary_mastersZ
master_idxZ
failed_masterZfactory_kwargs�tries�attempts�conn�last_excrr)�pub_channelrJZtransrC)r=r�rD�eval_master	s��
�
�
��������

�
����




��



��

��
����
���
M�

�


��

�
��

���zMinionBase.eval_masterc
s�|jdtdkr�|jddur�tjj��}i}t|jd�dd��D]@}z%|d7}t�	d|�|�
|���|sFt�
|jd�dd	��nWnWq#tyc}z
t�d
|�WYd}~nd}~ww|r�|j�di��dd
�}|dvr}t�d|�dS|jd�di�}|��D]5\}}|D](�t�fdd�|��D��}	|d
kr�t|	�s�|	t|�kr��d|jd<dSq�q�dSdSdSdS)z�
        Discover master(s) and decide where to connect, if SSDP is around.
        This modifies the configuration on the fly.
        :return:
        rZ	discoveryFr��r"z)Attempting %s time(s) to discover masters�pause�zSSDP discovery failure: %sN�match�any)r��allzPSSDP configuration matcher failure: unknown value "%s". Should be "any" or "all"�mappingcs*g|]\}}��di��|�|kr|�qS)r�r�)rRrv�value�Z
proto_datarCrDrTjs
�z0MinionBase._discover_masters.<locals>.<listcomp>)r=rr@r1ZssdpZSSDPDiscoveryClient�ranger/r4r�rHZdiscoverr6r7r�r5rjr��bool)
r�Zmaster_discovery_client�mastersZattrA�policyr��addrZmappingsZcntrCr�rDr�DsT����
�����
zMinionBase._discover_mastersc	Cs�d}|j�d�rFzt�|jd|jd�}||}t�d||�|WStyEt�d|jd|jd�t�|td�tdYSwt�||j�d��|j�d�S)z�
        Based on the minion configuration, either return a randomized timer or
        just return the value of the return_retry_timer.
        z+Minion return retry timer set to %s secondsZreturn_retry_timer_maxZreturn_retry_timerz%s (randomized)zeInvalid value (return_retry_timer: %s or return_retry_timer_max: %s). Both must be positive integers.)	r=r/r��randintr4r<rGr5r)r�r)Zrandom_retryZ	retry_msgrCrCrD�_return_retry_timerts(���zMinionBase._return_retry_timer�FN)r�TFF)�__name__�
__module__�__qualname__r�r��staticmethodr�r�r@r�r�r��	coroutiner�r�r�rCrCrCrDr��s
3

<0r�cs"eZdZdZd�fdd�	Z�ZS)�SMinionz�
    Create an object that has loaded all of the minion module functions,
    grains, modules, returners etc.  The SMinion allows developers to
    generate all of the salt minion functions and present them with these
    functions for general use.
    Ncs�ddl}|j�|�|d<t��|��j�dd�dks"�j�dd�r?|jjj	j
��}|jjjj
�fdd���|��fd	d
���jd|d��jddkr�j�d
d�r�ddl}tj��jdd�}tj�|�snt�|d�tj�|d�}�jddur��jd}nd}|�jddgii}|jj�|d��}	|jjj||	td�t�|d�Wd�n1s�wYtj�|d�}
|jj�|
d��}	|jjj�jd|	td�t�|
d�Wd�dS1s�wYdSdSdS)Nrr�rrrFc3s$��j�jdd�V\}}|��dS)zb
                Wrap eval master in order to close the returned publish channel.
                T�r�N)r�r=r�)rr��r�rCrDr��s�z%SMinion.__init__.<locals>.eval_mastercs��S�NrCrC)r�rCrD�<lambda>�sz"SMinion.__init__.<locals>.<lambda>T)r�r�Zminion_pillar_cacherbr�i�ztop.slsr��baser��cache�wb)�encodingi�z	cache.sls)�salt.loaderr{r��superr�r=r/r�r��ioloop�IOLoop�currentr�rZrun_syncr�Zsalt.utils.yamlrUrVrWrYrZr1�files�fopenZyamlZ	safe_dumprr^)r�r=r�r@r�ZpdirZptopZpenvZ	cache_top�fp_Z	cache_sls��	__class__)r�r�rDr��sJ�

����"��zSMinion.__init__r)r�r�r�__doc__r��
__classcell__rCrCrrDr�src@s0eZdZdZ						d	dd�Zd
dd�ZdS)�MasterMinionz�
    Create a fully loaded minion function object for generic use on the
    master. What makes this class different is that the pillar is
    omitted, otherwise everything else is loaded cleanly.
    TNcCsFtjj|d||d�|_||_||_||_||_||_|j	dd�dS)NZ	conf_file)�ignore_config_errorsT)r�)
r@ZconfigZmminion_configr=�	whitelist�mk_returners�	mk_states�mk_rend�
mk_matcherr�)r�r=r�r�r��matcherrrrCrCrDr��s

�zMasterMinion.__init__FcCs�tj�|j�|_tjj|j|j|j|d�|_tj�|j�|_|jr+tj�	|j|j�|_	|j
r<tj�|j|j|j|j�|_|jrItj�
|j|j�|_|jrTtj�|j�|_|j|jd<dS)r�)r1rr�r�N)r@r{r1r=r�rr�r�rr�rr�rr�r�rr�r�)r�r�rCrCrDr��s&
��zMasterMinion.gen_modules)TTTTNT�F)r�r�rrr�r�rCrCrCrDr�s	
�rcs�eZdZdZ�fdd�Zdd�Zdd�Zejj	j
jdd	��Z	
	
	
	ddd
�Z
dd�Zddd�Zejj	j
jdd��Zdd�Zedd��Zdd�Zdd�Z�ZS)�
MinionManagerz�
    Create a multi minion interface, this creates as many minions as are
    defined in the master option and binds each minion object to a respective
    master.
    cszt��|�|jd|_|jd|_g|_g|_tjj	j
j��|_
tdd�|_|j
j|jjfiddi��d|_d|_dS)Nr��acceptance_wait_time_maxZMultiMinionProcessManager��name�asynchronousT)r
r�r=�	auth_wait�
max_auth_wait�minions�	jid_queuer@r�r�rrrr�r�process_manager�spawn_callback�run�event_publisher�eventr�rrCrDr�s��
zMinionManager.__init__cC�|��dSr��destroyrrCrCrD�__del__ �zMinionManager.__del__cCsPtjjj|j|jd�|_tjjjd|j|jd�|_|j�d�|j�	|j
�dS)N�r�r�)r=r�r)r@r1r.ZAsyncEventPublisherr=r�r-�	get_event�	subscribe�set_event_handler�handle_eventrrCrCrD�_bind%s�
�zMinionManager._bindc
#sR�z�fdd�|jD�VWdSty(}z
t�d|�WYd}~dSd}~ww)Ncsg|]}|����qSrC)r8)rRr���packagerCrDrT4sz.MinionManager.handle_event.<locals>.<listcomp>zError dispatching event. %s)r(r�r4r5)r�r;rJrCr:rDr81s���zMinionManager.handle_eventNTc	Cst|||||||d�S)�F
        Helper function to return the correct type of object
        �r��loaded_base_namer)�load_grains)�Minion�r�r=r�r�r�r>r)r?rCrCrD�_create_minion_object8�
�z#MinionManager._create_minion_objectcCs|js
t�d�dSdS)zQ
        Check the size of self.minions and raise an error if it's empty
        z7Minion unable to successfully connect to a Salt Master.N)r(r4r5rrCrCrD�_check_minionsOs�zMinionManager._check_minionsr�c
Cs�|jd}|jddvst|jdt�s|g}d}|D]4}t�|j�}||d<d|d<||d<|r3d}|j||dd|jd	�|d�|jd
�}|j�	|j
|�q|j�||j�dS)zH
        Spawn all the coroutines which will sign in to masters
        rr�r�T�multimasterr�FZauth_timeoutzsalt.loader.{})r�r>r)N)
r=rhr~r��deepcopyrBr�r8r)r+�_connect_minion�
call_laterrD)r�r�r�r�r�s_optsr�rCrCrD�_spawn_minionsVs.

��zMinionManager._spawn_minionsc
csv�d}|jd}d}	z9|j�dd�r|jdd�|j�dd�r&|jdd�|j�dd	�d
kr6|j|d�V|jdd�|j�|�WdSty}}z-|�	�d}t
�d
|jd|�t��}||j
krj||j7}tjjj�|�VWYd}~nAd}~wty�|�	�d�|jd�}t
�|�YdSty�}z|�	�d}t
jd|jddd�WYd}~nd}~wwq)zL
        Create a minion, and asynchronously connect it to a master
        rr�FT�beacons_before_connect��before_connect�scheduler_before_connectr�r}r�r)�startzeError while bringing up minion for multi-master. Is master at %s responding? The error message was %srNr%�'Unexpected error while connecting to %s��exc_info)r=r/�
setup_beacons�setup_scheduler�connect_master�tune_inr(rlrr1r4r5r6r'r&r@r�r�r�r7rr8r�r�)r�r��lastr&r�rJrA�erCrCrDrGss`�
�

���
����zMinionManager._connect_minioncCs|��|��|j��dS)a
        Bind to the masters

        This loop will attempt to create connections to masters it hasn't connected
        to yet, but once the initial connection is made it is up to ZMQ to do the
        reconnect (don't know of an API to get the state here in salt)
        N)r9rJr�rOrrCrCrDrV�szMinionManager.tune_incCs|jD]}|jrdSqdS)NTF)r(�restart�r�r�rCrCrDrY�s

�zMinionManager.restartcCsp|jD]}|j��|j�|�|j��|��q|jdur'|j��d|_|jdur6|j��d|_dSdSr)	r(r*Zstop_restartingZsend_signal_to_processesZ
kill_childrenr1r-r�r.)r��signumr�rCrCrD�stop�s








�zMinionManager.stopcCsP|jD]}|��q|jdur|j��d|_|jdur&|j��d|_dSdSr)r(r1r-r�r.rZrCrCrDr1�s






�zMinionManager.destroy�NNNT)r�)r�r�rrr�r2r9r@r�r�r�rr8rBrDrJrGrV�propertyrYr\r1rrCrCrrDr!	s*


�


3
r!cs~eZdZdZ						dj�fdd�	Zdd�Zdkd
d�Zejj	j
jdldd
��Zejj	j
jdd��Z
ejj	j
jdd��Zdd�Z	dmdd�Zdd�Zejj	j
jdd��Zejj	j
jdd��Z		dldd�Zejj	j
j								dndd��Z								dnd d!�Zejj	j
jd"d#��Zd$d%�Zed&d'��Zd(d)�Zed*d+��Zed,d-��Zdod/d0�Zejj	j
jdpd1d2��Zdpd3d4�Z dqd5d6�Z!d7d8�Z"d9d:�Z#ejj	j
jd;d<��Z$drd=d>�Z%d?d@�Z&dAdB�Z'dCdD�Z(ejj	j
jdrdEdF��Z)dGdH�Z*dIdJ�Z+dKdL�Z,dMdN�Z-dOdP�Z.ejj	j
jdQdR��Z/dSdT�Z0dUdV�Z1dldWdX�Z2dldYdZ�Z3dsd\d]�Z4d^d_�Z5dtd`da�Z6dbdc�Z7ddde�Z8dfdg�Z9dhdi�Z:�Z;S)ur@z}
    This class instantiates a minion, runs connections for a minion,
    and loads all of the functions into the minion
    r�TNc	s�t��|�||_||_d|_tjj��|_	||_
d|_d|_d|_
|dur(gn||_i|_d|_|dur?tjjjj��|_n||_trMtdkrMt�d�tjj��s_|r^tj�|�|jd<n"|j� dd�rpt�d�d|jd<|j� dd�r�t�d	�d|jd<t�!d
�|jdr�t"�#d|jd�}t�!d
||jd�t$�%|�t&dd�|_'|jj(|j'j)fiddi��tjj��s�|j�(tj*j+|j|j'�t,�-t,j.�t,j/ur�t,�,t,j.|j0�t,�-t,j1�t,j/ur�t,�,t,j1|j0�dSdS)z*
        Pass in the options dict
        NF)r��z�You have a version of ZMQ less than ZMQ 3.2! There are known connection keep-alive issues with ZMQ < 3.2 which may result in loss of contact with minions. Please upgrade your ZMQ!r�rKzM'beacons_before_connect' is not supported for proxy minions. Setting to FalserNzO'scheduler_before_connect' is not supported for proxy minions. Setting to FalsezCreating minion process managerZrandom_startup_delayrzWMinion sleeping for %s seconds due to configured startup_delay between 0 and %s secondsZMinionProcessManagerr#r%T)2r
r�r�r��_runningr@r1r�ZSubprocessList�subprocess_listr>r�rY�readyr)�periodic_callbacks�req_channelr�r�rrrr�rrr4r9r�Zis_proxyr{r�r=r/r�r�r�r6r7rr*r+r,ZenginesZ
start_engines�signal�	getsignal�SIGINT�SIG_DFL�_handle_signals�SIGTERM)	r�r=r�r�r>r�r)r?Z
sleep_timerrCrDr��sl�
��
�


�
��zMinion.__init__cCs,d|_|j�||�t�d�t�d�dS)NFr"r)r`r*rir6r7r�r�)r�r[ZsigframerCrCrDri<s
zMinion._handle_signalsFcs�d�_t�d��fdd�}�j|d��_�j�|�|r&�j�|�jj�z�j�	�Wn
t
y:���Ynw�j��rX�j�
�}|rX�j��}|d�|d|d��|rc�jduretd	��d
Sd
S)z:
        Block until we are connected to a master
        F�sync_connect_mastercsd�_�j��dS�NT)�_sync_connect_master_successr�r\)�futurerrCrD�on_connect_master_future_doneJszAMinion.sync_connect_master.<locals>.on_connect_master_future_donerrr"r_z$Failed to connect to the salt-masterN)rmr4r<rUZ_connect_master_futureZadd_done_callbackr�rHr\rO�KeyboardInterruptr1�done�	exceptionrR�with_tracebackr)r�r�r�roZfuture_exceptionrRrCrrDrkCs(
�


�zMinion.sync_connect_masterccs��t|d�r&|jr&|j�d�t|jd�r|jj��t|jd�r&|j��t|d�r6|jr6|j��d|_|�|j|j	|j
|�V\}|_tjj
jj|j|jd�|_t�d�|j��V|�|�VdS)zX
        Return a future which will complete when you are connected to a master
        r�Nr�r�rdr4z,Connecting minion's long-running req channel)r_r��on_recvr��
invalidater�rdr�r=r�r�r@r�r��AsyncReqChannelr�r�r4r<r��_post_master_init)r�r�rrCrCrDrUcs&�

�
�
zMinion.connect_masterccs&�|j�|�||�V|j��dSr)ZpayloadsrlZpayload_ack�notify)r��payloadZ
reply_funcrCrCrD�handle_payloads�
zMinion.handle_payloadccsr�|jr.||jd<tjj|j|jd|jd|jd|j�d�d�}|��V|jd<|��|js6|�	�n#|jrY|jdrY|�
�\|_|_|_
|_t|d�rY|j|j_|j|j_t|d�sqtjjj|j|j|jtd	d
�gd�|_|jdr�d
|jvr�|jjdd
|jdddd|j�dd�d�idd�t�d�n|jjddd�|jddk�r|jddk�r|j�r|jjtd	|jdd�d|jdddd|jddd�d�idd�|jd �rd!|jv�r|jd|jd!dk�r|jjtd"d
�d#|jd$dddd|jd!did�idd�d%S|jjtd"d
�dd�d%S|jjtd	|jdd�dd�|jjtd"d
�dd�d%S)&a�
        Function to finish init after connecting to a master

        This is primarily loading modules, pillars, etc. (since they need
        to know which master they connected to)

        If this function is changed, please check ProxyMinion._post_master_init
        to see if those changes need to be propagated.

        Minions and ProxyMinions need significantly different post master setups,
        which is why the differences are not factored out into separate helper
        functions.
        rr�r�r�r�r�r�r�r��r�)�cleanupZmine_enabledzmine.updateZ__mine_intervalZ
mine_intervalTr_Zmine_return_jobF)�function�minutes�jid_include�
maxrunningZrun_on_start�
return_job)�persistzAdded mine.update to schedulerr��tcpr�r�r�r�
status.masterr"�rr��r}�secondsrr�r�rcr�r�r��status.ping_masterr�N)r�r=r@r��get_async_pillarr/r�r1rb�_setup_core�
_load_modulesr�r�r�r�r_r�r1�Scheduler��add_jobr4r��
delete_job)r�r�async_pillarrCrCrDrw�s��

�
�




���
�
���
����

��zMinion._post_master_initcCs.i}|j��D]
\}}|dkrq|||<q|S)zG
        Returns a copy of the opts with key bits stripped out
        �logger)r=rj)r��mod_optsrvrwrCrCrD�_prep_mod_opts�s
zMinion._prep_mod_optscCs�d}|s	|j}d}d}|�dd�dkrHtrHtrHt�d|d�d}t�tj�}t	�
t����
�dd�\}	}
|	|
|d}t�tj||f�n|�dd�dkr^tsWt�d	�ts^t�d
�t|d�rg|j}nd}|duroi}|durtjj||||d�|d
<tjj|||d�|_|�dd�r�t�|�}
tjj|
|j||j||d�}ntjj||j|||d�}tjj||||d�}i}d|vr�|d}|�d�|dur�t�tj|�tjj||||d�}|r�||_||||fS)za
        Return the functions and the returners loaded up from the loader
        module
        TF�modules_max_memoryrPrz1modules_max_memory set, enforcing a maximum of %sNr_z>Unable to enforce modules_max_memory because psutil is missingz@Unable to enforce modules_max_memory because resource is missingr��r�r�r�rE)r1r�r>rxr��r1rxr�r��_errors)r=r/�
HAS_PSUTIL�HAS_RESOURCEr4r<�resourceZ	getrlimitZ	RLIMIT_AS�psutilZProcessrU�getpidZmemory_infoZ	setrlimitr5r_r�r@r{r�r1r�rFr�r>r�rXr�)r��
force_refreshrxr�r=r�Zopt_inr�Z
old_mem_limitZrssZvmsZ	mem_limitr�rIr�r��errorsr�rCrCrDr�s|���

�
�	�
zMinion._load_modulescCs�|jdr$t�d�tj�|jdd�}tj�|tj	j
�|��}||d<tjj
jd|jdd��?}tt���}t�d	|�|j|d
|�d|jd��|d
�|jd|��|d�}|rjt�d|�|dWd�Std��1sqwYdS)N�minion_sign_messages�+Signing event to be published onto the bus.�pki_dir�
minion.pem�sigr�T�r=�listen�Send request to main id=%s�__master_req_channel_payload/�/r�r��__master_req_channel_return/)�tag�wait�Reply from main %sr?zRequest timed out)r=r4r:rUrVrWr@�crypt�sign_messager��msgpack�	serializer1r.r5r}�uuid�uuid4�
fire_event�TimeoutError)r��loadr��minion_privkey_pathr�r.�
request_idr?rCrCrD�_send_req_syncZs2

�����zMinion._send_req_syncc	cs.�|jdr%t�d�tj�|jdd�}tj�|tj	j
�|��}||d<tjj
jd|jdd��]}tt���}t�d	|�|j|d
|�d|jd��|d
�Vt��}t��||kry|jd|��dd�}|rhntjjj�d�Vt��||ks[td��t�d|�tjjj�|d��1s�wYdS)Nr�r�r�r�r�r�Tr�r�r�r�rr�r�)r�Zno_blockg333333�?zDid not recieve return eventr�r?)r=r4r:rUrVrWr@r�r�r�r�r�r1r.r5r}r�r��fire_event_asyncr6r�r�r�r7r�r�)	r�r�r�r�r�r.r�rOr?rCrCrD�_send_req_asyncts:�

��
���zMinion._send_req_asyncccst�|jdr%t�d�tj�|jdd�}tj�|tj	j
�|��}||d<|jj
|||jdd�V}tjjj�|��)z�
        Send a request to the master's request server. To be called from the
        top level process in the main thread only. Worker threads and
        processess should call _send_req_sync or _send_req_async as nessecery.
        r�r�r�r�r��return_retry_tries�r�r�)r=r4r:rUrVrWr@r�r�r�r�r�rd�sendr�r�r�r��r�r�r�r�r�r?rCrCrD�_send_req_async_main�s�

��zMinion._send_req_async_maincs��jdd|�jd�}|r||d<n|r|r||d<||d<n|s,|r,i|d<||d<ndS|rD�fdd��j�d	i���D�}||d	<|S)
Nr�Z
_minion_event)r��cmd�pretagr��eventsrrr�cs$i|]\}}|�jdvr||�qS)�start_event_grainsr�)rR�k�vrrCrD�
<dictcomp>�s
�z/Minion._fire_master_prepare.<locals>.<dictcomp>r�)r=r�r/rj)r�rrr�r�r��include_startup_grainsr�Z
grains_to_addrCrrD�_fire_master_prepare�s(�



�zMinion._fire_master_preparec
cs:�|�|||||�}|durdd�}	|	}|�||�VdS)NcWst�d�dS)N�Efire_master failed: master could not be contacted. Request timed out.T)r4r��r�rCrCrD�handle_timeout�s�z0Minion._fire_master_main.<locals>.handle_timeout)r�r�)
r�rrr�r�r�r��timeout_handlerr�r�r�rCrCrD�_fire_master_main�s�
�zMinion._fire_master_mainc	Csj|�|||||�}z	|�||�WdStjjy#t�d�YdSty4t�dt�	��YdSw)zQ
        Fire an event on the master, or drop message if unable to send.
        r�Fzfire_master failed: %sT)
r�r�r@�
exceptionsrr4r�r��	traceback�
format_exc)	r�rrr�r�r�r�r�r�r�rCrCrD�_fire_master�s
�
���zMinion._fire_masterc
	cs��d|vrt�d|d|d|d�nt�d|d|d�t�d|�t�d|j�|jdurR|d|jvr:dS|j�|d�t|j�|jd	krR|j�d
�t	|dt
�ru|ddkru|��\|_|_
|_|_|j|j_|j
|j_
|j�d�r�t|d
�r�|j}nd}tjj|jd|d�|jd<|j�d�}|d
kr�ttjj�|j��}||kr�t�d|d�tjjj�d�Vttjj�|j��}||ks�|}d}|j�dd�}d�|d�}|�rtjj �!�r�d}tj"j#j$}t%t&j't&j(��t)|j*|||j||j+|fd�}	Wd�n	1�swYnt,j-|j*||j||j+|f|d�}	|�rDt%t&j't&j(��|	�.�Wd�n	1�s>wYn|	�.�|j/�0|	�dS)�b
        Override this method if you wish to handle the decoded data
        differently.
        �userz(User %s Executing command %s with jid %sr��jidz Executing command %s with jid %szCommand details %szStarted JIDs: %sNZminion_jid_queue_hwmrr�Zgrains_refresh_pre_execr�T)r�r�r��process_count_maxzFMaximum number of processes reached while executing jid %s, waiting...�
�multiprocessingzProcessPayload(jid={}))�targetr$rg)r�rgr$)1r4r�r<r:r)rlr�r=rXrhr}r�r�r�r�r�r�r/r_r�r@r{r�r1r�Zrunningr9r�r�r�r7r8r�Zspawning_platformr��	AsyncAuth�	creds_maprrergrjr�_targetr��	threading�ThreadrOra�add)
r�rrr�r�Z
process_count�instancer�Zmultiprocessing_enabledr$r�rCrCrD�_handle_decoded_payload�s���
�


���

����
��zMinion._handle_decoded_payloadcCsBt��}|�|jj���|�|jj���|�|jj���|S)zG
        Return a single context manager for the minion's data
        )�
contextlib�	ExitStack�
enter_contextr�Zcontext_dict�cloner�r�)r�Z	exitstackrCrCrD�ctx]s
z
Minion.ctxc	Cs,|r|tjj_|sI||dd�}||_t|d�s/|j|dd�\}}}}	||_||_||_	|	|_
t|d�sItjjj
|�dd�d�}
t|d	|
d
�|_dd�}tjjj�t�t||d
����0tjjj�|j��||||�Wd�n1swwYWd�dSWd�dS1s�wYdS)NF)r?r�r�)r��proc_dirr��r�rb�rOcSs8t|dt�st|dt�rt�|||�St�|||�S)Nr�)rh�tupler~r@�_thread_multi_return�_thread_return)�minion_instancer=rrrCrCrD�run_func}sz Minion._target.<locals>.run_func)rrr=)r@r�r�r�r�r_r�r�r�r�r�r1r��get_uidr/rer�r�r��
stack_contextZStackContext�	functools�partialrr�)�clsr�r=rrr�r�r�r�r�r�rOr�rCrCrDr�gs:

�

���"�zMinion._targetcCs|d}|jr!|jd�dd�r!|jd�dg�}|dkr!||vr!d}|jd�dd�r=|jd�dg�}|dkr=||vr=d}|rCtd��||jvrV|j|}t|||�\}	}
n|}||}	}
d	|jjd
d<t|t�rn|g}nt|t	�ru|s|td�
|���|�d
d�r�|ddkr�d|d<t�d|�|D]'}|�d�}||j
vr�td|�d���|j
|||||	|
�}
|
dur�|
Sq�dS)z�
        Executes a function within a job given it's name, the args and the executors.
        It also checks if the function is allowed to run if 'blackout mode' is enabled.
        Fr�Zminion_blackoutZminion_blackout_whitelistzsaltutil.refresh_pillarTr�z�Minion in blackout mode. Set 'minion_blackout' to False in pillar or grains to resume operations. Only saltutil.refresh_pillar allowed in blackout mode.r�__context__�retcodezDWrong executors specification: {}. String or non-empty list expectedZ	sudo_userrrPZsudozExecutors list %sz.executez
Executor 'z' is not availableN)r�r=r/r
r�rx�packrhr}r~r8r4r:r�)r��
function_name�
function_argsr�r=rrZminion_blackout_violationrrqrgrcr$�fname�return_datarCrCrD�_execute_job_function�sR�



��

�zMinion._execute_job_functioncs���tj��j|d�}tjj�|j	�d��dt�
�i}|�|�t�
d|d|d�tjj�|d��}|�tj�|��Wd�n1sKwYddi}|d	�|d
}|�d�pmt�dg�pm|�ddg�}	t��fd
d�|	D��}
��jvs�|
du�rSz����||	||��t�tj�r�d}i}�D]8}
t|
t�r�t|t�r�|�|
�n	|s�g}|�|
�t|dd|dt|�gd�}d|
i}��||�|d7}q�||d<n�|d<�jj d�dtj!j"j#�}|tj!j"j#k�rz
t$�fdd�dD��}Wnt%�y
d}Ynw|�stj!j"j&}||d<|tj!j"j#k|d<W�n|t'�yW}z(d��d�}tj(|dd�|�d|��|d<d|d <tj!j"j&|d<WYd}~�nLd}~wt)�y�}z#tj*d!�|t+j,d"�d#|��|d<d|d <tj!j"j&|d<WYd}~�nd}~wt-�y�}z%tj*d$�|t+j,d"�d%��d&|��|d<d|d <tj!j"j&|d<WYd}~n�d}~wt.�y�}z"t�*d$�|�d%��d&|��|d<d|d <tj!j"j&|d<WYd}~n�d}~wt/�y}z+d'�0�|�j�j1�p�d(�}tj2|t+j,d"�||d<d|d <tj!j"j&|d<WYd}~n�d}~wt%�yRd)}tj2|dd"�tjj*j3tj4�5|�||d*�|�dt6�7���|d<d|d <tj!j"j&|d<YnMw�jd+��d,��}|�rm�j�8��|�<||d<n#�j�8��|d<��9d-�d}|�j:v�r�|dd.�0�j:|�7<d|d<tj!j"j&|d<d|d <|d|d<|d	|d	<|d
|d/<d0|v�r�|d0|d0<d1|v�r�|d1|d1<d2|v�r�t|d2t��r�|d2|d2<nt�2d3��j;�r��<|�t|�d�t��r|d4�rd5�|d4|df�|d4<n|d|d4<t�(d6|�|d4�r�t|d4t��r�d7|v�r)|d7|d7<d8|v�r4|d8|d8<|d|d<t=|d4�9d5��D]F}z#|�d9�}|�j>v�rY�j>||�n
�j>�8|�}t�*d:||�W�qCt%�y�}zt�?d;|d|�WYd}~�qCd}~wwdSdSdS)<�s
        This method should be used as a threading target, start the actual
        minion side execution.
        r�z._thread_return�pidz!Starting a new job %s with PID %s�w+bN�successFr�ru�module_executors�direct_callcs0g|]}|�d��jvr�j|�d����qS)z.allow_missing_func)r�)rR�executor)r�r�rCrDrT�s��z)Minion._thread_return.<locals>.<listcomp>Tr�progr��job�returnr"r�r�c3s�|]	}��|d�VqdS)TNr�r�)r�rCrD�	<genexpr>s�
�z(Minion._thread_return.<locals>.<genexpr>��resultr�zCommand required for 'z' not foundrQz: �nested�outz#A command in '%s' had a problem: %s)Zexc_info_on_loglevelzERROR: zProblem executing '%s': %szERROR executing 'z': z%Passed invalid arguments to {}: {}
{}rz'The minion function caused an exception)rzsys.doc�*ryz Possible reasons: '{}'�fun_argsr��	master_id�metadataz6The metadata parameter must be a dictionary. Ignoring.r?�,zminion return: %s�
ret_config�
ret_kwargs�	.returnerz#Returner %s could not be loaded: %s� The return failed for job %s: %s)@r�rUrVrWr�r@r1r��appendproctitler�r�rHr4r�rr�writery�dumpsr/r�r�r�r�rh�types�
GeneratorTyperirlrr}r�r�r�r��EX_OKr�r�r�rr<rr5�logging�DEBUGr
rrr8rr9Zfire_exceptionr�ZMinionErrorr�r�Zmissing_fun_stringrzr�r��_return_pub�setr�rr)r�r�r=rrrd�sdatarr?r�r�Zallow_missing_funcs�indZiretZsingler�Z
event_datar��func_resultrJr)Zdocs�mod_name�returnerZreturner_strZreturner_errrC)r�r�r�rDr��sd
�

����	
�



���
�����������
�
��








��� ���zMinion._thread_returncsd|��tj�|j|d�}tjj�|j	�d��dt�
�i}|�|�t�
d|d�tjj�|d��}|�tj�|��Wd�n1sHwY|�dd�}t|d	�}|rkdg|dg|dg|d
��niiid
��|�d�p�t|dg�p�|�ddg�}	td
|�D]�}
|d	|
}|d|
}|s�d�d|<zU|�|||	||�}
|r�|
n|d	|
�|
�d�<|jjd�dd
�}|d
kr�zt��fdd�dD��}Wnty�d}Ynw|s�d}|�d�<|d
k�d�<Wn1t�y$}z$t��}t�d|�|�r|�d|
<n
|�d|d	|
<WYd}~nd}~ww|d�d<|d	�d	<|d�d<d|v�rB|d�d<q�d|v�rN|d�d<|j�rW|� ��|d�r�d|v�rg|d�d<d|v�rr|d�d<t!|d�"d��D]4}|d �d <z
|j#|�d!���W�q{t�y�}zt�$d"|d|�WYd}~�q{d}~wwdSdS)#r�r�z._thread_multi_returnr�zStarting a new job with PID %sr�N�multifunc_orderedFr�)rr�r�rrrrur�rr�r�c3s$�|]
}�d��|d�VqdS)rTNr�r��rvr?rCrDr�s
��
�z.Minion._thread_multi_return.<locals>.<genexpr>rTr"z+The minion function caused an exception: %srr�rr?rrrr�rr)%r�rUrVrWr�r@r1r�rr�r�rHr4r�rrrryrr/r�r�r�r�r�r�r�r�r�r�r9r�rrrzr�r5)r�r�r=rrrdrrr#Z	num_funcsr�rr�r�r�r�r rJZtrbr"rCr$rDr��s�
��

��
������
�




 ���zMinion._thread_multi_return�_returnc

Cs�|�d|�d��}|�d|�d��}|jdr6tj�|j|�}tj�|�r6zt�|�Wn	ty5Ynwt	�
d|�t	�d|�|dkr�||jd	|||�d
�|�d�|�d�|�d
�d�}d|vrj|d|d<i|d<|��D]\}}|�
d�r|qr||d|<qrn||jdd�}|��D]\}}|||<q�d|vr�t|dt�r�|d|d<n't	�d|d�nz|j|j}	Wntttfy�Yn
wt|	t�r�|	|d<|jdr�|ddkr�tjj�|j�|d<tjj�|j|d|�|S)Nr��__jid__r��__fun__r��!Returning information for job: %szReturn data: %s�_syndic_returnrOru�tgt�tgt_type�__load__)r�r�r�r�rur*r+r��
__master_id__r
r�__r�)r�r�r
�+Invalid outputter %s. This is likely a bug.�
cache_jobs�req)r/r=rUrVrWr��isfile�remove�OSErrorr4r�r:rjr�rhr}r5r��
__outputter__r|r�rr@r1r�Zgen_jidr�r0)
r�r?�ret_cmdr�r�rdr�rvr��oputrCrCrD�_prepare_return_pub�sb
��

�
�

zMinion._prepare_return_pubc#s��|�d|�d���|�||�}|jdsdS�fdd�}z
|j||d�V}Wnty5|�d}Ynwt�d|�tjj	j
�|��)	Nr�r&�pub_retrc�t�d��dS�Nz�The minion failed to return the job information for job %s. This is often due to the master being shut down or overloaded. If the master is running, consider increasing the worker_threads value.T�r4r9r��r�rCrDr�+	�
�z0Minion._return_pub_main.<locals>.timeout_handlerr��ret_val = %s)r/r8r=r�rr4r:r@r�r�r�r��r�r?r6r�r�r��ret_valrCr=rD�_return_pub_main$	s�

�zMinion._return_pub_maincsv|�d|�d���|�||�}|jdsdS�fdd�}z	|j||d�}Wn
ty2|�YdSwt�d|�|S)	�P
        Return the data from the executed command to the master server
        r�r&r9rcr:r;r<r�r=rCrDr�F	r>z+Minion._return_pub.<locals>.timeout_handlerr�r?)r/r8r=r�rr4r:r@rCr=rDr=	s

�zMinion._return_pubcs�t|t�s|g}i}|D]�}|�d|�d���|�d|�d��}|jdrDtj�|j��}tj�|�rDzt�	|�Wn	t
yCYnwt�d��|�
�i�}	|dkr�|	ss|	�|jd�||�d	�|�d
�|�d�|�d�id
��d|vr}|d|	d<|��D]\}
}|
�d�r�q�||	d|
<q�n|	�d|jdi�|��D]\}
}||	|
<q�d|vr�t|dt�r�|d|	d<n't�d|d�nz|j|j}Wntttfy�Yn
wt|t�r�||	d<|jdr�tjj�|j|	d|�q|t|���d�}	�fdd�}
|�rz	|j|	|d�}Wn1t�y|
�YdSwtjj j!�"|
��|j#|	|dd�d�}Wd�n	1�s=wYt�$d|�|S)rCr�r&r�r'r�r(r)r�rur*r+r,)r�r�r�rur*r+r�rr-r
r.rr
r/r0)r�r�cr:r;r<r�r=rCrDr��	r>z1Minion._return_pub_multi.<locals>.timeout_handlerr�rcS�dSrrC)�frCrCrDr�	�z*Minion._return_pub_multi.<locals>.<lambda>)r��callbackNr?)%rhr~r/r=rUrVrWr�r2r3r4r4r��
setdefaultrHrjr�r}r5r�r5r|r�rr@r1r�r0�valuesr�rr�r�r��ExceptionStackContextr�r:)r�Zretsr6r��sync�jidsr?r�rdr�rvr�r7r�rArCr=rD�_return_pub_multiX	s�

���
�
�

�
�
��zMinion._return_pub_multicCs�|jdrb|j�dd�dkr|j�dd�dkrt�d�dSd|j�d	d
�d�}|jddkr<d
|d<|jdg|d<n|jddkrPd|d<|jdg|d<nd|d<g|d<|j�|j|�dSdS)zX
        Execute a state run based on information set in the minion config file
        Zstartup_statesr�r}r�rrzpCannot run startup_states when 'master_type' is set to 'disable' and 'file_client' is set to 'remote'. Skipping.r1Z
ext_job_cacher)r�r?Zslsz	state.slsr�Zsls_listru�topz	state.topZtop_filezstate.highstateN)r=r/r4r9r��add_callbackr��r�rrrCrCrD�
_state_run�	s"
��zMinion._state_runcCsNd|j�di�vr%d|jvri|jd<|jd�ddidg|d�i�dSdS)z�
        Create a loop that will fire a pillar refresh to inform a master about a change in the grains of this minion
        :param refresh_interval_in_minutes:
        :return: None
        Z__update_grainsr�z
event.fire�grains_refresh)r}rgr~N)r=r/rH)r�Zrefresh_interval_in_minutesrCrCrD�_refresh_grains_watcher�	s


����zMinion._refresh_grains_watcherccsz�d}|jdr
d}|jdr!|jd�|jdt���d|d�V|jd�|jdt���t|jdd	gd
�|d�VdS)NFr�T�enable_legacy_startup_eventszMinion {} started at {}r�Zminion_start)r�rOr�)r=r�r8r6�asctimer)r�Zinclude_grainsrCrCrD�_fire_master_minion_start�	s�

��z Minion._fire_master_minion_startcCsVt|d�sdSt�d|�|j||d�\|_|_}|_|j|j_|j|j_|��dS)�6
        Refresh the functions and returners.
        r�NzRefreshing modules. Notify=%s)rx)	r_r4r<r�r�r�r�r��beacons_refresh)r�r�rxr�rCrCrD�module_refresh�	s
�

zMinion.module_refreshcCs,|jsdSt�d�tj�|j|j�|_dS)rWNzRefreshing beacons.)r�r4r<r@r��Beaconr=r�rrCrCrDrX
s
zMinion.beacons_refreshcCst�d�tj�|j�|_dS)z&
        Refresh the matchers
        zRefreshing matchers.N)r4r<r@r{r�r=rrCrCrD�matchers_refresh
s
zMinion.matchers_refreshcCs�t|�D]	}||vr
||=qi}|D]-}||}dd�|D�}||vr?tjjj||||d�}|�d�r;||||<q|||<q|D]}||vrN||||<qB|S)z0
        Refresh the schedule in pillar
        cSsg|]	}|�d�r|�qSr�)r�)rR�itemrCrCrDrT'
sz2Minion.pillar_schedule_refresh.<locals>.<listcomp>)�ignore�new)r~r@r1Z
dictdifferZ	deep_diffr/)r�rr^r\Zpillar_scheduleZ
schedule_itemr]�diffrCrCrD�pillar_schedule_refresh
s*�
�
��zMinion.pillar_schedule_refreshccs0�|�|�|jrft�d�tjj|j|jd|jd|jd|j�d�|d�}z8z|�	�V}Wnt
y>t�d�Ynw|jd�d	i�}|�d	i�}|�||�|d	<||jd<W|�
�n|�
�w|��|��tjjjd
|jdd��}|jd
ditjjjd�Wd�dS1s�wYdS)z$
        Refresh the pillar
        zRefreshing pillar.r�r�r�r�)r��clean_cachezDPillar data could not be refreshed. One or more masters may be down!r�r�r�Fr�ZcompleteT)r�N)rYr�r4r<r@r�r�r=r/r�rr5r`r1r[rXr1r.r5r�r�r�ZMINION_PILLAR_REFRESH_COMPLETE)r�r�rar�Z
new_pillarZcurrent_scheduleZnew_scheduleZevtrCrCrD�pillar_refresh9
sF�


����
��"�zMinion.pillar_refreshcCs|�dd�}|�dd�}|�dd�}|�dd�}|�dd�}|�dd�}d||ffd	||ffd
|||ffd|ffd|ffd
||ffd|ffd||ffd||ffd||ffd|ffd|ffdd|ffd||ffd�}	z|	�|�\}
}t|j|
�|�WdSty�t�d|�YdSw)rWrqNr$r��wherer�r�r�r��
modify_jobZenable_scheduleZdisable_schedule�
enable_job�run_job�disable_job�postpone_job�skip_job�reloadr~)�
save_schedulerC�get_next_fire_time�
job_status)�deleter��modify�enabler�rerfrgrhrirjr~rkrlrmz4Function "%s" is unavailable in salt.utils.scheduler)r/r�r�rr4r5)r�r�rrrqr$r�rcr�r��funcs�alias�paramsrCrCrD�manage_schedulec
s8






��zMinion.manage_schedulec
Cs�|jsdS|�dd�}|�dd�}|�dd�}|�dd�}|�dd�}d||d�fd	||d�fd
d|ifdifdifd
d|ifdd|ifd||d�fdifd||d�fdifd�}z|�|�\}	}
t|j|	�di|
��WdSty~t�d|�YdSw)z 
        Manage Beacons
        Nrqr$�beacon_data�include_pillar�include_optsZ
add_beacon)r$ruZ
modify_beaconZ
delete_beaconZenable_beaconsZdisable_beacons�
enable_beacon�disable_beaconZlist_beacons)rwrvZlist_available_beacons�validate_beacon�reset)r�rornrpr�rxryr~Zlist_availablerzr{z2Function "%s" is unavailable in salt.utils.beaconsrC)r�r/r�r�rr4r5)r�r�rrrqr$rurvrwrqrrrsrCrCrD�manage_beacons�
s:


����zMinion.manage_beaconscCsP|�dd�}|durdS|�dd�}|�dd�}ddlmm}|�|||�S)z
        Set the salt-minion main process environment according to
        the data contained in the minion event data
        �environNF�false_unsets�	clear_allr)r/Zsalt.modules.environ�modulesr}Zsetenv)r�r�rrr}r~rZmod_environrCrCrD�environ_setenv�
szMinion.environ_setenvc
Cs�|jdur	d|_n$|jdurt�d|jj|jj�dS|jdur-t�d|jj|jj�dSzt�d|jjtjj�	��WdSt
ye}ztjtjj��rPt
jpRt
jd|jj|d�WYd}~dSd}~ww)	z�
        Set the minion running flag and issue the appropriate warnings if
        the minion cannot be started or is already running
        NTFz7This %s was scheduled to stop. Not running %s.tune_in()z4This %s is already running. Not running %s.tune_in()z%s is starting as user '%s'z)Failed to get the user who is starting %srQ)r`r4r5rr�r�r@r1r�Zget_userr�r�Z
is_windowsrr�ERROR)r�rArCrCrD�	_pre_tune�
s>

�
�

����zMinion._pre_tunec	Cs�tjjj�|j��3}|j|d<z|j||��|jdd�}|WWd�St	y;t
�d�YWd�dSw1s?wYdS)z.
        Send mine data to the master
        r�r�r�Nz#Unable to send mine data to master.)r@r�r��
ReqChannelr�r=r�r�r�rr4r9)r�r�rrr�r?rCrCrD�
_mine_send�
s 
��	
�	�zMinion._mine_sendc
	cs\�|jstjjj���tjjj�	|�\}}d|vr7|j
�d�dkr7|d}||jvr1tjjj���|j|}n|}t
�d|j
d|�|�d�rY|j|�dd�|�d	d�d
�dS|�d�r�|�dd
�dd�\}}||j
dkr�d}z|jj||��|j
dd�V}Wntjjy�t
�d||�tjjj���wtjjjd|j
dd��}	|	�d|id|���VWd�dS1s�wYdSt
�d|d||�dS|�d�r�|j|�dd�|�dd�d�VdS|�d�r�|��dS|�d�r�|��dS|�d��r
|�||�dS|�d��r|�||�dS|�d ��rA|�dd��s1|j|j
d!k�r?|jd"d#�|j
d!|_dSdS|�d$��rO|� ||�dS|�d%��r]|�!||�dS|�d&��r�|j"�r�t
�d'|d(|j
d�|�#|d)|d(|d*|d+�VdSt
�d,|j
d|d(�dS|�t$d-d.���s�|�t$d/d.���rF|�t$d-d.���r�|d|j
dk�r�tjjj���|�t$d/d.���r�|d|j
d0d1k�r�t%d2�&|d|j
d���|d|j
dd1k�r�t%d3�&|d���|j"�rDd|_"t
�'d4|j
d�|j
d5d6k�r|j(j)t$d7d.�d8�t
�'d9�t*|d:��rC|j+�,d�t*|j+d;��r7|j+j-�.�t*|j+d<��rC|j+�/�t*|d=��rU|j�rU|j�/�d|_z|j0|j
d"|�t$d/d.��d>�V\}
|_+Wn
t1�ytYnw|j"�r:|
|j
d<t
�'d?|j
d�tj2j3j4j5|j
|j6d@�|_|j�7�V|j(�8dA�|j
dA<|�9�\|_:|_;|_<|_=|j:|j(_:|j+�,|j>�|�?�Vt
�'dB�|j
d5d6k�r4dC|j
dDd"dd|j
dd"dE�dF�}|j(j@t$d7|j
ddG�|dH�|j
dI�r6d0|j
v�r8|j
d|j
d0d1k�r'dJ|j
dKd"ddd|j
d0d1idF�}|j(j@t$d/d.�|dH�dS|j(j)t$d/d.�d"dL�dSdSdSdSd"|_A|j6�B�dSdS|�t$dMd.���r�|j"�s�|j
dNdOk�r�t
�'dP|j
d�d"|_"|j
d5d6k�r�dC|j
dDd"dd|j
dd"dE�dF�}|j(j@t$d7|j
ddG�|dH�dSdSdSdS|�dQ��r�|dA�t$d7dRdG���r�|dS�r�t
�dT|dA�Ct$d7dRdG��d�|jD|dUdV�VdS|�dW��r�|j"�r�t
�dX|�|�#||�VdSdS|�dY��rtE|dZ�}t
�d[|tjFjGjH�|�|d\�|d\tjFjGjHtE|dZ�<dS|�d]��r*|j"�r,t
�d^�|j#|d_d`�VdSdSdS)azO
        Handle an event from the epull_sock (all local minion events)
        �proxy_target�	metaproxyZ
deltaproxyz)Minion of '%s' is handling event tag '%s'rrYr�Frx)r�rxZ__master_req_channel_payloadr�r_r"Nr�r�z3Timeout encountered while sending %r request. id=%sr�r�r?r�z5Skipping req for other master: cmd=%s master=%s id=%sr�rbra)r�rarXr[rtr|rRr�T)r�r�Z_minion_mineZfire_masterz Forwarding event %s to master %sr�rrr�r�z-Master %s is not connected, dropping event %sr�r{r�r�rz*Bad master '{}' when mine failback is '{}'zAlready connected to '{}'zConnection to master %s lostr�r�r�r#z1Trying to tune in to next master from master-listr�r�r�rd)r=r�r�z,Re-initialising subsystems for new master %sr4r��$Minion is ready to receive requests!r�r�r�r�r�)r$r�r�r�r�)r$r�r�r�r�z&Connection to master %s re-establishedZ__schedule_returnrrzConnected to master %sr%)r6Z_salt_errorz"Forwarding salt error event tag=%szsalt/auth/credsrvz#Updating auth data for %s: %s -> %sZcreds�__beacons_returnzFiring beacons to masterr�)r�)Irbr@r�r�r�r�r1r.Z	SaltEvent�unpackr=r/Zdeltaproxy_objsr4r<r�rY�rsplitrdr�r�r�rr5r5r�rbrXr[rtr|�grains_cacher�r�r�r�r�r	r8r�r�r�r_r�rtr�rur�r�rr�r�rvr�r�r��optionr�r�r�r�r��_handle_payloadrVrdrYr\rzrBr�r�r�r�)
r�r;r�rrr�Z_minionr�Z
job_masterr?r.rr�rvrCrCrDr8�
s��




�
�����"��


�


���
�������


��
�
��


����
�

���b�	
��
����
�zMinion.handle_eventcCs*t��|j��|jr|j��dSdS)z<
        Clean up subprocesses and spawned threads.
        N)r�Zactive_childrenrar|r��cleanup_subprocessesrrCrCrDr�s

�zMinion.cleanup_subprocessescCs�|jsK|��\|_|_|_|_|��|_tj	�
|j�|_
|jr)tj
�|j|j�|_
tjjj|j�dd�d�}t|jd|d�|_|jd|_d|_dSdS)za
        Set up the core minion attributes.
        This is safe to call multiple times.
        r�Nr�rbr�r�T)rbr�r�r�r�r�r�r�r@r{r�r=r�r�rZr1r�r�r/rer�r�)r�rOrCrCrDr�s �

�zMinion._setup_corecsd�jsdS����jd}d�jvr0tj��j�j��_�fdd�}|r(|���d|�dSdS)zR
        Set up the beacons.
        This is safe to call multiple times.
        Nr�r�cs�d}z���j�}Wntytjddd�Ynw|rCtjjjd�j	dd��}|�
d|id�Wd�dS1s<wYdSdS)	NzThe beacon errored: TrQr�Fr�r�r�)r�r�r�r4r�r@r1r.r5r=r�)r�r.rrCrD�handle_beacons4s��"��z,Minion.setup_beacons.<locals>.handle_beacons)	r�r�r=rcr@r�rZr��add_periodic_callback)r�rMr�r�rCrrDrS&s


�zMinion.setup_beaconsc
s�����jd�d�jvryd�jvri�jd<t�d�s2tjjj�j�j�j	�jt
dd�gd��_z�jdrKt�d�jd���
t�jd��Wntyd}zt�d|�WYd	}~nd	}~ww��fd
d�}|rq|���d|�d	Sd	S)zT
        Set up the scheduler.
        This is safe to call multiple times.
        r�r�r�r{)r1r|Zgrains_refresh_everyz;Enabling the grains refresher. Will run every %d minute(s).z[Exception occurred in attempt to initialize grain refresh routine during minion tune-in: %sNcs�����dSr)r�rC�r�r�rCrD�handle_scheduleksz/Minion.setup_scheduler.<locals>.handle_schedule)r�r=rcr_r@r1r�r�r�r�r�r4r<rS�absr�r5r�)r�rMrJr�rCr�rDrTGsB





�
������zMinion.setup_schedulerr"cCs<||jvrdStjjj�||d�|j|<|j|��dS)z�
        Add a periodic callback to the event loop and call its start method.
        If a callback by the given name exists this method returns False
        F��T)rcr@r�r�r�PeriodicCallbackrO)r�r$�method�intervalrCrCrDr�ts


�zMinion.add_periodic_callbackcCs&|j�|d�}|dur
dS|��dS)z~
        Remove a periodic callback.
        If a callback by the given name does not exist this method returns False
        NFT)rcrXr\)r�r$rGrCrCrD�remove_periodic_callback�s
zMinion.remove_periodic_callbackc	s����t�d�jd�|r-�j�dd�r�jdd��j�dd�r)�jdd�����jr<�j	�
�j�t�d�t
�trGtjj�������������d	�j��j�d
d�d}|dkr|�jr|�fd
d�}��d���d||�t�d�r��jdur��j��j�n
�j�d�dkr�t�d�|r�z�j	���jr����WdSWdStt fy����YdSwdS)�k
        Lock onto the publisher. This is the main event loop for the minion
        :rtype : None
        zMinion '%s' trying to tune inr�rKFTrLrNr�r|�
ping_intervalrr�csNz�fdd�}�jj�jdd|d�WdSty&tjdtjd�YdSw)Ncsl�j�dd�r4t�d��j�dd�}t�d|�z�jdt��WdSty3t�d�YdSwdS)	N�
auth_safemodeFz5** Master Ping failed. Attempting to restart minion**Zrandom_reauth_delayr�z delaying random_reauth_delay %sszservice.restartz�ping_interval reached without response from the master, but service.restart could not be run to restart the minion daemon. ping_interval requires that the minion is running under an init system.)	r=r/r4r5r�r�r�r|r9)r��delayrrCrD�ping_timeout_handler�s�
���zAMinion.tune_in.<locals>.ping_master.<locals>.ping_timeout_handler�pingZminion_ping)r�zAttempt to ping master failed.)Zexc_on_loglevel)r�rOr�r�r4r9rr)r�rrCrD�ping_master�s���z#Minion.tune_in.<locals>.ping_masterr�r�Nr�r�z;No connection to master found. Scheduled jobs will not run.)!r�r4r<r=r/rSrTrkr�r�rOrVr�r�HAS_WIN_FUNCTIONSr@r1Z
win_functionsZenable_ctrl_logoff_handlerrQr�r�r�r_r�rtr�r5rOrYr1rp�RuntimeError)r�rOr�r�rCrrDrV�sP

!

�
���zMinion.tune_incCs^|dur)|ddkr+|�|d�r|�|d�dS|jdr-t�d|d�dSdSdSdS)N�enc�aesr�Z
zmq_filteringz8Broadcast message received not for this minion, Load: %s)�_target_loadr�r=r4r:�r�ryrCrCrDr��s
��zMinion._handle_payloadcCs�d|vsd|vsd|vsd|vrdSd|vrJ|j�d�|d�d�}|dur(dS|ddvr@|�d	t�}||d|d
�s>dSdS||d�sHdSdS|jd|d�sUdSdS)
Nr*r�r�ruFr+z{}_match.match)ZgrainZ
grain_pcrer��	delimiter)r�zglob_match.matchT)r�r/r8r)r�r�Z
match_funcr�rCrCrDr��s,	���zMinion._target_loadcCs�d|_t|d�r
|`t|d�r|jdur|j�d�|j��t|d�r.|jdur.|j��t|d�r?|j��D]}|�	�q8dSdS)z&
        Tear down the minion
        Fr�r�Nrdrc)
r`r_r�r�rtr�rdrcrIr\)r��cbrCrCrDr1
s




�zMinion.destroycCr/rr0rrCrCrDr2*
r3zMinion.__del__)r�TNNNT�NFr )FFNNN)NNNNr�NF)r%)r%r�)r%r�T)FF)r"�T)<r�r�rrr�rirkr@r�r�r�rrUrzrwr�r�r�r�r�r�r�r�r�r��classmethodr�r�r�r�r8rBrrMrQrSrVrYrXr[r`rbrtr|r�r�r�r8r�r�rSrTr�r�rVr�r�r1r2rrCrCrrDr@�s��_

 



s
�U




�
�
�

`

!>
I

[
8

^


	
!)$(&




!
-
Z#r@cs�eZdZdZ�fdd�Zdd�Zdd�Zdd	�Zej	j
jjd
d��Z
dd
�Zdd�Zdd�Zej	j
jjdd��Z�fdd�Z�ZS)�Syndicz�
    Make a Syndic minion, this minion will use the minion keys on the
    master to authenticate with a higher level master.
    csnd|_d|_|�d�|_d|_d|d<d|d<t�j|fi|��tj�	|�|_
t�|_i|_
g|_d|_dS)NZ	interfaceTr�r"r�)r�forward_eventsr/Z_syndic_interfaceZ_syndicr
r�r@r�r�mminionr�jid_forward_cacherL�
raw_eventsZ
pub_future)r�r=rcrrCrDr�6
s
zSyndic.__init__cCsJt|�d|jd��d|d<|�dd�|j�dd�kr#|�|�dSdS)r��tor�r"r
rN)r3r/r=�
syndic_cmdrPrCrCrDr�E
s�zSyndic._handle_decoded_payloadc
Cs�d|vrd|d<d|�dg�i}dD]}||vr||||<qdd�}tjjj�|��-|jj|d|d|d	|d|d
|d|df|jd
d�d�|��Wd�dS1s[wYdS)zM
        Take the now clear load and forward it on to the client cmd
        r+�globZ	auth_list)r
r�cWst�d|d�dS)NzUnable to forward pub data: %sr"Tr<)rgrCrCrDr�b
sz*Syndic.syndic_cmd.<locals>.timeout_handlerr*r�rur?r�r�cSrDrrCr�rCrCrDrp
rFz#Syndic.syndic_cmd.<locals>.<lambda>)r�rGN)	rXr@r�r�r�rJrZ	pub_asyncr�)r�rrrc�fieldr�rCrCrDr�P
s0���
�"�zSyndic.syndic_cmdcCs`|jdr$t�d�tj�|jdd�}tj�|tj	j
�|��}||d<|jj
|||jdd�S�Nr�r�r�r�r�r�r�)r=r4r:rUrVrWr@r�r�r�r�r�rdr�)r�r�r�r�r�rCrCrDr�t
s

��zSyndic._send_req_syncccsh�|jdr%t�d�tj�|jdd�}tj�|tj	j
�|��}||d<|jj
|||jdd�V}|Sr�)r=r4r:rUrVrWr@r�r�r�r�r��async_req_channelr�r�rCrCrDr��
s�

��zSyndic._send_req_asynccCsZ|jdr|�d�|jdt���d�|�d�|jdt���t|jddgd��dS)NrTzSyndic {} started at {}r�Zsyndic_startrOr$)r=r�r8r6rUrrrCrCrD�fire_master_syndic_start�
s
��zSyndic.fire_master_syndic_startcCsTtjj|jd|jd�|_|j�|j�tj	jj
�|j�|_tj	jj
�|j�|_dS)z�
        Executes the tune_in sequence but omits extra logging and the
        management of the event bus assuming that these are handled outside
        the tune_in sequence
        �_minion_conf_filer4N)r@r��get_local_clientr=r�rr�rt�_process_cmd_socketr�r�r�rdrvr�rrCrCrD�tune_in_no_block�
s�zSyndic.tune_in_no_blockcCs8|dur|ddkrt�d�|�|d�dSdSdS)Nr�r�zHandling payloadr�)r4r:r�r�rCrCrDr��
s
�zSyndic._process_cmd_socketccs��t|d�r|j�d�t|jd�r|j��|`|j|jd�V\}|_|jr8||jd<|j�|j�t�	d�t
jjj
�|��)Nr�r�r�rr�)r_r�rtr�r�r=r�r�r4r�r@r�r�r�r�)r�rrCrCrD�	reconnect�
s�



zSyndic.reconnectcsFt���|jdur|j��d|_|jdur!|j��d|_dSdS)z-
        Tear down the syndic minion
        N)r
r1rr�r\rrrCrDr1�
s





�zSyndic.destroy)r�r�rrr�r�r�r�r@r�r�r�rr�r�r�r�r�r1rrCrCrrDr�0
s$




r�cs�eZdZdZdZdZd�fdd�	Zdd�Zej	j
jjdd	��Z
d
d�Zd d
d�Zddd�Zddd�Zdd�Zdd�Zdd�Zdd�Zdd�Zdd�Z�ZS)!�
SyndicManagera�
    Make a MultiMaster syndic minion, this minion will handle relaying jobs and returns from
    all minions connected to it to the list of masters it is connected to.

    Modes (controlled by `syndic_mode`:
        sync: This mode will synchronize all events and publishes from higher level masters
        cluster: This mode will only sync job publishes and returns

    Note: jobs will be returned best-effort to the requesting master. This also means
    (since we are using zmq) that if a job was fired and the master disconnects
    between the publish and return, that the return will end up in a zmq buffer
    in this Syndic headed to that original master.

    In addition, since these classes all seem to use a mix of blocking and non-blocking
    calls (with varying timeouts along the way) this daemon does not handle failure well,
    it will (under most circumstances) stall the daemon for ~15s trying to forward events
    to the down master
    r�Ncs�d|d<t��|�d|_tj�|�|_|j�dd�|_	|j�dd�|_
|jd|_|jd	|_t
��|_t�|_|durGtjjjj��|_n||_g|_i|_g|_i|_dS)
Nr"r�F�syndic_moderK�syndic_failoverr�r�r")r
r��_closingr@r�rr�r=r/r�r�r&r'r��EventZ_has_masterrr�r�r�rrrr�r��job_rets�delayed�pub_futures)r�r=r�rrCrDr��
s"

zSyndicManager.__init__cCsTt�|_|jd}t|t�s|g}|D]}t�|j�}||d<|�|�|j|<qdS)zI
        Spawn all the coroutines which will sign in the syndics
        rN)r�_syndicsr=rhr~r��_connect_syndic)r�r�rrIrCrCrD�_spawn_syndicss

�zSyndicManager._spawn_syndicsc
cs�d}|d}d}	t�d|d�z#t||jd|jd�}|j|d�V|��|��t�d	|d�WnQt	yh}z'd}t�
d
|d�t��}||jkrU||j
7}tjjj�|�VWYd}~n"d}~wttfyq�ty�d}tjd|ddd
�Ynwq
tjjj�|��)zL
        Create a syndic, and asynchronously connect it to a master
        rr�FTz"Syndic attempting to connect to %sr)r�r�r�rz#Syndic successfully connected to %szPError while bringing up syndic for multi-syndic. Is the master at %s responding?NrPrQ)r4r<r��SYNDIC_CONNECT_TIMEOUTr�rUr�r�r�rr5r6r'r&r@r�r�r�r7rp�
SystemExitr�r�r�)r�r=rWr&r�r$rJrCrCrDr�sR���

�
���'zSyndicManager._connect_syndiccCs>|j|��r|j|��}|��|j|<dSt�d|�dS)zL
        Mark a master as dead. This will start the sign-in routine
        zAAttempting to mark %s as dead, although it is already marked deadN)r�rqrr�r4r�)r�rr$rCrCrD�_mark_master_deadLs�zSyndicManager._mark_master_deadrCc	Cs�|duri}d}|�|�D]9\}}|��r|��r!t�d||�q
zt|��|�|i|��d}Wq
tyFt�d||�|�|�Yq
w|sQt�	d|�dSdS)zd
        Wrapper to call a given func on a syndic, best effort to get the one you asked for
        NF�5Unable to call %s on %s, that syndic is not connectedT�*Unable to call %s on %s, trying another...z!Unable to call %s on any masters!)
�iter_master_optionsrqrrr4r5r�rrr�r�)r�rqrgrcr
Z
successfulr�
syndic_futurerCrCrD�_call_syndic[s*���zSyndicManager._call_syndiccCs�d}|�|�D]`\}}|��r|��rt�d||�q|j�|d�\}}|durO|��s4||kr3dSq|��rOt�d||�|�|�|j|=|j�	|�qt
|��|�|d|��dd�}||f|j|<d	SdS)
zl
        Wrapper to call the '_return_pub_multi' a syndic, best effort to get the one you asked for
        rMr�r�NFr�r))r�rKT)
r�rqrrr4r5r�r/r�r��extendr�rr�)r�rIr
rqrr�rnrrrCrCrD�_return_pub_syndicus:��
�z SyndicManager._return_pub_syndicccsn�t|j���}|jddkrt�|�||jvr|�d�}n|�|�	||j|fV|s1dS|�d�}q%)zA
        Iterate (in order) over your options for master
        r�r�rTN)r~r�ror=r�r�rXr3)r�r
r�rCrCrDr��s�



�z!SyndicManager.iter_master_optionscCsi|_g|_dSr)r�r�rrCrCrD�_reset_event_aggregation�s
z&SyndicManager._reset_event_aggregationcCs$|jj�|j�}|j�||j�dSr)rr.r7�_process_eventr��
add_future�reconnect_event_bus)r�Z	somethingrnrCrCrDr��sz!SyndicManager.reconnect_event_buscCs�|��tjj|jd|jd�|_|jj�d�t	�
d|jd�i|_g|_|�
�|jj�|j�}|j�||j�tjjj�|j|jdd�|_|j��t�|j��dS)	zU
        Lock onto the publisher. This is the main event loop for the syndic
        r�r4rz$SyndicManager '%s' trying to tune inr�Zsyndic_event_forward_timeoutr�N)r�r@r�r�r=r�rr.r6r4r<r�r�r�r7r�r�r�r�r�rr��_forward_eventsr�rOr)r�rnrCrCrDrV�s$�
�
zSyndicManager.tune_incCs�|jj�|�\}}t�d|�|�d�}t|�dkr�|ddkr�tjj	�
|d�r�|ddkr�d	|vr�d
|vr9dS|jdkrR|�dd
�|j
�dd�krRt�d�dS|�d�}|j�|i��|i�}|s�|�d�|d<|d
|d<i|d<d�|j
d�}|d
|jvr�|d�|jj||d
��|j�|d
�t|j�|j
dkr�tt|j��}|�d
�t|�|_|dur�||d<i}	dD]}
|
|vr�||
|	|
<q�|	||d<dS|jdkr�d|vr�|j�||d��dSdSdS)NzGot event %sr��r"rr_r�r?rr�Zclusterr
rz7Return received with matching master_id, not forwardingr�r'r&r,z{}.get_loadZmaster_job_cacheZsyndic_jid_forward_cache_hwmr-)rr�r�rr�rKr�)rrr�)rr.r�r4r:rzr�r@r1r�Zis_jidr�r/r=r<r�rHr8r�rHr�r�r��sortedr~rXrr�rl)r��rawZmtagrrZ	tag_partsrZjdictZfstr�tmpr?rvrCrCrDr��s^
���

�

�
�zSyndicManager._process_eventcCs�t�d�|jr#|j}g|_|jd|t|jddd�|��dd�d�|jr1|�|j�}|r1g|_t	|j
���D]}t	|j
|���}|j||d	�}|rP|j
|=q8dS)
NzForwarding eventsr�r�r$)rT)r�r�r�rK)rc)r
)
r4r:r�r�rr=r�r�r�r~r�rorI)r�r��resrrIrCrCrDr�s.
��	��zSyndicManager._forward_eventscCs0|jdurdSd|_|jdur|j��dSdSrl)r�rr1rrCrCrDr1,s

�zSyndicManager.destroyr)rCNN)r�r�rrr�ZSYNDIC_EVENT_TIMEOUTr�r�r@r�r�r�rr�r�r�r�r�r�r�rVr�r�r1rrCrCrrDr��
s$

0


) ;r�c@s"eZdZdZ				ddd�ZdS)�ProxyMinionManagerzA
    Create the multi-minion interface but for proxy minions
    NTc	Cst||||||dd�S)r<Tr=)�ProxyMinionrArCrCrDrB9rCz(ProxyMinionManager._create_minion_objectr])r�r�rrrBrCrCrCrDr�4s	�r�cCsnd�|dtjjj�}tjj||d�}z|d}Wnty,d}t�d|d�Ynw|d|}||S)Nz{}.{}r�)r>r�r�zMNo metaproxy key found in opts for id %s. Defaulting to standard proxy minionry)	r8r@r{ZlazyZLOADED_BASE_NAMEr�r|r4r<)r=Zfn_namer>r�Zmetaproxy_nameZmetaproxy_fnrCrCrD�_metaproxy_callQs��r�c@s�eZdZdZejjjjdd��Z	ejjjjdd��Z
ddd�Zd	d
�Zdd�Z
ejjjjd
d��Zedd��Zedd��Zedd��ZdS)r�z�
    This class instantiates a 'proxy' minion--a minion that does not manipulate
    the host it runs on, but instead manipulates a device that cannot run a minion.
    cC�t|jd�}|||�S)a�
        Function to finish init after connecting to a master

        This is primarily loading modules, pillars, etc. (since they need
        to know which master they connected to)

        If this function is changed, please check Minion._post_master_init
        to see if those changes need to be propagated.

        ProxyMinions need a significantly different post master setup,
        which is why the differences are not factored out into separate helper
        functions.
        Zpost_master_init�r�r=)r�r�mp_callrCrCrDrwis
zProxyMinion._post_master_initcCst|jd�}||||�S)zT
        Function to finish init for the sub proxies

        :rtype : None
        �subproxy_post_master_initr�)r�Z	minion_idrOr�rCrCrDr�{sz%ProxyMinion.subproxy_post_master_initTcCr�)r�rVr�)r�rOr�rCrCrDrV�s
zProxyMinion.tune_incCr�)zQ
        Verify that the publication is valid and applies to this minion
        Ztarget_loadr�)r�r�r�rCrCrDr��s
zProxyMinion._target_loadcCr�)Nrzr�)r�ryr�rCrCrDr��s
zProxyMinion._handle_payloadcCr�)NZhandle_decoded_payloadr�)r�rrr�rCrCrDr��s
z#ProxyMinion._handle_decoded_payloadcCst|d�}|||||||�S)Nr��r�)r�r�r=rrr�r�r�rCrCrDr��s
zProxyMinion._targetcC�t|d�}|||||�S)NZ
thread_returnr��r�r�r=rrr�rCrCrDr���
zProxyMinion._thread_returncCr�)NZthread_multi_returnr�r�rCrCrDr��r�z ProxyMinion._thread_multi_returnNr�)r�r�rrr@r�r�r�rrwr�rVr�r�r�r�r�r�r�rCrCrCrDr�bs"




	



r�c@seZdZdZddd�ZdS)�SProxyMiniona
    Create an object that has loaded all of the minion module functions,
    grains, modules, returners etc.  The SProxyMinion allows developers to
    generate all of the salt minion functions and present them with these
    functions for general use.
    FNcCs�tjj�|jd�tj�|j�|jd<tjj|j|jd|jd|jd|j�	d�d��
�|jd<d|jdvrWd|jvrWdj|jdd	�}t�
|�d
|_ttjjj|d��d|jvrf|jdd|jd<tj�|j�|_tjj|j|j|d�|_tjj|j|jd
|j|d
�|_tjj|j|j|j|d�|_tj�|j�|_|j|jd<tjj|j|j|j|d�|_|jdd}|jd|jdd�|j|jjd<|j|jjd<|j|jjd<|jd|jjd<tjj|j|j|d�|_|j|jjd<|j��|�d�|jv�s
|�d�|jv�r%d�|�d}t�
|�d
|_ttjjj|d��|j�	|�d�dd���|_|j|d}||j�tjj|j|jd�|jd<|jd d!d�|jd|_d"|_d#S)$r�r�r�r�r�)r�r�r�r�z�No "proxy" configuration key found in pillar or opts dictionaries for id {id}. Check your pillar/options configuration and contents. Salt-proxy aborted.)r�Fr'r�r�r�r�Z	proxytypezsaltutil.sync_all)r�Z	__proxy__Z__salt__Z__ret__Z
__pillar__Z	__utils__z.initz	.shutdownz=Proxymodule {} is missing an init() or a shutdown() or both. z,Check your proxymodule.  Salt-proxy aborted.z.module_executorscSsgSrrCrCrCrCrDrrFz*SProxyMinion.gen_modules.<locals>.<lambda>)r�zsaltutil.sync_grainsrTN)r@r1ZextmodsrKr=r{r�r�r�r/r�r8r4r5r`r
r�r�r�r�r�r�r�r�r�r�r�Zreload_modulesrr�rb)r�r�r��errmsgZfq_proxynameZ
proxy_init_fnrCrCrDr��s�
����

���
���
�

zSProxyMinion.gen_modulesr�)r�r�rrr�rCrCrCrDr��sr�r�r�r)rrr�r�r�r�rr�rUr�rer[r�r�r6r�rr�r@Zsalt.beaconsZsalt.channel.clientZsalt.cli.daemonsZsalt.clientZ
salt.cryptZsalt.defaults.eventsZsalt.defaults.exitcodesZsalt.enginesZsalt.ext.tornadoZsalt.ext.tornado.genZsalt.ext.tornado.iolooprZsalt.loader.lazyZsalt.payloadZsalt.pillarZsalt.serializers.msgpackZ
salt.syspathsZsalt.transportZsalt.utils.argsZsalt.utils.contextZsalt.utils.dataZsalt.utils.dictdifferZsalt.utils.dictupdateZsalt.utils.errorZsalt.utils.eventZsalt.utils.extmodsZsalt.utils.filesZsalt.utils.jidZsalt.utils.minionZsalt.utils.minionsr0Zsalt.utils.platformZsalt.utils.processZsalt.utils.scheduleZsalt.utils.ssdpZsalt.utils.userZsalt.utils.zeromqZsalt._compatrZsalt.configrZ
salt.defaultsrZsalt.exceptionsrrrrr	r
rrr
Z
salt.templaterZsalt.utils.ctxrZsalt.utils.debugrrrZsalt.utils.odictrrrrrrr�r��ImportErrorr�r�Zsalt.utils.win_functionsr��	getLoggerr�r4rErKrerxr�r�r�r�rrr!r@r�r�r�r�r�r�rCrCrCrD�<module>s�,���

y
79
f<;Oi)^L