File: //opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/__pycache__/hgfs.cpython-310.pyc
o
�N�g�{ � @ s� d Z ddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl
Z ddl Z ddl
Z ddlZ ddlZ ddlZ ddlZ ddlZ ddlmZ ddlmZ ddlmZ dZdZzddlZd Z W n e yq d
Z Y nw e� e �Z d Z!d Z"d
d� Z#dd� Z$dd� Z%dd� Z&dd� Z'dd� Z(dd� Z)dd � Z*d d � Z+d d � Z,d!d"� Z-d#d$� Z.d%d&� Z/dDd'd(�Z0dDd)d*�Z1d+d,� Z2d-d.� Z3dEd/d0�Z4dFd2d3�Z5d4d5� Z6d6d7� Z7d8d9� Z8d:d;� Z9d<d=� Z:d>d?� Z;d@dA� Z<dBdC� Z=dS )Ga�
Mercurial Fileserver Backend
To enable, add ``hgfs`` to the :conf_master:`fileserver_backend` option in the
Master config file.
.. code-block:: yaml
fileserver_backend:
- hgfs
.. note::
``hg`` also works here. Prior to the 2018.3.0 release, *only* ``hg`` would
work.
After enabling this backend, branches, bookmarks, and tags in a remote
mercurial repository are exposed to salt as different environments. This
feature is managed by the :conf_master:`fileserver_backend` option in the salt
master config file.
This fileserver has an additional option :conf_master:`hgfs_branch_method` that
will set the desired branch method. Possible values are: ``branches``,
``bookmarks``, or ``mixed``. If using ``branches`` or ``mixed``, the
``default`` branch will be mapped to ``base``.
.. versionchanged:: 2014.1.0
The :conf_master:`hgfs_base` master config parameter was added, allowing
for a branch other than ``default`` to be used for the ``base``
environment, and allowing for a ``base`` environment to be specified when
using an :conf_master:`hgfs_branch_method` of ``bookmarks``.
:depends: - mercurial
- python bindings for mercurial (``python-hglib``)
� N)�datetime)�DEFAULT_HASH_TYPE��FileserverConfigError)�tagify)�branches� bookmarks�mixed)�base�
branch_method�
mountpoint�rootTF�hgfs)�hgc C sh t td vrdS tst�d� dS td tvr"t�dtd t� dS tjj� d�du r2t�d� dS t S ) z-
Only load if mercurial is available
Zfileserver_backendFzeMercurial fileserver backend is enabled in configuration but could not be loaded, is hglib installed?Zhgfs_branch_methodz6Invalid hgfs_branch_method '%s'. Valid methods are: %sr Nz2hgfs requested but hg executable is not available.)
�__virtualname__�__opts__�HAS_HG�log�error�VALID_BRANCH_METHODS�salt�utils�path�which� r r �H/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/hgfs.py� __virtual__Q s$ � �
r c C s dd� | � � D �}|S )z5
Returns all branches for the specified repo
c S �6 g | ]}t jj�|d �|d t jj�|d �f�qS �r � � �r r � stringutils�to_str��.0�xr r r �
<listcomp>q � (��z!_all_branches.<locals>.<listcomp>)r )�repor r r r �
_all_branchesj s �r* c �2 z� fdd�t | �D �d W S ty Y dS w )z9
Find the requested branch in the specified repo
c � g | ]
}|d � kr|�qS �r r r$ ��namer r r' } � z _get_branch.<locals>.<listcomp>r F)r* �
IndexError�r) r/ r r. r � _get_branchx �
�r3 c C s dd� | � � d D �}|S )z6
Returns all bookmarks for the specified repo
c S r r r! r$ r r r r' � r( z"_all_bookmarks.<locals>.<listcomp>r )r )r) r r r r �_all_bookmarks� s
�r5 c r+ )z;
Find the requested bookmark in the specified repo
c r, r- r r$ r. r r r' � r0 z!_get_bookmark.<locals>.<listcomp>r F)r5 r1 r2 r r. r �
_get_bookmark� r4 r6 c C s dd� | � � D �S )z1
Returns all tags for the specified repo
c S sR g | ]%}t jj�|d �dkrt jj�|d �|d t jj�|d �|d f�qS )r �tipr r � r! r$ r r r r' � s ��z _all_tags.<locals>.<listcomp>)�tags)r) r r r � _all_tags� s �r: c r+ )z6
Find the requested tag in the specified repo
c r, r- r r$ r. r r r' � r0 z _get_tag.<locals>.<listcomp>r F)r: r1 r2 r r. r �_get_tag� r4 r; c C s� |dkr| d }|| d ks|t � v rV| d dkr't| d |�p&t| d |�S | d dkr;t| d |�p:t| d |�S | d dkrVt| d |�pUt| d |�pUt| d |�S dS )z1
Return ref tuple if ref is in the repo.
r
r r r) r r F)�envsr3 r; r6 r2 r r r �_get_ref� s � �r= c C s dd� | j |d d�D �}|S )z
Get manifest for ref
c
S sL g | ]"}t jj�|d �t jj�|d �|d |d t jj�|d �f�qS )r r r r8 � r! r$ r r r r' � s ���z!_get_manifest.<locals>.<listcomp>r )�rev)�manifest)r) �refr@ r r r �
_get_manifest� s �
rB c C s t d��)zB
Fatal fileserver configuration issue, raise an exception
z$Failed to load hg fileserver backendr r r r r � _failhard� s rC c
C s
t j�td d�} d}g }i }tD ]
}ttd|� � �||< qtd D �]}}t�|�}t|t �r�t
t |��}dd� t j
j�|| ��� D �}|sQt�d|� t� |�d |d �} | tvrkt�d
| |d �t�� t� d}
d d
� |D �D ]}t�d||d �t�� d}
qt|
r�t� |�|� n|}t|t�s�t�d|� t� z
t j
j�|d �|d< W n ty� Y nw ttt�dt ��} | |� d��� � } t j�| | �}
t j� |
�s�t � |
� t �!|
�s�t"�#|
�}|�$� d}d}z�zt"�%|
�}W n2 t"jj&�y
t�d|
|� t� Y n t'�y& } zt�d||� t� W Y d}~nd}~ww z|j(dd�}W n t"jj)�y= d}Y nw |�sut j�|
dd�}t j
j*�+|d��}|�,d � |�,t j
j-�.d |� d ��� W d � n 1 �spw Y |�||| |
t j�td d| � d ��d �� |�/|� W |�r�|�$� q#|�r�|�$� w w |�rt j�td d!�}zCt j
j*�+|d��0}t0�1� �2d"�}|�,d#|� d �� |D ]}|�,t j
j-�.d$�3|d% |d& ��� �q�W d � n 1 �s�w Y W n t4�y� Y |S w t�5d'|� |S )(zE
Return a list of hglib objects for the various hgfs remotes
�cachedirr FZhgfs_Z hgfs_remotesc S s i | ] \}}|t |��qS r )�str)r% �key�valr r r �
<dictcomp>� s ��zinit.<locals>.<dictcomp>z�Invalid per-remote configuration for hgfs remote %s. If no per-remote parameters are being specified, there may be a trailing colon after the URL, which should be removed. Check the master configuration file.r zdInvalid branch_method '%s' for remote %s. Valid branch methods are: %s. This remote will be ignored.z, c s s � | ] }|t vr|V qd S �N)�PER_REMOTE_OVERRIDESr$ r r r � <genexpr> s � zinit.<locals>.<genexpr>z|Invalid configuration parameter '%s' for remote %s. Valid parameters are: %s. See the documentation for further information.TzZInvalid hgfs remote %s. Remotes must be strings, you may need to enclose the URL in quotesr � hash_typezutf-8Nz�Cache path %s (corresponding remote: %s) exists but is not a valid mercurial repository. You will need to manually delete this directory on the master to continue to use this hgfs remote.z<Exception '%s' encountered while initializing hgfs remote %ss paths)�namesz.hgZhgrc�w+z[paths]
z
default = �
z
.update.lk)r) �url�hashrD �lockfilezhgfs/remote_map.txtz%d %b %Y %H:%M:%S.%fz# hgfs_remote map as of z{} = {}
rQ rP z Wrote new hgfs_remote map to %s)6�osr �joinr rJ rE �copy�deepcopy�
isinstance�dict�next�iterr r �dataZrepack_dictlist�itemsr r rC �getr �updaterP Z strip_proto� TypeError�getattr�hashlibr �encode� hexdigest�isdir�makedirs�listdir�hglib�init�close�openZ ServerError� ExceptionZconfig� CommandError�files�fopen�writer" r# �appendr �now�strftime�format�OSError�info)�bp_Z
new_remote�reposZper_remote_defaults�param�remoteZ repo_confZrepo_urlZper_remote_confr Zper_remote_errorsrL Z repo_hashZrp_�clientr) �exc�refsZ
hgconfpathZhgconfigZ
remote_map�fp_� timestampr r r rh � s
������
� � �
�
����
� ���� ��
���� �� � rh c C s2 t j�td d�} zt �| �}W n ty g }Y nw t� }|D ]}z |�|d � W q! ty5 Y q!w g }|D ]}|dv rAq:t j�| |�}t j� |�rS|�
|� q:g }|r�|D ].}zt � |� W n ty� } zt
�d|| � |�
|� W Y d} ~ qZd} ~ ww t
�d|� qZ|D ]}
|�|
� q�t|�|fS )zC
Remove cache directories for remotes no longer configured
rD r rQ )rQ r| z0Unable to remove old hgfs remote cachedir %s: %sNz hgfs removed old cachedir %s)rS r rT r rf rt rh �remove�
ValueErrorrd rp �shutil�rmtreer r �debug�bool) rv Z cachedir_lsrw r) � to_remove�itemr �failed�rdirr{ Zfdirr r r �_clear_old_remotes� sF � �
��� r� c C s� t j�td d�} t j�td d�}g }| |fD ]-}t j�|�rEzt�|� W q tyD } z|�d|� d|� �� W Y d}~qd}~ww q|S )z%
Completely clear hgfs cache
rD r �file_lists/hgfszUnable to delete z: N) rS r rT r �existsr� r� rt rp )Z fsb_cachedir�
list_cachedir�errorsr� r{ r r r � clear_cache� s "���r� c C �� dd� }t | t�r
|| �S g }g }t� D ]S}zJ| rIzt�|d | �s+W W |d �� qW n tyH t�|d t| ��sFY W |d �� qY nw ||�\}}|�|� |�|� W |d �� q|d �� w ||fS )z�
Clear update.lk
``remote`` can either be a dictionary containing repo configuration
information, or a pattern. If the latter, then remotes for which the URL
matches the pattern will be locked.
c
S s dd� }g }g }t j�| d �r�z t �| d � W nU tyn } zI|jtjkrIz t�| d � W n' tyH } z ||| |� W Y d }~n d }~ww ||| |� W Y d }~||fS W Y d }~||fS W Y d }~||fS d }~ww d� | d �}t
� |� |� |� ||fS )Nc S s. d� |d |d |�}t�|� | �|� d S )Nz-Unable to remove update lock for {} ({}): {} rP rR )rs r r� rp )Zerrlistr) r{ �msgr r r �
_add_error� s
�
z6clear_lock.<locals>._do_clear_lock.<locals>._add_errorrR zRemoved lock for {}rP )
rS r r� r rt �errnoZEISDIRr� r� rs r r� rp )r) r� �successr� r{ r� r r r �_do_clear_lock� s4 ��
�
���
z"clear_lock.<locals>._do_clear_lockrP r) �rW rX rh �fnmatchri r_ rE �extend)ry r� Zclearedr� r) r� r� r r r �
clear_lock� s0
� ��
r� c C r� )z�
Place an update.lk
``remote`` can either be a dictionary containing repo configuration
information, or a pattern. If the latter, then remotes for which the URL
matches the pattern will be locked.
c
S s� g }g }t j�| d �sdz tjj�| d d�� W d � n1 s"w Y W n* tyR } z d�| d | d |�}t �
|� |� |� W Y d }~||fS d }~ww d�| d �}t �
|� |� |� ||fS )NrR �wz*Unable to set update lock for {} ({}): {} rP zSet lock for {}) rS r r� r r rm rn rt rs r r� rp )r) r� r� r{ r� r r r �_do_lock
s* ���
��
zlock.<locals>._do_lockrP r) r� )ry r� �lockedr� r) r� r� r r r �lock s0
� ��
r� c
C s� ddd�} t � \| d< }|D ]�}z�tj�|d �r,t�d|d |d � W |d �� qt|�\}}|rDt�d |d � W |d �� qt� d
|d � |d �
� |d � � }z|d � � W n t
y~ } ztjd ||d tjd � W Y d
}~nd
}~ww |d � � }|d |d kr�d| d< W |d �� n|d �� w t|� qtj�td d�}| �dd�du s�tj�|�s�tj�|�} tj�| �s�t�| � tdd�}
tjj�|d��} | � tj � |
�� t� d|� W d
� n1 s�w Y t�dd��r*tjj j!dtd tdd��} | �"| t#ddgdd�� W d
� n 1 �s%w Y ztj$�%tj�td d �t&� W d
S t'�yF Y d
S w ) z0
Execute an hg pull on all of the repos
Fr )�changedZbackendr� rR z�Update lockfile is present for hgfs remote %s, skipping. If this warning persists, it is possible that the update process was interrupted. Removing %s or running 'salt-run fileserver.clear_lock hgfs' will allow updates to continue for this remote.rP r) z7Unable to set update lock for hgfs remote %s, skipping.zhgfs is fetching from %sz1Exception %s caught while updating hgfs remote %s)Zexc_info_on_loglevelNr TrD � hgfs/envs.p)� ignore_cachezwb+zWrote env cache data to %sZfileserver_eventsZmasterZsock_dir)�opts�listenr^ �
fileserver)�prefix� hgfs/hash)(r� rS r r� r �warningri r� r r� rj r7 Zpullrk �logging�DEBUGr� rT r r] �isfile�dirnamere r< r r rm rn ro �payload�dumpsZtrace�eventZ get_eventZ
fire_eventr r� Zreap_fileserver_cache_dir� find_filert )
r[ rw r) �_r� Zcurtipr{ Znewtip� env_cacheZ env_cachedirZnew_envsr} r� r r r r^ 5 s�
� ��� ��� �
�� �
��r^ c C s t jjj| td td d�S )zc
Check if an environment is exposed by comparing it against a whitelist and
blacklist.
Zhgfs_saltenv_whitelistZhgfs_saltenv_blacklist)Z whitelistZ blacklist)r r r" Zcheck_whitelist_blacklistr )�envr r r �_env_is_exposed� s
�r� c C s | st j�td d�}tj�t|�}|dur|S t� }t� D ]e}z\|d � � |d dv rGt
|d �D ]}|d }||d krAd}|� |� q3|d d v rgt |d �D ]}|d }||d krad}|� |� qS|�
d
d � t|d �D �� W |d �� q |d �� w d d � t|�D �S )
z@
Return a list of refs that can be used as environments
rD r� Nr) r )r r r r
)r r c S s g | ]}|d �qS r- r r$ r r r r' � s zenvs.<locals>.<listcomp>c S s g | ]}t |�r|�qS r )r� r$ r r r r' � s )rS r rT r r r� Zcheck_env_cache�setrh rj r* �addr5 r^ r: ri �sorted) r� r� � cache_match�retr) �branchZ branch_nameZbookmarkZ
bookmark_namer r r r< � s0
r< r
c
K s: ddd�}t j�| �s|t� vr|S t j�td d|| �}t j�td d|| � d��}t j�td d|| � d��}t j�td d|| � d��}t j�|�}t j�|�} t j�|�stzt �|� W n t ys t �
|� t �|� Y nw t j�| �s�zt �| � W n t y� t �
| � t �| � Y nw t � D �]�}
�zw|
d r�| � |
d t jj
�s�W |
d
�� q�| t|
d �d � �t jj
�} |
d r�t j�|
d | �} |
d
�� t|
|�} | s�|
d
�� W |
d
�� q�tj�||� t j�|��rDt j�|��rDtjj�|d
��3}
|
�� }|| d k�r4| |d< ||d< |
d
�� |W d � W |
d
�� S W d � n 1 �s?w Y z|
d
jtjj � d| � ��g| d |d� W n t j j �yt |
d
�� Y W |
d
�� q�w tjj�|d�� W d � n 1 �s�w Y t!�!|�D ]}zt �
|� W �q� t"�y� Y �q�w tjj�|d��}
|
�#tjj �$| d �� W d � n 1 �s�w Y zt �
|� W n
t �y� Y nw | |d< ||d< z t%t �&|��|d< W n
t"�y� Y n w W |
d
�� | S W |
d
�� | S |
d
�� w |S )z
Find the first file to match the path and ref, read the file out of hg
and send the path to the newly cached file
� )r �relrD z hgfs/refsr� z.hash.*z.hash.blob_sha1z.lkr r) Nr
�rr r� r zpath:)r? �outputr� rN �stat)'rS r �isabsr< rT r r� rd re rt r rh �
startswith�sepri �len�lstriprj r= r r� Z wait_lockr� r rm rn �read�catr" �to_bytesrg r rl �globrk ro r# �listr� )r Ztgt_env�kwargs�fnd�destZ hashes_globZ blobshadestZlk_fnZdestdirZhashdirr) Z repo_pathrA r} Zsha�filenamer r r r� � s�
��
�
� �= �
3� �0� �
� !� �� ��
�� � �r� c s d� v r � � d� ddd�}t� fdd�dD ��s|S |d s!|S |d |d < � �d
d �}tj�|d �}tjj� |d ��9}|�
� d
� |� t d �}|rZtjj�
|�sZ|�t�}|rj|rjtjj�||�}||d
< ||d< W d � |S 1 syw Y |S )z?
Return a chunk from a file based on the data received
r� r� )r[ r� c 3 � � | ]}|� v V qd S rI r r$ ��loadr r rK � � z serve_file.<locals>.<genexpr>)r �loc�saltenvr r� r� �gzipN�rbr� Zfile_buffer_sizer[ )�pop�allr] rS r �normpathr r rm rn �seekr� r Z is_binary�decodeZ__salt_system_encoding__Z gzip_util�compress)r� r� r� r� Zfpathr} r[ r r� r �
serve_file s.
� � r� c
s" d� v r � � d� t� fdd�dD ��sdS dtd i}|d }|d }tj�td d
� d d �|td ��}tj�|�sktj j
� |td �|d
< tj j �
|d��}|�|d
� W d � |S 1 sdw Y |S tj j �
|d��}tj j�|�� �|d
< W d � |S 1 s�w Y |S )zL
Return a file hash, the hash type is set in the master config file
r� c 3 r� rI r r$ r� r r rK ; r� z file_hash.<locals>.<genexpr>)r r� r� rL r� r rD r� r� z
{}.hash.{}ZhsumrN Nr� )r� r� r rS r rT rs r� r r Z hashutilsZget_hashrm rn ro r" Z
to_unicoder� )r� r� r� �relpathr Zhashdestr} r r� r � file_hash3 s6
�
��
��r� c C s� d| v r | � d� tj�td d�}tj�|�s1zt�|� W n ty0 t� d|� g Y S w tj�|d�
| d ��}tj�|d�
| d ��}t j �
t|||�\}}}|dur[|S |r|i }t| �|d < t| �|d
< |rvt j �t|||� |�|g �S g S ) zD
Return a dict containing the file lists for files and dirs
r� rD r� zUnable to make cachedir %sz{}.pr� z.{}.wNrm �dirs)r� rS r rT r rd re rt r �criticalrs r r� Zcheck_file_list_cache�_get_file_list�
_get_dir_listZwrite_file_list_cacher] ) r� Zformr� Z
list_cacheZw_lockr� Z
refresh_cacheZ
save_cacher� r r r � _file_listsQ s2
�
� r� c C �
t | d�S )zR
Return a list of all files on the file server in a specified environment
rm �r� r� r r r � file_listr �
r� c C s� d| v r | � d� d| vs| d t� vrg S t� }t� D ]I}z@|d �� t|| d �}|rWt|d |d�}|D ] }tj� |d |d �}|�
d�sV|� tj� |d |�� q8W |d �
� q |d �
� w t|�S ) zO
Get a list of all files on the file server in a specified environment
r� r� r) �rA r> r
�../r )r� r< r� rh rj r= rB rS r r� r� r� rT ri r� )r� r� r) rA r@ �tupr� r r r r� y s$
� r� c C s g S )z>
Return a list of all empty directories on the master
r r� r r r �file_list_emptydirs� s r� c C r� )z8
Return a list of all directories on the master
r� r� r� r r r �dir_list� r� r� c C s. d| v r | � d� d| vs| d t� vrg S t� }t� D ]k}zb|d �� t|| d �}|ryt|d |d�}|D ]@}|d }|�dd�}t|�dkrxt j
� |d |d �}|d
krj|� d �sj|�
t j
�|d |�� |d �dd�}t|�dksJq8W |d �� q |d �� w |d r�|�
|d � t|�S )
z5
Get a list of all directories on the master
r� r� r) r� r> �/r r r
�.r� r )r� r< r� rh rj r= rB �rsplitr� rS r r� r� r� rT ri r� ) r� r� r) rA r@ r� �filepath�splitr� r r r r� � s4
�� r� rI )F)r
)>�__doc__rU r� r� r� ra r� rS r� r Zsalt.fileserverr Zsalt.utils.dataZsalt.utils.filesZsalt.utils.gzip_utilZsalt.utils.hashutilsZsalt.utils.stringutilsZsalt.utils.urlZsalt.utils.versionsZ salt.configr Zsalt.exceptionsr Zsalt.utils.eventr r rJ rg r � ImportError� getLogger�__name__r r Z__virtual_aliases__r r* r3 r5 r6 r: r; r= rB rC rh r� r� r� r� r^ r� r< r� r� r� r� r� r� r� r� r� r r r r �<module> sx % �
&&
>4M
i !