File: //opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/__pycache__/svnfs.cpython-310.pyc
o
�N�g2d � @ s� d Z ddlZddlZddlZddlZddlZddlZddlZddlmZ ddl Z
ddl Z
ddl Z
ddl
Z
ddlZ
ddlZ
ddlZ
ddlZ
ddlZ
ddlmZ ddlmZ ddlmZ dZdZz ddlZd Ze� � Z W n e ys Y nw e� e �Z!d
Z"d Z#d d
� Z$dd� Z%dd� Z&dd� Z'dd� Z(dd� Z)d3dd�Z*d3dd�Z+d d � Z,d d � Z-d4d d!�Z.d"d#� Z/d5d%d&�Z0d'd(� Z1d)d*� Z2d+d,� Z3d-d.� Z4d/d0� Z5d1d2� Z6dS )6a
Subversion Fileserver Backend
After enabling this backend, branches and tags in a remote subversion
repository are exposed to salt as different environments. To enable this
backend, add ``svnfs`` to the :conf_master:`fileserver_backend` option in the
Master config file.
.. code-block:: yaml
fileserver_backend:
- svnfs
.. note::
``svn`` also works here. Prior to the 2018.3.0 release, *only* ``svn``
would work.
This backend assumes a standard svn layout with directories for ``branches``,
``tags``, and ``trunk``, at the repository root.
:depends: - subversion
- pysvn
.. versionchanged:: 2014.7.0
The paths to the trunk, branches, and tags have been made configurable, via
the config options :conf_master:`svnfs_trunk`,
:conf_master:`svnfs_branches`, and :conf_master:`svnfs_tags`.
:conf_master:`svnfs_mountpoint` was also added. Finally, support for
per-remote configuration parameters was added. See the
:conf_master:`documentation <svnfs_remotes>` for more information.
� N)�datetime)�DEFAULT_HASH_TYPE��FileserverConfigError)�tagify)�
mountpoint�root�trunk�branches�tagsFT�svnfs)�svnc C s� t td vrdS tst�d� dS g } dD ]}tj�t| �r*| �d� |t| �� q| r>| D ]}t�|� q/t�d� dS t S )z.
Only load if subversion is available
Zfileserver_backendFzfSubversion fileserver backend is enabled in configuration but could not be loaded, is pysvn installed?)Z svnfs_trunkZsvnfs_branchesZ
svnfs_tagszJMaster configuration parameter '{}' (value: {}) cannot be an absolute pathz-Subversion fileserver backed will be disabled)
�__virtualname__�__opts__�HAS_SVN�log�error�os�path�isabs�append�format)�errors�paramr � r �I/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/svnfs.py� __virtual__M s* � ��
r c C sp z
t t�| d ��� �}W n% tjjttt fy2 } zt
� d| d | d |� W Y d}~dS d}~ww |d j S )z%
Returns revision ID of repo
�repozCError retrieving revision ID for svnfs remote %s (cachedir: %s): %s�urlN�revision)
�dict�CLIENT�info�items�pysvn�_pysvn� ClientError� TypeError�KeyError�AttributeErrorr r �number)r Z repo_info�excr r r �_revh s ���
r, c C s t d��)zB
Fatal fileserver configuration issue, raise an exception
z%Failed to load svn fileserver backendr r r r r � _failhardz s r- c C s t j�td d�} d}g }i }tD ]
}ttd|� � �||< qtd D �]}t�|�}t|t �rut
t |��}dd� t j
j�|| ��� D �}|sQt�d|� t� d} d d
� |D �D ]}t�d ||d �t�� d
} qZ| rot� |�|� n|}t|t�s�t�d|� t� z
t j
j�|d �|d< W n ty� Y nw ttt�dt��}
|
|�� � } t j�| | �} t j� | �s�t � | � t � | �s�zt �!|| � |�"| � d
}W nE t#j$j%y� }
zt�d||
� t� W Y d}
~
n,d}
~
ww zt �&| � W n t#j$j%�y }
zt�d| |� t� W Y d}
~
nd}
~
ww |�| || | t j�| d�d�� |�"|� q#|�r�t j�td d�}zCt j
j'�(|d��0}t)�*� �+d�}|�,d|� d�� |D ]}|�,t j
j-�.d�/|d |d ��� �qUW d � n 1 �svw Y W n t0�y� Y |S w t�1d |� |S ) zL
Return the list of svn remotes and their configuration information
�cachedirr FZsvnfs_Z
svnfs_remotesc S s i | ] \}}|t |��qS r )�str)�.0�key�valr r r �
<dictcomp>� s ��zinit.<locals>.<dictcomp>z�Invalid per-remote configuration for remote %s. If no per-remote parameters are being specified, there may be a trailing colon after the URL, which should be removed. Check the master configuration file.c s s � | ] }|t vr|V qd S �N)�PER_REMOTE_OVERRIDES�r0 �xr r r � <genexpr>� s � zinit.<locals>.<genexpr>z|Invalid configuration parameter '%s' for remote %s. Valid parameters are: %s. See the documentation for further information.z, Tz[Invalid svnfs remote %s. Remotes must be strings, you may need to enclose the URL in quotesr � hash_typez*Failed to initialize svnfs remote '%s': %sNz�Cache path %s (corresponding remote: %s) exists but is not a valid subversion checkout. You will need to manually delete this directory on the master to continue to use this svnfs remote.z update.lk)r r �hashr. �lockfilezsvnfs/remote_map.txt�w+z%d %b %Y %H:%M:%S.%fz# svnfs_remote map as of �
z{} = {}
r: r z Wrote new svnfs_remote map to %s)2r r �joinr r5 r/ �copy�deepcopy�
isinstancer �next�iter�salt�utils�dataZrepack_dictlistr# r r r- �updater Z strip_protor' �getattr�hashlib�getr � hexdigest�isdir�makedirs�listdirr! Zcheckoutr r$ r% r& �status�files�fopenr �now�strftime�write� stringutilsZto_strr �OSErrorr" )�bp_Z
new_remote�reposZper_remote_defaultsr �remoteZ repo_confZrepo_urlZper_remote_confZper_remote_errorsr9 Z repo_hashZrp_r+ Z
remote_map�fp_� timestampr r r �init� s�
����
� � �
����� �� ��� �� � r\ c C s2 t j�td d�} zt �| �}W n ty g }Y nw t� }|D ]}z |�|d � W q! ty5 Y q!w g }|D ]}|dv rAq:t j�| |�}t j� |�rS|�
|� q:g }|r�|D ].}zt � |� W n ty� } zt
�d|| � |�
|� W Y d} ~ qZd} ~ ww t
�d|� qZ|D ]}
|�|
� q�t|�|fS )zC
Remove cache directories for remotes no longer configured
r. r r: )r: Zrefsz1Unable to remove old svnfs remote cachedir %s: %sNz svnfs removed old cachedir %s)r r r> r rN rV r\ �remove�
ValueErrorrL r �shutil�rmtreer r �debug�bool) rW Z cachedir_lsrX r � to_remove�itemr �failed�rdirr+ Zfdirr r r �_clear_old_remotes sJ � �
���� rg c C s� t j�td d�} t j�td d�}g }| |fD ]-}t j�|�rEzt�|� W q tyD } z|�d|� d|� �� W Y d}~qd}~ww q|S )z&
Completely clear svnfs cache
r. r �file_lists/svnfszUnable to delete z: N) r r r> r �existsr_ r` rV r )Z fsb_cachedir�
list_cachedirr rf r+ r r r � clear_cache) s "���rk c C s� dd� }t | t�r
|| �S g }g }t� D ]2}| r6z
| |d vr!W qW n ty5 t| �|d vr3Y qY nw ||�\}}|�|� |�|� q||fS )z�
Clear update.lk
``remote`` can either be a dictionary containing repo configuration
information, or a pattern. If the latter, then remotes for which the URL
matches the pattern will be locked.
c
S s dd� }g }g }t j�| d �r�z t �| d � W nU tyn } zI|jtjkrIz t�| d � W n' tyH } z ||| |� W Y d }~n d }~ww ||| |� W Y d }~||fS W Y d }~||fS W Y d }~||fS d }~ww d� | d �}t
� |� |� |� ||fS )Nc S s. d� |d |d |�}t�|� | �|� d S )Nz-Unable to remove update lock for {} ({}): {} r r; )r r ra r )Zerrlistr r+ �msgr r r �
_add_errorC s
�
z6clear_lock.<locals>._do_clear_lock.<locals>._add_errorr; zRemoved lock for {}r )
r r ri r] rV �errnoZEISDIRr_ r` r r ra r )r rm �successre r+ rl r r r �_do_clear_lockB s4 ��
�
���
z"clear_lock.<locals>._do_clear_lockr )rA r r\ r' r/ �extend)rY rp Zclearedr r ro re r r r �
clear_lock9 s(
� ��
rr c C s� dd� }t | t�r
|| �S g }g }t� D ]6}| r:z t�|d | �s#W qW n ty9 t�|d t| ��s7Y qY nw ||�\}}|�|� |�|� q||fS )z�
Place an update.lk
``remote`` can either be a dictionary containing repo configuration
information, or a pattern. If the latter, then remotes for which the URL
matches the pattern will be locked.
c
S s� g }g }t j�| d �siz!tjj�| d d��
}|�d� W d � n1 s'w Y W n* tyW } z d� | d | d |�}t
� |� |� |� W Y d }~||fS d }~ww d� | d �}t
� |� |� |� ||fS )Nr; r< � z*Unable to set update lock for {} ({}): {} r zSet lock for {})
r r ri rD rE rP rQ rT rV r r ra r )r ro re rZ r+ rl r r r �_do_lock} s* ���
��
zlock.<locals>._do_lockr )rA r r\ �fnmatchr' r/ rq )rY rt �lockedr r ro re r r r �lockt s(
� ��
rw c
C sT ddd�} t � \| d< }|D ]y}tj�|d �r$t�d|d |d � qt|�\}}|r5t�d|d � qt�d |d � t |�}z t
� |d
� W n t j
jyj } zt�d |d |d |� W Y d
}~nd
}~ww t |�}tdd� ||fD ��r{q||kr�d| d< t|� qtj�td d�}| �dd�du s�tj�|�s�tj�|�} tj�| �s�t�| � tdd�}
tjj�|d��} | � tj � |
�� t� d|� W d
� n1 s�w Y t�dd��r
tjj j!dtd tdd��} | �"| t#ddgdd�� W d
� n 1 �sw Y ztj$�%tj�td d �t&� W d
S t'�y) Y d
S w ) z3
Execute an svn update on all of the repos
Fr )�changedZbackendrx r; z�Update lockfile is present for svnfs remote %s, skipping. If this warning persists, it is possible that the update process was interrupted. Removing %s or running 'salt-run fileserver.clear_lock svnfs' will allow updates to continue for this remote.r z8Unable to set update lock for svnfs remote %s, skipping.zsvnfs is fetching from %sr z1Error updating svnfs remote %s (cachedir: %s): %sr. Nc s s � | ]}|d u V qd S r4 r r6 r r r r8 � � � zupdate.<locals>.<genexpr>T� svnfs/envs.p)� ignore_cachezwb+zWrote env cache data to %sZfileserver_eventsZmasterZsock_dir)�opts�listenrG �
fileserver)�prefixz
svnfs/hash)(rg r r ri r �warningrw r ra r, r! rG r$ r% r& �anyrr r> r rJ �isfile�dirnamerM �envsrD rE rP rQ rT �payload�dumpsZtrace�eventZ get_eventZ
fire_eventr r~ Zreap_fileserver_cache_dir� find_filerV )
rF rX r �_r Zold_revr+ Znew_rev� env_cacheZ env_cachedirZnew_envsrZ r� r r r rG � s|
� ����
�� �
��rG c C s t jjj| td td d�S )zc
Check if an environment is exposed by comparing it against a whitelist and
blacklist.
Zsvnfs_saltenv_whitelistZsvnfs_saltenv_blacklist)Z whitelistZ blacklist)rD rE rU Zcheck_whitelist_blacklistr )�envr r r �_env_is_exposed� s
�r� c C s, | st j�td d�}tj�t|�}|dur|S t� }t� D ]n}t j�|d |d �}t j� |�r7|�
d� n t � d|d |d � t j�|d |d �}t j� |�r\|�
t �|�� n t � d
|d |d � t j�|d |d �}t j� |�r�|�
t �|�� q t � d |d |d � q d
d� t|�D �S )z@
Return a list of refs that can be used as environments
r. rz Nr r �basezdsvnfs trunk path '%s' does not exist in repo %s, no base environment will be provided by this remoter r
z2svnfs branches path '%s' does not exist in repo %sr z.svnfs tags path '%s' does not exist in repo %sc S s g | ]}t |�r|�qS r )r� r6 r r r �
<listcomp>' s zenvs.<locals>.<listcomp>)r r r> r rD r~ Zcheck_env_cache�setr\ rL �addr r rG rN �sorted)r{ r� � cache_match�retr r r
r r r r r� � s@
� � �r� c C s� |dkrt j�| d | d �}t j�|�r|S dS t j�| d | d �}t j�|�r8|t �|�v r8t j�||�S t j�| d | d �}t j�|�rW|t �|�v rWt j�||�S dS )z�
Return the root of the directory corresponding to the desired environment,
or None if the environment was not found.
r� r r Nr
r )r r r> rL rN )r �saltenvr r
r r r r � _env_root* s r� r� c K s ddd�}t j�| �s|t� vr|S t� D ]h}t||�}|du r!q|d r1| �|d t jj �s1q| t|d �d� � t jj�}|d rMt j�
|d |�}t j�
||�}t j� |�r}| |d< ||d< zt t �
|��|d< W | S ty| Y | S w q|S ) z�
Find the first file to match the path and ref. This operates similarly to
the roots file sever but with assumptions of the directory structure
based on svn standard practices.
rs )r �relNr r r� r �stat)r r r r� r\ r� �
startswith�sep�len�lstripr> r� �listr� � Exception)r Ztgt_env�kwargs�fndr �env_rootZ repo_path�fullr r r r� D s2
���r� c s d� v r � � d� ddd�}t� fdd�dD ��s|S |d s!|S |d |d < � �d
d �}tj�|d �}tjj� |d ��9}|�
� d
� |� t d �}|rZtjj�
|�sZ|�t�}|rj|rjtjj�||�}||d
< ||d< W d � |S 1 syw Y |S )z?
Return a chunk from a file based on the data received
r� rs )rF �destc 3 � � | ]}|� v V qd S r4 r r6 ��loadr r r8 z ry z serve_file.<locals>.<genexpr>)r �locr� r r� r� �gzipN�rbr� Zfile_buffer_sizerF )�pop�allrJ r r �normpathrD rE rP rQ �seek�readr Z is_binary�decodeZ__salt_system_encoding__Z gzip_util�compress)r� r� r� r� ZfpathrZ rF r r� r �
serve_fileq s.
� � r� c s� d� v r � � d� t� fdd�dD ��sdS � d }|dkr d}i }|d }|d
}|r2tj�|�s4|S td |d < tj�td d
d|d�|td ��}tj�|�r�t j
j � |d��&}|�
� �d�\}} tj�|�| kry||d< |W d � S W d � n1 s�w Y t j
j�|td �|d< tj�|�}
tj�|
�s�t�|
� t j
j � |d��}|�d�|d tj�|��� W d � |S 1 s�w Y |S )zL
Return a file hash, the hash type is set in the master config file
r� c 3 r� r4 r r6 r� r r r8 � ry z file_hash.<locals>.<genexpr>)r r� rs r� r� r r� r r9 r. r r: z
{}.hash.{}r� �:�hsumN�wz{}:{})r� r� r r r� r r> r ri rD rE rP rQ r� �split�getmtimeZ hashutilsZget_hashr� rM rT ) r� r� r� r� �relpathr �
cache_pathrZ r� �mtimeZ cache_dirr r� r � file_hash� sN
� � �
��r� c C s@ d| v r | � d� d| vs| d t� vrg S tj�td d�}tj�|�s>zt�|� W n ty= t �
d|� g Y S w tj�|d� | d ��}tj�|d� | d ��}t j
�t|||�\}}}|durh|S |�r t� t� t� d �}t� D ]y} t| | d �}
|
du r�qw| d
r�tj�|
| d
��tjj�}
tj�|
�s�qwt jj�|
�D ]J\} } }
tj�| |
�}tj�| d |�}|d kr�|d
�|� | s�|
s�|d �|� |
D ] }tj�tj�| |�|
�}|d �tj�| d |�� q�q�qw| d r�|d
�| d � |D ] }t|| �||< �q |�rt j
�t|||� |�|g �S g S )zY
Return a dict containing the file lists for files, dirs, emptydirs and symlinks
r� r� r. rh zUnable to make cachedir %sz{}.pz.{}.wN)rP �dirs�
empty_dirsr r �.r� r� rP )r� r� r r r> r rL rM rV r �criticalr rD r~ Zcheck_file_list_cacher� r\ r� �rstripr� rE Zos_walkr� r� r� Zwrite_file_list_cacherJ )r� Zformrj Z
list_cacheZw_lockr� Z
refresh_cacheZ
save_cacher� r r� r r� rP r� Z
dir_rel_fn�fnameZrel_fnr1 r r r � _file_lists� sb
�
�
��
r� c C �
t | d�S )zV
Return a list of all files on the file server in a specified
environment
rP �r� r� r r r � file_list s
r� c C r� )z>
Return a list of all empty directories on the master
r� r� r� r r r �file_list_emptydirs �
r� c C r� )z8
Return a list of all directories on the master
r� r� r� r r r �dir_list r� r� r4 )F)r� )7�__doc__r? rn ru rI �loggingr r_ r Zsalt.fileserverrD Zsalt.utils.dataZsalt.utils.filesZsalt.utils.gzip_utilZsalt.utils.hashutilsZsalt.utils.pathZsalt.utils.stringutilsZsalt.utils.urlZsalt.utils.versionsZ salt.configr Zsalt.exceptionsr Zsalt.utils.eventr r5 r r$ ZClientr! � ImportError� getLogger�__name__r r Z__virtual_aliases__r r, r- r\ rg rk rr rw rG r� r� r� r� r� r� r� r� r� r� r r r r �<module> sj �
(
;1L
-
- 7<