HEX

File: //opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/__pycache__/azureblob.cpython-310.pyc
o

�N�g}7�@s�dZddlZddlZddlZddlZddlmZddlZddl	Zddl
mZdZz
ddl
mZdZWn	ey;YnwdZe�e�Zd	d
�Z				d!d
d�Zdd�Zdd�Zdd�Zdd�Z	d"dd�Zdd�Zdd�Zdd�Zdd �ZdS)#a�
Use Azure Blob as a Pillar source.

.. versionadded:: 3001

:maintainer: <devops@eitr.tech>
:maturity: new
:depends:
    * `azure-storage-blob <https://pypi.org/project/azure-storage-blob/>`_ >= 12.0.0

The Azure Blob ext_pillar can be configured with the following parameters:

.. code-block:: yaml

    ext_pillar:
      - azureblob:
          container: 'test_container'
          connection_string: 'connection_string'
          multiple_env: False
          environment: 'base'
          blob_cache_expire: 30
          blob_sync_on_update: True

:param container: The name of the target Azure Blob Container.

:param connection_string: The connection string to use to access the specified Azure Blob Container.

:param multiple_env: Specifies whether the pillar should interpret top level folders as pillar environments.
    Defaults to false.

:param environment: Specifies which environment the container represents when in single environment mode. Defaults
    to 'base' and is ignored if multiple_env is set as True.

:param blob_cache_expire: Specifies expiration time of the Azure Blob metadata cache file. Defaults to 30s.

:param blob_sync_on_update: Specifies if the cache is synced on update. Defaults to True.

�N)�deepcopy)�PillarF)�BlobServiceClientT�	azureblobcCstsdStS)N)FznThe following dependency is required to use the Azure Blob ext_pillar: Microsoft Azure Storage Blob >= 12.0.0 )�HAS_LIBS�__virtualname__�rr�I/opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/azureblob.py�__virtual__Csr
�base�c
Cs&tj�tj�t�||��}td�|g�|gkriSt|||||�}	t�	d|	�|rdt�
d�|	��D]+\}
}t|���D] \}}|D]}
t
||
|
�}t�
d||
|
�t||	|
||
|�qCq=q3t�
d�tt�}|rrtj�||�gn|g|d|<dd�|dD�|d<t|t||�}|jd	d
�}|S)a�
    Execute a command and read the output as YAML.

    :param container: The name of the target Azure Blob Container.

    :param connection_string: The connection string to use to access the specified Azure Blob Container.

    :param multiple_env: Specifies whether the pillar should interpret top level folders as pillar environments.
        Defaults to false.

    :param environment: Specifies which environment the container represents when in single environment mode. Defaults
        to 'base' and is ignored if multiple_env is set as True.

    :param blob_cache_expire: Specifies expiration time of the Azure Blob metadata cache file. Defaults to 30s.

    :param blob_sync_on_update: Specifies if the cache is synced on update. Defaults to True.

    Zpillar_rootszBlob metadata: %sz-Syncing local pillar cache from Azure Blob...z%s - %s : %sz2Sync local pillar cache from Azure Blob completed.cSsg|]}d|vr|�qS)rr)�.0�xrrr	�
<listcomp>�szext_pillar.<locals>.<listcomp>�
ext_pillarF)�ext)�os�path�normpath�join�_get_cache_dir�__opts__�get�_init�log�debug�info�items�_find_files�_get_cached_file_name�_get_file_from_blobrrZ
__grains__Zcompile_pillar)Z	minion_idZpillar�	container�connection_string�multiple_env�environment�blob_cache_expireZblob_sync_on_updateZ
pillar_dir�metadata�saltenv�env_meta�files�	file_path�cached_file_path�optsZpilZcompiled_pillarrrr	rNsH�
�
����

�rc
Cs�t|�}t��|}tj�|�rtj�|�}nd}||k}t�d||r%dnd|||�|r7t|||||�}	nt	|�}	t�d|	�|	S)a
    .. versionadded:: 3001

    Connect to Blob Storage and download the metadata for each file in all containers specified and
        cache the data to disk.

    :param connection_string: The connection string to use to access the specified Azure Blob Container.

    :param container: The name of the target Azure Blob Container.

    :param multiple_env: Specifies whether the pillar should interpret top level folders as pillar environments.
        Defaults to false.

    :param environment: Specifies which environment the container represents when in single environment mode. Defaults
        to 'base' and is ignored if multiple_env is set as True.

    :param blob_cache_expire: Specifies expiration time of the Azure Blob metadata cache file. Defaults to 30s.

    rzQBlob storage container cache file %s is %sexpired, mtime_diff=%ss, expiration=%ss�znot z#Blob container retrieved pillars %s)
�_get_containers_cache_filename�timerr�isfile�getmtimerr�_refresh_containers_cache_file�_read_containers_cache_file)
r"r!r#r$r%�
cache_file�expZcache_file_mtimeZexpiredZpillarsrrr	r�s(
�	
�rcCs6tj�tdd�}tj�|�st�d�t�|�|S)zf
    .. versionadded:: 3001

    Get pillar cache directory. Initialize it if it does not exist.

    ZcachedirZpillar_azureblobz$Initializing Azure Blob Pillar Cache)rrrr�isdirrr�makedirs)�	cache_dirrrr	r�s


rcCs>tj�t�|||�}tj�tj�|��st�tj�|��|S)a%
    .. versionadded:: 3001

    Return the cached file name for a container path file.

    :param container: The name of the target Azure Blob Container.

    :param saltenv: Specifies which environment the container represents.

    :param path: The path of the file in the container.

    )rrrr�exists�dirnamer7)r!r'rr*rrr	r�s
rcCs0t�}tj�|�st�|�tj�||�d��S)z�
    .. versionadded:: 3001

    Return the filename of the cache for container contents. Create the path if it does not exist.

    :param container: The name of the target Azure Blob Container.

    z-files.cache)rrrr9r7r)r!r8rrr	r.�s	
r.c
s�zt�|�}|����Wnty%}z
t�d|�WYd}~dSd}~wwi�d
�����fdd�	��|d�tj�|�rCt�	|�t�
d�tjj
�|d	��}t��|�Wd��S1sbwY�S)a�
    .. versionadded:: 3001

    Downloads the entire contents of an Azure storage container to the local filesystem.

    :param connection_string: The connection string to use to access the specified Azure Blob Container.

    :param container: The name of the target Azure Blob Container.

    :param cache_file: The path of where the file will be cached.

    :param multiple_env: Specifies whether the pillar should interpret top level folders as pillar environments.

    :param environment: Specifies which environment the container represents when in single environment mode. This is
        ignored if multiple_env is set as True.

    �
Exception: %sNFrcs��j|d�}	zt|�}Wn
tyYdSwt�d|�|j�d�r+�|jd�q�r;|r3|dkr5dn|dd�}|�vrCi�|<��|vrOg�|�<�|��|�q)	N)Zname_starts_withTzRaw blob attributes: %s�/)�prefix�.r���)Z
walk_blobs�next�
StopIterationrr�name�endswith�append)r'r=Z	blob_listZblob��_walk_blobsr!�container_clientr&r#rr	rF*s&��z3_refresh_containers_cache_file.<locals>._walk_blobs)r'z%Writing Azure blobs pillar cache file�wb)rN)r�from_connection_string�get_container_client�	Exceptionr�errorrrr0�remover�salt�utilsr)�fopen�pickle�dump)r"r!r4r#r$�blob_service_client�exc�fp_rrEr	r2s,���



��r2cCsJt�d�tjj�|d��}t�|�}Wd�|S1swY|S)z�
    .. versionadded:: 3001

    Return the contents of the containers cache file.

    :param cache_file: The path for where the file will be cached.

    zReading containers cache file�rbN)rrrNrOr)rPrQ�load)r4rU�datarrr	r3Vs
	
��r3cCsRi}|��D] \}}||vrg||<dd�|D�}||dd�|D�7<q|S)z�
    .. versionadded:: 3001

    Looks for all the files in the Azure Blob container cache metadata.

    :param metadata: The metadata for the container files.

    cSsg|]}|d�qS)rBr�r
�krrr	rwsz_find_files.<locals>.<listcomp>cSsg|]	}|�d�s|�qS)r<)rCrYrrr	rys)r)r&�retr!rXZ
file_pathsrrr	rgs	rcCsZ||vr||ni}||vr||ni}|D]}t|�}d|vr*|d|kr*|SqdS)an
    .. versionadded:: 3001

    Looks for a file's metadata in the Azure Blob Container cache file.

    :param metadata: The metadata for the container files.

    :param container: The name of the target Azure Blob Container.

    :param saltenv: Specifies which environment the container represents.

    :param path: The path of the file in the container.

    rBN)�dict)r&r!r'rr(Zcontainer_metaZ	item_metarrr	�_find_file_meta~s��r]c
Cstj�|�r4t||||�}|rd�tttj|d���nd}t	j
j�|d�}t
�d|||�||kr4dSzt�|�}	|	�|�}
|
�|�}Wnty^}z
t
�d|�WYd}~dSd}~wwt	j
j�|d��}
|
�|�����Wd�dS1s|wYdS)	a�
    .. versionadded:: 3001

    Downloads the entire contents of an Azure storage container to the local filesystem.

    :param connection_string: The connection string to use to access the specified Azure Blob Container.

    :param metadata: The metadata for the container files.

    :param saltenv: Specifies which environment the container represents when in single environment mode. This is
        ignored if multiple_env is set as True.

    :param container: The name of the target Azure Blob Container.

    :param path: The path of the file in the container.

    :param cached_file_path: The path of where the file will be cached.

    r-ZetagN�md5z%Cached file: path=%s, md5=%s, etag=%sr;FrH)rrr0r]r�list�filter�str�isalnumrNrOZ	hashutilsZget_hashrrrrIrJZget_blob_clientrKrLr)rP�writeZ
download_blob�readall)r"r&r'r!rr+Z	file_metaZfile_md5Z
cached_md5rSrGZblob_clientrTZoutfilerrr	r �s< ���
��
��r )FrrT)Fr)�__doc__�loggingrrQr/�copyrZsalt.utils.filesrNZsalt.utils.hashutilsZsalt.pillarrrZazure.storage.blobr�ImportErrorr�	getLogger�__name__rr
rrrrr.r2r3rr]r rrrr	�<module>sD'�

�O5
�N