o

�N�g�j�@s>
dZd
dl
Z
d
dlZd
dlZd
dlZd
dlZd
dlZd
dlm	Z

d
dlZd
dl
Zd
dlZd
dlZd
dlZe�e�
ZdZdd�Zdd�Zd6d	d
�
Zdd�Zd
d�Zdd�Zdd�Zdd�Zdd�Zdd�Zdd�Zdd�Z dd�Z!dd �Z"d!d"�Z#d#d$�Z$d%d&�Z%d'd(�Z&d)d*�Z'd+d,�Z(d-d.�Z)d/d0�Z*d7d2d3�
Z+d4d5�Z,dS)8aj	
Amazon S3 Fileserver Backend

.. versionadded:: 0.16.0

This backend exposes directories in S3 buckets as Salt environments. To enable
this backend, add ``s3fs`` to the :conf_master:`fileserver_backend` option in the
Master config file.

.. code-block:: yaml

    fileserver_backend:
      - s3fs

S3 credentials must also be set in the master config file:

.. code-block:: yaml

    s3.keyid: GKTADJGHEIQSXMKKRBJ08H
    s3.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs

Alternatively, if on EC2 these credentials can be automatically loaded from
instance metadata.

This fileserver supports two modes of operation for the buckets:

1. :strong:`A single bucket per environment`

   .. code-block:: yaml

    s3.buckets:
      production:
        - bucket1
        - bucket2
      staging:
        - bucket3
        - bucket4

2. :strong:`Multiple environments per bucket`

   .. code-block:: yaml

    s3.buckets:
      - bucket1
      - bucket2
      - bucket3
      - bucket4

Note that bucket names must be all lowercase both in the AWS console and in
Salt, otherwise you may encounter ``SignatureDoesNotMatch`` errors.

A multiple-environment bucket must adhere to the following root directory
structure::

    s3://<bucket name>/<environment>/<files>

.. note:: This fileserver back-end requires the use of the MD5 hashing algorithm.
    MD5 may not be compliant with all security policies.

.. note:: This fileserver back-end is only compatible with MD5 ETag hashes in
    the S3 metadata. This means that you must use SSE-S3 or plaintext for
    bucket encryption, and that you must not use multipart upload when
    uploading to your bucket. More information here:
    https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html

    Objects without an MD5 ETag will be fetched on every fileserver update.

    If you deal with objects greater than 8MB, then you should use the
    following AWS CLI config to avoid mutipart upload:

    .. code-block:: text

        s3 =
          multipart_threshold = 1024MB

    More info here:
    https://docs.aws.amazon.com/cli/latest/topic/s3-config.html

.. note:: This fileserver back-end will by default sync all buckets on every
    fileserver update.

    If you want files to be only populated in the cache when requested, you can
    disable this in the master config:

    .. code-block:: yaml

        s3.s3_sync_on_update: False
�N�md5c
Cst�}t
|���
S)
z^
    Return a list of directories within the bucket that can be
    used as environments.
    )�_init�list�keys)
�metadata�r�H/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/s3fs.py�envsms
r	c
Cs�t�}t
�d
d�rIt�d�

|��D]/\}
}t|�
D]&}|��D]\}}|D]}t||
|�}t�d||
|�
t	||
|||�
q&q qqt�d�

dSdS)z/
    Update the cache file for the bucket.
    zs3.s3_sync_on_updateTzSyncing local cache from S3...z%s - %s : %sz#Sync local cache from S3 completed.N)
r�__opts__�get�log�info�items�_find_files�_get_cached_file_name�debug�_get_file_from_s3)r�saltenv�env_meta�bucket_files�bucket�filesZ	file_path�cached_file_pathrrr�updatexs&






�

�����r�basec
Ks�d
|vr	|�d
�

ddd�}t
�}|r|
|v
r|St||
�
}t�s)tj�|
|�}|D] }|��D]\}}||vrIt�	t
|�sI||d<||d<
nq1q+
|drT|dsV|St|d|
|�}	t||
|d||	�
|S)z�
    Look through the buckets cache file for a match.
    If the field is found, it is retrieved from S3 only if its cached version
    is missing, or if the MD5 does not match.
    �envN)r�pathrr)
�poprr�_is_env_per_bucket�osr�joinr�fs�is_file_ignoredr
rr)
rr�kwargs�fndr�	env_filesr�bucket_namerrrrr�	find_file�s.









�

r'cCs�d
|vr	|�d
�

i}d|v
r|Sd|
v
sd|
v
s|
ds|St
|
d|d|
d�}tj�|�
r?t|d<tjj�	|t�|d<|S)z4
    Return the hash of an object's cached copy
    rrrrZ	hash_typeZhsum)
rrrr�isfile�S3_HASH_TYPE�salt�utils�	hashutils�get_hash)�loadr$�retrrrr�	file_hash�s



�

r0cCs"
d
|vr	|�d
�

ddd�}d|v
sd|v
sd|v
r|Sd|
v
s$d|
v
r&|S|�
dd	�}t|
d|d|
d�}t|
dg
|d�d
|d<tjj�|d��9}|�|d�

|�	t
d
�
}|rktjj�|�
sk|�t
�
}|r{|r{tjj�||�}||d<||d<Wd	�
|S1s�w



Y
|S)z?
    Return a chunk from a file based on the data received
    r�)�data�destr�locrr�gzipNr
r3�rbZfile_buffer_sizer2)rrr�_trim_env_off_pathr*r+r�fopen�seek�readr
Z	is_binary�decodeZ__salt_system_encoding__Z	gzip_util�compress)r.r$r/r5r�fp_r2rrr�
serve_file�s2




�










�	�	r>c
Cs�d
|vr	|�d
�

g}
d|v
r|
S|d}t
�}|r||v
r |
St||�
D]}|��D]}dd�|D�
}|
t||�7}
q,q&|
S)zR
    Return a list of all files on the file server in a specified environment
    rrc
Ssg|]
}
t�
t|
�s|
�qSr)r!r"r
)�.0�
frrr�
<listcomp>
�zfile_list.<locals>.<listcomp>)rrr�valuesr7)r.r/rrr�bucketsrrrr�	file_list
s







�rEc


Cs
t�
gS)
z>
    Return a list of all empty directories on the master
    )
r)
r.rrr�file_list_emptydirs
srFc
Cs�d
|vr	|�d
�

g}
d|v
r|
S|d}t
�}|r||v
r |
St||�
D]}|��D]}t||dd�}|
dd�|D�
7}
q,q&|
S)z8
    Return a list of all directories on the master
    rrT)
�
trim_slashc
Ssg|]}
|
r|
�qSrr)r?Z_frrrrA=
�zdir_list.<locals>.<listcomp>)rr�
_find_dirsrCr7)r.r/rrr�dirsrrr�dir_list$
s




�rKcCs�d
tvrtd
n
d}dtvrtdn
d}
dtvrtdn
d}dtvr&tdn
d}dtvr0tdn
d}dtvr:tdn
d}d	tvrDtd	n
d}d
tvrNtd
n
d}||
||||||fS)z,
    Get AWS keys from pillar or config
    zs3.keyNzs3.keyidzs3.service_urlz
s3.verify_sslz
aws.kms.keyidz
aws.kmw.keyidzs3.locationz
s3.path_stylezs3.https_enable�
r
)�key�keyid�service_url�
verify_ssl�	kms_keyid�location�
path_style�https_enablerrr�_get_s3_keyB
s$





�






�rUcCsjt�}t
�
�t�d
d�}
d}ztj�|�
|
krt|�
}Wn	ty&


Yn
w|dur/t	|�
}t
|�

|S)zx
    Connect to S3 and download the metadata for each file in all buckets
    specified and cache the data to disk.
    zs3.s3_cache_expire�N)�_get_buckets_cache_filename�timer
rrr�getmtime�_read_buckets_cache_file�OSError�_refresh_buckets_cache_file�_prune_deleted_files)�
cache_file�exprrrrr^
s


�

�rcCstj
�td
d�S)z,
    Return the path to the s3cache dir
    ZcachedirZs3cache)rrr r
rrrr�_get_cache_dirx
sr`cCstj
�t�|
||�S)
z<
    Return the cached file name for a bucket path file
    �rrr r`)r&rrrrrr�
�rcCstj
�t�d
�S)z?
    Return the filename of the cache for bucket contents.
    zbuckets_files.cacherarrrrrW�
srWc

szt�
d
�

t�\}
}��}�
��i}|
|f��
���fdd�	}t�r�t���D]m\�}g}|D]`}i}	||�
}
|
s:q/dd�|
D�
|	|<|�|	�

|	|s�i}|
D]}d|vsZd|vr_|�|�

qPzt�d|d|�
Wq/t	y�


d|vr�t�d|d|�
Yq/t�d	�

iY

Swq/||�<q'n�t�D]�}||�
}
|
s�q�d
d�|
D�
}
|
s�i}|
D]}d|vs�d|vr�|�|�

q�zt�d|d|�
Wq�t	y�


d|vr�t�d|d|�
Yq�t�d	�

iY
Swdd�|
D�
}t
|�
}|D];��f
dd�|
D�
}�|v
�
r
g|�<d
}|�D]}	||	v�
r&|	||7<d}
n�
q|�
s4|��||i
�

q�q�t||�
|S)zb
    Retrieve the content of all buckets and cache the metadata to the buckets
    cache file
    zRefreshing buckets cache filec
s�gd
}}	td|
|||���
d��d|i
d�}g}|D]
}d|vr%
n|�
|�

q|�|�

tdd	�|D�
�
r<	|S|d
d}q)Nr1T�s3.queryF�marker)rMrNrQrrOrPrRZ
return_binrSrT�params�Keyc
Ssg|]
}
|
�dd
�d
k�qS)ZIsTruncated�false)
r)r?�headerrrrrA�
rBzF_refresh_buckets_cache_file.<locals>.__get_s3_meta.<locals>.<listcomp>���)�	__utils__�append�extend�all)rrMrNr/rd�tmp�headersrh)rTrRrSrOrPrr�
__get_s3_meta�
s8














�








���z2_refresh_buckets_cache_file.<locals>.__get_s3_metac
S�g|]}
d|
vr|
�qS�
rfr�r?�
krrrrA�
�z/_refresh_buckets_cache_file.<locals>.<listcomp>ZCode�Messagez'%s' response for bucket '%s'z2S3 Error! Do you have any files in your S3 bucket?c
SrqrrrrsrrrrA�
ruc
Ss(g|]}
tj
�|
d�
�d
d�d�qS)rf�
/�
r
)rr�dirname�splitrsrrrrAs(c
sg|]}
|
d���
r|
�qSrr)
�
startswithrs)
rrrrA#sFT)rrrUr�_get_bucketsrrkr�warning�KeyError�set�_write_buckets_cache_file)r^rMrNrQrrprDZbucket_files_listr&rZs3_metaZ
meta_responsertrZenvironmentsr%�foundr)rTrRrSrrOrPrr\�
s�
�


















�



�




�
���
�
0





�



�




�
��








�
��
r\c
Cs�
t�}
t
�}t
�}t�rE|��D]3\}}|D],}|��D]%\}}tj�|
||�}	tj�|	�
r3|�|	�

|D]}
|
d
}|�|�

q5qqqn-|��D](\}}t	�D]}tj�|
|�}	tj�|	�
rd|�|	�

qP|D]	}
|�|
d
�

qgqIt
�tj
�
r�ddl}t
�d|�|�
�
|D]f}	t�|	�
D]^\}
}}|D]V}tj�|
|�}tj�||	�}||v
r�t
�d|�
t
�d|�
t�|�

tj�|�
}||
kr�tt�|�
�
dkr�t
�d|�
t�|�

tj�|�
}||
kr�tt�|�
�
dks�q�q�q�dS)Nrfr
zcached file list:
%sz'File '%s' not found in cached file listz6File '%s' was deleted from bucket, deleting local copyz%Directory '%s' is now empty, removing)r`rrrrrr �exists�addr|r�isEnabledFor�logging�DEBUG�pprintrZpformat�walk�relpathr
�unlinkry�len�listdir�rmdir)r�	cache_dirZcached_files�rootsrZenv_data�bucket_metarZbucket_data�root�metarr�rrJr�	file_namer�ryrrrr]7sf










����




�
�








�





�����r]cCs~t�}t
j�|�
st
�|�

t
j�|
�
rt
�|
�

t�d
�

t	j
j�|
d��}t
�||�
Wd�
dS1s8w



Y
dS)z6
    Write the contents of the buckets cache file
    zWriting buckets cache file�wbN)r`rrr��makedirsr(�removerrr*r+rr8�pickle�dump)rr^r�r=rrrr�ps





"�r�c
Cs�t�
d
�

tj�|�
st�
d�

dStjj�|d��;}
zt	�
|
�
}Wn"t	jtt
ttttfyD
}
zt�
d|�
d}WYd}~n
d}~w
wWd�
|SWd�
|S1sXw



Y
|S)z7
    Return the contents of the buckets cache file
    zReading buckets cache filezCache file does not existNr6z*Exception reading buckets cache file: '%s')rrrrr�r*r+rr8r�r.ZUnpicklingError�AttributeError�EOFError�ImportError�
IndexErrorr~�
ValueError)r^r=r2�excrrrrZ�s8













�	
���
��
��rZc
Cs�g}
i}|D]<}|��D]5\}}d
d�|D�
}dd�|D�
}||v
r.d||<|
�
||i
�

q|
D]}||vr@|||7<
n
q0qq|
S)zA
    Looks for all the files in the S3 bucket cache metadata
    c
S�g|]}
|
d�qSrrrrsrrrrA�rHz_find_files.<locals>.<listcomp>c
Ssg|]	}
|
�d�
s|
�qS)
rw)
�endswithrsrrrrA�sT)rrk)rr/r��bucket_dictr&r2Z	filepathsrrrrr�s"









���rc
Cs�g}
i}|D]d}|��D]]\}}t
�}d
d�|D�
D]}d}|�d�
dd�D]}	||	d}
|�|
�

|
}q'q||v
rJd||<|
�|t|�
i
�

q|
D]}||vrh||t|�
7<tt
||�
�
||<
n
qLqq|
S)z�
    Looks for all the directories in the S3 bucket cache metadata.

    Supports trailing '/' keys (as created by S3 console) as well as
    directories discovered in the path of file keys.
    c
Sr�rrrrsrrrrA�rHz_find_dirs.<locals>.<listcomp>r1rwNriT)rrrzr�rkr)rr/r�r�r&r2Zdirpathsr�prefix�part�	directoryrrrrrI�s0









�





���rIc		Cs�||vr||n
i}i}|D]
}|
|vr||
}qttt
d
d�|��
�
}|D]'}d|vrM|d|krMz|d�d�
|d<W|
StyL


Y|
Swq&dS)zA
    Looks for a file's metadata in the S3 bucket cache file
    c

Ssd
|vS)Nrfr)
rtrrr�<lambda>�sz!_find_file_meta.<locals>.<lambda>rf�ETag�
"N)r�filter�stripr~)	rr&rrrr�rZ
files_metaZ	item_metarrr�_find_file_meta�s$



�


�

���r�cCsd
tvrtd
SiS)z*
    Return the configuration buckets
    z
s3.bucketsrLrrrrr|�rbr|cCs�
tj
�|�
}tj
�|�
st�|�

t�\}}}}	}
}}}
tj
�|�
r�t|||
|�}|r�|d
}|�d�
dkrM|}t	j
j�|t
�}||krFdSt�d|�
n�t�|�
}|j}tj�|j�
}tj�|dd�}|t|d�
kr�||kr�t�d	�

td
|||d|||	|tj�|�
|d||
d
�
}|du
r�d}}|d��D](\}}|��}|��}t|�
��dkr�tj�|d�}q�t|�
��dkr�t|�
}q�|r�||kr�|r�||kr�t�d||
|�
dStd
||||||	|tj�|�
|||
d�
dS)zw
    Checks the local cache for the file, if it's old or missing go grab the
    file from S3 and update the cache
    r��
-riNz-found different hash for file %s, updating...ZLastModifiedz%Y-%m-%dT%H:%M:%S.%fZZSizezicached file size equal to metadata size and cached file mtime later than metadata last modification time.rc�HEADT)
rMrNrQ�methodrrOrPrRr�
local_fileZfull_headersrSrTroz
last-modifiedz%a, %d %b %Y %H:%M:%S %Zzcontent-lengthzW%s - %s : %s skipped download since cached file size equal to and mtime after s3 values)rMrNrQrrOrPrRrr�rSrT) rrryr�r�rUr(r��findr*r+r,r-r)rr
�stat�st_size�datetime�
fromtimestamp�st_mtime�strptime�intrrj�urllib�parse�quoterr��str�lower)rrr&rr�
target_dirrMrNrOrPrQrRrSrTZ	file_metaZ	file_etagZfile_md5Z
cached_md5Zcached_file_statZcached_file_sizeZcached_file_mtimeZcached_file_lastmodr/Z
s3_file_mtimeZs3_file_size�header_nameZheader_value�name�valuerrrr�s�

�













�




�
�

�













�






�
�

�



�












�rFcs6t�rd
nt
|
�
d�|rdn
d
�
��
fdd�|D�
S)zH
    Return a list of file paths with the saltenv directory removed
    Nrxric
sg|]}
|
��
��qSrr)r?�
d�Zenv_lenZ	slash_lenrrrAkruz&_trim_env_off_path.<locals>.<listcomp>)rr�)�pathsrrGrr�rr7ds
r7c
Cs*t�}t
|t�r
d
St
|t�rdStd�
�
)z�
    Return the configuration mode, either buckets per environment or a list of
    buckets that have environment dirs in their root
    TFz)Incorrect s3.buckets type given in config)r|�
isinstance�dictrr�)
rDrrrrns





r)
r)
F)-�__doc__r�r�rr�rX�urllib.parser�Zsalt.fileserverZ
fileserverr!Zsalt.modulesr*Zsalt.utils.filesZsalt.utils.gzip_utilZsalt.utils.hashutilsZsalt.utils.versions�	getLogger�__name__rr)r	rr'r0r>rErFrKrUrr`rrWr\r]r�rZrrIr�r|rr7rrrrr�<module>
sPY











+%
(9 
q