> For the complete documentation index, see [llms.txt](https://cajac.gitbook.io/ctf-notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cajac.gitbook.io/ctf-notes/misc/cloud/aws-s3.md). # AWS S3 Amazon Simple Storage Service (S3) is a service offered by [Amazon Web Services](https://en.wikipedia.org/wiki/Amazon_Web_Services) (AWS) that provides [object storage](https://en.wikipedia.org/wiki/Object_storage) through a [web service](https://en.wikipedia.org/wiki/Web_service) interface. Amazon S3 uses the same scalable storage infrastructure that [Amazon.com](https://en.wikipedia.org/wiki/Amazon_\(company\)) uses to run its e-commerce network. Amazon S3 can store any type of object, which allows uses like storage for Internet applications, backups, disaster recovery, data archives, [data lakes](https://en.wikipedia.org/wiki/Data_lake) for analytics, and [hybrid cloud storage](https://en.wikipedia.org/wiki/Cloud_computing#Hybrid). ## Identification Default answer to `curl` is a simple status in JSON ```bash ┌──(kali㉿kali)-[/mnt/…/Hack_the_Box/HTB_Starting_Point/Tier_1/Three] └─$ curl s3.thetoppers.htb {"status": "running"} ``` ## awscli We can interact with S3 buckets with the `aws` command. Install it with `sudo apt install awscli` if needed. ### Initial configuration Before doing anything else we need to setup `aws` with some initial configuration. \ This is done with `aws configure`. ```bash ┌──(kali㉿kali)-[/mnt/…/Hack_the_Box/HTB_Starting_Point/Tier_1/Three] └─$ aws configure AWS Access Key ID [****************dmin]: htb AWS Secret Access Key [****************dmin]: htb Default region name [eu-central-1]: Default output format [json]: ``` ### List available buckets Add `--no-sign-request` for anonymous access. ```bash ┌──(kali㉿kali)-[/mnt/…/Hack_the_Box/HTB_Starting_Point/Tier_1/Three] └─$ aws --endpoint=http://s3.thetoppers.htb s3 ls 2023-12-17 10:00:46 thetoppers.htb ``` ### List content of a bucket To list the content of the bucket `thetoppers.htb` ```bash aws --endpoint=http://s3.thetoppers.htb s3 ls s3://thetoppers.htb ``` ```bash aws --endpoint=http://s3.thetoppers.htb s3 ls --recursive s3://thetoppers.htb ``` ls command - AWS S3 CLI: ### Copy files to/from a bucket Copies a local file or S3 object to another location locally or in S3. ```bash aws --endpoint-url http://challenge.ctf.games:32696 --no-sign-request s3 cp s3://bucket/47P9KwAlD4L33R4l local_file.txt ``` cp command - AWS S3 CLI: ### Sync files to/from a bucket Syncs directories and S3 prefixes. Recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files. ```bash ┌──(kali㉿kali)-[/mnt/…/Huntress_CTF/Huntress_CTF_2024/Miscellaneous/Malibu] └─$ aws --endpoint-url http://challenge.ctf.games:31541 --no-sign-request s3 sync s3://bucket local_dir ``` sync command - AWS S3 CLI: ## Usage information
aws --help ```bash ┌──(kali㉿kali)-[~] └─$ aws --help usage: aws [options] [ ...] [parameters] To see help text, you can run: aws help aws help aws help aws: error: the following arguments are required: command ```
aws s3 help ``` S3() S3() NAME s3 - DESCRIPTION This section explains prominent concepts and notations in the set of high-level S3 commands provided. If you are looking for the low level S3 commands for the CLI, please see the s3api command reference page. Path Argument Type Whenever using a command, at least one path argument must be specified. There are two types of path arguments: LocalPath and S3Uri. LocalPath: represents the path of a local file or directory. It can be written as an absolute path or relative path. S3Uri: represents the location of a S3 object, prefix, or bucket. This must be written in the form s3://mybucket/mykey where mybucket is the specified S3 bucket, mykey is the specified S3 key. The path argument must begin with s3:// in order to denote that the path argument refers to a S3 object. Note that prefixes are separated by forward slashes. For example, if the S3 object myobject had the prefix myprefix, the S3 key would be myprefix/myobject, and if the object was in the bucket my- bucket, the S3Uri would be s3://mybucket/myprefix/myobject. S3Uri also supports S3 access points. To specify an access point, this value must be of the form s3:///. For example if the access point myaccesspoint to be used has the ARN: arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint and the ob- ject being accessed has the key mykey, then the S3URI used must be: s3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/mykey. Similar to bucket names, you can also use prefixes with access point ARNs for the S3Uri. For example: s3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/mypre- fix/ Order of Path Arguments Every command takes one or two positional path arguments. The first path argument represents the source, which is the local file/directory or S3 object/prefix/bucket that is being referenced. If there is a second path argument, it represents the destination, which is the local file/directory or S3 object/prefix/bucket that is being operated on. Commands with only one path argument do not have a destination because the operation is being performed only on the source. Single Local File and S3 Object Operations Some commands perform operations only on single files and S3 objects. The following commands are single file/object operations if no --recur- sive flag is provided. o cp o mv o rm For this type of operation, the first path argument, the source, must exist and be a local file or S3 object. The second path argument, the destination, can be the name of a local file, local directory, S3 ob- ject, S3 prefix, or S3 bucket. The destination is indicated as a local directory, S3 prefix, or S3 bucket if it ends with a forward slash or back slash. The use of slash depends on the path argument type. If the path argument is a Local- Path, the type of slash is the separator used by the operating system. If the path is a S3Uri, the forward slash must always be used. If a slash is at the end of the destination, the destination file or object will adopt the name of the source file or object. Otherwise, if there is no slash at the end, the file or object will be saved under the name provided. See examples in cp and mv to illustrate this description. Directory and S3 Prefix Operations Some commands only perform operations on the contents of a local direc- tory or S3 prefix/bucket. Adding or omitting a forward slash or back slash to the end of any path argument, depending on its type, does not affect the results of the operation. The following commands will al- ways result in a directory or S3 prefix/bucket operation: o sync o mb o rb o ls Use of Exclude and Include Filters Currently, there is no support for the use of UNIX style wildcards in a command's path arguments. However, most commands have --exclude "" and --include "" parameters that can achieve the de- sired result. These parameters perform pattern matching to either ex- clude or include a particular file or object. The following pattern symbols are supported. o *: Matches everything o ?: Matches any single character o [sequence]: Matches any character in sequence o [!sequence]: Matches any character not in sequence Any number of these parameters can be passed to a command. You can do this by providing an --exclude or --include argument multiple times, e.g. --include "*.txt" --include "*.png". When there are multiple filters, the rule is the filters that appear later in the command take precedence over filters that appear earlier in the command. For exam- ple, if the filter parameters passed to the command were --exclude "*" --include "*.txt" All files will be excluded from the command except for files ending with .txt However, if the order of the filter parameters was changed to --include "*.txt" --exclude "*" All files will be excluded from the command. Each filter is evaluated against the source directory. If the source location is a file instead of a directory, the directory containing the file is used as the source directory. For example, suppose you had the following directory structure: /tmp/foo/ .git/ |---config |---description foo.txt bar.txt baz.jpg In the command aws s3 sync /tmp/foo s3://bucket/ the source directory is /tmp/foo. Any include/exclude filters will be evaluated with the source directory prepended. Below are several examples to demonstrate this. Given the directory structure above and the command aws s3 cp /tmp/foo s3://bucket/ --recursive --exclude ".git/*", the files .git/config and .git/description will be excluded from the files to upload because the exclude filter .git/* will have the source prepended to the filter. This means that: /tmp/foo/.git/* -> /tmp/foo/.git/config (matches, should exclude) /tmp/foo/.git/* -> /tmp/foo/.git/description (matches, should exclude) /tmp/foo/.git/* -> /tmp/foo/foo.txt (does not match, should include) /tmp/foo/.git/* -> /tmp/foo/bar.txt (does not match, should include) /tmp/foo/.git/* -> /tmp/foo/baz.jpg (does not match, should include) The command aws s3 cp /tmp/foo/ s3://bucket/ --recursive --exclude "ba*" will exclude /tmp/foo/bar.txt and /tmp/foo/baz.jpg: /tmp/foo/ba* -> /tmp/foo/.git/config (does not match, should include) /tmp/foo/ba* -> /tmp/foo/.git/description (does not match, should include) /tmp/foo/ba* -> /tmp/foo/foo.txt (does not match, should include) /tmp/foo/ba* -> /tmp/foo/bar.txt (matches, should exclude) /tmp/foo/ba* -> /tmp/foo/baz.jpg (matches, should exclude) Note that, by default, all files are included. This means that provid- ing only an --include filter will not change what files are trans- ferred. --include will only re-include files that have been excluded from an --exclude filter. If you only want to upload files with a par- ticular extension, you need to first exclude all files, then re-include the files with the particular extension. This command will upload only files ending with .jpg: aws s3 cp /tmp/foo/ s3://bucket/ --recursive --exclude "*" --include "*.jpg" If you wanted to include both .jpg files as well as .txt files you can run: aws s3 cp /tmp/foo/ s3://bucket/ --recursive \ --exclude "*" --include "*.jpg" --include "*.txt" SYNOPSIS aws s3 [ ...] OPTIONS None AVAILABLE COMMANDS o cp o ls o mb o mv o presign o rb o rm o sync o website S3() ```
## Resources Amazon S3 - Wikipedia: Amazon Web Services - Wikipedia: AWS CLI with MinIO Server: High-level commands - AWS S3 CLI: