> For the complete documentation index, see [llms.txt](https://cajac.gitbook.io/ctf-notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cajac.gitbook.io/ctf-notes/web/web-discovery/checklist-for-web-discovery.md).

# Checklist for Web Discovery

## Checklist

<details>

<summary>Manual checks</summary>

Manually browse to the web site and check for:

* Comments in the source code (e.g. with [http-comments-displayer](https://nmap.org/nsedoc/scripts/http-comments-displayer.html) Nmap script)
* Usernames or
* Other interesting stuff

</details>

<details>

<summary>Check for presence of robots.txt file</summary>

Check to see if there is a `robots.txt` file

```bash
curl http://192.168.216.52/robots.txt   
```

</details>

<details>

<summary>Check for presence of sitemap.xml file</summary>

Check to see if there is a `sitemap.xml` file

```bash
curl http://192.168.216.52/sitemap.xml
```

</details>

<details>

<summary>Check for files and directories with bruteforce dictionary attacks</summary>

Use tools such as:

* Dirb
* Gobuster

</details>

<details>

<summary>Crawl the web site with a web crawler</summary>

Use a [web crawler](https://en.wikipedia.org/wiki/Web_crawler) such as:

* Hakrawler

</details>