Web Crawler: Convert Websites to AI-Ready Markdown in Google Sheets

{
  "meta": {
    "instanceId": "3d7eb9567ae690bf8c9bba1cb43396e6e40c18e15eb5889cf9673ed1713da6db",
    "templateCredsSetupCompleted": true
  },
  "nodes": [
    {
      "id": "349e50cf-75b8-432c-818e-63f1ff3ead34",
      "name": "Overview Note",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        1696,
        3104
      ],
      "parameters": {
        "color": 4,
        "width": 600,
        "height": 1112,
        "content": "# Automated Website Crawler for AI Knowledge Bases\n\n## 📋 What This Template Does\nThis workflow crawls a website's homepage to extract all sublinks, filters images from content pages, scrapes and converts textual content to Markdown, then aggregates everything into Google Sheets—ideal for building AI-ready knowledge bases or company dossiers.\n\n## 🔧 Prerequisites\n- Google account with Sheets access\n- n8n instance\n\n## 🔑 Required Credentials\n\n### Google Sheets OAuth2 API Setup\n1. Go to console.cloud.google.com → APIs & Services → Credentials\n2. Create OAuth client ID for Web application\n3. Add n8n redirect URI: https://your-n8n-instance.com/rest/oauth2-credential/callback\n4. Add to n8n as Google Sheets OAuth2 API and grant Sheets scopes\n\n## ⚙️ Configuration Steps\n1. Import JSON into n8n\n2. Set target URL in Set Website node\n3. Assign Google credential to Sheet nodes\n4. Update documentId and sheetName to your spreadsheet\n5. Ensure sheet has columns: Website, Links, Scraped Content, Images\n6. Test manually\n\n## 🎯 Use Cases\n- Crawl company sites for knowledge base building\n- Extract content for AI agent training datasets\n- Gather competitor intel for market analysis\n- Archive dynamic sites for compliance\n\n## ⚠️ Troubleshooting\n- No links: Check homepage <a> tags and test URL\n- Sheet errors: Verify columns and permissions\n- Truncated content: Adjust slice limit or split rows\n- Rate limits: Add Wait node after scraping"
      },
      "typeVersion": 1
    },
    {
      "id": "eb43d67c-01fc-4d83-bb2c-099938a57468",
      "name": "Note: Trigger and Setup",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        2512,
        3072
      ],
      "parameters": {
        "color": 6,
        "width": 556,
        "height": 176,
        "content": "## 🖱️ Trigger & Setup Nodes\n\n**Purpose:** Manual Trigger starts the workflow; Set Website configures the target URL.\n\n**Note:** Update website_url in Set Website for your site; use Schedule Trigger for automation."
      },
      "typeVersion": 1
    },
    {
      "id": "3c8581cb-46cd-4f25-af5a-c52bc2f463c6",
      "name": "Set Website",
      "type": "n8n-nodes-base.set",
      "position": [
        2688,
        3296
      ],
      "parameters": {
        "options": {},
        "assignments": {
          "assignments": [
            {
              "id": "a652f57e-210e-421e-b20b-781d6f4dc240",
              "name": "website_url",
              "type": "string",
              "value": "https://example.com"
            }
          ]
        }
      },
      "typeVersion": 3.4
    },
    {
      "id": "18201858-7764-4a14-9f6b-12e36eaf158b",
      "name": "Manual Trigger",
      "type": "n8n-nodes-base.manualTrigger",
      "position": [
        2496,
        3296
      ],
      "parameters": {},
      "typeVersion": 1
    },
    {
      "id": "b7435481-bed3-439f-933c-1c5e0142ad5c",
      "name": "Scrape Homepage",
      "type": "n8n-nodes-base.httpRequest",
      "onError": "continueRegularOutput",
      "position": [
        2880,
        3296
      ],
      "parameters": {
        "url": "={{ $json.website_url }}",
        "options": {
          "redirect": {
            "redirect": {}
          },
          "allowUnauthorizedCerts": false
        }
      },
      "executeOnce": false,
      "typeVersion": 4.2,
      "alwaysOutputData": false
    },
    {
      "id": "ce13710d-24ca-47d4-a25c-8890c1592947",
      "name": "Note: Homepage Scraping",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        3168,
        3488
      ],
      "parameters": {
        "color": 5,
        "width": 396,
        "height": 192,
        "content": "## 🌐 Homepage Scraping Nodes\n\n**Purpose:** Scrape Homepage fetches HTML; Extract Links pulls hrefs from <a> tags; Split Links breaks array into items.\n\n**Note:** Handles redirects; targets all links for discovery."
      },
      "typeVersion": 1
    },
    {
      "id": "61a60f2c-f032-4b46-83ba-405df0ce05df",
      "name": "Extract Links from HTML",
      "type": "n8n-nodes-base.html",
      "position": [
        3088,
        3296
      ],
      "parameters": {
        "options": {
          "trimValues": true,
          "cleanUpText": true
        },
        "operation": "extractHtmlContent",
        "extractionValues": {
          "values": [
            {
              "key": "links",
              "attribute": "href",
              "cssSelector": "a",
              "returnArray": true,
              "returnValue": "attribute"
            }
          ]
        }
      },
      "typeVersion": 1.2
    },
    {
      "id": "582eeae0-fec0-4548-9c78-7c05ac5aaebc",
      "name": "Split Links",
      "type": "n8n-nodes-base.splitOut",
      "position": [
        3296,
        3296
      ],
      "parameters": {
        "options": {},
        "fieldToSplitOut": "links"
      },
      "typeVersion": 1
    },
    {
      "id": "17d59531-4d51-4494-8ae9-e91b81851a0b",
      "name": "Remove Duplicate Links",
      "type": "n8n-nodes-base.removeDuplicates",
      "position": [
        3520,
        3296
      ],
      "parameters": {
        "options": {}
      },
      "typeVersion": 2
    },
    {
      "id": "d50fa2a9-1a58-4dad-8bd0-cfbd31aeae91",
      "name": "Filter Real Hyperlinks",
      "type": "n8n-nodes-base.filter",
      "position": [
        3696,
        3296
      ],
      "parameters": {
        "options": {},
        "conditions": {
          "options": {
            "version": 2,
            "leftValue": "",
            "caseSensitive": true,
            "typeValidation": "strict"
          },
          "combinator": "and",
          "conditions": [
            {
              "id": "bd6c6da6-8af7-4809-b6cd-01a38d71953b",
              "operator": {
                "type": "string",
                "operation": "startsWith"
              },
              "leftValue": "={{ $json.links }}",
              "rightValue": "https://"
            }
          ]
        }
      },
      "typeVersion": 2.2
    },
    {
      "id": "cb121b70-a14a-4cbd-a54c-e55c6fc235b7",
      "name": "Note: Link Processing",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        3216,
        3056
      ],
      "parameters": {
        "color": 2,
        "width": 556,
        "height": 224,
        "content": "## 🔄 Link Processing Nodes\n\n**Purpose:** Remove Duplicate Links cleans list; Filter Real Hyperlinks keeps HTTPS; Separate Images and Links routes via regex.\n\n**Note:** Switch output 0: Images, 1: Content links; adjust regex for custom extensions."
      },
      "typeVersion": 1
    },
    {
      "id": "d69c0dc2-2c4c-474b-ba11-3d79e1390b12",
      "name": "Separate Images and Links",
      "type": "n8n-nodes-base.switch",
      "position": [
        2480,
        3680
      ],
      "parameters": {
        "rules": {
          "values": [
            {
              "outputKey": "Images",
              "conditions": {
                "options": {
                  "version": 2,
                  "leftValue": "",
                  "caseSensitive": true,
                  "typeValidation": "strict"
                },
                "combinator": "and",
                "conditions": [
                  {
                    "id": "16724958-4eea-489d-b494-3d76a3ba2562",
                    "operator": {
                      "type": "string",
                      "operation": "regex"
                    },
                    "leftValue": "={{ $json.links }}",
                    "rightValue": "=^https?:\\/\\/.*\\.(?:png|jpe?g|gif|webp|bmp|svg|ico)(?:\\?.*)?$"
                  }
                ]
              },
              "renameOutput": true
            },
            {
              "outputKey": "Links",
              "conditions": {
                "options": {
                  "version": 2,
                  "leftValue": "",
                  "caseSensitive": true,
                  "typeValidation": "strict"
                },
                "combinator": "and",
                "conditions": [
                  {
                    "id": "816392f0-96db-4134-8bee-4b74688ff929",
                    "operator": {
                      "type": "string",
                      "operation": "notRegex"
                    },
                    "leftValue": "={{ $json.links }}",
                    "rightValue": "=^https?:\\/\\/.*\\.(?:png|jpe?g|gif|webp|bmp|svg|ico)(?:\\?.*)?$"
                  }
                ]
              },
              "renameOutput": true
            }
          ]
        },
        "options": {}
      },
      "typeVersion": 3.2
    },
    {
      "id": "23896343-575e-4956-8e95-3b5e6e4c8ae7",
      "name": "Aggregate Images",
      "type": "n8n-nodes-base.aggregate",
      "position": [
        2736,
        3504
      ],
      "parameters": {
        "options": {},
        "fieldsToAggregate": {
          "fieldToAggregate": [
            {
              "fieldToAggregate": "links"
            }
          ]
        }
      },
      "typeVersion": 1
    },
    {
      "id": "fcad347b-60d7-4fa2-9b02-e96c2f27116d",
      "name": "Aggregate Links",
      "type": "n8n-nodes-base.aggregate",
      "position": [
        2736,
        3696
      ],
      "parameters": {
        "options": {},
        "fieldsToAggregate": {
          "fieldToAggregate": [
            {
              "fieldToAggregate": "links"
            }
          ]
        }
      },
      "typeVersion": 1
    },
    {
      "id": "fc5d6ce1-1765-4768-a9c7-de3677e8109d",
      "name": "Scrape Content Links",
      "type": "n8n-nodes-base.httpRequest",
      "position": [
        2736,
        3872
      ],
      "parameters": {
        "url": "={{ $json.links }}",
        "options": {}
      },
      "typeVersion": 4.2
    },
    {
      "id": "0d4b6a4e-b6cb-4e6c-9a22-bd0dc6a72027",
      "name": "Note: Content Scraping",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        2320,
        3984
      ],
      "parameters": {
        "color": 5,
        "width": 428,
        "height": 224,
        "content": "## 📄 Content Scraping & Aggregation Nodes\n\n**Purpose:** Scrape Content Links fetches pages; Convert to Markdown formats HTML; Aggregate Images/Links/Content combines outputs.\n\n**Note:** Markdown preserves structure for AI; slice content if exceeding sheet limits."
      },
      "typeVersion": 1
    },
    {
      "id": "349e5f7c-c81b-467b-a59b-ea40a47226f0",
      "name": "Convert to Markdown",
      "type": "n8n-nodes-base.markdown",
      "position": [
        2944,
        3872
      ],
      "parameters": {
        "html": "={{ $json.data }}",
        "options": {}
      },
      "typeVersion": 1
    },
    {
      "id": "24f22a31-03a3-4faf-81f4-3c38c0956ee4",
      "name": "Aggregate Scraped Content",
      "type": "n8n-nodes-base.aggregate",
      "position": [
        3136,
        3872
      ],
      "parameters": {
        "options": {},
        "fieldsToAggregate": {
          "fieldToAggregate": [
            {
              "fieldToAggregate": "data"
            }
          ]
        }
      },
      "typeVersion": 1
    },
    {
      "id": "a4d34aab-1af2-4196-85f5-1a2d832969dd",
      "name": "Add Images to Sheet",
      "type": "n8n-nodes-base.googleSheets",
      "position": [
        2944,
        3504
      ],
      "parameters": {
        "columns": {
          "value": {
            "Images": "={{ $json.links.join('\\n\\n') }}",
            "Website": "={{ $('Set Website').item.json.website_url }}"
          },
          "schema": [
            {
              "id": "Website",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Website",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Links",
              "type": "string",
              "display": true,
              "removed": true,
              "required": false,
              "displayName": "Links",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Scraped Content",
              "type": "string",
              "display": true,
              "removed": true,
              "required": false,
              "displayName": "Scraped Content",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Images",
              "type": "string",
              "display": true,
              "required": false,
              "displayName": "Images",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            }
          ],
          "mappingMode": "defineBelow",
          "matchingColumns": [
            "Website"
          ],
          "attemptToConvertTypes": false,
          "convertFieldsToString": false
        },
        "options": {},
        "operation": "appendOrUpdate",
        "sheetName": "your-sheet-name",
        "documentId": "your-document-id"
      },
      "credentials": {
        "googleSheetsOAuth2Api": {
          "id": "ZVbWK0SlohYDlZYO",
          "name": "Ewere"
        }
      },
      "typeVersion": 4.7
    },
    {
      "id": "6afbfad8-b80f-4a0d-81b4-9138cc2af46a",
      "name": "Add Links to Sheet",
      "type": "n8n-nodes-base.googleSheets",
      "position": [
        2944,
        3696
      ],
      "parameters": {
        "columns": {
          "value": {
            "Links": "={{ $json.links.join('\\n\\n') }}",
            "Website": "={{ $('Set Website').item.json.website_url }}"
          },
          "schema": [
            {
              "id": "Website",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Website",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Links",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Links",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Scraped Content",
              "type": "string",
              "display": true,
              "removed": true,
              "required": false,
              "displayName": "Scraped Content",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Images",
              "type": "string",
              "display": true,
              "removed": true,
              "required": false,
              "displayName": "Images",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            }
          ],
          "mappingMode": "defineBelow",
          "matchingColumns": [
            "Website"
          ],
          "attemptToConvertTypes": false,
          "convertFieldsToString": false
        },
        "options": {},
        "operation": "appendOrUpdate",
        "sheetName": "your-sheet-name",
        "documentId": "your-document-id"
      },
      "credentials": {
        "googleSheetsOAuth2Api": {
          "id": "ZVbWK0SlohYDlZYO",
          "name": "Ewere"
        }
      },
      "typeVersion": 4.7
    },
    {
      "id": "35ae2c30-a93a-4fd2-82b6-07d2f4c56c88",
      "name": "Add Scraped Content to Sheet",
      "type": "n8n-nodes-base.googleSheets",
      "position": [
        3344,
        3872
      ],
      "parameters": {
        "columns": {
          "value": {
            "Website": "={{ $('Set Website').item.json.website_url }}",
            "Scraped Content": "={{ $json.data.join('\\n\\n').slice(0, 50000) }}"
          },
          "schema": [
            {
              "id": "Website",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Website",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Links",
              "type": "string",
              "display": true,
              "removed": true,
              "required": false,
              "displayName": "Links",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Scraped Content",
              "type": "string",
              "display": true,
              "removed": false,
              "required": false,
              "displayName": "Scraped Content",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            },
            {
              "id": "Images",
              "type": "string",
              "display": true,
              "removed": true,
              "required": false,
              "displayName": "Images",
              "defaultMatch": false,
              "canBeUsedToMatch": true
            }
          ],
          "mappingMode": "defineBelow",
          "matchingColumns": [
            "Website"
          ],
          "attemptToConvertTypes": false,
          "convertFieldsToString": false
        },
        "options": {},
        "operation": "appendOrUpdate",
        "sheetName": "your-sheet-name",
        "documentId": "your-document-id"
      },
      "credentials": {
        "googleSheetsOAuth2Api": {
          "id": "ZVbWK0SlohYDlZYO",
          "name": "Ewere"
        }
      },
      "typeVersion": 4.7
    },
    {
      "id": "c3f7b022-db11-400c-baaa-77392acfb991",
      "name": "Note: Sheet Integration",
      "type": "n8n-nodes-base.stickyNote",
      "position": [
        3232,
        4048
      ],
      "parameters": {
        "color": 3,
        "width": 444,
        "height": 176,
        "content": "## 📊 Sheet Integration Nodes\n\n**Purpose:** Add Images/Links/Scraped Content to Sheet appends aggregated data to Google Sheets.\n\n**Note:** Matches on 'Website' column; update documentId/sheetName for your sheet."
      },
      "typeVersion": 1
    }
  ],
  "pinData": {},
  "connections": {
    "Set Website": {
      "main": [
        [
          {
            "node": "Scrape Homepage",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Split Links": {
      "main": [
        [
          {
            "node": "Remove Duplicate Links",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Manual Trigger": {
      "main": [
        [
          {
            "node": "Set Website",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Aggregate Links": {
      "main": [
        [
          {
            "node": "Add Links to Sheet",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Scrape Homepage": {
      "main": [
        [
          {
            "node": "Extract Links from HTML",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Aggregate Images": {
      "main": [
        [
          {
            "node": "Add Images to Sheet",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Convert to Markdown": {
      "main": [
        [
          {
            "node": "Aggregate Scraped Content",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Scrape Content Links": {
      "main": [
        [
          {
            "node": "Convert to Markdown",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Filter Real Hyperlinks": {
      "main": [
        [
          {
            "node": "Separate Images and Links",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Remove Duplicate Links": {
      "main": [
        [
          {
            "node": "Filter Real Hyperlinks",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Extract Links from HTML": {
      "main": [
        [
          {
            "node": "Split Links",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Aggregate Scraped Content": {
      "main": [
        [
          {
            "node": "Add Scraped Content to Sheet",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Separate Images and Links": {
      "main": [
        [
          {
            "node": "Aggregate Images",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Aggregate Links",
            "type": "main",
            "index": 0
          },
          {
            "node": "Scrape Content Links",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}