Picture by Writer
# Introduction
Internet crawling is the method of robotically visiting net pages, following hyperlinks, and accumulating content material from an internet site in a structured method. It’s generally used to assemble giant quantities of knowledge from documentation websites, articles, data bases, and different net sources.
Crawling a complete web site after which changing that content material right into a format that an AI agent can truly use shouldn’t be so simple as it sounds. Documentation websites usually comprise nested pages, repeated navigation hyperlinks, boilerplate content material, and inconsistent web page constructions. On prime of that, the extracted content material must be cleaned, organized, and saved in a method that’s helpful for downstream AI workflows corresponding to retrieval, question-answering, or agent-based techniques.
On this information, we are going to study why to make use of Olostep as a substitute of Scrapy or Selenium, arrange all the things wanted for the net crawling challenge, write a easy crawling script to scrape a documentation web site, and at last create a frontend utilizing Gradio in order that anybody can present a hyperlink and different arguments to crawl web site pages.
# Selecting Olostep Over Scrapy or Selenium
Scrapy is highly effective, however it’s constructed as a full scraping framework. That’s helpful whenever you need deep management, nevertheless it additionally means extra setup and extra engineering work.
Selenium is best identified for browser automation. It’s helpful for interacting with JavaScript-heavy pages, however it’s not actually designed as a documentation crawling workflow by itself.
With Olostep, the pitch is much more direct: search, crawl, scrape, and construction net information by means of one software programming interface (API), with assist for LLM-friendly outputs like Markdown, textual content, HTML, and structured JSON. Meaning you should not have to manually sew collectively items for discovery, extraction, formatting, and downstream AI use in the identical method.
For documentation websites, that can provide you a a lot quicker path from URL to usable content material since you are spending much less time constructing the crawling stack your self and extra time working with the content material you really need.
# Putting in the Packages and Setting the API Key
First, set up the Python packages used on this challenge. The official Olostep software program growth package (SDK) requires Python 3.11 or later.
pip set up olostep python-dotenv tqdm
These packages deal with the principle components of the workflow:
olostepconnects your script to the Olostep APIpython-dotenvmasses your API key from a .env filetqdmprovides a progress bar so you may monitor saved pages
Subsequent, create a free Olostep account, open the dashboard, and generate an API key from the API keys web page. Olostep’s official docs and integrations level customers to the dashboard for API key setup.
Then create a .env file in your challenge folder:
OLOSTEP_API_KEY=your_real_api_key_here
This retains your API key separate out of your Python code, which is a cleaner and safer technique to handle credentials.
# Creating the Crawler Script
On this a part of the challenge, we are going to construct the Python script that crawls a documentation web site, extracts every web page in Markdown format, cleans the content material, and saves it regionally as particular person recordsdata. We’ll create the challenge folder, add a Python file, after which write the code step-by-step so it’s straightforward to comply with and take a look at.
First, create a challenge folder in your crawler. Inside that folder, create a brand new Python file named crawl_docs_with_olostep.py.
Now we are going to add the code to this file one part at a time. This makes it simpler to grasp what every a part of the script does and the way the total crawler works collectively.
// Defining the Crawl Settings
Begin by importing the required libraries. Then outline the principle crawl settings, such because the beginning URL, crawl depth, web page restrict, embrace and exclude guidelines, and the output folder the place the Markdown recordsdata will likely be saved. These values management how a lot of the documentation web site will get crawled and the place the outcomes are saved.
import os
import re
from pathlib import Path
from urllib.parse import urlparse
from dotenv import load_dotenv
from tqdm import tqdm
from olostep import Olostep
START_URL = "
MAX_PAGES = 10
MAX_DEPTH = 1
INCLUDE_URLS = [
"/**"
]
EXCLUDE_URLS = []
OUTPUT_DIR = Path("olostep_docs_output")
// Making a Helper Perform to Generate Protected File Names
Every crawled web page must be saved as its personal Markdown file. To do this, we’d like a helper operate that converts a URL right into a clear and filesystem-safe file identify. This avoids issues with slashes, symbols, and different characters that don’t work effectively in file names.
def slugify_url(url: str) -> str:
parsed = urlparse(url)
path = parsed.path.strip("
if not path:
path = "index"
filename = re.sub(r"[^a-zA-Z0-9/_-]+", "-", path)
filename = filename.substitute(" "__").strip("-_")
return f"{filename or 'page'}.md"
// Making a Helper Perform to Save Markdown Recordsdata
Subsequent, add helper capabilities to course of the extracted content material earlier than saving it.
The primary operate cleans the Markdown by eradicating further interface textual content, repeated clean strains, and undesirable web page parts corresponding to suggestions prompts. This helps hold the saved recordsdata centered on the precise documentation content material.
def clean_markdown(markdown: str) -> str:
textual content = markdown.substitute("rn", "n").strip()
textual content = re.sub(r"[s*u200b?s*](#.*?)", "", textual content, flags=re.DOTALL)
strains = [line.rstrip() for line in text.splitlines()]
start_index = 0
for index in vary(len(strains) - 1):
title = strains[index].strip()
underline = strains[index + 1].strip()
if title and underline and set(underline) == {"="}:
start_index = index
break
else:
for index, line in enumerate(strains):
if line.lstrip().startswith("# "):
start_index = index
break
strains = strains[start_index:]
for index, line in enumerate(strains):
if line.strip() == "Was this page helpful?":
strains = strains[:index]
break
cleaned_lines: record[str] = []
for line in strains:
stripped = line.strip()
if stripped in {"Copy page", "YesNo", "⌘I"}:
proceed
if not stripped and cleaned_lines and never cleaned_lines[-1]:
proceed
cleaned_lines.append(line)
return "n".be part of(cleaned_lines).strip()
The second operate saves the cleaned Markdown into the output folder and provides the supply URL on the prime of the file. There may be additionally a small helper operate to clear previous Markdown recordsdata earlier than saving a brand new crawl consequence.
def save_markdown(output_dir: Path, url: str, markdown: str) -> None:
output_dir.mkdir(dad and mom=True, exist_ok=True)
filepath = output_dir / slugify_url(url)
content material = f"""---
source_url: {url}
---
{markdown}
"""
filepath.write_text(content material, encoding="utf-8")
There may be additionally a small helper operate to clear previous Markdown recordsdata earlier than saving a brand new crawl consequence.
def clear_output_dir(output_dir: Path) -> None:
if not output_dir.exists():
return
for filepath in output_dir.glob("*.md"):
filepath.unlink()
// Creating the Most important Crawler Logic
That is the principle a part of the script. It masses the API key from the .env file, creates the Olostep shopper, begins the crawl, waits for it to complete, retrieves every crawled web page as Markdown, cleans the content material, and saves it regionally.
This part ties all the things collectively and turns the person helper capabilities right into a working documentation crawler.
def principal() -> None:
load_dotenv()
api_key = os.getenv("OLOSTEP_API_KEY")
if not api_key:
increase RuntimeError("Missing OLOSTEP_API_KEY in your .env file.")
shopper = Olostep(api_key=api_key)
crawl = shopper.crawls.create(
start_url=START_URL,
max_pages=MAX_PAGES,
max_depth=MAX_DEPTH,
include_urls=INCLUDE_URLS,
exclude_urls=EXCLUDE_URLS,
include_external=False,
include_subdomain=False,
follow_robots_txt=True,
)
print(f"Started crawl: {crawl.id}")
crawl.wait_till_done(check_every_n_secs=5)
pages = record(crawl.pages())
clear_output_dir(OUTPUT_DIR)
for web page in tqdm(pages, desc="Saving pages"):
strive:
content material = web page.retrieve(["markdown"])
markdown = getattr(content material, "markdown_content", None)
if markdown:
save_markdown(OUTPUT_DIR, web page.url, clean_markdown(markdown))
besides Exception as exc:
print(f"Failed to retrieve {page.url}: {exc}")
print(f"Done. Files saved in: {OUTPUT_DIR.resolve()}")
if __name__ == "__main__":
principal()
Word: The total script is obtainable right here: kingabzpro/web-crawl-olostep, an internet crawler and starter net app constructed with Olostep.
// Testing the Internet Crawling Script
As soon as the script is full, run it out of your terminal:
python crawl_docs_with_olostep.py
Because the script runs, you will note the crawler course of the pages and save them one after the other as Markdown recordsdata in your output folder.
After the crawl finishes, open the saved recordsdata to examine the extracted content material. It’s best to see clear, readable Markdown variations of the documentation pages.
At that time, your documentation content material is able to use in AI workflows corresponding to search, retrieval, or agent-based techniques.
# Creating the Olostep Internet Crawling Internet Utility
On this a part of the challenge, we are going to construct a easy net software on prime of the crawler script. As a substitute of enhancing the Python file each time, this software offers you a neater technique to enter a documentation URL, select crawl settings, run the crawl, and preview the saved Markdown recordsdata in a single place.
The frontend code for this software is obtainable in app.py within the repository: web-crawl-olostep/app.py.
This software does a number of helpful issues:
- Enables you to enter a beginning URL for the crawl
- Enables you to set the utmost variety of pages to crawl
- Enables you to management crawl depth
- Enables you to add embrace and exclude URL patterns
- Runs the backend crawler immediately from the interface
- Saves the crawled pages right into a folder primarily based on the URL
- Exhibits all saved Markdown recordsdata in a dropdown
- Previews every Markdown file immediately inside the applying
- Enables you to clear earlier crawl outcomes with one button
To start out the applying, run:
After that, Gradio will begin an area net server and supply a hyperlink like this:
* Operating on native URL:
* To create a public hyperlink, set `share=True` in `launch()`.
As soon as the applying is operating, open the native URL in your browser. In our instance, we gave the applying the Claude Code documentation URL and requested it to crawl 50 pages with a depth of 5.
Whenever you click on Run Crawl, the applying passes your settings to the backend crawler and begins the crawl. Within the terminal, you may watch the progress as pages are crawled and saved one after the other.
After the crawl finishes, the output folder will comprise the saved Markdown recordsdata. On this instance, you’ll see that fifty recordsdata have been added.
The dropdown within the software is then up to date robotically, so you may open any saved file and preview it immediately within the net interface as correctly formatted Markdown.
This makes the crawler a lot simpler to make use of. As a substitute of adjusting values in code each time, you may take a look at totally different documentation websites and crawl settings by means of a easy interface. That additionally makes the challenge simpler to share with different individuals who might not need to work immediately in Python.
# Closing Takeaway
Internet crawling shouldn’t be solely about accumulating pages from an internet site. The true problem is popping that content material into clear, structured recordsdata that an AI system can truly use. On this challenge, we used a easy Python script and a Gradio software to make that course of a lot simpler.
Simply as importantly, the workflow is quick sufficient for actual use. In our instance, crawling 50 pages with a depth of 5 took solely round 50 seconds, which exhibits that you would be able to put together documentation information rapidly with out constructing a heavy pipeline.
This setup can even transcend a one-time crawl. You’ll be able to schedule it to run daily with cron or Activity Scheduler, and even replace solely the pages which have modified. That retains your documentation contemporary whereas utilizing solely a small variety of credit.
For groups that want this type of workflow to make enterprise sense, Olostep is constructed with that in thoughts. It’s considerably extra reasonably priced than constructing or sustaining an inner crawling resolution, and at the very least 50% cheaper than comparable options in the marketplace.
As your utilization grows, the price per request continues to lower, which makes it a sensible alternative for bigger documentation pipelines. That mixture of reliability, scalability, and powerful unit economics is why a number of the fastest-growing AI-native startups depend on Olostep to energy their information infrastructure.
Abid Ali Awan (@1abidaliawan) is a licensed information scientist skilled who loves constructing machine studying fashions. At the moment, he’s specializing in content material creation and writing technical blogs on machine studying and information science applied sciences. Abid holds a Grasp’s diploma in know-how administration and a bachelor’s diploma in telecommunication engineering. His imaginative and prescient is to construct an AI product utilizing a graph neural community for college students combating psychological sickness.
