Blog Folders

 

Introduction

Retrieval-Augmented Generation (RAG) is how production AI systems answer with facts instead of guesses: retrieve relevant, current documents, then let the LLM generate grounded responses. But a RAG system is only as good as the data flowing into its vector database — and for most commercial use cases (pricing assistants, product Q&A bots, market intelligence copilots), that data lives on the open web and changes daily.

This tutorial from Actowiz Solutions walks through the complete pipeline: scraped web content → cleaning → structuring → chunking → embedding → vector DB ingestion → refresh. Code examples use Python with widely adopted tooling, and every stage reflects the patterns we run at production scale for enterprise clients.

What Does "RAG-Ready" Actually Mean?

A dataset is RAG-ready when retrieval works — meaning a user's question reliably pulls back the right passages. In practice that requires five properties:

Clean text — no nav bars, cookie banners, or footer boilerplate polluting embeddings

Right-sized chunks — passages small enough to be precise, large enough to carry context

Rich metadata — source URL, timestamp, category, language, price fields — so retrieval can filter, not just search

Freshness — stale product prices in a vector DB are worse than no data; the pipeline must refresh

Deduplication — near-duplicate chunks waste index space and skew retrieval scores

Raw scraped HTML has none of these. The pipeline below adds all five.

Pipeline Architecture Overview

[Scraper / Actowiz Feed]

        ? raw JSON/HTML

        ?

[1. Extraction & Cleaning] → boilerplate removal, text normalization

        ?

[2. Structuring] → typed records + metadata

        ?

[3. Deduplication] → hash + near-dup filtering

        ?

[4. Chunking] → semantic, metadata-aware chunks

        ?

[5. Embedding] → batch embed with retry logic

        ?

[6. Vector DB Ingestion] → upsert with IDs (idempotent)

        ?

[7. Refresh Loop] → scheduled re-crawl + delta upserts

 

 

Step 1 — Start from Structured Scraped Records

The single biggest RAG shortcut: don't feed raw HTML to your pipeline. Start from structured extraction. A professionally scraped product record looks like this (illustrative sample):

{

  "url": "https://example-retailer.com/product/earbuds-x",

  "scraped_at": "2026-08-01T06:00:00Z",

  "title": "Wireless Earbuds Model X",

  "brand": "SampleBrand",

  "price": 79.99,

  "currency": "USD",

  "description": "Noise-cancelling earbuds with 30-hour battery...",

  "specs": {"battery_hours": 30, "bluetooth": "5.4"},

  "reviews_summary": "4.4 stars from 12,847 reviews",

  "category": "Electronics > Audio > Earbuds",

  "language": "en"

}

 

 

Actowiz delivers feeds in exactly this shape (JSONL/Parquet), which lets you skip weeks of HTML-parsing fragility. If you're scraping in-house, use a robust extractor and keep the raw HTML separately for reprocessing.

Step 2 — Clean and Normalize

Even structured text needs normalization before embedding:

import re, unicodedata

 

 

def clean_text(text: str) -> str:

    text = unicodedata.normalize("NFKC", text)

    text = re.sub(r"\s+", " ", text) # collapse whitespace

    text = re.sub(r"(Cookie Policy|Accept All|Subscribe to newsletter).*", "", text, flags=re.I)

    return text.strip()

 

 

Normalize currencies and units into metadata fields rather than leaving them in prose — "?1,299" and "Rs. 1299" should both become {"price": 1299, "currency": "INR"} so retrieval can filter numerically.

Step 3 — Deduplicate Before You Embed

Embedding costs money and duplicates poison retrieval. A two-tier approach works well:

import hashlib

from datasketch import MinHash, MinHashLSH

 

 

def exact_hash(text): 

    return hashlib.sha256(text.encode()).hexdigest()

 

 

lsh = MinHashLSH(threshold=0.85, num_perm=128)

 

 

def is_near_duplicate(doc_id: str, text: str) -> bool:

    m = MinHash(num_perm=128)

    for token in set(text.lower().split()):

        m.update(token.encode())

    if lsh.query(m):

        return True

    lsh.insert(doc_id, m)

    return False

 

 

Exact hashes catch identical pages; MinHash-LSH catches template near-duplicates (the same product page across regional subdomains, for example). At web scale, expect to drop 30–40% of records here — that reduction improves answer quality.