Price MonitoringCeleryRedisPython+3 more

How to Build an eCommerce Price Monitoring Scraper with Python and PostgreSQL

How to build a price monitoring scraper, then scale it into a production system: PostgreSQL for history, Redis and Celery queues for scheduling and retries, and a separate alert pipeline.

Aashish Timsina
November 17, 2026
22 min read

What a price monitor actually does

A price monitoring scraper visits a list of product pages on a schedule, reads the current price and stock status, compares it with the last value it saw, and records every change. The useful output is not a single scrape, it is the history: how a price moved over weeks, when a competitor ran a discount, and when an item went out of stock.

Retailers, resellers and marketplace sellers use this to react to competitor pricing, and buyers use it to wait for a real discount instead of a fake one. The build below is small enough to run on a cheap server, and the structure scales to thousands of products.

The pipeline

  1. Fetch the product page politely, with timeouts and retries.
  2. Parse name, price, currency and availability.
  3. Normalise the price into a number, not a string.
  4. Store it, and write a history row only when something changed.
  5. Alert when the change matters, for example a drop of 10% or more.
  6. Schedule the whole thing with cron.

Step 1: Design the database first

Two tables keep things simple. products holds the current state and price_history holds one row per observed change. Use NUMERIC for money, never floating point.

CREATE TABLE products (
  id          SERIAL PRIMARY KEY,
  url         TEXT UNIQUE NOT NULL,
  name        TEXT,
  last_price  NUMERIC(12,2),
  currency    TEXT,
  in_stock    BOOLEAN,
  updated_at  TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE price_history (
  id          BIGSERIAL PRIMARY KEY,
  product_id  INT REFERENCES products(id),
  price       NUMERIC(12,2),
  in_stock    BOOLEAN,
  scraped_at  TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX ON price_history (product_id, scraped_at DESC);

Step 2: Fetch pages without being a nuisance

Use one requests.Session, set a timeout on every request, identify your bot honestly, and back off when the server tells you to slow down.

import time
import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "price-monitor/1.0 (contact: [email protected])",
    "Accept-Language": "en",
})

def fetch(url, retries=3):
    for attempt in range(retries):
        resp = session.get(url, timeout=20)
        if resp.status_code == 200:
            return resp.text
        if resp.status_code in (429, 503):
            time.sleep(5 * 2 ** attempt)   # 5s, 10s, 20s
            continue
        resp.raise_for_status()
    return None

Step 3: Parse the price, preferring structured data

Before writing CSS selectors, check the page source for a script tag of type application/ld+json. Many shops publish a schema.org Product object there with the price, currency and availability. It is far more stable than class names, which change whenever the site is redesigned.

import json
from bs4 import BeautifulSoup

def extract_product(html):
    soup = BeautifulSoup(html, "html.parser")

    for tag in soup.select('script[type="application/ld+json"]'):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        for item in data if isinstance(data, list) else [data]:
            if item.get("@type") != "Product":
                continue
            offer = item.get("offers", {})
            if isinstance(offer, list):
                offer = offer[0] if offer else {}
            return {
                "name": item.get("name"),
                "price": parse_price(str(offer.get("price", ""))),
                "currency": offer.get("priceCurrency"),
                "in_stock": "InStock" in str(offer.get("availability", "")),
            }

    # Fallback: CSS selectors for sites without structured data
    price_el = soup.select_one(".product-price, .price")
    title_el = soup.select_one("h1")
    if price_el and title_el:
        return {
            "name": title_el.get_text(strip=True),
            "price": parse_price(price_el.get_text()),
            "currency": None,
            "in_stock": soup.select_one(".out-of-stock") is None,
        }
    return None

Normalise prices carefully

Prices arrive as text such as Rs. 1,25,000.00. Nepali and Indian shops often use lakh-style grouping, so never assume commas appear every three digits. Find the first number, strip the commas, and convert with Decimal.

import re
from decimal import Decimal

def parse_price(text):
    match = re.search(r"\d[\d,]*(?:\.\d+)?", text or "")
    if not match:
        return None
    return Decimal(match.group().replace(",", ""))

# parse_price("Rs. 1,25,000.00") -> Decimal("125000.00")
# parse_price("NPR 4,990")       -> Decimal("4990")

Step 4: Store only what changed

Writing a row on every run bloats the table with identical values. Compare against the last known price and write history only when the price or stock status moved. The function returns the previous price so the caller can decide whether to alert.

import psycopg2

conn = psycopg2.connect("dbname=monitor user=monitor")

def save(url, item):
    with conn, conn.cursor() as cur:
        cur.execute(
            "SELECT id, last_price, in_stock FROM products WHERE url = %s",
            (url,),
        )
        row = cur.fetchone()

        if row is None:
            cur.execute(
                """INSERT INTO products (url, name, last_price, currency, in_stock)
                   VALUES (%s, %s, %s, %s, %s) RETURNING id""",
                (url, item["name"], item["price"], item["currency"], item["in_stock"]),
            )
            product_id, old_price, old_stock = cur.fetchone()[0], None, None
        else:
            product_id, old_price, old_stock = row

        changed = old_price != item["price"] or old_stock != item["in_stock"]
        if changed:
            cur.execute(
                "INSERT INTO price_history (product_id, price, in_stock) VALUES (%s, %s, %s)",
                (product_id, item["price"], item["in_stock"]),
            )
            cur.execute(
                "UPDATE products SET last_price = %s, in_stock = %s, updated_at = now() WHERE id = %s",
                (item["price"], item["in_stock"], product_id),
            )
        return old_price

Step 5: Detect drops and alert

Not every change deserves a message. Alert on a meaningful drop, and keep the threshold in one place so it is easy to tune.

from decimal import Decimal

DROP_THRESHOLD = Decimal("0.10")   # 10%

def is_big_drop(old, new):
    if old is None or new is None or old == 0:
        return False
    return (old - new) / old >= DROP_THRESHOLD

Send the alert however suits you: an email through SMTP, a Telegram bot message, or a webhook into Slack. The monitor itself should stay unaware of the channel, so wrap it in a small notify(message) function.

Step 6: Put it together and schedule it

import random
import time

URLS = [line.strip() for line in open("urls.txt") if line.strip()]

def run():
    for url in URLS:
        try:
            html = fetch(url)
            item = extract_product(html) if html else None
            if item is None or item["price"] is None:
                print("PARSE FAILED", url)
                continue
            old = save(url, item)
            if is_big_drop(old, item["price"]):
                notify(f"{item['name']}: {old} -> {item['price']}
{url}")
        except Exception as exc:
            print("ERROR", url, exc)
        time.sleep(random.uniform(2, 5))   # be polite

if __name__ == "__main__":
    run()

Run it every six hours with cron:

0 */6 * * * /usr/bin/python3 /opt/monitor/run.py >> /var/log/monitor.log 2>&1

Cron is fine for a few dozen products. When the list grows, jump to the production architecture with Redis and Celery further down.

When plain requests is not enough

  • The price is rendered by JavaScript. If the price is missing from the raw HTML, look in the browser network tab for the JSON request the page makes and call that directly. If there is no such endpoint, drive the page with Playwright. I cover the trade-offs in Handling Dynamic Content with Selenium and Playwright.
  • You start getting blocked. Slow down first. Add delays, reduce concurrency and cache pages you do not need to refetch. Proxy rotation is a last resort, not a first move.
  • Prices depend on location or login. Make sure the monitor sees the same page a real customer in the target market would see, otherwise your data will quietly be wrong.

Keep it running: maintenance matters more than the first build

The first version is the easy part. Sites change their markup, and a scraper that silently returns nothing is worse than one that fails loudly. Three habits prevent most problems:

  • Count parse failures per run. If more than a small share of URLs fail to parse, alert yourself. That is almost always a layout change.
  • Sanity-check values. Reject a price of zero, or one that changed by more than, say, 80% in one step, and flag it for review instead of saving it.
  • Log the raw HTML of failures. Keeping the page that failed makes fixing the selector a five-minute job.

Scaling up: the production architecture

A single script with cron is a good start, but it breaks down as soon as the list grows. One slow site blocks everything behind it, a crash halfway through loses the rest of the run, you cannot add capacity without rewriting, and there is nowhere to hold an alert that failed to send. The fix is to stop thinking of the monitor as one program and split it into small, independent jobs connected by a queue.

The big picture

Price monitoring architecture: Celery Beat enqueues jobs into Redis, scrape workers save to PostgreSQL, and alert workers send messages to email, Telegram or Slack Celery Beat scheduler, every 5 min enqueue_due_products Redis broker + shared state queue: scrape rate:* limits alert:* dedupe queue: alerts Scrape workers fetch, parse, save scale to N workers Alert workers decide, send, retry price dropped: enqueue alert price + history PostgreSQL products, price_history alert_rules, alert_log Channels Email, Telegram, Slack
Beat claims due products from PostgreSQL and queues one job each. Scrape workers read and write PostgreSQL directly, and hand price drops to the alerts queue. Alert workers read the rules and deliver the messages.

Every box can fail or be restarted without losing work, and every box except the database and Redis can be duplicated to add capacity.

What each part is responsible for

  • PostgreSQL is the source of truth: which products exist, when each is next due, the price history, and the alert rules. Nothing important lives only in memory.
  • Celery Beat is the clock. It does not scrape anything. Every few minutes it asks the database which products are due and puts one small job per product on the queue.
  • Redis is the broker: the waiting room between the scheduler and the workers. It also holds short-lived coordination data that does not belong in PostgreSQL.
  • Scrape workers pick up one job at a time, fetch and parse one product page, save the result, and hand off an alert job if the price dropped.
  • Alert workers are a separate pool that only sends messages. A slow email server can never hold up scraping, and a scraping backlog can never delay alerts.

Why a queue instead of a loop

The queue turns "scrape 5,000 products" into 5,000 independent jobs. That gives you:

  • Isolation. One page that hangs or fails only affects its own job.
  • Retries per job. A blocked request is retried later with backoff, instead of being retried in a tight loop or lost.
  • Parallelism you control. Run 4 workers or 40, on one machine or several, with no code change.
  • Back-pressure and visibility. If the queue keeps growing, you are not keeping up, and you can see it.

What Redis is doing here

Redis plays four distinct roles in this design:

  1. Message broker. Celery stores each waiting task in a Redis list named after its queue (scrape, alerts). Workers pop from it.
  2. Per-domain rate limiting. A counter with an expiry lets every worker, on every machine, share one request budget per website. This is what keeps you polite at scale.
  3. Alert de-duplication. A key that is set only if it does not exist already stops the same price drop from being announced twice.
  4. Optional result store and cache. Task results, or a hash of the last page body so unchanged pages can skip parsing.

Two Redis settings matter in production. Turn on persistence (appendonly yes) so queued jobs survive a restart, and set maxmemory-policy noeviction so Redis refuses writes instead of silently evicting queue keys when memory is full.

What Celery is doing here

Celery is the framework that turns plain Python functions into queued, retryable jobs. It gives you task routing to named queues, automatic retries with exponential backoff and jitter, a scheduler (Beat), and worker processes you scale by starting more of them. You write normal functions and decorate them.

Building it: schema additions

The scheduler needs to know when each product is next due, and the alert system needs rules and a log.

ALTER TABLE products
  ADD COLUMN check_interval        INTERVAL    DEFAULT '6 hours',
  ADD COLUMN next_check_at         TIMESTAMPTZ DEFAULT now(),
  ADD COLUMN consecutive_failures  INT         DEFAULT 0;

CREATE TABLE alert_rules (
  id            SERIAL PRIMARY KEY,
  product_id    INT REFERENCES products(id),
  drop_percent  NUMERIC(5,2),    -- alert on a drop of at least this much
  target_price  NUMERIC(12,2),   -- or when the price falls to this level
  channel       TEXT NOT NULL,   -- 'email', 'telegram', 'slack'
  destination   TEXT NOT NULL    -- address, chat id or webhook URL
);

CREATE TABLE alert_log (
  id          BIGSERIAL PRIMARY KEY,
  rule_id     INT REFERENCES alert_rules(id),
  old_price   NUMERIC(12,2),
  new_price   NUMERIC(12,2),
  sent_at     TIMESTAMPTZ DEFAULT now()
);

The Celery app, queues and schedule

# celery_app.py
from celery import Celery

app = Celery(
    "monitor",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1",
)

app.conf.update(
    timezone="UTC",
    task_acks_late=True,               # ack only after the task finishes
    task_reject_on_worker_lost=True,   # re-queue if a worker dies mid-task
    worker_prefetch_multiplier=1,      # do not hoard tasks; keeps load even
    task_routes={
        "tasks.scrape_product": {"queue": "scrape"},
        "tasks.send_alert":     {"queue": "alerts"},
    },
    beat_schedule={
        "enqueue-due-products": {
            "task": "tasks.enqueue_due_products",
            "schedule": 300.0,         # every 5 minutes
        },
    },
)

task_acks_late is the important one. By default a task is acknowledged when a worker receives it, so a crash mid-scrape would lose the job. With late acknowledgement the job goes back on the queue instead. The trade-off is that a task can occasionally run twice, so every task must be safe to repeat. Ours is: saving the same price twice writes nothing the second time.

The scheduler job: claim the products that are due

One SQL statement both selects the due products and pushes their next check time forward, so two overlapping runs can never enqueue the same product. FOR UPDATE SKIP LOCKED makes concurrent claims skip rows another transaction already holds.

# tasks.py
from celery_app import app

@app.task(name="tasks.enqueue_due_products")
def enqueue_due_products():
    with conn, conn.cursor() as cur:
        cur.execute(
            """UPDATE products
               SET next_check_at = now() + check_interval
               WHERE id IN (
                   SELECT id FROM products
                   WHERE next_check_at <= now()
                   ORDER BY next_check_at
                   LIMIT 5000
                   FOR UPDATE SKIP LOCKED
               )
               RETURNING id, url"""
        )
        due = cur.fetchall()

    for product_id, url in due:
        scrape_product.delay(product_id, url)
    return len(due)

Sharing a rate limit across every worker

Celery's built-in rate_limit applies per worker, so ten workers would send ten times the traffic you intended. A shared counter in Redis fixes that: one budget per domain per minute, no matter how many workers exist.

import time
import redis

r = redis.Redis.from_url("redis://localhost:6379/2")

def acquire_slot(domain, per_minute=20):
    key = f"rate:{domain}:{int(time.time() // 60)}"
    count = r.incr(key)
    if count == 1:
        r.expire(key, 70)
    return count <= per_minute

The scrape task: retries, back-off and hand-off

In this version fetch() makes one attempt and raises on a bad status, so Celery owns the retry policy instead of a loop inside the function.

import random
import requests
from urllib.parse import urlparse
from celery_app import app

def fetch(url):
    resp = session.get(url, timeout=20)
    if resp.status_code == 404:
        return None                     # gone, do not retry
    resp.raise_for_status()             # 429 / 5xx raise HTTPError
    return resp.text

class ScrapeTask(app.Task):
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        # called once every retry has been used up
        mark_failed(args[0])            # bump consecutive_failures, flag at 5

@app.task(
    name="tasks.scrape_product",
    base=ScrapeTask,
    autoretry_for=(requests.RequestException,),
    retry_backoff=True,                 # 1s, 2s, 4s, 8s ...
    retry_backoff_max=600,              # never wait more than 10 minutes
    retry_jitter=True,                  # spread retries out randomly
    max_retries=5,
)
def scrape_product(product_id, url):
    domain = urlparse(url).netloc
    if not acquire_slot(domain):
        # budget for this site is used up: try again shortly as a fresh task
        scrape_product.apply_async((product_id, url), countdown=random.randint(5, 20))
        return "deferred"

    html = fetch(url)
    item = extract_product(html) if html else None
    if item is None or item["price"] is None:
        record_parse_failure(product_id)
        return "parse_failed"

    old = save(url, item)
    for rule_id in matching_rule_ids(product_id, old, item["price"]):
        send_alert.delay(rule_id, product_id, str(old), str(item["price"]))
    return "ok"

The alert system

Alerts are where a monitor earns trust or loses it. Too many and people mute them, too few and the tool is useless. Treat alerting as its own system with three jobs: decide, deliver, and never repeat yourself.

1. Decide: rules, not hard-coded thresholds

Store each person's conditions in alert_rules and match them when a price changes. A rule can be a percentage drop, a target price, or both.

def matching_rule_ids(product_id, old, new):
    if old is None or new >= old:
        return []                       # first sighting, or not a drop
    drop = (old - new) / old * 100
    with conn, conn.cursor() as cur:
        cur.execute(
            """SELECT id FROM alert_rules
               WHERE product_id = %s
                 AND (drop_percent IS NULL OR drop_percent <= %s)
                 AND (target_price IS NULL OR target_price >= %s)""",
            (product_id, drop, new),
        )
        return [row[0] for row in cur.fetchall()]

2. Deliver: a separate queue, retries and a channel adapter

The alert worker only sends messages. Keep each channel behind a tiny adapter so adding Telegram or Slack later does not touch the monitoring code.

@app.task(
    name="tasks.send_alert",
    autoretry_for=(Exception,),
    retry_backoff=True,
    max_retries=5,
)
def send_alert(rule_id, product_id, old, new):
    # 3. never repeat: one alert per rule per price per day
    key = f"alert:{rule_id}:{new}"
    if not r.set(key, 1, nx=True, ex=86400):
        return "duplicate"
    try:
        rule = load_rule(rule_id)
        product = load_product(product_id)
        CHANNELS[rule.channel](rule.destination, product, old, new)
        log_alert(rule_id, old, new)
    except Exception:
        r.delete(key)                   # allow the retry to send it
        raise

3. Never repeat yourself

SET key value NX EX 86400 succeeds only for the first caller, so if two workers process the same drop, or a task is re-delivered after a crash, only one message goes out. If delivery fails, the key is removed so the retry can still send it.

Keeping alerts useful

  • Ignore noise. Require a minimum drop (for example 3%) and reject values that look like parse errors, such as a price of zero.
  • Batch when needed. For many products, send one daily digest instead of dozens of messages.
  • Alert on the monitor itself. A site that fails five checks in a row is a different alert, aimed at you, not at the customer.

Running and operating it

Start separate worker pools for each queue, so you can size them independently, plus one Beat process:

celery -A celery_app worker -Q scrape -c 8 -n scrape@%h --loglevel=INFO
celery -A celery_app worker -Q alerts -c 2 -n alerts@%h --loglevel=INFO
celery -A celery_app beat --loglevel=INFO

# optional web dashboard (pip install flower)
celery -A celery_app flower

Run exactly one Beat process. Two Beats would enqueue every schedule twice. Scrape workers, on the other hand, you can start on as many machines as you like, all pointing at the same Redis.

As a starting point, here is the same setup in Docker Compose:

services:
  redis:
    image: redis:7
    command: ["redis-server", "--appendonly", "yes", "--maxmemory-policy", "noeviction"]
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: change-me
  worker-scrape:
    build: .
    command: celery -A celery_app worker -Q scrape -c 8 -n scrape@%h
    depends_on: [redis, db]
  worker-alerts:
    build: .
    command: celery -A celery_app worker -Q alerts -c 2 -n alerts@%h
    depends_on: [redis, db]
  beat:
    build: .
    command: celery -A celery_app beat
    depends_on: [redis]

Things that will bite you if you skip them

  • Redis visibility timeout. With the Redis broker, a task that is not acknowledged within the visibility timeout (one hour by default) is delivered again. Keep every task well under it, or raise broker_transport_options = {"visibility_timeout": ...}.
  • Idempotent tasks. Because late acknowledgement can run a task twice, make saving and alerting safe to repeat, as shown above.
  • Watch the queue depth. redis-cli LLEN scrape shows how many jobs are waiting. A number that keeps climbing means you need more workers or a longer interval.
  • I/O-bound work and concurrency. Scraping mostly waits on the network, so a thread or gevent pool can run many more tasks per worker than the default prefork pool. Test your database driver with whichever pool you choose.
  • Back up PostgreSQL, not Redis. The queue can be rebuilt from the database, because the scheduler re-enqueues anything that is due. The history cannot be rebuilt.

When to use this and when not to

For a few dozen products, the cron script above is enough, and simpler is better. Move to the queue-based design when you track hundreds or thousands of products, need alerts to different people, scrape several sites with different limits, or need to keep working while individual pages fail.

Scraping responsibly

Check the site's terms and robots.txt, keep your request rate low, and collect only the public product data you need. My guide to ethical web scraping and legal considerations goes through this in more detail.

Where to take it next

With history in PostgreSQL you can chart price movement, expose it through a small REST API, compare the same product across several shops, or trigger repricing rules in your own store. If you would rather have this built and maintained for you, see my web scraping services or get in touch with the sites and fields you need.

Written by

Aashish Timsina

Web scraper and backend developer based in Nepal — building automated data extraction pipelines, SERP scrapers, and scalable Node.js systems.

Need a Custom Web Scraping Solution?

Based in Nepal and working with clients worldwide — let's build a reliable data extraction pipeline tailored to your business.