Home / Blog / Amazon Price Tracking in Apache Airflow: XCom, Variables and the catchup Trap

Amazon Price Tracking in Apache Airflow: XCom, Variables and the catchup Trap

Run a daily Amazon Buy Box price check as an Apache Airflow TaskFlow DAG. The SellerMagnet side is unchanged from the Python quickstart (sellermagnet.py/retry.py, byte-identical); the real content is Airflow-specific: the API key as a Connection instead of a bare environment variable, the price baseline in a Variable (the only state the DAG needs), XCom carrying just a small dict between tasks (never the raw response), and a catchup default that quietly flips between Airflow 2.x and Airflow 3.

November 20, 2026
6 min read
SellerMagnet Team
Share & Bookmark
Terminal showing an Apache Airflow DAG checking an Amazon Buy Box price via the SellerMagnet API

A daily Amazon Buy Box check is a natural fit for Apache Airflow's TaskFlow API: three small @task functions (fetch the price, compare it to a baseline, notify on a drop), wired together with plain Python calls and run on a @daily schedule. sellermagnet.py/retry.py are unchanged from the Python quickstart - the part worth getting right is Airflow-specific: where the API key lives, where the price baseline persists between runs, what XCom is and is not for, and a scheduling default that is not even the same across Airflow versions.

Key Takeaways

  • Store the API key as an Airflow Connection (sellermagnet), not a bare os.environ read: Connections are Fernet-encrypted in the metadata database and masked in task logs the moment they are accessed; a plain env var read gets neither.
  • Persist the price baseline in an Airflow Variable, not XCom: XCom from one scheduled run is not available to the next run by default, while a Variable lives in the same metadata database across every run.
  • Push only a small derived dict through XCom between tasks, never the raw SellerMagnet response or the API key - XCom has no size limit of its own, only the metadata database's (1 GiB on Postgres in Airflow 2, 256 MiB in Airflow 3, 64 MiB on MySQL).
  • Set catchup=False explicitly on the DAG. Airflow 1.x/2.x defaults catchup_by_default to True (one backfilled DagRun per missed interval since start_date); Airflow 3's stable docs state the opposite default. Do not rely on either.
  • retry.py's own 429/500/502/503 retry loop lives INSIDE one task try; Airflow's separate retries/retry_delay in default_args only re-runs the whole task, and only after that inner loop still raises (or on a 401/403, which neither layer retries).

The DAG

Three @task functions inside one @dag-decorated factory: fetch_price calls retry.py's get_with_retry exactly like the quickstart, compare_and_alert reads and writes the Variable baseline and decides whether to alert, and notify prints the result (a Slack webhook call is a one-line swap, same shape as the Buy Box lost Slack alert post). buyBoxInfo.price can be the string "N/A" when no offer is featured - mapped to None, never 0, so a missing offer never reads as a free product or a false alert.

price_tracker_dag.py

"""price_tracker_dag.py - a daily Airflow DAG that polls one ASIN's Buy Box price via the
SellerMagnet API, keeps the last-seen price in an Airflow Variable (the only piece of state
the DAG needs), and alerts when the price drops. sellermagnet.py / retry.py are byte-identical
to the tested Python quickstart; this file only adds the Airflow-specific wiring (TaskFlow API,
XCom between tasks, Variable as the tiny persisted baseline, a Connection for the API key).
"""
from __future__ import annotations

import os
from datetime import datetime, timedelta

from airflow.decorators import dag, task
from airflow.hooks.base import BaseHook
from airflow.models import Variable

from retry import get_with_retry
from sellermagnet import ApiError, new_session

ASINS = ["B0CL61F39H"]
MARKETPLACE_ID = "ATVPDKIKX0DER"
CONN_ID = "sellermagnet"


def _session():
    """Read the API key from an Airflow Connection, not a bare env var: Connections are the
    purpose-built place for a third-party credential (Fernet-encrypted at rest in the metadata
    DB, or resolved straight from AIRFLOW_CONN_SELLERMAGNET with no DB row at all), and
    Airflow masks a Connection's password field in task logs the moment the Connection is
    accessed - a plain os.environ read gets no such masking.
    """
    conn = BaseHook.get_connection(CONN_ID)
    os.environ["SELLERMAGNET_API_KEY"] = conn.password
    return new_session()


def build_dag(*, retries: int = 3, retry_delay: timedelta = timedelta(minutes=2)):
    """Factory so the retry timing can be shortened for a real local test run (dag.test()
    really sleeps retry_delay between an up_for_retry task and its next attempt - there is no
    mock-clock hook for it) while the deployed DAG below keeps production-realistic values.
    """

    @dag(
        dag_id="sellermagnet_price_tracker",
        description="Daily Buy Box price check via the SellerMagnet API, alerts on a drop.",
        schedule="@daily",
        start_date=datetime(2026, 1, 1),
        catchup=False,  # see the catchup section below - the default for this flips between
                        # Airflow 2.x (True) and Airflow 3 (False), so a bare DAG with a past
                        # start_date can mean one backfilled DagRun or dozens, by version alone.
        max_active_runs=1,
        default_args={
            "retries": retries,
            "retry_delay": retry_delay,  # Airflow's own default is 300s (5 min) if left unset; 2 min
                                          # here is a deliberate shorter floor, not an unexplained value
            "retry_exponential_backoff": True,
            "max_retry_delay": timedelta(minutes=20),
            "execution_timeout": timedelta(minutes=3),  # retry.py's own retry loop can legitimately
                                                         # hold the task open close to the API's ~120s
                                                         # scrape allowance; leaving this unset means
                                                         # Airflow's default of no timeout at all
        },
        tags=["sellermagnet"],
    )
    def sellermagnet_price_tracker():

        @task
        def fetch_price(asin: str) -> dict:
            """One lookup. retry.py already retries 429/500/502/503 INSIDE this call (its own
            4-attempt cap with backoff); this task only re-runs from Airflow's side if the
            whole call still raises after that cap is exhausted, or on 401/403 (never retried
            by either layer - a bad key does not deserve 3 more billed attempts).
            """
            session = _session()
            data = get_with_retry(session, "amazon-product-lookup", asin=asin, marketplaceId=MARKETPLACE_ID)
            price = data["productInfo"]["buyBoxInfo"].get("price")
            return {"asin": asin, "price": price if isinstance(price, (int, float)) else None}

        @task
        def compare_and_alert(current: dict) -> dict:
            """XCom carries only this small dict between tasks, never the raw API response or
            the API key: XComs are serialized straight into Airflow's metadata database (no
            size limit of their own, but capped by the backend - 1 GiB on Postgres in Airflow
            2, 256 MiB in Airflow 3, 64 MiB on MySQL), so pushing a whole SellerMagnet payload
            through XCom on every run is the wrong place to put it even when it fits today.
            """
            var_key = f"sellermagnet_last_price_{current['asin']}"
            try:
                baseline = float(Variable.get(var_key))
            except KeyError:
                baseline = None
            price = current["price"]
            alert = baseline is not None and price is not None and price < baseline
            if price is not None:
                Variable.set(var_key, price)
            return {"asin": current["asin"], "price": price, "baseline": baseline, "alert": alert}

        @task
        def notify(result: dict) -> str:
            if result["alert"]:
                msg = f"ALERT {result['asin']}: {result['baseline']} -> {result['price']}"
            else:
                msg = f"ok {result['asin']}: {result['price']} (baseline {result['baseline']})"
            print(msg)
            return msg

        for asin in ASINS:
            notify(compare_and_alert(fetch_price(asin)))

    return sellermagnet_price_tracker()


price_tracker_dag = build_dag()

Two defaults worth overriding rather than leaving as-is: Airflow's own retry_delay default is 300 seconds (5 minutes) when default_args omits it, shortened here to 2 minutes deliberately; and execution_timeout has no default at all, so a hung task would otherwise run forever. retry.py's own retry loop can legitimately hold a task open for a while inside the SellerMagnet API's documented request allowance, so execution_timeout=timedelta(minutes=3) gives it headroom without leaving it unbounded.

The API key: a Connection, not os.environ

Every other automation post in this series reads SELLERMAGNET_API_KEY straight from os.environ, because that is all the platform offers. Airflow has a purpose-built place for a third-party credential: a Connection. Defined through the UI or CLI, its password field is Fernet-encrypted in the metadata database and masked in task logs the moment a task calls BaseHook.get_connection(...). Defined instead through the AIRFLOW_CONN_<CONN_ID> environment-variable convention, it skips the database and the UI entirely and resolves only on the worker process that needs it - a clean way to hand the key to workers via Docker or Kubernetes secrets without it ever reaching the scheduler or webserver.

Defining the Connection two ways

# via the CLI, stored (Fernet-encrypted) in the metadata database
airflow connections add sellermagnet \
  --conn-type generic --conn-host https://sellermagnet-api.com \
  --conn-password "$SELLERMAGNET_API_KEY"

# or via an env var, resolved only on the worker, never stored in the DB/shown in the UI
export AIRFLOW_CONN_SELLERMAGNET='{"conn_type": "generic", "password": "your-key", '\
'"host": "https://sellermagnet-api.com"}'
SellerMagnet status codes, billing and this DAG's reaction.
StatusMeaningBilledReaction
200successYesuse data
401 / 403key or creditsNofail the task, Airflow retries the whole task
404not on this marketplaceYesstore as missing, do not retry
429concurrency capNowait Retry-After, retried inside retry.py
500internal errorNo - refundedretry once inside retry.py
502 / 503upstream / breakerYeswait, retry inside retry.py, capped at 4
Blueprint flow diagram of the three Airflow tasks, the Variable baseline and the Connection
XCom carries only the small result dict between tasks; the Variable, not XCom, is what survives into tomorrow's run.

catchup: a default that is not even the same across Airflow versions

start_date=datetime(2026, 1, 1) on a DAG deployed weeks later reads as harmless until catchup enters the picture, and whether it backfills by default genuinely depends on which Airflow major version is running.

Table comparing the catchup default between Airflow 1.x/2.x and Airflow 3
Set catchup explicitly on every DAG; do not rely on either version's default.

XCom is for the small stuff, not the response

XCom values are serialized straight into Airflow's own metadata database, so "how big can an XCom be" is really "how big can one row be on this backend" - there is no separate Airflow-side cap. A full SellerMagnet lookup response fits easily today; it is still the wrong habit, because every additional field pushed through XCom is one more thing stored forever in a table Airflow itself queries constantly.

XCom's practical size ceiling, by metadata database backend.
BackendPractical XCom ceiling
SQLite (default local/dev)about 1 GB
PostgreSQL, Airflow 2.xabout 1 GiB
PostgreSQL, Airflow 3about 256 MiB
MySQLmax_allowed_packet, 64 MiB by default

dag.test() against a fake HTTP layer (3 real runs, real Variable, real XCom)

$ airflow dags test sellermagnet_price_tracker 2026-01-01   # run 1, no baseline yet
ok B0CL61F39H: 444.99 (baseline None)
  DagRun state: success  (1 SellerMagnet call billed, baseline -> 444.99)
$ airflow dags test sellermagnet_price_tracker 2026-01-02   # run 2, price drops
ALERT B0CL61F39H: 444.99 -> 399.0
  DagRun state: success  (ALERT B0CL61F39H: 444.99 -> 399.0, baseline -> 399.0)
$ airflow dags test sellermagnet_price_tracker 2026-01-03   # run 3, 1 bare 500 then 200
ok B0CL61F39H: 410.0 (baseline 399.0)
  DagRun state: success  (retry.py refunded-and-retried the 500 by itself, 2 calls, no Airflow-level retry)
  billed SellerMagnet calls this run: 2

Frequently Asked Questions

Does this DAG need its own retry logic, or does retry.py already handle it?

retry.py already retries a 429/500/502/503 up to 4 times inside one task try, exactly like the Python quickstart. The task-level retries set in default_args is a separate, outer layer: it only matters when the whole call still raises after that cap, or on a 401/403 that retry.py never retries at all - then Airflow re-runs the ENTIRE task after retry_delay, not just the failed HTTP call.

Where should the price baseline live between runs?

An Airflow Variable. A DAG run's own XCom values do not survive into the NEXT scheduled run by default, and a Cron Job or Function has no persistent disk either - a Variable, stored in the same metadata database Airflow already has, is the smallest thing that persists across runs without adding a separate datastore.

Is it safe to push the full SellerMagnet lookup response through XCom?

It is allowed, but it is the wrong default: XCom data is serialized straight into the metadata database with no size limit of its own, only the backend's row/column limits (see the table above), and masking of sensitive values only applies to Connections and Variables accessed directly, never to whatever a task happens to push as XCom. Push the small derived dict (asin, price, alert), never the raw response or the API key.

Why read the API key from a Connection instead of os.environ directly?

An Airflow Connection is the place built for a third-party credential: stored in the metadata database, Fernet-encrypted at rest, and masked in task logs the moment the Connection is accessed. A Connection defined via the AIRFLOW_CONN_<ID> environment variable convention skips the database and the UI entirely and is resolved only on the worker - useful for injecting the key through Docker/Kubernetes secrets without ever writing it to the scheduler or webserver.

What happens if I forget catchup=False on a DAG with a start_date weeks in the past?

It depends on the Airflow version, which is exactly the trap: Airflow 1.x/2.x's documented default is catchup_by_default=True, so the scheduler backfills one DagRun per missed daily interval between start_date and now - weeks of extra, billed SellerMagnet calls on first deploy. Airflow 3's stable docs state the opposite default (catchup False via the DAG's timetable). Set catchup explicitly either way instead of relying on whichever default a given Airflow version ships with.

Bottom line: the SellerMagnet half of this DAG is the same client/retry pair as every other post in this series. What Airflow adds is a Connection for the credential, a Variable for the one piece of state that needs to survive to tomorrow, XCom for the small handoff between today's three tasks, and a catchup flag that is worth setting explicitly rather than trusting to whichever Airflow major version happens to be running. The retry policy itself is covered in full in the rate limits and retries guide, and the same price-watch pattern on two other schedulers is in the AWS Lambda and GitHub Actions posts. Every endpoint is documented on the API documentation page.

Related Articles

Continue exploring topics related to this article.

Terminal window showing the Amazon Buy Box lost alert a Python script sent as an SMS via Twilio
Amazon API Buy Box Alerts

Amazon Buy Box Lost Alert via SMS: Poll Offers, Confirm, Then Text Yourself with Twilio

Get an Amazon Buy Box lost alert as a text message: poll the live offer listing for each ASIN, compare the featured offer's sellerId with yours, confirm the change on a second check, then send one SMS per run with Twilio's Messages API. Covers the one thing a webhook alert does not have to: Twilio's 2xx answer confirms the message was queued, not that it was delivered.

Nov 19, 2026 Read
Terminal showing a Python price-watch script pushing Amazon Buy Box metrics to a Prometheus Pushgateway for Grafana
Python Amazon API Monitoring

Amazon Buy Box Price Monitoring in Grafana via Prometheus Pushgateway

Push Amazon Buy Box price, availability and offer-count metrics from the SellerMagnet API to a Prometheus Pushgateway, then graph and alert on them in Grafana. sellermagnet.py / retry.py need zero changes from the Python quickstart. The real trap: using push_to_gateway (PUT, a full replace) instead of pushadd_to_gateway (POST, a merge) is what makes a dropped ASIN disappear from the dashboard instead of freezing there forever.

Nov 19, 2026 Read
Terminal running a Python script that computes Amazon's FBA dimensional weight from the lookup response and shows which weight governs shipping
Python Amazon API Reference

Amazon FBA Dimensional Weight Calculator via the API: Volume / 139 in Python

Amazon bills some FBA items on dimensional weight, not scale weight: volume (length x width x height in inches) divided by 139, with a 2-inch floor on bulky items. This guide pulls an item's size and weight from one product lookup, computes the dimensional weight the same way Amazon's own help pages do, and flags which weight actually governs shipping fees.

Nov 19, 2026 Read
Back to Blog

Ready to Extract Amazon Data at Scale?

Start building with SellerMagnet API today. Real-time product data, competitive pricing, and review analytics at your fingertips.

150 free API credits • No credit card required • Cancel anytime