@dag(start_date=datetime(2023,1,1), schedule=None, catchup=False) def xcom_exclusive_pipeline():
Last updated: 2025 Applies to Apache Airflow 2.0+ airflow xcom exclusive
A recurring theme in official Airflow documentation is the strict recommendation to use XComs only for . Because XComs are stored directly in the metadata database (such as PostgreSQL or MySQL), overloading them with large datasets—like massive Pandas DataFrames—can lead to severe performance degradation. Best Practices — Airflow 3.2.0 Documentation airflow xcom exclusive
class S3XCom(BaseXCom): @staticmethod def serialize(value): if size_of(value) > 1_000_000: s3_key = upload_to_s3(value) return "__s3_uri": s3_key return value airflow xcom exclusive
Imagine a financial DAG with 10 tasks: fetch_transactions → validate → feature_engineering → fraud_model → alert .