{"id":438,"date":"2026-05-04T00:00:00","date_gmt":"2026-05-04T00:00:00","guid":{"rendered":"https:\/\/wordpress.securinsight.ca\/index.php\/2026\/05\/04\/pipelines-pipelines-and-r2-data-catalog-now-supported-in-terraform\/"},"modified":"2026-05-04T00:00:00","modified_gmt":"2026-05-04T00:00:00","slug":"pipelines-pipelines-and-r2-data-catalog-now-supported-in-terraform","status":"publish","type":"post","link":"https:\/\/wordpress.securinsight.ca\/index.php\/2026\/05\/04\/pipelines-pipelines-and-r2-data-catalog-now-supported-in-terraform\/","title":{"rendered":"Pipelines &#8211; Pipelines and R2 Data Catalog now supported in Terraform"},"content":{"rendered":"<p><a href=\"https:\/\/developers.cloudflare.com\/pipelines\/\">Cloudflare Pipelines<\/a> ingests streaming data via <a href=\"https:\/\/developers.cloudflare.com\/workers\/\">Workers<\/a> or HTTP endpoints, transforms it with SQL, and writes it to <a href=\"https:\/\/developers.cloudflare.com\/r2\/\">R2<\/a> as Apache Iceberg tables. <a href=\"https:\/\/developers.cloudflare.com\/r2\/data-catalog\/\">R2 Data Catalog<\/a> manages those Iceberg tables, compaction, and compatibility with query engines like <a href=\"https:\/\/developers.cloudflare.com\/r2-sql\/\">R2 SQL<\/a>, <a href=\"https:\/\/developers.cloudflare.com\/r2\/data-catalog\/config-examples\/spark-scala\/\">Spark<\/a>, and <a href=\"https:\/\/developers.cloudflare.com\/r2\/data-catalog\/config-examples\/duckdb\/\">DuckDB<\/a>.<\/p>\n<p>You can now create and manage both products using Terraform, supported in the <a href=\"https:\/\/registry.terraform.io\/providers\/cloudflare\/cloudflare\/latest\/docs\" target=\"_blank\">Cloudflare Terraform provider v5.19.0<\/a>.<\/p>\n<p>This adds four new resources that let you define your entire data pipeline as infrastructure-as-code: a data catalog, a stream for ingestion, a sink that writes to R2 Data Catalog or R2, and a pipeline that connects them with SQL.<\/p>\n<p>The new Terraform resources are:<\/p>\n<ul>\n<li><a href=\"https:\/\/registry.terraform.io\/providers\/cloudflare\/cloudflare\/latest\/docs\/resources\/r2_data_catalog\" target=\"_blank\"><code>cloudflare_r2_data_catalog<\/code><\/a> \u2014 enable the data catalog on an R2 bucket<\/li>\n<li><a href=\"https:\/\/registry.terraform.io\/providers\/cloudflare\/cloudflare\/latest\/docs\/resources\/pipeline_stream\" target=\"_blank\"><code>cloudflare_pipeline_stream<\/code><\/a> \u2014 create a stream that receives events via HTTP or Worker bindings<\/li>\n<li><a href=\"https:\/\/registry.terraform.io\/providers\/cloudflare\/cloudflare\/latest\/docs\/resources\/pipeline_sink\" target=\"_blank\"><code>cloudflare_pipeline_sink<\/code><\/a> \u2014 create a sink that writes to R2 Data Catalog or R2<\/li>\n<li><a href=\"https:\/\/registry.terraform.io\/providers\/cloudflare\/cloudflare\/latest\/docs\/resources\/pipeline\" target=\"_blank\"><code>cloudflare_pipeline<\/code><\/a> \u2014 create a pipeline with SQL connecting a stream to a sink<\/li>\n<\/ul>\n<p>Here is a minimal example that creates a stream, an R2 Data Catalog sink, and a pipeline:<\/p>\n<div>\n<figure>\n<pre data-language=\"hcl\"><code class=\"language-hcl\"><div><div><span>resource<\/span><span> <\/span><span>\"cloudflare_pipeline_stream\"<\/span><span> <\/span><span>\"my_stream\"<\/span><span> <\/span><span>{<\/span><\/div><\/div><div><div><span>  <\/span><span>account_id<\/span><span> <\/span><span>=<\/span><span> var<\/span><span>.<\/span><span>cloudflare_account_id<\/span><\/div><\/div><div><div><span>  <\/span><span>name<\/span><span>       <\/span><span>=<\/span><span> <\/span><span>\"my_stream\"<\/span><\/div><\/div><div><div><span>  <\/span><span>format<\/span><span>     <\/span><span>=<\/span><span> <\/span><span>{<\/span><span> <\/span><span>type<\/span><span> <\/span><span>=<\/span><span> <\/span><span>\"json\"<\/span><span> <\/span><span>}<\/span><\/div><\/div><div><div><span>  <\/span><span>schema<\/span><span> <\/span><span>=<\/span><span> <\/span><span>{<\/span><\/div><\/div><div><div><span>    <\/span><span>fields<\/span><span> <\/span><span>=<\/span><span> [<\/span><span>{<\/span><\/div><\/div><div><div><span>      <\/span><span>name<\/span><span>     <\/span><span>=<\/span><span> <\/span><span>\"value\"<\/span><\/div><\/div><div><div><span>      <\/span><span>type<\/span><span>     <\/span><span>=<\/span><span> <\/span><span>\"json\"<\/span><\/div><\/div><div><div><span>      <\/span><span>required<\/span><span> <\/span><span>=<\/span><span> <\/span><span>true<\/span><\/div><\/div><div><div><span>    <\/span><span>}<\/span><span>]<\/span><\/div><\/div><div><div><span>  <\/span><span>}<\/span><\/div><\/div><div><div><span>  <\/span><span>http<\/span><span>           <\/span><span>=<\/span><span> <\/span><span>{<\/span><span> <\/span><span>enabled<\/span><span> <\/span><span>=<\/span><span> <\/span><span>true<\/span><span>, <\/span><span>authentication<\/span><span> <\/span><span>=<\/span><span> <\/span><span>false<\/span><span>, <\/span><span>cors<\/span><span> <\/span><span>=<\/span><span> <\/span><span>{}<\/span><span> <\/span><span>}<\/span><\/div><\/div><div><div><span>  <\/span><span>worker_binding<\/span><span> <\/span><span>=<\/span><span> <\/span><span>{<\/span><span> <\/span><span>enabled<\/span><span> <\/span><span>=<\/span><span> <\/span><span>false<\/span><span> <\/span><span>}<\/span><\/div><\/div><div><div><span>}<\/span><\/div><\/div><div><div>\n<\/div><\/div><div><div><span>resource<\/span><span> <\/span><span>\"cloudflare_pipeline_sink\"<\/span><span> <\/span><span>\"my_sink\"<\/span><span> <\/span><span>{<\/span><\/div><\/div><div><div><span>  <\/span><span>account_id<\/span><span> <\/span><span>=<\/span><span> var<\/span><span>.<\/span><span>cloudflare_account_id<\/span><\/div><\/div><div><div><span>  <\/span><span>name<\/span><span>       <\/span><span>=<\/span><span> <\/span><span>\"my_sink\"<\/span><\/div><\/div><div><div><span>  <\/span><span>type<\/span><span>       <\/span><span>=<\/span><span> <\/span><span>\"r2_data_catalog\"<\/span><\/div><\/div><div><div><span>  <\/span><span>format<\/span><span>     <\/span><span>=<\/span><span> <\/span><span>{<\/span><span> <\/span><span>type<\/span><span> <\/span><span>=<\/span><span> <\/span><span>\"parquet\"<\/span><span> <\/span><span>}<\/span><\/div><\/div><div><div><span>  <\/span><span>schema<\/span><span>     <\/span><span>=<\/span><span> <\/span><span>{<\/span><span> <\/span><span>fields<\/span><span> <\/span><span>=<\/span><span> [] <\/span><span>}<\/span><\/div><\/div><div><div><span>  <\/span><span>config<\/span><span> <\/span><span>=<\/span><span> <\/span><span>{<\/span><\/div><\/div><div><div><span>    <\/span><span>account_id<\/span><span> <\/span><span>=<\/span><span> var.cloudflare_account_id<\/span><\/div><\/div><div><div><span>    <\/span><span>bucket<\/span><span>     <\/span><span>=<\/span><span> <\/span><span>\"my-pipeline-bucket\"<\/span><\/div><\/div><div><div><span>    <\/span><span>table_name<\/span><span> <\/span><span>=<\/span><span> <\/span><span>\"my_table\"<\/span><\/div><\/div><div><div><span>    <\/span><span>token<\/span><span>      <\/span><span>=<\/span><span> var.catalog_token<\/span><\/div><\/div><div><div><span>  <\/span><span>}<\/span><\/div><\/div><div><div><span>}<\/span><\/div><\/div><div><div>\n<\/div><\/div><div><div><span>resource<\/span><span> <\/span><span>\"cloudflare_pipeline\"<\/span><span> <\/span><span>\"my_pipeline\"<\/span><span> <\/span><span>{<\/span><\/div><\/div><div><div><span>  <\/span><span>account_id<\/span><span> <\/span><span>=<\/span><span> var<\/span><span>.<\/span><span>cloudflare_account_id<\/span><\/div><\/div><div><div><span>  <\/span><span>name<\/span><span>       <\/span><span>=<\/span><span> <\/span><span>\"my_pipeline\"<\/span><\/div><\/div><div><div><span>  <\/span><span>sql<\/span><span>        <\/span><span>=<\/span><span> <\/span><span>\"INSERT INTO <\/span><span>${<\/span><span><span>cloudflare_pipeline_sink<\/span><span>.<\/span><\/span><span>my_sink<\/span><span>.<\/span><span><span>name<\/span><span>}<\/span><\/span><span> SELECT * FROM <\/span><span>${<\/span><span><span>cloudflare_pipeline_stream<\/span><span>.<\/span><\/span><span>my_stream<\/span><span>.<\/span><span><span>name<\/span><span>}<\/span><\/span><span>\"<\/span><\/div><\/div><div><div><span>}<\/span><\/div><\/div><\/code><\/pre>\n<div>\n<div><\/div>\n<\/div>\n<div><\/div>\n<\/figure>\n<\/div>\n<p>For a full end-to-end example that includes R2 bucket creation, data catalog setup, and scoped API token provisioning, refer to the <a href=\"https:\/\/developers.cloudflare.com\/pipelines\/reference\/terraform\/\">Pipelines Terraform documentation<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"<p>Cloudflare Pipelines ingests streaming data via Workers or HTTP endpoints, transforms it with SQL, and writes it to R2 as Apache Iceberg tables. R2 Data Catalog manages those Iceberg tables, compaction, and compatibility with query engines like R2 SQL, Spark, and DuckDB. You can now create and manage both products using Terraform, supported in the [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-438","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/posts\/438","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/comments?post=438"}],"version-history":[{"count":0,"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/posts\/438\/revisions"}],"wp:attachment":[{"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/media?parent=438"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/categories?post=438"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/tags?post=438"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}