Snowflake has released a new AI-powered migration tool within its CoCo coding agent that converts Apache Spark codebases to Snowflake's Snowpark Connect engine automatically.
Customers running workloads on Snowflake's engine see up to 5.1x faster performance and 42% lower costs compared to managed Spark services.
The tool migrates entire code directories in minutes from a single prompt, handling code rewriting, dependency updates, and validation—work that previously required hours of manual effort.
What happened
Snowflake released a spark-migration skill within its CoCo data-native AI coding agent that automatically converts Apache Spark and PySpark codebases to Snowpark Connect—Snowflake's engine for running Spark workloads. Users can now migrate an entire directory of files with a single conversational prompt instead of manual refactoring.
Why it matters
Running Spark traditionally requires managing clusters, patching infrastructure, and tuning environments. Moving to Snowflake eliminates that overhead; customers report up to 5.1x faster performance and 42% lower costs. The skill scans for incompatibilities, rewrites unsupported patterns, flags manual review items, and optionally validates the migrated code against the original—a workflow that previously took hours of manual effort now runs in minutes.
What to watch
The skill is bundled with CoCo at no additional setup cost and auto-activates when you mention Spark, PySpark, or code migration. It accepts Python, Scala, Java source files, and notebook files, and can also analyze and complete migrations started by other tools.
Snowflake's new spark-migration skill, bundled with its CoCo data-native AI coding agent, automates the process of converting Apache Spark and PySpark codebases to Snowpark Connect—Snowflake's engine for running Spark workloads natively on Snowflake. The tool is invoked through a simple conversational prompt; users can ask CoCo to "migrate this file to Snowpark Connect" or scope the effort with "assess the compatibility and level of effort required to migrate this codebase," and the agent handles the rest.
Behind the single prompt lies a multi-stage agentic workflow that mirrors the steps a migration expert would take. First, the skill performs an assessment and builds an inventory, scanning every source file for compatibility issues including RDD operations, certain UDF serialization patterns, and unsupported file formats. Each file, dependency, and unique API call are cataloged against a knowledge base built by Snowflake's engineering team. Next, it applies targeted fixes by dispatching parallel agents to rewrite incompatible patterns to DataFrame equivalents, update imports and session creation, and flag anything requiring manual review—such as validating reads and writes—with a detailed explanation. The skill then generates predictable reporting: issue logs, inventory reports, and a validation of the overall migration state, giving full visibility into every change. Finally, if the developer opts in, the skill validates the converted code's functionality by running both the original PySpark code and the migrated Snowpark Connect code against the same synthetic data, comparing their outputs table-by-table to catch behavioral differences. Any fixes discovered during validation are automatically committed back to the deliverable branch.
The spark-migration skill accepts Python, Scala, and Java code files, as well as build files and notebook files. It auto-activates when the developer mentions Spark, PySpark, or code migration, or can be invoked directly with the command spark-migration. Developers can also use the skill to validate an existing codebase before migration or to complete migrations started by other tools. Because it flags anything requiring manual review, the developer retains control over what changes.
According to Snowflake, moving Spark workloads to Snowflake delivers up to 5.1x faster performance and 42% lower costs, based on customer production use cases and proof-of-concept exercises comparing Snowpark versus managed Spark services between November 2022 and May 2025. Running Spark traditionally requires tuning clusters, patching infrastructure, and managing dependency environments—all of which disappear when workloads run on Snowflake's managed engine. The spark-migration skill is included with CoCo at no additional setup cost, and Snowflake reports that work once requiring hours of manual effort now takes minutes.
Snowflake's move to embed AI-assisted code migration within CoCo addresses a long-standing friction point for enterprises: moving workloads between data platforms. Spark is ubiquitous in data engineering but operationally expensive—clusters require constant tuning, patching, and infrastructure management. By offering a path to run existing Spark code on Snowflake's managed engine, the company removes that operational burden while claiming substantial performance and cost gains (up to 5.1x speed, 42% cost reduction). The key innovation is not the migration itself, but the depth of the AI agent's workflow: it scans for compatibility issues against a Snowflake-maintained knowledge base, applies targeted fixes in parallel, generates auditable reports, and optionally validates output correctness by running both versions against synthetic data. This end-to-end agentic approach—rather than a simple code rewriter—is designed to handle real-world complexity: RDD operations, UDF serialization patterns, unsupported file formats, and mixed-language codebases. The result is that what once required hours of manual effort and cross-file searching now runs in minutes from a single prompt, with the developer remaining in control of flagged items.
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytime
Ask AI anything about this article. Q&As are published on this page for other readers too.
Silicon Motion announced a private placement of US$1 billion in aggregate principal amount of 0% convertible s…
Sundar Pichai, CEO of Alphabet and Google, announced on August 11 that the Gemini app's monthly active users (…
OpenAI's ChatGPT crossed 1 billion monthly users some time ago and hit 1 billion weekly users in July, accordi…
OpenAI's Daybreak Red (GPT-5.6 Cyber) and Daybreak Blue (GPT-5.6 Sol) are now available to eligible customers…
IBM and Together AI have signed a $240 million agreement to build and operate an Nvidia-powered AI inference c…
NVIDIA and partners released multiple open-source AI models optimized for local execution throughout August, i…
The AI news that matters, in one minute each morning.
Sign up free