Snowflake has released a new AI-powered code migration tool called spark-migration within its CoCo coding agent that automatically converts entire Apache Spark codebases to run on Snowflake's platform.
Customers using this migration have reported up to 5.1x faster performance and 42% lower costs in production environments.
The tool scans code for compatibility issues, applies fixes in parallel, generates detailed reports, and optionally validates that migrated code produces identical results to the original—all from a single prompt, reducing what previously took hours of manual work to minutes.
What happened
Snowflake introduced a spark-migration skill within CoCo, its data-native AI coding agent, that can scan, rewrite, and validate entire PySpark codebases for compatibility with Snowpark Connect (Snowflake's engine for running Spark workloads) from a single conversational prompt.
Why it matters
Moving Spark workloads to Snowflake eliminates cluster management overhead and delivers up to 5.1x faster performance and 42% lower costs according to customer production use cases from November 2022 to May 2025. The automation replaces hours of manual migration work with a process that now takes minutes, letting developers avoid hand-coding compatibility fixes across multiple files.
What to watch
The skill handles Python, Scala, and Java code files, plus notebook files; it auto-flags anything requiring manual review (such as validating reads and writes) and can optionally run end-to-end validation by comparing outputs of the original PySpark code against the migrated Snowpark Connect code against synthetic data. The skill is bundled with CoCo and requires no additional setup.
Running Apache Spark workloads has historically required significant operational overhead: tuning clusters, patching infrastructure, and managing dependency environments. Snowflake's Snowpark Connect for Apache Spark allows existing Spark code to run on Snowflake's engine with minimal changes and no cluster provisioning, tuning, or patching required. Customers running Snowpark Connect have achieved up to 5.1x faster performance and 42% lower costs in production use cases measured between November 2022 and May 2025.
However, migrating large codebases from native Spark to Snowpark Connect has remained a manual and time-consuming process. Generic AI coding assistants can rewrite individual code snippets but lack the compatibility context to move an entire codebase: they do not know which patterns are unsupported, how to map them to DataFrame equivalents, or how to track what changed. Snowflake's new spark-migration skill, built into its CoCo data-native AI coding agent, is designed to address this gap.
The spark-migration skill works through a structured agentic workflow that mimics what a migration expert would do. First, it performs an assessment and builds an inventory by scanning every source file for compatibility issues—including RDD operations, certain UDF serialization patterns, and unsupported file formats—against a knowledge base built by Snowflake's engineering team. It inventories each file, dependency, and unique API call. Second, it applies targeted fixes by dispatching parallel agents to apply code fixes and verify results through multiple gates, including syntax compilation and evidence-based checks; incompatible patterns are rewritten to DataFrame equivalents, imports and session creation are updated, and items requiring manual review (such as validating reads and writes) are flagged with detailed explanations. Third, it generates reporting that provides full visibility into every change and what may still need attention, producing issue logs, inventory reports, and validation of the overall migration state. Finally, it optionally validates the converted code's functionality by running the original PySpark code and migrated Snowpark Connect code against the same synthetic data and comparing outputs table-by-table, catching behavioral differences and validating the code is ready to run; any fixes discovered during validation are automatically committed back to the deliverable branch.
The skill handles input files written in Python, Scala, or Java, as well as build files and notebook files. Developers can invoke it with a simple prompt such as "Migrate this file to Snowpark Connect" or ask it to perform other tasks such as "Assess the compatibility and level of effort required to migrate this codebase" or "Validate that this codebase will run with Snowpark Connect." The skill auto-activates when the developer mentions Spark, PySpark, or code migration, or can be invoked directly with the spark-migration command. The spark-migration skill is bundled with CoCo and requires no additional setup, allowing developers to reduce what previously took hours of manual effort to minutes while staying in control of the migration through flagged items requiring review.
The friction of migrating large codebases to a new execution engine has long been a barrier to adoption, even when the performance and cost case is strong. Generic AI assistants can rewrite small code snippets but lack the domain knowledge to handle architectural patterns, API mappings, and systematic compatibility issues across an entire codebase. Snowflake's spark-migration skill closes that gap by embedding the migration logic that Snowflake's engineering team would apply manually: systematic scanning for known incompatibilities (RDD operations, certain UDF serialization patterns, unsupported file formats), parallel application of targeted fixes, detailed reporting, and optional validation against synthetic data.
The result is that what once required hours of manual effort—searching across files, cross-referencing documentation, and hand-coding equivalent operations—now happens in minutes from a single prompt. The skill stays within CoCo's conversational interface, so developers do not need to learn a new tool; they invoke it either implicitly (when mentioning Spark, PySpark, or code migration) or directly with a spark-migration command. Because the tool flags items requiring manual review rather than making assumptions, developers retain control over what changes before running in production.
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytime
Ask AI anything about this article. Q&As are published on this page for other readers too.
The AI news that matters, in one minute each morning.
Sign up free