Optimizing migration performance

You can leverage the EDB Data Migration Service (DMS), a service included with the Hybrid Manager (HM), to migrate your data from existing database(s) (Oracle, self-managed Postgres, etc.) to a HM-managed database cluster. EDB DMS supports two migration methods: snapshot and snapshot + streaming, allowing you to migrate your data with minimal to no downtime.

Understanding migration workloads in HM

The EDB Data Migration Service runs within the same Kubernetes environment where your EDB Hybrid Manager is deployed (on-premises, AWS, Red Hat OpenShift, and Google Cloud). This means that most of the workloads required for migration operations are managed and optimized by your Kubernetes cluster.

Generally, you don't need to extensively fine-tune resources, as your Kubernetes environment meeting the general requirements for running the Hybrid Manager should ensure a smooth migration. However, for large datasets, optimizing specific components can significantly accelerate the process.

The key components to consider for resource allocation are:

• EDB DMS Reader: This component runs on your source database's host machine.

• Destination database cluster: This can be the database managed by HM or a different external self-managed database, however here we'll focus on an HM-managed database. This is where your data will reside.

• EDB DMS Writer: This component is only required if your migration target is an external (self-managed) database cluster, and it runs on that database's host machine.

Whether you perform a snapshot or snapshot + streaming migration also affects performance differently:

• Snapshot migrations: They primarily focus on efficiently transferring a large volume of existing data from a source database to the target. Performance for these is heavily influenced by total data size and sustained disk I/O.

• Snapshot + streaming migrations: They first transfer a snapshot of existing data and then continuously stream ongoing changes (transactions) from the source to the destination. Performance here is driven by the sustained transaction rate (TPS) and the ability to keep up with the source's write activity.

Optimizing migration performance for large datasets

The HM and DMS are designed to handle resource usage automatically. For datasets smaller than 100 GB, they are configured to perform fast migrations without further tuning. However, for datasets larger than 100 GB, a few targeted alterations can lead to significantly better performance and quicker migrations.

Tuning destination database performance

Optimizing the destination database cluster (whether an HM-managed database or a self-managed database) provides the most significant performance benefits from resource tuning.

Storage recommendations

During a migration, storage IOPS can peak at 2000, with an average throughput of roughly 300 MB/s and peaks up to 350 MB/s. To accommodate these demands:

• Separate WAL and data volumes: Aim for storage volumes that can handle a minimum of 2500-3000 IOPS and at least 200-250 MB/s of throughput for both your data and Write-Ahead Log (WAL) volumes. Using separate volumes for WAL and data is highly recommended for optimal performance.

• Unified storage and WAL volume: A unified storage and WAL volume can adversely affect performance. For example, using a single volume with 3000 IOPS and 125 MB/s throughput can increase migration time by over 40%. If you must use a unified storage and WAL volume, we recommend a throughput of 300-400 MB/s. Keep in mind that if the WAL becomes full, the migration will fail.

Memory recommendations

While 5 GB of memory is sufficient, a larger amount will facilitate optimal performance during a migration. Use at least 10 GB of memory for the destination database.

CPU recommendations

The destination database cluster should have at least 4 CPU cores. A lower core count will negatively impact migration performance.

Extensive testing has shown that the biggest factor affecting migration performance is volume speed (IOPS and throughput). While decreasing CPU below recommendations can reduce performance, it's not nearly to the degree that low IOPS and throughput for storage will. Halving the storage throughput can increase migration time by over 40%, whereas halving the CPU count to 2 cores might increase it by 10-15%. These recommendations are designed to yield the best migration performance; using lower values may still result in successful migrations but at a slower speed.

Tuning EBD DMS Reader and Writer performance

Optimal performance for the EDB DMS Reader and Writer is achieved when the host machines where these components are installed follow these specifications.

EDB DMS Reader

CPU: 3 CPU cores

Memory: 4 GB

EDB DMS Writer

CPU: 2 CPU cores

Memory: 2.5 GB

Performance benchmarking for migrations

To provide a clear understanding of expected migration performance, EDB conducted a series of tests to benchmark migration speeds under various conditions.

While the previous recommendations are derived from comprehensive testing across various migration configurations (including different table sizes, multiple tables, and streaming scenarios), this section presents results from a single 100 GB snapshot migration. Additional detailed performance benchmarks will be incorporated into future releases to further assist in planning your data migrations.

Approach

All migration performance tests were conducted using an automated test suite. The testing infrastructure was hosted on an EC2 t4i.4xlarge instance. Resources on this instance were shared across these Docker containers:

EDB DMS Reader container: 4 CPU, 16 GB memory

Oracle database: 8 CPU, 32 GB memory

All tests originated from an Oracle database and targeted an HM-managed EDB Postgres Advanced Server (EPAS) cluster. The HM-managed database was hosted on a Hybrid Manager deployment running on an m6a.2xlarge instance.

EPAS database cluster configuration

Volumes: Minimum 3,000 IOPS and 500 MB/s throughput.

Storage: Separate storage and WAL volumes were used for each test on the EPAS cluster, with volumes being approximately the same size.

GUC Settings: Default GUC settings were used on the PostgreSQL cluster. Further testing with increased memory allocations and fine-tuned GUC values did not show a significant impact on migration speed.

Migration data

The table contained two columns: one serving as the primary key and the other populated with random integers. A snapshot-only migration was performed several times to account for variability and specify tolerance ranges.

Migration procedure

1. Populated the Oracle database with the specified amount of tables.

2. Initiated and completed the migration process.

Results