How the replicator processes data.
GIFRÖST is based on Apache Kafka and Kafka Connect. Source connectors read changes from the source database or source tools, write them to Kafka topics, and target connectors consume those records and apply the matching operations to the target database.
Introduction
For transactional databases, the source connector does not have to poll business tables in a
loop. It reads changes recorded by the database engine in the transaction log. Depending on the
database technology, this position can be represented for example by PostgreSQL LSN or Oracle
SCN. The connector captures committed DML operations such as INSERT,
UPDATE, DELETE, and TRUNCATE, converts them into Kafka
Connect records, and writes them to Kafka.
The source connector runs inside Kafka Connect and is responsible for publishing changes to Kafka. The target connector, also running in Kafka Connect, consumes records from Kafka and applies the corresponding DML operations to the target database.
The database records committed changes in its transaction log. The CDC reader extracts row changes with minimal impact on the source system and without changing the source application.
The source connector turns log entries into structured change events and publishes them to Kafka topics. This is the point where source-side mappings and transformations can be applied.
Kafka stores the stream of changes, isolates producers from consumers, and gives target connectors a durable place from which they can continue reading.
The target connector reads Kafka records as soon as they appear and performs equivalent DML operations in the target database.
Impact on source database performance
CDC is based on reading native database transaction logs, such as Oracle LogMiner, PostgreSQL WAL, or equivalent mechanisms for the selected database technology. This means replication does not require triggers on business tables and does not run additional polling queries directly against those tables. As a result, the impact on source system performance is kept to a minimum.
CDC does not require triggers on replicated tables, so it does not add custom code executed synchronously inside each business transaction.
The connector reads the transaction log, so it does not generate cyclic table scans to detect changes in source tables.
The actual impact depends on environment characteristics, change volume, and configuration. The load can therefore be confirmed in the customer environment as part of a POC.
Based on our internal tests and customer-side validation work, a correctly configured CDC mechanism has not shown cases where source database load increased by more than 10%. This should be treated as an observation from previous tests, not as a guarantee for every production environment.
For critical systems, a POC is recommended with CPU, I/O, wait-event, replication-lag, and transaction-log volume metrics to confirm CDC impact under conditions close to production.
Event ordering
Ordering is preserved for a single table. The transaction log is read by a single-threaded process, and table changes are written to single-partition Kafka topics. Since Kafka preserves ordering within one partition, the target connector receives events for one table in the same order in which they were produced.
Within one table: DML events are processed sequentially, so updates, inserts, and deletes for the same table are delivered in order.
Between different tables: races can occur because independent topics and target-side tasks may progress at slightly different speeds.
Within one transaction for one table: event order remains stable for that table. Cross-table transactional ordering should not be treated as a global delivery guarantee.
Replication resume
Replication resume is handled natively by Apache Kafka Connect. Connect stores connector progress in a dedicated offset topic. For source connectors this offset is the last processed source position, for example PostgreSQL LSN or Oracle SCN. If a connector is stopped or fails, after restart it reads the saved offset and resumes replication from the last processed place, without rereading all data and without additional impact on the source system.
The source connector records the last transaction-log position that was successfully processed. After restart, it continues from that exact point.
Target connectors use an analogous offset mechanism. They store information about the last processed Kafka record and resume consumption from the correct place.
Target database error handling
When the JDBC Sink connector is used, the solution provides a Dead Letter Queue (DLQ) mechanism. DLQ is a dedicated, durable registry of records that could not be applied to the target database. Every failed operation, for example an integrity constraint violation, data type mismatch, or missing column, is automatically written to a dedicated Kafka topic that acts as a failed-transaction registry.
A record that could not be written to the target database is sent to a dedicated Kafka topic together with error information. The topic retention period is configurable, so DLQ content can be kept according to operational and audit requirements.
DLQ stores the context required to identify the source record, the processing stage, and the failure reason reported by the target database.
After the root cause is removed, DLQ records can be processed again without data loss, because the registry stores the original, complete change image.
Complete record content: change image with the operation type
INSERT, UPDATE, or DELETE, record key, and column
values before and after the change when available in the source event.
Record origin: Kafka topic, partition, and offset, which uniquely identify the source transaction or source event.
Target database error details: exception class, error message, database
error code such as ORA-00001, and full stack trace.
Processing context: the stage where the failure occurred, connector or component identifier, and diagnostic data required for incident analysis.
DLQ content is durable, searchable, and available from the administrative panel and API. Optionally, the error registry can also be written to a relational table indicated by the customer. In parallel, every execution error is recorded in the solution system logs with a configurable detail level, including the ability to log full SQL statement text when allowed by the environment security policy.
DDL replication
GIFRÖST supports capturing DDL operations executed on a source Oracle database. DDL statements are read from the redo stream through LogMiner, parsed, and stored in a dedicated schema history repository. This lets the system maintain the current definition of replicated table structures and continue replication after compatible schema changes on the source.
The system can automatically create target table schemas during replication initialization. This does not guarantee a 1:1 DDL mapping, so production deployments should create target structures manually or with dedicated scripts.
Additive schema changes can be propagated in a limited way. The typical supported scenario is adding new columns to replicated tables.
Renaming columns, dropping columns, and changing data types are handled in a controlled mode. The system signals a schema change, the connector stops, and a database administrator updates the target manually before processing is resumed automatically or manually.
Compression and encryption
GIFRÖST can use compression mechanisms at several levels of the pipeline: Kafka Connect worker configuration, source and target connector settings, Kafka broker and topic configuration, and Kafka on-disk log storage. This reduces network traffic, lowers topic storage usage, and helps the cluster handle larger change volumes more efficiently.
Source connectors that publish records to Kafka can use producer compression, for example
through compression.type. Common algorithms include gzip,
snappy, lz4, and zstd. The setting can be applied at
the Kafka Connect worker level or per connector when client override policy allows it.
Kafka stores records in log segments. Compressed batches sent by a producer can remain compressed when written to broker disk, and broker or topic settings can enforce a selected compression type. This means compression can reduce both network transfer and the amount of data stored in Kafka logs.
Target connectors consume records from Kafka without handling the compression format directly - decompression is performed by the Kafka client. On the target write side, batching and driver-level or protocol-level compression can be used when supported by the selected target technology.
Encryption in Kafka Connect and Kafka is implemented mainly through transport security and access control. Compression and encryption solve different problems: compression reduces the size of transferred and stored data, while encryption protects communication confidentiality and access to cluster resources.
TLS/SSL between Kafka Connect and Kafka brokers: worker, producer, and
consumer communication can use encrypted channels through security.protocol=SSL
or SASL_SSL together with truststore and keystore settings.
mTLS and client authentication: brokers can require client certificates, so Kafka Connect workers and connectors prove their identity when connecting to Kafka.
SASL and ACLs: authentication can use mechanisms such as SCRAM, GSSAPI/Kerberos, OAUTHBEARER, or PLAIN only over TLS. Access to topics, consumer groups, and cluster resources can be restricted through ACLs.
HTTPS for the Kafka Connect REST API: the Connect worker REST interface can be exposed over HTTPS, and administrative operations can be restricted through authentication and network controls.
Encrypted source and target database connections: connectors can use TLS/SSL in JDBC or native database client settings when the selected database supports it.
Secrets and data-at-rest encryption: passwords and tokens should be kept outside plaintext connector configuration, for example through a ConfigProvider or external secret store. Kafka log encryption at rest is typically implemented at the volume, filesystem, or storage/KMS layer, because Kafka writes log segments while at-rest encryption belongs to the infrastructure layer.
Scalability
GIFRÖST can be scaled vertically and horizontally. More CPU, RAM, and faster storage improve the capacity of Kafka, Kafka Connect, and supporting services. Additional Kafka Connect instances can also be added to increase the number of connectors that can run in parallel.
Increasing server resources improves the ability to process higher change volume, larger connector workloads, and longer retention windows.
Adding Kafka Connect instances increases worker capacity and allows more source and target connectors to run across the Connect cluster.
If strict event ordering is not required, Kafka topics can use more partitions to increase parallelism. For table-level ordered replication, single-partition topics remain the recommended model.