Oracle GoldenGate for Big Data 12.2.0.1 is Generally Available Now!

By Thomas Vengal-Oracle on Dec 22, 2015

https://blogs.oracle.com/dataintegration/entry/oracle_goldengate_for_big_data

Much awaited Oracle GoldenGate for Big Data 12.2 is released today and it is available for download at OTN.

Let me give you a quick recap on Oracle GoldenGate for Big Data. Oracle GoldenGate for Big Data streams transactional data into big data systems in real-time, raising the quality and timeliness of business insights. Oracle GoldenGate for Big Data offers also provides a flexible and extensible solution to support all major big data systems.

Oracle GoldenGate for Big Data

  • Same trusted Oracle GoldenGate architecture used by 1000’s of customers
  • Data delivery to Big Data targets including NoSQL databases
  • Support for Polyglot, Lambda and Kappa architectures for streaming data

Key Benefits

  • Less invasive on source databases when compared to batch processing such as Sqoop or ETL processes
  • Simple ingestion for 1:1 data architecture for populating “raw data” zones
  • Real-time data delivery for streaming analytics/apps
  • Reliable, proven at scale with high performance

Architecture – GoldenGate for Big Data 12.2 versus 12.1

New Features in 12.2.0.1:

New Java based Replicat Process 

The advantages of using Java based Replicat process are the following:

  1. Improved performance with Java based adapters
  2. Declarative design and configurable mapping
  3. Transaction grouping based on Operation count & Message size
  4. Improved check pointing functionality
    E.g.: CHECKPOINTSECS 1 (default 10 seconds)

Dynamic Data Handling

You no longer require to define SOURCEDEFS. DDL changes are automatically replicated to target. For example, if a new column named “mycolumn“ is added on the source database, it will be automatically replicated to the target without stopping and reconfiguring Oracle GoldenGate.

Pluggable Formatters

Oracle GoldenGate for Big Data can write into any Big Data targets in various data formats such as delimited text or XML or JSON or Avro or custom format. This can save users cost and time for staging data in ETL operations.

Example: gg.handler.name.format= <value>
values supported are “delimitedtext”, “xml”, “json”, “avro” or “avro_row”, “avro_op” or Custom Format. Extended class path needs to be included in the config file. <com.yourcompany.YourFormatter

Security Enhancement

Native Kerberos support is available in the 12.2.0.1 binaries.

Example of configuration:
gg.handler.gghdfs.authType=Kerberos
gg.handler.gghdfs.kerberosKeytabFile=/keytab/file/path
gg.handler.gghdfs.kerberosPrincipal=user/FQDN@MY.REALM

Declarative Design

Oracle GoldenGate for Big Data is able to provide mapping functionally between source table to target table and source field to target field for HDFS/Hive, HBase, Flume and Kafka. The metadata is also validated at Hive or using an Avro schema to ensure data correctness.

Example:
MAP GG.TCUSTOMER, TARGET GG.TCUSTMER2, COLMAP (USEDEFAULTS, “cust_code2″=cust_code,”city2″=city);

Kafka as target

Oracle GoldenGate for Big Data can write Logical change records data to a Kafka topic. Operations such as Insert, Update, Delete and Primary Key Update can be handled. It can handles native compression such as GZIP and Snappy in Kafka.

Example of defining Kafka Handler Properties:
gg.handlerlist=ggkafka
gg.handler.ggkafka.type=kafka
gg.handler.ggkafka.topicName=gg_kafka
gg.handler.ggkafka.mode=tx

Other Enhancements

  • Partition data by Hive Table and/or column. Partitioning into new file based on designated column values
    Example:

    • gg.handler.{name}.partitionByTable =true | false
    • gg.handler.{name}.partitioner.{fully qualified table name}={colname}
    • gg.handler.{name}.partitioner.{fully qualified table name}={colname1},{colname2}
    • gg.handler.<yourhandlername>.partitioner.dbo.TCUSTORD=region, rating
  • Configurable File Rolling Property for HDFS (file size, duration, inactivity timer, metadata change)
  • Configurable file output encoding into HDFS
  • Automatically create HBase table if it does not exist
  • Ability to treat primary key updates as a delete and then an insert in HBase
  • HBase row key generation
  • Treat Primary Key updates as delete and insert in Flume and HBase
  • New Time stamping functionality to include micro second precision as ISO-8601
  • Availability on additional OS platforms: Windows and Solaris
  • Certification for newer versions: Apache HDFS 2.7.x, Cloudera 5.4.x, Hortonworks 2.3, Kafka 0.8.2.0 and 0.8.2.1

For more details about new product features, you may refer to Oracle GoldenGate for Big Data 12.2.0.1 Release Notes and User Documentation.

For more information about Oracle GoldenGate for Big Data.

Feel free to reach out to me for your queries by posting in this blog or tweeting @thomasvengal

 

Sobre Alexandre Pires

ORACLE OCS Goldengate Specialist, OCE RAC 10g R2, OCP 12C, 11g, 10g , 9i e 8i - Mais de 25 anos de experiência na área de TI. Participei de projetos na G&P alocado na TOK STOK, EDINFOR alocado na TV CIDADE "NET", 3CON Alocado no PÃO DE AÇUCAR, DISCOVER alocado na VIVO, BANCO IBI e TIVIT, SPC BRASIL, UOLDIVEO alocado no CARREFOUR e atualmente na ORACLE ACS atendendo os seguintes projetos: VIVO, CLARO, TIM, CIELO, CAIXA SEGUROS, MAPFRE, PORTO SEGURO, SULAMERICA, BRADESCO SEGUROS, BANCO BRADESCO, BASA, SANTANDER, CNJ, TSE, ELETROPAULO, EDP, SKY, NATURA, ODEBRESHT, NISSEI, SICREDI, CELEPAR, TAM, TIVIT, IBM, SMILES, CELEPAR, SERPRO,OKI,BANCO PAN, etc
Esse post foi publicado em GOLDENGATE, ORACLE 11gR2 e marcado , . Guardar link permanente.

Deixe uma resposta

Preencha os seus dados abaixo ou clique em um ícone para log in:

Logotipo do WordPress.com

Você está comentando utilizando sua conta WordPress.com. Sair / Alterar )

Imagem do Twitter

Você está comentando utilizando sua conta Twitter. Sair / Alterar )

Foto do Facebook

Você está comentando utilizando sua conta Facebook. Sair / Alterar )

Foto do Google+

Você está comentando utilizando sua conta Google+. Sair / Alterar )

Conectando a %s