Monday December 16, 2019

What's New in Apache Kafka 2.4

On behalf of the Apache Kafka® community, it is my pleasure to announce the release of Apache Kafka 2.4.0. This release includes a number of key new features and improvements that we will highlight in this blog post. For the full list, please see the release notes.

What’s new with the Kafka broker, producer, and consumer

KIP-392: Allow consumers to fetch from closest replica

Historically, consumers were only allowed to fetch from leaders. In multi-datacenter deployments, this often means that consumers are forced to incur expensive cross-datacenter network costs in order to fetch from the leader. With KIP-392, Kafka now supports reading from follower replicas. This gives the broker the ability to redirect consumers to nearby replicas in order to save costs.

See KIP-392 and this blog post for more details.

KIP-429: Kafka Consumer Incremental Rebalance Protocol

KIP-429 adds Incremental Cooperative Rebalancing to the consumer rebalance protocol in addition to the original eager rebalance protocol. Unlike the eager protocol, which always revokes all assigned partitions prior to a rebalance and then tries to reassign them altogether, the incremental protocol tries to minimize the partition migration between members of a consumer group by letting consumers retain their partitions during a rebalance event. As a result, end-to-end rebalance times triggered by scaling out/down operations as well as rolling bounces are shorter, benefitting heavy, stateful consumers, such as Kafka Streams applications. 

See KIP-429 and this blog post for more details.

KIP-455: Create an Administrative API for Replica Reassignment

As a replacement for the existing ZooKeeper-based API, the new API supports incremental replica reassignments and cancellation of ongoing reassignments. This also addresses the current limitations in the ZooKeeper-based API like security enforcement and auditability. The new API is exposed via the AdminClient.

See KIP-455 for more details.

KIP-480: Sticky Partitioner

Currently, in the case where no partition and key are specified, a producer's default partitioner partitions records in a round-robin fashion. This results in more batches that are smaller in size and leads to more requests and queuing as well as higher latency. 

KIP-480 implements a new partitioner, which chooses the sticky partition that changes when the batch is full if no partition or key is present. Using the sticky partitioner helps improve message batching, decrease latency, and reduce the load for the broker. Some of the benchmarks which Justine Olshan discusses on the KIP show up to a 50% reduction in latency and 5–15% reduction in CPU utilization.

See KIP-480 and this blog post for more details.

KIP-482: The Kafka Protocol should Support Optional Tagged Fields

The Kafka remote procedure call (RPC) protocol has its own serialization format for binary data. The Kafka protocol currently does not support optional fields, nor does it support attaching an extra field to a message in a manner that is orthogonal to the versioning scheme. 

In order to support these scenarios, KIP-482 adds optional tagged fields to the Kafka serialization format. Tagged fields are always optional. KIP-482 also implements more efficient serialization for variable-length objects.

See KIP-482 for more details.

KIP-504: Add new Java Authorizer Interface

This KIP defines a Java authorizer API that is consistent with other pluggable interfaces in the broker. Several limitations in the current Scala authorizer API that could not be fixed without breaking compatibility have been addressed in the new API. Additional request context is now provided to authorizers to support authorization based on the security protocol or listener. 

The API also supports asynchronous ACL updates with batching. The new pluggable authorizer API only requires a dependency on the client’s JAR. A new out-of-the-box authorizer has been added, leveraging features supported by the new API. The additional context provided to the authorizer has been used to improve audit logging. Batched updates enhance the efficiency of ACL updates using the new authorizer when multiple ACLs are added for a resource. An asynchronous startup and updated APIs will enable Kafka to be used as the storage backend for ACLs once ZooKeeper is removed under KIP-500. In addition, authorizer implementations can now enable dynamic reconfiguration without broker restarts.

See KIP-504 for more details.

KIP-525: Return topic metadata and configs in CreateTopics response

Before, the CreateTopics API response only returned a success or failure status along with any errors. With KIP-525, the API response returns additional metadata, including the actual configuration of the topic that was created. This removes the need for additional requests to obtain topic configuration after creating the topic. 

Furthermore, this KIP enables users to obtain default broker configs for topic creation using CreateTopics with validateOnly=true. This is useful for displaying default configs in management tools used to create topics.

See KIP-525 for more details.

KAFKA-7548: KafkaConsumer should not throw away already fetched data for paused partitions.

When a partition is paused by the user in the consumer, the partition is considered "unfetchable." When the consumer has already fetched data for a partition and the partition is paused, then in the next consumer poll all data from "unfetchable" partitions will be discarded. In use cases where pausing and resuming partitions are common during regular operation of the consumer, this can result in discarding pre-fetched data when it's not necessary. 

Once the partition is resumed, new fetch requests will be generated and sent to the broker to get the same partition data again. Depending on the frequency of pausing and resuming of partitions, this can impact different aspects of consumer polling, including broker/consumer throughput, number of consumer fetch requests, and NIO-related garbage collection (GC) concerns for regularly dereferenced byte buffers of partition data. This issue is now resolved by retaining completed fetch data for partitions that are paused so that it may be returned in a future consumer poll once the partition is resumed by the user.

See KAFKA-7548 for more details.

What’s new in Kafka Connect

KIP-382: MirrorMaker 2.0

KIP-382 implements MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication engine based on the Connect framework. This tool includes several features designed for disaster recovery, including cross-cluster consumer checkpoints and offset syncs. Automatic topic renaming and cycle detection enable bidirectional active-active replication and other complex topologies. 

A new RemoteClusterUtils class enables clients to interpret checkpoints, heartbeats, and "remote topics" from other clusters.

See KIP-382 for more details.

KIP-440: Extend Connect Converter to support headers

KIP-440 adds header support to Kafka Connect. This enables the use of Kafka Connect together with Kafka producers and consumers that rely on headers for serialization/deserialization.

See KIP-440 for more details.

KIP-507: Securing Internal Connect REST Endpoints

KIP-507 brings out-of-the-box authentication and authorization to an internal REST endpoint used by Connect workers to relay task configurations to the leader. If left unsecured, this endpoint could be used to write arbitrary task configurations to a Connect cluster. 

However, after KIP-507, the endpoint automatically secures as long as the other attack surfaces of a Connect cluster (such as its public REST API and the underlying Kafka cluster used to host internal topics and perform group coordination) are also secure.

See KIP-507 for more details.

KIP-481: SerDe Improvements for Connect Decimal type in JSON

KIP-481 adds to the JSON converter decimal.format for serializing Connect’s DECIMAL logical type values as number literals rather than base64 string literals. This new option defaults to base64 to maintain the previous behavior, but it can be changed to number to serialize decimal values as normal JSON numbers. The JSON converter automatically deserializes using either format, so make sure to upgrade consumer applications and sink connectors before changing source connector converters to use the number format.

See KIP-481 for more details.

What’s New in Kafka Streams

KIP-213: Support non-key joining in KTable

Previously, the Streams domain-specific language (DSL) only allowed table-table joins based on the primary key of the joining KTables. Now, for a KTable (left input) to join with another KTable (right input) based on a specified foreign key as part of its value fields, the join result is a new KTable keyed on the left KTable’s original key. This supports updates from both sides of the join.

See KIP-213 for more details.

KIP-307: Allow to define custom processor names with KStreams DSL

Prior to this release, while building a new topology through the Kafka Streams DSL, the processors were automatically named. A complex topology with dozens of operations can be hard to understand if the processor names are not relevant. This KIP allows users to set more meaningful processor names.

See KIP-307 for more details.

KIP-470: TopologyTestDriver test input and output usability improvements

The TopologyTestDriver allows you to test Kafka Streams logic. This is a lot faster than utilizing actual producers and consumers and makes it possible to simulate different timing scenarios. Kafka 2.4.0 introduces TestInputTopic and TestOutputTopic classes to simplify the test interface. 

See KIP-470 and this blog post for more details.

Metrics, monitoring, and operational improvements

  • KIP-412 adds support to dynamically alter a broker's log levels using the Admin API. 
  • KIP-495 allows users to dynamically alter log levels in the Connect framework.
  • KIP-521 changes Connect to also send log messages to a file and rolls that file every day.
  • KIP-460 modifies the PreferredLeaderElection RPC to support unclean leader election in addition to preferred leader election.
  • KIP-464 allows you to leverage num.partitions and default.replication.factor from the AdminClient#createTopics API.
  • KIP-492 supports the security provider config, which can be used to configure custom security algorithms.
  • KIP-496 adds an API to delete consumer offsets and expose it via the AdminClient.
  • KIP-503 adds metrics to monitor the number of topics/replicas marked for deletion.
  • KIP-475 adds metrics to measure the number of tasks on a connector.
  • KIP-471 exposes a subset of RocksDB's statistics in Kafka Streams metrics, which enables users to find bottlenecks and tune RocksDB accordingly.
  • KIP-484 adds new metrics for the group and transaction metadata loading duration.
  • KIP-444 adds a few new metrics at the Streams instance level such as static version/commit-id as well as dynamic state.

ZooKeeper upgrade to 3.5.x

ZooKeeper has been upgraded to 3.5.x. support for TLS encryption added in ZooKeeper 3.5.x. This enables us to configure TLS encryption between Kafka brokers and ZooKeeper.

Scala 2.13 support

Apache Kafka 2.4.0 now supports Scala 2.13 while also remaining compatible with Scala 2.12 and 2.11.


We want to take this opportunity to thank everyone who has contributed to this release!

To learn more about what’s new in Apache Kafka 2.4, be sure to check out the release notes and highlights video.


That Apache Kafka is a great release that will wonder many people and developers i'm sure about it

Posted by mohd on December 18, 2019 at 09:48 AM UTC #

What Are The Significant Features Of Yahoo Help And Support Service There are plenty of benefits one can take leverage by just availing Yahoo Help and support service from a trusted source. Here, one can avail the services irrespective of time concerns as it remains active all the time. In addition, the service change is very nominal and can be delivered under the guidance of experts.

Posted by Lesli Wilson on December 19, 2019 at 08:56 AM UTC #

Cash App Hacked then Immediately Reach Us If your cash app hacked then just instantly contact us via call, email, or text and report the issues. Never ignore this sort of issue as it may also lead to the deduction of money from your account and undefined transaction. Our professionals take instant actions for resolving the payment app hacking issues.

Posted by Lesli Wilson on December 19, 2019 at 09:27 AM UTC #

That Apache Kafka is a great release

Posted by Rikko on December 24, 2019 at 03:30 AM UTC #

The Department of Justice today announced that beginning on 24 January 2014, applicants for the NBI Clearance can now apply and fill out ...

Posted by harry12 on December 25, 2019 at 09:23 AM UTC #

The Department of Justice today announced that beginning on 24 January 2014, applicants for the NBI Clearance can now apply and fill out ...http://nbiclearance.today/

Posted by harry12 on December 25, 2019 at 09:24 AM UTC #

hey, check out my website that aims at the art culture and history and how it fares in the modern world.

Posted by studio 66 on December 26, 2019 at 10:07 AM UTC #

Hi, your content is very unique and informative, Keep spreading this useful information. Meanwhile, Buy premium, authentic and genuine hookahs online. Check out our website

Posted by Buy shisha on December 26, 2019 at 10:08 AM UTC #

GreatPeopleMe login portal and different highlights offered by Kroger to their representatives, at that point in this article

Posted by https://greatpeople-me.website/ on December 26, 2019 at 11:21 AM UTC #

The Department of Justice today announced that beginning on 24 January 2014, applicants for the NBI Clearance can now apply and fill out ... https://theinsurancemenu.com

Posted by AaronWeb on January 04, 2020 at 11:29 PM UTC #

I guess i need more apache to get a good job ;)

Posted by Bandar Ceme Online on January 07, 2020 at 06:15 AM UTC #

Apache, you are awesome https://moralkahani.in/top-10-hindi-short-stories-with-moral/

Posted by Ibney Ali on January 08, 2020 at 08:37 PM UTC #

The Apache Kafka is an amazing release! https://www.landscapingelpasotexas.com/

Posted by Chris on January 14, 2020 at 05:51 AM UTC #

Is kafka 2.4 compatible with Spark 2.4.4 ?

Posted by Shadab on January 16, 2020 at 01:57 PM UTC #

Well, i can see this is the best release for this year. Good work Apache. Find out more about online game here, https://score88poker.bid/ and enjoy it!

Posted by Ruby on January 18, 2020 at 06:53 AM UTC #

Thanks for sharing these details about the new apache. https://call2shiftjaipur.in/

Posted by Devender Poonia on January 20, 2020 at 07:22 AM UTC #

Thanks for sharing these informative details with us. List of best forum submission sites is here - https://theblogstack.com/digital-marketing/top-high-da-forum-submission-sites-list/

Posted by Aakash Kumar on January 23, 2020 at 06:07 PM UTC #

Hey thanks for sharing and updating this, well do visit our website which is related to the login guides and much more related to the stuff https://krogerexperiencee.com/kroger-hr-express/

Posted by krogerfeedback on January 28, 2020 at 04:45 PM UTC #

Its good version. Thanks https://www.plerb.com/kimjones

Posted by Kim Jones on February 04, 2020 at 05:24 AM UTC #

Good new things here. https://www.lyricsmaze.com

Posted by Nikita on February 04, 2020 at 05:27 AM UTC #

Thank you for the information <a href="http://www.cliftonlandscape.com">Clifton Landscape</a>

Posted by Amanda Hablawi on February 05, 2020 at 07:31 PM UTC #

Thank you for the information <a href="http://www.cliftonlandscape.com">CliftonLandscape</a>

Posted by Amanda Hablawi on February 05, 2020 at 07:31 PM UTC #


Posted by Amanda Hablawi on February 05, 2020 at 07:32 PM UTC #

Your post is just amazing and thanks for sharing a such a great post

Posted by Warrenddd on February 06, 2020 at 07:44 AM UTC #


Posted by Maria Loui on February 06, 2020 at 08:58 AM UTC #


Posted by google on February 06, 2020 at 08:58 AM UTC #

https://www.contacthelpsupport.com/ https://www.contacthelpsupport.com/aol-mail-help/

Posted by komal on February 06, 2020 at 09:03 AM UTC #

Accounts Confidant is a U.S. registered firm having its delivery center

Posted by Accountants in New York on February 06, 2020 at 12:23 PM UTC #

How to set up Apache https://www.sitepoint.com/how-to-install-apache-on-windows/

Posted by qqscore88 on February 08, 2020 at 03:54 AM UTC #

How to configure an Apache web server https://opensource.com/article/18/2/how-configure-apache-web-server https://sbobet.blog/ https://sbobet338.win/ https://score88.vip/

Posted by score88 on February 08, 2020 at 04:11 AM UTC #

Currently, in the case where no partition and key are specified, a producer's default partitioner partitions records in a round-robin fashion https://tunasbola.org/link-alternatif-sbobet/ http://www.tunaspoker.online/

Posted by sbobet on February 08, 2020 at 04:41 AM UTC #

very good and very interesting article and this is one of the articles that is very useful and provides more knowledge, thank you , from : https://gembalapoker.biz/

Posted by gembalapoker on February 09, 2020 at 08:31 AM UTC #

Keep up the good work

Posted by Express hr on February 09, 2020 at 11:29 AM UTC #

I read Post and got it fine and informative. Please share more like that..

Posted by krogerexperiencee on February 12, 2020 at 10:27 AM UTC #

How to Install Apache Tomcat 9 (on Windows, macOS, Ubuntu) and Get Started with Java Servlet Programming https://www.ntu.edu.sg/home/ehchua/programming/howto/Tomcat_HowTo.html

Posted by https://www.papipoker.online/ on February 13, 2020 at 02:17 AM UTC #

Apache Tomcat Windows quick start guide https://www.mulesoft.com/tcat/tomcat-windows

Posted by https://www.slotdepositpulsa.co/ on February 13, 2020 at 02:19 AM UTC #

Awesome and interesting article. Great things you've always shared with us. Thanks. Just continue composing this kind of post. taiwan coronavirus https://www.islandlifetaiwan.com/taiwan-safe-to-visit-during-coronavirus-in-mainland-china/

Posted by osama shk on February 13, 2020 at 01:57 PM UTC #

Awesome and interesting article. Great things you've always shared with us. Thanks. Just continue composing this kind of post. taiwan coronavirus

Posted by Bull on February 14, 2020 at 12:15 AM UTC #

very good and very interesting article and this is one of the articles that is very useful and provides more knowledge, thank you , from : https://guidecrest.com.ng

Posted by Myname on February 14, 2020 at 12:17 AM UTC #

There's definately a lot to know about this issue.

Posted by Download Instagram Photos on February 15, 2020 at 07:21 AM UTC #

I was able to find good information

Posted by Download Instagram Video on February 15, 2020 at 07:22 AM UTC #

iCloud Login Process is Explained Here. If you have any issues in icloud sign in we are here to help you! Go to iCloud.com & login with your deatils.

Posted by icloud on February 17, 2020 at 04:54 AM UTC #

Through this step by step Guide, one can easily download Kik For PC (Windows 10,8, 7) using Bluestacks and Andy Emulator.

Posted by download kik for pc on February 17, 2020 at 04:55 AM UTC #

Hotmail Login is the most favorite among the corporate industry and students. Hotmail comes with many features

Posted by hotmail on February 17, 2020 at 04:55 AM UTC #

I think that thanks for the information and insights you have so provided here.

Posted by tell the bell on February 17, 2020 at 10:57 PM UTC #

Post a Comment:
  • HTML Syntax: NOT allowed



Hot Blogs (today's hits)

Tag Cloud