Release Notes - Kafka - Version 2.3.0

Below is a summary of the JIRA issues addressed in the 2.3.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

New Feature

[KAFKA-4730] - Streams does not have an in-memory windowed store
[KAFKA-7283] - mmap indexes lazily and skip sanity check for segments below recovery point
[KAFKA-7730] - Limit total number of active connections in the broker
[KAFKA-7904] - KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)
[KAFKA-7922] - Returned authorized operations in describe responses (KIP-430)
[KAFKA-8265] - Connect Client Config Override policy
[KAFKA-8365] - Protocol and consumer support for follower fetching

Improvement

[KAFKA-3816] - Provide more context in Kafka Connect log messages using MDC
[KAFKA-5505] - Connect: Do not restart connector and existing tasks on task-set change
[KAFKA-6161] - Add default implementation to close() and configure() for Serializer, Deserializer and Serde
[KAFKA-6399] - Consider reducing "max.poll.interval.ms" default for Kafka Streams
[KAFKA-6455] - Improve timestamp propagation at DSL level
[KAFKA-6474] - Rewrite test to use new public TopologyTestDriver
[KAFKA-6521] - Store record timestamps in KTable stores
[KAFKA-7236] - Add --under-min-isr option to describe topics command
[KAFKA-7243] - Add unit integration tests to validate metrics in Kafka Streams
[KAFKA-7320] - Provide ability to disable auto topic creation in KafkaConsumer
[KAFKA-7321] - ensure timely processing of deletion requests in Kafka topic (Time-based log compaction)
[KAFKA-7440] - Use leader epoch in consumer fetch requests
[KAFKA-7466] - Implement KIP-339: Create a new IncrementalAlterConfigs API
[KAFKA-7487] - DumpLogSegments reports mismatches for indexed offsets which are not at the start of a record batch
[KAFKA-7492] - Explain `null` handling for reduce and aggregate
[KAFKA-7747] - Consumer should check for truncation after leader changes
[KAFKA-7811] - Avoid unnecessary lock acquire when KafkaConsumer commits offsets
[KAFKA-7847] - KIP-421: Automatically resolve external configurations.
[KAFKA-7864] - AdminZkClient.validateTopicCreate() should validate that partitions are 0-based
[KAFKA-7875] - Add KStream#flatTransformValues
[KAFKA-7918] - Streams store cleanup: inline byte-store generic parameters
[KAFKA-7930] - StreamsResetter makes "changelog" topic naming assumptions
[KAFKA-7933] - KTableKTableLeftJoinTest takes an hour to finish
[KAFKA-7944] - Add more natural Suppress test
[KAFKA-7956] - Avoid blocking in ShutdownableThread.awaitShutdown if the thread has not been started
[KAFKA-7963] - Extract hard-coded Streams metric name strings to centralized place
[KAFKA-7986] - distinguish the logging from different ZooKeeperClient instances
[KAFKA-7992] - Add a server start time metric
[KAFKA-8006] - Guard calls to init() and close() within user code
[KAFKA-8007] - In-memory window store copies on fetch
[KAFKA-8014] - Extend Connect integration tests to add and remove workers dynamically
[KAFKA-8029] - Add in-memory bytes-only session store implementation
[KAFKA-8060] - The Kafka protocol generator should allow null default values for strings
[KAFKA-8094] - Iterating over cache with get(key) is inefficient
[KAFKA-8126] - Flaky Test org.apache.kafka.connect.runtime.WorkerTest.testAddRemoveTask
[KAFKA-8131] - Add --version parameter to command line help outputs & docs
[KAFKA-8168] - Add a generated ApiMessageType class
[KAFKA-8169] - Wrong topic on streams quick start documentation
[KAFKA-8215] - Limit memory usage of RocksDB
[KAFKA-8231] - Expansion of ConnectClusterState interface
[KAFKA-8294] - Batch StopReplica requests with partition deletion and add test cases
[KAFKA-8309] - KIP-465: Add Consolidated Connector Endpoint to Connect REST API
[KAFKA-8336] - Enable dynamic update of client-side SSL factory in brokers
[KAFKA-8346] - Improve replica fetcher behavior in handling partition failures
[KAFKA-8347] - Choose next record to process by timestamp
[KAFKA-8348] - Document of kafkaStreams improvement
[KAFKA-8428] - Cleanup LogValidator#validateMessagesAndAssignOffsetsCompressed to assume single record batch only

Bug

[KAFKA-3143] - inconsistent state in ZK when all replicas are dead
[KAFKA-3522] - Consider adding version information into rocksDB storage format
[KAFKA-4893] - async topic deletion conflicts with max topic length
[KAFKA-5061] - client.id should be set for Connect producers/consumers
[KAFKA-5141] - WorkerTest.testCleanupTasksOnStop transient failure due to NPE
[KAFKA-6569] - Reflection in OffsetIndex and TimeIndex construction
[KAFKA-6635] - Producer close does not await pending transaction
[KAFKA-7026] - Sticky assignor could assign a partition to multiple consumers (KIP-341)
[KAFKA-7455] - JmxTool cannot connect to an SSL-enabled JMX RMI port
[KAFKA-7652] - Kafka Streams Session store performance degradation from 0.10.2.2 to 0.11.0.0
[KAFKA-7703] - KafkaConsumer.position may return a wrong offset after "seekToEnd" is called
[KAFKA-7763] - KafkaProducer with transactionId endless waits when network is disconnection for 10-20s
[KAFKA-7801] - TopicCommand should not be able to alter transaction topic partition count
[KAFKA-7813] - JmxTool throws NPE when --object-name is omitted
[KAFKA-7831] - Consumer SubscriptionState missing synchronization
[KAFKA-7855] - Kafka Streams Maven Archetype quickstart fails to compile out of the box
[KAFKA-7896] - Add some Log4J Kafka Properties for Producing to Secured Brokers
[KAFKA-7912] - In-memory key-value store does not support concurrent access
[KAFKA-7921] - Instable KafkaStreamsTest
[KAFKA-7939] - Flaky Test KafkaAdminClientTest#testCreateTopicsRetryBackoff
[KAFKA-7946] - Flaky Test DeleteConsumerGroupsTest#testDeleteNonEmptyGroup
[KAFKA-7961] - Handle subscription changes with a rebalance in progress
[KAFKA-7962] - StickyAssignor: throws NullPointerException during assignments if topic is deleted
[KAFKA-7965] - Flaky Test ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
[KAFKA-7974] - KafkaAdminClient loses worker thread/enters zombie state when initial DNS lookup fails
[KAFKA-7976] - Flaky Test DynamicBrokerReconfigurationTest#testUncleanLeaderElectionEnable
[KAFKA-7989] - Flaky Test RequestQuotaTest#testResponseThrottleTimeWhenBothFetchAndRequestQuotasViolated
[KAFKA-8013] - Avoid buffer underflow when reading a Struct from a partially correct buffer
[KAFKA-8052] - Intermittent INVALID_FETCH_SESSION_EPOCH error on FETCH request
[KAFKA-8058] - ConnectRestExtensionContext does not return list of connectors from its ConnectClusterState
[KAFKA-8062] - StateListener is not notified when StreamThread dies
[KAFKA-8063] - Flaky Test WorkerTest#testConverterOverrides
[KAFKA-8091] - Flaky test DynamicBrokerReconfigurationTest#testAddRemoveSaslListener
[KAFKA-8098] - Flaky Test AdminClientIntegrationTest#testConsumerGroups
[KAFKA-8111] - KafkaProducer can't produce data
[KAFKA-8114] - Flaky Test DelegationTokenEndToEndAuthorizationTest#testNoGroupAcl
[KAFKA-8121] - ZooKeeperClient session expiry thread may leave clients behind after close
[KAFKA-8134] - ProducerConfig.LINGER_MS_CONFIG undocumented breaking change in kafka-clients 2.1
[KAFKA-8142] - Kafka Streams fails with NPE if record contains null-value in header
[KAFKA-8157] - Missing "key.serializer" exception when setting "segment index bytes"
[KAFKA-8181] - Streams docs on serialization include Avro header, but no content
[KAFKA-8187] - State store record loss across multiple reassignments when using standby tasks
[KAFKA-8190] - Keystore update without file change doesn't update SSLContext
[KAFKA-8199] - ClassCastException when trying to groupBy after suppress
[KAFKA-8204] - Streams may flush state stores in the incorrect order
[KAFKA-8208] - Broken link for out-of-order data in KStreams Core Concepts doc
[KAFKA-8209] - Wrong link for KStreams DSL in Core Concepts doc
[KAFKA-8210] - Missing link for KStreams in Streams DSL docs
[KAFKA-8227] - Missing Links in Duality of Streams and Tables section
[KAFKA-8229] - Connect Sink Task updates nextCommit when commitRequest is true
[KAFKA-8232] - Flaky test kafka.admin.TopicCommandWithAdminClientTest.testTopicDeletion
[KAFKA-8240] - Source.equals() can fail with NPE
[KAFKA-8241] - Dynamic update of keystore fails on listener without truststore
[KAFKA-8254] - Suppress incorrectly passes a null topic to the serdes
[KAFKA-8275] - NetworkClient leastLoadedNode selection should consider throttled nodes
[KAFKA-8277] - Fix NPE in ConnectHeaders
[KAFKA-8289] - KTable, Long> can't be suppressed
[KAFKA-8290] - Streams Not Closing Fenced Producer On Task Close
[KAFKA-8291] - System test consumer_test.py failed on trunk
[KAFKA-8298] - ConcurrentModificationException Possible when optimizing for repartition nodes
[KAFKA-8304] - Connect susceptible to deadlock while registering REST extensions
[KAFKA-8313] - KafkaStreams state not being updated properly after shutdown
[KAFKA-8320] - Connect Error handling is using the RetriableException from common package
[KAFKA-8323] - Memory leak of BloomFilter Rocks object
[KAFKA-8324] - User constructed RocksObjects leak memory
[KAFKA-8332] - Regression in handling of JoinGroupRequest disallows deterministic protocol selection based on order of preference
[KAFKA-8344] - Fix vagrant-up.sh to work with AWS properly
[KAFKA-8351] - Log cleaner must handle transactions spanning multiple segments
[KAFKA-8352] - Connect System Tests are failing with 404
[KAFKA-8363] - Config provider parsing is broken
[KAFKA-8376] - Flaky test ClientAuthenticationFailureTest.testTransactionalProducerWithInvalidCredentials test.
[KAFKA-8379] - Flaky test KafkaAdminClientTest.testUnreachableBootstrapServer
[KAFKA-8381] - SSL factory for inter-broker listener is broken
[KAFKA-8404] - Authorization header is not passed in Connect when forwarding REST requests
[KAFKA-8407] - Connector client overrides broken on client configs with type 'Class' or 'List'
[KAFKA-8415] - Interface ConnectorClientConfigOverridePolicy needs to be excluded from class loading isolation
[KAFKA-8418] - Connect System tests are not waiting for REST resources to be registered
[KAFKA-8422] - Client should not use old versions of OffsetsForLeaderEpoch
[KAFKA-8425] - KIP 421 Bug: Modifying Immutable Originals Map results in Java exception
[KAFKA-8426] - KIP 421 Bug: ConfigProvider configs param inconsistent with KIP-297
[KAFKA-8446] - Kafka Streams restoration crashes with NPE when the record value is null
[KAFKA-8449] - Restart task on reconfiguration under incremental cooperative rebalancing
[KAFKA-8463] - Fix redundant reassignment of tasks when leader worker leaves
[KAFKA-8473] - Adjust Connect system tests for incremental cooperative rebalancing and enable them for both eager and incremental cooperative rebalancing
[KAFKA-8475] - Temporarily restore SslFactory.sslContext() helper
[KAFKA-8483] - Possible reordering of messages by producer after UNKNOWN_PRODUCER_ID error
[KAFKA-8484] - ProducerId reset can cause IllegalStateException

Task

[KAFKA-7766] - Improve fail-fast behavior of Jenkins build
[KAFKA-8155] - Update Streams system tests for 2.2.0 and 2.1.1 releases
[KAFKA-8308] - Update jetty for security vulnerability CVE-2019-10241

Wish

[KAFKA-7950] - Kafka tools GetOffsetShell -time description

Test

[KAFKA-8118] - Ensure that tests close ZooKeeper clients since they can impact subsequent tests

Sub-task

[KAFKA-7832] - Use automatic RPC generation in CreateTopics
[KAFKA-7862] - Modify JoinGroup logic to incorporate group.instance.id change
[KAFKA-7972] - Replace SaslHandshake request/response with automated protocol
[KAFKA-7997] - Replace SaslAuthenticate request/response with automated protocol
[KAFKA-8090] - Replace ControlledShutdown request/response with automated protocol
[KAFKA-8220] - Avoid kicking out members through rebalance timeout
[KAFKA-8225] - Handle conflicting static member id
[KAFKA-8284] - Enable static membership on KStream
[KAFKA-8285] - Handle thread-id random switch on JVM for KStream
[KAFKA-8373] - Add group.instance.id field into Sync/Heartbeat/OffsetCommit protocols
[KAFKA-8386] - Use COORDINATOR_NOT_AVAILABLE to replace UNKNOWN_MEMBER_ID when the group is not available