Release Notes - Kafka - Version 0.8.2.0

Sub-task

[KAFKA-784] - creating topic without partitions, deleting then creating with partition causes errors in 'kafka-list-topic'
[KAFKA-1012] - Implement an Offset Manager and hook offset requests to it
[KAFKA-1240] - Add ability to existing system tests to use the new producer client
[KAFKA-1304] - unregistered mbean exception in new producer
[KAFKA-1316] - Refactor Sender
[KAFKA-1328] - Add new consumer APIs
[KAFKA-1337] - Rationalize new producer configs
[KAFKA-1359] - Add topic/broker metrics once new topic/broker is discovered
[KAFKA-1462] - Add new request and response formats for the new consumer and coordinator communication
[KAFKA-1471] - Add Producer Unit Tests for LZ4 and LZ4HC compression
[KAFKA-1472] - Add the compression ratio metrics in the new producer
[KAFKA-1611] - Improve system test configuration
[KAFKA-1723] - make the metrics name in new producer more standard
[KAFKA-1729] - add doc for Kafka-based offset management in 0.8.2
[KAFKA-1730] - add the doc for the new java producer in 0.8.2
[KAFKA-1731] - add config/jmx changes in 0.8.2 doc

Bug

[KAFKA-179] - Log files always touched when broker is bounced
[KAFKA-294] - "Path length must be > 0" error during startup
[KAFKA-472] - update metadata in batches in Producer
[KAFKA-703] - A fetch request in Fetch Purgatory can double count the bytes from the same delayed produce request
[KAFKA-847] - kafka appender layout does not work for kafka 0.7.1
[KAFKA-899] - LeaderNotAvailableException the first time a new message for a partition is processed.
[KAFKA-924] - Specify console consumer properties via a single --property command line parameter
[KAFKA-1002] - Delete aliveLeaders field from LeaderAndIsrRequest
[KAFKA-1022] - Double quotes in Domain name & Topic name
[KAFKA-1025] - Producer.send should provide recoverability info on failiure
[KAFKA-1026] - Dynamically Adjust Batch Size Upon Receiving MessageSizeTooLargeException
[KAFKA-1034] - Improve partition reassignment to optimize writes to zookeeper
[KAFKA-1041] - Number of file handles increases indefinitely in producer if broker host is unresolvable
[KAFKA-1045] - producer zk.connect config
[KAFKA-1061] - Break-down sendTime to multipleSendTime
[KAFKA-1096] - An old controller coming out of long GC could update its epoch to the latest controller's epoch
[KAFKA-1108] - when controlled shutdown attempt fails, the reason is not always logged
[KAFKA-1123] - Broker IPv6 addresses parsed incorrectly
[KAFKA-1124] - Sending to a new topic (with auto.create.topics.enable) returns ERROR
[KAFKA-1146] - toString() on KafkaStream gets stuck indefinitely
[KAFKA-1147] - Consumer socket timeout should be greater than fetch max wait
[KAFKA-1179] - createMessageStreams() in javaapi.ZookeeperConsumerConnector does not throw
[KAFKA-1180] - WhiteList topic filter gets a NullPointerException on complex Regex
[KAFKA-1189] - kafka-server-stop.sh doesn't stop broker
[KAFKA-1192] - Enable DumpLogSegments tool to deserialize messages
[KAFKA-1193] - Data loss if broker is killed using kill -9
[KAFKA-1209] - System test transient failure due to ProducerPerformance's connection to servers kept broken
[KAFKA-1210] - Windows Bat files are not working properly
[KAFKA-1212] - System test exception handling does not stop background producer threads
[KAFKA-1213] - Adding fetcher needs to be avoided upon make-follower when replica manager is shutting down
[KAFKA-1235] - Enable server to indefinitely retry on controlled shutdown
[KAFKA-1238] - New producer hangs in a loop detecting metadata for auto created topics
[KAFKA-1276] - Provide a list of config overrides available when running kafka.topics
[KAFKA-1280] - exclude kafka-clients jar from dependant-libs dir
[KAFKA-1282] - Disconnect idle socket connection in Selector
[KAFKA-1297] - releaseTarGz target needs signing task
[KAFKA-1301] - system testcase_0206 fails using the new producer
[KAFKA-1302] - cleanup logging in new producer
[KAFKA-1305] - Controller can hang on controlled shutdown with auto leader balance enabled
[KAFKA-1307] - potential socket leak in new producer and clean up
[KAFKA-1318] - waiting for producer to stop is not reliable in system tests
[KAFKA-1319] - kafka jar doesn't depend on metrics-annotation any more
[KAFKA-1364] - ReplicaManagerTest hard-codes log dir
[KAFKA-1369] - snappy version update 1.1.x
[KAFKA-1375] - Formatting for "Running a task on a particular version of Scala" paragraph in README.md is broken
[KAFKA-1376] - transient test failure in UncleanLeaderElectionTest
[KAFKA-1378] - transient unit test failure in LogRecoveryTest
[KAFKA-1381] - transient unit test failure in AddPartitionsTest
[KAFKA-1382] - Update zkVersion on partition state update failures
[KAFKA-1383] - transient unit test failure in SocketServerTest
[KAFKA-1389] - transient unit test failure in ProducerFailureHandlingTest
[KAFKA-1391] - transient unit test failure in DeleteTopicTest.testPreferredReplicaElectionDuringDeleteTopic
[KAFKA-1395] - fix unit tests in AutoOffsetResetTest
[KAFKA-1396] - fix transient unit test ProducerFailureHandlingTest.testBrokerFailure
[KAFKA-1397] - delete topic is not working
[KAFKA-1400] - transient unit test failure in SocketServerTest
[KAFKA-1406] - Fix scaladoc/javadoc warnings
[KAFKA-1409] - oversized messages can slow down the brokers
[KAFKA-1410] - MetadataCache cleanup
[KAFKA-1412] - transient unit test failure in ProducerSendTest.testAutoCreateTopic
[KAFKA-1418] - transient unit test failure in ProducerFailureHandlingTest
[KAFKA-1422] - Fix transient unit test SocketServerTest
[KAFKA-1424] - transient unit test failure in testSendWithDeadBroker
[KAFKA-1431] - ConsoleConsumer - Option to clean zk consumer path
[KAFKA-1432] - Make num.producerThreads configurable on new MirrrorMaker
[KAFKA-1433] - transient unit test failure in ZookeeperConsumerConnectorTest
[KAFKA-1434] - kafka.admin.TopicCommand missing --delete topic command
[KAFKA-1437] - ConsumerMetadataResponse should always include coordinator information
[KAFKA-1438] - Migrate kafka client tools
[KAFKA-1439] - transient unit test failure in testDeleteTopicDuringAddPartition
[KAFKA-1442] - RBTools post-review is deprecated
[KAFKA-1444] - kafka.javaapi.TopicMetadata and PartitionMetadata doesn't forward the toString method
[KAFKA-1451] - Broker stuck due to leader election race
[KAFKA-1453] - Add a channel queue jmx in Mirror Maker
[KAFKA-1459] - kafka.tools.ConsumerOffsetChecker throws NoNodeException
[KAFKA-1468] - Improve perf tests
[KAFKA-1469] - Util.abs function does not return correct absolute values for negative values
[KAFKA-1473] - transient unit test failure in testRequestHandlingDuringDeleteTopic
[KAFKA-1481] - Stop using dashes AND underscores as separators in MBean names
[KAFKA-1482] - Transient test failures for kafka.admin.DeleteTopicTest
[KAFKA-1487] - add test jars to gradle build for packaging and release
[KAFKA-1490] - remove gradlew initial setup output from source distribution
[KAFKA-1491] - ConsumerMetadataResponse is not read completely
[KAFKA-1502] - source jar is empty
[KAFKA-1503] - all partitions are using same broker as their leader after broker is down
[KAFKA-1510] - Force offset commits when migrating consumer offsets from zookeeper to kafka
[KAFKA-1511] - Malformated link and consumer/producer mixup in documentation
[KAFKA-1515] - Wake-up Sender upon blocked on fetching leader metadata
[KAFKA-1529] - transient unit test failure in testAutoCreateAfterDeleteTopic
[KAFKA-1531] - zookeeper.connection.timeout.ms is set to 10000000 in configuration file in Kafka tarball
[KAFKA-1533] - transient unit test failure in ProducerFailureHandlingTest
[KAFKA-1539] - Due to OS caching Kafka might loose offset files which causes full reset of data
[KAFKA-1542] - normal IOException in the new producer is logged as ERROR
[KAFKA-1544] - LogCleaner may take a long time to shutdown
[KAFKA-1549] - dead brokers coming in the TopicMetadataResponse
[KAFKA-1557] - ISR reported by TopicMetadataResponse most of the time doesn't match the Zookeeper information (and the truth)
[KAFKA-1558] - AdminUtils.deleteTopic does not work
[KAFKA-1562] - kafka-topics.sh alter add partitions resets cleanup.policy
[KAFKA-1567] - Metric memory leaking after closing the clients
[KAFKA-1571] - MetadataeTest hangs
[KAFKA-1574] - unit tests can hang on socketserver shutdown
[KAFKA-1577] - Exception in ConnectionQuotas while shutting down
[KAFKA-1578] - Controller should de-register all listeners upon designation
[KAFKA-1580] - Reject producer requests to internal topics
[KAFKA-1585] - Client: Infinite "conflict in /consumers/"
[KAFKA-1598] - Fix variable typo in Kafka main class
[KAFKA-1600] - Controller failover not working correctly.
[KAFKA-1603] - MirrorMaker consumer does not put the message key into ProducerRecord
[KAFKA-1609] - New producer metadata response handling should only exclude a PartitionInfo when its error is LEADER_NOT_AVAILABLE
[KAFKA-1616] - Purgatory Size and Num.Delayed.Request metrics are incorrect
[KAFKA-1619] - perf dir can be removed
[KAFKA-1622] - project shouldn't require signing to build
[KAFKA-1624] - bump up default scala version to 2.10.4 to compile with java 8
[KAFKA-1635] - Java doc of makeLeaders in ReplicaManager is wrong
[KAFKA-1637] - SimpleConsumer.fetchOffset returns wrong error code when no offset exists for topic/partition/consumer group
[KAFKA-1642] - [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost
[KAFKA-1645] - some more jars in our src release
[KAFKA-1647] - Replication offset checkpoints (high water marks) can be lost on hard kills and restarts
[KAFKA-1653] - Duplicate broker ids allowed in replica assignment
[KAFKA-1662] - gradle release issue permgen space
[KAFKA-1663] - Controller unable to shutdown after a soft failure
[KAFKA-1670] - Corrupt log files for segment.bytes values close to Int.MaxInt
[KAFKA-1671] - uploaded archives are missing for Scala version 2.11
[KAFKA-1673] - potential java.lang.IllegalStateException in BufferPool.allocate()
[KAFKA-1674] - auto.create.topics.enable docs are misleading
[KAFKA-1681] - Newly elected KafkaController might not start deletion of pending topics
[KAFKA-1699] - autoRebalanceScheduler.shutdown() causes deadlock while controller shutting down
[KAFKA-1717] - remove netty dependency through ZK 3.4.x
[KAFKA-1721] - Snappy compressor is not thread safe
[KAFKA-1724] - Errors after reboot in single node setup
[KAFKA-1732] - DumpLogSegments tool fails when path has a '.'
[KAFKA-1738] - Partitions for topic not created after restart from forced shutdown
[KAFKA-1742] - ControllerContext removeTopic does not correctly update state
[KAFKA-1743] - ConsumerConnector.commitOffsets in 0.8.2 is not backward compatible
[KAFKA-1761] - num.partitions documented default is 1 while actual default is 2
[KAFKA-1764] - ZookeeperConsumerConnector could put multiple shutdownCommand to the same data chunk queue.
[KAFKA-1766] - Ecosystem docs subsection has wrong anchor
[KAFKA-1781] - Readme should specify that Gradle 2.0 is required for initial bootstrap
[KAFKA-1787] - Consumer should delete offsets and release partition ownership for deleted topics
[KAFKA-1799] - ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG doesn't work
[KAFKA-1806] - broker fetch request uses old leader offset which is higher than current leader offset causes error
[KAFKA-1819] - Cleaner gets confused about deleted and re-created topics
[KAFKA-1841] - OffsetCommitRequest API - timestamp field is not versioned
[KAFKA-1851] - OffsetFetchRequest returns extra partitions when input only contains unknown partitions
[KAFKA-1855] - Topic unusable after unsuccessful UpdateMetadataRequest
[KAFKA-1861] - Publishing kafka-client:test in order to utilize the helper utils in TestUtils
[KAFKA-1864] - Revisit defaults for the internal offsets topic
[KAFKA-1868] - ConsoleConsumer shouldn't override dual.commit.enabled to false if not explicitly set
[KAFKA-1869] - Openning some random ports while running kafka service
[KAFKA-1870] - Cannot commit with simpleConsumer on Zookeeper only with Java API
[KAFKA-1873] - scalatest_2.10-1.9.1.jar of core build path is cross-compiled with an incompatible version of Scala (2.10.0)
[KAFKA-1874] - missing import util.parsing.json.JSON
[KAFKA-1876] - pom file for scala 2.11 should reference a specific version
[KAFKA-1897] - Enhance MockProducer for more sophisticated tests
[KAFKA-1902] - fix MetricName so that Yammer reporter can work correctly

Improvement

[KAFKA-404] - When using chroot path, create chroot on startup if it doesn't exist
[KAFKA-997] - Provide a strict verification mode when reading configuration properties
[KAFKA-1028] - per topic configuration of preference for consistency over availability
[KAFKA-1204] - Add KAFKA_LOG_DIR to allow LOG_DIR to be outside of code dir
[KAFKA-1264] - Make ConsoleProducer compression codec configurable
[KAFKA-1278] - More flexible helper scripts
[KAFKA-1287] - enable log4j in command line tools using the new producer
[KAFKA-1291] - Make wrapper shell scripts for important tools
[KAFKA-1294] - Enable leadership balancing by default
[KAFKA-1298] - Controlled shutdown tool doesn't seem to work out of the box
[KAFKA-1312] - Update gitignore to ignore IDE generated files
[KAFKA-1325] - Fix inconsistent per topic log configs
[KAFKA-1344] - Kafka-console-producer.sh support snappy compression
[KAFKA-1348] - Producer's Broker Discovery Interface
[KAFKA-1353] - report capacity used by request thread pool and network thread pool
[KAFKA-1361] - enable controlled shutdown by default
[KAFKA-1390] - TestUtils.waitUntilLeaderIsElectedOrChanged may wait longer than it needs
[KAFKA-1392] - all TestUtiles.waitUntilTrue() should throw an exception if the return value is false
[KAFKA-1414] - Speedup broker startup after hard reset
[KAFKA-1419] - cross build for scala 2.11
[KAFKA-1425] - provide logStartOffset in addition to LogEndOffset metric
[KAFKA-1430] - Purgatory redesign
[KAFKA-1446] - Consumer metrics for rebalance
[KAFKA-1454] - Update for a Scala 2.11 build
[KAFKA-1456] - Add LZ4 and LZ4C as a compression codec
[KAFKA-1480] - support log.roll.minutes
[KAFKA-1488] - new metrics for measuring the ratio when the new producer is block for space allocation
[KAFKA-1493] - Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[KAFKA-1513] - Cleanup KafkaServerStartable code
[KAFKA-1519] - Console consumer: expose configuration option to enable/disable writing the line separator
[KAFKA-1535] - return all live brokers in TopicMetadataResponse
[KAFKA-1555] - provide strong consistency with reasonable availability
[KAFKA-1576] - Make "deleteTopic" a bit more user friendly
[KAFKA-1618] - Exception thrown when running console producer with no port number for the broker
[KAFKA-1620] - Make kafka api protocol implementation public
[KAFKA-1644] - Inherit FetchResponse from RequestOrResponse
[KAFKA-1692] - [Java New Producer] IO Thread Name Must include Client ID
[KAFKA-1706] - Adding a byte bounded blocking queue to util.
[KAFKA-1719] - Make mirror maker exit when one consumer/producer thread exits.
[KAFKA-1768] - Expose version via JMX
[KAFKA-1769] - javadoc should only include client facing packages
[KAFKA-1797] - add the serializer/deserializer api to the new java client
[KAFKA-1835] - Kafka new producer needs options to make blocking behavior explicit
[KAFKA-1872] - Update Developer Setup
[KAFKA-1879] - Log warning when receiving produce requests with acks > 1

New Feature

[KAFKA-1000] - Inbuilt consumer offset management feature for kakfa
[KAFKA-1227] - Code dump of new producer
[KAFKA-1384] - Log Broker state
[KAFKA-1443] - Add delete topic to topic commands and update DeleteTopicCommand
[KAFKA-1512] - Limit the maximum number of connections per ip address
[KAFKA-1597] - New metrics: ResponseQueueSize and BeingSentResponses
[KAFKA-1784] - Implement a ConsumerOffsetClient library

Task

[KAFKA-1399] - Drop Scala 2.8.x support
[KAFKA-1559] - Upgrade Gradle wrapper to Gradle 2.0
[KAFKA-1728] - update 082 docs

Test

[KAFKA-1079] - Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually
[KAFKA-1285] - enable log4j in unit test
[KAFKA-1411] - remove unused test cases

Wish

[KAFKA-1308] - Publish jar of test utilities to Maven
[KAFKA-1370] - Gradle startup script for Windows
[KAFKA-1371] - Ignore build output dirs
[KAFKA-1485] - Upgrade to Zookeeper 3.4.6 and create "shim" for ZKCLI so system tests can run