Releases Β· confluentinc/librdkafka Β· GitHub
Skip to content

Releases: confluentinc/librdkafka

v2.15.1

Choose a tag to compare

@emasab emasab released this 09 Sep 17:12
c58bbed

librdkafka v2.15.1 is a maintenance release:

  • Update bundled OpenSSL and libcurl dependencies, and refresh the Windows
    build toolchain (msys2, vcpkg) (#5579).
  • Fix int_latency metric calculation, that was reporting bogus values when
    message.timeout.ms is set to 0 (infinite) (#5335).
  • IPv6 addresses are wrapped in square brackets following RFC 3986, allowing to connect to those compressed IPv6 addresses (#5544).
  • IPv6 addresses are correctly passed to OpenSSL as IPs not as hostnames with brackets and the certificate is validated against its iPAddress entries instead of against dNSName entries (#5544).

Upgrade considerations

  • If you're parsing the nodename as received by connect_cb, stats_cb, ssl_cert_verify_cb or throttle_cb, make sure you're correctly parsing it following RFC 3986. Metadata API calls are unaffected: they return the host and port as separate fields.
  • Make sure the brokers' certificates carry the IP in an iPAddress entry, not a dNSName entry. Previously the certificate was validated against dNSName (falling back to the subject CN) whenever the address could not be parsed as an IP literal: with OpenSSL < 3.0 or BoringSSL for any IP address, and on all OpenSSL versions for an IPv6 address configured in brackets in bootstrap.servers ([::1]:9092), which never verified. A scoped address (fe80::1%eth0) was likewise dNSName-matched on every version and now has the zone stripped before matching. IPv6 addresses ending in :: could not be connected to at all, as the nodename became host:::port and failed name resolution. In the remaining cases, a bare IPv4 or IPv6 literal on OpenSSL >= 3.0, the iPAddress entry was already used, and behaviour is unchanged.

Security considerations

Bundled dependencies were further upgraded as follows:
OpenSSL 3.5.6 β†’ 3.5.7 for source/autoconf builds, and 3.6.2 β†’ 3.6.3 for
vcpkg-based packages; libcurl 8.20.0 β†’ 8.21.0, now used by both
source/autoconf builds and vcpkg (previously pinned to 8.19.0 in vcpkg,
so vcpkg-based packages also pick up the fixes below that source/autoconf
builds already got from 8.20.0).

Fixes

Producer fixes

  • Issues: #5555.
    Fix int_latency metric calculation. It was derived from the message
    timeout timestamp (now + message.timeout.ms - rkm_ts_timeout), which
    yields a large negative value when message.timeout.ms is 0 (infinite)
    and rkm_ts_timeout is INT64_MAX. It's now computed directly as
    now - rkm_ts_enq, the actual time the message spent in the queue,
    regardless of the timeout setting.
    Happening since 0.11.0 (#5335).

Checksums

Release asset checksums:

  • v2.15.1.zip SHA256 1ca308c10c4234962695fbf9c7f1ab0649ab43c7ad03d384f1645db7b74c92ce
  • v2.15.1.tar.gz SHA256 23c8575c7d1ced07246cb9cf200c11325b72201fd4134a02414ca869fbdd8ed3

v2.15.0

Choose a tag to compare

@PratRanj07 PratRanj07 released this 30 Jun 19:23
9a94e11

KIP-932 Queues for Kafka – Now in Preview

  • Added a preview implementation of the share consumer (Queues for Kafka,
    KIP-932).
    Members of a share group cooperatively consume from the same partitions with
    per-record acquire/acknowledge semantics and redelivery, providing queue-like
    consumption on top of Kafka.
  • New rd_kafka_share_* public API, with a dedicated rd_kafka_share_t handle
    created via rd_kafka_share_consumer_new():
    • Subscription: rd_kafka_share_subscribe(), rd_kafka_share_unsubscribe(),
      rd_kafka_share_subscription().
    • Batch polling: rd_kafka_share_poll() returns an rd_kafka_messages_t
      batch (rd_kafka_messages_count() / rd_kafka_messages_get() /
      rd_kafka_messages_destroy()).
    • Acknowledgement: rd_kafka_share_acknowledge(),
      rd_kafka_share_acknowledge_type(), rd_kafka_share_acknowledge_offset()
      with ACCEPT / RELEASE / REJECT types, and
      rd_kafka_message_delivery_count().
    • Commit: rd_kafka_share_commit_sync(), rd_kafka_share_commit_async() and
      the acknowledgement-commit callback
      (rd_kafka_share_set_acknowledgement_commit_cb()).
    • Lifecycle: rd_kafka_share_consumer_close(),
      rd_kafka_share_consumer_close_queue(), rd_kafka_share_destroy().
  • Two acknowledgement modes selected by share.acknowledgement.mode
    (default implicit; explicit requires the application to acknowledge every
    record before the next poll).
  • New max.poll.records property (default 500) and adjusted defaults for
    several network properties for share consumers (receive.message.max.bytes,
    connections.max.idle.ms, reconnect.backoff.ms,
    reconnect.backoff.max.ms).
  • See the Share consumers (Queues for Kafka) section of
    INTRODUCTION.md, the
    Share consumer section in rdkafka.h, and the
    examples/share_consumer* programs.

Note

The KIP-932
share consumer is currently in Preview and should not be used in
production environments. The public interfaces may change before General
Availability, and known limitations apply (see
INTRODUCTION.md). The
share consumer is single-threaded and not thread-safe by design. It requires
a broker with share groups enabled (generally available in Apache Kafka
4.2.0).

Enhancements

  • Add aws_iam option to sasl.oauthbearer.metadata.authentication.type, with a defensive stub that fails when no token-refresh callback is registered.

Fixes

General fixes

  • Issues: #5135.
    Fix compilation with CMake when CURL is disabled.
    The OAuthBearer OIDC code included <curl/curl.h> under #ifdef WITH_OAUTHBEARER_OIDC, but
    CMake always defines that macro (to 0 or 1), so CURL was required even when it was turned off.
    Happening since 2.11.0 (#5136).
  • Issues: #5282.
    Fix rd_atomic32_set/rd_atomic64_set returning the new value instead of the previous one in CMake builds.
    CMake never defined HAVE_ATOMICS_{32,64}_ATOMIC, so the setters used a non-atomic fallback that
    returned the new value, which prevented the ALL_BROKERS_DOWN event from being raised under CMake.
    Happening since 2.11.1 (#5136).

Consumer fixes

  • Issues: #5541.
    Improve error handling in the KIP-848 consumer group protocol:
    • Defer the leave heartbeat until the assignment is revoked, so a member that
      exceeds max.poll.interval.ms rejoins cleanly instead of being rejected
      with a fatal INVALID_REQUEST.
    • Treat GROUP_ID_NOT_FOUND in the ConsumerGroupHeartbeat response as fatal,
      except while the member is already leaving.
    • Surface unexpected permanent broker-level heartbeat errors instead of
      retrying them in a loop; internal transport/timeout codes keep their
      existing retry/reconnect handling.
      Happening since 2.12.0 (#5488)

Checksums

Release asset checksums:

  • v2.15.0.zip SHA256 d0655c770f7edc5699855f65b391d1e20dc24fc03ecf2dd680665934dd4d1cc1
  • v2.15.0.tar.gz SHA256 259015220cdca708afe838b5aa79ebf1a5fb710fb4179cf918d390aed85d5dbc

v2.14.2

Choose a tag to compare

@Devarsh010 Devarsh010 released this 03 Jun 10:28
6516057

librdkafka v2.14.2 is a maintenance release:

  • Fix duplicate groups in ListConsumerGroups when multiple brokers
    return the same group (#5417).
  • Fix data race in timers (#5089).
  • Update bundled OpenSSL, libcurl, zstd, zlib and cJSON
    dependencies (#5346).

Security considerations

Bundled dependencies were upgraded as follows (see #5346):
OpenSSL 3.0.15 β†’ 3.5.6 (LTS) for source/autoconf builds, and to 3.6.2 in
vcpkg-based packages (no LTS available in vcpkg); libcurl 8.10.1 β†’ 8.20.0
for source/autoconf builds and to 8.19.0 in vcpkg; zlib 1.3.1 β†’ 1.3.2;
zstd 1.5.6 β†’ 1.5.7; cJSON 1.7.14 β†’ 1.7.19.

Fixes

General fixes

  • Issues: #5082.
    Fix data race in timers. The callback and its argument could have been modified after the lock is released.
    Happening since 1.x (#5089).

Consumer fixes

  • Fix crash (SIGSEGV) in rd_kafka_cgrp_handle_LeaveGroup() when coordinator
    is unavailable during consumer close. The error logging path dereferenced
    a potentially NULL broker pointer. Happening since 1.x.

Admin client fixes

  • Issues: #5417.
    Fix duplicate groups in ListConsumerGroups when multiple brokers return the same group.
    Happening since 1.x (#5417).

Checksums

Release asset checksums:

  • v2.14.2.zip SHA256 2c0a563a39d5c1bc2e7b3ae81bbad9aca23c586ddab9f659b51983d4dc67cffb
  • v2.14.2.tar.gz SHA256 d7eec9c31c817fa44402f679c252dfbf97e4c338a849a25c3579a31fd127beb8

v2.14.1

Choose a tag to compare

@Devarsh010 Devarsh010 released this 15 Apr 05:22
e0da09c

librdkafka v2.14.1 is a maintenance release:

  • Bundle prebuilt binaries for linux-s390x (#5365).

Checksums

Release asset checksums:

  • v2.14.1.zip SHA256 b6f52ae7a743e504e416e620c94e1192906daf15eafe661e4e2e6a7b793efc06
  • v2.14.1.tar.gz SHA256 bb246e754dee3560e9b42bf4e844dc05de4b146a3cae937e36301ffacdc456e7

v2.14.0

Choose a tag to compare

@emasab emasab released this 01 Apr 13:57
f1b831e

librdkafka v2.14.0 is a feature release:

  • KIP-768 Extend SASL/OAUTHBEARER to support OIDC claim mapping beyond the default sub claim (#5336).

Checksums

Release asset checksums:

  • v2.14.0.zip SHA256 372589ac63b06f9cac5d9b50d4ed1998f46b1a6cca991b691527ccc62b7cb7dd
  • v2.14.0.tar.gz SHA256 c05c03ef00a13a8463fac3e8918c04843c416f11ced58c889d806a88ca92cf99

v2.13.2

Choose a tag to compare

@PratRanj07 PratRanj07 released this 02 Mar 18:32
901af7c

librdkafka v2.13.2 is a maintenance release:

  • The librdkafka.redist NuGet package now includes binary for alpine-arm64 (#5237, @mclayton7)
  • Remove CPU usage regression when a subscription matches no topics (#5324).
  • Fix rd_kafka_consume_batch_queue incorrectly updating the application position on EOF or error messages (#5213).
  • Fix compilation without getentropy (Oleg Babin (@olegrok), Leo Singer (@lpsinger), #5288).
  • Use a truly random seed for pseudo-random number generation whenever available (#5288).
  • Fix rd_list destroy callback type mismatch by changing rd_kafka_assignor_destroy to take a void * argument, as expected by rd_list_init() destroy callbacks, and casting internally to rd_kafka_assignor_t * (#5195) (#5278).

Fixes

General fixes

Consumer fixes

  • Issues: #5324. Remove CPU usage regression when a subscription matches no topics. The increased CPU usage (~30%) was seen in particular when there are many topics in the clusters and the given subscription regex doesn't match any. Happening since 2.10.0 (#5324).
  • Issues: #4844. Fix rd_kafka_consume_batch_queue incorrectly updating the application position when receiving EOF or error messages, causing the position to move forward and likely be stored and committed. When storing the application offset the leader epoch is also considered for correct offset ordering in case of log truncation. Happening since 2.2.0 (#5213).

Checksums

Release asset checksums:

  • v2.13.2.zip SHA256 1b71b01a33f54c5d3e359935f73445fde4010215efdb899e3a746475d6aa178a
  • v2.13.2.tar.gz SHA256 14972092e4115f6e99f798a7cb420cbf6daa0c73502b3c52ae42fb5b418eea8f

v2.13.0

Choose a tag to compare

@k-raina k-raina released this 05 Jan 12:59
59b2f66

librdkafka v2.13.0 is a feature release:

  • KIP-482 Upgrade CreateAcls, DescribeAcls, DeleteAcls to the first version supporting this KIP (#5081).
  • KIP-482 Upgrade DescribeGroups, DeleteTopics, DeleteRecords, CreatePartitions, DeleteGroups to the first version supporting this KIP (#5083).
  • Strip trailing dot of hostname to fix SSL certificate verification issue (#5253).
  • Fix memory management for interceptors in rd_kafka_conf to prevent
    double-free errors (#5240).
  • Fix for the pseudo-random generator seed on Windows involving as well
    the uniqueness of the new consumer group protocol member id (#5265).
  • Add secure random generation functionality used for UUID uniqueness
    and secure salt generation in rd_kafka_UserScramCredentialUpsertion
    using OpenSSL or the POSIX or WIN32 equivalent calls when it
    isn't available (#5265).

Fixes

General fixes

  • Issues: #4348.
    Strip trailing dot of hostname to fix SSL certificate verification issue.
    Happening since 1.x (#5253).
  • Issues: #4142.
    Fix memory management for interceptors in rd_kafka_conf to prevent double-free errors.
    In case the client instance fails the users needs to destroy the configuration
    data structure, it was causing a double-free because the interceptors were
    already freed in the constructor.
    Happening since 1.x (#5240).
  • Issues: #5263, #3929.
    Fix for the pseudo-random seed on Windows. The function rand_r isn't present
    on Windows and the global seed wasn't based on the current microseconds and thread
    id. Also it wasn't called on every thread as required on this platform but
    only once per process. The fix allows on this platform the uniqueness of client side
    member id generation in next-generation consumer group protocol.
    Happening since 1.x (#5265).

Checksums

Release asset checksums:

  • v2.13.0.zip SHA256 73d731322b34c59fb5245d27172c71824e9323acd934e214d70a77954749e79d
  • v2.13.0.tar.gz SHA256 3bd351601d8ebcbc99b9a1316cae1b83b00edbcf9411c34287edf1791c507600

New Contributors

Full Changelog: v2.12.1...v2.13.0

v2.12.1

Choose a tag to compare

@emasab emasab released this 21 Oct 15:59
e1db7ea

librdkafka v2.12.1 is a maintenance release:

  • Restored macOS binaries compatibility with macOS 13 and 14 (#5219).

Fixes

General fixes

  • Fix to restore macOS 13 and 14 compatibility in prebuilt binaries present in librdkafka.redist.
    Happening since 2.12.0 (#5219).

Checksums

Release asset checksums:

  • v2.12.1.zip SHA256 da7571a0c1dc374aabb18af6ca01411d4bc597d321977980c8d3211ec5adf696
  • v2.12.1.tar.gz SHA256 ec103fa05cb0f251e375f6ea0b6112cfc9d0acd977dc5b69fdc54242ba38a16f

v2.12.0

Choose a tag to compare

@emasab emasab released this 08 Oct 18:36
2f22088

librdkafka v2.12.0 is a feature release:

KIP-848 – General Availability

Starting with librdkafka 2.12.0, the next generation consumer group rebalance protocol defined in KIP-848 is production-ready. Please refer the following migration guide for moving from classic to consumer protocol.

Note: The new consumer group protocol defined in KIP-848 is not enabled by default. There are few contract change associated with the new protocol and might cause breaking changes. group.protocol configuration property dictates whether to use the new consumer protocol or older classic protocol. It defaults to classic if not provided.

Enhancements and Fixes

  • Support for OAUTHBEARER metadata based authentication types,
    starting with Azure IMDS. Introduction available (#5155).
  • Fix compression types read issue in GetTelemetrySubscriptions response
    for big-endian architectures (#5183, Faidon Liambotis (@paravoid)).
  • Fix for KIP-1102 time based re-bootstrap condition (#5177).
  • Fix for discarding the member epoch in a consumer group heartbeat response when leaving with an inflight HB (#4672).
  • Fix for an error being raised after a commit due to an existing error in the topic partition (#4672).
  • Fix double free of headers in rd_kafka_produceva method (blindspot (@blindspotbounty), #4628).
  • Fix to ensure rd_kafka_query_watermark_offsets enforces the specified timeout and does not continue beyond timeout expiry (#5201).
  • New walkthrough in the Wiki about configuring Kafka cross-realm authentication between Windows SSPI and MIT Kerberos.

Fixes

General fixes

  • Issues: #5178.
    Fix for KIP-1102 time based re-bootstrap condition.
    Re-bootstrap is now triggered only after metadata.recovery.rebootstrap.trigger.ms
    have passed since first metadata refresh request after last successful
    metadata response. The calculation was since last successful metadata response
    so it's possible it did overlap with the periodic topic.metadata.refresh.interval.ms
    and cause a re-bootstrap even if not needed.
    Happening since 2.11.0 (#5177).
  • Issues: #4878.
    Fix to ensure rd_kafka_query_watermark_offsets enforces the specified timeout and does not continue beyond timeout expiry.
    Happening since 2.3.0 (#5201).

Telemetry fixes

  • Issues: #5179 .
    Fix issue in GetTelemetrySubscriptions with big-endian
    architectures where wrong values are read as
    accepted compression types causing the metrics to be sent uncompressed.
    Happening since 2.5.0. Since 2.10.1 unit tests are failing when run on
    big-endian architectures (#5183, Faidon Liambotis (@paravoid)).

Consumer fixes

  • Issues: #5199
    Fixed an issue where topic partition errors were not cleared after a successful
    commit. Previously, a partition could retain a stale error state even though the
    most recent commit succeeded, causing misleading error reporting. Now, successful
    commits correctly clear the error state for the affected partitions
    Happening since 2.4.0 (#4672).

Producer fixes

Checksums

Release asset checksums:

  • v2.12.0.zip SHA256 9b2f373e03f3d5d87c2075b3ce07ee9ea3802eea00cea41b99d8351a68d8a062
  • v2.12.0.tar.gz SHA256 1355d81091d13643aed140ba0fe62437c02d9434b44e90975aaefab84c2bf237

v2.11.1

Choose a tag to compare

@emasab emasab released this 18 Aug 20:51
69b1865

librdkafka v2.11.1 is a maintenance release:

  • Made the conditions for enabling the features future proof (#5130).
  • Avoid returning an all brokers down error on planned disconnections (#5126).
  • An "all brokers down" error isn't returned when we haven't tried to connect
    to all brokers since last successful connection (#5126).

Fixes

General fixes

  • Issues: #4948, #4956.
    Made the conditions for enabling the features future proof, allowing to
    remove RPC versions in a subsequent Apache Kafka version without disabling
    features. The existing checks were matching a single version instead of
    a range and were failing if the older version was removed.
    Happening since 1.x (#5130).

  • Issues: #5142.
    Avoid returning an all brokers down error on planned disconnections.
    This is done by avoiding to count planned disconnections, such as idle
    disconnections, broker host change and similar as events that can cause
    the client to reach the "all brokers down" state, returning an error and
    since 2.10.0 possibly starting a re-bootstrap sequence.
    Happening since 1.x (#5126).

  • Issues: #5142.
    An "all brokers down" error isn't returned when we haven't tried to connect
    to all brokers since last successful connection. It happened because the down
    state is cached and can be stale when a connection isn't needed to that
    particular broker. Solved by resetting the cached broker down state when any
    broker successfully connects, so that broker needs to be tried again.
    Happening since 1.x (#5126).

Checksums

Release asset checksums:

  • v2.11.1.zip SHA256 4a63e4422e5f5bbbb47f0ac1200e2ebd1f91b7b23f0de1bc625810c943fb870e
  • v2.11.1.tar.gz SHA256 a2c87186b081e2705bb7d5338d5a01bc88d43273619b372ccb7bb0d264d0ca9f