Apache Samza

Tuesday November 27, 2018

Announcing the release of Samza 1.0

We’re thrilled to announce to the release of Apache Samza 1.0.

Today Samza forms the backbone of hundreds of real-time production applications across a multitude of companies, such as LinkedIn, VMWare, Slack, Redfin among many others. This release of Samza adds a variety of features and capabilities to Samza’s existing arsenal, coupled with new and improved documentation, code snippets, examples, and a brand-new website design! Here are a few selected highlights:

  • Stable high level APIs that allow creating complex processing pipelines with ease.

  • Beam Samza Runner now marries Beam’s best in class support for EventTime based windowed processing and sophisticated triggering with Samza’s stable and scalable stateful processing model.

  • Table API that provides a common abstraction for accessing remote or local databases. Developers are now able to “join” an input event stream with such a Table.

  • Integration Test Framework to enable effortless testing of Samza jobs without deploying a Kafka, Yarn, or Zookeeper cluster.

  • Support for Apache Log4j2 allowing improved logging performance, customization, and efficiency.

  • Upgraded Kafka client and consumer.

  • An interactive shell for Samza SQL for seamless formulation, development, and testing of SamzaSQL queries.

  • Side-input support that allows using log-compacted data sources to populate KV state for Samza applications.

  • An improved website with detailed documentation and lots of code samples!

In addition, Samza 1.0 brings numerous bug-fixes, upgrades, and improvements listed below.

New features

Samza 1.0 brings full-feature support for the following:

Improved Stable High Level APIs

Samza 1.0 brings Descriptor APIs that allows applications to specify their input and output systems and streams in code. Samza’s new Context APIs provide applications unified access to job-level, container-level, task-level, and application-level context and capabilities. This also simplifies Samza’s ApplicationRunner interface.

This API evolution requires a few simple modifications to application code, which we describe in detail in our upgrade steps

Beam Runner Support

Samza’s Beam Runner enables executing Beam pipelines over Samza. This enables Samza applications to create complex processing pipelines that require event-time based processing, varying types of event-time based windowing, and more. This feature is supported in both the YARN and standalone deployment models.

Joining Streams and Tables

Samza’s Table API provides developers with unified access to local and remote data sources such as Key-Value stores or web services, while providing features such as rate-limiting, throttling, and caching capabilities. This provides first-class API primitives for building Stream-Table join jobs. Learn more about the use, semantics, and examples for Table API here.

Test Samza without ZK, Yarn or Kafka

Samza 1.0 brings a test framework that allows testing Samza applications using in-memory input and output. Users can now setup test and testing pipelines for their applications without needing to setup any other services, such as Kafka, YARN, or Zookeeper.

Log4J2 support

Samza now supports Apache Log4j 2 for system and application logging. Log4j 2 is an upgrade to Log4j that provides significant improvements over its predecessor, Log4j 1.x, such as better throughput and latency, custom log levels, and a pluggable logging architecture.

Kafka upgrade

This release upgrades Samza to use Kafka’s high-level consumer (Kafka v0.11.1.62). This brings latency and throughput benefits for Samza applications that consume from Kafka, in addition to bug-fixes. This also means Samza applications can now better their utilization of the underlying Kafka cluster.

SamzaSQL Shell

SamzaSQL now provides a shell for users to type-in their SQL queries, while Samza does the heavy-lifting of wiring the inputs and outputs, and sizing the application in the background. This is great for testing and experimenting with queries while formulating your application-logic, specially suited for data-scientists and tinkerers.

Side-inputs

Samza 1.0 brings the ability to leverage existing log-compacted data sources (e.g., Kafka topics) to populate KV state for Samza applications. If your data processing pipeline involves Hadoop-to-Kafka push, this feature alleviates the need for your Samza job to create separate Kafka-topics to back KV state.

Improved website, documentation, and samples

We’ve re-designed the Samza website making it easier to find details on key Samza concepts and patterns. All documentation has been revised and rewritten, keeping in mind the feedback we got from our customers. We’ve revised and added sample application code to showcase Samza 1.0 and the use of its new APIs.

Enhancements and Upgrades

This release brings the following enhancements, upgrades, and capabilities:

API enhancements and simplifications

SAMZA-1789: unify ApplicationDescriptor and ApplicationRunner for high- and low-level APIs in YARN and standalone environment

SAMZA-1804: System and stream descriptors

SAMZA-1858: Public APIs for shared context

SAMZA-1763: Add async methods to Table API

SAMZA-1786: Introduce the metadata store abstraction

SAMZA-1859: Zookeeper implementation of MetadataStore

SAMZA-1788: Add the LocationIdProvider abstraction

Upgrades and Bug-fixes

SAMZA-1768: Handle corrupted OFFSET file

SAMZA-1817: Long classpath support for non-split deployments SAMZA-1719: Add caching support to table-API

SAMZA-1783: Add Log4j2 functionality in Samza

SAMZA-1868: Refactor KafkaSystemAdmin from using SimpleConsumer

SAMZA-1776: Refactor KafkaSystemConsumer to remove the usage of deprecated SimpleConsumer client

SAMZA-1730: Adding state validation in StreamProcessor before any lifecycle operation and group coordination

SAMZA-1695: Clear events in ScheduleAfterDebounceTime on session expiration

SAMZA-1647: Fix race conditions in StreamProcessor

SAMZA-1371: Some Samza Containers get stuck at \“Starting BrokerProxy\”

SAMZA-1648: Integration Test Framework & Collection Stream Impl

SAMZA-1748: Failure tests in the standalone deployment

A source download of Samza 1.0 is available here, and in Apache’s Maven repository.

Community Developments

A symposium on Stream processing with Apache Samza and Apache Kafka was held on July 19th and on October 23rd. Both were attended by more than 350 participants from across the industry. It featured in-depth talks on Samza’s Beam integration, its use at LinkedIn for real-time notifications, a talk on Kafka-replication at Uber, and Kafka cruise control, and many others.

Samza was also the focus of a talk at Strange Loop'18, focussing in depth on its scalability, performance, extensibility, and programmability.

Comments:

Thank you very much for this posting.

Posted by Samsun Rock on January 31, 2019 at 06:17 AM GMT #

Great to see this hijab collection. me and my brother are always loved to play this cool game, this is also a math games for kids in educational use. that the fantastic way to entertain our self.

Posted by math game on January 31, 2019 at 06:19 AM GMT #

This is a really useful version, thank you for sharing

Posted by drawing games on March 20, 2019 at 01:24 AM GMT #

https://www.lechateauguillestre.com

Posted by youssou on May 02, 2019 at 02:19 AM GMT #

Apache guidelines for students exist in substantial part since wireless transmissions are limited and open. You can't utilize wavelengths that another person is asked to apply for forbidden files to utilize them. Web foundation and servers are commonly not open. Privately owned businesses paid for, assembled, and claim the greater part of that framework. Indeed, even at that, you can't go to a TV station or paper and request they give you broadcast appointment. You'd just look like a silly guy and be left the structure by the police.

Posted by essay agency on May 06, 2019 at 08:37 AM GMT #

I have a question , it is the cost of your production

Posted by run 3 on May 13, 2019 at 02:01 AM GMT #

Thanks for useful release of Samza 1.0! It is really nice

Posted by Homepage on June 18, 2019 at 08:41 AM GMT #

Thank you for this update. I always like to read posts that share useful information.

Posted by swords and souls 2 on June 26, 2019 at 03:05 AM GMT #

Thank you for this update. I always like to read posts that share useful information.

Posted by film izle on June 29, 2019 at 08:41 PM GMT #

Download AOS TV App for Android and Firestick devices from the official aos tv website to enjoy movies and tv shows.

Posted by aos tv download on July 06, 2019 at 02:09 PM GMT #

Nice!!! Truly stunning.

Posted by nice on July 19, 2019 at 10:35 PM GMT #

I think I saw this in Enschede, is that possible? Excellent presentation!

Posted by ewafharfa1976 on July 19, 2019 at 11:40 PM GMT #

Gods work Some great features in this!

Posted by onlegtifor1986 on July 20, 2019 at 03:37 PM GMT #

These are great! Thanks for sharing your sketches too! Nice work!!!!

Posted by istotoca1979 on July 20, 2019 at 05:39 PM GMT #

amazing.!! very good graphics, gives modern feel to the design!

Posted by ziecumarin1978 on July 20, 2019 at 06:10 PM GMT #

So freaking cute!!! Yay, this is so cool!!

Posted by glibbadespdank1986 on July 20, 2019 at 10:46 PM GMT #

Great project ! Amazaing work, congratz

Posted by reicommalin1978 on July 20, 2019 at 11:53 PM GMT #

Not a fan of this green, but you made it work here. Thank you Noha.@Noha Morgan

Posted by calsunbsusa1986 on July 21, 2019 at 02:06 AM GMT #

Muito bom! OMG, amazing.

Posted by viepadora1983 on July 21, 2019 at 03:44 AM GMT #

Woahh! Perfect! it's brilliant ;)

Posted by gasnatabmo1985 on July 21, 2019 at 05:04 AM GMT #

Very nice work + + + Very nice branding and cool project! My congratulations, @Jefferson Paganel!

Posted by hardvosucol1983 on July 21, 2019 at 05:39 AM GMT #

very beautiful work Fantastic! I am also based in Ams, might come check this out if you guys have a pop up store :)

Posted by easarimtu1973 on July 21, 2019 at 06:18 AM GMT #

Great!!! love the presentation, and photos, cool

Posted by bridosytes1982 on July 21, 2019 at 06:50 AM GMT #

amazing style! Well done!

Posted by taicakerac1972 on July 21, 2019 at 07:55 AM GMT #

I love how the identity can work with every color, cold or warm. Good job! Hermosas ilustraciones ! Felicitaciones, saludos desde Argentina =)

Posted by sitachorli1975 on July 21, 2019 at 08:27 AM GMT #

WOW! Great! I really like it) Oh yeah, still love pastell colors ❤️

Posted by wanttanlaco1981 on July 21, 2019 at 08:59 AM GMT #

One Punch Man yay!

Posted by thoughmitledesf1976 on July 21, 2019 at 09:32 AM GMT #

looking nice.... great work (y) Keep it Up

Posted by detywhechild1982 on July 21, 2019 at 10:05 AM GMT #

love the So cool !!

Posted by scotigodsteel1981 on July 21, 2019 at 10:38 AM GMT #

Congratulations on the opportunity and achievement! IT looks amazing :) Make some tutorials bro... really nice work!

Posted by nerliareli1975 on July 21, 2019 at 11:11 AM GMT #

Post a Comment:
  • HTML Syntax: NOT allowed

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation