Replace Kafka With TMS for metrics
- Chengfeng Mao
Checklist
- User Stories Documented
- User Stories Reviewed
- Design Reviewed
- APIs reviewed
- Release priorities assigned
- Test cases reviewed
- Blog post
IntroductionÂ
Use TMS to publish and fetch metrics.
Goals
Better replication and less code with TMS.
User StoriesÂ
Â
Design
Currently, metrics are published by KafkaMetricsCollectionService to and processed by KafkaMetricsProcessorService. In KafkaMetricsProcessorService, MetricsMessageCallback is called to process Kafka messages and store metrics in a metrics data HBase table. MetricsMessageCallback also updates the offset of the latest processed message in a metrics meta table. To replace Kafka with TMS, metrics need to be published and fetched in the same TMS topic, and processed metrics will be stored in the same metrics data HBase table. A new metrics meta table is also needed to store the messageId of the latest processed message. However, if the user upgrade CDAP from the version using Kafka for metrics to the version using TMS, unprocessed messages in Kafka still need to be processed.
Approach
Â
Security ImpactÂ
What's the impact on Authorization and how does the design take care of this aspect
Impact on Infrastructure OutagesÂ
System behavior (if applicable - document impact on downstream [ YARN, HBase etc ]Â component failures) and how does the design take care of these aspect
Test Scenarios
| Test ID | Test Description | Expected Results |
|---|---|---|
| Â | Â | Â |
| Â | Â | Â |
| Â | Â | Â |
| Â | Â | Â |
Releases
Release 4.1.0
Related Work
- Work #1
- Work #2
- Work #3
Â