Debugging CDAP system services

The intent of this wiki page is to document debugging tools/endpoints for CDAP system services.

Thread stack trace endpoint

A Java thread dump is a snapshot of what every thread in the JVM is doing at a particular point in time. Each thread in the JVM is listed with it's name and id, it's current state and the Java call stack showing what monitor it has locked or is waiting on. This is especially useful if CDAP system services seems to hang when running under load, as an analysis of the dump will show where the threads are stuck, either by deadlock or other thread contention.

CDAP exposes below REST endpoint to get thread stack trace of CDAP system services. 

GET v3/system/services/{service-name}/stacks?depth=10
where,
service-name is name of CDAP system service 
depth is the maximum number of entries in the stack trace to be dumped. Integer.MAX_VALUE could be used to request the entire stack to be dumped.


Below are the REST endpoints to get thread stack trace for each CDAP system service.

System serviceREST Endpoint
App Fabricsystem/services/appfabric/stacks?depth=2147483647
Dataset Executorsystem/services/dataset.executor/stacks?depth=2147483647
Explore Servicesystem/services/explore.service/stacks?depth=2147483647
Log Saversystem/services/log.saver/stacks?depth=2147483647
Messaging Servicesystem/services/messaging.service/stacks?depth=2147483647
Metadata Servicesystem/services/metadata.service/stacks?depth=2147483647
Metricssystem/services/metrics/stacks?depth=2147483647
Metrics Processorsystem/services/metrics.processor/stacks?depth=2147483647