Goals:

1. Improve operability in the Hydrator Studio (Improvements to logs, metrics, debuggability)
2. Improve usability in the Hydrator Studio (Redesign of bottom panel, etc)

...

1. Timeline:
   1. Starts at the program/service start time. Ends at the program/service end time (past) or current.
   2. Time range indicated by two sliders on each side. Time range can be selected by sliding these sliders.
   3. Updating slider position causes a refresh of the log viewer to show logs in the selected range with the selected filters
   4. If program/service is still running, the right/bottom end of the slider indicates current time, and if the slider is at this position, logs are updated live. The timeline keeps updating to reflect that.
   5. Sliders must not cross each other
   6. Label on the selected time range indicates the selected time range
   7. The timeline is marked with time range with granularity that depends on the duration of the log (which is the duration of the program run). 
   8. In the selected (or default time range), there should be symbols on the time line for the errors and warnings, as well as for events that match the filter in the search box. Clicking on such a symbol should navigate you to the corresponding event in the table. The graph may look like so:
      
2. Filters:
   1. Filter by lowest log level: 
      1. If ERROR is selected, then we show only ERROR
      2. If WARN is selected, then we show ERROR and WARN
      3. If INFO is selected, then we show ERROR, WARN and INFO
      4. If DEBUG is selected, then we show ERROR, WARN, INFO and DEBUG
      5. If TRACE is selected, then we show ERROR, WARN, INFO, DEBUG and TRACE
         
   2. Filter by search keywords:
      1. Search box that filters logs by the search text.
      2. This is a simple filter that applies on the message column
3. Log viewer Table:
   1. Columns:
      1. Timestamp
      2. Lowest Log Level
      3. Source - Only in CDAP - This column should not be shown in Hydrator
      4. Message (also contains stack trace).
   2. Default view shows single line messages, with / buttons to expand individual messages if they have more content
   3. Ability to suppress/show stack trace with a similar / buttons.
   4. Ability to expand all messages
   5. Ability to only view  the message column
4. Top Bar:
   1. Shows information/summary of the log
   2. Indicates program/service name
   3. Summary of total messages with number of warnings and errors
   4. Download button to download entire log
   5. Search box for filtering.

Required Backend support:

...

• Make DAG the hero
• Not have widgets that occupy real-estate but only show messages like "No XXX for this XXX"
• Not occupy real-estate statically with capabilities that are add-ons but not requirements for creating pipelines
• Support incremental pipeline development:
• Basics first: Configure nodes and get the pipeline working
• Add-ons later: Adding a schedule, adding post-run actions, etc
• Clearly demarcate Studio/Detail page into two sections:
• Canvas: Has the DAG, and capabilities to modify/view/update/reference node level information
• Pipeline section: Configure/update/view pipeline level information
• Canvas occupies majority real-estate by default. If you want to view/modify pipeline details, that reduces canvas size
• For pipeline section, there are two views for most capabilities:
• Compact view: Shows the selected pipeline capability (logs, metrics, pipeline configuration) in a smaller drawer, but also shows the canvas (and the DAG).
• Full-screen view: Hides the canvas, and only contains the header, footer and the selected capability.
• Switching between these two views is supported
• Reduce the disparity between the pipeline details view and the studio
  • This will help to add the capability to edit a pipeline after publishing

Use Case

...

3: Debuggability/Testing/Preview

User Stories:

•  As a Hydrator user, I want to preview my pipeline with a specified set of input records fetched directly from the source for validating my pipeline configuration and behavior before I publish the pipeline.
• As a Hydrator user, I want to be able to save snapshots of data previously fetched during preview, and use them later, so I don't have to connect to the source every time I want to preview my pipeline.
• As a Hydrator user, I want to test individual plugins in my pipeline by sending in sample input records and viewing the resultant output.
• As a Hydrator user, I want to be able to smoke test my pipeline from the UI
• Provided with a golden set of input and output, the pipeline should be able to validate itself
• As a Hydrator user, I want to be able to smoke test a single plugin from the UI
• Provided with a golden set of input and output, the plugin should be able to validate itself
• As a Hydrator user, I want to be able to store smoke test data for later use
• As a Hydrator user, I want to be able to validate my pipeline more effectively
  • If clicking on a 'Validate' button returns successful, then publishing the pipeline should not fail.

Design:

Use Case 4: Complex schema management

User Stories:

Design: 

Scratch Pad:

Work Streams:

 Tech Debt

• Simplify Config Store
• Simplify DAG component ~ Ajai's hack
  

Moving hard-coding/logic to backend

• Drafts
• Default plugin version
• For a stage, define whether it can accept an Input, Output or both
• Single APIs for status/logs/metrics for hydrator pipelines

New features

• Preview
• Log Viewer

...

   Possible solutions  for Log Viewer

• Tabular view: Columns for date, Class Name/Thread Name, Log Level, Log Line
• Alternate row background colors
• Vertically expandable with scrolling
• Searchable (Filter-able) columns
• Clear demarcation of rows
• Snippet with expand - especially for stack traces
• Picking only 1 or more log level -INFO, DEBUG, WARN, ERROR, ALL
• Ability to view and download raw logs if required
• Ability to view and expand only the "content" column of a log line

...