Table of Contents

...

1. Timeline:
   1. Starts at the program/service start time. Ends at the program/service end time (past) or current.
   2. Time range indicated by two sliders on each side. Time range can be selected by sliding these sliders.
   3. Updating slider position causes a refresh of the log viewer to show logs in the selected range with the selected filters
   4. If program/service is still running, the right/bottom end of the slider indicates current time, and if the slider is at this position, logs are updated live. The timeline keeps updating to reflect that.
   5. Sliders must not cross each other
   6. Label on the selected time range indicates the selected time range
   7. The timeline is marked with time range with granularity that depends on the duration of the log (which is the duration of the program run). 
   8. In the selected (or default time range), there should be symbols on the time line for the errors and warnings, as well as for events that match the filter in the search box. Clicking on such a symbol should navigate you to the corresponding event in the table. The graph may look like so:
      
2. Filters:
   1. Filter by lowest log level: 
      1. If ERROR is selected, then we show only ERROR
      2. If WARN is selected, then we show ERROR and WARN
      3. If INFO is selected, then we show ERROR, WARN and INFO
      4. If DEBUG is selected, then we show ERROR, WARN, INFO and DEBUG
      5. If TRACE is selected, then we show ERROR, WARN, INFO, DEBUG and TRACE
         
   2. Filter by search keywords:
      1. Search box that filters logs by the search text.
      2. This is a simple filter that applies on the message column
3. Log viewer Table:
   1. Columns:
      1. Timestamp
      2. Lowest Log Level
      3. Source - Only in CDAP - This column should not be shown in Hydrator
      4. Message (also contains stack trace).
   2. Default view shows single line messages, with / buttons to expand individual messages if they have more content
   3. Ability to suppress/show stack trace with a similar / buttons.
   4. Ability to expand all messages
   5. Ability to only view  the message column
4. Top Bar:
   1. Shows information/summary of the log
   2. Indicates program/service name
   3. Summary of total messages with number of warnings and errors
   4. Download button to download entire log
   5. Search box for filtering.

Required Backend support:

...

• Use tabs with customized expansions for complex fields.
• Simple types string, int, long, float, double, boolean, bytes can be defined as today
• Enum: When an enum is selected, the field name becomes clickable. Expansion allows you to enter enum values.
• Array: When an array is selected, the field name becomes clickable. Expansion accepts a data type (which in turn could be a complex type as well, in which case the same flow rules defined here would apply).
• Map: When a map is selected, the field name becomes clickable. Expansion accepts a key type and a value type.
• Record: When a record is selected, the field name becomes clickable. Expansion allows you to specify a nested record
• Union: When a union is selected, the field name becomes clickable. On clicking, you can add schemas. Each schema is of type record. 
• For viewing, main screen only shows first level (string, int, long, float, double, boolean, bytes, enum, array, map, record and union) data types. For complex types, field names are clickable, and expand to read-only views of the expansions described above.

Use Case 5: Pipelines Listing or Pipelines Dashboard

...

User Stories:

• As a Hydrator operations team member, I would like to view all the pipelines running across multiple namespaces that I am authorized for
  • Would also like to be view only the pipelines that I have access for.
• As a Hydrator operations team member, I would like to see following fields for each pipeline
  • Pipeline Fields
    • Namespace
    • Pipeline Name
    • Status (RUNNING | SUCESS | FAILED)
    • Number of Active Runs 
    • Total Number of Runs
    • Last Start Time
    • Last Run Finish Time
    • User 
• As a Hydrator operations team member, I should be able to filter on above specified fields to get to the pipeline status
• As a Hydrator operations team member, I should be able to filter all the pipelines (including pipelines across namespace) based on Start date, End date and  Status
• As a Hydrator operations team member, I should be able to filter on a specified field and continue to filter the results using other fields. (nested filtering)

Use Case 6: System Mode

User Stories:

• As a Hydrator user, I want to know if I'm working in distributed or stand-alone mode at all times.
  
• As a Hydrator user, I want to know if I'm working in secure or secure mode at all times.

Design: 

Use Case 7: Join Node

Use Case 8: Run Configuration

Scratch Pad:

Work Streams:

...