Table of Contents

...

1. Timeline:
   1. Starts at the program/service start time. Ends at the program/service end time (past) or current.
   2. Time range indicated by two sliders on each side. Time range can be selected by sliding these sliders.
   3. Updating slider position causes a refresh of the log viewer to show logs in the selected range with the selected filters
   4. If program/service is still running, the right/bottom end of the slider indicates current time, and if the slider is at this position, logs are updated live. The timeline keeps updating to reflect that.
   5. Sliders must not cross each other
   6. Label on the selected time range indicates the selected time range
   7. The timeline is marked with time range with granularity that depends on the duration of the log (which is the duration of the program run). 
   8. In the selected (or default time range), there should be symbols on the time line for the errors and warnings, as well as for events that match the filter in the search box. Clicking on such a symbol should navigate you to the corresponding event in the table. The graph may look like so:
      
2. Filters:
   1. Filter by lowest log level: 
      1. If ERROR is selected, then we show only ERROR
      2. If WARN is selected, then we show ERROR and WARN
      3. If INFO is selected, then we show ERROR, WARN and INFO
      4. If DEBUG is selected, then we show ERROR, WARN, INFO and DEBUG
      5. If TRACE is selected, then we show ERROR, WARN, INFO, DEBUG and TRACE
         
   2. Filter by search keywords:
      1. Search box that filters logs by the search text.
      2. This is a simple filter that applies on the message column
3. Log viewer Table:
   1. Columns:
      1. Timestamp
      2. Lowest Log Level
      3. Source - Only in CDAP - This column should not be shown in Hydrator
      4. Message (also contains stack trace).
   2. Default view shows single line messages, with / buttons to expand individual messages if they have more content
   3. Ability to suppress/show stack trace with a similar / buttons.
   4. Ability to expand all messages
   5. Ability to only view  the message column
4. Top Bar:
   1. Shows information/summary of the log
   2. Indicates program/service name
   3. Summary of total messages with number of warnings and errors
   4. Download button to download entire log
   5. Search box for filtering.

Required Backend support:

...

• As a Hydrator Product Team, I want to better plan the Hydrator real-estate so it is not statically allocated for configurations/views that are not commonly used/mandatory to be updated for creating pipelines
• e.g. Pipeline configurations like post run actions, engine, schedule
• As a Hydrator Product Team, I want to better design the Hydrator UI to lay more emphasis on the DAG
• As a Hydrator user, I do not want to switch back-and-forth between the DAG and the bottom panel repeatedly for building my pipeline
  • I should be able to provide node-level details right near the node
  • I should be able to simultaneously view details for multiple nodes both while editing a pipeline as well as viewing it.
• As a Hydrator user, I want to be able to build my pipeline incrementally. I want mandatory information to be more obvious.
• Build the pipeline with mandatory fields only to start off
• Incrementally add schedule, post run actions, etc
• As a Hydrator Product Team, I want remove to reduce the disparity between the pipeline detail view and the studio view. This will facilitate the move towards being able to edit a pipeline after publishing
  • e.g. Reference is unavailable in the pipeline details view
    
• As a Hydrator user, I want the messaging regarding multiple runs from the Hydrator UI to be clearer.
• Does Hydrator only always show the last run?
• If so, what is the "History" view for
• As a Hydrator Product Team, I want to reduce duplication
  • The console is not very useful today, it just shows messages. Can it be reconciled with the notification center?
• As a Hydrator user, I want related actions to appear together. 
  • e.g. "Export" is available in the bottom panel, but other pipeline controls are in the top bar.
• As a Hydrator Product team, I want to bring Jump buttons to Hydrator to make them the primary method of viewing entities in different contexts across CDAP, Hydrator and Tracker
  • Jump from pipeline details view in Hydrator to program details view in CDAP
  • Jump actions for source/sink in Hydrator:
    • View in Dataset Details page in CDAP
    • View in entity details page in Tracker
    • Explore Dataset (if possible) in CDAP

...

• As a Hydrator user, I want to be able to set complex schemas for my pipeline
  • I would like to have fields with enum, array, map, record and union types and would like an efficient method to create/manage them from the UI
• As a Hydrator user, I would like to be able to view complex schemas for my pipeline

Design: 

Design Principles: 

1. No static real-estate allocation. Users can add rows with a  button.
2. Breadcrumbs as a primary means of creating/viewing complex schemas.

Design

• Use a combination of dedicated screens with breadcrumbs for complex types
• Main screen should show the top level data type: Use tabs with customized expansions for complex fields.
• Simple types string, int, long, float, double, boolean, bytes , enum, array, map, record and unionSimple types string, int, long, float, double, boolean, bytes can be defined in the main screen itself, which is represented in breadcrumbs by the name of the stage for which schema is being definedcan be defined as today
• Enum: When an enum is selected, a sub-screen should accept the field name becomes clickable. Expansion allows you to enter enum values. Breadcrumb: <field-name> (enum)
• Array: When an array is selected, a sub-screen should accept a the field name becomes clickable. Expansion accepts a data type (which in turn could be a complex type as well, in which case the same flow rules defined here would apply, and breadcrumbs would be updated). Breadcrumb: <field-name> (array)
• Map: When a map is selected, a sub-screen should accept a key-the field name becomes clickable. Expansion accepts a key type and a value - type (which in turn could be complex types as well, in which case the same flow rules defined here would apply for each, and breadcrumbs would be updated). Breadcrumb: <field-name> (map). 
  • Simple types for map key and value can be configured on the same map subscreen.
  • Breadcrumb for complex map key: key (<selected complex type>)
  • Breadcrumb for complex map key: value (<selected complex type>)
• Record: When a record is selected, a sub-screen identical to the main screen shows up which accepts a nested record. Breadcrumb: <field-name> (record).
• Union: When a union is selected, a sub-screen should accept the two schemas whose union is needed. Breadcrumb: <field-name> (record)
• Simple types for both schema1 and schema2 can be selected on the same subscreen
• Complex types for schema1 takes you to a sub-screen identical to the record sub-screen. Breadcrumb: schema1 (<selected complex type>)
Complex types for schema2 takes you to a sub-screen identical to the record sub-screen. Breadcrumb: schema2 (<selected complex type>).
• Record: When a record is selected, the field name becomes clickable. Expansion allows you to specify a nested record
• Union: When a union is selected, the field name becomes clickable. On clicking, you can add schemas. Each schema is of type record. 
• For viewing, main screen only shows first level (string, int, long, float, double, boolean, bytes, enum, array, map, record and union) data types. Complex For complex types, field names are clickable, and take you expand to viewread-only views of the expansions described above with breadcrumbs for navigation.

Scratch Pad:

Work Streams:

...