Introduction 

The schema associated with HDFS Files may vary based on set of wrangler directives. Current implementation of datasets do not apply wrangler directives while exploring datasets or reading records using hydrator pipeline. 

Goals

Better User experience: The goal is to apply wrangler directives while reading data through explore and hydrator pipelines from datasets. 

User Stories 

• The HDFS files from existing Dataset can be sampled and wrangled through wrangler. The final output schema and wrangler directives should be applied while exploring the dataset or reading records from the dataset using hydrator pipeline. If there are any further changes to wrangler directives or output schema, it should be reflected through explore queries and hydrator pipeline. 
• The records from Table Dataset can be sampled and wrangled through wrangler. The final output schema and wrangler directives should be applied while exploring the dataset or reading records from the dataset using hydrator pipeline. If there are any further changes to wrangler directives or output schema, it should be reflected through explore queries and hydrator pipeline. 

Design

Cover details on assumptions made, design alternatives considered, high level design

Approach

Approach #1

Approach #2

API changes

New Programmatic APIs

New Java APIs introduced (both user facing and internal)

Deprecated Programmatic APIs

New REST APIs

Deprecated REST API

CLI Impact or Changes

• Impact #1
• Impact #2
• Impact #3

UI Impact or Changes

• Impact #1
• Impact #2
• Impact #3

Security Impact 

What's the impact on Authorization and how does the design take care of this aspect

Impact on Infrastructure Outages 

System behavior (if applicable - document impact on downstream [ YARN, HBase etc ] component failures) and how does the design take care of these aspect

Test Scenarios

Releases

Release X.Y.Z

Release X.Y.Z

Related Work

• Work #1
• Work #2
• Work #3

Future work