Introduction 

The current metadata search does not allow for users to search for entities by dates. The feature presented in this design doc aims to allow users to query entities by their creation date as well as accepted forms of user-defined date properties. A new search syntax will be introduced to facilitate this. The feature will be implemented through Elasticsearch.

Goals

Allow users to search for entities based on a date property. 

User Stories 

• As a pipeline developer, I want to be able to see entities that were created before or after a certain date.

• As a pipeline developer, I want to be able to see entities that I made between one date and another.

• As a pipeline developer, I want to be able to see entities that are set for deployment on a specific date.

• As a pipeline developer, I defined my own property with a date value, X, and now I want to get entities whose value X is after a specific date.  

Design

• To accommodate for this type of search, a new syntax will be introduced. Relevant methods in the QueryParser class will be modified to be able to detect this syntax, and the QueryTerm class will include a new field to indicate if the term is a date search.  

• Modify the current search method such that when a date query is supplied, entities created with a date-value condition are found. 

• In case there is a similarity between date-search syntax and other user-created metadata labels, the program will conduct a regular string search for the query in addition to the date search.

Implementation

• Currently the QueryBuilder object used for searches is a BoolQueryBuilder, which has been used to distinguish which terms are required and optional in the search results. ElasticSearch also has a RangeQueryBuilder which will be applied to search for creation times as a range over an hour, day, or multiple days. 

• A date query will be assumed to be a required query and so the RangeQueryBuilder object can be added into the main BoolQueryBuilder’s required searches. Since the RangeQueryBuilder will be embedded in the BoolQueryBuilder, there will still be only one search object for the entirety of the user's query. 

• The user’s search date / time will be converted from the predetermined UI syntax into a long to represent the millis since the epoch (unix time). This is the format that is currently used to store creation time of entities and is supported by Elasticsearch. 

• The metadata Property class will now include an additional field called date which will be a Long. If a user defines a metadata property with a date-format value, the value will be parsed into unix time and stored as date. This will allow for a date search over user defined dates. 

Approach

Approach #1

• User writes their query with the new syntax
• Once we detect that it is a date query we indicate it in QueryTerm class with an enum: SearchType
• But we keep the term as it is originally
• Assumption: all date queries are required, whether indicated or not
• Modify createMainQuery() in ElasticsearchMetadata so that it creates a date subquery when necessary
• When iterating over the QueryTerms, if it is a DATE SearchType, call dateQuery = createDateQuery  and add the result to create a new range query builder. 
• Continue to call  termQuery = createTermQuery  as earlier, for possible conflict reasons as mentioned in the design section.
• Place both objects (dateQuery and termQuery) into the boolQuery search, checking, as in the current implementation, to see if the query is required or optional.  
• createDateQuery()
• This method will create a range query from the supplied QueryTerm. The QueryTerm’s term (string) will be parsed to extract the date range that is meant to be searched for. The extracted range will be in the form of a long representing unix time.  
• The method will consider whether a user-defined field is provided for the search. If so, only that field will be searched for given date values. Otherwise search over creation times and all user-defined dates - creation time results will be presented first. 

API changes

QueryTerm

Introduce the following fields:

public enum SearchType { STANDARD, DATE }

private SearchType searchType;

private Long date;

Accommodate a new constructor that takes a SearchType as a parameter. Existing constructor will default the searchType to be SearchType.STANDARD

QueryParser.parseQueryTerm()

Create a check for the new keywords (using an if-statement, similar to checking for required terms). If the term contains the new keywords, return a QueryTerm indicating SearchType.DATE in its construction, otherwise SearchType should be STANDARD. 

UI Impact or Changes

• Users create their date query starting with "DATE:" 

• The colon can be followed by a date-field name to search over, followed by another colon

• The supported date format (for both defining and querying dates) is:
  • YYYY-MM-DD

• The date can immediately follow "DATE:" or it can follow the field name and colon

• To indicate conditions such as before or after a date, use the following comparison operators immediately before the date:
  • > (after)
  • >= (on or after)
  • <= (on or before)
  • < (before)
  • == (on - if no condition is specified this is the assumed condition)

• Examples searches and their results: 

  • “DATE:2019-07-01” or "DATE:==2019-07-01"

    • Returns entities created on July 1st, 2019 or with equivalent date property values. 

  • “DATE:<2019-07-01”

    • Returns entities created before July 1st, 2019 or with equivalent date property values. 

  • “DATE:<=2019-07-01”

    • Returns entities created on or before July 1st, 2019 or with equivalent date property values. 

  • “DATE:>2019-07-01”

    • Returns entities created after July 1st, 2019 or with equivalent date property values. 

  • “DATE:>=2019-07-01”

    • Returns entities created on or after July 1st, 2019 or with equivalent date property values. 

  • "DATE:my_date:2019-07-01"
    • Returns entities that have a property called "my_date" the value of which is equivalent to the queried July 1st, 2019 date. 
    • All other comparison operators work similarly with this type of field search.

Related Work

• Work done on the Required Search Fields feature was the first work that allowed for new syntax to be easily defined for the UI. 

Future Work

• Currently only YYYY-MM-DD format is supported for date search and definition. Support for other date syntaxes should be implemented in the future.