Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Checklist

  • User Stories Documented
  • User Stories Reviewed
  • Design Reviewed
  • APIs reviewed
  • Release priorities assigned
  • Test cases reviewed
  • Blog post

Introduction 

The current metadata search does not allow for users to search for entities by dates. The feature presented in this design doc aims to allow users to query entities by their creation date as well as accepted forms of user-defined date properties. A new search syntax will be introduced to facilitate this. The feature will be implemented through Elasticsearch.

Goals

Allow users to search for entities based on a date property. 

User Stories 

  • As a pipeline developer, I want to be able to see entities that were created before or after a certain date.

  • As a pipeline developer, I want to be able to see entities that I made between one date and another.

  • As a pipeline developer, I want to be able to see entities that are set for deployment on a specific date.

  • As a pipeline developer, I defined my own property with a date value, X, and now I want to get entities whose value X is after a specific date.  

Design

  • To accommodate for this type of search, a new syntax will be introduced. Relevant methods in the QueryParser class will be modified to be able to detect this syntax, and the QueryTerm class will include a new field to indicate if the term is a date search.  

  • Modify the current search method such that when a date query is supplied, entities created within a given range of time are found.

  • In case there is a similarity between date-search syntax and other user-created metadata labels, the program will conduct a regular string search for the query in addition to the date search.

Implementation

  • Currently the QueryBuilder object used for searches is a BoolQueryBuilder, which has been used to distinguish which terms are required and optional in the search results. ElasticSearch also has a RangeQueryBuilder which will be applied to search for creation times as a range over an hour, day, or multiple days. 

  • A date query will be assumed to be an optional field, unless otherwise indicated by the user’s syntax, and so the RangeQueryBuilder object can be added into the main BoolQueryBuilder’s optional searches. Since the RangeQueryBuilder will be embedded in the BoolQueryBuilder, there will still be only one search object. 

  • The user’s search date / time will be converted from the predetermined UI syntax into a long to represent the millis since the epoch (unix time). This is the format that is currently used to store creation time of entities and is supported by Elasticsearch. 

Approach

Approach #1

  • User writes their query with the new syntax
  • Once we detect that it is a date query we indicate it in QueryTerm class
    • But we keep the term as it is originally
    • Edge case: term is indicated as both required and date by user
    • Assumption: all date queries are required, whether indicated or not
  • Modify createMainQuery() in ElasticsearchMetadata so that it creates a date subquery when necessary
    • When iterating over the QueryTerms, if it is a DATE SearchType, call dateQuery = createDateQuery  and add the result to create a new range query builder. 
    • Continue to call  termQuery = createTermQuery  as earlier, for possible conflict reasons as mentioned in the design section.
    • Place both objects (dateQuery and termQuery) into the boolQuery search, checking, as in the current implementation, to see if the query is required or optional.  
  • createDateQuery()
    • This method will create a range query from the supplied QueryTerm. The QueryTerm’s term (string) will be parsed to extract the date range that is meant to be searched for. The extracted range will be in the form of a long representing unix time.  

API changes

QueryTerm

Introduce the following fields:

public enum SearchType { STANDARD, DATE }

private SearchType searchType;

Accommodate a new constructor that takes a SearchType as a parameter. Existing constructor will default the searchType to be SearchType.STANDARD

QueryParser.parseQueryTerm()

Create a check for the new keywords (using an if-statement, similar to checking for required terms). If the term contains the new keywords, return a QueryTerm indicating SearchType.DATE in its construction, otherwise SearchType should be STANDARD. 

UI Impact or Changes

  • Users create their date query starting with the word DATE: 

  • Keywords are followed by a colon : and either a date or a time measurement 

    • before, after, and on take a date

    • between takes two dates which are also separated by : 

    • newer_than and older_than take a time measurement 

  • Date representation

    • MM/DD

    • MM/DD/YY

  • Time measurement representation

    • #m for minutes

    • #h for hours

    • #d for days

  • Examples searches and their results: 

    • “DATE:before:04/15”

      • Returns entities created before April 15th of this year

    • “DATE:after:04/15/18”

      • Returns entities created after April 15th, 2018

    • “DATE:newer_than:3d”

      • Returns entities that are less than 3 days old


  • Choices to be made:
    • Should the user input the date MM/DD or DD/MM format?

Related Work

  • Work done on the Required Search Fields feature was the first work that allowed for new syntax to be easily defined for the UI. 
  • No labels