...
Solr can be run as either a separate Twill Runnable using logic like https://github.com/lucidworks/yarn-proto/blob/master/src/main/java/org/apache/solr/cloud/yarn/SolrMaster.java or can be housed inside the DatasetOpExecutorTwillRunnable as well. This decision depends on some prototyping . Solr will be started to use HDFS for persistence.
Standalone Mode
Solr supports a standalone mode, which starts up a separate Solr process. However, we will prefer to use EmbeddedSolrServer, in the same process as standalone CDAP.
...
CDAP will use EmbeddedSolrServer in in-memory mode.
Data Flow
Like in 3.5, there would be a call to update the index everytime the metadata of an entity is updated. Unlike in 3.5 though, this call would be an HTTP call to the Search Service (running Solr in 4.0).
Note: Since this call is now an HTTP call,
- should it be asynchronous?
- it will happen outside of the transaction to update the Metadata Dataset.
Index Sync
Since the persistence stores for metadata and the search index will be different, we will need a utility to keep them in sync. This could be a service/thread that runs periodically (preferred), or a tool that is invoked manually.
Upgrade
There should be a way to upgrade existing indexes to be stored in the new Search backend. The index sync tool should be developed in a way that it can be run via the Upgrade Tool to update existing metadata in the new search backend.
Indexes (TBD)
- What is the schema of data to be indexed in the new search backend?
REST API changes
The following changes would be made in the metadata search RESTful API:
...