HTTP Query - SODA SoQL

Owned by Todd Greenstein
Aug 29, 2016

Introduction 

Open Datasets from Federal, State, and Municipal Government agencies have largely standardized on Socrata's platform to expose data to developers.  Socrata has an HTTP REST API known as SODA for querying and filtering data from a given dataset.   SODA includes simple filter and a complex query language available over http known as SoQL (not to be confused with salesforce object query language with the same abbreviation).  Extending the capabilities of Hydrator Pipelines to include freely available government Datasets will open new use cases.  

 

Use-case
A User in their Hydrator Pipeline can query government datasets to analyze trends across a broad spectrum of datasets.  

User Stories

  • As a Hydrator User I want to use the SODA HTTP REST API to query specific open datasets and use simple filters to retrieve information so that I can include it in my Hydrator Pipelines
  • As a Hydrator User I want to use the SODA HTTP REST API to issue a SQL-like query to query a specific open dataset to retrieve information so that I can include it in my Hydrator Pipelines
  • As a Hydrator User I want to use my SODA application token when connecting to an open dataset so that my throughput can be improved.
  • As a Hydrator User I want the ability to disable pagination in queries I issue to the SODA HTTP REST API so that I can retrieve data without pagination restrictions.  

Example

Example for how the plugin should work

Implementation Tips

Design 

Design: