Overview

...

• Source: "Sourcename Tpfs"

• Format: Avro Record with fields - ts, attributes

• Cube Name: “SourceNameCube”

...

• Example

  • Github: “GithubTPFS”

    • Format: Avro Record with fields - ts, repo, stars, forks, watchers, pulls

    • Cube Name: “GithubCube”

API

External Apis to be used

Api Calls

• Use a Workflow Custom Action to run periodic RESTful calls to  APIs
• A spark job can read the data from filesystem and update the cube
• In order to allow different scheduling of different calls, each call will have its own workflow

...

• UI should allow refining all metrics to different time granularities (Hourly, Daily, Weekly, Monthly, Every three months, Every year)

• Visualize metrics in the form of Dashboard (Widgets - Line, Bar, Pie, Scatter, ...)

• Dashboard and backend should support overlaying week-over-week, month-over-month or year-over-year for any metric

• Backend should  allow for raw querying of data through SQL commands

...

• Email and text notifications can be sent using SendGrid or Amazon SNS service

• Users can unsubscribe or subscribe using front end backed by apis

• Generate daily and weekly digest report and email them to stakeholders

• Export data into PDF/Excel available for download in UI

...