Twitter: Task Overview

[cyri113]

Overview

The Web3 Foundation has contracted TogetherCrew to deliver a Twitter Dashboard for monitoring community health.

High Level Solution

• Create an endpoint /auth/twitter
• The user is prompted to authenticate with Twitter
• Upon successful authentication, the user should be directed to the page /twitter/:handle
• Cases
  • No data - call the api endpoint /twitter/:handle/refresh
  • Data
    • Less than x hours - show data
    • More than x hours - show data + propose refresh (button)
• While refreshing data, we should show a progress bar
  • /twitter/:handle/refresh should create a saga, initiate the choreography and respond with a sagaId
  • The frontend should open a socket to /saga/:sagaId
  • The progress bar should be updated based on the status of the saga transactions
  • States:
    • TWITTER_BOT - NOT_STARTED
    • TWITTER_BOT - IN_PROGRESS
    • TWITTER_ANALYZER - NOT_STARTED
    • TWITTER_ANALYZER - IN_PROGRESS
• Refresh (/twitter/:handle/refresh)
  • Create a saga with TWITTER_REFRESH choreography with transactions:
    • { queue: QUEUE.TWITTER_BOT, event: Event.TWITTER_BOT.EXTRACT }
    • { queue: QUEUE.TWITTER_ANALYZER, event: Event.TWITTER_ANALYZER.RUN }
    • { queue: QUEUE.TWITTER_BOT, event: Event.TWITTER_BOT.SEND_MESSAGE }

Workflow

sequenceDiagram
    title Twitter Integration

    actor user
    participant frontend
    participant api
    participant rabbit-mq
    participant twitter-bot-server
    participant twitter-bot-redis
    participant twitter-bot-worker
    participant twitter-analyzer-server
    participant twitter-analyzer-redis
    participant twitter-analyzer-worker
    participant twitter

    user -->> frontend: Goes to /twitter
    frontend -->>+api: Authenticate using Twitter
    api -->>+twitter: Authenticate
    twitter -->>-api: Response
    api -->>-frontend: Authenticated

    frontend -->> frontend: Redirect to /twitter/:handle
    
    frontend -->>+api: GET /twitter/:handle/refresh
    api -->> rabbit-mq: Publish(Queue.TWITTER_BOT, Event.TWITTER_BOT.EXTRACT)
    api -->>- frontend: Response (SagaId)
    frontend --> api: Subscribe to socket /saga/:sagaId
    
    rabbit-mq -->> twitter-bot-server: Subscribe(Queue.TWITTER_BOT, Event.TWITTER_BOT.EXTRACT)
    twitter-bot-server -->> twitter-bot-redis: Enqueue Job
    twitter-bot-redis -->>+twitter-bot-worker: Dequeue Job
    twitter-bot-worker -->> twitter-bot-worker: Run extraction script
    twitter-bot-worker -->>-rabbit-mq: Publish(Queue.TWITTER_ANALYZER, Event.TWITTER_ANALYZER.RUN)

    rabbit-mq -->> twitter-analyzer-server: Subscribe(Queue.TWITTER_ANALYZER, Event.TWITTER_ANALYZER.RUN)
    twitter-analyzer-server -->> twitter-analyzer-redis: Enqueue Job
    twitter-analyzer-redis -->>+twitter-analyzer-worker: Dequeue Job
    twitter-analyzer-worker -->> twitter-analyzer-worker: Run analytics script
    twitter-analyzer-worker -->>-rabbit-mq: Publish(Queue.TWITTER_BOT, Event.TWITTER_BOT.SEND_MESSAGE)

    rabbit-mq -->> twitter-bot-server: Subscribe(Queue.TWITTER_BOT, Event.TWITTER_BOT.SEND_MESSAGE)
    twitter-bot-server -->> twitter-bot-redis: Enqueue Job
    twitter-bot-redis -->> twitter-bot-worker: Dequeue Job
    twitter-bot-worker -->>+ twitter: Send message
    twitter -->>- twitter-bot-worker: Response

    frontend -->>+ api: GET /twitter/:handle/followers
    api -->>- frontend: Response
    frontend -->>+ api: GET /twitter/:handle/posts
    api -->>- frontend: Response
    frontend -->>+ api: GET /twitter/:handle/metrics
    api -->>- frontend: Response

Loading

By Service

api

• add passport-twitter
• create account in neo4j (opportunity to migrate away from mongodb)
• redirect to /twitter/:handle upon successful authentication
• endpoints:
  • twitter/:handle/refresh
    • create saga
    • publish Event.TWITTER_BOT.EXTRACT on Queue.TWITTER_BOT
    • respond with saga id
  • twitter/:handle/metrics/followers (tbd)
  • twitter/:handle/metrics/posts (tbd)
• websocket:
  • /saga/:sagaId

broker libraries (py + ts)

• add TWITTER_REFRESH choreography
• add Queue.TWITTER_BOT queue
• add Queue.TWITTER_ANALYZER queue
• add Event.TWITTER_BOT.EXTRACT event
• add Event.TWITTER_BOT.SEND_MESSAGE event
• add Event.TWITTER_ANALYZER.RUN event

twitter-bot

based on python-service

server

• subscribe to Event.TWITTER_BOT.EXTRACT on Queue.TWITTER_BOT
• add to queue
• acknowledge

worker

• run extraction (what variable do we need to pass? i.e. timeframe)
• store data to neo4j database
• publish Event.TWITTER_ANALYZER.RUN on QUEUE.TWITTER_ANALYZER

twitter-analyzer

based on python-service

server

• subscribe to Event.TWITTER_ANALYZER.RUN on Queue.TWITTER_ANALYZER
• add to queue
• acknowledge

worker

• run analysis
• store data in neo4j
• publish Event.TWITTER_BOT.SEND_MESSAGE on Queue.TWITTER_BOT

frontend

• views
  • /auth/twitter endpoint (authentication)
  • /twitter/:handle endpoint (dashboard)
• websocket:
  • /saga/:sagaId

[amindadgar]

We've defined a base event for bots which is FETCH. Isn't it better to use the term FETCH instead of EXTRACT for TWITTER_BOT?