Content Synchronisation

The content synchronization process is normally used to index the Starmind content for corporate search engines. You can also consider using OpenSearch, to add Starmind to your search results (you can find more details here: Open Search). There are other possible scenarios for a content synchronisation, but one should avoid using this technique for immediate user interaction. The reason being that our algorithms will not capture this kind of user interaction and we cannot learn this way.

Content synchronisation process

Initial steps

  1. Get a valid token with the credentials that we provide to you with.
  2. Get all question ids from the GET /questions/updates endpoint.
  3. Loop through all question ids via our GET /questions/:id/complete endpoint. Make sure that you don't index deleted content ("is_deleted": true) and don't forget to also grep the attachments.
  4. Feed your system with the content.

Periodically:

This process can be triggered e.g. every 4 hours.

  1. Get a new valid token if the old one is expired.
  2. Get all question ids from the GET /questions/updates endpoint with a content update since the last crawl.
  3. Loop through all new question ids via our GET /questions/:id/complete endpoint.
  4. Make sure that you don't index deleted content ("is_deleted": true) and delete content from your index if it got deleted on Starmind. Also don't forget to grep new or modified attachments.
  5. Feed your system with new/updated/deleted content.