The purpose of this project is to develop a core platform and a set of associated technologies to support observation, analysis, and visualisation of social machine activity. To fulfil this ambition, we have drawn together existing Web standards and recent advances in big data technologies (e.g. Storm). At the core of the Social Machine Observatory is the Web Observatory platform which orchestrates several components for data ingestion, integration, storage, and streaming.
The platform, which is part of a multi-site network of Social Machines observatories, allows for social machine researchers to capture and share data on the Web, and to reuse this data to build analytical applications and visualisations. Furthermore, the network of Web Observatories allows for multi-institute data querying and access. The project work has lead to several research contributions, including methods for data integration and re-streaming in real-time, and methods for data storage and retrieval in the context of collecting large scale social Web data.
We have also been working with several other institutes including RPI (USA) in order to establish the Web Observatory Schema.org Metadata schema, which is an extension of the widely used data metadata vocab.