Social Media Prediction (SMP)

Overview

People are interested in predicting the future. For example, which films will bomb or who will win the upcoming Grammy awards? Making predictions about the future in many aspects is not only fun matters but can bring real value to those who correctly predict the course of world events, such as which stocks are the best purchases for short-term gains. Predictive analytics is thus a field that has attracted major attention in both academia and the industry.

As social media has become an inseparable part of modern life, there has been increasing interest in research of leveraging and exploiting social media as an information source for inferring rich social facts and knowledge. In the literature, a large number of social media datasets have been established for various research tasks and helped lead to great advancements in social media technology and applications. However, most of the existing dataset are limited in content coverage, i.e. the collected data are often biased to the particular problem in question, and lacking of cross-task generalization.

Therefore, as a joint activity with the research teams from the Chinese Academy of Sciences, Academia Sinica, and Microsoft Research Asia, we are releasing a large-scale social media dataset for sociological understanding and predictions, namely Social Media Prediction (SMP) dataset, with over 850K posts and 80K users in total. Our goal is to make the SMP dataset as varied and rich as possible to thoroughly represent the social media “world”. Particularly, we aim to record the dynamic variance of social media data. For example, the social media posts in the dataset are obtained with temporal information to preserve the continuity of post sequences.

Task Description

This year we will focus on two particular tasks, Popularity Prediction and Tomorrow’s Top Prediction. Meanwhile, we are open for innovative self-proposed topics.

  • Task1: Popularity Prediction 
    The task is designed to predict the impact of sharing different posts for a publisher on social media. Given a photo (a.k.a. post) from a publisher, the goal is to automatically predict the popularity of the photo, e.g., view count for Flickr, Pin count for Pinterest, etc.
  • Task2: Tomorrow’s Top Prediction
    The task is designed to discover top-k popular posts on social media. Suppose to have a set of candidate photos and the history data of past photo sharing, the goal is to automatically predict which will be the most popular photos in the next day.
    The contestants are asked to develop their prediction system based on the SMP dataset provided by the Challenge (as training data), plus possibly additional public/private data, to address one or both of the two given tasks. For the evaluation purpose, a contesting system is asked to produce prediction results of popularity. The accuracy will be evaluated by pre-defined quantitative evaluation. The contestants need to introduce their systems and datasets in the conference.
  • Additional Task: Open Topics
    To encourage the exploration of the SMP application scope, we also accept innovative topics proposed by the participants themselves, e.g., behavior prediction, interest mining, etc. For open topics, the participants need to clearly define the topic, demonstrate the technical advancement of their proposed solutions, specify the evaluation protocols, and provide SMP based experimental results.
Dataset

The SMP dataset contains two subsets collected from Flickr (a photo sharing platform), SMP-T1 and SMP-T2, for the two particular tasks, respectively. For each task, we split the data with time-order, resulting in 90% for training and 10% for testing. The tables below show the statistics of SMP-T1 subset and SMP-T2 subset.

Dataset #Post #User Temporal
Range
(Years)
Avg. Title
Length
Avg. Tag
Count
Avg.
Description
Length
Avg. Views
SMP-T1 400K 135 6 20 9 114 131

 

Dataset #Post #User #Categories Temporal
Range
(Months)
Avg. Title
Length
#Tags #POIs Avg. Views
SMP-T2 450K 80K 11 16 26 669 103K 306

*In the SMP dataset, we provide the category information for each photo.

Submission Format

Each team is allowed to submit the results of at most three runs and selects one run as the primary run of the submission (we do not guarantee to evaluate additional runs), which will be measured for performance comparison across teams.

Each submission is required to be formatted in a JSON File as follows.

Task1 (ascending by post_id):
{

"version": "VERSION 1.2", "result":[
{
"post_id": "post6374637",
"popularity_score": 2.1345
},
...
{
"post_id": "post3637373",
"popularity_score": 3.1415
}
],
"external_data":{
"used": "true", # Visual Features. True indicates used of external data.
"details": "VGG-19 pre-trained on ImageNet training set" # Details of your external data.
}

}

Task2 (ascending by post_id):
{

"version": "VERSION 1.2", "result":[
{
"post_id": "post6374637",
"ranking_position": 1,
"popularity_score": 2.1345
},
...
{
"post_id": "post3637373",
"ranking_position": 5,
"popularity_score": 3.1415
}
],
"external_data":{
"used": "true", # Visual Features. True indicates used of external data.
"details": "VGG-19 pre-trained on ImageNet training set" # Details of your external data.
}

}

Note: comments in blue are illustrative and help us to provide inline detailed explanations. Please avoid them in your submissions. Participants please strictly follow the submission format.

Evaluation Metric

The evaluation provided here can be used to obtain performances on the testing set of SMP. It contains multiple common metrics, including Spearman Ranking Correlation, MAE, MSE, ACC@5, and NDCG@10.

By quantitative evaluation, we measure the systems submitted to this challenge on a testing set. Our evaluation protocol is applied on the following criteria:

  • Prediction Correlation: whether the predicted popularity more correlates with the actual value of the popularity?
  • Prediction Error: to judge the error of the score prediction
  • Ranking Relevance: to measure the ranking relevance between top-k items and predicted ranking
Participation

The Challenge is a team-based contest. Each team can have one or more members, and an individual can not be a member of multiple teams.

At the end of the Challenge, all teams will be ranked based on both objective evaluation and human evaluation described above. The top three performing teams will receive award certificates and/or cash prizes. At the same time, all accepted submissions are qualified for the conference’s grand challenge award competition.

Timeline
  • April 15, 2017: Dataset available for download (training set)
  • June 1, 2017: Test set available for download
  • June 10, 2017: Results submission (for Task1 and Task2)
  • June 11 - June 25, 2017: Objective evaluation and human evaluation (for Task1 and Task2)
  • June 30, 2017: Evaluation results announce (for Task1 and Task2)
  • July 14, 2017: Paper (all tasks) submission deadline (please follow the instructions on the main conference website)
Paper Submission

Please follow the guideline of ACM Multimedia 2017 Grand Challenge for the paper submission.

Contact

Wen-Huang Cheng (), Academia Sinica
Bo Wu (), Chinese Academy of Sciences
Yongdong Zhang (), Chinese Academy of Sciences
Tao Mei (), Microsoft Research Asia