Martech Challenges: The Results

Between May 29 and July 29, 2019, SaaS Advisor organized the first Martech Challenges focused on Social Listening. We would like to first congratulate the seven participating vendors, all pure-players and among the leaders in the market, who agreed to participate and be compared feature by feature. This exercise was not easy, and is quite rare in this sector and within the Martech industry in general.

Here, we present the methodology, key results, and insights from this challenge.

Objectives and Methodology of the Martech Challenges

The primary goal of this challenge was to provide clarity in the market. SaaS Advisor, as a trusted third party, aimed to:

• Provide a European Perspective: Offer a market view from a "European" lens, different from existing analyses (e.g., G2Crowd, Forrester).
• Utilize Realistic Briefs: Employ briefs closely aligned with real-world conditions and conduct a “feature versus feature” comparison.
• Offer an Objective View: Present an objective assessment of the strengths and weaknesses of each solution.

The 7 Participants:

To evaluate and rate these vendors, we assembled a jury of 8 professionals from various backgrounds (advertisers, agencies, research institutes, etc.) to cover all usage scenarios.

The Jury:

We created three briefs to cover different use cases:

1. Local Brand Brief: Focused on WWF (World Wildlife Fund) in the US, to assess the vendors' ability to listen in a defined market.
2. Global Brand Brief: Included two sports clubs (Liverpool FC and the Golden State Warriors) to measure the vendors' ability to collect global information in various languages.
3. Advanced Queries Methodologies Brief: Centered on daily mobility (trips of less than one hour) to evaluate the vendors' ability to provide additional services for brands.

 

Martech Challenges: Evaluation Criteria

We determined the evaluation criteria with the jury, covering the entire Social Listening value chain. The criteria were categorized as follows:

Our goal was to identify one winner per criterion, and subsequently, a grand winner per category based on the number of criteria won. This led us naturally to the rating process.

• Each jury member received a “Rating Card” for the various criteria mentioned above.
• No criterion was weighted (each was considered equally important for our study).
• Ratings were based on both quantitative and qualitative data.
• Data collection and measurement were conducted from May 29 to July 29, 2019. Any features released after this date were not considered.
• Business models (including pricing) were also not factored into the evaluation.

Social Data Results

The volume and relevance of data were critical criteria for this challenge. We anticipated that results, particularly for Twitter, would be quite close among vendors. This was not the case. Furthermore, there was no consistent performance among leaders and challengers across different briefs. Each vendor had strengths and weaknesses depending on social networks, language, geographic region, and client industries. Consequently, we decided not to award a grand winner for this category.

In our comprehensive study, we detailed:

• Data collection policies by network (what is possible to collect for each tool).
• Official partnerships between vendors and social networks.
• Basic and on-demand data (especially for China).
• The best tools by network and language.

We detailed the data collection policies and official partnerships for Twitter, Facebook, Instagram, YouTube, VKontakte, Weibo, the web in general (blogs, forums, media), and reviews. We also illustrated the strengths and weaknesses of each vendor in terms of data (both quantitative and qualitative) for each source: global web, Twitter, Instagram, Facebook, and for three languages/geographies: China, Russia, and Arabic.

Disclaimer:

SaaS software evolves rapidly, and Social Listening tools are no exception. Most vendors release new features and major updates monthly. Thus, there is no perfect timing for such a challenge. For instance, since September 2019, Synthesio and Brandwatch have updated their products extensively. These updates could have significantly impacted our results. Our study, concluding in July 2019, should be viewed as a snapshot of Social Listening solutions at that time rather than a definitive assessment.

Winners by Criterion

Here are the winners by criterion. Detailed ratings for each criterion are included in our comprehensive study. Additionally, we identified a winner per major category of criteria.

As mentioned in the article, no criteria were weighted. We understand that depending on each brand's needs and specifics, different criteria might be crucial or disqualifying. Our study aimed to be cross-sectional, so we aggregated the points awarded by the jury across all criteria to provide an overall ranking by points.

Key Takeaways:

• No “Winner Takes All” on the market or during the contest.
• The lack of standardization in the Social Listening industry is a challenge for brands and users.
• Data collection conditions frequently change at the discretion of social networks, complicating matters for vendors. Brands must have a clear definition of their data needs before selecting a tool.
• Social Listening is transforming to offer less "self-centered" KPIs and better align with the needs of Marketing and Digital departments to gain traction within organizations.
• Automation remains a challenge.
• Finding a single software solution for a brand on a “global scale” is nearly impossible, thus necessitating the combination of multiple platforms.

Want to Learn More? Contact us for a detailed and personalized presentation of the results.

Further Reading

Click here to explore our preliminary market analysis leading up to the Martech Challenges to better understand the Social Listening market.