Now, there’s a hot topic. Measuring engagement. One of the most widely debated topics in web analytics.
What is engagement and how do we measure it?
Engagement, unfortunately, is not derived from a single measure. It’s not time on site. It’s not how many pages they viewed. It’s not bounce rates and it’s not about conversions.
Engagement is about a lot of things. What is an engaged visitor and how do you measure engagement?
“Visitor Engagement is an estimate of the depth of visitor interaction against a clearly defined set of goals.” Eric T. Peterson and Joseph Carrabis.
A while ago, I came across their paper through Web Analytics Demystified, entitled “Measuring the Immeasurable: Visitor Engagement”. While I won’t go into it in any detail, I will suggest that you read it, as it’s the background of this post.
The premise of the paper is that visitor engagement is made up of 7 different metrics, and expressed through one formula:
where:
Engagement can be expressed as the average of the sum of indexes, across specific segments, according to a:
- Click Depth Index – which captures the contribution of page and event views
- Duration Index – capturing the contribution of time spent on site
- Recency Index – which captures the visitor’s “visit velocity”—the rate at which visitors return to the web site over time
- Loyalty Index– the level of long-term interaction the visitor has with the brand, site, or product(s)
- Brand Index – the apparent awareness of the visitor of the brand, site, or product(s)
- Feedback Index – qualitative information including propensity to solicit additional information or supply
- Interaction Index – visitor interaction with content or functionality designed to increase level of Attention
According to Eric T. Peterson, “Visitor Engagement is a function of the number of clicks (Ci), the visit duration (Di), the rate at which the visitor returns to the site over time (Ri), their overall loyalty to the site (Li), their measured awareness of the brand (Bi), their willingness to directly contribute feedback (Fi) and the likelihood that they will engage in specific activities on the site designed to increase awareness and create a lasting impression (Ii).”
When applied at the visitor level, on a per-visitor basis, they combine to form a pretty good proxy for visitor engagement.
Having read the paper, I was intrigued, and decided to use Discover to implement this…to some pretty insightful results.
A summary of the indexes
Click Depth Index
The percentage of your overall audience that has a minimum threshold of an acceptable number of page views per session. If you see that on average, visitors “convert” after viewing at least 5 pages, then your minimum threshold would be 5 pages per visit.
Duration Index
The percentage of your overall audience that has a minimum threshold of an acceptable amount of time on site per session. If you see that on average, visitors “convert” after spending at least 10 minutes on your site, then your minimum threshold would be 10 minutes.
Recency Index
The percentage of your overall audience that returns and converts within an acceptable amount of time (generally days). If you notice that most visitors convert between 1 and 10 days, then you’d be looking for visitors with a return frequency of <= 10 days.
Loyalty Index
The percentage of your overall audience that has a repeat visit frequency in excess of a minimum threshold. For example, if you notice that many visitors convert after visiting your site more than three times, then your threshold would be a visit count of at least 3.
Brand Index
The percentage of your overall audience that comes to your site either directly, or through branded search terms.
Feedback Index
The percentage of your overall audience that completes feedback on your site, or participates in rating or reviewing content, or commenting on blogs.
Interaction Index
The percentage of your overall audience that interacts with specific content on your site, or engages in an activity on your site. There are no thresholds for this index – they are simply counts of.
Note: while you can count pre-defined activities on your site, it is better to score visitor interaction. I’ll be doing a post on visitor scoring shortly.
Using Discover
Discover was built for this! It’s very easy to create segments within Discover and apply them across various views to gain insight.
Firstly what we did was to look at some of the thresholds to understand what our “Anonymous” segment of traffic does. Our anonymous segment is made of non-student and non-staff traffic, which we already have segments for in both Discover and SiteCatalyst.
We figured out what our minimum page views per session should be, the average duration, frequency of visit etc, by looking at them from a conversion standpoint…i.e. how many pages does a converter see, on average, before converting.
Once we’d done that, our 7 segments were easy to define as follows:
- Click Depth Index – Visitor container, Path length > 10
- Duration Index – Visitor container, Seconds spent per visit > 1800 (30 minutes)
- Recency Index – Visitor container, Return Frequency <= 7-14 days
- Loyalty Index – Visitor container, Visit number >= 2
- Brand Index – Visitor container, Organic Search Keyword contains “Murdoch” or Visit without referrer
- Feedback Index – (we don’t use this one)
- Interaction Index – Visitor container, any of the following events: Lead Complete, Application Complete, Form Complete, Tool Name
The 8th segment was All Visits. In each case, we used the Visitors metric to view the number of visitors that were part of each index.
If we view this against referring sites, what we end up with is the number of visitors that match each segment rule:
Export to Excel
What we need to do now is export the data to Excel to do the averages and generate the final engagement value.
Simply select the first item “None”, click Ctrl+A for select all, then click Ctrl+C for copy.
Open Excel, and paste the raw data into a new sheet.
Then it’s simply a matter of calculating one columns percentage as a percentage of the All Visits – Visitors column.
Once you’ve done that for each column, you have the indexes for each segment. Now you just average all of the indexes to get an engagement metric.
For example, we see in the above that Direct Traffic, “none” in the above report, has an overall engagement value of 23%. But if we look at the other columns, we also see that they are at the median value on Click Depth, whereas traffic from deewr.gov.au is well above the median.
A couple of interesting things have also been highlighted in the above, for example, traffic from Google Singapore is actually far more engaged than traffic from Google Australia – now that’s interesting.
Of course, you should always look at engagement via larger segments, for example, by Campaign, by Site, by Time of Day, Day of Week etc.
While Tuesday comes out overall for a better engaged visitor, I’ve highlighted other interesting things, such as on Saturday and Sunday visitors click more, but more visitors spend time on Saturdays. During the week is better for branded search term visitors, and Wednesdays seems to be better overall for key interactions.
Multiple Sites
If you have multiple sites, such as microsite etc, you might want to check engagement across them to see if they are dramatically different, so you can then begin to try to understand why.
In the above calculation, I’ve removed the Feedback and Interaction indexes from the calculation, as they would skew the results. It’s interesting that while the main University site has an engagement index of 23.67, versus a median of 20.43%, sites like Mobile and Maps have a very high engagement value.
Segmentation
Once you’ve got the basic engagement working and you’re looking at things from the overall perspective, you can then easily begin to look at engagement by different segments.
In Discover, you can create segments on-the-fly and apply them across your other segments, by simply dragging the new segment onto the filtered workspace. For example, we segment the above by Anonymous visitors, after we’ve built the overall segments. We can do the same for Converting Visitors, or Social Network visitors, or Campaign Visitors, or just Mobile visitors, or different content areas across the site etc. Discover makes it very easy to do this.
Once the spreadsheet is set up, all you need to do is copy the data back into the sheet and you’ve re-run the engagement metric – in about 5 seconds.
Discover just rocks for this real-time, conscious stream of thought, type of analysis.
In summary
There’s lots of different ways to look at engagement, and hopefully, this will help you understand that there is no single metric, and engagement values change based on various lenses. But, with the above combination of metrics from the very useful paper by Eric T. Peterson, I believe that we’re closer to understanding engagement, which will help us to modify our sites, or target content better, to try to achieve better levels of engagement by those who are below the medians.
As an aside, part two of this post will be about Visitor Scoring, which is a better Interaction Index than the one demonstrated above – and can be used directly in SiteCatalyst reports. It involves a bit of custom code for your s_code, and a bit of forethought, but easy enough to do…but I’m saving that for a bit later this week.
Drop me a line or comment below with ways that you are measuring visitor engagement.
