CCi Mainstreaming and RHD Jams Outcomes Report

Abstract The ARC Centre of Excellence in Creative Industries and Innovation (CCi) commissioned two jams –online collaborative events – to discover new ideas for improving outcomes in two important aspects of the Centre’s work. The first jam – known as the CCi Mainstreaming Challenge - was held for two weeks from 8th to 23rd June 2009 and focused on findings ways to improve the mainstream impact of the Centre’s research. The event involved 112 participants from the CCi’s nodes around Australia and internationally, CCi Advisory Board members and a select group of external ‘critical friends’ (see Appendix A for a full participant list). Participants posted their own ideas, and commented and voted on the ideas of others, on a challenge question related to improving the mainstream impact of the Centre’s research. In total the jam generated 22 substantial new ideas, 103 votes and 96 comments. The second jam – known as the CCi RHD Challenge – was held from June 8th to 24th 2009 to elicit ideas and opinions from the Centre’s Research Higher Degree (RHD) community on how to improve their professional development experience and outcomes during their student tenure. The event invited 75 RHD students to participate and was kept exclusive to this group (i.e. no involvement from supervisors or Centre management). In total the community generated 17 ideas, 26 comments and 78 votes. This report summarises the key results and insights generated from both Challenges, and is divided into three parts: Section One describes the challenges around which these two events were focused and participation rates achieved, while Section Two summarises the outcomes in terms of statistics and trends by theme, as well as the value of the tool for ‘capturing the long tail’ of collaborative processes. Section Three concludes by presenting the top ideas by popularity and activity, as well as identifying the ideas that most polarised opinion and emergent themes.


Introduction
The ARC Centre of Excellence in Creative Industries and Innovation (CCi) commissioned two jams -online collaborative eventsto discover new ideas for improving outcomes in two important aspects of the Centre"s work.
The first jamknown as the CCi Mainstreaming Challenge -was held for two weeks from 8 th to 23 rd June 2009 and focused on findings ways to improve the mainstream impact of the Centre"s research. The event involved 112 participants from the CCi"s nodes around Australia and internationally, CCi Advisory Board members and a select group of external "critical friends" (see Appendix A for a full participant list). Participants posted their own ideas, and commented and voted on the ideas of others, on a challenge question related to improving the mainstream impact of the Centre"s research. In total the jam generated 22 substantial new ideas, 103 votes and 96 comments.
The second jamknown as the CCi RHD Challengewas held from June 8 th to 24 th 2009 to elicit ideas and opinions from the Centre"s Research Higher Degree (RHD) community on how to improve their professional development experience and outcomes during their student tenure. The event invited 75 RHD students to participate and was kept exclusive to this group (i.e. no involvement from supervisors or Centre management). In total the community generated 17 ideas, 26 comments and 78 votes.
This report summarises the key results and insights generated from both Challenges, and is divided into three parts: Section One describes the challenges around which these two events were focused and participation rates achieved, while Section Two summarises the outcomes in terms of statistics and trends by theme, as well as the value of the tool for "capturing the long tail" of collaborative processes. Section Three concludes by presenting the top ideas by popularity and activity, as well as identifying the ideas that most polarised opinion and emergent themes.
Section One: Challenge Summaries

1.1a
Context and question for CCi Mainstreaming Challenge The challenge context and question for the CCi Mainstreaming Challenge were developed in close consultation with Centre management and Terry Cutler, Chair of the CCi Advisory Board, and were designed to reflect the fact that there exists considerable diversity amongst Centre participants in what constitutes "impact". The primary challenge question posed by the jam was thus kept deliberately broad:

How can we increase the CCI's mainstream impact?
The challenge question was accompanied by a short context-setting text to provide background and guidance on responding to the challenge question: Terry Cutler, Chair of the CCI"s Advisory Board, wants to hear your ideas for how the Centre can have greater mainstream impact.
This follows from observations that collectively, the CCI has a unique set of perspectives on social, technological, institutional and cultural change, but these are not always sufficiently focused on mainstream issues in our respective fields.
What should we do to ensure that our work at the Centre has the greatest impact possible, within our own disciplines, within business and policy worlds, and in media and public perception? How can we better leverage the assets and capabilities of the CCI? Are we doing everything we can to take advantage of cross-disciplinary synergies?
Increasing the CCI's mainstream impact might involve looking to international models for inspiration, or adopting some of the practices of think tanks or the private sector. It might mean developing brand new "offerings" from the Centre. Or it may mean simply getting better at celebrating what we already do. Importantly, we want to think about what kind of legacy the Centre leaves and what kind of springboard it provides for all of us involved as we move on to other endeavours in the future. This challenge is broad in remit because we recognise that creating greater impact on mainstream issues means such different things to each of us. That"s why we"re interested to hear your ideas, and your opinions on the ideas of others. We"ve also invited a number of trusted friends and collaborators into our discussion to hear their thoughts as well.
The challenge context and question were consciously positioned as widely and openly as possible in order to stimulate a broad range of responses and minimise the risk of disenfranchising particular cohorts within the participant group.

1.1b Context and question for CCi RHD Challenge
The RHD Challenge was structured around three key themes reflecting current areas of concern in the experience of Research Higher Degree students at the Centre. The wording of the themes was designed to be as meaningful as possible to the RHD participants in the event: The CCI's interdisciplinary research agenda, including what you understand it to be and whether it has helped you in your own program of work. What you need in regards to professional development, above and beyond disciplinary scholarship and direct contact with supervisor(s). This could include: skills to disseminate work to different audiences conferences, networking and profile-building opportunities access to jobs both during and after life as a research student taking advantage of formal courseware Your hopes and expectations for future workspecifically the career position you hope to reach after graduation and how your degree will help you get there.
Participants were also provided with a "free space" area where any other ideas not captured in the above listed themes could be posted.

1.2a Participation in CCi Mainstreaming Challenge
Of the 112 people invited to participate in the CCi Mainstreaming Challenge, 31.25% made contributions in the form of ideas, comments and or votes. This is an above average participation rate compared to standard "ideation" events which rarely achieve participation rates of more than 20%.
Of those who did participate in the CCi Mainstreaming Challenge, just over ten percent contributed by vote only (i.e. did not post any ideas or comments). This highlights that the jam process provides two distinct ways for individuals to participatethrough written contributions and through expressing opinions in the forms of votes. The CCi Mainstreaming community generated a vote-to-idea ratio that approximately matches the average of other jam events, voting at a rate of about 5 to 1 1 .

Figures 1 and 2 below
show that participation over the duration of the event was strongly influenced by the communications programme designed to stimulate an increase in participation rates. Votes and comments in particular spiked significantly after the mid-point email reminder sent on 15-06-09. Figure 1 also shows that posting of ideas started high at the commencement of the event but tailed off over timethis is a very typical pattern for idea contribution held over a multi-day period. However it is worth noting that in this case the communications programme was sufficiently effective to cause a small spike in new ideas posted on 15-06-09. It is typically very difficult to achieve this spike in idea contributions in the second week of a jam.

1.2b Participation in CCi RHD Challenge
Of the 75 people invited to participate in the CCi RHD Challenge, 32% made contributions in the form of ideas, comments and or votes. This is an above average participation rate compared to standard "ideation" events which rarely achieve participation rates of more than 20%.
Of those who did participate in the Challenge, more than 45% contributed by vote or comment only, with 8% contributing by comment only and 37.5% contributing by vote only. This highlights that voting was an extremely important mechanism for expressing opinions for this particular community. However the CCi RHD community generated a voteto-idea ratio that is slightly lower than the average of other jam events, voting at a rate of about 4.5 votes per idea on average.

Figures 3 and 4 below
show that participation over the duration of this event was also strongly influenced by email reminders. Votes and comments spiked significantly after the mid-point email sent on 16-06-09, albeit with a delayed response of about 24 hours. It is also worth noting that the commenting rate in particular stayed high in the second week of this event, which may be attributed to the effectiveness of the reminder email or to the characteristic behaviours of the group involved 2 .  In total, the CCi Mainstreaming Challenge generated 22 ideas, 103 votes and 96 comments, while the RHD Challenge generated 17 ideas, 78 votes and 26 comments. These are particularly strong results considering that no face-to-face events, such as workshops, were held to augment the online jams prior to or during their execution (although results for both were presented and discussed at a research symposium held after both jams had closed).

2.1
Outcomes by theme Figure 5 below shows the distribution of ideas, comments and votes contributed in the CCi RHD Challenge grouped by theme. (Note that this breakdown is not possible for the CCi Mainstreaming Challenge as it involved a single challenge question.) The "future work opportunities and challenges" and "free space" themes clearly captured the most activity in terms of ideas, comments and votes. It is interesting to note that no ideasand therefore no comments or voteswere posted against the theme related to the Centre"s interdisciplinary research agenda. This indicates that this particular theme may have been mismatched with the actual concerns of the RHD community. As shown in Figures 6, 7 and 8 below, the dominance of the "free space" or "miscellaneous" theme in attracting ideas, comments and votes indicates that the challenge themes were not particularly well matched to the types of concerns and opportunities perceived by the group. This may be due to the fact that this group had previously been engaged by the Centre via surveys, and some of the categorisation of "issues" identified through that mechanism may have inadvertently bled into the design of the jam process. This highlights that survey/questionnaire activities are fundamentally different to ideation initiatives, in that the former counts and categorises responses to known questions whereas the latter seeks potential solutions to problems or challenges broadly defined.
A recommendation arising from this outcome is that a single challenge (i.e. non-themed) jam design should be used unless there is strong and accurate knowledge of the topic categories that are relevant for the particular combination of challenge and participant group.

Long tail of contributions
Contributions to both Challenges conformed to a "long tail" distribution (also known as a power law distribution). This means that a small number of people were responsible for the majority of the activity in each event. This pattern of contribution is seen in many online and interactive phenomena such as Wikipedia, social networking and internal firm collaboration processes.
In the CCi Mainstreaming Challenge, the top 3 individual contributors (in terms of the number of comments and ideas submitted) contributed just under 40% of the total ideas and comments generated by the jam. This is illustrated in Figure 9 below.

Figure 9: Proportion of contributions (ideas and comments) from most to least prolific contributor in CCi Mainstreaming Challenge
It may be the case that those contributors who are relatively prolific in an online context are much less likely to contribute in traditional face-to-face settings, so that their "voices" can be more clearly "heard" through a web-based process such as the jam. It is only possible to demonstrate this conclusively, however, by comparison to data on face-to-face contribution rates, which are not currently known for these groups. The concentration of popular ideas in the long tail of the distribution shows that the jam provides an effective mechanism for capturing valuable ideas from individuals who are relatively less prolific in their contributions compared to others. For example, the second most popular idea in the CCi Mainstreaming Challenge was submitted by one of the least prolific contributors in the event.

Section Three: Ideas
This section presents the top ideas by popularity and activity, as well as identifying the ideas that most polarised opinion and emergent themes.

3.1
Top ideas by popularity Votes cast during a jam can be negative (a thumbs-down or "Demote") or positive (a thumbs-up or "Promote" for the idea), resulting in an overall net score per idea. The top ideas by popularity (i.e. with the highest net score) for each Challenge are summarised in Tables 1 and 2  An increased focus on creative production processes Phil Graham  Prizes were awarded to the contributor of the most popular idea in each eventa bottle of fine wine in the case of the CCi Mainstreaming Challenge; and a $1,000 travel bursary in the case of the RHD Challenge.

Top ideas by activity
A key principle in making the most of a jam process is to recognise that the most popular ideas are not necessarily the most valuable to the organisation.
We can also rank ideas by the volume of overall activity they attracted in terms of both votes and comments. In Table 3 below, ideas are ranked by the number of votes they received, whether negative, positive or indifferent (i.e. "No Opinion"), as shown in the column "All Votes", plus the number of comments generated, as shown in the column "Comments".

3.3
Controversial ideas The jam voting system allows participants to express three types of opinion about an idea: positive ("Promote"), negative ("Demote") or indifferent ("No Opinion"). The total number of votes received by an idea may not correlate with its popularity if the idea receives a relatively large number of negative or indifferent votes. This is demonstrated in the discrepancy between the values in the columns "All Votes" and "Net Score" in Table 3 above.  The above breakdown of activity by idea emphasises that it is important to take a holistic view of each ideaand the qualitative and quantitative responses that each has generatedrather than focusing solely on relative popularity as given by the net voting score.

Emergent Themes
As discussed, the Mainstreaming Challenge used a single question design due to the wide divergence of views known to be held on the jam topic amongst the invited participants. While the three specific themes originally chosen for the CCi RHD Challenge were successful in eliciting a high response rate and strong voting levels, the relatively dominant use of the "free space" or "miscellaneous" theme indicates that the three original themes were a less than perfect match for the ideas and discussion subsequently generated -although all clearly fell under the umbrella theme of "improving the RHD experience at CCi".
In both cases there was a strong rationale for conducting a "meta-analysis" to identify emergent themes that collectively describe all of the ideas put forward during the events, and assist with deciding on specific actions going forward. These emergent themes are based on "bottom-up" analysis of the jam content itself rather than on the "top-down" themes that were originally conceived for structuring the discussion (in the case of the RHD Challenge) or the wide scope created by the deliberately broad challenge question set for the Mainstreaming jam.
To demonstrate this approach, six "emergent themes" from the Mainstreaming Challenge are identified: