The Commercial Haskell SIG members want to help people adopt
Haskell. What would help? Data beats speculation, so FP Complete
recently emailed surveys to over 16000 people interested in
Haskell. The questions were aimed at identifying needs rather than
celebrating past successes, and at helping applied users rather
than researchers.
Over 1240 people sent detailed replies, spending over 250
person-hours to provide their answers.
This rich data set includes extensive information on Haskell
user needs. We are open-sourcing the entire anonymized data set,
downloadable by clicking
here [.zip]. There are numeric ratings and extensive textual
comments. Feel free to analyze the data -- or just read the
summaries -- and share your most helpful and actionable conclusions
with the community. We will too.
First Results
Although we have completed only basic analysis, here are some of
our first findings -- those so clear that they show up on even the
most basic examination of the aggregate data.
- Satisfaction with the language, the compiler, and the community
are high.
- Among non-students, 58% would recommend Haskell for a project
at their workplace, but only 26% actually use it at work -- partly
due to colleagues unfamiliar with Haskell who see it as requiring
skills that are hard to obtain, or who need to see more success
stories. Would improvement to colleagues' perceptions make a
difference in the team's choice of Haskell for a project? 33% of
respondents rated this "crucial" and another 26% said it would be
"important", while only 16% said it would be a "slight help" or no
help.
- Package management with cabal is the single worst aspect of
using Haskell. Asked if improvements to package management would
make a difference to their future choice of Haskell for a project,
38% said it would be "crucial" and a further 29% said it would be
"important". Comments connected cabal with words like hell, pain,
awful, sucks, frustrating, and hideous. Only this topic showed such
grave dissatisfaction.
- Documentation improvements are a very high priority. For
example, users need more concrete tutorials and templates showing
them exactly what kinds of problems Haskell is good at solving, and
exactly how to implement such programs completely. 65% of
respondents said improvements to documentation and learning
resources would be crucial or important, and a further 23% said
they would be helpful. However, comments did not begin to approach
the level of concern seen with cabal.
- Skills are a priority. Users need to see that people with
Haskell skills are readily available, and that Haskell skills are
quite feasible to learn. A majority of respondents said an
improvement in availability of skilled personnel would make an
important or crucial difference to them, and many also expressed
concern about their or colleagues' abilities to learn the needed
concepts and skills.
We have started deeper statistical analysis of the data, and we
hope that some readers of this post will perform -- and share --
even better analyses than we can. New issues may become clearer by
clustering or segmenting users, or through other statistical
techniques. Also, we may find more clarity about needs through
deeper study of textual responses. Follow-up studies are also a
possibility.
We propose that the community, given this large and detailed
data set, should set some of its priorities in a data-driven manner
focused on user-expressed needs. This effort should be
complementary to the ongoing research on issues of historic Haskell
strength such as language design and runtime architecture.
Areas for Further Work
We request that useful findings or insights derived from the
open-sourced data set be shared with the community, including
attribution of the source of the data and the methods of analysis
used.
The data collected strongly invites clustering or segmentation,
so as to identify needs of different sub-populations. FP Complete
has already begun one such study.
The data collected includes extensive textual remarks which
should be studied by knowledgeable people for insights. Automated
text analysis methods may also be applicable.
Cost-benefit analysis seems worthwhile: based on identified
needs, what improvements would help the most people, to the
greatest degree, at the least cost and/or the least delay? A method
to match volunteer contributors with identified high-payoff
projects also seems worthwhile.
It would be useful to merge the data from versions 0.1 and 0.2
with version 1.0 of the survey, since 1.0 includes only 71% of the
total answers received. Differences between the questions and
scales make this a nontrivial, but still realistic, goal.
If important new hypotheses require testing, or if further
detail is needed, we intend to conduct follow-up research at some
future date among users who volunteered their email addresses for
follow-up questions.
A future repeat survey could determine which improvement efforts
are successful.
Methodology Notes
This was not a survey of the general public, but of a population
motivated to provide feedback on Haskell. Invitees included 16165
non-opted-out email addresses gathered from FP Complete's website,
in randomized order. Due to privacy considerations this list will
not be published, but FP Complete was able to use it to contact
these users since the survey was directly related to their
demonstrated interest in Haskell. The high quality of the list is
reflected in the extremely high response rate (7.7%), the low
bounce rate (1.9%), and the low unsubscribe rate (also 1.9%).
Surveys were conducted using SurveyGizmo.com, with an email
inviting each participant to click a link to a four-page Web-based
survey. Survey form 0.1 invitations went to 1999 users of whom 190
completed the survey. Survey form 0.2, incorporating some edits,
went to 2000 users of whom 170 completed the survey. Survey form
1.0, incorporating further edits, went to 12166 users of whom 894
completed the survey.
Form 0.2 incorporated edits to eliminate questions yielding
little information about how to help users, either because
satisfaction was very high (the language itself, the compiler, the
community) or because two questions were redundant. Also, new
questions inspired by textual responses to form 0.1 were
included.
Form 1.0 incorporated further such edits. Also, the rating scale
was changed to ask about helping the user's (and team's) future
choice of Haskell rather than current usefulness/difficulty. The
ratings questions were displayed under the heading "Would
improvements help you and your group to choose Haskell for your
future work?"
Responses were processed anonymously, but users were given the
option to fill in their email address if they would accept
follow-up questions, and the option to name their
company/organization. Users were informed that the survey results,
without these fields, would be shared with the community.
Acknowledgments
We are grateful to the many, many people who spent their
valuable time and expertise completing and returning their survey
forms. Thanks to Dr. Tristan Webb and Ms. Noelle McCool-Smiley,
both of FP Complete, for their material help in formulating and
conducting the survey. Thanks to FP Complete's corporate customers
for providing the revenues that allow us to fund this and other
community projects. Thanks to the Commercial Haskell SIG for
providing the motivation behind this project. Thanks to the many
volunteers who've spent absolutely huge amounts of time and
expertise making Haskell as good as it is today, and who continue
to make improvements like those requested by the survey
participants. Thanks to the companies that allow some of their
staff to spend company time making such contributions to the common
good. Special thanks to the late Professor Paul Hudak; may we all
strive to live up to his example.
Subscribe to our blog via email
Email subscriptions come from our Atom feed and are handled by Blogtrottr. You will only receive notifications of blog posts, and can unsubscribe any time.
Do you like this blog post and need help with Next Generation Software Engineering, Platform Engineering or Blockchain & Smart Contracts? Contact us.