[TOS] Help put Research Minion Mel to work - what do you want to measure?

[Mel Chua]

> Well... "valuable" w.r.t. measurement usually implies that you know
> what the question is. Collecting data without knowing the question is
> a problem. The spirit of your questions seem to be "what is the
> activity/pulse of this community?" If that's the question, then that
> might make it easier to think of what to measure.

I think that's accurate - "is the TOS community alive and helping 
professors teach open source classes?" Max and I pow-wowed for a bit 
after lunch today and I decided I was going to limit myself to measuring 
5 things, every week:

1. TOS presence at events - that's 
http://mchua.fedorapeople.org/tos-scorecard/ambassadors.png, except 
expanding it to be all TOS community members rather than just students.

2. TOS publications - that's 
http://mchua.fedorapeople.org/tos-scorecard/publications.png, basically 
any paper any TOS community member writes about TOS.

3. TOS Planet activity - # of total posts, # of individuals posting, # 
of individual classes represented by these posts.

4. TOS list activity - # of total messages, # of individuals posting, # 
of subscribers (which I don't think I have permission to view, actually 
- this is something the infra team would have to decide to give me 
access to, else I'll drop that metric).

5. POSSE module attendance - how many folks show up each week to the 
online classes we're teaching? (Just going for sheer numbers right now; 
once we get a gauge on what types of folks show up - profs, students, 
devs, etc - we might also start tracking proportions.)

Lots of open questions about how precisely to get accurate numberss on 
these; need criteria, etc and there are some things to address like 
"boy, it's hard to count Planet posts when Planet is down!" but it gives 
me a solid target to aim for.

I'll be running these numbers for the first time for this week's stats, 
and we'll see how hard they are to generate.

> If you genuinely thought that the study of FOSS communities was going
> to play a role in your dissertation work, then a dashboard-like
> project could be a very powerful way to get research data. However,
> you can likely pursue a dissertation with three case studies just as
> well as a firehose of incomprehensible commit logs... it all depends
> on your question. But, in the meantime, if you designed an amazing,
> best-of-breed, FOSS participation dashboard... yeah, I'd use it. :D

I'm hoping that I'll work on this stuff and a few years from now look up 
and go "...wait, 70% of my dissertation data has already been collected 
by shell scripts over the past 3 years, I just need to do a bunch of 
analysis and writing!" I'd love to have a ton of quantitative data to 
get some general trends across classes and institutions and help us zone 
in on more interesting places to look for case studies... but that's 
wishful thinking from someone who's not yet learned to do proper 
research and does not have a properly-formed research question, and I 
realize all this is potentially a ton of work and I need to figure out a 
smallish bite-sized bit that will actually let me graduate before I retire.

I'll start very small, with metrics that I personally care about and can 
manually capture in a very short amount of time every week, then see how 
that goes before I even think about the automation. Not getting all 
starry-eyed about a groundbreaking new research platform right now or 
anything... just trying to consistently measure *something* and see if 
it gets any insights, and then going from there.

--Mel