What Makes an Expert?

Book Journey - Part 2

2022, Jan 27    

Preamble

While a paper summary would usually end up at @paper_takeaways, this one has too many good points, worth a blog post.

Towards a Theory of Software Development Expertise, D.O.I 10.1145/3236024.3236061, written by Sebastian Baltes and Stephan Diehl.

Note: yeah, I was supposed to be reading Guidelines for performing Systematic Literature Reviews in Software Engineering and Systematic literature reviews in software engineering – A tertiary study, and be working on my own review paper about Systems Debugging, but I think you’ll like this one just like I did.

Towards a Theory of Software Development Expertise, the paper

The authors constructed a Grounded Theory of what makes a software engineer a field expert, based on a set of concepts and relationships.

Concepts are the core points of the theory, the “thing” that in this case makes an expert, and relationships are points that connect these concepts, so a relationship is an action/event, that improves/generates/acts upon two or more concepts.

The authors used datasets from both GitHub and StackOverflow, broken into three sets of users with varying levels of experience in software engineering, always focused on the Java programming language.

  1. the first set (S1) contained 122 participants, 67.2% being software developers and 13.9% software architects
  2. the second set (S2) contained 127 participants, 64.6% software developers and 13.4% software architects
  3. the third set (S3) contained 91 participants, comprised of developers with the most experience in all sets, ages between 55 and 80 years old and a median of 34 years of general programming experience (⊙_⊙). 61.6% reported being a software developer and 17.5% a software architect

Questions

For S1, the questionnaire contained seven open-ended and four closed-ended questions around two central points:

RQ1: Which characteristics do developers assign to novices and which to experts? RQ2: Which challenges do developers face in their daily work?

For S2 and S3 the questionnaire was later refined:

RQ1.1: Typical character traits of experts. RQ1.2: Traits that are supportive for becoming an expert. RQ1.3: Situations where developers’ performance declined over time. RQ1.4: Tasks that an expert should be good at. RQ2: was changed to better represent questions around Work Context and Motivation. RQ3.1: Questions about monitoring (oneself). RQ3.2: Questions about mentoring (another).

General Research Design

The participants from S1 were sent the questions described above (RQ1 and RQ2), and with their answers the authors formed the initial layout of their Grounded Theory, which brought forth the following concepts and relationships:

Concepts

  • Experience: most answers indicated that an expert ought to have lots of actual experience in the field, in terms of QUANTITY, but some other answers also pointed out to PROFESSIONAL EXPERIENCE, as in, shipped code to production, and worked on SHARED CODE
  • Knowledge: since the research was done with the Java programming language, the first topic here was LANGUAGE-SPECIFIC, like Java frameworks. But also DEPTH OF KNOWLEDGE, and then BROAD KNOWLEDGE, meaning an expert should know a bunch of things pretty well
  • Quality of source code: experts create WELL STRUCTURED, READABLE and MAINTAINABLE source code
  • Behavior, character traits, and skills: here, COMMUNICATION SKILLS was the biggest pointer of an expert, someone who knows how to communicate clearly with the broader team/organization
  • Work context: the work-related issues that most affected participant’s work was PEOPLE and TIME PRESSURE, and then REQUIREMENTS not being well specified or hard to understand, closely related to COMMUNICATIONS because users don’t know what they want/need

Relationships

After going through the participants’ answers and creating the concepts above, the authors came up with the following relationships:

Certain behaviors plus character traits and general skills help gain more knowledge and experience which then produces a higher quality of source code, and the work context influences behavior from the get-go.

After this initial phase, the authors turned their research to S2 and S3, with an improved questionnaire, now guided by the initial theory layout. The answers from S2 and S3 in their turn, further refined the theory to its final form, with the following concepts and relationships:

This isn’t even my final form

Concepts

According to the answers given by S2 + S3 (so 218 participants), these are the core areas in which an expert Software Engineer should be proficient at:

  • Tasks: an expert should be good at performing theses tasks:

    • DESIGNING SOFTWARE ARCHITECTURE (95)
    • WRITING SOURCE CODE (91)
    • ANALYSING AND UNDERSTANDING REQUIREMENTS (52)
    • TESTING (48)
    • COMMUNICATING (44)
    • STAYING UP-TO-DATE (28)
    • DEBUGGING (28)¹

  • Experience, knowledge, and performance:

    • GENERAL KNOWLEDGE (35)
    • SPECIFIC KNOWLEDGE (30)
    • EXPERIENCE (26)

    And these would be the qualities of an expert’s source code:

    • MAINTAINABILITY (22)
    • CLEAR STRUCTURE (12)
    • PERFORMANCE (9)

  • Individual differences: experts should be:

    • OPEN-MINDED (42)
    • TEAM PLAYERS (37)
    • CURIOUS (35)
    • have ATTENTION TO DETAIL
    • be PATIENT (26)
    • have SELF-REFLECTION (20)

    Regarding skills that an expert should possess:

    • PROBLEM-SOLVING (84)
    • ABSTRACTION/DECOMPOSITION (30)
    • ANALYTICAL THINKING (20)
    • ASSESS TRADE-OFFS (19)
    • LOGICAL THINKING (17)

  • Mentoring: more than half (54.3%) of S2 and S3 participants had a mentor that made them a better developer. The most important traits of a mentor are:

    • CHARACTER (29)
    • SKILLS (19)
    • EXPERIENCE (16)
    • give FEEDBACK(20)
    • MOTIVATE (19)

  • Monitoring and self-reflection: how should an expert monitor their own performance:

    • PEER REVIEW (16)
    • TIME TRACKING tools (14)
    • PROJECT TRACKING tools (14)
    • ISSUE TRACKING (14)

    There’s also references to COMMIT FREQUENCY, LINES OF CODE ADDED/DELETED and ISSUES RESOLVED, but I highly disagree with using these as metrics to anything but themselves. One participant noted:

    I do not think that measuring commits LOC […] automatically is a good idea to rate performance. It will raise competition, yes-but not the one an employer would like. It will just get people to optimize whatever is measured

    See Goodhart’s law.

  • Motivation: what motivates an expert:

    • RESULT (55) that they bring with the solution
    • PROBLEM SOLVING (46), because it’s rewarding to solve an issue
    • HELPING OTHERS (37)
    • QUALITY (22) of the solution implemented/proposed
    • CREATING SOMETHING NEW (19)

  • Work context: characteristics of a good work environment that enable a person to more easily become an expert:

    • ENCOURAGE LEARNING(70)
    • ENCOURAGE EXPERIMENTATION (61)
    • IMPROVE INFORMATION EXCHANGE (53)
    • GRANT FREEDOM (42)
    • pays and incentivizes TRAINING COURSES (34)
    • pays and incentivizes to attend CONFERENCES (15)

  • Performance decline:

    • DEMOTIVATION (34), like NON-CHALLENGING WORK (8)
    • WORK ENVIRONMENT (32), like BAD MANAGEMENT (8), STRESS (6) and TEAM STRUCTURE (5)
    • AGE-RELATED DECLINE (13)
    • CHANGES IN ATTITUDE (10), like changing one’s priorities in life (focus more on family, friends and health, for example)
    • SHIFTING TOWARDS OTHER TASKS (7)

Relationships

The only relationship added was mentoring, which improves motivation and gives feedback towards expertise.

Programming Experience vs. Expertise

The paper wasn’t able to draw any correlation regarding years of programming experience with one’s actual expertise in the field.

The actual summary

It took you long enough, Sam and Frodo rest after a good day's work

And so, at last we get to an actual summary. Now in my own words and interpretation, a Software Engineer Expert should:

Write well architected, maintainable and tested projects, while clearly communicating what’s happening and setting proper expectations with the team (management or otherwise), understanding requirements, and not just what the customers or PMs have told them.

An expert should be open to suggestions and criticism from anybody else, senior or junior, new or old on the team, should be at all times teaching and mentoring less experienced engineers, and also reflecting upon their own performance, motivation, behaviors and the like.

For me, an important step on the expertise ladder would be to get all of this that we just discussed, and attempt to do a little bit of everything at all times. You don’t need to write a well architected project, just write a better architected project than last time. You don’t need to mentor a junior engineer everyday, just give’em a couple of hints and help every once in a while.

Below, a simple diagram to summarize the concepts and relationships we’ve seen so far, and how they connect to each other:

Diagram summarizing Concepts and their Relationships

Best References

Somewhat unrelated to the post, these are the references that caught my eye the most:

  • [3] John Anvik, Lyndon Hiew, and Gail C. Murphy. 2006. Who Should Fix This Bug?. In 28th International Conference on Software Engineering (ICSE 2006). 361ś370.

  • [8] Sarah Beecham, Nathan Baddoo, Tracy Hall, Hugh Robinson, and Helen Sharp. 2008. Motivation in Software Engineering: A systematic literature review. Information and Software Technology 50, 9 (2008), 860ś878.

  • [13] Kathy Charmaz. 2014. Constructing grounded theory (2nd ed.). Sage.

  • [19] Juliet Corbin and Anselm Strauss. 2008. Basics of qualitative research (3rd ed.). SAGE Publications.

  • [22] Oscar Dieste, Alejandrina M. Aranda, Fernando Uyaguari, Burak Turhan, Ayse Tosun, Davide Fucci, Markku Oivo, and Natalia Juristo. 2017. Empirical evalua- tion of the effects of experience on code quality and programmer productivity: An exploratory study. Empirical Software Engineering 22, 5 (2017), 2457ś2542.

  • [36] Barney G. Glaser and Anselm L. Strauss. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Transaction.

  • [40] Daniel Graziotin, Xiaofeng Wang, and Pekka Abrahamsson. 2015. How do you feel, developer? An explanatory theory of the impact of affects on programming performance. PeerJ Computer Science 1 (2015), e18.

  • [62] Thomas D. LaToza, Gina Venolia, and Robert DeLine. 2006. Maintaining mental models: A study of developer work habits. In 28th International Conference on Software Engineering (ICSE 2006). 492ś501.

  • [63] Paul Luo Li, Andrew J. Ko, and Jiamin Zhu. 2015. What Makes A Great Software Engineer?. In 37th International Conference on Software Engineering (ICSE 2015).

  • [69] David C. McClelland. 1987. Human motivation. Cambridge University Press.

  • [90] Richard M. Ryan and Edward L. Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.

  • [96] Janice Singer, Timothy C. Lethbridge, Norman G. Vinson, and Nicolas Anquetil. 1997. An examination of software engineering work practices. In 1997 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON 1997).

  • [97] Dag I. K. Sjoberg, Tore Dyba, Bente C. D. Anda, and Jo E. Hannay. 2008. Building theories in software engineering. In Guide to Advanced Empirical Software Engineering. 312ś336.

  • [117] Minghui Zhou and Audris Mockus. 2010. Developer Fluency: Achieving True Mastery in Software Projects. In 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2010). 137ś146.

¹: I can’t place DEBUGGING on a scale compared to the other tasks mentioned, but I believe it should be much more cited. On my day to day, DEBUGGING goes along with WRITING SOURCE CODE, while TESTING and DESIGNING SOFTWARE ARCHITECTURE are more spaced, so there’s that.

# paper # expertise # paper-journey # book-journey