Harmonize GitHub labels and usage

TLDR

This is an attempt to harmonize GitHub labels and their usages across Parse repos. Below are the new labels and usage suggestions, up for discussion. Since this impacts every team member, your feedback is most welcome!

Issue

Currently there is a variety of labels and usages on GitHub across Parse Platform repos:

  • available labels vary across repos
  • no prioritization for bugs according to criticality
  • some labels are de-facto unused and clutter the label list
  • there is no guideline on how to use labels for common understanding
  • some repos use labels also on PRs, others only on issues

This can lead to issues such as:

  • issues may not get the appropriate attention according to their criticality (eg. #6728)
  • labels may be confusing for contributors and followers
  • security issues that should be submitted confidentially may stay published publicly for long periods of time

There is currently no way to highlight issues that should be fixed with high priority:

  • most repos have a common bug label
  • some repos have an additional unbreak now label for critical bugs that seems unused and there is no guidance on when to apply it
  • there is a distinct process to expedite security issues, not applicable to non-security bugs

Suggestion

  • Define a uniform set of GitHub labels across all Parse repos (excluding the docs repo due do its speciality of referencing multiple repos).
  • Introduce bug severity levels in GitHub labels to increase awareness for important issues, expedite fixing these issues and merging associated PRs.
  • Introduce a guideline for team members on when to apply which label to facilitate uniform label usage.

(The following part is continuously updated to show the latest consensus; please see the edit history for any changes incorporated and see the thread for the change reason.)

Uniform GitHub labels

1. Initial status

  • no label (issue has not yet been look at by a Parse team member)
  • needs more info (more info is needed from author or more investigation is needed by Parse team member to classify the issue)

2. Issue classifiers:

  • :bug: bug (impairs a documented feature or a functionality that is not explicitly documented but one would strongly expect the general user to infer from a documented feature)
  • :rocket: feature (a new feature that does not exist yet)
  • :dna: enhancement (performance improvement or functional enhancement of existing feature)
  • :grey_question:question (not an issue but a question; should be in Community Forum but we can’t easily transfer, so we refer the author to the Community Forum)
  • :wrench: troubleshooting (not an issue but asking a code-level question; should be in StackOverflow; for simple issues we may give a quick tip / solution to the author, in general we refer to StackOverflow to educate the author about where to post code-level questions)
  • :speech_balloon: meta (issues regarding CI and tests or pinned calls for high visibility like “maintainers wanted”; other meta discussions are preferably held in the Community Forum)

2.1 Augmenting labels for bug severity

  • :rotating_light: S1 (catastrophic - blocks development or testing, may impact more than 25% of users, causes data loss, no workaround available, potential chemspill although we forward the issue into our distinct process for security related issues if found to be so)
  • :zap:S2 (serious - major functionality severely impaired and a satisfactory workaround doesn’t exist)
  • S3 (normal - blocks non-critical functionality and a work around exists)
  • S4 (small / trivial - minor significance, cosmetic issues, low or no impact to users

2.2 Augmenting labels for any classifier

  • duplicate (duplicate of already reported issue; references and closes the issue to encourage discussion in one place)
  • documentation (requires a change in the docs; since docs tend to be neglected, a distinct label should bring more attention to keeping the docs updated)
  • up for grabs (a bug, enhancement or feature that is not in the works; once a feature is being actively worked on, it is expected to infer that from the comments and the label can be removed; the label will be reapplied after prolonged time of inactivity)
  • on hold (the issue cannot be resolved anytime soon because it depends on a 3rd party or another issue that needs to be resolved first)
  • won’t fix (the issue won’t be fixed; the reason for this should be clearly stated)

Remove all other existing labels (non exhaustive list)

  • discussion, because every thread is essentially a discussion and every discussion is either regarding a bug, an enhancement, a feature or a question.
  • in the works, because we can never be sure whether a feature is still actively being worked on and we rather have more people looking at an open issue than less (see “up for grabs” label); the current status of an issue should be inferred from the last comments, not from a label that is likely to be outdated.
  • good first issue, because it seems to be rare that someone picks up an issue just to make a PR; more often an issue is picked up because someone faces the same roadblock.
  • long term, because it is unclear what it means; fixing a bug is long-term when no-one is working on it; other issues that currently use that label are not parse server issues and would be better placed in the community forum; if an issue cannot be solved because of an external dependency then a distinct “on hold” label makes it more clear.
  • needs investigation, because an issue usually needs investigation and it is a temporary label that needs to be removed at some point anyway.
  • pr submitted, because a PR is referenced in the thread anyway and a submitted PR doesn’t say anything about it’s viability or readiness for merge (sometimes PRs are submitted even before opening an issue and asking for community feedback).

Framework

  • Bug severity (S1-S4) can only be assigned by a triage team to ensure proper classification of and attention to critical issues. This triage team will be formed from the core team members. The purpose of restricting severity classification is to prevent inflationary severity use and ensure timely detection of critical issues.
  • A bot notifies core team members when an issue is not classified (as bug, enhancement, feature, etc) after 3 days of being opened, to mitigate the risk of missing a critical bug.
  • A bot notifies core team members when an issue that is classified as bug does not have a severity level assigned by the triage team within 2 days, to mitigate the risk of missing a critical bug.
  • A bot adds the up for grabs label to issues classified as feature or enhancement after 30 days of inactivity.
  • Issue transfers between repos happen by team members who have write access to the target repo, otherwise by asking @parse-community/core-maintainers in the comments.
  • Adding, editing, deleting labels is restricted by guideline (cannot be restricted by repo permission).

Governance

  • Labels play a fundamental role in issue severity classification and can have a significant impact on the product, therefore any future changes of the labels or the framework governing their usage requires a majority vote by the core team members.
  • Addition, change and removal of labels, their usage or bot labelling mechanisms require consideration of the following gatekeeper questions:
    • How does the change improve the quantity / quality of issue handling?
    • How does the change benefit the maintainers / reviewers / authors / developers?
  • Labels are used (interpreted or applied) by virtually all active team members on almost a daily basis, therefore changes of labels and their usage requires a non-binding feedback opportunity for team members.

The feedback period for the suggested changes above is until Sep 30, 2020, after which the latest consensus will be presented to the core team members for a final vote. A majority vote by the core team members is required to adopt the changes.

2 Likes

Suggestion for improvement of:

  • A bot notifies core team members when an issue is not classified (as bug , enhancement , feature , etc) after 3 days of being opened, to mitigate the risk of missing a critical bug.

Change to:

  • A bot notifies team members with write access to the repository when an issue is not classified (as bug , enhancement , feature , etc) after 3 days of being opened, to mitigate the risk of missing a critical bug. After 6 days of being opened, the bot notifies the core team members. The duration times double for issues with the needs more info label. A team member can always elevate an issue to the core team to expedite issue classification, regardless of any notification intervals.

Reason:

  • As seen in this issue, we may be waiting for more info from the author to be able to reproduce a bug or classify an issue as a bug. If we pause the bot indefinitely, we may miss critical issues, so the suggested alternative is to extend the period.
  • To avoid flooding the core team with notifications from all repos, a 2 step notification process (repo team members - core team members) may better distribute the work load.

Hi mtrezza,

I like the idea of consolidating the labels for the github repos to have a common understanding throughout the repos.
I’m part of the .NET SDK team and wondered how we may cover additional information about target platforms or frameworks (e.g. Unity, Xamarin, .Net Standard) with labels or if we should at all.
Some of the issues in the .NET SDK repo are related to specific target platforms, although only few really come down to be bugs and most others are questions which should be referred to stack overflow or the community forum I found it helpful to have those issues labeled for categorisation of some sort.

I assume this is kind of a special case with .NET but how may this circumstance be handled with your suggestions, May the teams introduce additional purely informative labels for such things as affected target platforms?

Cheers
Tobias

@TobiasPott Thanks for your feedback.

Sure, we can deviate from the suggested standard set of labels if you think it makes sense for the .NET repo. To find out whether it makes sense I think the gatekeeper questions would be:

  • How does the label improve the quantity / quality of issue handling?
  • How does the label benefit the maintainers / reviewers / authors / developers?

We actually have a similar situation with the Parse iOS SDK which covers iOS, macOS, tvOS, watchOS. @noobs2ninjas, @drdaz, @Tom what are your thoughts on platform labels for the iOS SDK?

I think platform labels could be helpful - I’m not active enough with the iOS SDK to give a thoughtful opinion. Similarly though, I do find with the docs repo that tagging issues & PRs with the guide they correspond to.

I think the difference here is that a docs tag actually points to a particular file in the docs, which is helpful since the docs are not yet consolidated. In iOS SDK / .NET SDK, the sub platforms share a common code base, so I’d assume that an issue usually affects all sub platforms.

@TobiasPott Do you see any benefits in platform labels for the .NET repo?

@mtrezza Yes I definitely see benefits of platform specific labels for the .net SDK repo.
The .net SDK covers .NET as it is (.NET Standard/.NET Core) and Xamarin and Unity which both behave not identical to the .NET Standard framework and might not even behave completely identical when speaking of an app build with Unity targeting Android or iOS.
This led to some issues which occured only on one of all the possible target platforms. I would expect such issues to still occur in the future and assume having labels help distinguish them and their impact and provide info to other users and us maintainer where to look at for and what to consider for fixes (and perhaps allow us to pre-determine if and who may need to take a look on an issue due to more experience with the separate target platforms).

Although I’m not sure in which detail such platform labels might be introduced. If sth. like “Unity”, “Xamarin”, “Android”, “iOS”, “Windows”, “Linux”, “macOS” could be used and a combination of these will provide the necessary info (e.g. “Xamarin” + “Android” indicates an issue with Android apps based on Xamarin) or if they need to be more explicit. I would tend to former and combining them as required.

I think when looking for fixes, one would inevitably read the thread and the issue report. To make sure we have specific platform information, we can simply add an environment field to the GitHub issue template.

What do you think would be the benefit of not only having this info in the issue template but also in the Github issue list?

If we introduce platform labels, I would expect them to always be applied, which means it is an additional label to manage. Looking at the current .NET issue list, the platform labels seem to be applied inconsistently and the majority of issues doesn’t have these labels. What do you think is the reason for that?

You are right, the platform information could be put into environment information of an issue. I don’t have access to the settings and did not think about issue templates.

I think the inconsistency is/was caused by the vacuum of active maintainers on the .NET SDK repo since the last version 1.7 was released and alex picked up the rewrite and me working with the SDK who could look for the use and application of labels in new issues.
I also think that the lack of a template or a guide for creating new issues leaves the decision to the user to apply any label and skip or miss this step initially.

Now I’m unsure about the requirement of explicit platform labels. Perhaps it might become helpful to still distinguish between Xamarin, Unity and .NET Standard/.NET Core but that info might also be into the issues themselves instead of of labels.

I will open a PR with issue templates for the .NET repo for you to review.

I’d propose we add the platform question to the issue template and see how it goes. I am not sure whether essential info about the issue should we in the label, because historically we see that managing labels consistently can be a challenge, because it is purely administrative work that doesn’t help with solving the issue.

The challenge I see with platform labels specifically is that it requires a technical assessment and investigation to know which platforms are affected. If the platform label is assigned early in the process it Is more likely to be incorrect, if it is assigned way into the investigation the label may not be updated and be incorrect as well. Therefore I personally would not trust these platform labels in general.

I agree, it’s hard to establish an argument in favor of platform labels so far. I suggest until we can do that, we add it to the issue template and see how it goes. We can add them later on once we can determine their advantage.

Yes you are right @mtrezza . Using platform specific labels would rely on the reporting user and/or the maintainer to look after to be helpful in the later process.
It will be easier to go without them for know and see if we encounter specific lack of information in the future.
I’m looking forward to your issue templates PR.

1 Like

Suggestion for improvement of:

  • needs more info (more info is needed from author to classify the issue)

Change to:

  • needs more info (more info is needed from author or more investigation is needed by Parse team member to classify the issue)

Reason:

This effectively combines the existing labels needs more info and needs investigation. The reason is that the process is practically a co-investigation by author and team member.

PR as discussed.

I think it would be appropriate to move the deadline for feedback to end of September, as August may be a month where members are less active. @Tom would you agree?

Sorry I haven’t responded. Happy to push back to end of Sept.

Deadline has passed, next steps:

  1. I will update the labels with the comments in this thread soon
  2. Core team votes on final labels
  3. Implement the new labels in the Github repos
1 Like

Are we ready to vote on this yet? If so you could just just drop a poll in as a reply.

Also I think next step 2.5 should be setting up a bot to do the labelling reminders etc as discussed. Btw we have a peril server on Heroku which does the auto dependency merging. It could probably do what we want but I don’t know much about it. We can work out the specifics of that once core have voted.

1 Like

If you let me work any comments from the thread into the list at the top, then it’s ready to be voted on. Apologies for the delay, the last summer days got the best of me :slightly_smiling_face:

1 Like

The final draft is complete:

Final draft

Uniform GitHub labels

1. Initial status

  • no label (issue has not yet been look at by a Parse team member)
  • needs more info (more info is needed from author or more investigation is needed by Parse team member to classify the issue)

2. Issue classifiers:

  • :bug: bug (impairs a documented feature or a functionality that is not explicitly documented but one would strongly expect the general user to infer from a documented feature)
  • :rocket: feature (a new feature that does not exist yet)
  • :dna: enhancement (performance improvement or functional enhancement of existing feature)
  • :grey_question:question (not an issue but a question; should be in Community Forum but we can’t easily transfer, so we refer the author to the Community Forum)
  • :wrench: troubleshooting (not an issue but asking a code-level question; should be in StackOverflow; for simple issues we may give a quick tip / solution to the author, in general we refer to StackOverflow to educate the author about where to post code-level questions)
  • :speech_balloon: meta (issues regarding CI and tests or pinned calls for high visibility like “maintainers wanted”; other meta discussions are preferably held in the Community Forum)

2.1 Augmenting labels for bug severity

  • :rotating_light: S1 (catastrophic - blocks development or testing, may impact more than 25% of users, causes data loss, no workaround available, potential chemspill although we forward the issue into our distinct process for security related issues if found to be so)
  • :zap:S2 (serious - major functionality severely impaired and a satisfactory workaround doesn’t exist)
  • S3 (normal - blocks non-critical functionality and a work around exists)
  • S4 (small / trivial - minor significance, cosmetic issues, low or no impact to users

2.2 Augmenting labels for any classifier

  • duplicate (duplicate of already reported issue; references and closes the issue to encourage discussion in one place)
  • documentation (requires a change in the docs which are part of the current repo; this is not to be used for changes related to the separate docs repo; since docs tend to be neglected, a distinct label should bring more attention to keeping the docs updated)
  • up for grabs (a bug, enhancement or feature that is not in the works; once a feature is being actively worked on, it is expected to infer that from the comments and the label can be removed; the label will be reapplied after prolonged time of inactivity)
  • on hold (the issue cannot be resolved anytime soon because it depends on a 3rd party or another issue that needs to be resolved first)
  • won’t fix (the issue won’t be fixed; the reason for this should be clearly stated)

Remove all other existing labels (non exhaustive list)

  • discussion, because every thread is essentially a discussion and every discussion is either regarding a bug, an enhancement, a feature or a question.
  • in the works, because we can never be sure whether a feature is still actively being worked on and we rather have more people looking at an open issue than less (see “up for grabs” label); the current status of an issue should be inferred from the last comments, not from a label that is likely to be outdated.
  • good first issue, because it seems to be rare that someone picks up an issue just to make a PR; more often an issue is picked up because someone faces the same roadblock.
  • long term, because it is unclear what it means; fixing a bug is long-term when no-one is working on it; other issues that currently use that label are not parse server issues and would be better placed in the community forum; if an issue cannot be solved because of an external dependency then a distinct “on hold” label makes it more clear.
  • needs investigation, because an issue usually needs investigation and it is a temporary label that needs to be removed at some point anyway.
  • pr submitted, because a PR is referenced in the thread anyway and a submitted PR doesn’t say anything about it’s viability or readiness for merge (sometimes PRs are submitted even before opening an issue and asking for community feedback).

Framework

  • Bug severity (S1-S4) can only be assigned by a triage team to ensure proper classification of and attention to critical issues. This triage team will be formed from the core team members. The purpose of restricting severity classification is to prevent inflationary severity use and ensure timely detection of critical issues.
  • A bot notifies core team members when an issue is not classified (as bug, enhancement, feature, etc) after 3 days of being opened, to mitigate the risk of missing a critical bug.
  • A bot notifies core team members when an issue that is classified as bug does not have a severity level assigned by the triage team within 2 days, to mitigate the risk of missing a critical bug.
  • A bot adds the up for grabs label to issues classified as feature or enhancement after 30 days of inactivity.
  • Issue transfers between repos happen by team members who have write access to the target repo, otherwise by asking @parse-community/core-maintainers in the comments.
  • Adding, editing, deleting labels is restricted by guideline (as it cannot technically be restricted by repo permission).

Governance

  • Labels play a fundamental role in issue severity classification and can have a significant impact on the product, therefore any future changes of the labels or the framework governing their usage requires a majority vote by the core team members.
  • Addition, change and removal of labels, their usage or bot labelling mechanisms require consideration of the following gatekeeper questions:
    • How does the change improve the quantity / quality of issue handling?
    • How does the change benefit the maintainers / reviewers / authors / developers?
  • Label colors are chosen uniformly across repositories; the parse-server repository is the leading repo defining the colors for other repositories to adopt.
  • Labels are used (interpreted or applied) by virtually all active team members on almost a daily basis, therefore changes of labels and their usage requires a non-binding feedback opportunity for team members.

The following suggestions from the thread have been worked into the draft:

Suggestions
  • Platform labels (@TobiasPott): since we could not establish an concrete argument in favor of platform labels, they are not entered into the draft. They may however be added at a later point in time under the framework also suggested in the draft.
  • Gatekeeper questions (@mtrezza): to give some guidance for the future change / addition / removal of labels the gatekeeper questions established in the thread have been added to the draft.
  • Needs more info (@mtrezza): definition expanded to include info from author or investigating team member

@core-contributors can now vote on the proposal to:

  • adopt the final draft
  • and change the repository labels of parse server, parse client SDKs and their sub-repositories such as adapters or modules accordingly (special repositories such as docs or blog are exempt).
  • Aye :+1:
  • Nay :-1:

0 voters

1 Like