Maybe you are a new engineering leader at red-hot startup. The founders hired you on account of your previous experience at a successful tech company, they brought you in to take engineering to the next level. After a few weeks of onboarding, you now have a list of changes you want to implement. How do you find a way to propose that without making the old guard feel alienated from the process?
Or maybe you are part of the old guard yourself. You have shown interest in stepping up and leading the engineering team from a scrappy group of people working 7 days a week to a more mature organization. You were promoted to a position where you finally have the ability to tackle the root cause for the growing pains you all are experiencing. One question still remains, though: how can you make sure that your fellow engineers don’t feel that you are imposing your views like a tyrant?
Or it could be that those ideas aren’t even yours. You are a manager worried about the amount of technical debt and frequent production incidents caused by people rushing to implement their ideas withouth having them double-checked by a second pair of eyes. When you casually remind them about the benefits of collaboration, you hear about how they are are afraid that a reviewer will waste everyone’s time pushing for the perfect solution, and we need the first iteration of this thing out as soon as possible.
In sitations like these, you are usually asking yourself how can you foster a culture that is more accepting and kind towards change?
In my experience, one of the most effectives things one can do to achieve that is establish a structured process for feedback on ideas, designs, and architectures. Honoring a long tradition in software engineering, I call this a Structured Request For Comment (RFC) Process.
Introducing an RFC process
Your organization already has various formal and informal ways to share ideas, from formal presentations to casual chatter over lunch or beers. Something I have oserved in the various startups I have worked with is that thse channels tend to break down when the organization reaches something like 70-100 engineers. At this size, people still reach out for feedback from those who they know—for example people who have worked at the organization for a long time or maybe people who joined at the same time and bonded during onboarding—but these networks are more like cliques than peer groups.
This is when, as a leader in the engineering organization, I tend to establish the structured RFC process. RFC stands for Request For Comments. The term has a long history in engineering, but outside formal standard bodies it is normally used to refer to a document describing and idea, written by someone who expects feedback on it from their peers. This kind of interaction happens all the time amongst engineers, but I believe that a well-defined and structured process helps set expectations that is is an expected part of the engineering workflow. It also makes it easier for people to take part on the process, as they don’t have to second-guess if their opinions are welcomed or when to bother more senior people for feedback. Teams tend to use this process to gather feedback on the design for a new system, a strategy for upgrading shared libraries, new coding conventions, changes to the code review process, etc.
After introducing such process in various startups, I have compiled the lessons that my teams and I have learned into an RFC itself, so that the team experience it first hand while discussing if they should adopt it or not. You can find it in full as a Google Document here. Please feel free to copy this format, make whatever changes make sense and use it in your organization.
In the process described above, the author writes a document describing the proposal, following a template that aims at making sure that some fundamental questions are answered before inviting people to give feedback. They will then ask other engineers for written feedback, usually by sending an email to a well-known mailing list. People reviewing the document provide the author with their opinion, anecdotes from previous experience, and facts related to the proposal. This feedback is considered informational, meaning that the authors of the RFC are free to do incorporate it into their proposal or not.
There is no guarantee that the feedback will be ultimately incorporated into the proposal, but we don’t want reviewers thinking that they have wasted their time commenting on it. That is why the process described here requires the authors to acknowledge every piece of feedback given. The authors must also commit to revisiting their final decision at some point in the future, sharing the lessons they have learned.
We have recently introduced this process at Meetup, and in the first few weeks it was already clear that there was demand for something like it:
The document above should contain everything you need to start a structured RFC process on your own. The remainder of this article is an annotated version of it, adding some nuance and historical background that isn’t fully captured in the RFC. It adds some color and background on the key points of this process based on my experience implementing it at ThoughtWorks, SoundCloud, DigitalOcean, and now Meetup.
The annotated RFC
Authors: Phil Calçado To be reviewed by: 10/5/2018 Revisit Date: 04/17/2019 State: Feedback Requested
A header like this might look antiquated, but I find still incredibly useful. At a glance, this provides me with who the authors are—important for accountability, which we will discuss later—and a few important dates to keep in mind.
Most of this document was taken verbatim from the one we used to introduce the RFC process at Meetup. It is likely that your organization has some of the needs stated here, but you might want to be more specific about your needs.
A healthy engineering organization demands a culture of asking for and welcoming feedback on our work. In smaller organizations, sharing plans, designs, and decisions is much easier. As we grow, it has become clear that this organic process won’t suffice.
This paragraph acknowledges a common pain point in hyper-growth companies. When your team was small, there was a straightforward way to share ideas between engineering, product, and even founders—just have a conversation! As you hire more people, suddenly engineers find themselves with a feeling that we can summarize as “I don’t know what’s going on anymore.” While RFCs won’t solve all of your problems, it establishes a well-defined process to share and consume information about engineering decisions and ideas.
Currently, various teams already write down their plans and designs in documents that could be usually called an RFCs (Request for Comments). Without shared and clear guidance or process, these vary drastically in format, contents, and objectives. There is also a lot of variance on how these are advertised to other engineers who would be good candidates for feedback givers. At the moment, there is no easy way for an engineer to know what topics are being discussed at a given time, or how could they give input on such decisions.
Your team is already sharing information one way or another. Unfortunately, the lack of a standard for how and where to ask for and give feedback makes it such that these documents often don’t reach people who would be the most helpful or impacted by it in a timely fashion.
At Meetup, for example, our Web Architecture team was planning to build a GraphQL-based API to boost mobile productivity. They had a meeting with our mobile team to share the good news and talk about the project; the expectation as that the mobile team would adore the idea. Instead, the GraphQL proposal was received with confusion voiced as questions like “So.. does that mean we should stop our refactoring of the HTTP clients?” It turns out that the mobile engineers had decided to solve the productivity problem themselves by changing their HTTP client to make it super-productive to use our REST API. They had done some amazing thinking about how to improve the current state of things, and even shared the idea as an RFC. Unfortunately, this RFC was never shared with any other team, and people who own the API platform had no idea that this initiative was going on.
There is also no clarity on collaboration versus decision-making. An RFC process, by definition, is meant to collect feedback on a proposal. There will always be different opinions, and we must encourage people to expose their ideas and have them debated. Nevertheless, we operate in a very competitive landscape and we have no time to waste in analysis paralysis. We believe that speed of iteration beats quality of iteration, and to iterate quickly we absolute clarity about who has the decision-making responsibility on a proposal.
One of the most important aspects of any change management process, especially when trying to increase transparency and engagement, is to avoid design by committee. Whatever feedback gathering process you end up adopting, you must make sure that there is an explicit acknowledgment of who is the decision maker, the one accountable for the outcome and with veto power over it. Feedback givers must always keep in mind that their opinion will be taken in consideration, but there is no guarantee that they will be incorporated into the proposal.
Moreover, existing RFCs and similar documents often get into too much detail about the “how” and not enough on the “what” of the proposal. There are many different ways to materialize an idea, and implementation details are better left to be decided by those who are actually doing the work.
You want to both gather feedback from a diverse audience and make sure that the reviewers aren’t missing the big picture and focusing on implementation details. To achieve that, you need to make sure that your document doesn’t spend too much time on distractions and focuses on the most important aspects of the proposal. When I am coaching managers and leaders, I tend to summarize this as don’t invite people to conversations you don’t want to have with them.
One example of this going bad was when, at SoundCloud, my Platform team published an RFC describing the changes that application developers would face as we moved from our own datacenters to the cloud. The document was full of important and potentially contentious information about how application developers would have to change their mindset about latency, availability, and even simple things like trusting that there was a durable file system in their servers. Nevertheless, the one paragraph that everyone in the company decided to comment on was one that causally mentioned that we would write some tooling in Python because that is the de-facto canonical AWS SDK. This was an implementation detail, completely irrelevant to anyone who wasn’t in that team. Still, they were introducing a new programming language and that sparked a heated debate that went on over the weekend. At SoundCloud, our teams had autonomy to decide whatever tools made sense to them, and the mistake this team made in the RFC as to invite an engineering department full of very opinionated people to give feedback on their programming language preferences.
In summary, we need a clear and simple process that allows people to share their ideas, receive feedback on them, and defines how the decision-making process works.
This sentence summarizes everything that this process tries to address. People need a safe space to get feedback on their ideas, and feedback givers must know how their input will be used.
The recommended approach to fulfill the needs presented in the previous section is a structured RFC process. In this document, a person or group of people will author a document describing a proposal and asking for feedback on it from the rest of the organization.
As mentioned before, in my own experience a well-managed RFC process can address the needs stated previously. The trick here is that the term Request for Comments means different things to different people. To make it clear what we mean by ot, this section tries to be prescriptive and opinionated about how to build such a process.
Feedback vs. Approval
The RFC process is a tool that can be used during the decision-making process, and everyone is encouraged to share rough and early ideas and proposals as RFCs.
The more polished a document looks, the softer and less impactful reviews tend to be. When facing a well-written document, our brains enter into a sunk cost fallacy mindset, thinking “Ugh, I think this is a horrible idea, but this person has put so much effort into it…“. This leads us to focus on smaller, irrelevant details instead of addressing any elephants in the room. It is generally easier to give more candid and useful feedback on something on its earlier stages, maybe just a list of bullet points and a back-of-the-napkin drawing.
As a leader, it is probably common for people to share with you their plans and ideas over Slack, email, or meetings. When this happens to me, I spend some time listening and asking some preliminary questions about the proposal, but soon enough I say “This sounds interesting, do you mind putting it in a two-page document using the RFC format?”. I tend to work with them for a few iterations on it, antecipating questions that I believe will come from the wider audience, and then ask them to send it to the wider group for peer feedback. If the person is resistant to sharing it widely, I coach them into sharing it first with people they are more comfortable with, and widening the circle until the whole organization is engaged.
Even if the proposed change on the RFC is extremely well-received, it doesn’t mean that it is approved to be worked on or that it will be prioritized. Authors of the RFC must make sure that they have whatever approval or sponsorship they need from management, leadership, stakeholders, collaborators, and their own team before any actual work is done.
The biggest caveat of a peer review process like the one described here is that just because something has gotten good feedback and people may be super-excited to see the change implemented, it doesn’t mean that it is the right thing to do, or that it should be prioritized.
This is where the push for everyone to use RFCs and to publish early work can backfire. It is not uncommon for engineers to try and use the process as a way to sell an idea that hasn’t been approved by their stakeholders or managers. They try to gather support from the other members of staff, transforming something that should be a purely technical matter into a popularity contest.
That is why one needs to make it absolutely clear that RFCs are not a decision-making process. RFCs are merely for feedback on a proposal, and there is no commitment that a well-received RFC will be implemented or that a poorly received one won’t.
Just like with any other engineering problem, it is also helpful to be explicit about any constraints before asking people to find a solution. One way in which I have done this in the past is taking responsibility for writing the Need section of the RFC. You should use that as an opportunity to make sure that not only the technical and functional aspects of what is needed are expressed there, but also explicit acknowledgment of the other constraints one is under. For example, you should make it clear that the desired solution needs to be delivered within a given timeframe, or under some budget.
RFCs are expected for any change that extends beyond a team or department, as it gives the people who would be affected an opportunity to learn more about the change and give feedback.
Thoughout the document, I try to reiterate over and over the idea that RFCs should be used at any time whenever people can benefit from feedback on an idea. This section takes a more prescriptive stance, explicitly setting the expectation that RFCs will be used when a change impacts more than just a team or any other cohesive group of people. This draws a line on what autonomy means in practice, setting a safeguard that is triggered when a team’s decision might impact other individuals.
In my experience, enforcing this rule is seldom necessary. In fact, it is more common that the problem is other around: is not that people need to be told when to write an RFC, they need coaching identifying when this is not the best course of action.
This might sound conter-intuitive. If we are so convinced that the RFC process brings value to the organization, why don’t we want to have RFCs for almost everything? In my experience, there are two main problems to this approach.
The first problem is that it can create an avalanche of RFCs that spam our inboxes. As discussed in the introduction, the RFC process here won’t scale very well if there are too many changes to be reviewed at a given point in time. People get overwhelmed and will quickly disengage from the process.
A second and more dangerous problem I have observed is something that can be mapped to the phenomenon called diffusion of responsibility. That is when engineers start using the RFC process as means to protect themselves from any bad consequences. “Everybody reviewed it and gave their ‘ok’“ feels like an efficient shield to use when asked hard questions. Autonomy doesn’t work without accountability, and if your engineers are using RFCs as an ass-covering tool you probably need to revisit how your culture deals with failure.
One way to tackle this problem is with coaching. I expect my technical leadership, e.g. tech leads, architects, Staff/Principal engineers, etc., to invest a lot of their time in reviewing and helping prepare RFCs. To me, engineering leaders do their job when they are helping others with their RFCs like this, not when they are writing RFCs themselves.
Authorship, Accountability, and Responsibility
The authors of an RFC can be an individual, team, or any other group of people. Being an author means that a person or team sponsors the initiative and are accountable for it.
I usually ask teams to assume collective ownership of the RFCs they produce. While it’s normal or one person or maybe a pair to take the lead on responding to feedback and managing the process, the ownership of an RFC should be treated the same way as they do with code. Every now and then an RFC would be owned by a single person, but this shouldn’t be the norm.
While non-authors may be responsible for implementing the results of an RFC, its authors are accountable for it, as per the definitions below:
The main difference between responsibility and accountability is that responsibility can be shared while accountability cannot. Being accountable not only means being responsible for something but also ultimately being answerable for your actions. Also, accountability is something you hold a person to only after a task is done or not done. Responsibility can be before and/or after a task.
This section makes it explicit that, while the authors might not be the ones actually doing the work of implementing the change, they are accountable for making sure that the RFC process is executed well and, more importantly, for the change being proposed.
The concepts of accountability and responsibility are fundamental to a healthy organization and deserve their own article. If your organization hasn’t yet developed a good understanding of what these terms mean, you migh want to expand this section and include some more details.
RFCs must be sent to a mailing list called email@example.com. All engineers are automatically part of this list, and people from other groups are welcome to join and participate.
These days many organizations are trying to completely switch to real-time communication tools like Slack. I personally prefer an asynchropnous tool, such as email, for the RFC process. I have also seen teams using Github issues and wiki pages for this.
Comments and feedback should focus on the technical content. As long as they don’t impact the content, collaborators should avoid commenting on formatting, writing style and other maybe relevant, but not critical aspects. Such comments can be sent directly to the author to avoid polluting the comment and storming people with notifications.
As a Brazilian citizen who has been working in English-speaking environments for more than ten years, I know first-hand the challenges of having English as a Second Language. While we always welcome feedback as a way to get better at expressing ourselves, and RFC isn’t the best forum for it. It is perfectly fine to ask authors and feedback givers to rephrase a sentence that is a little confusing, but please refrain from using this interaction as a way to find teaching moments.
Similarly, don’t be obsessed with formatting. It is great when RFCs look the same, it makes it easy to quickly parse and check if you shoukld invest time on it, but it isn’t mandatory.
Authors must address all comments written by the deadline. This doesn’t mean every comment and suggestion must be accepted and incorporated, but they must be carefully read and responded to. Comments written after the deadline may be addressed by the author, but they should be considered as a lower priority.
This goes back to the our desire to make sure that people who have invested their time inr eading and commenting on the document don’t feel like they have wasted their time and that their opinions aren’t even going to be taken into consideration.
Something to be aware of is that, in my experience, platforms with in-line commenting such as Google Docs or Github Pull Requests can create a habit of commentiong-as-you go. This can be extremely annoying to RFC authors, as they keep receiving notifications and scriolling through comments that are answered in the document itself if the reviewer just read a few paragraphs more. There are a few technology options that can help with this, such as Github Reviews, but to me this is a behavior better addressed by feedback and coaching.
Every RFC has a lifecycle. The lifecycle has the following phases:
- Draft: The authors are working on the RFC before asking for wider feedback
- Feedback Requested: The RFC has been sent to the mailing list is waiting for feedback from stakeholders
- Active: The deadline for comments on this RFC has passed and the authors have decided to go ahead with it
- Abandoned: The authors have decided not to move forward with the changes proposed in this RFC.
- Retired: The changes proposed on this RFC aren’t in effect anymore, the document is kept for historical purposes
The lifecycle of an RFC is meant as a tool that people can use to enforce a window in which feedback is expected and create a discrete point when the authors can say “Thanks everyone” and move on, either implementing the changes or deciding that it wasn’t such a great idea.
The draft stage is aimed at creating a safe space for people to gather early feedback on an idea. As mentioned before, engineers can be really resistant to sharing half-baked thoughts until they can defend their opinions and designs from criticism, and this might take a long time. People seem generally more comfortable with sharing something in its early stages if is clearly marked as a draft, though, and this can lead to faster feedback cycles.
I would generally recommend that once an RFC moves away from Feedback requested, it is considered a historical artifact, if not discarded completely. RFCs aren’t great as documentation, once the feedback period is over I usually ask the authors to document any relevant parts somewhere else like a wiki or even a different Google Doc.
Each RFC has a revisit date, by when the authors will update the mailing list on what they have learned since the feedback phase. This is a natural point for an RFC to be retired and a new approach proposed.
The most important lesson that I have learned as a change agent in organizations is that people are much more welcoming to change if they know that the decisions and assumptions will be revisited at some point in the future.
I love the way that Linda Rising describes this as a Pattern in her great book Fearless Change: Patterns for Introducing New Ideas:
You’re getting worn out as you attempt to address the concerns people have about the new idea because it doesn’t look like the questions and objections are going to end anytime soon.
There are people in the organization who are expressing an endless supply of objections to the new idea. It would be a daunting, or even impossible, task to try to ease everyone’s worries before the new idea is adopted.
Fear is often what keeps us talking and questioning but stops us from doing anything. However, even though people may be fearful of change, they usually love to experiment. Change means risk. An experiment is something you can undo and walk away from when you are all the wiser.
Ideas that can be tested on an installment plan are generally adopted more rapidly than those that are not. If people are offered a trial period, they will have the opportunity to experiment with the innovation under their own conditions. This is likely to ease their uncertainties and give meaning to something that was previously seen as only an abstract idea.
It’s more effective to let people convince themselves through sight and touch than to try to convince them with words and logic. For “test purposes” is a convenient label for temporarily transferring “unacceptable” ideas into an “acceptable” category, until such time that the idea can gain the persuasive power to become part of the established way of doing things.
Suggest that the organization, or a segment of the organization, try the new idea for a limited period as an experiment.
Having an expiry date and a commitment from the authors to revisit the decision is one way to implement this Pattern in your organization.
The RFC document itself is where comments and decisions are recorded. It should be a Google Doc, and everyone should have access rights to comment on it.
As mentioned previously, this process can be implemented using various publishing software, from Google Docs to Github Pull Requests. I personally like the idea of Google Docs because it makes it easier to apply the same RFC process outside engineering. Say, for example, that you want to propose a change on the job description for engineers in your organization. If you use a tool familiar to your HR folks you can keep the conversation in a single document, instead of having to translate back-and-forth between what engineers are giving feedback on and an endless email thread with your People Team.
A good RFC will describe the scope and the approach. It should not contain a list of specific tasks or project plan.
This is a soft requirement, trying once more to reiterate that the what is often more important than the how for an RFC. It is perfectly fine to ask for feedback on a project plan, though, but I would suggest that the authors try and divorce the feedback on the objectives from the discussion about the project plan, it schedule, staffing and resources—the latter should derive from the former once that is established.
To avoid overloading the document with implementation details, RFCs should follow the Stanford Research Institute’s NABC model, making sure that they cover four points:
An NABC comprises the four fundamentals that define a project’s value proposition:
- Need: What are our client’s needs? A need should relate to an important and specific client or market opportunity, with market size and end customers clearly stated. With DARPA, for example, we are required to state a critical Department of Defense (DoD) need. The market should be large enough to merit the necessary investment and development time.
- Approach: What is our compelling solution to the specific client need? Draw it, simulate it or make a mockup to help convey your vision. As the approach develops through iterations, it becomes a full proposal or business plan, which can include market positioning, cost, staffing, partnering, deliverables, a timetable and intellectual property (IP) protection. If we are developing a product, it must also include product specifications, manufacturing, distribution and sales. DARPA usually demands paradigm-shifting approaches that address a specific DoD need (e.g., a 10-times improvement).
- Benefits: What are the client benefits of our approach? Each approach to a client’s need results in unique client benefits, such as low cost, high performance or quick response. At DARPA, the benefit might be an airplane that turns faster, goes higher, costs less or is safer. Success requires that the benefits be quantitative and substantially better - not just different. Why must we win?
- Competition/alternatives: Why are our benefits significantly better than the competition? Everyone has alternatives. We must be able to tell our client or partner why our solution represents the best value. To do this, we must clearly understand our competition and our client’s alternatives. For a commercial customer, access to important IP is often a persuasive reason to work with us. At DARPA, our competition is usually other research laboratories and universities across the United States. But, whether to a commercial or government client, we must be able to clearly state why our approach is substantially better than that of the competition. Our answer should be short and memorable.
The NABC format was introduced to SoundCloud by Gavin Bell, who has learned about it during his time on research labs. I was very skeptical of it at first, but nowadays it is my go-to format for proposals.
One of my favorite features of this model is the requirement that authors give some thought to alternatives and how they compare to the proposal. Something I like to enforce myself is that every RFC must consider the alternative of doing nothing. Every change requires investment of time, energy, and resources, and before implementing anything new we should consider what happens if we don’t do anything a all.
The first significant benefit of the approach described above is making a clearer distinction between decision-making and feedback gathering. With a clearly appointed accountable team, we can create a disagree and commit culture. We will carefully hear all positions and reckons from everyone, but ultimately a decision will be made by a specific person or team. Once the decision is made, everybody, irrespective of any differences in opinion during the RFC process, will commit to implementing and championing the decision. If it turns out that the decision wasn’t a good one, the revisit date on the RFC is there to make sure another discussion will be held in the near future.
Another important benefit of the proposed RFC process is openness. We have fantastic engineers, and we need to use our collective knowledge as leverage. None of us is as smart as all of us. To make collaboration work, we need to make it easy for all engineers to see what RFCs are being proposed and we need to make it a safe environment to collaborate, where comments focus on factual benefits and tradeoffs.
The NABC format is an industry tool used for making structured ‘pitches’. Using this tool will likely lead us to discuss the what without losing ourselves in the ocean of technical detail.
These paragraphs summarizes a lot of what I have discussed in this annotated version, but in a concise way aimed at the reviewer. I find myself referring back to it a lot when people start off-topic discussions on RFCs.
Competition (or Alternatives)
We should consider the option of not making any change and keeping the ad-hoc model we currently have for RFCs.
The main issues with this option were described in the Need section of this document. Unless something changes, the problems there will remain.
The “Do Nothing” option for the RFC process is highly contextual, but something that I believe most organizations will face is that, in order to keep communication manageable, people will either communicate in silos or stop discussing their ideas altogether. Even if you don’t adopt this process in particular, you should consider implementing an alternative.
Adopt IEEE RFC Model as-is
Although any collaborative development process will have feedback as a core component, the name RFC was made popular by the process used by the IETF to document fundamental standards for what eventually became the Internet. We could follow the IETF RFC model, and maybe even require authors to use terms like MUST, SHOULD, and MAY as formally specified by RFC2119 to avoid ambiguity.
The main reason to avoid this style is that IETF RFCs have evolved into “the Internet documents of record”, containing “very detailed technical information” about standards that browser vendors and network middleware need to implement. These documents will impact the whole industry and hence warrant a complex publishing workflow. The process we propose in this document, on the other hand, is about putting forward an idea as early as possible and receiving feedback on it by a wide audience. With this goal in mind, a less formal process like the one described here is preferred.
I’ve always been fascinated with RFC2119, so much that more than ten years ago I used it as a model when writing one of the first unit testing frameworks available for Clojure. After a few attempts at using the “official” RFC framework, though, I have found that even if you simplify the workflow it is very hard for people to be productive when bounded by it. Moreover, there is often no need for an RFC to be so precise, the authors are often the only people implementing the change and reviewers will benefit more from a more fluid, conversational prose than focusing on strict use of keywords.
Use Architecture Decision Record
Michael Nygard published a model to document and manage change in software architecture called Architecture Decision Record (ADR). Its motivation, format, and lifecycle are very similar to what this document proposes.
Nygard’s model is specialized in software architecture work. This is reflected in its usage of engineering tools such as repositories and Markdown files, which only make sense in a software project. We want the RFC process to be a tool useful in areas other than software development, which makes harder to implement some of the more specialized areas of the process. Nevertheless, ADRs can be used together with the RFC process described here when developing software systems.
I haven’t personally used ADR as proposed by Michael Nygard, and I am very interested in hearing experience reports from folks who have tried it. At the moment, I am not convinced that it is a good replacement for the RFC process described here. People often bring it up when reviewing the RFC process, though, so I wanted to address it from the beginning.
- 11/20/2018 - First published