CTO is pushing for trunk based development, team is heavily against the idea, what to do?

Trunk based development doesn't mean you're just committing everything directly to main. You can still use branches and PRs. The idea is simply that you don't have long lived branches. This is actually smart development, as the older a branch is the more the entropy increases and merges can become very difficult or nearly impossible in some cases, if enough has changed, particularly if there's cross over requiring resolving conflicts.

If your team is resistant to it, they either don't understand what it is, or they're doing things they shouldn't be anyways. You should be iteratively developing features, and merging them in quickly, as standard practice. If you're worried about something not ready to actually go live, that's what feature flags are for.

Can you ask him what he’s trying to accomplish by moving to trunk based development and pair programming? Can you provide alternatives to accomplish his goal? 

I did 100% pairing (XP) for awhile. Got exhausting. Literally left the company because of it. Btw, it didn’t make me a better engineer. When I was working with someone better than I was, I had a hard time keeping up. I wasn’t doing any critical thinking; rather, I was just trying to understand at a high level what was happening. I was not catching bugs. And when I was paired with someone less experienced, it felt slow and excruciating. I got way better after I left for a job where I could work at my own pace on interesting problems.

Don’t get me wrong, pairing ad hoc can be really valuable. But a draconian process doesn’t make bad developers good. Instead, hire good programmers, establish good standards, and encourage a quality-minded culture. I think it’s just easier to prescribe a process than it is to teach a philosophy, so that’s what managers do.

Trunk based development is unquestionably a good practice, however, your CTO needs to realize that one does not simply "do" trunk based development. There's a whole lot of other organizational capabilities that must be in place in order for trunk based development to be done in a way that increases velocity rather than the opposite. More specifically it's imperative for trunk based development that your team has the capability to decouple code releases from feature releases, and it sounds like you're quite far from that right now. Do you dark launch code? Do you utilize feature flags? Do you have automated testing? If not, you're miles away from successfully pulling off trunk based development.

As for pair programming, I think it's criminally underrated, but it also shouldn't be strictly mandated as an all day every day working style. That's pretty extreme.

It sounds to me like your CTO means well, but isn't enacting large organizational changes particularly tactfully, and it also sounds to me like your senior devs aren't particularly strong leaders and might be a bit lacking technically if they think trunk based development is inherently risky. I would expect the senior most ICs under the CTO to basically explain what I did that there's a whole maturity model that needs to be walked before trunk based development, and be able to get to the bottom of why the CTO wants pair programming so badly and work backwards from the goal instead of accepting a solution without knowing the problem.

You might find that trunk based development is closer to your way of working than you think. It doesn't mean to commit straight to the trunk/main/master branch. It's just that the branches have fewer change and are shorter lived. For example, one feature might be broken up into a pull request to add a column to the database, another that makes the backend use it, another for some refactoring, and a last one to update the frontend.

You can still create a branch per pull request, get a code review, use feature flags, run all the tests, and maybe even have your own manual QA gate before merging into the trunk and deploying.

Pair programming, that's an entirely different conversation depending on team/company culture and preferences.

I was working as a consultant with a Fortune 100 consumer brand, about as famous as such brands get. One of their tech leads was hell-bent on doing trunk based development. But they also had no internal review environment. So he literally wanted design and brand teams to review things in production but have it all sequestered via feature flags. Also, he had never actually run a project that way, it was just the new cool thing.

Pure madness. I talked them out of it but it didn’t help internal relations. So keep that in mind.

The problem with all of these “old way is trash, silver bullet is X” approaches is that, even if right, the tooling is years behind. And abandoning branches means you’re losing an entire ecosystem of proven tools and techniques.

Re: full-time pair programming, I would quit if anyone asked me to do that. And no, CI / CD does not require that in the slightest.

But you may be able to torpedo that by pointing out that LLMs are basically the new pair programmers. Put another way, use his shiny-new-thing obsession to make the unpalatable shiny new thing look old.

Or… just hang loose and watch him loose the the trust of the business after he makes the production systems unreliable, alienates the developers, and the executives see programmers killing time like cops on a stakeout.

For context my team looks after a critical area of our platforms (the type where if we screw up serious money can be lost and we'll have regulators to answer to.

Plenty of teams do and they do trunk based development. I'd want to hear more about your team's usage of git flow before rending a final opinion, but I frequently see teams that use it without actually really doing dedicated scheduled releases or full release testing... and that's kind of defeating the point of the branching model. Can't know for sure, but it smells like you guys are using it in an atypical way.

In fact, if your team is has good enough testing practices (ideally automated ones), trunk based development tends to be faster and less painful model for getting work out the door. The only sensible pushback to my mind to not be doing trunk based development would be that the act of releasing/testing requires so much dedicated coordination or time that something like git flow's release branches as an interim make a ton more sense for your team.

It's hard to know without more information, but honestly, I think you might want to hear out your CTO on both of these issues. I'm not hearing a particularly compelling argument from you or your team about why the git workflow change is bad, just that you guys don't like it. In fact, that you have so much automated testing to me begs the question of why you wouldn't be operating that way.

How much of this is really just getting lost in translation? Can you elaborate more or your team's practices in this area.

Way back when, I was quite against trunk based. After doing it for a few years now, it’s hard to go back.

Dont resist and be open to experimenting, then measure the results and revisit. You might be surprised, or not. But at least you tried.

There might be some pre-requisites to list off before going full trunk based though.

Trunk based dev is 100% the best strategy, when combined with effective flagging and other release processes that embrace it.

Pair programming 100% of the time? Lame as hell, leave me alone, I get my best work done after hours anyways. I'll pair program like 5-10% of the time, when the situation calls for it.

Edit: I work for a company that does both, the teams that do gitflow are always fucking up, the teams that do trunk based are also fucking up, but in way smaller chunks that are easier to manage.

I.e. in a gitflow team, someone sits in a branch for 1-2mo, they merge up daily, then when a release branch is made, 3-4 of these 2mo old branches smash together like semi-trucks on the highway. But the release schedule is fixed and everyone is scrambling to clean up the mess. Combine that with varying Beta/GA feature sets, and it's a fucking nightmare.

The trunk based teams are aiming to get their shit into trunk, if they don't get into trunk when a release is cut, they make the next one. There is no rushing the release, shit makes it or doesn't. And if it misses it by a day it can still be cherry-picked to the release or delivered to trunk, it doesn't have to wait in a branch for another 1-2mo for the "next release".

Unless you outrank the CTO then your only option is to try to convince them otherwise and, failing that, do as they say. I’m sure you’ll get lots of advice here about leaving. You’ll have to decide whether this is a hill worth dying on.

Detailed responses here are really good, so here's a more meta response:

Your team need to talk to the CTO about their need to feel safe. Trunk-based development (tbd) can absolutely be done keeping things very very safe, but it has to be worked out. As /u/spoonraker says, one does not simple do tbd - not without a plan. How will code reviews happen before release? How will tests be run? How will the manual QA happen? It can all work, but you have to talk to the CTO about satisfying those needs that your current process satisfies.

And, on the other side, any "senior" dev who thinks tbd is 'madness' is probably one of those 5yoe "seniors" I keep hearing about around here. It's pure ignorance, and there's lots of places to educate oneself on the practices of tbd (ie, Dave Farley's youtube channel, also his books, Jez Humble's books, the Accelerate book, Kent Beck etc). A lot of good stuff there!

You dont need full pair programming to achieve CI/Cd and it sound exhausting AF.

Gitflow with feature branches IS trunk based development, the scaled version:

https://trunkbaseddevelopment.com/

So what does your team and CTO both think need to actually change?

full time pair programming

RED FLAG.

Pair Programming is a tool in an engineer's toolbelt, not something to be dictated from someone who doesn't have to do it.

When I worked on a team that was 100% pairing, it was emotionally draining. I'd end my days with no desire to do anything because I was spent talking to and debating some of the absolute stupidest stuff with another engineer.

It doesn't reduce bugs. It doesn't make better engineers. And it certainly doesn't get things done faster.

The pair programming sounds like a waste of time and unpalatable. I would just do it for a bit then document the issues and give feedback. The maybe stop doing it or more likely negotiate to bring it down to a low amount.

Document the risks of making this change in your current environment. Document the requirements for successfully making this change. Ask your CTO what problem he’s trying to solve. Suggest better ways to achieve the same result with less risk.

Pair programming full time is a horrible idea. You're going to burn people out.

Trunk based development can work, but the team(s) need to be on board.

Your CTO is a dipshit.

My suggestion?

Just keep doing things your normal way. If it gets brought up, "oh, it's our next priority"

New CTO coming in trying prove useful, playing stupid games. 100% pair programming has its uses but they are for specific scenarios: otherwise you end up having half the work done.

I've done it in the past, and it was a hot mess tbh.

Forced paired programming is the biggest waist of resources and can be horrible experience when the peer has a poor command of English. I agree, it's exhausting and deprives of yoiu of learning and discovering experiences. It takes 2-3 times longer to do a task that normally can be done by a single dev. I took quit a job because of this forced policy. Code reviews work just fine and everyone can work out their own comfortable pace.

I think your CTO is too in the weeds with the actual processes the teams are utilizing and should be way higher up the stack.

Pairing sucks ass. Wouldn't do it full time and I suggest you fight it.

Trunk based is a great practice, the issue seems to be that y'all aren't feature flagging features and maybe the code base is not in a place to allow that type of work, yet. Raise the based points and maybe work towards that first before you all in for trunk development

For what it's worth, research suggests that Trunk-Based Development absolutely improves software delivery performance (as in, you can deliver more frequently and with better stability). That same research program, however, has found that Trunk-Based Development increases burnout (which is not a good thing), and is thinking about removing it from their core model.

My own thought is that these practices (Pair Programming and TBD) are good things, but can't be implemented in isolation. You need a whole bunch of other stuff to make it work (such as continuous integration, unit tests, acceptance tests, etc). Also there are several flavors of trunk-based dev some of which make it easier to ease into.

If you're in that group of sites that trusts all commits in master, uses pushes to master to trigger deployment, and so on...

I've seen many occasions where what was pushed to master failed to deploy correctly, or fully, leaving commits in master that need to be marked after-the-fact as unsound, and skipped from then on for git bisect and the like, not to mention leaving a set of hosts/VMs that were load-balancing the service with conflicting versions of the software. I have not yet been in a company that had any kind of process around what to do when the illusion of master's inviolability faceplanted. I recommend looking into git notes for ways to add meta data to commits without changing them, so that this kind of information can be tracked and used by automated systems.

Pair coding is exhausting for most pairs, so mandating it will shed staff and generally make most devs unhappy.

"trunk-based" wasn't fully defined, but having a single clear line of commits on the trunk, frequently pushing modular commits (where each commit is sufficiently self-complete to be harmless while the rest is being worked on) of partially completed features into the trunk, and not having the hairball of multi-year-old branches with two-way merges, are all good things that do fit very well with continuous integration. CI does best when any code being worked on in a branch has been updated to be a descendant of whatever the current trunk branch is with git rebase. Not that I'm not saying which branch that is, since that depends on the devs, whether the QA team is separate, and some other things.

The core failure of the gitflow use - in relation to CI - that you describe with features not merging in until proven complete is that at the point of merging, any number of new integration bugs may arise. Whereas, if devs are all continuously pulling in changes from the trunk head beneath their own work, far more integration-type bugs are likely to be found before the final merge. I've seen developers at these moments try to do some cowboy hack in the merge commit itself, making it no longer a clean fast forward and potentially complicating some operations, and I've seen the worst things happen when that big merge happened into master and trigger a deploy before all the integration bugs were found. So those are fun.

Dude I get tired after more than an hour of pair programming. All day every day would destroy me.

forced pair programming is absolutely brainless. One time I worked at this place, they wanted to "live code" as a group in vscode.... I don't even use vscode -_-.... the experience was anxiety-inducing AF, they didn't care, assholes.

Oh god, this was my first job. 1 lead who spent all day hiding alone in a closet. 6 juniors. Trunk-based development. No tests. No tests at all. Everything was just broken, all the time.

One dev came in late at night to 'work flexibly', altered the date schema to another format - like integer or something, they thought date looked messy - and actually got the app in a state it wouldn't even build for about four days while various others had to step in.

It was complete chaos. I left after six months because I was concerned all I'd learn were bad habits. It was like the giant buckets of 'punch' people made at house parties in my teens - chuck it all in, brilliant, it sort of 'works' and gets the job done, but you'll lose a ton of braincells, hours and probably days of time and might sustain permanent damage.

Now I'm not saying I haven't met people in my career I would do trunk-based with, with the right environmental setup. But they are very few.

And a quick note on pair programming. As a junior, truth is a lot of the time the 'senior' just wanted to do the work as quickly as possible and never let me have a go. Best I could hope for was them narrating what they were doing as I just watched. They'd cringe if I asked to write a damn line. One just stuck his headphones on and ignored me.

When I began to learn how to focus and get into a flow state, pairing would ruin that. So it took me a lot longer to learn to focus.

Again I'd still do it today, with certain people, on certain tasks, for a certain amount of time, but for some people and some tasks it's just not a good use of everyone's time.

I would absolutely leave a company mandating pair programming

It sounds like your CTO wants to improve cycle times, improve knowledge / awareness of features / code bases and thinks the patterns / practices he’s putting forth won’t significantly raise risk.

It sounds like the teams are fearful they’ll be responsible / accountable for screwups, intentional or otherwise, and also prefer working independently

Trunk based development isn’t a hill to die on. You can still manually gate deployment to prod until automated testing is in place

Pair programming, depending what it entails could slow productivity / reduce effectiveness. The only way to know is to try.

I’d message to the team that they aren’t the accountable ones, to step onboard and prove / disprove their fears and help mitigate them

The timeline is insane.

We commit to repos that are contributed to by multiple teams and basically use a simplified version of Gitflow with feature branches merging into master only when fully reviewed & tested and considered prod ready. 

FYI this is called GitHub flow.

to enable CI/CD

You need to enable this tooling before you do trunk. Your PR checks need to run unit & integration tests, you need to perform static analysis etc. 

More substantial is you need to have feature flags. Trunk only works effectively if you are doing a feature canary in your production environment otherwise you are just releasing untested code. 

You also need to have a credible strategy for how you quickly deploy a diff. If the duration between diffs is lower than canary time you need to figure out how to stack/combine diffs so you don't end up with an infinity queue for deployment.

This usually takes about 6 months if you know what you are doing. Lots of tools and processes to teach.

Like it or not, it sounds like it’s coming. Start planning what you’d need to put in place to make it work. Eg analysis for bugs found in manual testing to understand why they couldn’t be found by automated tests, re designing parts of the system to make feature flags work etc. Then present it to the bosses, get the work prioritised and start working through them.

Hopefully either the amount of work will put them off or it will take you so long so do the work that you’ll never get it done.

Feature flags should get you the stability you're looking for with far less potential for git conflicts than long lived branches.

They can be enabled on QA environments while left disabled on production.

That sounds like the key missing piece to this story, I'd look at spreading that idea around and getting reactions from the team.

Pair programming 100% of the time is absolutely exhausting and I would push back on that with the CTO. That said, most teams I've seen could benefit from communicating more than they do, and pairing more often will lead to that. Be ready and willing to push some additional pairing onto your team. That increase can happen slowly over time as they're willing to adapt.

Trunk based development can still include short lived feature branches and PR’s. You can also make release branches from your main branch.

Trunk based development is good because it forces teams to do other quality practices (automated unit, integration testing, static analysis, on every pipeline run). It also forces and incentivizes coding patterns like feature flagging, keystone interfaces, and small chunking of work.

Of course all of these things are modern software engineering practices the team will need to learn. But they are measurably effective practices.

On the practical level, I’d recommend your CTO buy each team member a copy of Accelerate or Continuous Delivery by Jez Humble. Fantastic resource for addressing the “why” your team may be thirsting for.

Constant required pair programming sounds like a recipe for burnout to be honest. I’m all for doing it from time to time for a lot of reasons but there are definitely days where I get, say, 4 hours of really good work in and I’m kind of done. I don’t need a situation where a peer is hanging around me who might be ready to go for 8 that day, every day. Even if PP is faster, which is sometimes the case but often not, I sometimes just need recharge time or else, frankly, time spent doing something else while my subconscious percolates over an issue (where yes, you can talk it out with your pair and sometimes that’s fine but sometimes it’s not). A lot of why I like development in general is because there tends not to be a lot of busywork, that is, even if you’re doing some very basic CRUD work to throw a grid on a page, it’s because there’s a need for all of that somewhere. There becomes a point to where talking out something you don’t have an answer to yet is time best spent either doing something else if you think you can figure it out or researching if you don’t.

To each their own I guess.

Here's an analogy that might help your arguments. Companies that can do trunk-based development well are like really high-quality watches. These types of watches are so precisely manufactured that they need virtually no lubrication. All of the mechanical components are machined to such precise measurements that they all click together perfectly with no need for it.

Companies using other branching strategies are like slightly less high-quality watches. Things like Gitflow are the lubrication that allows these watches to function. The components have some friction, but the watch still tells time just fine.

Trying to impose trunk-based development on a company that isn't set up for it is like taking a lower-quality watch and removing all the lubrication. You're probably going to destroy the watch.

Trunk-based development is the end result of doing a lot of things right. At a minimum, you need to have rigorous automated testing and robust observability systems in place. Implementing trunk-based development without those is a recipe for disaster.

What is your objection to trunk-based development?

My concept of "Trunk based development" matches what I just found in the top Google hit:

https://trunkbaseddevelopment.com/

... And it's completely identical to what your described: A light GitFlow with (short-lived) feature branches for individual developers.

So... Unless your branches last longer than some definition of "short lived," not seeing the conflict here.

If your feature branches diverge for weeks..? Then the problem isn't that you're not doing trunk development but that your feature granularity isn't small enough. And if that's close to the real situation, then I might tend to agree with your CTO there.

Even then you can mitigate any real problems by frequently rebasing your feature branches onto the trunk.

But pair programming? Ugh.

Occasional pairing is great. It's fun, even. But except among extremely low skill developers, productivity when pairing never makes up for the loss in productivity of having only a single keyboard per two developers.

Think of it this way: A competent developer shouldn't be able to have their productivity more than doubled by telling someone else what to do--or being told what to do.

A lot of developers will enjoy 100% pairing. But many developers get a rush from getting things done, and many don't like the social aspect of pairing. So expect a lot of people to quit over time--and for those people to, on average, be your stronger developers.

But that a CTO would push for this is a huge red flag. It means he is used to working for companies with lower quality developers. So unless you can convince him to change his approach, you'll likely see a lot of people leaving anyway.

Polish up that resume.

Ask him if he’d like a full time co-CTO

What is the problem the CTO is trying to solve?

Stop arguing about coding, and start experimenting with coding. Give it a try. The makeup (and resistance) of the team itself is definitely a factor, so there will be learning, trying, even failing. Be okay with that. Include everyone in the process.

For example, at my last coding gig (trunk-based), before marking a story card (in the discovery tree) as done, we'd bring QA in. Give them a demo. Give them a chance to kick the tires and poke holes. It's a whole-team effort. We kept refining how we did it.

Get a coach to teach you how to do team programming. You can't just pair/mob like you've been coding before. It'll be ineffective and tiring. You can totally do this with a technical coach. Without a technical coach, it could suck and then you'll blame the idea, when the reality was the team didn't know what they were doing.

This IS madness, but not for the reason your devs think.

It is perfectly possible to maintain highly secure, available, and high quality systems using tbd and full blown ci/cd. It's not really possible to do that on any kind of reasonable time scale without the teams being bought in and acclimated in a safe way to that way of working.

Is there a strategy in place to train everyone and gradually implement these changes?

I'm pretty sympathetic to the CTO, but this kind of top down mandate won't work

If this CEO is really coming in and mandating this then it will be a flaming disaster and they'll get fired in under a year. People hate been forced to do things. They'll drag their feet, half-ass things, and when the inevitable speedbumps come up people will say the whole thing is a failure.

Speaking as someone who has failed at this more often than I've succeeded (honestly not sure I've ever really succeeded): It is incredibly difficult to change tech culture and the only way I've ever seen it work is by starting small, showing the value with those people, and using their excitement to grow adoption over time.

TBD:

So after breaking down the two scenarios I think Im with you and I would be reluctant to pursue TBD.

Modeling it with 2 feature branches A and B:

You do a long feature, you send feature A to QA, QA cant parallelize so B will naturally have to wait, by the time they can do B, A will be merged so they test first A and second A+B. Sounds fair.

In trunk based, they will receive multiple requests for QA and full regression tests for both A and B. Their work will be many times repeated and the intermediate steps wont add any value anyway since that will likely be a dead code dangling in production anyway.

If it takes weeks to develop A or B, what happens in production when you deploy half-baked A and half-baked B? Are they just dead code dangling until they are finished? What did we earn then?

Pair programming:

I dont think I could do pair programming for more than 2 hours a day. If you design your org like that from the scratch and make it explicit during interviews and have acceptance of it - sure. But as a sudden policy change it isnt fair. Kinda feels like requesting everyone to wear speedos and bikinis at work - it works if you hire swimsuit models for swimsuit modelling but to force that upon any group, not fair.

I would have the senior developer stand up and explain precisely why it is madness and give examples

"basically use a simplified version of gitflow" - I would also have them explain this and why this is fine but trunk based isn't

I would also ask the CTO as part of their strategy why they want to change and what benefits the change will bring

Then meet in the middle

Process training without adequate expectations or training is pointless.

People like throwing out the term pair programming but rarely ever state their definition of it. Are they expecting one person to do the speaking and thinking and the other person types? Is it one person with a ticket and the other is there to validate or invalidate an approach or ideas? What is the expected output drop of the team while everyone is getting used to the new way of doing things?

If none of these types of questions are answered, the person suggesting it has not spent enough time evaluating their own assumptions.

Did 100% pair programming for 5 years, was the best job I ever had. I leveled up so fast and my soft skills improved so much that I’m in senior leadership now. Learning how others think, and being able to convey your own ideas verbally and in writing will take you so far

I’d be very suspicious of the “senior” dev who referred to TBD as madness but thinks GitFlow makes perfect sense.

Trunk based development is great. It’s the old saying - if something is painful, do it more often. Manual QA belongs in the bin - it’s only pretending to make things safer, in reality the only reason manual QA prevents production incidents is because it becomes such a bottleneck to making changes that the incidents slow down because you’re making less changes.

You may increase MTBF (mean time between failures) but this should be offset by an improvement in MTTR (mean time to recover) and hopefully MTTD (mean time to detect). There will be pain, and the pain will force you to improve your observability/alerting methods, as well as decouple rollouts from the code delay (e.g you feature flagging that works with schema changes)

Forcing pair programming practice sucks though. Teams should be able to self organise this.

You're basically already doing trunk based lol. Write an automation to create the release branches and the workflow is identical.

Trunk based is really feature flagging which you should be doing already based on your requirements.

Yes, sometimes the requirements of the job will change. If you don't like it, find somewhere else.

As for pair programming... I Iike it. Others don't. You're paid to do what is asked of you. Give it an honest shot and let it fail on its own, fighting the tide and whining will just get you in trouble.

Trunk based development is stupid and an invitation to unimaginably fuck your code beyond repair. GitFlow is the best and most recommended way to work.

I joined a team temporarily which followed trunk based development and believe me it was a horror show. All sorts of garbage was pushed to master by 100 other devs which was impossible to control. I was tasked to improve the code quality and employ strict coding standards to stabilise a very critical project. First thing i did was throw away trunk based approach and employ all kinds of restrictions and quality gates in ci/cd pipelines and MR configurations. Many junior devs got pissed as they were told to believe that trunk based is the only way to work. I had to forcibly throw them out of the project and hire good devs.

You can trial trunk based development which is guaranteed to make your stable code into a hot mess at which point you can tell your cto to fuck off and let senior devs and leads handle development and let cto handle posting motivational messages on linkedin.

(12yrs of pure development experience with big product companies)

I just moved a company from trunk based development to feature development, things are working so much better now. Trunk based development and feature flags created so many problems that all just went away with feature branches.

You can expect more bugs as more non production code is committed, you can also expect releases to be missed over and over again since you can't isolate features until they are at some level of readiness.

Pair programing will tank productivity, kill moral and cause will most of the senior developers to be gone within the year (or as soon as they can, maybe a bad market will keep them a bit longer), Junior developers will enjoy it.

This is totally a decision a CTO is allowed to make and its not out of the ordinary at all

ran my team on the idea of trunk based development and they're heavily against it including the senior devs (one of whom called it 'madness').

Calling a pretty standard development methodology "madness" is completely overboard. Your team is trying to gaslight the CTO here.

So answering your question, what to do- you do what your paid to do and follow through on the direction you've been given

Aversion to trunk based from senior devs screams "skills gap" issue. Bigger battle there. The CTO should know:

In the game of thrones, you either win or you die.

I legit do not understand why the CTO is enforcing engineering methodologies at all

A CTO should not be so close to the codebase that they can legitimately enforce such details. They may recommend them, or approve them, but imo these sorts of practices should be implemented by someone closer to the codebase. Engineering managers and/or "directors" who are also actually coding from time to time.

Of course, every company is different.

Does CTO at least have a plan for migration to new processes that include things like training, and adding support tooling, procedures, and systems? You can't just jump headfirst into TBD. And 100% pairing sounds exhausting.

Pair programming: which dev gets their name on the ticket and the credit for the work? Or are you using a system that allows you to track multiple devs on a task? I learned late in my career that "we before me" doesn't work if your name isn't on the task and helping others doesn't help you when it comes to evaluations and layoffs. The bean counters all work without context and have no clue what you do.

Seems like your DORA metrics will shape the story. If your stories are very thorough and have great acceptance criteria and your test automation is great then it will be a super success. If you need your users to show you all the use cases that you missed or did wrong, then you will do a lot of hotfixes which will lead to less frequent releases.

At my shop we settled on once a week releases. We have a complicated funnel with so many different paths and call so many services it is not worth it to write tests for all of them, having to set up so many products and customers and pricing by region. It is not worth the time to automate it all.

Better to just let the branch bake in the test environment for a week and use a combination of automated and manual tests, both at the API level and selenium/cypress level.

I'd push for a transitional version of gitflow that uses ephemeral environments or preview builds to do QA/UAT on. Gets you the best thing about trunk-based (trunk is always release-ready) while being a lot more familiar and a lot more safe.

Trunk based development doesn't necessarily require continuous deployment, though the two often go together. They definitely aren't madness, but they aren't something you can just decide to do one day.

For CD to be successful you need really solid automated testing. Not just a handful of unit tests, but integration tests and functional or e2e tests. Tests need to be part of the CI pipeline. You also need feature flagging so that you can build complex features piecemeal without exposing them to clients until they're ready. Ideally you also need the ability to deploy your PR's so that QA and/or product have a way to see and sign off on UI changes BEFORE they're merged.

It's all very doable and IME is a great way to work, but if CTO just says "do it" it might be messy. I'm assuming that you guys are a more traditional org - slower pace of releases, more reliance on manual QA, etc. You can't just flip a switch and change all your processes and the mindset of your teams. It's going to be a process, ideally with some guidance from some senior technical leaders who have strong experience with automated testing, etc.

While pairing full time sounds exhausting, I could see it as a temporary thing to get to trunk based development. Actually, non-trunk based development makes me nervous nowadays, not the opposite. Any time a branch is alive and not merged is time in which risk is going up. Non-trunk based work just serves to make people feel good about work (not to mitigate risk), all while moving slower. Much slower.

With that said, a lot must be done beforehand before you can do full time trunk work. The number one thing is getting to a high level of tests coverage (60-80% coverage, verified in CI). If you are not already doing that, I’d call your team irresponsible.

The second thing is creating a CI/CD pipelines that provides you with the guarantee that if tests pass, then you are good to merge.

Once that’s done, you need to make people work with short-lived branches and feature flags as a way of life.

Once those things are done, your CTO is right. You’ll gain more velocity, and you should see error rates go down dramatically. Maybe try to work with him to convince the team to change dev practices? They sound very old fashioned.

If it were me, I’d tell your CTO you think it’s a good idea, create a 3 step plan to get there, starting with creating a CI/CD pipeline that has a manual push to trunk button. Once you cross the test threshold, I’d push to remove the deploy button from staging environment leaving it there for prod until the team feels more comfortable with the new process.

The main thing against branches for development is that it's very hard to do good continuous integration. Ideally you want to be able to merge your changes with everyone else's as often as possible so you know quickly whether it causes problems. If you go away for a couple of days or more on a feature branch, you don't find out until the end, and that's where merge bombs happen.

On the other hand, just saying a a new CTO "right, you all have to do trunk-based development, now" isn't really the right way to bring it in. If I became a CTO I'd think of different ways to encourage this behaviour.

I read many comments seemingly misunderstood trunk-based development. I'd recommend reading through this link, it is a good discipline to follow. https://trunkbaseddevelopment.com/

As for pair programming, I think it's only helpful for complex designs/features etc, doing it all day everyday will lose its potency, just like having too many meetings, people will stop paying attention and it becomes ineffective.

Trunk-based development and monorepos work in special cases where all the software is tightly coupled and everything deploys together at the same time. Most enterprises work in silos, deploy once in a while and have explicit, loosely coupled dependencies, so a distributed system like git/Gitflow should be perfectly acceptable.

Github flow (small feature branches merging to master after a pr review) is one form of trunk based development.

Add a feature flag workflow and decouple your deployments from branch names & voila - comparable system that makes everyone happy.

Everyone resists change.

A CTO this deep in the weeds that he micromanages something like this is a huge red flag.

Mandating development practices from above with minimal developer input is a great way of risking losing a large percentage of engineers, starting from the most marketable ones. Id the CTO wants to do a disguised layoff, this is the way to do it, and it doesn't matter which specific practices he picks: His choices could be great, and it's still not the right way to go around it.

It sure doesn't seem that there is any actual convincing attempt, or any work to make the change doable (for instance, trunk based development without a high quality, low latency feature flag system is madness)

As for pair programming all day, every day, it's again, a guarantee that you will churn people. It's a perfectly workable system, but it's not for everyone, so people will bounce. Either the CTO wants to replace existing staff, or they are quite inexperienced here, because I can't imagine how this approach doesn't lead to a lot of churn.

Dave Farley is grinning right now.

You need to tell your CTO that the team isn’t onboard and the CTO needs to provide context and justification for their actions. The CTO needs to convince the team they are right, not just order the team to do it.

Why aren't your feature flags foolproof? 

Sounds like new CTO wants to look good for the boss. Hate execs like this, usually a red flag. With any new abrupt changes I always as the three whys. Get deep into their thinking and finally ask how it’s going to impact the problems we have now. Preferably do this in front of the boss if you can.

On the topic of "risk", trunk-based development with CI/CD has shown to reduce risk. Smaller and more frequent releases is less risky than "big bang" releases where a lot of changes are batched together. With small and frequent releases it is also much easier to diagnose a breakage when something does go wrong in production. Most tech companies have figured this out a long time ago and there is a lot of evidence of the benefits of CI/CD and trunk-based development out there. Important: short-lived feature branches are still used with trunk-based development.

From my experience, at companies that follow CI/CD there is no ceremony around releases since every code change that is reviewed and merged in will typically flow to production (assuming automated tests pass and no one has manually blocked the pipeline). Teams do have to invest in automated testing. Customer-impacting issues were rare, and when they did occur it was viewed as a learning opportunity to prevent similar problems from happening again.

On the topic of "full-time pair programming", I think that is a terrible idea for most teams. I think it is worth pushing back on pair programming, but you should absolutely work towards adopting trunk-based development and CI/CD.

1. TBD is useful when done well

2. Pairing is useful but exahusting, it has to be done well and carefully

There is no silver bullet.

I did my best job at a company doing pair programming every day and trunk based development. We wrote so many test that we were positive the change wouldn’t break in production. We monitored the hell of it as well with logs and stats. We didn’t have manual QAs. Devs and business did all the validation needed. When and if there was a problem we could roll back the change or implemente the fix quickly because of all the tests we had. We did also code reviews. Every once in a while would do a feature alone.

We do trunk based and it works great (we did gitflow ~4 years ago). It makes sense for our product though and the team decided on this approach. Other teams do it differently.

Also we don't do full-time pairing. We pair 'ad hoc' when it makes sense, and let people focus individually when it makes sense. We largely work async using chat.

I think a CTO should be smart enough to realise there's no one-sized-fits-all solution, and that this is borderline micromanaging. He should at minimum support the idea of trials/phases in case it doesn't work.

CI/CD your trunk-based dev to a QA environment. Production pushes become manual.

Considering how tough it is to implement changes from the inside of a team with small scale ramp up and experiments that are evaluated in checkpoints maybe you can offer that to him too. Make him prove the goal of his changes to you and agree to slowly roll it out with success criteria surely he can’t complain then?

In my experience (4-5yrs) pair programming is fantastic when you have

a) two humble people with a willingness to share and learn and
b) there is an expectancy of knowledge sharing, for example the domain expert paired with a new hire

Do this for 1-3 sessions to solve a task / deliver a feature, where maybe the new hire is doing the work outside of the pair programming sessions.

I've been on both sides (the domain expert & the new hire) and it was really enjoyable and impactful. But I can't see it as a routine effort that "should be mandatory for x amount of hours each week"

With the right people, a workflow with pair/ensemble programming and trunk based development is great. With the wrong ones, it can be a nightmare. Talk to the CTO and figure out what he plans to do about the people. If he is aware and aims to replace the people until the workflow is great, then your choice basically becomes to board the train or leave the station.