How do you manage UAT without freezing a release branch in an agile environment with CI/CD?

[MalcolmSequeira]

I’ve been reading more about how agile shops that use CI/CD are frequently at odds with frozen code. How is UAT handled when there isn’t frozen code, is my query. Can features be evaluated in flight without having any code frozen and yet have the product approve a release?

Similar concerns for QA exist with regard to manual testing, particularly regression tests on intricate functionality where dependencies might be present. If a build wasn’t tested in a production setting, how can QA certify it?

[MatthewShun]

Pre-production (perhaps with production data) or staging (maybe with non-production data) environments are typically what you desire. In order to test “in prod” before releasing to the public, certain companies will heavily employ feature flags (but this should obviously be tested for safety in order environments first).

The utilization of ephemeral Preview environments is another choice that has recently gained increasing popularity. In other words, create a separate instance of your application for each pull request and/or feature change so that you may give the feature your approval before it is merged in and made available.

[MalcolmSequeira]

@MatthewShun, the main point I’m trying to make is how to carry out UAT without adding delay. My development team needs the freedom to launch deployments into production whenever a task or story is finished. I disputed this, raising potential problems with publishing code that might be devoid of bugs yet still fail UAT. They respond that this will only cause the system to experience delay. I’m not used to working in this manner; I’m used to product acting as the production gatekeeper and UAT taking place on a frozen branch before a deployment.

[JaneWinfred]

@MalcolmSequeira, I concur. You should under no circumstances consent to such production adjustments.

There are advantages and disadvantages to each of those solutions. If you must test at production grade, you will need to purchase something pricey. Since we haven’t really needed anything of a production-grade kind, we haven’t really needed to spin up any AWS servers. I oversee a payment processing system where latency is critical. We simply know that we will demand improved production performance.

[MalcolmSequeira]

@JaneWinfred, not necessarily meaning production power or speed (or even production data). I use the term “production” in the sense that if QA has approved the finished version, I can be sure my UAT was comprehensive, and no new features won’t be included in a later deployment that could damage it.

[JaneWinfred]

Ah! Now I understand what you’re saying. I apologise for the confusion. For each level of testing, in my opinion, you need staging servers, which can all have various CD-managed code versions. If you don’t have any stakeholders or consumers to obtain UAT certification from, just QA certification will do. If not, there would be too much risk of failure.

Most likely, you and your technical team can agree on labelling a release as simple or difficult in order to justify the risk you’re ready to take.

[TinaGreist]

@JaneWinfred

You should under no circumstances consent to such production adjustments.

Although I often have the same thoughts, it is true that certain stores operate in this manner. A developer is given the authority to submit changes to production as soon as they meet with the DoD on a story. The coders are clear that quality control is entirely their responsibility, and those organisations frequently lack dedicated QA personnel.

I don’t see how my team will get there from where they are right now. I might not want to. It’s hardly unheard of, though.

[JaneWinfred]

Interesting! This snippet for small projects or when a team is just getting started was new to me.

[TinaGreist]

For a while, we had a director of development who had experience in that field and was seriously considering firing our entire QA team. He didn’t take any drastic action because it would have been a HUGE step up from the current procedures in place and instead moved on to another opportunity.

[MariaWilson]

Feature flags are your buddies in this situation. Even if it’s simply your QA, you can have the feature enabled for beta users in Prod. Before your Beta testers provide their approval, you are not required to release to all of your clients.

Automation for complex regression should be built into your workflow. If you only meant exploratory testing, though, then having a preprod environment for them to test in is preferable; if not, as I’ve already indicated, feature flags can be used to restrict access in prod.

[Nathanendicott]

There will be some additional latency if you want to do another step before deployment.

It’s important to keep latency as low as possible and to make sure that the team supports your desire to act as a “gatekeeper.”

I don’t believe that making the product the “gatekeeper” is the best strategy, but perhaps your team might be better served by a feature-flag strategy (they can release the new feature, but just to a limited audience until you authorise “allowing” it for everyone).

[MichaelYoffe]

Just to unpack the “CD” portion. There are two kinds of “D”: Delivery and Deployment. I like to differentiate between these when working with clients. I’m a CD coach (both kinds).

In continuous delivery, you typically have gate keeping which would use some kind of branching strategy like git-flow. This approach is typically gated through UAT environments and such. It’s common to map branches to environments in this approach, and it often forces you into code freezes and merge issues.

In continuous deployment, every commit can theoretically go straight to production. In reality however, teams still do some level of testing, it just doesn’t happen on the main branch and is therefore non blocking. If you can have each feature/bug fix be developed, tested and deployed independently, then you can have a continuous flow of deployments.

To achieve both low latency and a continuous flow of deployments, you would need to employ 2 techniques.

The first is to have a production-like test harness (a cut down version of prod with all external dependencies faked) that gets deployed in each feature branch. This test harness allows your QA team to perform verification and exploratory testing of isolated changes in a non-prod-like environment. This gives you enough (not total) confidence to progress code to production. You can complicate your test harness accordingly to increase your confidence by improving the fakes and making use of contract testing.

This technique would be used for feature that would take a max of 1 day of testing. Any more than that you risk long lived branches which leads to merge issues, so you need to use…

The second technique, which is to use Feature Flags for things that need a longer time to develop & test with QA, UAT and alpha/beta customers. You would set the FF to only expose the features to the required groups of people. In effect, what you’re doing here is partitioning a single environment with a single data store to be used for PROD, UAT and QA. The advantage to doing this is that you don’t have long-lived branches in the code. And you also get a fully integrated environment to test in.

The challenge in the second technique is to slice the features in the first place in a way where they can be incrementally delivered.

Note that without sufficient automated test coverage it’s futile to even attempt continuous deployment. It will be a constant struggle of regression defects entering the system as the code becomes more complex.

I saw some other responses referring to blue/green. This is a good technique as you can choose when to release a batch of work to production, however it’s forcing a batch-release approach into your system which is ultimately blocking deployments and is therefore not continuous.

Hope that answers your question.

[MichellePlowman]

Have a second Have a second environment identical to prod?

ETA: Before each UTA iteration of the user acceptance test, I would create a fresh prod clone with the identical operations, data, and permissions as prod.

[KaneMorgan]

@MichellePlowman, This is the real answer, Blue/Green environments are the future. It’s more expensive to implement, but reduces risk if done right.

[Pankaj-Jain]

Agile has an answer for you on this! DoD, or Definition of Done, is what it’s called.

A narrative or task is often designated as complete when the marketing is ready, user or beta testing has been completed, the support team is aware of any changes, and any communication plan or document revisions have been made, not when it has been tested or is bug-free ( privacy policy, website feature lists, FAQ etc. which require updates )

Depending on the features, we used different DoDs in my products ( for new features a quite substantial one will be there, for incremental updates, some are not there, specifically the communication plan and docs - Marketing and others are alerted and depending on the size beta may or may not be run completely, for bugs or hotfixes it is pushed right through and usually change is updated after the fact as a release note internally )