One of the key things that auditors want to see in validation documents is traceability between features, tests of those features and the evidence that the tests were executed. Normally this is presented as a requirements specification, a set of test scripts which reference the specification and records that the test scripts were executed. For product management and for an auditor this provides assurance that you planned what you wanted to create, you developed acceptance criteria and you tested the results.

At TrialGrid we record our specifications as Issues in a GitLab repository and we reference the unique identifiers for these issues in our automated test scripts and in commit messages. I outlined this in our recent blog post. For this discussion the important thing to know is that for every Issue we have a summary page in our validation portal which pulls data from GitLab to display a description, history and links to requirements tests for that Issue.

What about traceability between bugs and bug-fixes?

We also use Issues to track bugs. Some bugs have a visual component and we will include tests that these issues are fixed as part of our automated requirement test scripts. These "Bug Issues" then have tracability from the Issue to the tests that show they are fixed just like a "Requirement Issue".

However, not all bug fixes have a visual component so we would end up with some Issue pages with no hyperlink traceability to any proof that the bug had been fixed and tested.

Is this a big deal? In my experience, organizations don't provide this kind of traceability. Bug fixes are treated separately to requirements and references to bugs fixed go in the release notes. There may be a unit test that proves the bug was fixed but maintaining a traceability matrix between unit tests and bugs fixed is more work than anyone is looking for.

Still, it bothered us that there was this lack of traceability in our validation docs. In our unit testing code we would make reference to an issue. For example:

1
2
3
4
5

# issue 1359 
def test_result_suffix_inactive_field(self): 
    """Diagnostic should generate one result for suffix when field is inactive""" 
    self.params["ignore_inactive"] = "false" 
    ...

With a comment like issue 1359 we were making reference to the issue - what we needed was a way to find these kinds of references and match them up with the pages in the validation docs with a hyperlink to the code.

Code is data

Our source code coverage analysis system already generates hyperlinked code listings which we include in our validation package. These listings allow us to create hyperlinks which will take the reader to an exact line of a particular file. What we needed was a way to read all our source code, identify the issue comments and make an association between an issue and the file and line where the issue is referenced.

Happily for us, python has good support for analysing source code so we were able to get this working in just a few hours. The end result is hyperlinks from an issue page:

to the source code that exercises the issue:

Summary

We don't expect that auditors will want to review the source code of unit tests but we think that providing traceability to bug fixes is useful and meets the needs of auditors who want to see evidence.

We're not yet done with improving traceability in our validation package. More on our ongoing efforts in a future blog post.

Back in 2017 I wrote a blog post outlining our Continuous Validation procedure. This week Hugh O'Neill wrote an article on the PHARMASEAL Blog describing their process of Continuous Validation and it sparked some conversation on LinkedIn.

While our process remains much the same, it has been refined and tested through audits and I hope that an update to how our process works will be helpful to other organizations in adopting a Continuous Validation process or at least in accelerating their existing process.

Our practice is a refinement of an approach used by our former colleagues at Medidata Solutions. We didn't invent it and we are still open to learning new and better ways to perform validation.

Who are we?

To understand our process it helps to understand what kind of company we are.

• We have a small, very experienced team. We need everyone in the team to be able to do everything - coding, database admin, code review, testing etc. We don't have silos of responsibility.
• We are a geographically distributed team. We don't have a central office or any company owned physical assets. The company doesn't own a filing cabinet - it's all in the cloud with qualified SaaS providers. Everything has to be accessible via the web - our product, our systems, our procedures and policies and our validation documentation.
• We don't handle Subject/Patient Clinical Data. That makes us inherently less risky than, say, an EDC system.
• We know we are not world-leading experts in Validation. None of us have an auditing background. Our interactions with Auditors are an opportunity to learn and improve. Here we describe what is working for us.

If you don't have a Quality Management System, you're Testing not Validating

I said this before but it's worth repeating. Validated software is software that has been developed within the guiding framework of a Quality Management System (QMS). The QMS describes what you do, why and how you do it. An Auditor will compare your documented procedures to evidence that you followed the procedures.

Developing a QMS is beyond the scope of a blog post, but write to me if you're interested in that topic.

Continuous Integration

ThoughtWorks defines Continuous Integration as:

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

"Automated Build" means that when changes are received from a developer they are automatically compiled into a working system. If that process succeeds, further automated tests are run on the system to see that it works as expected. These steps are usually organized as a pipeline with each step performing a test and then passing onto the next step in the pipeline if the tests pass.

Here we see part of the TrialGrid CI pipeline showing 3 stages, some with multiple parallel tasks.

Continuous Delivery

Continuous Delivery expands on this idea with the end of the pipeline being a package that can be deployed to customers at the push of a button.

Commit Code > Compile > Run Tests > Assemble Deployable Package

Organizing a development team so that it can use Continuous Delivery is a big undertaking. It means automating every step in the process so that the output deployment package contains everything required. This is easier if you don't have hand-offs between groups. If your process involves programmers writing code and then technical writers authoring the help material this will be hard to coordinate. Much better if the help material is written alongside the code and they are dropped into the pipeline together.

Continuous Validation

Continuous Validation takes the idea of Continuous Delivery one step further and adds an additional output to the pipeline - a Validation Package that bundles up all the evidence that the software development practices which are mandated in the Quality Management System have been performed.

We want to do more than just drop a zip file on an auditor, by collecting evidence as data (just as Hugh O'Neill describes in his blog post) we can create a hyperlinked website, a Validation Portal which is easy to navigate and contains all the evidence required to perform a virtual audit of the software.

Evidence

Auditors have a simple rule:

If there is no evidence that something happened, it didn't happen.

Some of this evidence is generated as part of the pipeline - the output of tests for example but evidence of our activities such as code review need to be fed into the pipeline as data.

When we do something mandated by our Quality Management System we want to record that we did it. Ideally recording this evidence should not put a greater burden on us, it should be a by-product of the action.

Requirements and Traceability

In validated software everything starts with a requirement. When a user points to a widget on a screen and says "Why is this here?" you should be able to direct them to a written requirement that explains its function.

This will lead to the question "How do I know what you have implemented is functionally correct?" and you should be able to guide them to a plan for testing that feature and from there to evidence that the plan was executed.

Requirement > Test Plan > Evidence of Testing

Maintaining this traceability matrix is a challenge and it's a core element of the Validation Package that auditors will inspect so this too has to be captured.

Tools

To organize our development efforts we use GitLab. GitLab is an open-source product which you can install yourself or use their pay-to-use hosted version. It combines many of the tools required for the software development process:

• Source code repository
• Bug tracker / issues list
• Continuous Integration pipeline management

And a lot more besides. GitHub and BitBucket are similar products.

We use git for managing changes to source code. All you really need to know about git is that it:

1. Allows you to pull down a complete copy of the "master" copy of your source code.
2. Tracks every change you make
3. Packages up a set of changes into a "commit" with a comment on why these changes were made
4. Manages merges of your committed changes into the master copy

GitLab provides a way to coordinate this activity and collect data on it. When I want to merge changes into the master copy I create a record in GitLab called a "Merge Request". Here's a screenshot:

Notice the #1182, #1183 and #1184 - these are references to Issues stored in the GitLab issue tracker. We also put these references into our commit messages:

Here's the related Issue for #1184:

We use GitLab issues both for bug reports and for features. You can see that this one has been labelled as a Client Suggestion and as a Feature. You can also see that GitLab is finding references to this Issue in the merge request and in the commits which are part of that merge request.

Traceability from tests to requirements

Using Issue reference numbers gives us traceability between the requirement for a feature (the Issue) and the changes to the code that addressed that feature. This is cool but probably too much information for an Auditor. They want to see a link between this requirement, a test plan and testing evidence.

We use human-readable (and computer executable) tests to exercise UI features and we tag them with @issue to create a link between a test and an issue.

Here's a test definition (a plan) for the issue 1184:

1
2
3
4
5
6

  @issue:1184
  Scenario: iMedidata.com should be the default iMedidata URL for Draft Import
    When I view the Drafts list for my Project
    And I click the "Import Draft" button
    Then I should see "iMedidata" is the Login Type
    Then I should see "https://imedidata.com" as the iMedidata URL name

When this test is run it will click through the TrialGrid application just as a human would and take screenshots as evidence as it goes.

The programs that generate our validation package extract issue references from the Merge Requests and Commits and match them with the test definitions that exercise those issues and their related test evidence.

Here's a short sequence that shows the linkage between tests / evidence / issues and merge requests in the validation package.

Using this method it is easy to maintain traceability and the hyperlinked navigation has been well received by auditors used to dealing with requirements documents written in Word, traceability matrices managed in Excel spreadsheets and evidence in ring-binders.

Code Review

At the end of the previous video I scroll through the commits and their comments which formed part of a Merge Request.

An example comment:

When a Merge Request is created it displays all the changes made to the code and provides an opportunity to review and make comments. A Merge Request is blocked and code cannot be merged into the master copy until all comments have been resolved. This is our opportunity to perform code review. All comments and responses, including changes to the code as a result of comments are captured by GitLab.

We use GitLab API's to pull out the comment history on Merge requests and include them in the validation package as evidence of code review and of acceptance of the changes made to the code.

Generating Validation Packages

At TrialGrid we practice Continuous Delivery. That doesn't mean that if a code change passes our pipeline it is deployed direct to the Production environment. We decide when to deploy to production and trigger the automated deployment at times agreed with our customers.

We do deploy immediately to our Beta environment when the CI pipeline succeeds and a Merge Request is approved.

In addition, every Merge Request that passes the pipeline is deployed to our Development environment where it can be smoke tested by the reviewer before they approve the contents of the Merge Request.

We generate a validation package for every run of the pipeline. We only keep and sign-off on packages that relate to Production deployment but we are constantly generating these packages.

In effect, we are making the validation package part of the software development process. If a developer changes code in such a way that it affects the generation of the validation package then the pipeline will fail. The software may be fine, all tests may pass but if the validation package isn't generated correctly the whole pipeline fails and the developer must fix it.

For production releases we have a manual process of review for the validation package and sign off via electronic signature. Our CI pipeline has around 40 steps, many running in parallel, and completes in about 60 minutes. The validation package it generates weighs in at 1.3GB with about 500MB of images. This gets deployed automatically to Amazon S3 as a password protected resource ready for virtual audit.

Summary

I hope this glimpse into our Continuous Delivery / Continuous Validation approach is helpful. We are constantly refining this process and welcome new ideas.

In customer audits we have been able to demonstrate that we have our software development process under control and provide evidence that we are doing the things required by our Quality Management System.

One of the things that makes writing software for the clinical trials market different to writing software for, say, the consumer market is the need to Validate the product. I write it in scary bold font because I think this requirement does deter would-be startup founders in our industry. So how are we building software at TrialGrid in a way that meets these requirements?

First of all, there is a fundamental difference between Validated software and software that has been developed using best practice. Validated software is software that has been developed within the guiding framework of a Quality Management System (QMS). The QMS describes what you do and how you do it. An Auditor (internal or external) will compare your documented procedures to evidence that you followed the procedures.

Developing a QMS is beyond the scope of a blog post but I want to give you an overview of our process for creating quality software while collecting our documented evidence that we're following procedures.

Activities and Evidence

Imagine I am a TrialGrid programmer. I have received a requirement and I have written some code to meet that requirement so now I want my code merged into the master copy of the product so that it can be made available to my customers.

Here are the steps that have to be covered under our process:

1. Code Review (Manual)

First I need my code peer-reviewed. Have I missed anything? Is there a better (more efficient, clearer) way to implement the feature. Does the peer understand the code well enough to maintain it in the future? The output of this step will be documented evidence of review, comments and perhaps suggestions on changes to the code. If changes are needed the cycle starts again. We do this code review in our source code control system, so that we have evidence of who proposed the code, who reviewed it and what their review comments were. The source code control system is set up so that any comments must be resolved (answered by explanation or change to the code) before the code can go on.

2. Coding Standards Check (Automated)

Every company has a set of coding standards which dictate how code should be written and formatted. A lot of time can be spent arguing the merits of one style over another so to avoid that we have an independent arbiter, a code compliance checking software robot that enforces our preferred style. If the submitted code doesn't make the grade then "No Soup for you!" and it is rejected. The output of this step is evidence that the checker was run and any findings it reports.

3. Complexity Testing (WIP, Automated)

This one is a work in progress. Code should be understandable. That means that we should have code units with a clear purpose, ideally performing just one function. A complexity measurement examines the code to determine how many logical paths there are through that code. Thomas McCabe wrote a paper defining a logical score. He says a complexity score of 10 is a good (but not magical) upper limit. We currently have 37 functions in our 23,000 lines of code with a score greater than 10. Our worst case has a score of 42. That represents our high bound. Once we have simplified that code and reduced its score we'll set the threshold to our next highest with the goal of getting to a max of 10. Any code over the limit is rejected. As before, the output of this step is evidence that the checker was run and any findings it reports.

4. Unit/Integration Testing (Automated)

We have a simple rule. If you write code, you write a unit/integration test to prove that it does what was intended. This is enforced through running unit tests with a code-coverage tool. Code-coverage checks to see that every logical path of the code has been exercised. We have a 95% threshold (100% unit test coverage is not usually regarded as being useful - you end up writing tests just to achieve coverage rather than helping ensure the code works as intended). As I write, our coverage figure is 97% with more than 1,200 - one thousand, two hundred - unit tests. If code coverage falls below that threshold the code is rejected. The evidence from this step is a detailed report and a set of hyperlinked pages of the code which show the coverage for each file and line individually.

5. Requirements Testing (Automated)

Our Unit and Integration tests are code-based tests, code checking code but we need to provide traceability between user requirements, tests that show those requirements are met and evidence that the tests have been executed. I remember one company I worked for recording videos of testers running through manual tests scripts. Happily those days are behind us. We write requirements in a fixed style:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

Feature: Login
    As a security auditor I want to be sure that logins to the system are secure
    so that I have assurance that the system is not inappropriately accessed

    Background: Test Setup
        Given a user exists with username "login_test" and password "password1234"

    Scenario: Failed Login due to bad password
        When I visit the login page
        And I enter "login_test" as my username
        And I enter "BADPASSWORD" as my password
        And I try to log in
        Then I should see "Please enter a correct username and password. Note that both fields may be case-sensitive."

the steps of these tests are then automated, another software robot driving a web browser (quite eerie to watch) clicking through the application, checking what it sees on the screen (by examining the HTML content of the pages) and taking evidence screenshots. The output of this step is a set of detailed logs showing which steps passed (or failed) and a set of evidence screenshots which are later packaged into the Validation Evidence Package (see later). Of course, if any step fails here, we're back to the beginning.

6. Documentation Review (Automated)

TrialGrid isn't just code, it's also online help. When you write a new feature you write the help that goes along with it. This is reviewed in Step 1 but there's an (automated) process of assembling the individual help pages into a hyperlinked website. Cross-links are checked and if there are any problems (missing pages, bad links) the build process fails and we start again from Step 1.

7. License Checks (Automated)

Modern software is built on a foundation of libraries. For instance, to read a SpreadSheetML file I don't want to have to write a general-purpose XML parser, I'll use an Open Source one or license a commercial product that someone else has written. Package managers make adding Open Source libraries to a project very easy. As part of the documentation and build system we have a requirements file that lists our required libraries and their versions:

1
2
3
4
5
6
7
8

amqp==1.4.9
anyjson==0.3.3
appdirs==1.4.0
asgi-redis==1.0.0
asgiref==1.0.0
attrs==16.3.0
autobahn==0.17.1
...

But not all Open Source licenses are created equal. Some do not permit use in Commercial Products like ours. As you can see from that list, there is no licensing information there. To protect ourselves and our clients we need that so we have a step in our pipeline that uses an API to pull information about these packages from their setup manifests. Most packages list their license (MIT, BSD, etc) and we compare these to a list of pre-approved licenses. Some don't list their license and in that case the system warns us so that we can manually review. It might take some internet searching to find those packages and we can add them to an exclusion list of manual approvals. Note that we pin to exact versions like amqp==1.4.9 if this library changes to version 1.5.0 and we just automatically upgraded we might find that the license changed between versions. At the moment this tool does not reject code that makes use of non-approved or not-found licenses but it already gives us early warning if unapproved libraries are sneaking in so we can find or write alternatives. The output of this stage is a compliance report which goes into the Validation Evidence Package.

8. Create Validation Evidence Package (Automated)

The final step of the process is the generation of a Validation Evidence Package. This gathers up all the evidence that was generated in the previous steps into a set of hyperlinked documents so that Auditors who want to review our evidence can browse our license management, unit test, code coverage and requirements test evidence in a single place. Once generated this file is automatically packaged up and retained in our build system.

Summary

We didn't invent Continuous Integration (automatic running of tests on checked-in code) and many of the steps I describe are performed routinely by other companies as part of best practice. Nor is this list comprehensive, to get to production the code would have to go through Installation Qualification (IQ) and Operational Qualification (OQ) steps. I think we are unusual in being so rigorous at such an early stage of our company life and in our drive to automate so much of the process. We pay a price for that rigor, it is very annoying to have a build fail because we failed to put an opening brace in the correct place but we think the long-term benefits are worth it.