From smarter search to faster access: A deep dive into the enhanced Gateway

What is the Gateway?

The Health Data Research Gateway (the Gateway) is an online platform providing researchers with a simplified journey to search, discover and request access to health and health-related data and associated sharable resources. The Gateway is funded by Health Data Research UK (HDR UK) and its design and development is coordinated by our Technology Team.

The Gateway does not hold or store any health or patient data itself, nor does it approve users to access sensitive data for research. Think of the Gateway as a digital library for health data. It brings together and connects researchers to various resources, like books in a physical library, that are owned and managed by different teams and organisations. These resources can then be used by researchers to help them answer important questions that drive improvements to people’s lives.

In November 2024, we released a major new version of the Gateway aimed at transforming how the research community engage with and use health data. The update is packed with enhancements to improve user experience, search functionality, resource discovery, and metadata management. Let’s explore these improvements in more detail and hear firsthand insights from our Technology Team, the people behind the transformation!

A user-friendly experience: Explore the Gateway’s new interface

Thoughtfully designed with both researchers and data custodians in mind, the new Gateway interface creates a more supportive and user-friendly experience. The platform now features an intuitive, streamlined approach to navigation that helps you to find exactly what you need, faster and more efficiently.

You can explore a growing number of resources directly from the homepage and easily switch between them using the horizontal tabs on the search page.

We’ve added lots more helpful tool tips across the site for quick, in-context explanations, and if you need further support, this is just one-click away with the new yellow support button now available on every page of the Gateway.

There’s also a dedicated support section for data custodians with comprehensive onboarding and management guidance, making it easier than ever to contribute and maintain your Gateway assets.

Stephen Lavenberg, a Business Analyst within the Tech Team focussed on understanding the needs of Gateway users, says:

“The redesign brings user-centricity to the forefront of continuous improvement of the Gateway. We are also better positioned to adapt quickly to user feedback whether that’s improving an existing feature or introducing something new.”

For the best experience on the Gateway, Stephen recommends: “My advice is to create an account. This allows you save your searches, create a library of datasets, and share and manage your resources. Each time you login, whether it’s your first visit or your fiftieth, you’ll be able to build on your previous work and enhance your research journey.”

Improved metadata onboarding

A dataset on the Gateway is summary information, or metadata (data about data), about a particular dataset, not the health, patient, or sensitive data itself. Metadata is added to the Gateway by data custodians and can be administrative, descriptive, structural, and technical. This information helps users decide whether the data itself would be useful for their research project and whether they would be eligible to access the dataset.

Managing and onboarding metadata through the enhanced Gateway is simpler and more efficient, thanks to a series of recent updates designed to give data custodians more control and flexibility.

With automatic integrations now in place, onboarding metadata and managing standards across platforms is faster and more streamlined. Plus, a new ‘Developer’ team role allows you to set up and self-manage integrations directly from your Team Management dashboard, making the process more efficient for your team. For those who prefer manual onboarding, we’ve significantly improved the metadata onboarding form to include new fields, an adaptive interface, and better guidance to make the experience smoother.

As our Service Support Manager, Jemma Mitchell, spends a lot of time with data custodians, supporting them with the onboarding process. She says:

“The improvements to metadata onboarding have made it easier for users to update and view information about their datasets. Whether they prefer automated Gateway integrations or more manual methods, we’ve made it simpler and faster for data custodians to upload and publish their metadata on the Gateway. We’ve also introduced additional data fields – which when populated – make it easier for researchers to find what they are looking for in a particular dataset.”

Taryn Aspeling, a Senior Intelligence Analyst for NHS Northwest London, manages metadata onboarding for Discover-NOW, part of the NHS London Secure Data Environment, comments:

“We use the Gateway to make our data more discoverable to researchers who would like to find out what health and care data we have available for them to use. We welcome the automation updates to metadata onboarding. It makes a big difference to our capacity and keeps the metadata standardised and always up to date .”

Smarter search

We’re excited that Elastic (open-source search engine software) is now integrated into the Gateway, transforming how you search for datasets and related resources. This advanced technology generates search results that are faster, more relevant, and improve over time. Try a single keyword search, a multi-term query or search by medical classifications and codes! You can also use the improved filters to refine your search results further, including new data type categories and dynamic value filtering.

Branwen Snelling, one of our Senior Research Software Engineers who was responsible for building the Gateway’s updating search functionality, says:

“The integration of Elastic is fantastic! It’s providing a much more sophisticated search engine service for the Gateway than previous solutions. My favourite things are the simple but elegant features we added that make a big difference to the user experience, like the handling of multi-word queries and fuzzy* matching. I am excited to add query builder logic – and, or, and not instructions – that would enable users to construct more complex queries themselves.”

Branwen’s top tip for searching the Gateway is: “Use the table view of results to get an overview and easily compare datasets. It condenses and presents the key information in a clear but compact way.”

Discover more with better linked resources

You can now find resources related to a particular dataset, like publications that have used it previously for example, much more easily thanks to smarter recognition between entities and an updated metadata page – click on the interactive chart to start exploring! Plus, you can search for related research and publication beyond what’s listed on the Gateway with the integration of Europe PMC – an open access database of biomedical and life sciences research – into the platform.

There are also new collection-based features so you can easily explore all the resources linked to a single organisation or group of organisations in one place, such as the NHS Research Secure Data Environment (SDE) Network which brings together health data assets from one national SDE and 11 regional SDEs.

Our Chief Solutions Architect, Chris Milner, champions the Gateway’s enhanced resource discovery, commenting:

“The best thing about the linked resources functionality is its growing value over time. As more resources are added and connected to each other, researchers will be able to access an increasingly rich and comprehensive network of information, right at their fingertips.”

He emphasises that “data is key” and explains that the team “spent a lot of time working with data custodians to bring additional data in to create these relationships, and automation.” The team is working on automating a number of linked resources to predictively surface related information. This will further improve the solution, reducing the effort required by data custodians to collate information and enabling researchers to more easily discover and explore it.

Simplified data access

The Gateway now makes it easier to connect with data custodians about specific datasets. Whether you need to send a simple general enquiry or a more detailed feasibility query, you can make contact directly from the platform. This means you can ask questions and gain deeper insights into datasets of interest to determine their suitability for your project before submitting a full data access request. To get started, use the new enquiry buttons located at the top of the updated metadata page and continue your conversation over email.

Our Senior Technical Programme Manager, Clara Fennessy, has worked on improving the Gateway’s data access feature since the beginning. She says:

“We want to make the data access process as seamless and user-friendly as possible for both researchers and data custodians. One of the most valuable improvements we’ve made – inspired by user feedback – is the option to send a single, general enquiry to multiple custodians at once. This change saves users time and effort. Even better, researchers can build these enquiries using their saved library of datasets, making the process even more efficient. We’re excited to be taking this a step further soon by extending this capability to full data access submissions – so stay tuned!”

Let us know what you think

These enhancements are part of our ongoing commitment to simplify the process of finding and using health data so that researchers can spend more time answering important questions that drive improvements to people’s health.

We’re excited to see how these changes support your work and improve your experience with the Gateway. We’d love to hear your thoughts – try out the new features and let us know how they’re working for you!

A final word from our Chief Technology Officer, Emily Jefferson:

“We first launched the Gateway in 2020, and our goal was to help researchers discover health datasets more easily. Fast forward to 2024, and we’ve gone far beyond data discovery. Our latest enhancements are designed to streamline the entire researcher journey – from finding datasets to analysing data and sharing insights – to make it smoother and more efficient than ever before.”

 

“A massive thank you to our amazing Tech Team, partners, and collaborators for bringing this vision to life. Together, we’ve created a platform that truly supports researchers at every step of their journey, connecting them with health datasets, analysis scripts, software, publications, and research projects from across the UK and beyond. We couldn’t be prouder of what we’ve achieved so far – and what’s still to come!”

Take me to the enhanced Health Data Research Gateway!