DORA document comparison as a complex, data-rich case

Click here to view the German version

Setting out detailed requirements for IT systems, the EU’s Digital Operational Resilience Act (DORA) is designed to enhance the resilience of IT operations and allow insurance companies and other financial services firms to avoid internal disruptions and respond effectively to external threats such as cyberattacks.¹

Following the regulation’s entry into force in January 2023, all major financial institutions launched projects to ensure compliance with DORA’s requirements, beginning with gap analyses using their own IT documentation and process records as well as published regulatory requirements. As these were written in natural language, the use of generative AI was an attractive option from the outset, reducing cost and effort while maintaining high-quality results.

This DORA gap analyses example is representative of many compelling generative AI (GenAI) use cases. Complex and data-rich, they differ from simpler data-intensive tasks in two ways:

• Diverse source types. The content originates from different types of sources with different functions – in DORA’s case, legal texts and IT documentation
• Cross-document comparison. The goal is not merely to extract answers from a single source, but to identify discrepancies across documents.

These complexities place higher demands on both GenAI models and the overall solution design. To illustrate this further, let’s focus on another gap analysis required on the road to DORA compliance – the management of outsourcing documentation.

DORA and outsourcing management

DORA sets out highly specific requirements for contracts with external IT service providers – for instance, mandatory termination clauses. As a result, firms face the challenge of reviewing all their IT contracts against these obligations.² Even smaller organizations may have hundreds of contracts, differing in structure, length, and detail.

Without generative AI, this review process would require considerable manual effort. So how can this complex, data-rich use case be effectively addressed using generative AI? The first step is to identify the relevant DORA provisions for IT contracts and convert them into actionable rules.

For example, check whether the contract contains a termination clause. Such rules are then formalized within prompts. At this point, the approach resembles traditional rule-based AI, used in areas like invoice checking since the 1990s. The key difference is that, because GenAI models understand natural language, the need for highly detailed rule sets is reduced.

Moreover, subject matter experts gradually refine prompts until results improve. Experience shows that even minor prompt changes can enhance results. However, adjusting longer prompts may also cause unintended side effects. This makes it essential for domain experts to have at least a basic technical understanding of GenAI, or to work closely with AI specialists. Agile collaboration between business and IT is therefore well-suited to GenAI projects. Given that improvements often depend on fine-tuning, these projects typically require more frequent interaction than other agile initiatives.

AI working with human experts

In complex cases such as DORA contract reviews, GenAI serves as a valuable support tool but does not replace human judgment. A three-step approach has emerged in practice:

1. Create and refine a prompt through extensive testing until results are reliable
2. Apply the refined prompt ‘in production’ to real contracts
3. Have experts validate the AI outputs.

To simplify the expert’s review, AI outputs are typically presented in an Excel file with color coding: green for compliant clauses, red for deviations, and yellow for unclear cases. This visual format allows experts to focus quickly and reduces workload significantly when prompts are well-designed. Experts therefore remain central – but the effort they are required to expend is reduced thanks to AI pre-processing.

Prompts should be tested against real contracts from the outset. Although GenAI can easily create sample contracts, real agreements often contain unexpected variations. Only signed contracts have legal force – and many older agreements may exist only as scanned paper documents. These must first be converted into machine-readable formats.

Major hyperscalers now offer powerful services for this purpose. Yet, challenges can still arise, especially with documents containing tables or graphics. Project planning should therefore allow sufficient time to prepare all relevant contracts in the correct format.

In the next article in this series, we will examine another complex, data-rich use case with a slightly different technical focus. Importantly, the tools required for smooth implementation of such projects have already been developed by Capco and successfully applied in numerous client engagements. If you have questions or would like to discuss further, we would be delighted to hear from you.

References
¹ Publications Office
² See Article 30 (2) h) of the regulation: Publications Office