Non-Functional Requirements Consultative Group meeting minutes – 19 June 2024

Chair Introduction

Mark Verstege, the Chair of the Non-functional Requirement Consultative Group (NFR CG) welcomed everyone to the meeting and acknowledged the traditional owners of the lands. 

The Chair noted that the initial trial period for this group is coming to an end after the next meeting.  Whilst this trial has demonstrated that the approach is effective there is still a lot or progress to be made.  He proposed to members to extend the trial for further 6 meetings, subject to the Data Standards Chair approval. The members agreed. 

ACTION:  The Chair to seek approval from the Data Standards Chair to extend the trial.

The Chair noted that Andrew Ferris has changed roles at AGL was stepping down from the committee.  They are waiting for AGL to nominate another representative for consideration. 

The Chair noted that Jim Basey (Basiq) was an apology for this meeting.

Minutes

The Chair thanked the group for their comments on the minutes from 22 May 2024 meeting.  The Minutes were formally accepted and will be made publicly available on the Consumer Data Standards website.  

Action Items                                                    

The Chair provided an update on the outstanding Action Items as follows: 

• Inviting Xero to a future meeting – in progress
• To invite a large DH (via AEC) to join the group – no progress to date. The DSB suggested they would raise this at the next DSAC meeting

ACTION:  DSB to invite DSAC members who represent the energy sector to join the NFR Consultative Group

Problem Analysis

Information synthesis

Nils Berge from the DSB presented a summary of the main drivers, issues and opportunities that emerged over the previous meetings, covering topics such as data sharing arrangements, bulk data, low velocity data, event driven approaches, NFR settings, performance and pagination.

The DSB noted that next steps are looking at the drivers, issues and opportunities: 

Driver # 1:  Business use cases (energy in particular).

Issues:  Business consumers may have larger number of accounts and data points, meaning they could require a significantly larger volume of data to be shared within standardised NFRs.

Opportunities include: limit the number of accounts that can be included in a single arrangement; limit the amount of data to return in a single response, to remain within the current NFR; and introduce an alternative sharing pattern tailored to the requirement.

Driver # 2: ADRs want to maintain an accurate, up to date view of consumer data through their use case.

Issues: current poll / request mode of data sharing requires constant checking for new or updated data, so a proportion of requests may be made for data that has not changed; ADR may not know the expected velocity of change for certain datasets, ADR immediately repeats rejected requests with the aim of ultimately receiving all the data they need for their use case.

Opportunities include: specify known low-velocity endpoints to limits that can be defined and adhered to; and defining a CDR-specific response header to provide a hint of change velocity determined by the data holder, potentially per endpoint per customer.

Driver # 3: ADRs want to reduce the number of requests they make (while still maintaining an accurate view of consumer data).

Issues: Calls must be made to multiple endpoints to provide a complete view of data; historical data may change, forcing the ADR to make repeated requests in case it has changed.

Opportunities include: consolidate fields to allow fewer requests for related data; and consider a push-based model/event based, or a ‘last update’ endpoint to help determine whether further requests are necessary.

Driver # 4: The NFRs are intended to support different retrieval patterns by having ADRs indicate when a customer present request is being made through their use case.

Issues: The customer-present indicator is not passed through to secondary data holders, so they cannot tailor their service to meet the stricter NFR requirements.

Opportunities include: secondary data holder use a heuristic model to determine when to apply the customer present NFR; and requiring an indicator to be passed to the secondary data holder without disclosing private information.

One member conducted analysis for Energy API calls to determine how big the issue is around repeat calls or ADRs requesting the data. 95% of service points are repeat requests this month to date and less than 5% are used on a single day which equates to 0.3 % of all requests made. This gives a good indication of the size of the issue.

Traffic forecasting

The DSB noted that a key topic that has been discussed is if we are prepared for expected step changes in traffic volumes.  The key two issues from a banking perspective are the migration of screen scraping load over to the CDR and what if a large collector of data comes on board.

The DSB noted that there a number of different ways this can be modelled but they kept coming back to the issue that was raised in issue # 541.  This is when operating a single software product where the standards currently impose a limit of 50 transactions per second, and assuming they can exhaust and utilise that connection.  If a software product serves a particular use case, that puts a hard limit on the number of API calls that can be made for any given software product within a 24-hour period.  They looked at other assumptions like TPS limits which were discounted.

The DSB presented some data modelling based on the change requests and the expected load from screen scraping migration and large data collectors.  They also highlighted some of the opportunities and challenges for different data collection patterns and NFR settings. 

The DSB discussed potential solutions to address issues with the current API, including a bulk transaction details API to get the delta of changed or new transactions since the last call, implementing events signalling, and introducing a consent API etc. 

The group discussed the ongoing collection of data for customers that are already connected and the initial full collection of data over 7 years of history for new customers. They also discussed the need for better guidance from the DSB to help assist ADRs on knowing what core patterns to use.  One of the key themes was the way data is communicated between data recipients and data holders.   

The group discussed a worst-case scenario of collecting all account data for a new customer and its impact on the number of consumers that can be onboarded in a particular day; the usage of direct debits and scheduled payments APIs and the challenges of collecting data on an ongoing basis and a one-time basis and providing customers with data at a specific point of time.

The DSB discussed the data collection process, its limitations and the time it takes to collect data from a full historical perspective.

The Chair noted that the next step is for the DSB to come back to the group with a cohesive view of the primary drivers, issues and opportunities.  In the next meeting through an activity, they will get the groups feedback on the most valuable opportunities to pursue as we move towards problem solving, decision proposals and change requests.  They also need to do some data analysis and start to overlay what the current TPS for organic traffic and start to superimpose the figures they have. 

ACTION: DSB to provide a cohesive view of the primary drivers, issues and opportunities to the group for further input

ACTION:  DSB to complete the next stage of data analysis to overlay current TPS for organic traffic with new figures

One member presented a batch-based model for providing bulk data for customers which is used for customers who require a large amount of information that cannot be serviced by APIs.

The group discussed the data limits and the next steps for the DSB.  They talked about the number of sessions per day, the TPS limit and the event driven architectures and how to deal with bulk data.

The group discussed the batch file model for servicing large amounts of transactional data to institution customers and the possibility of using a file-based approach to extract transactional data for large customer. They noted that this approach is different from the current API based approach and is more similar to the Xero model. 

The group discussed the approaches for sharing data between systems, one being bulk data sharing which involves aggregated data from multiple consumers and sharing it with accredited data recipients.

One member presented the different aspects of real time eventing model and load average respectively. The eventing model only publishes transactional changes and that separate event topics are needed for different types of events.

One member shared some graphs showing the TPS patterns for different ADRs and software products and highlighted the gap between peak and average load and the potential issues with rate limiting and scheduling.  It showed a significant gap between peak and average TPS highlighting the need for capacity.  They also discussed the issue of peak usage and the importance of controlling outgoing TPS.

The group agreed to continue the conversation around load management and how to address the gap between peak and average TPS and the different use cases and access patterns. 

ACTION:  Continue the conversation around load management and explore solutions to manage peak and average TPS effectively

Meeting Schedule

The Chair noted that the next meeting is scheduled for Wednesday 17 July. 

Any Other Business

No further business was raised.

Closing and Next Steps

The Chair thanked the group for participating for attending the meeting. 

The meeting closed at 15:59