Skip to main content
Broadpin

A Year of Listening Instead of Guessing: What 90,000 Hours of User Behavior on a Legacy System Taught Us

This blog was written by Stephan La Rocca.

Before modernizing legacy applications, many organizations lack a crucial foundation: reliable data on which parts of the application are actually being used. Decisions are often based on assumptions, with the corresponding risks in terms of effort, costs, and user acceptance.

As the name suggests, legacy applications have been around for quite some time. Not much remains of the initial euphoria surrounding the drafting of the 1.0 requirements specification and the enthusiastic team that, based on the data model, carried out the first validations and created the user interface, which today feels more than a bit clunky. Generations of users and a business environment subject to constant change have added new chapters to the requirements specifications. And these were not always consistent. Developers came and went, and various trends in software development have likely left their mark on the software. Technical shortcomings are obvious but also too deeply ingrained to be easily fixed in the next release. The same fate befell the users, however,they came and went, the documentation was rarely up to date, and the verbal training was quickly forgotten.

We've examined and systematically analyzed countless applications, and one thing has become particularly clear: this pattern is most evident in in-house applications.

Before making any decisions about the future direction of software development, it helps to get a clear picture of the current facts.

One central premise holds, independent of any software documentation or user manual: You can only trust the code itself and the way it is actually being used.

The code itself can be captured quickly and systematically. Technical deficiencies such as unreachable code, redundancies, and architectural missteps can be identified and addressed using software parsers (e.g. PITSS.CON Pro Find).

User behaviour, however, is considerably harder to analyse. Interviews often fall short here: a desired process may be presented instead of the actual one, individuals may not have the full picture, key users may hold a skewed view of what the broader user base actually does, or certain features may simply be unknown to them.

There is also the ever-present risk of misunderstandings and subjective interpretation when relying on written or verbal accounts.

To generate more concrete facts, we ran a logging process for over a year on an in-house application, a student record management system at a university in England, capturing every single interaction in every user session. In total, this amounted to nearly 90,000 hours of monitoring across 25,000 sessions, roughly 200 working days with an average of 120 users active each day. The logging was minimally invasive: not a single line of application code had to be changed, recompiled, or redeployed. Infrastructure-level performance tuning allowed data to be captured without users noticing any impact. The collected data contained no reference to the logged-in user, making it impossible to trace any activity back to a specific individual.

The results were striking. Of the application's 360 form modules, 136 were never opened during the entire period. 73% of the total codebase was never executed even once, and 4 out of 5 triggers or events never fired at all. The savings potential and room for optimisation hiding in those numbers is enormous.

At the same time, activity was detected in a part of the application that had supposedly already been fully migrated to a new technology. Before the project had even kicked off, the IT department's opening position had been: "We don't really need that section anymore, there's a new app for that now." Had part of the user base simply missed the memo? Were there functions overlooked during the redevelopment? Or was the old way simply faster for many users than learning the new app? Thanks to the granular level of detail, which made it possible to distinguish regular usage from accidental or one-off interactions, this question was answered quickly.

Of course, 26% code coverage over this period does not automatically mean that 74% of the code can simply be deleted. But it does enable an important shift in perspective. In a first pass, unused features can be assigned to the category of "infrequent activities" year-end closing, annual inventory counts, and the like. This is where the environment-agnostic nature of the approach pays off: monitoring data from different systems production and Q&A can be combined. Individual features can then be deliberately exercised in the Q& A environment and folded into the overall picture.

What remains is the large bin of "we used to need that" and "we do it differently now" features. Once those have been cleaned out and the remaining processes are well understood and modelled, the best possible foundation for a successful legacy modernisation is in place.

How do you approach software modernisation? Interviews, code analysis, logging, AI? What are your go-to tools for those first steps?

Free one-on-one consultation

Discuss your individual Oracle Forms challenges with our experts and turn them into actionable solutions you can apply immediately.