What do you think about running trial transactions through a system to see how it behaves and using the observed behavior as a baseline for enhancement requests?
There are lots of "creative" things you can do to understand current functionality when there is no up-to-date documentation. Over the years, I've seen folks use the trial transaction approach you are considering. I've also seen them wade through user training manuals for the system, or examine regression test bed inputs and expected responses. And, very often, I've seen analysts ask a developer to go wading through the code to find out what the system is capable of doing.
Running trial transactions is as good a starting point as any when there is no up-to-date documentation of what the system does. Yet, you also need to understand the context for why it behaves the way it does. When enhancing an existing system, there's always a risk that the current behavior of the system – especially an older system – was based on business or regulatory or design or project constraints which were valid at the time it was built, but which no longer apply. If you understand these constraints and their context, you have an opportunity – if the benefits outweigh the costs -- to ensure that the enhancement will bring the system closer to what the business wants and needs.
So, before you rely solely on trial transactions or look at tests or code as a starting point, be sure to talk to the business subject matter experts or product owner. These folks will not only give you information about the current functionality, but may also help you understand the reasons why it works the way it does.
If there is no person who can provide the context for the current systems behavior, look for business workflows (as opposed to systems training manuals), which sometimes include some context along with the sequence of the work which need to be accomplished. Check for whether there is a history of the changes which were applied to that portion of the system, which may include the business or technical reasons for the changes.
Whether or not there is anyone available to provide "why it works that way," it's still important understand the business reasons for the system behavior going forward. So, as you work on the enhancement with your current subject matter experts or product owner, be sure to clarify the objectives or goals which the enhancement supports as well as what they expect to be different once the system is enhanced.