It should be possible to re-save various version of FileMaker Pro databases, but only older format to newer one and not the other way. So a .fm5 to .fmp12 conversion should be doable using FileMaker Pro 12 or possibly later.
Convert FileMaker 5 database file to FileMaker Pro database file.
We found 2 software records in the database related to .fm5 to .fmp12 file format conversion.
Microsoft Windows software - convert fm5 to fmp12 on Windows
A program used to create and manage information databases
Convert from fm5 file format
The tables with software actions are a good pointer for what a certain program does with various file types and often may give users a good hint how to perform certain file conversion, for example the above-mentioned fm5 to fmp12. However, it is not perfect and sometimes can show results which are not really usable because of how certain programs work with files and the possible conversion is thus actually not possible at all.
Apple macOS / Mac OS X software - convert fm5 to fmp12 on OS X
A database management program for Mac