If you need to create a DMG installer from a APP file on Mac OS X, you can for example use the App2DMG utility that can perform this task simply by dragging the *.app file to the programs windows. It is not actual conversion, more of creating one file from another.
Convert MAC application package or core system service file to Apple Mac disk image.
We have found one software suitable in our database for this conversion.
Apple macOS / Mac OS X software - convert app to dmg on OS X
A simple APP to DMG creator
Convert from app file format
The table with program actions contains information about what each program is capable of doing with their files. This may be a good pointer for further searches for certain file conversions and help you find the converter you are looking for if our search result is insufficient for you. However, it is far from perfect and may sometimes show results which are not really usable because of the nature of the software handles the files. So a conversion that appears as possible, just because of matched actions, may in fact not be possible.