One of our clients has a lab in which has proprietary code which performs advanced vision processing algorithms. It is currently being provided to an outside firm on a trial basis, in the form of a DLL, and is written in C++. The outside firm will be writing an application (Windows 10) which will include their own code but will reference our client's DLL. If the application becomes a product and is distributed to the firm's customers, the client wants to ensure that its DLL is only used in the manner intended, and is not pirated, widely distributed, or reverse engineered by parties other than the licensed partner.
Our client is looking for some limited consulting time from an appropriate expert to explore options and seek a recommendation on the best path forward. There is a spectrum of techniques that could be used, that vary in the level of risk containment, developer difficulty, customer experience issues, and cost. We are looking for more quantifiable metrics in terms of each of these parameters. We are looking for benchmarks of how other firms have approached similar issues and what their practical experiences have been with the solution chosen.
Options could include: encryption, obfuscation, and API design. To prevent reverse engineering of internals, there exist commercial protection schemes for this purpose, like Themida, VMProtect or Enigma and probably more. Side effects may include debuggers halting execution, interference with anti-virus tools, memory and performance issues. Beyond a DLL, a library can also be linked statically and compiled. Another option could be an online licensing system verifying a key with a server.