This returns a 128-bit MD5 hash of the string s, which may contain binary zeroes. Unlike the utils.hash function this returns the result as a straight 16-byte (128-bit) field (that is, not converted to printable hex). If you want it in readable form you must then convert it yourself (eg. with utils.tohex).
eg.
print (utils.tohex (utils.md5 ("nick gammon")))
--> result: 9A380FD967D936AC99ED73B4A038CE8C
You can write a small Lua program to do the same thing that the md5sum program does (in Linux, Cygwin etc.):
f = io.open ("docs/RegularExpressions.txt", "rb")
if f then
print (utils.tohex (utils.md5 (f:read ("*a"))))
f:close ()
end -- if
--> result: 3764E22E2AC5BA67997C42C288253101
Compare this to the output from md5sum using Cygwin:
$ md5sum RegularExpressions.txt
3764e22e2ac5ba67997c42c288253101 *RegularExpressions.txt
The hash is the same, apart from not being in lower case, which you can change with the string.lower function if you want.