The upvoted answer does not work anymore. To get it to work you need
to manually add an axis set to [0, 0, 1, 1], or remove the patch under figure.

import matplotlib.pyplot as plt

fig = plt.figure(figsize=(5, 5), dpi=20)
ax = plt.Axes(fig, [0., 0., 1., 1.])
fig.add_axes(ax)
plt.imshow([[0, 1], [0.5, 0]], interpolation="nearest")
plt.axis('off')                                # same as: ax.set_axis_off()

plt.savefig("test.png")

Alternatively, you could just remove the patch. You don’t need to add a subplot in order to remove the paddings. This is simplified from Vlady’s answer below

fig = plt.figure(figsize=(5, 5))
fig.patch.set_visible(False)                   # turn off the patch

plt.imshow([[0, 1], [0.5, 0]], interpolation="nearest")
plt.axis('off')

plt.savefig("test.png", cmap='hot')

This is tested with version 3.0.3 on 2019/06/19. Image see bellow:

A much simpler thing to do is to use pyplot.imsave. For details, see luator’s answer bellow