HandRefiner: Fixing Deformed Hands in AI-Generated Images 134

Solving the problem of hand deformities in AI image generation. Current image generation models perform excellently in generating images, but they often encounter problems when generating images of human hands, such as incorrect finger counts or bizarre hand shapes.

HandRefiner project address: https://github.com/wenquanlu/HandRefiner/

HandRefiner paper address: https://huggingface.co/hr16/ControlNet-HandRefiner-pruned

HandRefiner model download: https://huggingface.co/hr16/ControlNet-HandRefiner-pruned

HandRefiner proposes a method to correct abnormally shaped hand images without changing other parts of the picture. It uses a conditional inpainting approach to fix malformed hands, capable of identifying the correct hand shape and gesture, and reapplying this correct information to the original incorrect hand image.

Main features of HandRefiner:

- Accuracy: HandRefiner can accurately identify and correct malformed hands in generated images, providing an effective post-processing solution.

- Consistency: While correcting hands, it maintains consistency with other parts of the image and does not affect the overall image quality.

- Utilization of synthetic data: The research discovered a phase transition phenomenon in ControlNet, which enables HandRefiner to effectively utilize synthetic data for training without being impacted by the domain gap between real and synthetic hands. This means HandRefiner can also learn many different hand appearances, so no matter how strange the hand is, it can find a suitable way to correct it.

- Applicability: Although HandRefiner primarily targets hand images, its underlying principles and techniques can be applied to other image generation tasks that require fine corrections. For instance, this method can also be used to correct other parts, such as feet or ears.

How HandRefiner works:

1. Hand recognition and reconstruction: Identify the problem: First, HandRefiner identifies the abnormally shaped hands in the generated image. Hand reconstruction: Using a hand mesh reconstruction model, HandRefiner redraws a correct hand based on how a human hand should look. It can reconstruct the correct hand shape and gesture. This is thanks to the model's training data based on normal hands, which allows it to generate reasonable reconstruction results even from malformed hand images.

2. Conditional inpainting: Inpainting process: HandRefiner uses a conditional inpainting approach to handle the identified problem hands. It generates a depth map that contains important information about the hand's shape and position. Integration and correction: This depth map is then used as guidance and integrated into the diffusion model through ControlNet. HandRefiner will place the redrawn hand back into the original artwork, replacing the incorrectly drawn hand while leaving other parts untouched, maintaining the original artwork's style and content.