ControlNet’s OpenPose is the constraint type that attracted everyone’s attention at the beginning. By extracting the action pose skeleton diagram of the character in the original image, we can more accurately control the posture of the imaged character. Even some action poses that cannot be described by text description alone can be easily achieved.
OpenPose preprocessor & examples
OpenPose currently has six preprocessors: openpose, openpose_full, openpose_hand, openpose_face, openpose_face_only, and dw_openpose_full.
Let’s look directly at imaging examples to compare the differences in sensitivity (accuracy) of each preprocessor in different action postures, feature extraction, and final imaging.
First of all, let’s test with the easiest-to-recognize front-facing portrait of a person. (The prompt words are all blank)
openpose: Only extract the approximate skeleton position of the character
Since Openpose only extracts the position of the character’s whole body skeleton and lacks hand joint information, the final image may have the character’s left hand as shown below. There is no pocket on the waist, but the entire palm is missing (not drawn).
openpose_hand: Extract the approximate skeleton position of the character + hand joints
openpose_face: Extract the approximate body skeleton position of the character + facial outline/facial features position
Although the hand joints were not extracted, the SD left hand was very natural and reasonable when I painted the picture this time.
openpose_face_only: Extract only facial contours/features positions
openpose_full: Extract the approximate full body skeleton position of the character + hand joints + facial outline/facial features position
dw_openpose_full: Extract the approximate body skeleton position of the character + hand joints + facial outline/facial features position
Like openpose_full, the skeleton, face and hand joints are all extracted, but in this picture dw_openpose_full also detects the right hand where only a small part of the back of the hand is exposed. At present, most of the full-body portraits have been tested, and the sensitivity (accuracy) of dw_openpose_full is the best.
The action postures of the characters in the original picture above are relatively simple and easy to identify, and the pictures produced by each preprocessor are not very different (all can produce normal and reasonable pictures). But when the movements and postures of other characters are more complex later (depth or overlapping limbs need to be considered), the gap between the preprocessors can be clearly seen.
Next, let’s test a more difficult action posture. In the original picture below, the character’s left and right feet stagger forward and backward. (The prompt words are all blank)
As can be seen from the preprocessing diagram below, openpose_full failed to extract the right hand joints, while dw_openpose_full successfully extracted all action posture features (body skeleton, joints of both hands, face).
But as mentioned before in the depth constraint article, the character skeleton image information of Openpose is flat. For the spatial position of the left and right feet staggered forward and backward, the final imaging image cannot fully restore the effect of the original image. (Although the right foot is staggered behind, it is slightly leaning out rather than completely tucked in behind).
Maybe the difference in the depth of the character’s action poses in the original picture above is not obvious, so let’s test a more obvious example to see if the Openpose preprocessor will overturn.
(This picture needs some clues: a girl taking selfies with smart phone)
Like the previous example, the extraction effect of dw_openpose_full is better than that of openpose_full (although dw_openpose_full does not fully extract the hand joints correctly, at least both hands are recognized). However, in addition to the hand problem in the two final images, the character’s head-to-body ratio and front-to-back depth are still very weird/unnatural.
Now that we think there is a problem with the depth perspective, let’s try adding a depth constraint to assist imaging.
After adding the depth constraint, the overall feeling of the picture becomes more natural and reasonable. Or actually, taking the actions of the characters in this picture, to restore the action postures of the characters in the picture, just use a depth control constraint. openpose is not practical.
Finally, let’s take a look at the effects of feature extraction on different types of character images through openpose.
A group photo of many people in the picture: (Prompt word: man and women looking at a tablet cheerfully). This SD naturally automatically draws the tallest skeleton as a man, and the other skeletons as women.
Although there is a slight error in the final imaging of dw_openpose_full (the front and back order of the overlapping arms of the man and woman in the red box is slightly different from the original image), there is no big problem overall. Openpose_full loses more action posture information when multiple people appear in the screen at the same time and are very close to each other.
From the previous case of a group photo, we found that it seems that it is particularly easy to misidentify the intersection of the characters’ limbs in the picture, so let’s try another photo that finally caused all the preprocessors to overturn:
When it came to this picture, I was surprised that dw_openpose_full, which had always outperformed openpose_full, overturned more seriously. Neither of these two preprocessed pictures could be used directly.
But can we only be limited by the limitations of the existing preprocessor effects, and cannot generate a character skeleton diagram that fully meets our needs regardless of any action posture?
Of course not~ When all the preprocessors in Openpose cannot meet our needs, we can also use various plug-in extensions to adjust the skeleton diagram to correct the parts where the preprocessor judgment/extraction fails, or we can generate the skeleton diagram ourselves. OpenPose’s Model reads and generates images.
OpenPose skeleton diagram modification/adjustment plug-in tool
Search for “openpose” under the extensions page, and you can see that there are currently 4 related plug-in extensions available. For how to install plug-in expansion, please refer to the previous article:
→ AI Drawing-Stable Diffusion 007- Installation, update, removal and backup of plug-in extensions
Here we first install the first “sd-webui-openpose-editor-editing” to test. After the installation is complete and the SD is restarted, click “Edit” in the lower right corner of the Openpose preprocessing image window to enter the page for modifying/adjustment of the preprocessing image:
This editing screen feels straightforward to use. Slide the mouse wheel on the right canvas area to zoom in and out, and use the space bar + left mouse button to move the canvas. In the lower left corner, you can add or remove character groups by yourself, which is the same as the layer operation in Photoshop.
The only thing to note is that when you want to zoom in, out, rotate and move the entire part group (the right hand in the picture below), when the drop-down menu is not expanded, just click on the part position and the outline of the deformation operation will appear. But if the drop-down menu is expanded, you have to use the drag box to select all the joints of the hand.
If you want to make adjustments to individual nodes in the group (such as adjusting the position of the right hand joint as shown below), you must first expand the group menu, and then click on each node to operate.
After spending some time, the skeletons and hand joints of the father and daughter were finally completed. The skeleton diagram is then transferred back to ControlNet.
First, only use a single OpenPose to constrain the imaging. (Prompt word: father holding little girl, the canvas is changed to the same length and width as the original image. All other parameter settings are directly used as default)
The skeleton diagram restores the “flat” action posture, but the staggered front and back positions of the character’s limbs cannot be controlled. ~ The little girl’s left hand should cover her right hand, and the man’s right hand should be behind the little girl’s right foot to hold her buttocks ~
Maybe just by relying on the flat skeleton map, there is a chance that the front and rear positions can be exactly the same as the original image (but I only saw 1 or 2 after dozens of paintings @@), but if you just rely on luck to paint the map, it will not be The purpose of us spending more time learning SD.
Therefore, let’s try it with other constraint types.
Add one more depth constraint. Just looking at the depth preprocessing picture, at least the little girl’s right foot should be in front of the man’s right hand, but no matter how you try (even if you increase the weight of the depth constraint + depth’s Control Mode, use ControlNet More importantly), everything is still the same, and the front and back positions that the original image should have cannot be displayed.
Since the depth constraint does not work, let’s try to match it with normal. Intuitively, the effect of the normal preprocessing map should be more accurate than depth for the positional relationship that is close to the front and back. But in the final drawing, only the part where the little girl’s left hand covered her right hand was corrected, but the man’s right hand was still stubbornly drawn in front of the little girl’s right foot.
It’s really hard to understand here. The pre-processed pictures clearly indicate that the little girl’s right foot is more protruding than the man’s right hand (even if it is more important to increase the weight of the normal constraint + normal’s Control Mode and use ControlNet), Why does your SD always have to draw your right hand forward~ This way of holding is not ergonomic, right? Your right hand should be dragged on the little girl’s butt to hold her firmly~~~
Neither depth nor normal worked. Finally, I tried another one, especially adding line constraints (Lineart) to the part where the right hand is placed behind the little girl.
With Lineart (Lineart constraint weight preset 1 + Control Mode with Balanced), there is finally almost half a chance of correctly drawing the man’s right hand behind the little girl’s right foot, and the little girl’s left hand is also covering the right hand.
Moreover, since I have modified the Lineart preprocessing image here to remove lines other than non-action poses (in order to prevent the characters from looking too similar to the original image), at this time, try not to set the weight of the Lineart constraint too high or It is more important to use the Control Mode of Lineart and not to set ControlNet, because there is a high chance that the third and fourth heads will be drawn inexplicably. This is because you let the SD place too much emphasis on the Lineart pre-processed image, but in the image There are too many places where no line instructions are given, so SD has to use its imaginative skills to draw some weird and deformed pictures for you.
Finally, I want to increase the probability of correct action postures in the pictures, and open three ControlNet types at a time to constrain the imaging.
When the three ControlNets are used together, the action poses of the pictures produced each time are almost exactly what I want. However, for some reason, the pictures produced become more and more like retro old photos. It seems that this is another issue that needs to be dealt with later…
For this action, I spent a whole afternoon swiping pictures, and I sighed in my heart, give me a camera. Sometimes it’s quite tiring to guess at the pictures for a long time@@~ Why not go out and take real photos~ Take pictures and think In addition to taking photos, you can also relax your mind and body~
(PS. Use SD to paint pictures, especially if you have been using ControlNet. After using it for a long time, sometimes you feel that the pictures you paint are weird (the following image is a parameter that is abnormally adjusted too high or too low), but it is obviously It’s not that the parameters are not adjusted correctly, but it’s more like shutting down and restarting the SD when the SD is exposed. After a lot of time, all the problems that have been bothering you will disappear~)
As for the other three plug-in extensions that modify and adjust the skeleton diagram, I have only used the 3D Openpose Editor tab. If you think the other two are easier to use, you can try them out. For me, the first two (“sd-webui-openpose-editor-editing” and 3D Openpose Editor tab) are enough (and easier to operate).
The operating interface of the 3D Openpose Editor tab may not be easy to control for people who have not used 3D software. However, because it has more parameters to adjust, action postures can be adjusted more finely. Those who are interested can first go to its online web version to try out the operation, and then install it on the SD if they find it necessary (easy to use).
