Borrow a picture's structure ControlNet (canny)

Keep the shape of one picture and change everything else about it. A detector traces its edges, and the prompt decides the rest.

Same lesson, three subjects

Step by step

Skill level

  1. Source image

    1 · Attach the structure image

    keeps the shape of a picture you already have and changes everything else about it.Start by attaching the picture whose composition you want to keep, in . The example used the text-to-image result, scaled down to 512×512 to match the size the request asks for.The picture is yours to attach; nothing on this step comes from the example.
  2. Source image

    2 · Set the mode to ControlNet

    Press ControlNet in the Source image card.Your picture is now read by a detector: a small program that draws a map of the picture's structure, such as its outlines or the pose of a person in it. The new image follows that map, and your prompt decides everything else about it: the style, the colours, the materials.
  3. Source image

    3 · Choose the control type

    The control type says how a detector reads your picture.This example used canny, which traces the edges into a line map and asks the model to draw something new that follows those lines.The others read depth, pose, surface direction or rough scribbles instead, and each keeps a different part of your picture:
    • edges keep the outlines
    • depth keeps the arrangement in space
    • pose keeps the body
  4. Source image

    4 · Set the denoising strength

    is the same slider as in img2img, and here it has nothing to keep.With a control type set, the worker starts from a blank canvas. Your picture is only read by the detector that draws the map, so there is nothing from your picture left in the image for denoise to preserve.Leave it at 1, which is what the example sent as 0.75. is the setting that says how closely the image follows the map.

  5. Prompt

    5 · Prompt for the new treatment

    Ask for the look you want, not the look of the picture you attached.The lines are already being followed, so the prompt decides everything else: medium, palette, lighting.This example asked for ink and watercolour illustration of a woman in her 30s with a sleek silver bob haircut, round copper goggles resting up in her silver hair, a patched teal canvas coat, a small brass compass on a cord around her neck, at the wooden railing of a zeppelin airship's open gondola deck, the canvas envelope curving overhead, a sea of clouds below, loose brush strokes, soft washes of colour, visible paper texture, and put photograph, 3d render, lowres, blurry, deformed hands, extra fingers, watermark, text in the negative so the model cannot fall back on the realism it is tracing.

  6. Model & style

    6 · Pick an SD 1.5 model

    This one is not a free choice.ControlNet is served for Stable Diffusion 1.5 models across the horde, so an SDXL model leaves your request sitting in the queue with nobody able to take it.This example used Deliberate.

  7. Size

    7 · Match the size to the source

    Press the 512 preset so matches the picture you attached.512×512 is both the size of the source and the size an SD 1.5 model was trained at, so the two agree here.Sampling is unchanged from the earlier examples: k_euler_a, 30 steps, guidance 7.5, seed 1234567890.

  8. 8 · Send it to the horde

    Press Generate. The edge map is made on the worker, from the picture you sent, so there is nothing else to prepare.Fewer workers carry ControlNet than carry plain image models, so this one usually waits longer than the others: anything from a few seconds to a few minutes. This run took about 89 seconds and cost 23 kudos.

Where next

You can now borrow the structure of one picture for another. The next lesson is about a smaller repair: faces come out soft when they are small in the frame, and a face fixer is a post-processor that redraws them after the image is made.