The unexpected result

[pribadihcr]

I have trained controlnet using this tutorial https://github.com/huggingface/diffusers/blob/main/examples/controlnet/README.md with custom dataset

The conditioning image

the generated image:

But the X-Adapter didn't produce the expected result.

I made some change in inference_controlnet.py to adapt with the condition_type arg

if args.condition_type == "canny":
        controlnet_path = args.controlnet_canny_path
        canny = CannyDetector()
    elif args.condition_type == "depth":
        controlnet_path = args.controlnet_depth_path  # todo: haven't defined in args
        depth = MidasDetector.from_pretrained("lllyasviel/Annotators")
    elif args.condition_type == "mask":
        controlnet_path = args.controlnet_mask_path
    else:
        raise NotImplementedError("not implemented yet")

    prompt = args.prompt
    if args.prompt_sd1_5 is None:
        prompt_sd1_5 = prompt
    else:
        prompt_sd1_5 = args.prompt_sd1_5

    if args.negative_prompt is None:
        negative_prompt = ""
    else:
        negative_prompt = args.negative_prompt

    torch.set_grad_enabled(False)
    torch.backends.cudnn.benchmark = True

    # load controlnet
    print(controlnet_path)
    
    controlnet = ControlNetModel.from_pretrained(
        controlnet_path, torch_dtype=weight_dtype
    )
    print('successfully load controlnet')

    input_image = Image.open(args.input_image_path)
    # input_image = input_image.resize((512, 512), Image.LANCZOS)
    input_image = input_image.resize((args.width_sd1_5, args.height_sd1_5), Image.LANCZOS)
    if args.condition_type == "canny":
        control_image = canny(input_image)
        control_image.save(f'{args.save_path}/{prompt[:10]}_canny_condition.png')
    elif args.condition_type == "depth":
        control_image = depth(input_image)
        control_image.save(f'{args.save_path}/{prompt[:10]}_depth_condition.png')
    elif args.condition_type == "mask":
        control_image = input_image
        control_image.save(f'{args.save_path}/{prompt[:10]}_mask_condition.png')

the command:

python inference.py --plugin_type "controlnet" --prompt "a metal_nut with a bent" --condition_type "mask" --input_image_path ".mvtec/metal_nut/bent/source/179_triple.png"  --controlnet_condition_scale_list 1.0 2.0 --adapter_guidance_start_list 1.00 --adapter_condition_scale_list 1.0 1.20 --height 1024 --width 1024 --height_sd1_5 512 --width_sd1_5 512

the screenshot of conditioning and the generated image: