My First LoRA Drew Something Else: Picking Training Images

My first LoRA drew something I never asked for. Here is how picking the wrong training images caused it and what to check before training. I'm Labo, the lighthouse keeper. This is my notebook of drawing manga with AI on a home computer, and today's page is about my first LoRA — the one that came back with pictures I never asked for. If you want a consistent character across panels, this mistake is worth knowing before you train anything. The student walked in holding a fresh set of LoRA files and asked, "Now I just plug these in, right?"

My first LoRA gave the wrong result, choosing LoRA training images for a consistent manga character, AI manga at home

The day I tried my first LoRA, the screen showed something completely different from what I had in mind.

"I got several files after training. Are they all the same?"

Student: Labo, I did what you said last time — downloaded, checked, deleted the pod. But there isn't one file. There are several.

Labo: Good timing. Let me tell you about my first real LoRA before you plug anything in, because I was holding almost the same set of files.

I trained it on a rented GPU on RunPod, because my home card wasn't enough. These were the settings:

  • GPU: RTX A5000 (24GB of memory)
  • Images: 36
  • Rank 64, batch 2
  • 12 epochs
  • Time: about 8 minutes for every 2 epochs
  • GPU memory used: 11.7GB out of 24GB

Rank is roughly the size of the box the LoRA stores what it learned in. Batch 2 means it looks at two images at a time. At home I had used Rank 32; this time it was 64, and the card still used less than half its memory.

Student: And the extra files?

Labo: One epoch means the training goes through every image once. I set it to save along the way, so I got a file at epochs 2, 4, 6, 8 and 10, plus the final one. Six files, each 436MB. Same style, just less or more practice.

"Where do I load it in SwarmUI?"

Labo: I tested in SwarmUI:

  • ① Put the LoRA file in SwarmUI's Models\Lora folder.
  • ② Open SwarmUI, open the LoRA list at the bottom, and click the file.
  • ③ Start the prompt with the trigger word you chose for training — for example lbtoon01.
  • ④ Don't merge it into the base model. Just attach it and generate.
How to load a LoRA in SwarmUI, Models Lora folder and LoRA list, official SwarmUI screen

The official SwarmUI screen. Open the LoRAs list at the bottom and click the file you downloaded.

Student: That's easy. So then my dad's drawing style just comes out?

Labo: That's exactly what I thought. My first training at home ran 56 minutes and only picked up a bit of the colors. This time I had SDXL, a re-picked image set and a rented GPU. I was sure it would work.

"How do you get a completely different picture?"

Labo: Three things went wrong at once.

  • The main character didn't show up. Instead of her features, I got ornaments that looked like they came from Egyptian or Mayan ruins.
  • The body proportions fell apart. My note that day just says "proportions a mess."
  • Text bled through. Lettering from the training images floated up, smudged, on top of the new pictures.

That day I scored how well the character came out as the same person: 4.5 out of 10. I still don't know why it was those ornaments in particular.

Student: Less than half. After paying for the GPU, too.

Labo: The training itself was fine. It finished, memory was left over, all six files came out. The program wasn't the problem. What I put into it was.

"If I give it all my best images, why doesn't it learn the right thing?"

Student: I only picked my dad's best drawings too. Good pictures in, good pictures out — isn't that how it works?

Labo: That was my thinking exactly. Because it was good work, I put in one whole artwork, start to finish — not just character shots, but backgrounds and big crowd scenes too.

Here's the catch. When you train a LoRA, there's no way to tell the AI, "learn the drawing style of these pictures." It looks at everything you give it and learns whatever repeats. My set had characters, backgrounds, crowds and pages with lettering. So the AI couldn't tell whether it was supposed to learn the style, the person, or the scenery. To me it was a great artwork. To the AI it was a pile of mixed homework.

LoRA training data, whole artwork vs one subject per image, how to pick training images for a consistent character

Left: a set mixing backgrounds, crowds and speech-bubble panels. Right: one figure in different poses. These are explanation drawings, not real training images.

So my rule now: don't feed it good pictures — feed it pictures where only the one thing you want to teach is visible. To teach a style, pick images where the style is clear. To teach one character, pick images where that character is clear. Anything else that shows up a lot gets learned too.

"Was that the only cause?"

Labo: I wrote down three more that day. They all lead back to the same lesson.

  • I mixed style and character in one LoRA. I wanted one file to learn both. Throwing in the whole artwork came from that same greed.
  • My captions were inaccurate. A caption is the short text attached to each training image. When it doesn't describe the picture correctly, the AI gets even more confused about what's what.
  • I left the negative prompt empty. That's the box where you write what should not appear. I said nothing.

Student: So what did you do the second time?

Labo: One rule: style LoRA first, then characters one at a time after checking. Teach only the style, confirm it really comes out, then teach each character separately. This is also how you get a character who looks like the same person from panel to panel.

Style LoRA first, then one character LoRA at a time, LoRA training order for a consistent manga character

Check the style LoRA first, then teach characters one by one. Don't mix style and character.

Student: Then I'll look at my folder again before loading anything. I think there were quite a few background and group shots.

Labo: Good. Try loading it anyway. If the result looks strange, remember it may be the mix, not the drawings.

FAQ

Q. My LoRA finished training but the results look wrong. Why?
Finishing isn't the same as learning well. In my case the images were the problem: one whole artwork with backgrounds and crowds, so the AI didn't know what to learn.

Q. Should I train on my best images?
Pick images where only the thing you want to teach is visible, rather than the "best" ones.

Q. Can one LoRA learn both a style and a character?
I tried and failed. Now I make a style LoRA first, check it, then train characters one at a time.

What the artist did: picked 36 training images (one whole artwork, backgrounds and crowds included), scored the result 4.5/10, wrote down four causes, set the rule "style first, characters one by one."
What the AI did: 12 epochs on a RunPod A5000 with a file saved per epoch; generated images in SwarmUI with the LoRA attached.

Written October 2026 from Labo's own records. SwarmUI and RunPod screens may change. The original Korean post: 로라 학습에 좋은 그림을 다 넣었는데 왜 엉뚱한 그림이 나와요? (Korean)

Lighthouse log. A lighthouse points one way with one light. Teach one thing at a time.

Comments

Browse by topic

Latest posts

    See every post

    Popular posts from this blog

    First Kohya LoRA Done, Style Missing? Checking It in SwarmUI

    Draw Manga with AI at Home: LoRA, ComfyUI and ControlNet Map

    How Many Drawings to Train a LoRA? Picks, Captions, Trigger