The Architecture for Serving 100 Fine-Tuned Models on One GPU

...explained with code.

Sep 11, 2026

One hundred fine-tuned variants of a 7B model can occupy 1.5 TB as merged weights, or about 19.3 GB when they share one base model.

The difference comes from what the serving stack loads into GPU memory. A rank-8 LoRA adapter contains about 40 MB of task-specific weights. Merging it produces another 15.2 GB copy of the model.

A one-endpoint-per-variant layout also creates 100 separate scaling pools. Each one has its own workers, cold starts, and idle capacity.

Keeping the adapters separate changes the layout. One 15.2 GB base model and roughly 4 GB of adapters leave about 60 GB of an 80 GB GPU for KV cache.

Each request still reaches its fine-tuned variant. vLLM applies the requested adapter while every variant reuses the same base weights.

Two memory layouts on the same 80 GB GPU. Left shows a single merged 15.2 GB model filling one slot with the rest of the card unusable for other variants. Right shows one 15.2 GB base block plus a strip of 100 small adapter blocks totalling 4 GB, with the remaining 60 GB labelled as KV cache. Both annotated with total capacity in variants served.

Memory is the obvious savings, but worker reuse matters more in production. A shared endpoint routes every request to the same pool, while separate endpoints can leave one model warm as another request waits for a new worker.

In this article, we will test both layouts under the same traffic. We will train the adapters on a Pod, serve them through vLLM on Runpod Serverless, and compare one shared endpoint with separate per-model endpoints.

Runpod Serverless works well for this test because flex workers scale to zero and bill per second. Its worker states and request timing let us separate GPU startup time from time spent waiting for capacity.

Four deployment types are available:

Four deployment shapes side by side, each showing what sits in GPU memory: merged (one full model per variant, GPU count scales with variant count), unmerged registered-at-startup (shared base plus preloaded adapters), unmerged resolved-at-request-time (shared base plus adapters fetched from storage on first use), and hosted per-tenant (provider-managed base and adapters, billed per token); labels highlight that only the merged shape multiplies GPU count by variant count.

Merged, one endpoint per variant → The adapter becomes part of a complete copy of the base model.
Unmerged, registered at startup → One base model is shared, and every adapter loads when the worker starts.
Unmerged, resolved at request time → The engine fetches an adapter when a request first needs it.
Hosted per tenant → The provider manages the base model and adapters behind an API.

We will compare all four, build the shared-base layout on Runpod, and measure it against one endpoint per model. To follow along, create a Runpod account and work through the build below.

A quick note on the terminology:
Adapter is a small set of trained matrices that store the fine-tuning updates.
An endpoint is the API and scaling configuration. A worker is the running process attached to a GPU. So one endpoint may start several workers, and each worker loads the configured model weights.

Let’s begin!

1) Merged, one endpoint per variant

Merging adds the adapter’s changes into the base model so the serving engine no longer needs to load or apply the adapter separately.

That simplicity costs memory since each fine-tuned variant becomes another 15.2 GB copy of the model. Three variants require 45.6 GB, even though each one started as a 40 MB adapter.

Each variant also has its own endpoint and workers in this setup. When no worker is ready, the request must wait for Runpod to start a GPU worker and load the model.

A horizontal stacked bar breaking a cold worker's boot time into five sequential phases, waiting for GPU capacity, pulling the container image, moving weights into GPU memory, torch.compile plus CUDA graph capture, and allocating KV cache, with the compile phase shown as the dominant chunk, next to a shrunk version of the same bar showing the 324s to 91s reduction, to make clear compilation (not weight size) drives cold start cost.

RunPod measured this startup delay at 324 seconds in one test with a 32B model on two H200s, and four configuration changes reduced it to about 91 seconds without changing the model’s speed once it was running.

Separate endpoints also prevent workers from helping one another. An idle worker holding the extraction model cannot process a request for the SQL model. The SQL request starts another worker while the first GPU remains unused.

Three separate autoscaling pools, one per variant, each with its own boot cost, compile cache and idle floor. A request for variant B arrives while a worker holding variant A sits idle, with an arrow showing the request bypassing that idle worker and triggering a fresh boot.

Merging still makes sense when variants need different base models, require different GPUs, or need complete isolation. When all the adapters share one base model, separate copies mostly duplicate memory and workers.

2) Unmerged, registered at startup

The base model is loaded once. Instead of merging every fine-tune into a complete model, the worker loads the smaller adapters beside it.

Each request invokes the adapter it needs. vLLM applies that adapter to the shared base model and generates the response. Extraction, SQL, and routing requests can therefore use the same GPU without loading three complete models.

S-LoRA's unified memory pool: adapter weights paged between host memory and GPU memory on one side, KV cache growing and shrinking on the other, both drawing from a single shared GPU memory pool so neither fragments the other, contrasted with a split/fragmented pool where separate reservations waste space.

Every worker loads the same base model and adapter set, so any worker can process any variant. The worker pool grows with traffic rather than the number of fine-tunes.

One decode step on a single GPU with four concurrent requests. All four pass through one shared block of base weights. Each additionally routes through its own rank-r adapter pair, and the batched kernel processes the four different adapters in a single pass, outputs rejoining into one result.

Adding another variant now requires about 40 MB of adapter memory instead of another 15.2 GB model.

There are two practical limits.

Three ways the unmerged/registered-at-startup option breaks: adapter memory eating into KV cache capacity (4 GB of adapters costing about 70k tokens of context), a single rank-64 adapter forcing the engine to allocate at the largest rank for an entire rank-8 family, and the hard requirement that every adapter in the set share the same base model.

All adapters must come from the same base model. They also consume GPU memory, leaving less room for active requests.

3) Unmerged, resolved at request time

Registering adapters at startup (technique 2) works when the catalogue is small and rarely changes.

A larger catalogue may contain hundreds of customer adapters, while only a few receive requests at any given time.

Request-time loading keeps those adapters in storage instead of loading all of them when the worker starts.

When a request needs an adapter that is not loaded, vLLM fetches it and adds it to the running worker.

Later requests can reuse that adapter without restarting the endpoint. New adapters can also be added without another deployment.

Runtime adapter resolution flow: an incoming request names an adapter unseen by the engine, a resolver plugin fetches it from a local directory or hub repository, and it loads into GPU memory which holds only the current working set, not the full catalogue; contrast a cold adapter (fetch + load delay before first token) against a warm adapter (already resident, immediate token); takeaway is that the catalogue becomes unbounded but the first request per cold adapter pays a latency tax.

This gives you a much larger catalogue, but the first request to an unloaded adapter takes longer. It must wait for the adapter to be fetched and copied into GPU memory.

Use request-time loading when adapters change frequently, or the full catalogue cannot fit in GPU memory. For a small and stable catalogue, loading everything at startup is simpler and avoids the first-request delay.

4) Hosted per-tenant

A provider stores the adapters and serves them behind an API, usually priced per token rather than per GPU hour. The provider controls the deployment layout and limits what you can measure or tune.

There is nothing to deploy yourself. The rest of this issue assumes you want control of the GPU.

Here are all four options we discussed, summarized in a table:

Comparison of the four deployment shapes across what sits in GPU memory, catalogue limit, what a new variant costs, and what breaks each one.

And this graph shows how the required worker count changes as the catalogue grows:

Line chart with number of variants on the x-axis and worker/replica count on the y-axis, showing merged-per-variant scaling linearly upward (workers = variants times replicas) while registered-at-startup and resolved-at-request-time stay flat as variant count grows; takeaway is that only the merged topology's infrastructure cost grows with catalogue size.

The merged setup needs more workers for every new variant, while both unmerged setups reuse the same worker pool and add workers only when request volume increases.

Building option two on RunPod

This experiment measures whether one shared endpoint starts fewer workers and causes fewer cold starts than three separate endpoints.

To test this, we will first train three adapters for extraction, SQL generation, and request routing. Then we will deploy them in two ways:

One endpoint that loads the base model and all three adapters
Three endpoints that each load the base model and one adapter

Both setups will receive the same requests. We will then compare how long those requests wait, how many workers start, and how much GPU time each setup uses.

We will run it on a Runpod Pod and then move the adapters to Runpod Serverless for the deployment test.

Three stages left to right. A Pod holding the frozen base model and three task datasets produces three adapter files. The hub holds three adapter repositories. A Serverless endpoint pulls the base model plus all three adapters into every worker it starts, with a client sending requests that differ only in the model field.

Serverless reports whether workers are running, starting, throttled, or waiting for GPU capacity. That helps us separate cold-start time from model execution.

Runpod’s vLLM worker also accepts adapter settings through environment variables, so we can test both layouts without building a custom container.

The experiment uses a 1.5B Qwen model to keep the cost low. The earlier example used a 7B model, but the shared-base mechanism remains the same.

Step 1: Keys:

Create a Runpod API key. Create a Hugging Face token with write access. Export both in your local terminal:

export RUNPOD_API_KEY=”your-key-here”

export HF_TOKEN=”your-huggingface-token”

The first key creates and manages Runpod resources. The second is passed into the training Pod so it can upload the finished adapters.

Step 2: Provision a Pod:

We used Runpod’s command-line client instead of creating the machine in the dashboard. Install it and connect it to your account:

wget -qO- cli.runpod.net | sudo bash # Install on Linux
brew install runpod/runpodctl/runpodctl # Install on macos

bash <(curl -sL cli.runpod.io)

runpodctl config --apiKey “$RUNPOD_API_KEY”

Now create one Pod from Runpod’s PyTorch template:

runpodctl pod create \
  --name lora-training \
  --template-id runpod-torch-v21 \
  --gpu-id “NVIDIA GeForce RTX 4090” \
  --volume-in-gb 30 \
  --env “{\”HF_TOKEN\”:\”$HF_TOKEN\”}”

This call starts one RTX 4090, mounts a 30 GB working volume, and makes the Hugging Face token available inside the Pod. The 1.5B model does not need a larger training GPU.

The command returns a Pod ID. Once the Pod is running, retrieve its SSH command and connect to it:

runpodctl ssh info YOUR_POD_ID

Steps 3 through 6 run in that SSH session. The working volume survives a stopped Pod, but Runpod deletes it when the Pod is terminated. We upload the adapters before terminating the machine.

Step 3: Install the training packages

The PyTorch template already includes a CUDA-compatible copy of PyTorch. Keep it. We only need the fine-tuning and dataset libraries:

pip install -U trl peft datasets accelerate huggingface_hub

This is the whole software setup. trl runs supervised fine-tuning, peft creates LoRA adapters, and datasets reads the training files.

Step 4: Create three small datasets

We want routing mistakes to be obvious.

Three parallel task shapes side by side: extraction turning a shipping note into JSON output, query generation turning a schema plus question into SQL, routing turning a support message into a queue name and reason, making the point that a wrong adapter is visible in the output shape itself, not only in a metric.

So the extraction adapter returns JSON, the SQL adapter returns a query, and the routing adapter returns a support queue. These are implemented below:

import json

from pathlib import Path

def extract(i):
    note = f"Order ORD-{40000+i} has {i % 5 + 1} items. Ship to Lisbon by DHL."
    answer = json.dumps({”order_id”: f”ORD-{40000+i}”, “city”: “Lisbon”,
                         “quantity”: i % 5 + 1, “carrier”: “DHL”})
    return “Extract the order details. Reply with JSON only.”, note, answer

def sql(i):

    question = f”How many shipped orders have amount over {100+i}?”
    answer = f”SELECT COUNT(*) FROM orders WHERE status = ‘shipped’ AND amount > {100+i};”
    return “Answer with one SQLite query only.”, question, answer

def route(i):

    messages = ["My card was charged twice.", "Tracking has not moved.", "I cannot sign in."]
    queues = [”billing”, “shipping”, “account”]
    choice = i % 3
    ticket = f”Ticket {1000+i}: {messages[choice]}”
    return “Return the support queue only.”, ticket, queues[choice]

Next, we generate the data:

Path(”data”).mkdir(exist_ok=True)

for task, make_example in {”extract”: extract, “sql”: sql, “route”: route}.items():
    with open(f”data/{task}.jsonl”, “w”) as file:

        for i in range(600):
            system, user, answer = make_example(i)
            row = {
                “prompt”: [{”role”: “system”, “content”: system},
                           {”role”: “user”, “content”: user}],
                “completion”: [{”role”: “assistant”, “content”: answer}],
            }
            file.write(json.dumps(row) + “\n”)

Once done, you should see 600 rows in each file. TRL recognizes this prompt-and-completion format and trains on the answer rather than asking you to assemble a chat transcript by hand.

Step 5: Train one adapter per task

Next, we have the training script and the first half of it defines the shared base model and the LoRA adapter:

import argparse
import torch
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

parser = argparse.ArgumentParser()
parser.add_argument(”task”, choices=[”extract”, “sql”, “route”])
task = parser.parse_args().task
base = “Qwen/Qwen2.5-1.5B-Instruct”

adapter = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=[”q_proj”, “k_proj”, “v_proj”, “o_proj”,
                    “gate_proj”, “up_proj”, “down_proj”],
    task_type=”CAUSAL_LM”,
)

The rank, r=16, controls the size of the adapter. The target list tells PEFT which parts of the transformer may receive those small updates. The base weights remain frozen.

Next, the trainer:

training = SFTConfig(
    output_dir=f”adapters/{task}”,
    num_train_epochs=3,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=2,
    learning_rate=2e-4,
    max_length=512,
    save_strategy=”no”,
    report_to=”none”,
    model_init_kwargs={”dtype”: torch.bfloat16},
)

trainer = SFTTrainer(
    model=base,
    args=training,
    train_dataset=load_dataset(”json”, data_files=f”data/{task}.jsonl”, split=”train”),
    peft_config=adapter,
)

trainer.train()

trainer.save_model(f”adapters/{task}”)

A transformer block with the four attention projections and three MLP projections highlighted as the trainable LoRA targets while the rest of the frozen base model is greyed out, with an arrow to a small saved output labeled adapter config plus weights, tens of MB, separate from the base.

Train the three adapters one after another:

python train.py extract
python train.py sql
python train.py route

Each output directory should contain adapter_config.json and adapter_model.safetensors. At rank 16 across all seven projection modules, each adapter_model.safetensors is about 70 MB.

It does not contain another multi-gigabyte copy of Qwen. That is the first check that we kept the adapters separate.

Step 6: Upload the adapters

The Hugging Face CLI creates each repository if it does not exist and uploads the matching folder. Replace YOUR_USERNAME before running these commands:

hf upload YOUR_USERNAME/qwen15-lora-extract adapters/extract .
hf upload YOUR_USERNAME/qwen15-lora-sql adapters/sql .
hf upload YOUR_USERNAME/qwen15-lora-route adapters/route .

Open each repository and check that both adapter files are present. You can now stop or terminate the training Pod.

Step 7: Deploy the shared Serverless endpoint

Return to the local terminal from step 1. Search the Runpod Hub for the current vLLM worker:

runpodctl hub search vllm

Copy the ID of Runpod’s vLLM listing. Save it along with your Hugging Face username and the adapter list:

export VLLM_HUB_ID=”id-from-the-search-result”
export HF_USER=”your-hugging-face-username”

export LORA_MODULES=”[
  {\”name\”:\”extract\”,\”path\”:\”$HF_USER/qwen15-lora-extract\”},
  {\”name\”:\”sql\”,\”path\”:\”$HF_USER/qwen15-lora-sql\”},
  {\”name\”:\”route\”,\”path\”:\”$HF_USER/qwen15-lora-route\”}
]”

Now create the endpoint:

runpodctl serverless create \
  --hub-id “$VLLM_HUB_ID” \
  --name qwen-lora-shared \
  --gpu-id “NVIDIA GeForce RTX 4090” \
  --workers-min 0 \
  --workers-max 3 \
  --idle-timeout 5 \
  --env MODEL_NAME=Qwen/Qwen2.5-1.5B-Instruct \
  --env ENABLE_LORA=true \
  --env MAX_LORAS=3 \
  --env MAX_LORA_RANK=16 \
  --env “LORA_MODULES=$LORA_MODULES”

When you deploy from a Hub ID, the GPU type and container disk come from the Hub release.

The --gpu-id flag overrides that default.
ENABLE_LORA turns on adapter support.
MAX_LORAS=3 lets requests for all three adapters share one batch.
LORA_MODULES registers the names that later appear in each request’s model field.

Two distinct configuration surfaces for the shared endpoint: one panel listing the four environment variables (ENABLE_LORA, MAX_LORAS, MAX_LORA_RANK, VLLM_EXTRA_ARGS) that carry adapter configuration into the worker, and a separate panel showing endpoint settings (min workers 0, max workers 3) that controls scaling.

The endpoint starts with no workers and can scale to three. This lets it return to zero after its five-second idle period, which we use for the cold-start test in step 10.

The command returns the new endpoint ID. Save it and inspect the deployment:

export ENDPOINT_ID=”id-from-the-create-result”

A registration check comparison showing a correct response listing four model ids, one base plus three adapters, next to a silent-failure response listing only the single base id, labeling the missing adapter ids as the signal that every downstream measurement is quietly running against the base model alone.

Step 8: Confirm the adapters loaded

curl -s "https://api.runpod.ai/v2/$ENDPOINT_ID/openai/v1/models" \
  -H "Authorization: Bearer $RUNPOD_API_KEY" | python -m json.tool

This request starts a worker if none is running, so expect it to take a while the first time. The response should list the base model plus extract, sql, and route. If you see only the base model, stop here. The endpoint did not load the adapters, so every later measurement would test the wrong thing.

Step 9: Send one request to each adapter

The URL stays the same. Only the model field changes:

curl -s “https://api.runpod.ai/v2/$ENDPOINT_ID/openai/v1/chat/completions” \
  -H “Authorization: Bearer $RUNPOD_API_KEY” \
  -H “Content-Type: application/json” \
  -d ‘{”model”:”sql”,”messages”:[{”role”:”user”,”content”:”How many shipped orders have amount over 500?”}],”max_tokens”:48}’

Run the same request with extract and route, using a matching prompt for each task. If the outputs have the expected formats, one endpoint is switching between three fine-tunes.

You can automate a warm latency check with a small Python script. Install openai in that local shell if you do not already have it. This measures the complete request, which is enough to catch a large switching penalty:

import os, statistics, time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ[”RUNPOD_API_KEY”],
    base_url=f”https://api.runpod.ai/v2/{os.environ[’ENDPOINT_ID’]}/openai/v1”,
)

prompts = {
    “extract”: “Extract: Order ORD-40100 has 2 items. Ship to Lisbon by DHL.”,
    “sql”: “How many shipped orders have amount over 500?”,
    “route”: “Tracking has not moved.”,
}

for model, prompt in prompts.items():

    times = []
    for _ in range(5):

        started = time.perf_counter()
        client.chat.completions.create(
            model=model,
            messages=[{”role”: “user”, “content”: prompt}],
            max_tokens=48,
            temperature=0,
        )
        times.append(time.perf_counter() - started)
    print(model, f”{statistics.median(times):.2f}s median”)

In our warm tests, the adapters stayed within the base model’s normal variation. The serving layout saved memory without adding a visible request-time penalty.

The script prints one timing result for each adapter. Each number is the median time across five requests:

Measured warm-path results: time to first token and output tokens per second for the base model and the three adapters.

It does not print the generated answers. Run the script against your endpoint and replace these placeholders with the three values from your terminal. Those are the numbers to compare with the separate-endpoint test in step 11.

Step 10: Measure a cold start

Step 9 measured requests while a worker was already running. This test starts after the endpoint has returned to zero workers.

Check the endpoint from your local terminal:

curl -s “https://api.runpod.ai/v2/$ENDPOINT_ID/health” \
  -H “Authorization: Bearer $RUNPOD_API_KEY” | python -m json.tool

The workers section should show zero workers in every state, including throttled:

“workers”: {”idle”: 0, “initializing”: 0, “ready”: 0,
            “running”: 0, “throttled”: 0, “unhealthy”: 0}

Next, we run this script to measure the cold start stats:

import os, time, requests

base = f”https://api.runpod.ai/v2/{os.environ[’ENDPOINT_ID’]}”
headers = {”Authorization”: f”Bearer {os.environ[’RUNPOD_API_KEY’]}”}

payload = {”input”: {
    “openai_route”: “/v1/chat/completions”,
    “openai_input”:{
        “model”: “sql”,
        “messages”: [{”role”: “user”, “content”: “Count shipped orders over 500.”}],
        “max_tokens”: 48,
        “temperature”: 0,
    },
}}

response = requests.post(f”{base}/run”, headers=headers, json=payload, timeout=30)
response.raise_for_status()
job = response.json()

while True:
    response = requests.get(f”{base}/status/{job[’id’]}”, headers=headers, timeout=30)
    response.raise_for_status()
    result = response.json()

    if result[”status”] in {”COMPLETED”, “FAILED”, “CANCELLED”, “TIMED_OUT”}:
        break
    time.sleep(2)

if result[”status”] != “COMPLETED”:
    raise RuntimeError(result)

print(”status:”, result[”status”])
print(”delayTime:”, f”{result[’delayTime’] / 1000:.2f}s”)
print(”executionTime:”, f”{result[’executionTime’] / 1000:.2f}s”)

The terminal will print this:

status: COMPLETED
delayTime: 171.89s
executionTime: 0.56s

delayTime covers everything before a worker picks up the job, including worker startup and any wait for GPU capacity.
executionTime covers the request after pickup.

In this run, 99.7 percent of the measured time came before inference. The pickup delay was roughly 307 times the execution time. A model that responds in half a second can therefore take almost three minutes to reach the user after the endpoint scales to zero.

Since this test began with zero workers and one queued request, the startup path dominated unless Runpod marked the worker as throttled.

Repeat the health check and script three times. Let the endpoint return to zero workers before each run, then record both values.

Our three clean trials waited 409, 245, and 214 seconds before pickup. The median cold start was 245 seconds. Execution stayed near half a second.

That spread is uncomfortable, but useful. Most of the uncertainty came from starting the worker, not running the model.

Those numbers are also far longer than a 1.5B model on a 24 GB GPU should need, and the setup explains why. Nothing is cached: each cold start pulls the vLLM worker image, then downloads the base model and all three adapters from Hugging Face before the engine can warm up.

Runpod’s own post on the linked configuration changes cut a much larger model from 324 to 91 seconds without touching the code. Baking the model into the image or caching it on a network volume, and keeping FlashBoot enabled, would remove most of the delay we measured here. We left the default configuration in place so the two layouts stay comparable, but treat 245 seconds as an unoptimized baseline, not a property of the platform.

If Runpod reports a throttled worker, discard that trial. The request was waiting for GPU capacity, so delayTime no longer represents worker startup alone.

Step 11: Compare one endpoint with three

The shared endpoint loads all three adapters. We now create three separate endpoints that load one adapter each.

Return to the local terminal where VLLM_HUB_ID and HF_USER are already set. Define a small shell function so the common settings stay identical:

create_single_adapter_endpoint() {
  adapter=”$1”
  runpodctl serverless create \
    --hub-id “$VLLM_HUB_ID” \
    --name “qwen-lora-$adapter” \
    --gpu-id “NVIDIA GeForce RTX 4090” \
    --workers-min 0 \
    --workers-max 3 \
    --idle-timeout 5 \
    --env MODEL_NAME=Qwen/Qwen2.5-1.5B-Instruct \
    --env ENABLE_LORA=true \
    --env MAX_LORAS=1 \
    --env MAX_LORA_RANK=16 \
    --env “LORA_MODULES=[{\”name\”:\”$adapter\”,\”path\”:\”$HF_USER/qwen15-lora-$adapter\”}]”
}

Run the function once for each adapter:

create_single_adapter_endpoint extract
create_single_adapter_endpoint sql
create_single_adapter_endpoint route

Each command prints the endpoint it created. Copy the three endpoint IDs into environment variables:

export EXTRACT_ENDPOINT_ID=”id-from-extract-create-result”
export SQL_ENDPOINT_ID=”id-from-sql-create-result”
export ROUTE_ENDPOINT_ID=”id-from-route-create-result”

Confirm each endpoint registered its adapter the same way as in step 8:

for id in \
  “$EXTRACT_ENDPOINT_ID” \
  “$SQL_ENDPOINT_ID” \
  “$ROUTE_ENDPOINT_ID”
do

curl -s “https://api.runpod.ai/v2/$id/openai/v1/models” \ 
     -H “Authorization: Bearer $RUNPOD_API_KEY” | python -m json.tool

done

Every endpoint uses the same base model, GPU, worker limits, and five-second idle timeout. Only LORA_MODULES changes. MAX_LORAS is now 1 because each endpoint serves one adapter.

We now have two layouts:

Two topologies side by side receiving the identical mixed request stream: a single shared endpoint whose workers each hold all three adapters versus three separate single-adapter endpoints, with a background sampler polling worker counts on each to accumulate worker-seconds for comparison.

Shared layout: one endpoint, three adapters.
Separate layout: three endpoints with 1 adapter each

Send the same five-minute stream to both layouts at one request per second. Rotate through extraction, SQL, and routing. For every job, save its delayTime. Also record the highest worker count shown by Runpod and the billed worker time for the run.

Keep the rate low enough that warm workers can keep up. If requests queue behind other requests, a long delayTime may look like a cold start even when no worker is booting.

This is what happened in our run:

Measured topology comparison: peak workers, worker-seconds billed, and requests that paid a boot, for one shared endpoint against three single-adapter endpoints.

Note that the separate layout used fewer billed worker-seconds, but not because its workers idled out. At one request every three seconds per endpoint, a warm worker never reaches the five-second idle timeout. The separate endpoints spent most of the five minutes cold-starting, so requests queued behind the boot and far fewer completed.

It was cheaper during this short run because it completed fewer requests and made most of them wait more than two minutes.

The shared workers stayed alive because the combined traffic kept reaching the same pool. That is the operational advantage the memory calculation misses.

Step 12: Turn the measurements into a cost estimate

Use your current Runpod GPU rate rather than copying a price from this article. Rates depend on GPU type and worker mode.

For flex workers, estimate monthly cost as:

GPU rate × (request execution hours + startup hours + idle hours)

Request execution should be similar in both layouts because the models generate the same tokens. Startup and idle time are paid once per endpoint. Three separate endpoints can therefore pay that overhead three times during the same traffic cycle.

For active workers, multiply the monthly price of one always-on worker by the number of endpoints. One shared endpoint needs one warm floor. Three separate endpoints need three if every variant must respond immediately.

A monthly cost breakdown showing boot and idle overhead as a fixed per-endpoint cost block that stays constant while the traffic-driven compute block shrinks at quiet-hour request rates, illustrating why fixed overhead dominates the bill at low traffic.

Use the 245-second cold start from step 10 only as an example. Replace it with your median, then calculate busy and quiet periods separately. Startup consumes a much larger share of the bill when traffic is sporadic.

Wrapping up

A LoRA fine-tune is small until you merge it. Merging creates a complete model copy for every variant, and separate endpoints give each copy its own worker pool.

That trade is reasonable when models need different hardware or complete isolation. It wastes memory and workers when every adapter comes from the same base.

For a family of adapters on one base, my default is to keep them separate. Load one base model, attach the adapters, and let every request use the same worker pool.

The catalogue still consumes GPU memory, and all adapters must match the base, but adding a variant no longer means adding another full model and endpoint.

The experiment also gives us a better way to discuss serverless latency. Record waiting time and execution time separately. A model that runs in half a second still feels slow if the endpoint has to pull an image and download weights every time it wakes, which is why cold-start configuration deserves as much attention as the serving layout.

Runpod made that distinction visible in this test. Pods handled training, Serverless exposed the worker behaviour, and the ready-made vLLM worker let us compare the layouts without maintaining four container images.

Thanks for reading, and thanks to Runpod for partnering with us on this issue.

Daily Dose of Data Science

Discussion about this post

Ready for more?