Introduction
Labeling costs often dominate machine learning projects, especially when one class is rare. This article explains how combining active learning with adaptive sampling can cut those costs for imbalanced datasets while keeping model quality high. The goal is practical: concrete steps, code snippets, and trade offs to help you design an efficient labeling pipeline.
Why class imbalance and labeling cost pair poorly
Imbalanced data means the signal for minority classes can be overwhelmed. Random sampling for labels tends to waste budget on redundant majority examples. Manual labeling time can be expensive and slow. Active selection of informative examples plus adaptive adjustments to sampling probability can focus human effort where it matters most.
Inefficient learning when the model rarely sees minority examples
Label budget waste on majority class examples with low information gain
Evaluation mismatch if metrics are not aligned with the business objective
Core idea: active learning with adaptive sampling
Combine two concepts
Active learning: select examples that the current model finds most informative, such as highest uncertainty
Adaptive sampling: change sampling probabilities to favor underrepresented or high value regions, for example increase sampling weight for minority clusters
Together they reduce labeling cost by steering label requests toward items that simultaneously improve minority class decision boundaries and reduce model uncertainty.
Simple pipeline
Initial seed: small stratified labeled set or synthetic examples to bootstrap a classifier
Model: a robust classifier that supports incremental updates or quick retraining
Acquisition function: uncertainty sampling, margin sampling, or expected model change
Adaptive sampler: compute sampling weights by combining class imbalance and model uncertainty, then sample the unlabeled pool
Labeling loop: query labels for selected items, retrain, update weights, repeat until budget exhausted or target metric reached
Hands on example with Python
The example uses a synthetic imbalanced dataset and an active learner that queries by uncertainty. Sampling weights are adjusted every iteration to boost minority examples and items near the decision boundary. The code purposely avoids heavy strings to be easy to adapt.
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, f1_score
import numpy as np
from imblearn.under_sampling import RandomUnderSampler
# fast active learning helper
def uncertainty_scores(probs):
# smallest margin between top two predicted probabilities
part = np.sort(probs, axis=1)[:, -2:]
margins = part[:, 1] - part[:, 0]
return 1.0 - margins
# build synthetic imbalanced data
X, y = make_classification(n_samples=5000, n_features=20, n_informative=6,
weights=[0.97, 0.03], flip_y=0.01, random_state=0)
# initial labeled seed: small stratified sample
X_train_pool, X_test, y_train_pool, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=1)
X_seed, X_pool, y_seed, y_pool = train_test_split(X_train_pool, y_train_pool, test_size=0.98, stratify=y_train_pool, random_state=2)
clf = RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=0)
clf.fit(X_seed, y_seed)
budget = 300
batch_size = 20
labeled_X = X_seed.copy()
labeled_y = y_seed.copy()
for _ in range(budget // batch_size):
probs = clf.predict_proba(X_pool)
u_scores = uncertainty_scores(probs)
# base sampling weight: favor minority using current predictions and imbalance
preds = clf.predict(X_pool)
minority_mask = preds == 1
base_weights = np.where(minority_mask, 1.5, 1.0)
# combine uncertainty and base weights to create selection probability
combined = u_scores * 0.7 + base_weights * 0.3
combined = combined / combined.sum()
# sample candidate indices
idxs = np.random.choice(len(X_pool), size=batch_size, replace=False, p=combined)
# simulate oracle labeling
new_X = X_pool[idxs]
new_y = y_pool[idxs]
# update labeled set
labeled_X = np.vstack([labeled_X, new_X])
labeled_y = np.hstack([labeled_y, new_y])
# remove queried from pool
mask = np.ones(len(X_pool), dtype=bool)
mask[idxs] = False
X_pool = X_pool[mask]
y_pool = y_pool[mask]
# optional undersample majority for training step to balance
rus = RandomUnderSampler(sampling_strategy=0.5, random_state=0)
X_bal, y_bal = rus.fit_resample(labeled_X, labeled_y)
clf.fit(X_bal, y_bal)
# evaluate
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))Design choices and why they matter
Every component has trade offs. Consider these points when designing a production workflow.
Acquisition function: uncertainty sampling is simple and effective, but it can focus on outliers. Margin or entropy variants can be more stable
Sampling adjustment: boosting minority weight helps find rare events, but too high weight can degrade overall performance
Batch size: larger batches reduce retraining overhead while increasing label redundancy. Small batches improve responsiveness
Stopping criteria: stop on validation metric plateauing or when marginal gain per label falls below a threshold
Evaluation tips for imbalanced scenarios
Use metrics that reflect business needs. Precision, recall, and F1 for the minority class are often more informative than plain accuracy. Consider cost sensitive evaluation if labeling or misclassification have asymmetric costs.
Precision and recall for minority class
Area under precision recall curve when the positive rate is tiny
Calibration checks to ensure probability outputs are informative for sampling
Scaling to larger pools
When the unlabeled pool is large, compute selection scores on a representative subset or use approximate nearest neighbors to find boundary regions. Parallelize scoring and prefer models with fast inference for iterative loops.
Practical pitfalls
Label noise can mislead the acquisition function. Add redundancy or consensus labeling for uncertain items
Overfitting the seed if initial labeled set is too small or biased
Class drift in production demands periodic rebalancing of sampling strategy
Monitoring and human in the loop review are essential when minority class examples are high risk or rare by nature.
Extensions and libraries
Several libraries can accelerate development. For classical workflows consider sklearn and imblearn for resampling. For active learning specific helpers try modAL or ALiPy. Deep learning setups can integrate uncertainty via Monte Carlo dropout or ensembles and request small batches for labeling.
Checklist before you start
Define clear success metric aligned with business goals
Prepare a small but representative initial labeled set
Choose an acquisition function and a conservative adaptive weight schedule
Plan for evaluation on held out data and for label quality assurance
Following a structured approach helps keep annotation costs reasonable while improving minority class performance. Iteration and measurement are the best safeguards against over optimist expectations.