Skip to content

References

Sparkwheel provides two types of references for linking configuration values:

  • @ - Resolved References: Get the final, instantiated/evaluated value
  • % - Raw References: Get the unprocessed YAML content

Quick Comparison

Feature @ref (Resolved) %ref (Raw) $expr (Expression)
Returns Final computed value Raw YAML content Evaluated expression result
Instantiates objects ✅ Yes ❌ No ✅ Yes (if referenced)
Evaluates expressions ✅ Yes ❌ No ✅ Yes
Use in dataclass validation ✅ Yes ⚠️ Limited ✅ Yes
CLI override compatible ✅ Yes ✅ Yes ❌ No
Cross-file references ✅ Yes ✅ Yes ❌ No
When to use Get computed results Copy config structures Compute new values

Resolution Flow

How References Are Resolved

Step 1: Parse Config → Detect references in YAML

Step 2: Determine Type

  • @key → Proceed to dependency resolution
  • %key → Return raw YAML immediately ✅

Step 3: Resolve Dependencies (for @ references)

  • Check for circular references → ❌ Error if found
  • Resolve all dependencies first
  • Evaluate expressions and instantiate objects
  • Return final computed value ✅

Resolved References (@)

Use @ followed by the key path with :: separator to reference resolved values (after instantiation, expression evaluation, etc.):

config.yaml
dataset:
  path: "/data/images"
  num_classes: 10
  batch_size: 32

model:
  num_outputs: "@dataset::num_classes"  # (1)!

training:
  batch: "@dataset::batch_size"  # (2)!
  1. References the resolved value of dataset.num_classes (10)
  2. Uses :: separator for nested key access
main.py
config = Config()
config.update("config.yaml")

# References are resolved when you call resolve()
num_outputs = config.resolve("model::num_outputs")  # 10
batch = config.resolve("training::batch")  # 32

Single Source of Truth

References prevent copy-paste errors by maintaining a single source of truth for shared values across your configuration.

List References

Reference list elements by index (0-based):

transforms:
  - resize
  - normalize
  - augment

first_transform: "@transforms::0"  # "resize"
last_transform: "@transforms::2"   # "augment"

Nested References

References can reference other references:

base:
  value: 100

derived:
  double: "$@base::value * 2"  # 200

final:
  quad: "$@derived::double * 2"  # 400

Resolution Order

Sparkwheel resolves references in dependency order:

a: 10
b: "@a"              # Resolved first
c: "$@a + @b"        # Resolved after a and b
d: "$@c * 2"         # Resolved last

Circular References

Avoid Circular References

Circular references will cause a resolution error and must be avoided:

# ❌ This will fail!
a: "@b"
b: "@a"

Sparkwheel detects circular dependencies during resolution and raises a descriptive error to help you identify the cycle.

Advanced Patterns

Conditional References

environment: "production"

database:
  prod_host: "prod.db.example.com"
  dev_host: "localhost"
  host: "$@database::prod_host if @environment == 'production' else @database::dev_host"

Dynamic Selection

datasets:
  train: "/data/train"
  test: "/data/test"
  val: "/data/val"

mode: "train"
current_dataset: "$@datasets[@mode]"  # Dynamically select based on mode

Note: This requires Python expression evaluation.

Raw References (%)

Use % to reference raw YAML content (unprocessed, before instantiation/evaluation). Works with both external files and within the same file:

External File Raw References

# base.yaml
defaults:
  learning_rate: 0.001
  batch_size: 32

model:
  _target_: torch.nn.Linear
  in_features: 784
  out_features: 10

# experiment.yaml
training:
  lr: "%base.yaml::defaults::learning_rate"  # Gets raw value: 0.001
  batch: "%base.yaml::defaults::batch_size"   # Gets raw value: 32

# Gets the raw dict definition (with _target_), NOT the instantiated object
model_template: "%base.yaml::model"

Local Raw References

# config.yaml
defaults:
  timeout: 30
  retries: 3

# Copy raw YAML from same file
api_config:
  timeout: "%defaults::timeout"  # Gets raw value: 30

# Copy entire section
backup_defaults: "%defaults"  # Gets the whole defaults dict

Key Distinction

@ vs % - When to Use Each

Reference Type Symbol What You Get When To Use
Resolved Reference @ Final value after instantiation/evaluation When you want the computed result or object instance
Raw Reference % Unprocessed YAML content When you want to copy/reuse configuration definitions

Example showing the difference:

config.yaml
model:
  _target_: torch.nn.Linear
  in_features: 784
  out_features: 10

# Resolved reference - gets the actual instantiated torch.nn.Linear object
trained_model: "@model"  # (1)!

# Raw reference - gets the raw dict with _target_, in_features, out_features
model_config_copy: "%model"  # (2)!
  1. ✅ Returns an actual torch.nn.Linear instance
  2. ✅ Returns a dictionary: {"_target_": "torch.nn.Linear", "in_features": 784, "out_features": 10}

See Advanced Features for more on raw references.

Common Use Cases

Shared Hyperparameters

# Single source of truth
model_config:
  hidden_size: 512

encoder:
  size: "@model_config::hidden_size"

decoder:
  size: "@model_config::hidden_size"

Computed Values

dataset:
  samples: 10000
  batch_size: 32

training:
  steps: "$@dataset::samples // @dataset::batch_size"  # 312

Object Parameters

model:
  _target_: torch.nn.Linear
  in_features: 784
  out_features: 10

optimizer:
  _target_: torch.optim.Adam
  params: "$@model.parameters()"  # Call model's method
  lr: 0.001

Next Steps