run_data_extraction

Overview

run_data_extraction downloads historical OHLCV (Open, High, Low, Close, Volume) and saves it to blob storage. Almost every QuantLab run starts here. Say which tickers, dates, and source (massive or yahoo) you want — the agent fills the config.

DatesYou normally give only the sample start and end — not a three-date train / mid / test story.

Without ML/DL: do not invent a separate testing-end beyond that end date. After portfolio optimization, the stretch you evaluate is the full window except the first n trading days, where n is covariance_window (default 40).
With ML/DL: same idea — start and end of the sample. A train cut inside the window (Learning_end) is only for the model fit vs score split; you still are not working with three user-facing period ends.

Parameters

object

required

Data extraction configuration.

Show config fields

object

required

Core date and source settings.

Show backtest_params fields

string

required

Ticker selection mode. Use "custom" to pass an explicit list via the custom field, or a preset key like "nasdaq100" or "sp500" (lists defined at the top level of config).

string

required

Data source. Supported values: "massive" (default) or "yahoo".

Value	Notes
`"massive"`	High-quality market data source under the hood. Splits adjusted by the API; dividends back-adjusted by the job.
`"yahoo"`	Free via `yfinance`, fallback if massive data is not available. Splits + dividends already adjusted.

If massive coverage for a ticker is broken (gaps, renames), the job may fall back to Yahoo for that ticker and record it in SourceUsed.

string

required

Start of the data window. Format: YYYY-MM-DD.

string

required

End of the data window. Format: YYYY-MM-DD. On the no-ML path this is simply the end of the sample — not a separate “test period” flag. Evaluation after PO covers everything after the first covariance_window days.

string

ML/DL only. Train/test cut inside the window. Format: YYYY-MM-DD. Omit entirely when you are not running features + ML/DL (default path).

array

List of ticker symbols used when Tickers is "custom".Example: ["AAPL", "MSFT", "GOOGL"]

array

Predefined list of some Nasdaq-100 tickers (not all 100). Used when Tickers is "nasdaq100".

array

Predefined list of some S&P 500 tickers (not all 500). Used when Tickers is "sp500".

Returns

{
  "status": "Succeeded",
  "output_url": "https://stqsnpprod.blob.core.windows.net/data/data_extractor_2b510101b9b7.json",
  "output_name": "data_extractor_2b510101b9b7.json",
  "execution_name": "job-data-extraction-worker-abc123xyz"
}

Field	Description
`status`	Job terminal status (`Succeeded`)
`output_url`	Full HTTPS URL to the output blob — pass to `run_feature_worker` or `run_po_job`
`output_name`	Blob filename
`execution_name`	Job execution ID for audit/debugging

The blob includes Data, Tickers, SourceUsed, and (when Learning_end was set) Train / Test / TestSize.

Example — no ML (start + end only)

{
  "config": {
    "backtest_params": {
      "Tickers": "custom",
      "Data_source": "massive",
      "Learning_start": "2020-01-01",
      "Testing_end": "2026-03-01"
    },
    "custom": ["AAPL", "MSFT", "GOOGL", "NVDA"]
  }
}

Example — ML/DL (add a train cut)

{
  "config": {
    "backtest_params": {
      "Tickers": "custom",
      "Data_source": "massive",
      "Learning_start": "2020-01-01",
      "Learning_end": "2024-01-01",
      "Testing_end": "2026-03-01"
    },
    "custom": ["AAPL", "MSFT", "GOOGL", "NVDA"]
  }
}

Resources

Resource	Value
Container Apps Job	`job-data-extraction-worker`
Container name	`job-data-extraction-worker`
Env var injected	`CONFIG` (JSON-serialized config)
Output blob prefix	`data_extractor_`

Next Step

Default path (no ML): pass output_url to run_po_job as input_url.
ML-enhanced path: pass output_url to run_feature_worker as input_url.

Overview

How To

Tools description

Overview

Parameters

Returns

Example — no ML (start + end only)

Example — ML/DL (add a train cut)

Resources

Next Step

​Overview

​Parameters

​Returns

​Example — no ML (start + end only)

​Example — ML/DL (add a train cut)

​Resources

​Next Step

Overview

Parameters

Returns

Example — no ML (start + end only)

Example — ML/DL (add a train cut)

Resources

Next Step