> ## Documentation Index
> Fetch the complete documentation index at: https://docs.quantspace.limex.pro/llms.txt
> Use this file to discover all available pages before exploring further.

# run_data_extraction

> Download OHLCV market data and save to blob storage

## Overview

`run_data_extraction` downloads historical OHLCV (Open, High, Low, Close, Volume) and saves it to blob storage. Almost every QuantLab run starts here.

Say which tickers, dates, and source (`massive` or `yahoo`) you want — the agent fills the config.

<Note>
  **Dates**

  You normally give only the sample **start** and **end** — not a three-date train / mid / test story.

  * **Without ML/DL:** do not invent a separate testing-end beyond that end date. After portfolio optimization, the stretch you evaluate is the **full window except the first `n` trading days**, where `n` is `covariance_window` (default 40).
  * **With ML/DL:** same idea — **start and end** of the sample. A train cut inside the window (`Learning_end`) is only for the model fit vs score split; you still are not working with three user-facing period ends.
</Note>

***

## Parameters

<ParamField body="config" type="object" required>
  Data extraction configuration.

  <Expandable title="config fields">
    <ParamField body="backtest_params" type="object" required>
      Core date and source settings.

      <Expandable title="backtest_params fields">
        <ParamField body="Tickers" type="string" required>
          Ticker selection mode. Use `"custom"` to pass an explicit list via the `custom` field,
          or a preset key like `"nasdaq100"` or `"sp500"` (lists defined at the top level of config).
        </ParamField>

        <ParamField body="Data_source" type="string" required>
          Data source. Supported values: `"massive"` (default) or `"yahoo"`.

          | Value       | Notes                                                                                                           |
          | ----------- | --------------------------------------------------------------------------------------------------------------- |
          | `"massive"` | High-quality market data source under the hood. Splits adjusted by the API; dividends back-adjusted by the job. |
          | `"yahoo"`   | Free via `yfinance`, fallback if massive data is not available. Splits + dividends already adjusted.            |

          If `massive` coverage for a ticker is broken (gaps, renames), the job may fall back to Yahoo for that ticker and record it in `SourceUsed`.
        </ParamField>

        <ParamField body="Learning_start" type="string" required>
          Start of the data window. Format: `YYYY-MM-DD`.
        </ParamField>

        <ParamField body="Testing_end" type="string" required>
          End of the data window. Format: `YYYY-MM-DD`.
          On the **no-ML** path this is simply the end of the sample — not a separate “test period” flag. Evaluation after PO covers everything after the first `covariance_window` days.
        </ParamField>

        <ParamField body="Learning_end" type="string">
          **ML/DL only.** Train/test cut inside the window. Format: `YYYY-MM-DD`.
          Omit entirely when you are not running features + ML/DL (default path).
        </ParamField>
      </Expandable>
    </ParamField>

    <ParamField body="custom" type="array">
      List of ticker symbols used when `Tickers` is `"custom"`.

      Example: `["AAPL", "MSFT", "GOOGL"]`
    </ParamField>

    <ParamField body="nasdaq100" type="array">
      Predefined list of some Nasdaq-100 tickers (not all 100). Used when `Tickers` is `"nasdaq100"`.
    </ParamField>

    <ParamField body="sp500" type="array">
      Predefined list of some S\&P 500 tickers (not all 500). Used when `Tickers` is `"sp500"`.
    </ParamField>
  </Expandable>
</ParamField>

***

## Returns

```json theme={null}
{
  "status": "Succeeded",
  "output_url": "https://stqsnpprod.blob.core.windows.net/data/data_extractor_2b510101b9b7.json",
  "output_name": "data_extractor_2b510101b9b7.json",
  "execution_name": "job-data-extraction-worker-abc123xyz"
}
```

| Field            | Description                                                                      |
| ---------------- | -------------------------------------------------------------------------------- |
| `status`         | Job terminal status (`Succeeded`)                                                |
| `output_url`     | Full HTTPS URL to the output blob — pass to `run_feature_worker` or `run_po_job` |
| `output_name`    | Blob filename                                                                    |
| `execution_name` | Job execution ID for audit/debugging                                             |

The blob includes `Data`, `Tickers`, `SourceUsed`, and (when `Learning_end` was set) `Train` / `Test` / `TestSize`.

***

## Example — no ML (start + end only)

```json theme={null}
{
  "config": {
    "backtest_params": {
      "Tickers": "custom",
      "Data_source": "massive",
      "Learning_start": "2020-01-01",
      "Testing_end": "2026-03-01"
    },
    "custom": ["AAPL", "MSFT", "GOOGL", "NVDA"]
  }
}
```

## Example — ML/DL (add a train cut)

```json theme={null}
{
  "config": {
    "backtest_params": {
      "Tickers": "custom",
      "Data_source": "massive",
      "Learning_start": "2020-01-01",
      "Learning_end": "2024-01-01",
      "Testing_end": "2026-03-01"
    },
    "custom": ["AAPL", "MSFT", "GOOGL", "NVDA"]
  }
}
```

***

## Resources

| Resource           | Value                             |
| ------------------ | --------------------------------- |
| Container Apps Job | `job-data-extraction-worker`      |
| Container name     | `job-data-extraction-worker`      |
| Env var injected   | `CONFIG` (JSON-serialized config) |
| Output blob prefix | `data_extractor_`                 |

***

## Next Step

* Default path (no ML): pass `output_url` to [`run_po_job`](/tools/po-job) as `input_url`.
* ML-enhanced path: pass `output_url` to [`run_feature_worker`](/tools/feature-worker) as `input_url`.