Hack to extract historical data from TOS

curiousjeff

Active Member
This will allow to extract information as far back as is available in TOS for most equities, futures and ETFs.
This applies to futures with rolling contracts, indicators like the ones below, studies like ForceIndex, etc...
You can extract prices of options on a 1 minute interval over the last 10 days.

1601544878355.png

Anything that can output a single value in a chart or study should work.
For complex options positions, you can concatenate all the premiums and price of underlying for example in a single string and process after.

This is the link to the post where I found this hack:

STEPS:

1) Place the "Strategy" in a chart. !!!! This is a STRATEGY !!!, not a STUDY

1601545111594.png

2) Modify the STRATEGY to output the data you want.
3) Select the time frame and the amount of years you want to extract as you do in any chart (I have only tested using a daily time frame)
4) In the chart, right click on the text of the strategy and select "show report" and let it prepare the report
1601545285618.png
5) Save the report using the export button

1601547337211.png


6) Clean up the saved file.

I have done this manually in NOTEPAD and Excel. I am including a Python script to automate this, but you will need to have Python on your system for this to work.

My file has semi-colons instead of commas (maybe because I am in Europe ?).

In Notepad
Remove commas if they are used as a thousand separator
Replace semicolon with commas
Remove the following text surrounding the data .

1601546011906.png

The result:
1601546175773.png

The rest of the data (columns, rows) must be removed in Excel.

This is the end result once reordered from most recent to oldest.

1601546292778.png



!!!!!!! VERY IMPORTANT: When you have finished cleaning your file, you must SHIFT all the data up by ONE row to match correctly the data and the date.

The Python script will handle all the cleaning and shifting.

Here is the STRATEGY to past into the TOS chart.

To change the output, you need to change the value of

def MY_DATA = (close("$TVOLND"));

In this example, it is the daily close of $TVOLND: Nasdaq total volume

To use a study, you need to place the Thinkscript for the study above the line.
I have used the TOS Pricer for the STT and have extracted the price for each leg and SPX. This works well for the last 10 days.
(This is a little long to explain, if interested, write to me)

def MY_DATA = (close("$TVOLND"));

and make sure that the value is passed to MY_DATA

Included is the Python script.
To use the script, rename the report: "MY_INPUT.csv" and place it in the same folder as the script. Run the script.
The output will be : "MY_OUTPUT.csv"

The result will be reordered from latest to oldest.

I have used it to get:
ForceIndex using /ES
Many of Ron's studies in the Market_Conditions charts
/ES hourly price over a single year
Etc...
 

Attachments

  • TOS_EXTRACT.txt
    634 bytes · Views: 0
Last edited:
Top