This panel defines how the Divine Engine backtest is performed. But most importantly, it allows you to tell the Divine Engine the criteria you want in the strategies it produced. This panel shows up only when the Divine Engine is enabled.
Behind the scene, there are so many things going on in this panel. In this guide, we'll explore every concept thoroughly. We'll discuss how exactly the Evolutionary progression is performed, the tournaments, the cross-breeding and mutation, the rotational fitness, the calculation of fitness score, and many more.
This page is divided into four segments:
Here's the full list of properties contained in this panel (top to bottom, based on how they appear):
It's highly recommended that you familiarize yourself first with the Divine Engine and The Boss by following the tutorials here.
Note this is a huge page, so if some images don't show up, please refresh/reload the page.
1. Search Algorithm – This is the most important property, as it defines the search algorithm used by the Divine Engine.
This property fundamentally changes how the Divine Engine works in finding the best strategies. With “Evolutionary” algorithm, the Divine Engine utilizes the Evolutionary progression to create higher performing strategies: it builds up from earlier, lower-performing strategies as the foundation. Whereas the “Random” algorithm lets the Divine Engine to employ brute force to achieve its goal.
By changing this property you change most properties on the Backtest Parameterization panel, to conform with the search mode you selected. Therefore we'll start with the Evolutionary mode and discuss the properties related to it.
Evolutionary mode is best suited to create strategies from scratch, or to augment an existing strategy with new rules and filters.
2. Population Size – This property defines the amount of strategies contained within each generation.
Let us discuss first what happens at the beginning of the evolution: the Divine Engine populates the 1st Generation with diverse strategies. In the screenshot above, for example, it creates 64 different strategies.
There are two ways to do this: if you start from scratch without supplying any rules and filters, the Divine Engine adds random rules and filters (along with their random values) for each one of the 64 strategies. On the other hand, if you have an existing strategy, the Divine Engine creates other strategies as variations of your “baseline” strategy.
Such variations may come from:
Notice, the value you set on “Population Size” must be a multiple of 4. This is due to the tournament mechanism that requires 4 strategies to do battle (will be explained later). Also, it's best to set “Population Size” to a smaller value, like 64 or 128, and then do multiple Divine Engine backtests simultaneously.
We've found that a smaller Population Size may increase the likelihood of finding better performing strategies, in contrast to a large bunch of population at once (like 256). By doing multiple backtests with smaller Population Size, you can simulate “the island effect”: an evolutionary idea where different groups of people (or organisms) develop independently of each other, without being dragged down by the others' “bad gene” or fierce competition.
Then, to find the best 20 strategies from the multiple backtests, take a look at the Overall Top Results Tab:
Notes:
3. Cross-breeding Rate – This property defines the percentage of offspring created through cross-breeding (sex between parent strategies). Or put another way, it defines the amount of parent strategies that are willing to marry.
But we haven't discussed what happens after the 1st Generation was created. So let's continue our Evolutionary discussion: after the 1st Generation was created, these 64 strategies are backtested to find their Metrics' values. Therefore each strategy has their Score, CAGR, Drawdown, MAR, Sharpe, and other Metrics reported.
Once done, they'll do the tournaments to define the survival of the fittest. Each tournament consists of 4 strategies battling each other. These 4 strategies are pulled randomly from the pool of 64. Therefore with a Population Size of 64, we'll have 16 tournaments.
During the tournament, the 4 strategies are sorted top to bottom based on the Fitness Function criteria that you set. For example, if you used CAGR as the Fitness Function, the 4 strategies are sorted based on their CAGR values. The top two strategies with the highest CAGR are kept as survivors, while the bottom two are destroyed. The tournament repeats for other strategies until the pool is empty, and we're left with the 32 survivors. The pair of strategies that survived each tournament will become parents for their offspring.
These 32 survivors are now part of the 2nd Generation. The Divine Engine will then “repopulate” this Generation with their offspring. This is when each pair (the two survivors of each tournament) cross-breed. But not all parents will cross-breed; some choose to simply clone themselves to produce their offspring. This parameter “Cross-Breeding Rate” defines what percentage of parents will cross-breed, and the rest will simply clone themselves.
For example, with a “Cross-Breeding Rate” of 80%, there will be 80% of survivors to cross-breed, while 20% will clone themselves. Regardless, every pair will create another pair of offspring. Thus 32 survivors will produce exactly 32 offspring, replenishing the Population Size to its original 64 size.
Now, what happens exactly during cross-breeding? Cross-breeding essentially means: the two parent strategies pooled their respective rules together, and the children are given random rules picked from that pool. For more clarity, take a look at these illustrations:
As you can see above, cross-breeding happens in all System Settings, Buy Filters, Ranking, and Sell Filters panels. For the System Settings panel, children are given random rules (picked from the parents' System Settings pool) until their System Settings panel is entirely populated & valid.
For the Buy Filters panel, children are also given filters picked randomly from the parents' Buy Filters pool. The amount they're given is anywhere from the 1st Parent's amount to the 2nd Parent's amount. Let's say the 1st Parent contains 3 Buy Filters, while the 2nd Parent has 5 Buy Filters. A child therefore may contain 3, 4, or 5 Buy Filters. The Sell Filters and Ranking Rules have the same mechanism. Note, a child is not allowed duplicate rules, i.e two rules with the exact same type and values.
This mechanism is further influenced by the Mutation Rate, that is, with a higher mutation, a child is more likely to get a different amount of rules than its parents. Continuing with the example above, instead of a random amount between 3 and 5 filters, the child may be assigned anything from 1 to 10 amount of filters (if you use a high mutation rate, that is). In other words, the range is extended by decreasing the lower bound (3), and increasing the upper bound (5). It is so, because with higher mutation we want the offspring to be more different than their parents. And since the parents have only 8 distinct rules between them (3 and 5), should the child be given more than 8 slots, it will be capped at 8.
Essentially, cross-breeding is not about modifying the rules' values, but simply plucking random rules from the parents. A lower Cross-Breed rate means cloning is more prevalent. This may negatively affect the Evolutionary progression as strategies have less variations between them, thus little or no performance growth throughout the generations.
Notes:
4. Mutation Rate – This property defines the percent chance that a “randomizable thing” will be randomized again.
You may wonder, what's a “randomizable thing”? To answer that question, let's continue with our Evolutionary discussion: after the offspring are created (through cross-breeding and cloning), they'll undergo the mutation phase. That is, mutation does not happen for the parent strategies.
During mutation, a strategy's “randomizable things” may become randomized again. Such “randomizable things” include:
So in essence, each of those “randomizable things” takes a dice roll on whether it will be randomized again. A higher “Mutation Rate” means there's a higher chance that it will get mutated to obtain a new value (or a new rule). Hence more variation in the rules & values between strategies (diverse gene).
Be careful if you set the “Mutation Rate” too high: the Evolution may not be able to build from the previous generations, as the “DNA” is wrecked and replaced by new ones completely different from the parents.
5. Min. Generations – This property defines the minimum amount of Generations the Divine Engine will create.
We're at the full 2nd Generation now, let's continue where we left off: After the offspring are created through cross-breeding, cloning, and subsequently mutated, they will be backtested to find their Metrics' performance. This is similar to what happened previously (in the 1st Generation), except that now only the offspring are backtested. Parent strategies obviously have been backtested in the 1st Generation, so we don't need to check their Metrics' performance again.
Then all strategies within this 2nd Generation, parents and offspring alike, do battle between each other. The tournaments happen again with the exact same mechanism as already described. The survivors then become parents for the 3rd Generation, and these parents cross-breed and clone themselves to produce offspring in the 3rd Generation. Offsprings are then mutated, and the whole 3rd Generation do tournaments all over again. This cycle continues for at least the “Min. Generations” that you set here.
Therefore “Min. Generations” is the absolute minimum amount of generations that the Divine Engine creates, no matter what. Let's say you set it to 20, then 20 Generations it will be.
The larger the “Min. Generations”, the longer the backtest time will be.
6. Max. Generations – This property defines the absolute maximum amount of generations (that may or may not be created).
After the “Min. Generations” are created (in our example 20 Generations) then the Exit Conditions kick in. We'll explain about them in a moment. But essentially, they're the criteria for stopping the Evolutionary progression should it only produce lower and lower performing strategies. Like a hawk, they watch each generation's performance, deciding whether or not the Evolution must be stopped. For example, if at the 20th Generation the Exit Conditions are hit, there'll be no more tournaments, no more cross-breeding, cloning, and mutation. There will be no 21st Generation.
Conversely, if none of the Exit Conditions are hit, the cycle continues until the “Max. Generations”. Let's say you set it to 70, then 70 Generations will be created. That's the absolute maximum. There will be no more after that. The cycle stops no matter what.
A higher “Max. Generations” mean a longer backtest time. But the Evolution could mature enough with such a longer “timeline”, and you'll be rewarded with high-performing strategies. But sometimes, a longer “timeline” is a waste of compute power if subsequent generations only yield lower (or at least the same) performing strategies over and over again. That's the reason you use Exit Conditions. If you don't use Exit Conditions at all, the Divine Engine will always create strategies up to the “Max. Generations” that you specified.
7. Exit Conditions – This section of the Backtest Parameterization Panel allows you to add and customize the Exit Conditions.
So we've touched the general idea of Exit Conditions. We'll get into the details, but first we must finish our Evolutionary discussion. We'll discuss the concept of “Average Fitness Score”, and how the top strategies are picked:
The 1st Generation contains a set of population, let's say 64 strategies; and each strategy has its own Fitness Score. If you use CAGR as the Fitness Function, then each strategy's Fitness Score is calculated from the in-sample CAGR. This CAGR value is then normalized into a factor of 1, to yield the final Fitness Score.
The Divine Engine then calculates the average of the Fitness Score from those 64 strategies. This is the “Average Fitness Score” of the 1st Generation. In addition, the strategy with the highest Fitness Score is recorded as one of the “Top Strategies”, which you'll eventually see at the Divine Engine Results Tab.
This process (of finding the Average Fitness Score and the Top Strategy) continues for subsequent generations. From the 2nd Generation onward, the top strategy that's picked is the best strategy from each Generation's offspring.
That's because the 1st Generation's top strategy, being a survivor and a top parent, will always be the top strategy of the next generations unless there's an offspring strategy that can beat it. So the Divine Engine continually watches (and picks) the best offspring of each Generation, to see if they can beat the old man.
When the Evolution reaches its end, the Divine Engine picks 20 (or less) high-performing strategies from the last Generation, not just the Top Strategy there. It's so that results from the ultimate terminus of the Evolutionary progression are not wasted, based on the idea that the last Generation should benefit from all Generations behind it.
Thus our Evolutionary discussion ends, with the Top Strategies served to you. Now you may wonder, what use is the “Average Fitness Score” then? Well it serves as the input for the Exit Conditions, and we'll get to it as we explain each Exit Condition.
So let's see what types of Exit Condition we have, by clicking the button Add Exit Condition:
That opens a dialog, where you see we have four different Exit Conditions:
Select any of those Exit Conditions (by ticking their checkbox), and press OK to add them.
Average In-Sample Population Fitness Rate of Change:
This Exit Condition stops the evolution if the Average Fitness Score drops (across many generations) below the threshold Rate of Change that you specified here.
Do you recall the Average Fitness Score we discussed earlier? This Exit Condition tracks the percentage change (also known as the Rate of Change) of the latest Generation's “Average Fitness Score” and the “Average Fitness Score” of N Generations ago. Let's say the Evolutionary progression is currently at Generation 25, and its “Average Fitness Score” is 0.70. We look back ten generations (including the current generation), that's Generation 16, and its “Average Fitness Score” is 1.5. Thus from Generation 16 to 25 we have a drop of -53%. Such a large drop warrants the termination of the Evolutionary progression.
The first parameter defines the “period”, so to speak, for the Rate of Change. So if you set it to 10, the Exit Condition looks back “10 generations”, including the latest Generation. Obviously this is a moving window, so if we're currently at Generation 25, then it looks back at Generation 16. And if the Exit Condition is not met, the Divine Engine creates the Generation 26, thus we look back at Generation 17, and so on and so forth.
It is best to set a longer “period” here, so the Exit Condition is more forgiving (wider tolerance). If you set a shorter period, thus a tighter “window”, you'll get stopped out all the time as extreme fluctuations usually happen in the short term. It may reduce the chance of finding better performing strategies as the Evolutionary progression hasn't matured enough.
The second parameter defines the threshold Rate of Change (in percent). If the RoC (Rate of Change) across 10 generations falls below this threshold RoC (the percentage change is less than this threshold), the Exit Condition is hit and the Evolution stopped. Usually, we want to set this parameter at either 0%, or a big negative percentage (let's say -50%). The former indicates that we only want a “generally increasing” Fitness Score to happen instead of a decline; even if there are declines, we hope they happen for only a few generations, not a long term decline. The latter (-50%) is actually more useful, as the Evolution won't be stopped unless there's extreme RoC drop, in which case the next generations are more likely to be losers.
We don't want this parameter set to a high RoC, as that means we don't tolerate the little gradual changes (even the little negative changes) inevitable in an Evolutionary progression. High RoC is not sustainable in the long run: as the Evolution can't keep up with such a high standard, it may get stopped prematurely. Remember that this is an Exit Condition, whose sole purpose is to stop a bad Evolution, not as a fitness criteria that you want in a strategy.
In-Sample/Out-of-Sample Moving Average Rate of Change Divergence:
Quite an intimidating name for this Exit Condition, and rightfully so, as it's a combination of five concepts:
The idea here is to stop the Evolutionary search if there's great inconsistency (great divergence) between the in-sample performance and the out-of-sample performance throughout the generations. That is, in-sample “Average Fitness Score” only increases throughout generations, while the out-of-sample “Average Fitness Score” (for the exact same strategies) only decreases throughout the generations. If we continue the Evolution, the Divine Engine may yield top performing strategies indeed, but only for the in-sample period. The out-of-sample performance, on the other hand, is underwhelming. We don't want such inconsistent strategies, as their backtest performance may as well be fake, and you may lose your hard-earned money investing in them. Note, such inconsistency (great divergence) may also apply if the out-of-sample performance is much greater than the in-sample.
The first parameter defines the “period” for both the Moving Average and the Rate of Change. If you set it to 10, for example, there are 10 generations' worth of “Average Fitness Score” that will be smoothed-out to produce the Moving Average. Ditto the Rate of Change compares the Moving Average that spans 10 generations.
You may wonder why are we using Moving Average instead of the Average Fitness Score themselves? The answer is that a Moving Average gives you a more objective view for the long term progression, so the Rate of Change won't get whipsawed by outlier Average Fitness Score (spike up/down) that usually happens short-term.
It's recommended to use a longer “period” for this parameter, so the Evolution won't be stopped prematurely due to such extreme outlier values. Remember this is an Exit Condition, and we don't want the Evolution to be stopped, unless there's a chronic (long term) divergence.
The second parameter defines the threshold divergence (a.k.a. difference, or delta) between the in-sample RoC and the out-of-sample RoC. Let's say we put this threshold at 50%, and the former RoC is 30% while the latter RoC is -40%, thus a delta of 30% – (-40%) = 70%. That's a large divergence, much greater than our threshold at 50%, so the Evolution is stopped, as future generations may only yield greater inconsistency between the in-sample and out-of-sample performances.
Preferably, set this threshold at a higher percentage, so the Evolution won't get stopped by temporary small divergences that are bound to happen.
Moving Average In-Sample Population Fitness
With this Exit Condition, the Evolution stops if the latest Generation's “Average Fitness Score” drops below the Moving Average line. You're already familiar with the concept of Moving Average for the “Average Fitness Score”. Similar as in stock prices, the Moving Average acts as a “support line” where the Average Fitness Score may rebound and resume its “uptrend”. Therefore if it breached such a strong support, there's a good chance that later generations will have lower and lower score.
Note the “Average Fitness Score” is taken from the in-sample period.
The only parameter here defines the “period” for the Moving Average. Similar as in stock prices' Moving Average, the longer the period, the stronger it acts as the support line. Therefore if a breach happens, it's a significant signal that a downtrend is imminent, instead of a false positive.
Moving Average Out-of-Sample Population Fitness
This is similar to the previous one, in that the Evolution is stopped if the “Average Fitness Score” drops below the Moving Average. But the “Average Fitness Score” is calculated from the out-of-sample period.
The idea is, we don't want subsequent generations to have lower and lower performance for the out-of-sample period. Remember, the Divine Engine was blind to the out-of-sample performance as it was creating the strategies (to avoid overfitting). Thus, as the Divine Engine creates higher performing strategies based on the in-sample period, this Exit Condition makes sure the out-of-sample performance are not decreasing, at the very least.
But that's just half the story, as the Divine Engine may also create lower and lower performance for the in-sample period. So the best way is to use this Exit Condition in tandem with the previous Exit Condition (Moving Average In-Sample Population Fitness). Thus we make sure that both in-sample and out-of-sample performance are not downtrending.
The one and only parameter here defines the “period” for the Moving Average. As you already know, a higher value is preferable for a truer signal.
No New High for Out-of-Sample Moving Average
This Exit Condition looks at the out-of-sample Average Fitness Score (the green line on the Fitness Series chart), that is: During the specified “period” (including the latest generation), if there's no new all-time high for the OOS Avg. Fitness Score, then the evolution is aborted.
It could be that the green line (during the period) is all lower than the previous ATH (which is outside the period), or, it simply reached the same score as the previous ATH; in both cases the evolution will be aborted. The idea is that each new high, provided it's not too far separated from the previous one, usually signifies further improvements for the out-of-sample Avg. Fitness Score.
So at least the out-of-sample performance for the strategies created are getting better; if not, abort before it consumes too much of your compute hour credit.
There's only one parameter for this, which defines the “period” (the number of generations) to look back, starting from the latest generation. As usual, you don't want to set a short period here, since this is an Exit Condition and you must give some leeway for the evolution to mature.
Now, despite the name “… Moving Average”, this Exit Condition does not actually employ Moving Average at all. Instead, it implies the “moving window of generations” for the Average Fitness Score OOS.
Notes:
1. Search Algorithm – We'll now delve into the Random search algorithm. Obviously, we'll set this property to “Random” instead of Evolutionary.
So now we have different properties shown on the Backtest Parameterization Panel, to conform with the Random mode:
The Random mode is much simpler than Evolutionary. In fact it's the original form of the Divine Engine before Evolution was developed. If you're familiar with this tutorial, then you know a lot about the Random mode. This page is only for those craving the nitty gritty technical details.
The Random mode is best suited for fine-tuning a strategy's values. So you have an existing strategy, and you believe in its rules and filters, but you want to find values (of the parameters) that give the best possible performance. Through brute force, the Divine Engine exhaustively tests all possible values and combinations (within your parameterization ranges) and give you the best ones.
Sometimes this is the most sensible thing to do. You don't want to waste time finding a completely new system. You have a system that you trust for years. Let's say, you were a manual trader who used momentum oscillators, like RSI and CMO, along with Retracement lines as support and resistance. So you keep those filters, but let the Divine Engine find the best values. Who knows, maybe the 14-days RSI you've been using is not the optimal period? Or that the RSI levels of 70 and 30 are not the best representation of overbought and oversold conditions?
Note that a “Random” backtest can only be executed locally in your computer. You do this by clicking the button “Start Local Test”, not the button “Start the Boss”. A confirmation dialog then shows up, allowing you to add a memo for this backtest, along with a warning that a local test may hog all your computer resources:
If you don't want this obtrusive dialog to appear each time you start a local backtest, tick the checkbox “Don't show again”. Once done, click the button “Continue” to actually start the backtest. A memo is optional, as it's simply a description of each backtest you did. Each backtest (and its memo) can be seen under the Divine Engine Results Tab:
2. Execute n Random Backtests… – This property allows you to set the amount of iterations to be backtested.
An iteration consists of one strategy: a variant of your strategy with its distinct values and combinations. For example, “Iteration 1” contains the RSI Filter with the settings “10 days” and “85 RSI level”. Then “Iteration 2” has the settings “28 days” and “65 RSI level”. And so on and so forth.
This property contains three fields (of which, only one is user editable):
The idea for this calculation is that: even if you backtest only the absolute minimum of iterations, not only all values will be tested (parameterized values across all parameters), but some of their combinations as well.
Preferably, the amount shown here matches the “Number of Parameter Combinations”. You do this by adjusting the editable field previously explained.
Matching them means all parameterized values and all of their combinations will be backtested. Thus you tell the Divine Engine to do its most exhaustive search; not a single value and not a single combination will be left out. But often this means an impossibly huge iteration amount, thus an impossibly long backtest time. If that's the case, you can set the previous user-editable field to a value of 2, 3, or 5.
3. Number of Parameter Steps – This property is not editable. It shows you the amount of values (steps) that you parameterized, across all parameters.
For example on the “RSI Filter”, you parameterized its period parameter with a range of 5 to 25 days and a step of 5. That means there are 5 values parameterized (5, 10, 15, 20, and 25 days).
And then the RSI threshold is also parameterized with a range of 0 to 100 and a step of 10. That means 11 different values are parameterized (0, 10, 20, 30, 40, 50, 60, 70, 80, 90, and 100).
The ranking rule “PB Pattern Score” also has its third parameter parameterized to include all values there: “PB Velocity Score”, “PB Pattern Score”, and “PB Velocity*Pattern Score”. Thus we have 3 values parameterized here.
So across three parameters above, we have a total of 19 different values parameterized (5 + 11 + 3). As shown on the “Number of Parameter Steps”:
If you recall about the property “Execute n Random Backtests…” earlier, even if you set it to a value of 1, not only the 19 values are backtested, but some of their combinations as well. As attested by the final amount of iterations (33) that well exceeds the value of 19:
Note, when a rule is set to “Replaceable” (or “Random”), that means 2 values are parameterized.
4. Number of Parameter Combinations – This property is not editable. It shows the total amount of combinations from the parameterized values.
So, continuing from the previous example (no. 3 above), we have a total of 165 combinations. This is calculated by multiplying the amount of parameterized values across all parameters: 5 x 11 x 3 = 165.
One “combination” is a distinct mix of different values. For example, the RSI period of “5 days” is combined with the RSI Threshold of “50” along with the value “PB Velocity Score”. That's one combination. The next combination could be the RSI period of “10 days”, RSI Threshold of “70”, and the value “PB Pattern Score”, for example. And so on and so forth.
Essentially, 1 combination is 1 iteration (1 distinct strategy). That's why as stated earlier, it's best to set the property “Execute n Random Backtests…” to match this property “Number of Parameter Combinations”, so that each and every combination is tested.
5. Pulling the Top Strategies – Now that we've set the amount of iterations and start the Divine Engine backtest, Top Strategies will be pulled (recorded) to the Divine Engine Results Tab, even as the Random search is progressing.
Note, instead of “Gen.” (Generation) as in the Evolutionary search, Top Strategies are listed based on their “Iteration” number. The Iteration that beats a previously listed Iteration (in terms of its “Fitness IS” score) will then be listed on this Divine Engine Results Tab.
Unlike Evolutionary search where backtests are done in the order of earlier Generations toward the later Generations, Random search backtests lower-numbered Iterations and higher-numbered iterations at the same time. Remember that Random search assigns random values & combinations for each Iteration; there's no such thing that higher-numbered Iterations are building up from the lower-numbered Iterations.
This is how the Random search backtest is performed:
So why pull multiple top strategies instead of just the one best strategy, if the idea of Random search is to find the best values & combinations? Remember that the Divine Engine is blind to the out-of-sample performance. Thus it could be that the best of the best strategy, despite performing well on the in-sample period, is actually a loser for the out-of-sample period. Besides, different strategies may have different metrics performance (CAGR, Drawdown, etc). So you're given multiple “flavors” to choose from, to suit your preferences.
Note:
6. The Mini Chart – When the backtest is finished, each parameter (that you parameterized) has a Mini Chart. It shows the parameter's performance across the different values that you parameterized.
To open this chart, click on the colorful button beside each parameter. The Mini Chart then pops up. Note that Mini Charts are exclusive for Random search; the Evolutionary search doesn't have them, even if you parameterized the parameters.
Let's now discuss the Mini Chart's components:
If you have so many values parameterized, the horizontal axis will be crowded that some values don't have their own labels. But they're still there, plotted correctly.
We don't show the axis' labels (the Fitness Score scale) as it would clutter the chart more. Instead, you can hover your mouse on the chart to show a tooltip. The tooltip shows the various Fitness Score of a given value.
So, you hover the mouse over a certain value, let's say “25”. That value of “25” has four Fitness Score: “Avg. Fitness IS”, “Avg. Fitness OOS”, “Best Fitness IS”, and “Fitness OOS for Best Fitness IS”. Let's discuss them now:
Let's say, of all the strategies that used the value “25”, their Fitness Score (IS), averaged, is 4.1.
For example, strategies that use the value “25” (for this parameter) have the Fitness Score during the out-of-sample period averaged as 3.47.
For example, of all strategies that used the value “25”, there's one strategy with the highest Fitness Score (IS) of 4.96. Thus 4.96 is the “Best Fitness IS” for the value 25.
If you look at a parameter's Mini Chart, a value with the highest “Best Fitness IS” (highest orange line) is the value that's deemed best for that parameter and thus applied for it. Such value is shown outside the parentheses (as seen on the screenshot above).
For example, of all strategies that used the value “25”, one strategy has the highest Fitness Score in-sample. And that strategy's Fitness Score out-of-sample is shown in this brown line.
Now, sometimes you may encounter a parameter with multiple Mini Charts:
This happens if a parameter with built-in values (such as “Above” and “Below”) is parameterized. Thus another parameter (e.g. period parameter) may show multiple Mini Charts, to show its different performance when “Above” and “Below” are used.
Keep in mind, the multiple Mini Charts only account for another parameter within the same filter.
Notes:
The “Fitness Function” section allows you to define the criteria you want in the strategies created by the Divine Engine.
A Fitness Function is essentially a metric, such as CAGR, Max.Drawdown, Avg.Position Gain, Sharpe Ratio, etc, which you can find on the Metrics Tab. The metric(s) that you choose, then, becomes the basis for measuring the strategies' performance. The Divine Engine will try its best to fulfill the metric's performance that you want.
In achieving that goal, the Divine Engine employs Fitness Function in different areas:
1. The very first time a strategy has its Divine Engine enabled, a Fitness Function is already added, which is “Score”.
“Score” is good overall as a Fitness Function, but it isn't specific. You may delete it with the trash-bin icon.
To add the Fitness Function(s), click the button “Add Fitness Function”:
A dialog appears, allowing you to choose whichever metric(s) you desire:
Once they're selected, press the OK button. They're then listed under the “Fitness Function” section:
Note that you must have at least one Fitness Function for the Divine Engine to work. This applies to both Random and Evolutionary search algorithms.
2. Now we'll define the criteria. We do this by modifying the Fitness Functions' parameters:
Each Fitness Function has three parameters, as shown above. We'll discuss each parameter next.
3. The first parameter defines how the metrics' value (score) will be pursued. You can pursue as high as possible a value, as low as possible, or target a certain value as close as possible. You can also set a cap (ceiling or floor) in chasing the metric's score.
As you can see above, we have five values for this parameter:
For example, if you choose “At Least” on the CAGR Fitness Function, the Divine Engine will try to create strategies with as high a CAGR as possible.
If you choose “At Most” for the Max Drawdown, it will create strategies with the lowest possible Max Drawdown, if necessary 0%.
If you choose “Goal” for the Avg. Trades Per Year, the Divine Engine will create strategies whose amount of trades average about X trades per year. X is the Avg. Trades Per Year you want, let's say 25 trades per year. So the strategies will have as close to 25 trades per year as possible, nothing more, nothing less.
Now, “Minimum” and “Maximum” are a bit more complicated. “Minimum” takes the metric score higher to reach the minimum value that you want, but not really pursuing any higher. It could be useful for some metrics like “Avg. Price”, “Avg. Trades per Year”, “Total Trades”, and “Total Trades/Rule Count”. With “Maximum” it's the opposite: the metric score is suppressed lower to reach the maximum value you want, but anything lower is just a bonus. It could be useful to the drawdown metrics.
In other words, they're a bit like “Goal” except they don't mind a better score. If the Maximum (or Minimum) target is easy to reach, for example a “maximum of 50% Avg. Drawdown”, then the Divine Engine will pick whatever strategies as long as their Avg. Drawdown is less than 50%, be they 30%, 10%, or 49%. It won't go out of the way to suppress the drawdown as low as possible.
Notes:
4. The second parameter defines the metric score that you want. This is the score that the Divine Engine will try at least to achieve.
For example, as shown above, the Divine Engine will try to create strategies with at most 30% Max Drawdown. Any strategies with Max Drawdown above 30% will be “punished”. They're less likely to survive in the tournaments. And any strategies with Max Drawdown less than 30% (the smaller the better), are likely to survive and generate offspring.
Ditto the CAGR: the Divine Engine will try to achieve at least that 40% CAGR, but the higher the better. As for the Avg. Trades Per Year, it's already explained before. That is, anything less than or more than 25 trades per year will be punished.
Keep in mind, the “punishment” or “reward” is merely semantics. What happens is that, any strategy that falls short of your metric threshold (let's say 35% CAGR instead of 40%), will have a Fitness Score less than 1. And any strategy that exceeds that threshold will have a Fitness Score greater than 1. It could be the case that your threshold is so high (let's say 200% CAGR), that most strategies fall short of that. In that case, even if they have the Fitness Score of 0.6, for example, they'll still trump those strategies with the Fitness Score of 0.4 (besting them in the tournaments). But generally, try not to find a unicorn with your Fitness Functions.
Later on, we'll discuss the Fitness Score, and the how & why the metric score is normalized into 1.
5. The third parameter defines the “Weight” for each Fitness Function, a.k.a. its “importance”. You can set each weight anywhere from 0 to 999,999. The actual weight applied is shown inside the parentheses (in relation to other Fitness Functions, which add up to 100%).
This is especially true if you're using multiple Fitness Functions, so you can set their importance relative to each other. Essentially, weighting allows you to reduce or magnify a Fitness Function's effect in relation to the others (if you only have one Fitness Function, you can set here any number and its actual weight will always be 100%).
This could be useful if you believe a certain Fitness Function could overwhelm other Fitness Functions, hence a lopsided strategy performance. For example, if you maximize Avg. Trades Per Year, its Fitness Score could be well over 1000 (even after it's normalized into 1), whereas CAGR merely has a Fitness Score of 1.5. Therefore the Divine Engine will pursue more trades per year instead of higher CAGR. So the solution is to decrease the weight on Avg. Trades Per Year into a very small value, like 1%, and increase the CAGR's weight to 150%, for example.
So logically, weighting is used to tell the Divine Engine which Fitness Function you'd like to be emphasized. If you like CAGR to be pursued more, then increase the CAGR's weight, and/or reduce the Drawdown's weight. Because if they both have the same weight, it could be that strategies have good Drawdown but underwhelming CAGR.
Good weighting comes after trial and error. So take a look at the Divine Engine's result, and see which metric has abnormally high score, especially if it's not the primary metric you want. Take a look at the Divine Engine Results Tab:
In the case of Drawdown metrics, see if they're not abnormally small. If any metric shows aberration, you can adjust its “Weight”, or at least set it to “Goal”. Then do another Divine Engine backtest.
6. Fitness Score: Now we'll discuss how the metric(s) score are normalized into a factor of 1, to yield the final Fitness Score.
As stated earlier, Fitness Functions serve as the basis for tournaments, exit conditions, and selection of top strategies. But to do those things, Fitness Functions' performance must be converted into a Fitness Score. So for example, it's not just a strategy's raw CAGR of 28.73%, the Max. Drawdown of 19.94%, and R² of 98.81% that will be used. It's these metrics score combined and normalized into 1, which will serve as the basis for tournaments, exit conditions, and top strategy selection.
And here's how the Fitness Score is calculated:
Take a look at the example below, a Top Strategy from Generation 41:
It has a Fitness Score (In-Sample) of 1.17, which comes from a CAGR of 47.59% and the R² of 97.16%. Here are the Fitness Functions used:
So how do those percentages translate into a Fitness Score of 1.19?
First we must calculate the Fitness Score for each metric (one by one) before we combine them to yield the final Fitness Score. Understand that, in-sample metric scores become in-sample Fitness Score, whereas out-of-sample metric scores become out-of-sample Fitness Score.
The in-sample Fitness Score will be the one used for most purposes, and that's what we'll calculate. Out-of-sample is only useful for some Exit Conditions (like Moving Average Out-of-Sample and In-Sample/Out-of-Sample M.A ROC exit conditions). Hence the Divine Engine is usually blind to the OOS performance.
We'll calculate the CAGR's Fitness Score first. Here's the formula:
In our case, the 47.59% CAGR obviously “exceeds” the 35% threshold. Remember that we want a minimum of 35% CAGR, the higher the better. So if the CAGR is 23.52% for example, then it “misses” the threshold, thus we'd be using the second formula. In our case, we'll use the first formula:
As you can see above:
So the Fitness Score for CAGR is:
Next, we'll calculate the Fitness Score for R². If you notice, R² uses the “Goal” option. When the “Goal” option is used, we're certain to use the second formula. That's because a metric score is almost guaranteed to miss the threshold value; either it misses above, or below the threshold value. Remember that “Goal” tries to target the threshold as close as possible, nothing higher, nothing lower. If in the rare case that the metric score matches exactly the threshold value, then we just give it a Fitness Score of 1, no questions asked.
In our case, the metric score for R² is 97.16%, with a threshold of 95%. It has a weight of 75%.
So, the Fitness Score for R² is:
Thus we have the Fitness Score for each metric:
Now we'll find the final Fitness Score from those two combined. The formula is pretty simple:
Since we have only 2 Fitness Functions, the calculation is thus:
So that's how we get the final Fitness Score of 1.17. This Gen. 41 strategy has gone through tournaments and turns up a winner.
Notes:
Those columns can't be removed unless you delete the Fitness Functions.
Last but not least, we'll discuss the rest of the properties in this Backtest Parameterization Panel:
We'll discuss them based on their order of importance.
1. Rule Set Parameterization – This defines whether the Divine Engine is allowed to slap random new rules (filters) to your strategy. There are three values for this property:
If you choose this value (Manual), each filter may be assigned these flags (at the leftmost parameter, as shown above):
If you choose this value (Random Selection), any existing filter you have may be assigned these flags (at the leftmost parameter, as shown above):
Note, the random addition of filters may be affected by the number of parameters (and the value ranges) of each filter. More complex filters may have a higher chance of being added to the strategies. This is because with the wide and deep array of values, it's more difficult for such filters to find good value combinations. On the other hand, less complex filters may have an easier time getting good value combinations.
With this option, the rules & filters added by the Divine Engine are exclusively of this “Cyber Code” type (none of the built-in indicators), as opposed to the option “Random Selection”, which adds rules & filters of the built-in type only. So, they are mutually exclusive; either you want the Cyber Code indicators, or the built-in indicators, not both.
You can define how the new indicators are created through the Cyber Code Configuration window, accessed with the button highlighted in the screenshot above. Cyber Code Genetic Programming is a huge new feature by itself, so we'll dive deep into it in its own dedicated help page.
Note:
The option “Random Selection” can also be used for the Random Search Algorithm. That is, aside from fine-tuning existing parameters, the Divine Engine will also add random rules/filters (with their random values) to the strategies. There is no natural selection and evolution at play here, simply slapping random rules/filters for every iteration, in the hope it may beef up the strategies. Obviously rules/filters will be added for as long as there are iterations, as you see on the property “Execute n Random Backtests…”.
Therefore it's highly dependent on your parameterization of the existing parameters.
2. Use Current Rule Set as Baseline – This checkbox is used in tandem with the option “Random Selection”, explained above.
With this checkbox enabled (along with the option “Random Selection”), the Divine Engine is allowed to add new rules, and weigh whether the existing ones are useful. It's perfect for augmenting your Baseline strategy with new rules, utilizing the Evolutionary search algorithm. Ticking it ensures the Divine Engine will always give you results that are at least as good as the baseline strategy. But you must have a valid baseline strategy (complete, with its required rules; e.g. a Ranking Rule is a must). If you don't have a complete baseline and let the Divine Engine find the rest of the rules for you to complete the strategy, turn this checkbox off.
The existing filters may have the flag “Replaceable” and “Not Replaceable” similar to the previous one. But the difference lies in the “Replaceable” flag: instead of discarding the filter, the Divine Engine will weigh whether it's useful or not. The filter is more likely to be kept, but with parameter values assigned by the Divine Engine itself.
When a filter is set to “Replaceable”, you can't parameterize its parameters yourself. It's like saying to the Divine Engine, “You decide whether this filter is useful; and if it is, you decide the best values for it”. It's more efficient than flagging it as “Not Replaceable” and do the parameterization yourself; unless of course you want that filter to stay no matter what.
Such techniques can be used to guide (or kick-start) the Divine Engine to follow a certain path. For example, to avoid hindsight overfitting, the Divine Engine is supplied with a certain filter that prevents it from buying undervalued stocks, when such stocks were not yet part of an Index Portfolio (e.g. S&P 500). Such a filter could be the “Average Currency Volume Filter” set to buy above a certain dollar volume.
3. Randomize Ranking Rule Offsets – This checkbox defines whether the Ranking Rules “Offset” parameters will be randomized.
The default for these “Offset” parameters is “0 days”. If the checkbox is enabled (ticked), the Divine Engine will apply random days to those parameters. If the checkbox is disabled, they'll be applied 0 days as usual. To understand about the “Offset” parameter, read about it here.
Keep in mind, the randomization happens only to the Ranking Rules added by the Divine Engine (or if a rule is set to “Replaceable”). For those Ranking Rules you added yourself, their “Offset” parameters are unaffected by this checkbox. If you wish to randomize their “Offset”, you must parameterize them yourself.
4. Tournament Fitness Function Rotation – This property defines whether Rotational Fitness is used in the tournaments.
Rotational Fitness is a mechanism by which every tournament is given a Fitness Function different from the other tournament. As you may recall earlier, each tournament uses Fitness Functions to rank the 4 strategies' performance, thus defining the 2 survivors.
For example we have three Fitness Functions, CAGR, R2, and MAR:
The idea for Rotational Fitness is twofold:
Obviously, Rotational Fitness is only useful if you have multiple Fitness Functions. If you have but one Fitness Function, you can disable Rotational Fitness:
Now, let's examine the technical difference between the two flavors there:
Enabled – with this, the Divine Engine makes a list of all the Fitness Functions used; in our case, the list would be:
Then: the first tournament gets the first item on that list (CAGR), the second tournament gets the second item (R2), and the third tournament gets the third item (MAR). Now that the list is fully “consumed”, it'll start from the beginning again. So the fourth tournament gets CAGR, the fifth tournament R2, the sixth tournament MAR, and so on and so forth until all tournaments are done.
Note, a Fitness Function's “Weight” also determines the amount it's listed there, in relation to the amount of other Fitness Functions. For example, if all Fitness Functions are weighted equally, each will be listed once; just like what we see in the list above. But if CAGR has a weight of 100%, while R2 is 75%, and MAR is 50%, then the list becomes:
Notice CAGR is listed 4 times, R2 3 times, and MAR 2 times, to account for their different Weights relative to each other. In other words, the “Weight” parameter defines how often that Fitness Function will be assigned to all tournaments–aside from the fact that “Weight” is also used as a multiplier to the Fitness Score itself. That means, as usual, that lower “Weight” indicates less importance in defining the character of the strategies.
Enabled: use permutations – This is similar to the previous one, except the list now includes the permutations (combinations) of the Fitness Functions, aside from the individual Fitness Functions themselves. Hence the list looks like:
As usual, the “Weight” parameter also affects how the list is constructed.
Now you may wonder “What's the tangible difference for me, if I choose this or that flavor?”. Here's the answer:
5. Tournament Type – This property further modifies the way tournaments are executed.
Just like Rotational Fitness that modifies the tournaments' mechanism, this property also affects how winners are selected. When it's set to “Default”, tournaments are conducted as previously explained so far. But if you choose “Winner Takes All”, similar Fitness Functions are compared against each other (between the 4 strategies). For example, Strategy A's CAGR is compared against Strategy B's CAGR, and R2 against R2, etc.
The losing Fitness Function is then discarded altogether. It won't have any effects on that strategy's final Fitness Score. Finally, the 4 strategies are compared again as usual (based on this modified Fitness Score) to define the eventual survivors.
So each strategy doesn't have a fixed Fitness Score, which are then sorted as usual. Instead, their final Fitness Score depends on their opponent (the literal duel). By comparing the Fitness Functions individually, you basically prevent a 2nd best score to add to the final Fitness Score. That in turn, may give your strategies a slight advantage.
Notes:
Please confirm you want to block this member.
You will no longer be able to:
Please note: This action will also remove this member from your connections and send a report to the site admin. Please allow a few minutes for this process to complete.