System Messages

This forum is for members of the Apiary Investment Fund. To learn more about how to get involved, sign up now or fill out the form below and preview the training right now!

MS ML Callout

 
Back to Forums
 
 
mike_smith

MS ML Callout

https://github.com/gomlfx/apiaryfund

Legal Disclaimer: My Callout is for educational purposes only. It is not a recommendation to buy or sell. That is a personal decision that should be made by you only.

The ML callouts represent the evolution of my current trading skill. I don't claim they exist in a state of perfection or any given callout will be perfect.

 

Wed, 10/09/2019 - 1:17am
 
mike_smith

default estimators, 1.09834655 1.10213875 1.09432795

 
Rookie

Hay Pythons did u catch the bullish move to 1.09855?? On the money bro, great call.
----------------------------------------------
and it's running! on bad us/china news... surprise!

EU confluence of events.PNG
 
mike_smith

Increasing the n_estimators doesn't look like it changes the target prediction much. I use a 1971-2019 EOD dataset. So, this means I need to add parameters to increase accuracy, or change the model if adding parameters doesn't increase accuracy.

The three values I give represent close, high, low. But the market may manipulate those targets (p.close=a.low, etc), so I just give three values likely to hit. If 1 (the price in the middle (close)) hits, the other two are hedged. If 2 hit (close and 1 extrema), hopefully there's a retracement on the high or low so 1 order is full profit, and 2nd hit is some profit after 3rd target loss. If after the day a price hit an extrema but not the other two, or without hitting the middle target(price has negative distance) and either extrema, that's a loss day.

Not much difference using 100 vs 500 estimators: [[1.0983426 1.1021141 1.09444992]].
or using 5000: [[1.09838565 1.10215485 1.09445814]]

 
mike_smith

Oct 11, 0-24 GMT: 1.0992081 1.10298645 1.0953056 (default n_estimators)

You choose which is the high, low, close! Good luck!

 
Rookie

yo python, it's your call out! you chose the highest probability.

 
mike_smith

I'll keep that in mind.

 
mike_smith

Well that definitely wasn't a sell day. :)
The low was missed by 8 pips leaving 2 sells still open. I will use the old sells and change the new targets on them if the new prediction today shows two sells. If it shows two buys I will close one sell (at smallest current negative) at a loss.

 
mike_smith

I will make new predictions and see if I can limit the loss on the two open sells.

 
mike_smith

Oct 14, 0-24GMT prediction: 1.1081257, 1.10054863, 1.10431218, 0.24

The first three numbers are data points the model thinks are probable to complete. The number at the end is an additional metric for probability from 0 to 1 to by or sell. I'll start observing the last number, didn't yet. This is just the regular random forest. I cut one sell for -20pip. I am thinking alternative is to target only close each day. Close after a day either way, loss or profit. Then observe the equity curve. I could train many features vs one. Then measure which way is better.

 
robunited

Mike, what is this based on to get the numbers?

Can you give a brief rundown of it?

 
mike_smith

You mean the strategy in the code? It's based in python 3.0.
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.Rando...

I'll focus on closing price and probability for close direction. They seem to make the most sense. As the high and low are an extrema range and one side may not hit. But the closing price, which is always between the extrema range, appears to consistently hit. This way I can observe and record the close price directional probability. Currently using random forest, but will replace with bayesian optimized XGBRegressor.

Oct 15 0-24GMT: [[1.1078539 1.1002715 1.1040558 0.145 ]] I changed yesterday's open orders T/P's for today's T/P values.

 
mike_smith

A pattern I'm starting to see is the predicted close is used as a next day extrema, and the probability for direction is more consistent than a predicted close price. Since one extrema is predicted to close, I can choose the one in the predicted direction. So this can narrow to 2 high probability targets, since I know the predicted close is either way a high probability target.

Oct 16 0-24 GMT: 1.10823805 1.10062945 1.10446635 0.2

So, the above reads highest probability target =1.10446, and highest probability direction is down with target = 1.10062.

 
mike_smith

Well my model clearly needs work! The direction today hit the high not the low.

This leaves highest value prediction as the close. It appears as the most consistent target, even though on the actual day the predicted close might be a high, low, or something in between. So I'll continue to report all numbers but just report one highest probability target (p.close).

Additionally, I can see a pattern emerging, as I noted before it appears either a predicted high or low is typical to hit, but not both. On this day it was the high, although I predicted it would be the low. Even though I can't use it, as I don't know which of one will hit, but it is a significant observation that it does occur 'to the pip' (today's predicted max: 1.1082, actual: 1.1085).

better results are in the pipeline with XGBRegressor, reinforcement learning, model selectors via Sagemaker, etc.

 
mike_smith

Until the Sages set the forests ablaze... 1.10452 prediction remain.

Re: Oct 17 0-24 GMT [[1.10839225 1.10065035 1.10452 0.29 ]]

 
mike_smith

You may choose to use the three predicted values as support and resistance lines if your daytrading.

2019-10-16 23_32_24-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M5].png
 
mike_smith

Well the predicted value did not hit at all. I can experiment with GradientBoostingRegressor and hopefully this will improve from RandomForestRegressor. Using the predictions as support and resistance lines was a nice little gem I discovered yesterday. Quite nice...

 
Rookie

SageMaker, your EU chart just above is similar to my trading charts for the 5M. My charts would add a few more S&R; lines as needed and I trade between the lines/channel.

I draw multiple lines to define one channel zone from another. Typically the EU will trade in all for channels during a 4H session. It's important to know when the price action is changing channels so that one does not get overextended.

In Jeff's CATT class he teaches us to make a price and time cycle box and place one on each side pf the price point. Then divided the box into quadrants to get a feel for the percentage and momentum of the PA movement.

Food for programming thought to add time cycles to your channels.

 
mike_smith

Pls, just call me Mike! I change my title too much :D

 
mike_smith

Price and time cycles I agree would be useful inputs. A lot of models, like basic ones, need to predict off datasets. Like the ones I use now. But models like RL, reinforcement learning, which I hope to use soon, could technically learn by rules and an incoming data stream only. I mention this because I don't think Jeff or anybody has a dataset of price and time cycles. Or a way they can be generated?

 
mike_smith

I had an interesting result for GradientBoostingRegressor for Oct 16, where it fit the day more than RandomForestRegressor. The models are based on totally different techniques but are considered both effective. So, today I'll start using GBR and see if I can prove a theory I have that they're superior to RFR (https://scikit-learn.org/stable/supervised_learning.html#supervised-lear...).

initial Oct18 0-24GMT: 1.10904, 1.11673, 1.11422

 
mike_smith

Today's r/s?

2019-10-17 11_51_44-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M5].png
 
jdiazfar

This topic and ideas seem most interesting. I will follow closely. As an Engineer, probability and statistics, numbers and science and of course method/systems are my interests.

 
Rookie

@jdiazfar, as a student od these things you would really like Jeffs, group which meets on Sat am but a lot of the lesser stuff is in CATT classe on M-W-F.

 
mike_smith

probability of error 1.34 percent to hit 1.1142? Take this with a grain of salt! I was really tweaking the parameters. I learned how to use iterations, so having a blast w/ differ parameters. Here I'm testing max_features=log2. But I need to get the error ratings, accuracy scores, etc. coded so I can see what I'm doing.

2019-10-17 20_27_34-Prediction.ipynb - Colaboratory.png
 
mike_smith

I can't believe the f^#*ing morons I meet even in data science. I just met one guy online that said RL is his hobby but it couldn't work on forex. What a f^#*ing dunce.

 
mike_smith

GBRegressor did accurately predict the high and a gave a good close (hlc) target. The Random Forest didn't predict the direction right in the last two days. I may be close to proving my theory that Gradient Boosting model is superior to Random Forest.

Pls remember these are very good results, since I did not code the error score reporting yet. I'm working on that now. And since I didn't use default parameters. GBR is precursor to XGBR.

2019-10-18 13_11_11-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M5].png 2019-10-18 13_15_32-pythons EURUSD daily callout _ Apiary Fund.png
 
robunited

Would be interesting to see what the prediction for the GBP from your system were compared to the actual price we get on Monday, after the draft EU deal and UK votes over the weekend.

How do you factor in exceptional scheduled events like elections, votes etc.?

 
mike_smith

I can include GBP predictions if you like. Personally, I think eurusd is the most complex structurally, that's why I use it. There is NLP for natural language processing, assuming you can ingest the data. I don't have sources for all that type of data your talking about. I only download metatrader csv. I work off the premise that learning models like RL will recognize and manage adequately movements on a chart. I also believe to an extent that events like news are directed by predictions. In other words, predictions are calculated, then the human announcement is made. Either way, we all know price action outcomes don't always match the announced news. So, I don't see any conflict to rely on data and models for prediction, the more data the better, is the only review.

And I trust risk vs fundamentals. In fact I don't trust fundamentals at all, the shorter timeframe, the less I trust them. Unless they were data tables so I could use them as additional features for my data prediction.

2019-10-19 23_27_19-Smith's Strategy Log _ Apiary Fund.png
 
mike_smith

GBR default: 1.12148, 1.11415, 1.11964, .99

XGBR (http://tqchen.com) coming very soon! -iksne outdated.

 
Rooster

Nice Work. Keep up the passion.

 
mike_smith

LGBM: 1.12434, 1.11532, 1.12298, 1 (Light Gradient Boosting Machine Regressor. 100,000 estimators in <5 min.) vs

LGBM default: 1.12097, 1.11646, 1.12091, .99

 
mike_smith

Since the targets were up, it looks like the trend lost momentum. Of course this is a symptom which can occur with the prediction. If the momentum stays the same, the targets are good. But if it's time to change the trend, you can also see that, such as today. I noticed the LGBM model shifted the prediction downward at a near mirror angle it shifted it upward yesterday. I'm new to using LGBM.

LGBM 22Oct: 1.11556,1.11003,1.11061,0
n_estimators=5000, 6 features

LGBM 22Oct default: 1.11452,1.10990, 1.11060,0
default=100 n_estimators

I'm also going to keep an LGBM 1 monthly, weekly, daily prediction (attached).
n_estimators=5000, 1 feature

As a data scientist I need to clean data, so I spend a lot of time observing datasets. At this time a particular metrader refused to download a whole 1 week of data, so I was missing it and had to find a differ mt4 to use it's data, then when I opened it I found each price on it's daily data table was different by the other mt4 by a whopping 10 pips. Yet while the volume number was absolutely identical! I crawl thinking how corrupt these brokers are. But if the volume is a universally identical number could I use it as a dependable feature?

2019-10-21 21_53_53-1050903_ RedexHoldings-Demo - Demo Account - [EURUSD,Daily].png
 
mike_smith

I think today was a regular day. I think the actual extrema would have been closer to the target predictions if the data I took didn't distort the price 10 pips, as I previously noted above. I found also the broker was using a 10 pip fixed spread. It's just like data scientists say, use clean your data!

 
mike_smith

23 Oct 0-24 GMT: 1.11516, 1.11082, 1.11280

Spread is <1 pip on this dataset.
Default LGBM

 
mike_smith

There's a trend to use classifiers vs regressors in prediction. That's because they give only two outcomes. Yes, no, true, false, etc. Because of that popularity, I have more material how to fit the model for error, accuracy score, etc. So I'll give one classifier prediction for the daily direction and one regressor prediction for the extreme in the direction of the classifier prediction.

 
mike_smith

Low and Close to the pip! Give it up!

2019-10-23 16_14_03-pythons EURUSD daily callout _ Apiary Fund.png
 
mike_smith

I will be pausing any more predictions so I can develop a new Classifier and an image prediction model. The classifier will give buy,sell prediction, the image prediction will give the full day image prediction. After either is fully developed I will record their outputs here at callout.

 
danielfowler91

I like your style

 
danielfowler91

The model will be skewed on Friday November, 1st. The news release Non-Farm Payroll will send Price Action atypically. It only happens once a month.

Best Wishes and be cautious during that time.

 
mike_smith

Oct31 0-24GMT: 1.11786,1.11064,1.11539

2019-10-30 16_00_42-Smith's Strategy Log _ Apiary Fund.png
 
mike_smith

My opinion (the three arrows.. not made from ML, only the three horizontal lines were.) of this prediction is to expect more price action on the down side today, potentially hitting the predicted low, then continuing the trend up for a little higher than open price action, then closing a little higher than the open.

2019-10-30 16_04_27-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M5].png
 
mike_smith

Seems we did have the little price action on top the open as predicted, however it is still waiting to fill that empty space to hit the predicted low.

There's 3 hours left, so not sure it can do it, but anything's possible. Look at yesterday's range fit 70 pips in less than 1 hour.

2019-10-31 10_48_48-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

Nov1 0-24 GMT: 1.11936, 1.11206, 1.11542

The Close has been playing 'stable genius' recently. I think gap to low's more likely to fill first since is smaller size vs gap to high (and it did that yesterday), then target up to the high from the low.

The lines are ML generated, the arrows are only my personal guess.

2019-10-31 17_37_03-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M5].png
 
mike_smith

As you can see, the actual looked quite close to the predicted. At least it did make the low first then the high second, as predicted. And made the close higher than the open, as predicted. The predicted low was 10 pips away from actual low, the predicted high unfortunately was 20 pips away, but the actual price remained within the predicted range and shape.

In essence, this is simply a commentary of the effectiveness of using an LGBM model. Changing to better models and other better engineering will produce better results.

2019-11-01 17_38_13-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M5].png
 
mike_smith

Nov4 0-24GMT: 1.11993, 1.11489, 1.11823 (h,l,c)

The close is higher vs open, but also close is near predicted high value. More gap on the up side. So I expect low target to hit first then make it's way up to the high, then close above open.

 
mike_smith

Sorry, I misnamed the date, that prediction above is for Nov 4 (today). I will be doing some feature engineering. I think I can get some features from datetime.

 
mike_smith

Looks like trend change. As you see it looks to abnormally reach beyond the low prediction, and is not interested in the high and higher close value. This could be symptomatic of trend change (multi-day, 100+ pips).

2019-11-04 12_59_23-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M5].png
 
alexstuehmer

Thank You!

 
mike_smith

2019-10-5 0-24GMT: 1.117, 1.1138, 1.1117

high, close, low arrangement

The gradient boosting is still indicating up.

Be very cautious if buying under 1.1117.

2019-11-04 15_27_00-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M5].png
 
mike_smith

As we can see, a recurring pattern developing, to use the close prediction for today's extrema, and one of today's extrema prediction as polarity.

This is one case which results from the prediction.
As I predicted in the prior forum post, be careful buying under low prediction. Activity under an extrema prediction has usually led to polarity.

2019-11-05 13_06_36-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

2019 11 06 0-24GMT: 1.11073, 1.10553, 1.10848

 
skidplate69

Hey anyone catch the down trend that just occurred? It was an AB=CD pattern. I caught part of the CD move(1.1130 - 1.1095). Segment D ended at 1.1075 and I originally had my limit set at 1.1075 but I got out early at 1.1095 because I had to wait till the next morning to see 1.1075.
Look up AB=CD pattern and how to find it on a chart

 
mike_smith

Well, what did you expect with only a 28 pip range today.

2019-11-06 18_03_34-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

good shot skidplate69

 
mike_smith

I have some various new combinations I can implement, I think... I'll try them if I have time.
I am interested to combine the open columns of the 7 majors, or maybe just use all columns. Volume data can get messy though, perhaps I will exclude it. The target will be EURUSD only. The closing price proves to be statistically significant. Though deviation from open is minor.

 
pysquaredmgmt

Thanks for sharing. I recently created a python script that pulls forex data and identifies setups for my trades. I haven't done any price predictions though so I may have to look into this area. I pull my data from https://www.alphavantage.co . Any advice on where a novice can get started on learning price prediction modeling?

 
mike_smith

Forgot to include in Callout! Remember this is OHLC predictions which should be used as resistance/support trading lines.

2019-12-02 20_26_35-Smith's Strategy Log _ Apiary Fund.png
 
mike_smith

No volatility. Wonder if I can predict it... but the direction was right? ...

The total range today was a measly 20 pips.
It would be a good strategy to predict high vs low volatility + direction.

2019-12-03 08_00_07-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

However, if we can get close to intraday price structure prediction, it will also be true for the structural intraday volatility prediction. We can use it to better predict the intraday price structure (where it will be volatile even the direction is false). If the date-time features can be mapped effectively to the price, I think this'll also supply predictability.

Why? On lower time frames, all intraday data is an array. If any array values can be predicted >1400 n-values, then this can apply to any given non-random array feature.

 
mike_smith

A full review of all pairs shows GBP is up and USD is down for next week. Given that, eurusd also appears to be sitting at the bottom of an uptrend. This is related to the observation about gbpusd potentially breaking it's uptrend, but now seems that's likely uptrend continuation. Using polarities, I'd call a target for at least 1.12.

2020-01-12 13_52_40-3514059_ IronFXBM-Demo1 - Demo Account - [EURUSD,H4].png
 
Jon Paarlberg

Hi Mike,

I concur with your assessment based on median line retracement on a daily chart. I'll enter at around 1.1115 and put my take profit target at 1.1253, stop loss around 1.1065. Hopefully make around 140 pips during the week or maybe early next.

Chart is attached as a .pdf.

Cheers! And happy trading.

AttachmentSize
Chart 1-14-2020 4-02-54 PM.pdf 75.3 KB
 
mike_smith

I learned one thing from Shawn, focusing on one pair is easier than 27. I might try googling for analyst sources (27 analysts 1 pair vs 1 analyst 27 pairs) for EURUSD, and measuring the consensus, follow it daily to trade. It would be an interesting experiment to see what type of risk stats it made.

 
Rookie

Mike, u got that right!

 
Jon Paarlberg

When I was stuck in Gold 1, I decided to un-complicate my trading... I was all over the board. I stuck with one system, and one pair, the EURUSD, as it is popular, stable, liquid, and the spreads are tight. I've learned a lot by trading just that one pair, in fast times and slow, and it has forced me to trade setups that are not picture perfect. Totally a good idea to do in depth studies of a pair by spending time with just one.

 
mike_smith

It's Mike Smith your favorite Data Scientist and machine learning engineer! I'm back and will be making more predictions for eurusd! Stay tuned!

 
mike_smith

It's clear from the median line snapshots that there's not much for eurusd to be attracted to in the minima areas for the longest ranges. Therefore, all attractors are at the upside. The nearest long range maxima attractor is 1.15. As the 1st photo shows. However I anticipate focus on nearer term median lines, photo 2, 'sell on the rally' strategy. Photo 3 shows a price gap where I anticipate median values and price action to converge towards. I submit 1.105 as a target, and continue to reprocess the new median lines to follow the price action.

2020-02-01 17_40_51-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,Monthly].png 2020-02-01 17_44_14-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,Daily].png 2020-02-01 17_31_06-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,H1].png
 
hwhdanvers

mike are you a programmer?

 
mike_smith

machine learning engineer.

 
mike_smith

Exiting a trade is more important than entering one

https://www.moneycontrol.com/news/business/personal-finance/exiting-a-tr...

2020-02-03 21_05_09-Exiting a trade is more important than entering one - Moneycontrol.com_.png
 
Rookie

Mike, I could not agree more with that sentiment.

The attached chart is from a Kevin class on money management.

What's the alternative, big, very big losses...

off to listen to the shared podcast. Thanks.

Understand Four Fears.PNG
 
FXGold54

Respectfully, why can't we have precision entries and exits?

Allen

 
Rookie

WIP, work in progress...:))

 
mike_smith

WIP,

I adjusted the multi linear regressions after some target hits, adjusted new targets. This method allows me to adjust targets for both directions. I just recently thought of this so I'm testing its outcome. It seems to make sense. I will report any errors.

This method isn't considering strategic stops or entries.

2020-02-04 17_35_03-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,H1].png 2020-02-04 17_39_20-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,Daily].png
 
mike_smith

does anyone know if the beeline ever changes from this, or does it stay like this for the rest of our apiaryfund life?

2020-02-04 17_42_00-Mike Smith _ Apiary Fund.png
 
FXGold54

Mike, I have been starring at that for 5 years....

 
mike_smith

Ya, I noticed it never changed.

 
mike_smith

On MN1 thought it looks like the pair is looking down, zooming in the price is under all linear regression lines. Since I buy towards lines, I am looking to buy and all prior sells were closed hitting their target and there is no selling in sight for me.

2020-02-05 06_30_25-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,Monthly].png 2020-02-05 06_33_31-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
Rookie

Mike, I sure hope you have been trading the EU!
It has been very reliable lately.

 
mike_smith

As you can see the linear regression snapshots are pointing down more now, but the price is still deep under all of them. Therefore the only alternative is to buy back to all the regression lines... This could be scaling in strategy but the risk parameters for Alveo don't allow any drawdowns over 5%. Is the D1 regression line at 1.15 a potential target to move currently losing targets? Or should the strategy be dismissed due to unworkability with the 5% risk rule? I'm thinking the latter.

2020-02-10 17_18_29-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,Daily].png 2020-02-10 17_25_36-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,H1].png
 
mike_smith

The green and red arrows represent a type of mtf calculus. I'll run it through https://apiaryfund.com/stats and let you know after.

I've been thinking on it recently, the price action in forex isn't that hard. Like you said rookie, it's quite dependable sometimes. So may not need machine learning, at least could be considered overkill for long term trading, if some common sense from our dear friend Sir Isaac Newton will work (calculus/physics). The indi above is similar to a multi time frame overlay of derivatives (calculus/physics).

For those that can write (calculus) derivatives to code, that should work just fine.

 
mike_smith

I don't mean I'd stop doing machine learning, but for long range trading I think derivatives work nicely as a totally different solution.

 
robunited

Ok, before you start derivatives ...

I would be interested to see how far you have implemented your machine learning/coding into your current strategy.

As a kind of summary, what framework/strategy have you built so far ... and what are you planning to add/change/modify?

What is your core strategy at the moment you are working on?

I love to peek into this thread to follow your coding/estimates, but now I get a bit lost what the exact trading plan is ;-)
If you could clarify and update a bit on this, that would be great.

 
mike_smith

The start of the uptrend to the regression lines?

2020-02-11 16_29_52-2100683976_ JAFX-Demo3 - Demo Account - EURUSD,M15.png
 
mike_smith

derivatives is simply using calculus. No problem with trying to be good at both. Machine learning is a derivative of maths. I'm also interested to see which I can get up faster. But coding derivatives to work correctly is a mathematician's skill. I'm more in the (data science/ml) coding side right now, so you probably will see more (data science) first.

I was just throwing out why the easy setup multi linear regression strategy wasn't immediately responsive, and what had the potential to be (more responsive to direction, which I am sure derivatives do the trick).

 
mike_smith

This strategy was just a simple way to see where the price likely will be forced to in the future. It is not from my ml training, only some simple method I offered to see where the price will need to consider reverting to. Of course, real ml will look far more accurate and engineered. I'll give some examples.

Like you said what's my core strategy, probably then that's developing some machine learning strategies. Because that method I feel the least risk with. Ultimately, developing ml strategies to analyze/view the market in different ways and manage the various risk parameters will need to be in the core strategy. The model can be generalized for rapid deployment, then edited to evolve it.

Ha, writing the code to deploy an ml model needs discipline and ingenuity as much as learning trading does! So let's do some ML trading...
First, I'll use succession of mt5 timeframes to predict a likely price structure (train/test xgb,lgbm). Include accuracy scores.

2020-02-11 17_23_18-Clipboard.png
 
robunited

Mike "Ha, writing the code to deploy an ml model needs discipline and ingenuity as much as learning trading does!"

Yes, I can see that.

At the end, in my thinking, ML is trying to emulate a human person trading, or the world of human traders globally, so it has to have the same of discipline and ingenuity and everything else ... at least a 1:1 copy, right?

Machine learns from Human ... ML Machine learning ... you feed the machine.

I had the question, in a different way, what are you currently trading yourself, everyday? When not programming ML ;-)

 
mike_smith

Forgot to record the Callout this morning. But still waiting for that big run up..

2020-02-12 16_19_27-2100683976_ JAFX-Demo3 - Demo Account - EURUSD,M15.png
 
mike_smith

@robunited, Trading everyday? I'm not that disciplined. Maybe tomorrow!

Even if I used the indi above, which I've had for years (if anyone wants to audit it let me know), as you can see it doesn't change every day.

It makes me feel more confident anyway making a lot of money trading a lot less than trading a lot more.

 
robunited

Mike, I had this funny thought this morning ;-)

Imagine if all the traders in the world would take a break for 1 week, and only all the EA's, ML programs etc. would run the markets.

How would the chart look like after 1 week?

More structured, less chaotic, or the same?

 
robunited

To put my question the other way around.

If ML trading programs are based on exploiting/profiting from human trader behavior, what happens to that ML program if the human is out of the market?

What will it profit from? Other ML's emulating humans trading?

 
mike_smith

Flash crashes. They're caused by that. Since the trading algos are HFT, they trade in millisecond intervals, so I don't think most are responsive to human trading activity, unless you're moving the market significantly. I think the pattern left on the market will depend on the complexity of the algos, how they're designed to exploit any mode in the market. If no algos are designed to exploit a particular structure in the market, then the market will be left in limbo. This is what I think.

There are many examples how algos can create their own utility, such as genetic programming, NLP, etc.

https://nypost.com/2017/08/01/creepy-facebook-bots-talked-to-each-other-...

http://www.primaryobjects.com/2018/09/03/creating-self-assembling-code-w...

If the patterns they make aren't useful to people, then people intervene and make their changes.

2020-02-13 07_30_17-dow flash crash - Google Search.png 2020-02-13 07_33_30-dow flash crash - Google Search.png
 
robunited

Mike: "If no algos are designed to exploit a particular structure in the market, then the market will be left in limbo. This is what I think".

Yes, but in the FX market specifically, there are also other aspects to it.

Multinational companies have to close their quarters, have to pay their overseas transactions at a certain time, contracts expire, banks have to close their days/books, Millions of tourists are exchanging money on a daily base, people moving money out of fear into a safe havens, so many other things and aspects happen in daily FX transactions, without specifically looking to exploit a specific structure.

btw .. these Flash crash's look scary. I hope we don't see more of these as more and more sophisticated artificial HFT systems come online in future.

 
robunited

To continue that thought ... if the sum total of all those transactions are at the end funneled into the hands of 4-5 big players, they will try to off-load that volume at certain stages of the price cycles.

There are times, when they have already done that ... they will have to close their days/books, at any other stage.
And those 4-5 players, will not do it all at once.

By that, creating new patterns (to exploit).

I'm sorry, I realize that this might have been a distraction to your ML strategy, I was trying to figure out the impact of EA's/ML's into the overall concept and dynamics of the FX market. And whom better to ask as someone who is into programming this ;-)

 
mike_smith

Still going down? I'll let you know.

2020-02-16 19_32_57-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

@robunited, if you're interested in ml, have you tried Amazon's DeepAR? It seems easy to use, and reasonable.

 
mike_smith

But complete data science analysis and analytics will let you see more than a scalar forecast (w/ deepAR).

 
mike_smith

Huge jump both ways today. Let's see where this goes.

2020-02-18 07_20_34-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

Not up today... :D

2020-02-18 16_15_46-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

Still the same o...

It must be Trump trying to please the hillbilly's.

2020-02-19 07_00_23-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

Buy activated.

buyfeb19.png
 
mike_smith

The strategy.

2020-02-19 20_37_08-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

Sell again, trying to find it's way.

2020-02-20 02_04_23-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

Choppy: contraction mode. Is it finally the major ride up we've been waiting for?

2020-02-20 06_28_55-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

Seems the nearest high targets then are 1.15 and the closer 1.098.

But remember... momentum before midpoint!

:D

2020-02-20 07_00_48-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,Weekly].png 2020-02-20 07_02_54-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,H1].png
 
estelwillits

what just happened. I was 45 pips short and all of a sudden it jumped to -60 pips up

 
estelwillits

another jump up to -1o1 pips

 
estelwillits

make that -252

 
robunited

As a guess, spike came with German Flash PMI's ...

Sometimes a good opportunity for a quick scalp, if you are in front of your screen and can catch it.

Screen Shot 2020-02-21 at 3.35.23 PM.png Screen Shot 2020-02-21 at 3.37.19 PM.png sell pmi.jpg
 
mike_smith

Newton and Einstein are right. All you need is rate of change (derivatives).
Clearly that didn't change in forex.

But you can't avoid the midpoints. Probably will pause at 1.087 and next midpoint is 1.098.

Good luck!

If anybody is coding derivatives let me know I'd like to help. https://en.wikipedia.org/wiki/Derivative

 
mike_smith

https://www.centerspace.net/examples/nmath/csharp/core/differentiation-e...

I found this, but I'm not a .NET developer, I'm a python developer. Any takers?

 
mike_smith

Look to buy to 1.096. This target will decay slightly.

2020-02-24 07_21_09-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
Rookie

well maybe not quite, me thinks today's surprise is done, the market is rest and the USD can return bullish.

 
mike_smith

Like I said, the target could decay. Attached you can see a linear regression intercept at 1.096.

2020-02-26 06_57_32-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png 2020-02-26 07_02_26-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,Daily].png
 
mike_smith

In the context of all the regression lines captured, it is clear the pair is (minned out?) on oversold, and is merely skipping on the bottom.

2020-02-26 07_22_24-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,H4].png
 
mike_smith

Rate of change alert to sell. S/L 1.091. 1 lot unit.

26.png
 
mike_smith

Contracting... buy again. 1 lot unit +1. S/L 1.086.

22.png
 
mike_smith

Target hit. Now looking for new target.

Not as fun as machine learning, but works!

2020-02-27 06_54_55-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

Here are two new targets to look for: 1.15 and 1.076. Look for these to converge.

 
robunited

Mike

"Target hit. Now looking for new target.

Not as fun as machine learning, but works!"

;-) Good one!

 
mike_smith

24 hours to go! Let's see how much juice I can get out of this!

 
mike_smith

Updated the linear regression lines. Note how the lines act like support,resistance,chart patterns, though they are neither.

2020-02-27 22_25_14-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png 2020-02-27 22_30_25-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,H4].png
 
mike_smith

Target changed little, maintained at around 1.147. Now you can see more regression lines looking up to that target.

2020-03-01 15_09_04-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png 2020-03-01 15_09_39-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,H4].png 2020-03-01 15_10_46-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,Monthly].png
 
mike_smith

Sell Alert. 1.119 stoploss.

5.png
 
FxMoneyTrader

Hello Mike,
I am intrigued by your posts, but I confess I am a little confused because of a lack of knowledge. Know how I can learn more about linear regressions? BTW I Just scalped 20 pips from this pair!

 
mike_smith

Surprise change! Change to buy and increment order +1.

2020-03-03 07_02_39-Alveo.png 2020-03-03 07_03_26-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
mike_smith

The secret to my success with a good probability but still imperfect indicator is increasing the order entry by 1 for each new directional turn if prior order is a loss. You can do this with any indi that is naturally profitable yet demonstrates a low equity rise over time. Using the +1 order technique your equity improves significantly.

Look for an indi that has no high drawdown, doesn't turn frequently, can trend, and has a complex formula.

This technique actually makes me excited to be in the market, because I know once the market exits the chop and starts trending it will carry the incremental raises I added to it.

 
mike_smith

@FxMoneyTrader,

What do you want to know about linear regressions? The way I draw the lines are at my log "Smith's Strategy Log" https://apiaryfund.com/forum/smiths-strategy-log?page=4#new

I only use the lines for targets. They provide 0 directional information for me (0 short term, but yes they do long term). I am reading the directional timing from the indi (the red/green arrows) which is a type of derivative (calculus) based formula. It is not exactly calculus, more like rate of change on rate of change if you get my meaning, yes it needs serious redesign, but the point is derivative calculus is a key you can use.

 
mike_smith

@FxMoneyTrader,

Attached, you can see the expected min/max target for each direction. Why linear regression lines are good targets? For their midpoint value. The longer the time snapshot of the regression the stronger the target.

2020-03-03 07_24_59-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,H4].png
 
mike_smith

@FxMoneyTrader,

regarding midpoints in currencies, I encourage you to talk with triguylm1. He seems to understand it well.
TBH my interest is in derivatives and machine learning.
Just linear regression was a small stepping stone in the machine learning path.
As you see, if I pick up something I learn in ML and can throw it up quick here to be useful, I do so.

 
mike_smith

Strange, seems time has decayed the targets fast! Attached is new expected min/max TP.

2020-03-03 18_43_38-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,H4].png
 
mike_smith

Long view. Time decay, like that found in general relativity, shouldn't be surprising. Don't you agree Rookie?

2020-03-03 18_47_49-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
Rookie

Uhm, I'll get back to u on that..,))

100x100.png
 
mike_smith

Another way to see the price between the targets.

As the distance from a target increases, it becomes a preferred target.

2020-03-03 19_18_25-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,M15].png
 
robunited

I can follow your price cycles ... from consolidation to expansion to consolidation ... (believe it or not) ;-)

If i have to put my idea on it ... that's how I would trade it.

Screen Shot 2020-03-04 at 10.25.03 PM.png
 
robunited

Your linear's are matching the cycles of a Wobble process/cycle ... which are just the same as the levels of a price cycle.

Well done on your linear's

;-)

 
robunited

A perfect example where the strategy (indicators) is matching the price cycle stages of a Wobble cycle (risk/money management)

Thanks Mike ;-)

 
robunited

Mike, to avoid misconception ;-)

I'm not saying you have to Wobble it out ... that's a different approach.

What I wanted to point out is that the risk management framework 'only' is a nice overlap to your strategy and how you have
mapped out the price structure with your linears.

The way you trade and scale it stays the same as you do ...

(I'm not sure I should draw on other traders charts ... I hope it didn't confuse you. Just tell me if this was a wrong thing to do.
I'm just a little bit excited, because for the past year, I only saw a bunch of lines on your charts, and now they start to make sense to me)

 
mike_smith

I can use same strategy and risk on M1 for the indi. No relation to the linear regression of course.

 
robunited

Yes, I can see that.

Looking forward to the next set of linear regressions ....

 
mike_smith

Rob, each line is a guaranteed target. It is only that they'll adjust, so unknown their actual price at that time. You simply choose which line you prefer to hit, but the better idea is the one using the longer time snapshot, since it's harder to move itself.

The lines at the bottom are the shorter time frames, so they'll adjust first. The lines above are the longer timeframes, so they move less. Likewise, the lines that move less are going to be stronger attractors for price.

Since the lines are guaranteed to intersect the price somewhere in the future, my next question is which line has a substantial distance from the current price (= profit and potential to remain at that distance).

2020-03-05 18_59_00-Smith's Callouts _ Apiary Fund.png
 
mike_smith

Updated

2020-03-05 19_12_45-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,Monthly].png 2020-03-05 19_19_04-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,H1].png 2020-03-05 19_28_18-2100683976_ JAFX-Demo3 - Demo Account - [EURUSD,Daily].png
 
mike_smith

Yes, you can draw on my charts Rob!

I'm still trying to figure out where the 1.133 target went. I'll have to get back to you one that one.

 
mike_smith

Since I'm testing M1 now, I am not giving callouts on M15, thanks for your audit.

 
mike_smith

So anyway, it seems the 1.133 target did hit. I am interested in 1.08 now. The target adjusts due to convergence towards the current price. So let's see what happens.

 
robunited

Price cycle like this? Just a try, but it matches your linears again.

Wish you a great weekend!

Screen Shot 2020-03-07 at 1.03.37 PM.png
 
Rookie

Mike in your comment above,
"Long view. Time decay, like that found in general relativity, shouldn't be surprising. Don't you agree Rookie?"

"Until future proof is discovered, some of Einstein’s theories will continue to be simply that — theories, but they do provide us with an entirely different perspective of time."
https://blog.onlineclock.net/einstein-time-theories/
Thus time is a purely human invention from humans to measure cycles. From that, we can conclude only matter can decay or change its form the clock example.

https://www.investopedia.com/terms/t/timedecay.asp with liberties...
Time Decay—The Silent Killer, and paraphrased it will always be negative.
Then the value is important.
Intrinsic value is the difference between the market price of the underlying security,
So are the underlying values of Forex? At layer one, it's the Currencies that make up the markets.
The extrinsic value is more abstract than the intrinsic value, and it's more difficult to measure
So we could say the extrinsic value is the overtime of a currency pair is the increase or decrease of the pair's value.

There is so much data needed to compute the relative value of a currency and then put each currency on an Einstein type fabric is beyond my capabilities of collection and extraction of data to form any reasonable hypotheses for precisely what is going to occur.
That said we mear humans can track the movement of money, a currency(s) for the sale of commodities AKA Adam Smith's theory of money. And this is where we get into cycles based on what we observe. Remember it is not a reality until it is observed.

So for trades, some folks are very good at utilizing CME data and the movement of money, some folks follow the central banks, some folks the crowed and so on. But what really matters to a Forex Trader can be much simpler.
What do we need to know? Location, direction, and how far.
What can we control, hedging, sizing, entry, exit.
That's what your attem[pting to define with your linear analysis.
When you add price and time cycles to the analysis, we get the highest probability of where the price is going.
Back to Forex trading application, how do we increase the odds and maximize the profits of what we can know with our Price and time cycles computations in a particular time fractal?
We have to know what our winners and losers in a row are. If you never had a loser, why would you not risk 100% of the account equity? IE, we can change our sizing accordingly to fit the probability of when the next loser may occur.

Now we can calculate the probability of the next candle in a fractal, and we would think that some measurement of velocity and momentum would provide the distance of travel and your time decay question. Still, we also need to factor in the sentiment of the crowd and other entities' interests that may be influencing the sentiment; some of these entities are unknown to us as little people. Our equation becomes very flawed in construction and output.

So what the solution, trade what you see, and work with what we know now.
1. location 2, direction 3. How far (many traders use S&R or Fibs)
What we can control,
1, Entry 2. Exit 3. Sizing (Winners/losers in a row) 4. Hedging,
Then comes maximization,
We need to look at what we did with that data to adjust.

Learning to get to simple is not an easy task for many.
The bottom line, while everything may be relevant, not all relevance is needed.

BTW rumor control has it that one of our traders made 84% equity gain last month. I think that the trader has figured it out.

Enjoy the process.

 
mike_smith

Interesting Rob!

Please keep drawing on my charts, I'm starting to see your reasoning.

 
robunited

The reasoning, to me, is the natural price cycle.

Any indicator on the chart is an estimate of price structure calculation.

Sometimes they match exactly, sometimes just a little bit.

Once you got your indicators on the charts, you can map out how the price cycle currently matches your setup.

What we could do, as a test ... if you could cut out the plain chart only, without instrument, price and TF ... with your linear's only.

I can try to draw the current price cycles on the chart to see how it fits the calculation of your setup.

 
robunited

I like your linear's, because you have 3 levels .... and we have 3 key levels in the natural price cycle (well, at least in the price cycle I have in my mind).

Mike linear Levels.jpg