Re: How to code Natural Cubic Spline?

Just sent PM -- that sounds close to my guess, tick data vs. Bar Closing. I've found similar when porting my FSB strategies to trading platform.

27 (edited by krog 2010-06-25 05:07:49)

Re: How to code Natural Cubic Spline?

Looks like you're more on track on the other thread. I don't think I have much to add. But if you haven't tried this, it's best to turn off automatic control of the previous bar so you are sure what's being used:

Manually set Use Previous Bar (Strategy > Auto Control of "use prev bar value")
FSB:
- Open Slot set to "Bar Closing"
- Untick "Use previous bar value" box
- Set CS 3 logic as "bar closes above/below ..."

FST:
- Open Slot set to "Bar Opening"
- Tick "Use previous bar value" box
- Change CS 3 logic from "bar closes above/below ..." to "bar opens above/below ..."

The next bar's opening should be near to matching the previous bar's closing. This way you get the ending value of the Projected Y, with a comparable price value. It may help to avoid fluctuating CS 3 values that move from above or below the tick stream closing.

This might not work 100% since your screenshot appears to show your data has gaps between bars.

If you do this with 2 FSB's side by side, most of the trades should match, but the Bar Opening instance will show trades entered one bar later.

Of course, this might not work at all, I'm not sure how FST works or what it does with the strategies.

28 (edited by krog 2010-06-26 01:07:30)

Re: How to code Natural Cubic Spline?

Here's CS 4 -- in the screen shot, it looked really cluttered, so I've removed the green curve line which does nothing, and the Spline Curve Y Value which did nothing. You can set the Projected Y line to a color, so your fast and slow can be different colors. I jsut picked a bunch of colors off MSDN C# Color FromName list, you should be able to find a few that look distinct.

Taking this down due to entry bug.

29 (edited by footon 2010-06-25 15:26:50)

Re: How to code Natural Cubic Spline?

Thanks for the CS4, the colour option is wonderful, I just thought about it yesterday when I tracked the lines with great difficulty  big_smile

my testing results so far:

screenshots below are from trader. The red X marks the bar which should be an opening point of position as both logics concur, Z on the other hand should be the bar on which entry calculations are based on.

opening logics:     close above blue
                        open below purple

The red box shows the entries allowed which cannot be true (the Z bar doesn't close above blue line, it should allow to go short not long, and a similar mistake in screenshot 2).

http://s4.postimage.org/bT9RJ.jpg


http://s1.postimage.org/4iSv0.jpg

The graph is M5, but the most interesting part is that 1340 hours there should be a long trade opened, but it didn't open one, 1345 hours it opened a short one, I'm very puzzled to be honest.

I will compare the signals with builder next, but at the moment there is only one discrepancy in the 1340 bar if builder uses just the same strategy - set to bar opening, previous bar value ticked. (trader says 2x "yes" to long, builder says "no yes" and "yes no", screenshot below).


http://s1.postimage.org/4jdXi.jpg

Is my logic right? I just want to make sure I comprehend the working method of CS.

update: it's getting better all the time lol if the strategy is left the same, trades match in builder! Unfortunately the testing period was to short, only an hour, I'll send it to the battle next week, then the discrepancy shall appear again.

If "bar closing" is set and "previous bar value" is unticked, it behaves completely differently, trades are not even close and I can't tell how the signals are given.

Re: How to code Natural Cubic Spline?

I examined "bar higher y" logic and I think I found something - previous bar value is taken only for projected Y, that's a way to explain given signals, it takes the higher/lower values from current bar (from which a position should be opened), but CS projected Y value comes from preceding bar.

I might be completely wrong here as I'm not a programmer, I assume the signal is generated by comparing the high and low point of the bar with Y's reading (logic "bar higher Y) for the same bar.

I stumbled upon this when I started investigating a trade, which never took place by the builder, so if my assumptions above are right, trader reads indicators correctly, but builder fails to do so, and this is valid only for "bar higher Y" logic, at least for now.

31 (edited by krog 2010-06-26 19:20:21)

Re: How to code Natural Cubic Spline?

Thanks for the screen shots -- the more detailed real examples, the better it will help us figure this out. I think you're right on the current bar with previous bar's indicator values.

This is a bug in the Cubic Spline indicator. After some thought, you should not be able to have a "Bar Closes Above/Below" with a "Bar Opening" entry point.  Because it's using both an open and a close, and wants to enter at this bar's opening. It's like you wait for the current bar to close, then enter at the current bar's opening -- that would be like going back in time.

Looking at the code in Indicators_Base/Indicator.cs, I think FSB compares the current bar's prices with the current bar's indicator values. Or, if "Use Previous Bar", it compares the current bar's prices with the previous bar's indicator values. (I think you found that with "Bar is Higher" logic.)

An example for CS 4 with a "Close Above/Below":
For your FSB screenshot (3.jpg), for 13:40, marked with the red X, with strategy as:
- open is below purple, with "Use previous" ticked
- close is above blue, with "Use previous" ticked

it's following logic like this:

if (open[13:40] is below purple[13:35] AND close[13:40] is above blue[13:35])
    then show 2 Long Yes, enter Long at open[13:40]
else if (open[13:40] is above purple[13:35] AND close[13:40] is below blue[13:35])
    then show 2 Short Yes, enter Short at open[13:40]
else
    show 1 Yes, 1 No, and don't enter

This does not make sense -- you could not know close[13:40] (which would be at 13:44:57) and then go back in time to enter at open[13:40].

If "Use previous bar" meant previous indicator values and previous prices, it would be ok; but that's not the way it is. It's only use  previous indicator values and current prices.

I'll have to change "Close Above/Below" to "Previous Close Above/Below".

Re: How to code Natural Cubic Spline?

One other thing I'm concerned about -- in your screenshots, FSB Projected Y values appear .002 and .001 higher than FST. In a few cases that would be enough to cause them to not match exactly.

Re: How to code Natural Cubic Spline?

alright, followed "close above/open below" logic, and the same pattern there as well.
I made a mistake earlier:

trader reads indicators correctly, but builder fails to do so


My wishful thinking, actually they both fail because they read indicators the same way, just trader doesn't have all the info available contrary to builder, when it takes the closing and opening prices, it has just the opening price, so the opening price becomes a closing price as well.

Now I used "cold data" (market closed, prices are not updating), and trader versus builder signals are equal, yesterday when running "hot" there was discrepancy.

One other thing I'm concerned about -- in your screenshots, FSB Projected Y values appear .002 and .001 higher than FST. In a few cases that would be enough to cause them to not match exactly.

Excellent spotting, Krog, checked this as well but seems in check (screenshot below), but maybe  discrepancy appears again when going live, will see it through in next week's battle.

http://s3.postimage.org/vmLTA.jpg

I think we've made considerable progress, but have we cracked it? Can't come up with nothing else at the moment, but I'll think it through again to be sure.

Stay sharp, stay frosty! as my motto states  big_smile

34 (edited by krog 2010-07-05 21:49:15)

Re: How to code Natural Cubic Spline?

footon wrote:

trader doesn't have all the info available contrary to builder, when it takes the closing and opening prices, it has just the opening price, so the opening price becomes a closing price as well.

Now I used "cold data" (market closed, prices are not updating), and trader versus builder signals are equal, yesterday when running "hot" there was discrepancy.

I think this was the problem. I did a bit more reading about "Use Previous" and looked at some samples of other indicators. It was due to combining enter position at Bar Opening with a "Bar Closes" and "Bar Opens" logic. The "Bar Closes" is not known until the end of the bar -- so FST would not enter the trade. But then with cold data, FST has all the info like FSB in backtesting, so the trades would sync up.

I always use Bar Closing, which will work with "Bar Close" logic, so never encountered this problem. But it's good to find it in order to code the indicator so users won't run into it.

Find CS 5 attached. It will only allow "Bar opens" logic, and no "Bar Closes" logic for entering the position. Then, let FSB and FST auto control "Use previous". It's best to use CS 5 and remove previous versions, they all have the go-back-in-time entry issue.

As for the different values, if there is a difference, it's the first thing to check. It would be like one signal out of 100 or 1000 would not match, or some frequency that would appear rare and random.


----
Removing this indicator, may not work properly.

Re: How to code Natural Cubic Spline?

thanks for the new version, I put it to the test, not so stratospheric like previous versions, but seems workable, is the "close above" option totally out of question (especially when used with "bar opening")?

I'll definitely go live as fast as I can, if something fails, I'll let you know.

Re: How to code Natural Cubic Spline?

I hope that one takes care of most of the non-matching issues.

Yes, "close above" should be totally removed, especially using it with FST. What happens is you wait for the close, which triggers the signal in the current bar. Then, it wants to enter the position at the opening of the current bar. FSB can do that, because it has all the bars from cold data. FST cannot do that with hot data -- it would be going back in time.

Since the gap is very small between bars, think of the open of the current bar the same as the close of previous bar. So to use the close for the signal, use "Bar Opening" with "Use Previous" ticked. It will compare current bar's opening (virtually same as previous bar's close) with previous bar's indicator value.

Thanks for sticking through this. This is where bugs come from -- one guy creates and uses it only one way, and it works fine. It's not until someone uses it a different way that bugs are found. I never would have thought of the bar opening vs bar closing issue because I always used it the same way.

Re: How to code Natural Cubic Spline?

krog wrote:

Yes, "close above" should be totally removed, especially using it with FST. What happens is you wait for the close, which triggers the signal in the current bar. Then, it wants to enter the position at the opening of the current bar. FSB can do that, because it has all the bars from cold data. FST cannot do that with hot data -- it would be going back in time.

Since the gap is very small between bars, think of the open of the current bar the same as the close of previous bar. So to use the close for the signal, use "Bar Opening" with "Use Previous" ticked. It will compare current bar's opening (virtually same as previous bar's close) with previous bar's indicator value.

You're right, I got a bit confused (thought that earlier results were somewhat close to the real thing, but not in this world, it's actually kind of funny - I got excellent results like for a week in real money trading lol).

krog wrote:

Thanks for sticking through this. This is where bugs come from -- one guy creates and uses it only one way, and it works fine. It's not until someone uses it a different way that bugs are found. I never would have thought of the bar opening vs bar closing issue because I always used it the same way.

No problems, mate. I'm very glad I could help, I learned a few things as well, so a real win-win situation. Stay sharp! big_smile

Re: How to code Natural Cubic Spline?

alright, conclusion of today's tests:

TF M5, "opens above after opening below" + "opens above", 4 trades all match, but problem in Y values:

http://s3.postimage.org/ALUGS.jpg

Another pair, TF M5, "previous higher" + "rises", trader made 10 trades, builder 7. Y values are differing as well, but the interesting part is that it in 2 trades values matched, but builder didn't carry out the trade, screenshot is with a slight difference in Y values.



http://s1.postimage.org/dg8I0.jpg

Any ideas what could be the weak spot?

Re: How to code Natural Cubic Spline?

First good to double check the simple things, to verify the Base Price and the Smoothing methods are the same in both builder and trader. Most likely they are, but I've found at times a quick double check can save a lot of time.

Bigger problem, more likely cause -- the closing prices between builder and trader do not match. In Yvalues.jpg, trader close = 1.50672 and builder = 1.50646. In nomatch.jpg, trader close = 89.276, builder close = 89.275. Since the indicator is based on the closes, it would not be expected to match (I think you could verify this with any indicator, they will all mismatch slightly).

If it's just one bar, it would probably only affect trades with in the lookback of the CS 5, about 25 bars based on your slow cubic. Yesterday's screenshot closes matched, so I don't know how frequent your closes mismatch. If one knot close input value is off, even slightly, then the indicators from builder to trader will not match, nor would we expect them too. In fact, if they did, that would mean there's a whole different problem !! The differences are very small, but since the comparison is done all the way down to .1 pip, the difference is enough to make trades mismatch.

Re: How to code Natural Cubic Spline?

Krog,

do you mind sharing your Cubic Spline indicator?

Thank you



krog wrote:

Hi Footon, thanks for taking an interest, I really appreciate the discussion and feedback.
Posting what I have now -- Cubic Spline 3. "3" does not mean anything, that's the number of the version of coding I've got to that works or makes sense.
For more info on cubics, see http://en.wikipedia.org/wiki/Cubic_spline. Basically, you have "knots", which are fixed points, like 5 or so, then draw a smoothly curved line that passes through the knots.
In Cubic Spline 3, you can select the number of knots and the "spacer". Spacer is the number of bars between the knots.
It draws the spline curve on the prices as a green line. Because of the way it's calculated, you only see the curve line for the part. Before that, it is on the close for the bar. If you select a large number for spacer, it will be more visible.
There is also a blue line for the "Projected Y value". That is, go to the last knot, use its cubic values (they describe the slope of the curve), and extend it forward one knot -- as if the spline curve were projecting out into the future. It's shifted back to the current bar.
Ex: spacer = 10, current close = 1.1234, Projected Y = 1.290: this means in 10 more bars, if the spline curve were drawn forward, the next knot would be at 1.290, above the current close.
At first, the Projected Y looked very bumpy, so I added a parameter to smooth it out. It takes a moving average.
It's an Open/Close Filter, so have to add after "Bar Closing". I've kept it simple for now, if there's more promise I may go back and add Open/Close slot.
Logic is simliar to Moving Average -- it rises, falls, the price opens above or below the Projected Y, or closes above or below the Projected Y. And -- "bar is higher/lower" - the whole bar has to be above or below the Projected Y.

In using it, it looks like a momentum indicator. It seems to go up and down with the waves, overshooting the peaks and undershooting the troughs. And it seems to pass through the prices at the middle of the wave. This is what led me to think it's not very useful -- it seems to exaggerate the movements, with no advance signal of price reversal. Or it's like it is more volatile and adds noise instead of reducing noise as an indicator should. Also, it could be just curve fitting which looks great in back testing but never works in real trading.

Feel free to try it out, perhaps there is something else to do with it that I have not thought of. Let me know, I'll see if it I can put it in.