forex software

Create and Test Forex Strategies

forex software

Skip to forum content

Forex Forum

Forex Software, Forex Strategies, Expert Advisors Generator

You are not logged in. Please login or register.


Forex Forum → Forex Strategy Builder Professional → The handling capacity and calculation power of FSB

Pages 1

You must login or register to post a reply

RSS topic feed

Posts: 14

Topic: The handling capacity and calculation power of FSB

I would like to know the limits of FSB in terms of calculation power.



What I have - I'm developing a custom indi, which gathers data points from price data and then sorts and identifies them in data arrays, in short it collects data. After that I manipulate the array data into patterns in order to make very basic statistical analysis. There are no price signals or chart objects, in essence pure calculation.

The main part consists of 4 nested for loops. I wouldn't say it is such a heavy calculation. What tends to be extreme is array sizes:

int[] freq1 = new int[66666666];

Yes, it is 66 million and there are quite a few of them. But I'm not touching the max array size with it, and I had already system out of memory exceptions, which made me lower my expectations and numbers. I'm using those sizes because of indexing (patterns are indexes and so it makes life easier for me).

Now the main point: I can't finish the basic steps of this indi because when I try to add another for loop, I can't get FSB to finish the calculation even with very little actual data. A method call, which would substitute that last for loop, produces expectedly the same result.

Legacy FSB crashes, Pro keeps working but CPU usage >50% remains even when I remove my indicator.

In terms of errors I have only seen index out of range errors, which do not come from my code. Raising the size of one array fixed that ghost error, but I'm not sure why because the code is not exceeding the limits.

So no errors, just stalling/crashing with high CPU usage. When I remove the for loop, it works.

Yes, it is a bit heavy, calculation time for 100k bar data takes 10 seconds, but why can't it finish the whole code in 15 seconds? I don't care, it might as well do it 30, but it doesn't.

In the light of this, I would like to know where is exactly FSB's limit? I tend to understand it is not in my hardware or C# limits. Is it the program and where are its limits?

Re: The handling capacity and calculation power of FSB

Hello Footon,

Please share the actual code and I may try the issue. I may also try to make some performance test to see what exactly overflows. I don't remember any fixed limits right now, but if there are I may find them.

Re: The handling capacity and calculation power of FSB

Sending via email right now.

Re: The handling capacity and calculation power of FSB

I made some tests. It looks like the compiler enters into the nested `for` loops and never come out. what I see it is pure calculations without using structures of FSB Pro. I may assume that simply it needs a longer time to finish the calculations except if there are .NET limitations. I read somewhere that a single list may contain up to 2GB of data (but this may not be true).

Anyway, the algorithm of this indicator can be improved. For sure you don't need to calculate it Bars * Bars * 8 * 8 times (except for experimental purposes)


However, to be completely sure that this is the case, I copy/paste only the nested `for` loops code in a separate program out of FSB Pro and simulated it for 500 bars of data. It raised an "OutOfMemory" exception. It suggested compiling it for explicit 64 bits CPU support, but I could not manage to do it for 20 seconds.

https://image-holder.forexsb.com/store/out-of-memory-exception.png

If you want, link the problem you want to solve and we may find a solution or a workaround.

Re: The handling capacity and calculation power of FSB

Found something:

An array could theoretically have at most 2,147,483,647 elements, since it uses an int for indexing. The actual limit is slightly lower than this, depending on the type contained within the array.

However, there is a 2GB maximum single object restriction in the .NET CLR, even in 64bit. This was done by design.

Re: The handling capacity and calculation power of FSB

Firstly, thank you for taking a look!

Yeah, I know about the array limit and my array sizes are below that.

If there would be array bigger than 2GB, it would throw the exception, right? But it doesn't. And I am *sure* that I'm not blowing the size of arrays. I've read something about not finding the right amount of memory space to store the last file, there's plenty of memory but not in a suitable chunk to put the file, so the experts write at least, I don't know.

For sure you don't need to calculate it Bars * Bars * 8 * 8 times

It is not Bars, but the size of the pattern array. I can tell you a real example: I have 16k bars (Bars) and from that I get 1850 patterns, so it is 1850*1850*8*8.
And yes, I need it to calculate this way, that's the whole point smile I take initial pattern from 1-8 length and compare them to distinctive patterns of 1-8 lengths that follow. I've been thinking quite a bit, but still no good way found to achieve it more efficiently.

A memory problem... Drop int arrays and use int lists instead?

Re: The handling capacity and calculation power of FSB

You know best what you want to achieve. However, I cannot grasp the idea. If you calculate a pattern, you then calculate from it to the end of the data series. At the next, let's say "main" pattern, you calculate again patterns to the end.
I see you collect them by 8 in a raw, but cannot understand why.

Isn't it possible to calculate the main patterns to a particular period and then to continue with the alternative ones? Or to calculate the alternative patterns form one main pattern to the next one. I can propose to try to solve the problem with two linear sets of patterns - one main and one auxiliary. Anyway, it will be best if you can have only a single array of patterns or even only a single array of signals.

You may try a [Bar x 8] matrix for storing the different patterns by bars and types.

You may also try a [Bar] array where to store a separate list for the sub-patterns. You may expand them dynamically.

Another possibility is to make an object Pattern with properties { type, bar, subPatterns[], ... } and to store them in a list.

The Pattern type can be "byte" or to make an enumeration. Patterns also can be `Tuple`s for better memory efficiency.

8 (edited by footon 2020-05-28 13:14:34)

Re: The handling capacity and calculation power of FSB

...

Nevermind my previous posts, I used online compiler, which enabled me to see better, I know the area to work on at least.

Re: The handling capacity and calculation power of FSB

For me development from scratch is some tough shit, pardon my French. But I did it! The current concept is working! 100k of bar data is done in 15 seconds! It is on the border of memory usage but that is sufficient. I am afraid though that I'm going to hit the wall by having it write the collected data into text files. This remains to be seen.

Re: The handling capacity and calculation power of FSB

Yayyyyyy   I think this is quite a feat that you have accomplished.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: The handling capacity and calculation power of FSB

C# is not suitable, really. I just finished checking last things and let it write a file of the base data; the initial, most primitive stuff, which comes right off price data. What can I say, it hurts to be dumb and my computer agrees with me - the csv file is 700MB large. And it was just a test with 20k bars. I thought to use 200k, then considering the limitations 100k bars, but now...

The concept itself is quite fascinating, so much or little I've seen is showing some very interesting relationships between data points. I'll most probably need to dumb it down, scale back. The number of patterns currently is 8ˇ6. Long patterns are so much rarer that those don't repeat so often to be actually useful. I have to find a good cut off point. Can purpose-built database programs handle 700MB files? I'll probably save the climate for today and try it in the coming days.

Re: The handling capacity and calculation power of FSB

I am wondering if Excel will help.  (guessing)

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: The handling capacity and calculation power of FSB

Not for me smile It cuts off data at 65th kilo row.

Re: The handling capacity and calculation power of FSB

SQLite is very good software. It can create and manage a DB in the Memory, which is the fastest possible. it can also store the DB if it is necessary.

I'm using a similar technique for EA Studio and it is very fast. However, I'm using my own implementation.

C# is not suitable, really. I

Years ago I was sure that C# is the best language. Now, when I spend at least 8 hours per day studying computer science, I'm sure even more that it is not smile))))

This is somewhat better smile)

https://image-holder.forexsb.com/store/easl-call-ops-from-list-thumb.png

Posts: 14

Pages 1

You must login or register to post a reply

Forex Forum → Forex Strategy Builder Professional → The handling capacity and calculation power of FSB

Similar topics in this forum