My Algorithmic Trading Project
My Memories, and Challenges I Faced
Hi everyone. I am a Data Scientist and Machine Learning Engineer, formerly a Developer and Industrial Engineer. Every tech person’s dream is to write an algorithmic trading bot. We have implemented, and here are my experiences. I hope they can be helpful to people who want to work in the same field.
This trade bot is developed by Alparslan Mesri and Hakan Elbas.
Sections
1- Have Domain Knowledge and Decide Your Bot Type
2- Confidentiality
3- Study Your Lesson Well, Collect Requirements Good and Do Concept of Proof
4- What Was Our Target?
5- About the Trade Duration
6- Attention to Data Leak!
7- Getting First Results
8- Inventing New Algorithms
9- Code and Algorithm Optimization
10- Optimum Percent of the Capital to Invest in Every Trade
11- The Ship Metaphor
12- Team Job
13- Concerning the Nature of Money
14- Final Boss: Can Your Presence Change the Market Behavior?
15- Latest Numbers
16- Inflation and Crisis Forecast
17- Conclusion
1- Have Domain Knowledge and Decide Your Bot Type
I began to think. In that moment, I was manually executing trades based on certain patterns. I particularly love the “diamond” pattern. I had a collection of diamond patterns in my Excel file. I manually scrutinized historical market data, taking note of each diamond pattern and its results. I had a close friend, Data Scientist Hakan Elbas. I shared the idea with him, and we decided to embark on this journey together, attempting it for two months. So, my initial concept was as follows:
“Why don’t I write a script to identify all diamonds, rising wedges, and falling wedges in the market?”
“If I identify them, I can gather statistics on the success rate of these patterns”
“If I were to enter trades using these patterns and certain parameters, what would be my profit?”
“If I identify them, I can develop an algorithm to pinpoint the parameters that optimize my profit”
However, everything took a different turn. At that juncture, I had a conversation with one of my old friends, trader Selim, who manages his own investment company. With enthusiasm, I shared my idea with him. He proposed a different approach. He introduced me to a trading doctrine — a variation of price action trading. In that moment, I had have basic information about that style, although I didn’t know the specifics. Selim had devised that variation and created several hours of training videos on his YouTube channel.
2- Confidentiality
Selim suggested that I engage in manual trading. This proposition would resonate with most individuals. However, possessing developer skills, I was inclined to script a solution and assess its effectiveness. Selim requested that we refrain from making this script publicly available and limit its usage to our personal trades. This stipulation is well-founded, as widespread adoption of the same approach could potentially dilute the efficacy of the tactic. Confidentiality is essential in every professional sphere, but I believe that trading, particularly algorithmic trading, demands an even greater level of confidentiality.
3- Study Your Lesson Well, Collect Requirements Good and Do Concept of Proof
With Elbaş, we didn’t begin coding right away. A research indicates that the cost of mistakes is much lower during the requirement identification phase compared to the coding phase. First, you need to thoroughly understand your lesson. How did we achieve that? We began by watching Selim’s videos, which were approximately 6 hours long. We didn’t watch them just once, but multiple times. We engaged in discussions about different scenarios. In videos, Selim explained tactics in real cases using price charts. We would pause the videos and contemplate the next steps. We brainstormed extensively about trades. The videos presented a few main tactics, but as we focused on trades, we began identifying extraordinary situations and documenting them. We accumulated a set of rules on paper, which surprised me. Initially, the doctrine seemed simple, consisting of only a few rules. As we progressed, I gained a deeper appreciation for the complexity of the human brain. Those few rules were straightforward for humans, but the computer required more intricate rules. We meticulously noted everything down on paper.
Now we have the rules in place. What comes next? At this point, you might be eager to jump into coding. However, I advise against it. First, a concept of proof is necessary. We created an Excel file and prepared a manual backtesting document. We selected random dates and examined what would have happened if we had followed this trade doctrine manually, as human traders did in the past. After choosing example dates, we decided whether to enter a trade or not without looking into the future, and we documented the results.
The results were promising. In the videos, we often witnessed the doctrine was working. Yet, this was the first time we experienced it ourselves and became witnesses to its success.
What did we learn from this phase? What could we have done better? What advice can I offer to new algorithmic traders?
During this phase, we took it seriously and invested a significant amount of time — perhaps a few weeks. Looking back, I now realize that we needed even more time. When you conduct experiments, you’ll observe that the concept works, but it demands more effort. More tests, better documentation, and improved standardization were needed. Before beginning to code, it’s crucial to create the necessary metrics from these manual tests.
Of course, it’s easy to say that you should perform these operations better. These tasks are often mundane and lack creativity. When we talk about quality, we often overlook how some quality-related operations can be dull, draining our energy and motivation. In my personal opinion, it’s worth sacrificing some money to delegate these monotonous testing tasks to others.
If you watched Silicon Valley series, you will remember how Jin Yang was unhappy when they requested from him to scrap meal photos from internet to train a meal identifier app.
Despite everything, our concept of proof was ready. We were prepared to move on to the next step.
4- What Was Our Target?
When Selim introduced this trading doctrine, he presented us with his trade results from the last two years. Using that data, we calculated the return on investment (ROI). Our primary question was: Is this ROI figure realistic? Will it also be effective in different time intervals? There was only one way to find out: write a script and do backtest! We initiated the script from a specific date, let’s say 1995, without any knowledge of the future. When the script encountered a trading opportunity, it had to decide whether to enter or not. If our bot entered the trade, it had to determine the details and parameters on its own. The bot would follow Selim’s doctrine in its behavior. This process is also referred to as a backtest. Essentially, we conducted a backtest on historical data.
5- About the Trade Duration
In the trading business, there are different timeframes, such as minute, hourly, and daily bars on the charts. The doctrine was designed to work on daily bars, and we also found it appropriate to work with daily bars. When trading based on daily bars, a trade can conclude in a few days or even a few months. While we didn’t have a clear experience in this regard, I assumed that there would be more competition and require more technical expertise when dealing with hourly and minute bars. There was a scene in the movie “Leon” where Leon provides information about weapons to a young girl. He said, “You have to start with a sniper to learn”. He mentioned that long-range weapons are the easiest to learn, and the knife is the last weapon you will learn. Similarly, we thought that there would be less competition in long-range trades. Of course, there was side effects to this approach, which we realized later on. We will tell about side effects.
6- Attention to Data Leak!
After two months, we obtained some remarkable results. The algorithm was producing very impressive numbers on past data without knowing future. At that moment, I was deeply surprised, and this was just the beginning. We hadn’t even optimized the hyperparameters yet. Upon seeing the initial numbers, I started running inside my home in excitement. It was an incredibly thrilling moment. While running I was rising one of my hand up. This feeling was like Dr Emmett Brown got enjoyed after sending Marty McFly back to 1985.
After calming down, I began to review and scrutinize the results in more detail. The numbers seemed excessively successful to me. There had to be a mistake somewhere. I reported the initial results to Elbaş, and we rechecked everything. We tested and manually repeated some trades. Everything appeared to be in order. It was a strange situation. This phase lasted at least a week.
Within the results, I noticed some minor anomalies. These anomalies were quite infrequent and didn’t significantly affect the outcomes, but my suspicions grew. I kept saying to myself, “I’ve checked all of the code, there shouldn’t be any anomalies”. Eventually, I discovered the reason behind these anomalies. Among thousands of lines of code, there were two incorrect lines placed very closely to each other. The first faulty line was flipping the entire dataset, and after a certain function, it was flipping it back to its original state. In the flipped dataset, the dates were arranged from the most recent to the oldest.
#that case was more difficult to detect, but lets put here a simple example
#to understand the issue
df = df[::-1] #df reversed
execute_one_function(df) #reversed_df used in some function
df = df[::-1] #reversed df, reversed again In other words, our system was making trades by foreseeing the future, but then immediately reverting the data to its original state to leave no evidence. These two incorrect lines in thousands of lines of code were not only producing highly profitable trade results but also flawlessly operating the program, leaving no trace of a bug.
Discovering and fixing this bug led me into deep contemplation. On one hand, I felt relieved because I had resolved the issue. On the other hand, witnessing how a single gap in quality could potentially negate all the efforts made, I became somewhat less optimistic. I realized how a single bug could lead the entire ship astray in the wrong direction, and it could do so almost invisibly, without leaving much of a trace.
I realized, if code quality is not excellent, the software can murder us in cold blood, on the ship without leaving any evidence.
7- Getting First Results
I don’t recall the exact results now, but as far as I remember, they weren’t high enough to necessitate entering an actual trade the next day. However, they were definitely promising. Perhaps the ROI (return on investment) could have been around 22%. Considering the hundreds of trades we had made in the USD/EUR market since 1995, an average trade could yield around a 22% gain. We know speak about latest roi later, but the initial number should have been something around 22%.
The EUR was introduced after 1999. Yet, a calculation method was used to estimate the EUR value even before 1999. We might have reached these figures maybe by the end of the first 2.5 months. I don’t think it was a bad outcome. However, there was still some room for improvement. Numerous optimizations were possible. We hadn’t achieved the trade performance that Selim had achieved manually, but we had only just begun to improve the system.
We had to keep moving forward as the initial numbers were promising.
Maybe a lesson can be drawn from here: you will make rapid progress in the first steps. but keep in mind that as the system gets more complex you will probably start to progress less in the same amount of time.
So, after getting these promising numbers, we needed spent efford beyond of that 2 months trial period.
8- Inventing New Algorithms
To replicate the doctrine, we needed to develop certain algorithms. Initially, we found and utilized some small algorithmic pieces from the internet. In my view, the most challenging aspect of the doctrine was “identifying a trend rule”. If our bot were a car, this “identifying a trend rule” part would be its engine. Looking at price charts with human eyes might make it seem easy, but in reality, it’s far from straightforward.
At the outset, to identify trends, we searched for and incorporated algorithmic pieces we found online. Some parts of our system were original, but others were adopted from various sources. The end result resembled a sort of “Frankenstein”. You know, a body composed of different and non-matching parts. That’s how I remember it, at least. But would Frankenstein come to life? Yes, absolutely. It was moving its limbs, walking around. Initial results indicated that it was indeed alive. However, the problem was that it wasn’t fully alive nor entirely functional. The outcomes were not satisfactory. The components weren’t perfectly designed for it and lacked harmony. We also didn’t thoroughly test it. It’s not very predictable. Could it suddenly decide to go rogue? Well, from data leaks, we realized that it was quite possible.
So, we resolved to redesign its algorithms from the ground up. This time, we wouldn’t rely on code snippets from the internet; we would construct its engine ourselves. This process was time-consuming. The results were better. We introduced more parameters and robust algorithms. But then other issues started cropping up. As we developed this code, it became increasingly intricate. Sometimes, I would look at the code and exclaim, “Oh, what a mess.” I knew that the code needed to be clean, and we tried, but it still proved challenging to manage. The software now resembled a massive factory boiler. As long as we kept tossing things into it and kept it running, there was no problem. However, I was aware that if we paused the project for a week or two, if the boiler stopped, getting it back up and running would take several days.
While attempting to simplify the code, another problem emerged. The code started running sluggishly! Waiting 10 minutes for results was no longer feasible during the development phase. We needed to take action.
Detecting the Dynamic Key Points?????
9- Code and Algorithm Optimization
During the development phase, one of the most interesting and enjoyable aspects, at least for me, was the part related to code optimization.
According to Pareto principle 80% of problems are often caused by 20% of the reasons. I recall a concept from the book “The Goal,” where factory production was hampered due to some bottlenecks. Similarly, our software was transferring data across numerous Python pages. We needed to find these bottlenecks and fix them. We inserted many print(time) checkpoints into the script. The culprit behind the slowdown was clearly identified: the k-means algorithm.
K-means was slowing us down. Initially, I had the thought, “And now are we really supposed to optimize the k-means library???” and wondered if it was even feasible. Of course, it’s quite possible that we couldn’t directly optimize the k-means library. However, luck was on our side, and our research led us to the k-means+ algorithm, which is faster than traditional k-means. It resulted good. Yet, we needed more optimizations. If you cant measure a system, you cannot control it. Thus, we focused on measuring the system. Drawing from my training, I recalled data flow diagram. One key factor contributing to computational complexity in software is data types. We decided to create a data flow diagram, illustrating which information with specific data types fed into which methods. We developed something akin to the following in Visio:
Following this, we began altering variable data types. For instance, we shifted from dataframes to numpy arrays. Naturally, this adjustment wasn’t straightforward. We had to adapt our algorithms accordingly. Yet, the process yielded positive results. In the end, the script ran much faster, and we also gained a clear map of the software’s architecture.
Then, we started to refactor the algorithms. We calculated the computational costs of the algorithms and attempted to reduce these costs by making necessary adjustments. The results were good. The optimization had worked. The program was now running much faster than before.
10- Optimum Percent of the Capital to Invest in Every Trade
Before, when I used to make amateur trades, I would divide my investment capital into a few portions, let’s say four pieces, and risk one of them each time. Additionally, when a trade failed, I would exit the trade with a stop loss. That was my approach. But this doctrine was different. It didn’t involve a stop loss. If a trade failed, the trade money was lost. So, we needed to divide our portfolio into multiple pieces. But how many pieces? At that point, we were fixated on the ROI metric, like a hunting dog focused on its prey. However, a data scientist should consider all the necessary metrics together.
Then, a question emerged in our minds: What percentage of our capital should we use for each trade? But before delving into that, there’s another crucial point to mention. Every algo trader should pay attention to an important question and answer it: Does their doctrine lose its efficiency by the time?
Having a great ROI might seem impressive, but if your doctrine loses its efficiency, it loses its meaning. We looked at the data from that perspective. The doctrine was solid. Its success was relatively evenly distributed since 1995. This was positive. However, one thing was clear. Some crisis years were really bad times to enter the market. It was evident, for instance, in the years 2000, 2008, and 2020.
We often tend to think about the worst-case scenario. What if we enter the market right at the start of a financial storm? This might be a highly unlikely scenario, but theoretically possible. So, we decided to write a function, create a simulation, and test it. This way, we could figure out the percentage of capital we could use for each trade.
We came up with a zombie case metaphor while discussing this problem. Imagine you’re in the game “Resident Evil”. You have a limited number of bullets. When you kill a zombie, you get a few new bullets. That’s good. And let’s say your average accuracy in killing zombies is high. It is also good. But sometimes, a door suddenly breaks, and a group of zombies attacks you all at once. You still have a limited amount of ammo. How many shots can you miss consecutively? If the world enters a crisis, how many failed trades can you afford consecutively?
This means your timing for entering the market for the first time is also crucial. We wrote a function and ran a simulation. Starting from a decided time (for example, 1999), every day the computer decided to become a trader and entered the market. I don’t recall the exact numbers, but it’s helpful to describe the concept.
According to the results, if you allocate 3% of your capital for each trade, you’ll have around 33 “bullets” (100/3). And your chance of going bankrupt is 0%. It doesn’t matter which day you entered the market since 1999. When we increased this number to 4%, if you entered the market on a random day, your chance of going bankrupt would rise to 6.5%. In such a scenario, you’d have only 25 bullets. So, we concluded that 6.5% risk is a good number.
11- The Ship Metaphor
There is a situation that people who enter the trading field later on don’t foresee. Normally, people act based on their emotions rather than their logic. It is more performance/cost efficient in daily life, but not in this case. Most of the time, they make decisions using their emotions and then justify these decisions with their logic. In a trade that lasts for weeks or months, relying on emotions will lead to a shipwreck. Today you might be motivated. When you board the ship to explore America, you are motivated. Will you still be motivated after 1 month? Will you still be motivated after 2 months? If motivation is the driving force behind your actions, it’s not a good sign. Beyond motivation, you should rely on other factors. No human boards a ship for an exploration journey with the intention of “I will feel hopeless and want to turn back after 1 month”. Regardless of anything, you must stick to the doctrine. As someone who has traded before, I was accustomed to sailing on the ship for a few months without seeing land. But the results showed that this journey could be more challenging. Sometimes, going from negative to zero could take 2 years. Once you set sail for America, the only way is to continue on the same path. Going back, especially after consuming half of your supplies, is not possible. In such a situation, all you can do is continue on the path you’ve set, hoping your calculations are correct. We had indeed invested a considerable amount of time for quality. But I wondered if the effort of two people was enough. If a person experiences three consecutive failed trades, they may psychologically break down. Were we prepared to face 24 consecutive failed trades in the worst case? Are you ready to face 6 months of consecutive failed trades? I believe trading is similar to jumping off a cliff. If your calculations are correct, there’s a safety net at the end of the cliff. The net saves you, and you receive your reward. So, before jumping, you need to make sure your calculations are accurate. You can’t just jump saying, “It’s probably right,” “There’s probably a safety net at the end of the cliff”. Here, jumping off the cliff is a metaphor. Imagine that after this jump, you find that there’s no safety net at the end of the path but you’re suffering. After jumping off the cliff 7 times in a row and falling to the ground each time, can you still maintain your optimism?
So, make sure your calculations are very accurate. Because even if a trade turns out to be profitable, it’s a painful process. And when you suffer a lot, you start questioning your calculations. The quality of your calculations must be excellent, and you must have documented this quality well. This way, when you have doubts, you can look at the documents again, confirm that everything has been well thought out and calculated, and eliminate your doubts, continuing your journey. This is where we realized something. This is more than a two-person job. Yes, the program we wrote, the ship we built, the route we calculated are probably correct. If trading was about waiting for results by firing a single shot, it would be easy to tolerate this uncertainty. But embarking on a ship journey that will last for 2 years with this uncertainty must be very difficult.
12- Team Job
When we received this doctrine from Selim, we had promised not to mention it to others. Therefore, we couldn’t add new team members. But this wasn’t the only constraint. We had a budget to hire one Python programmer to help us. However, the job wasn’t just about one person. To develop the code, we needed one Python developer. To identify bugs, we needed one tester. For testing the trades in real or dealing with issues that might arise during trade execution, we needed a trader. For monitoring metrics, for observing a data shift, a data scientist was necessary. And for documenting the software and creating diagrams, a business analyst was required.
Stealing a trade secret from a factory might not be very useful to you because you don’t have the necessary production tools to implement that trade secret. However, stealing and leaking a trade bot by its employees is much easier. As an amateur entrepreneur, what can you do to prevent this? Are we going to establish a company and even a security department for this? This wasn’t feasible for us. Such a measures are beyond of us.
13- Concerning the Nature of Money
There are certain images that, when stared at for an extended period, transform into three-dimensional forms. Depth increases, revealing an entirely different tableau. For me, gazing at money over time had a similar effect. I began to see angles and aspects that I hadn’t noticed before. Despite not being an economist by profession, I possess a solid understanding of economics. In the past, I had extensively studied economics for various reasons. I had predicted and anticipated the 2020 March crisis, even before the emergence of the coronavirus, through an economist’s perspective (at the time, he was unknown to the public). The PMI data were plummeting, yield curves were inverting (November 2019), despite there being money in the market, banks had stopped lending to each other, and a survey with industry leaders indicated that a potential crisis was expected in the first quarter of 2020.
Focusing on money for an extended time allowed me to see different perspectives. A significant portion of people in my country used to measure their investments based on the local currency (although this situation might have changed now). This is a flawed perspective. A group with a better grasp of economics would measure their investments based on the dollar. If they had $40,000 before and it increased to $50,000 through an investment, they considered it a success. This view is one floor upper but also erroneous. The dollar experiences inflation too. Individuals who are aware of dollar inflation factor that into their calculations. But it’s still not accurate. The shadow inflation of the dollar exists as well. In the 1980s, the calculation method for inflation was altered to make it appear lower. But does it end there? No, there are upper layers. In an environment where the monetary base is increased fourfold, can an investor who increases their money in the stock market by 25% be considered as having performed well? I’m not entirely sure. Even if the shadow inflation is 0%, that investor can now buy 25% more eggs from the grocery store. However, due to limited assets likely becoming four times more expensive, they probably won’t be able to purchase the same quantity of limited assets.
In such a scenario, as a successful investor, you can buy more water, more eggs, more bread, but that’s about it. You can’t acquire more limited assets. Consequently, a result emerges: are the investments we believe to be successful actually not successful? Is such a situation possible? Are many people caught in an illusion where they think they’re progressing, but in reality, they’re regressing?
Let’s examine this graph. The top 1% of the world has increased their share of total wealth. I believe that most individuals might actually be losing out.
Scenarios like these have caused a shift in my perspective towards money. Even if there had been a 50% profit within two years, I’m no longer entirely sure of how successful we would have truly been. Christopher Columbus discovered America and was proven correct. But was this discovery profitable for him? The route calculations were accurate. But were their foresights correct about whether the Spanish political leadership would honor their promises of governorship?
14- Final Boss: Can Your Presence Change the Market Behavior?
The title evokes the famous double-slit experiment. In this experiment, the rules of physics change depending on whether you participate or not. Now, let’s assume you’ve perfectly calculated everything for your trade bot. You’ve conducted all quality tests and you’re confident, ready to start actual trading.
The perspective discussed under this title is likely something most amateur investors haven’t realized. Will the market’s behavior change when you enter the scene? There’s no way to conduct this experiment in advance. There’s no way to conduct this experiment without paying a price. There’s no way to conduct this experiment by paying a small fee either.
I can hear you saying, “Will the grand market change its behavior for my tiny bet?”
You’ve tested all your hypotheses, all your experiments on a market where you weren’t present. But it’s not the same market as when you’re in. So, let’s ask our question again: Will the grand market change its behavior for your small bet?
Yes, it can change…
There used to be a saying that “everyone will be famous for a moment in the future”. The market, even for a brief moment, can alter the course of history for you.
Let’s simulate this scenario. The bots of the big players moving the market. These bots can see where you entered, your position, and where you’ll likely lose the trade. Let’s say you opened a trade with $1000. The cornerstone of our doctrine is entering the trade at resistance and support points. Similar to many trading tactics, we enter trades at critical points, and slight differences can easily lead to trade losses. Let’s say you’ve entered a trade. You’ve opened a position at a critical point. If the price drops by 0.5% just below that, your trade will result in a loss. Why wouldn’t the algorithm on the other side, even if for just a second, adjust the price to burn your $1000? Maybe it’s not feasible to temporarily decrease the market price by 0.5% for $1000. Also dont remember. Your 1000$ is not just 1000$. You used leverage. If you win, it will be 8000$. Perhaps someone else at your level also opened a $1000 bet. But the market makers didn’t consider $1000 feasible, so they didn’t change the market. But now, when combined with your $1000, it becomes $2000, and you’ve exceeded the feasibility threshold. Can the market, even if for just a second, change the price slightly for you to burn $2000?
We don’t know… We haven’t tried this. As you can see, we can’t experiment with this without losing money and time. Furthermore, we can’t experiment with a small amount of money. The market that didn’t react to $100 we spent for the experiment might react to $1000.
In an old Turkish movie, there were two actors named Kemal Sunal and Şener Şen. In a movie with them, the duo decides to trick villagers on a certain matter. For some reason, the villagers want Şener to die. Kemal fakes Şener’s death cleverly. They meticulously plan everything and fool the villagers. Now comes the burial seramony. Beforehand, the duo installs a mechanism where Şener’s supposed to be buried in the graveyard. This way, Şener can get oxygen through a hose underneath the ground. However, the hose is placed on the north side of the burial site. During the funeral, the imam says that the coffin should face south according to traditions. Kemal insists that the coffin should face the north, but due to the villagers listening to the imam, the coffin is buried facing south. The hose ends up on Şener’s ass side instead of his head, and he struggles slightly under the ground in the coffin.
Remember, even if your calculations are correct and your program works flawlessly, at the end of the road, the market might have one last trap waiting for you.
15- Latest Numbers
latest numbers were better than first results. In EUR/USD and GBP/USD markets, this algorithm suceeded 0.48 ROI in 514 Trades. It means if you enter a trade with 1000 dollars, in average you will get 1480 dollars back.
16- Inflation and Crisis Forecast
This project ended around June 2022. In May 2022 USA inflation reached to 8%. At that moment we thought that there is possiblity to rise into 20%(Altough nowadays it is declining still there is chance to rise). Also interest yield curve is reversed. So in following months we might experience another economic crisis. In such a situation, despite of the success of the trading bot, we decided to not making any investment for the it and dont make any real trade.
Imagine a hypotetical case, you invested for trading and after 2 years later you made your 100$ into 150$. But meanwhile dollar lost 30% of it value. We concluded it is not worth to take the risk.
Around February 2022, about the inflation I wrote such a article.
17- Conclusion
In this article, I discussed our experience with algorithmic trading and the challenges we encountered. I talked about the difficulties we faced in this project. Mentioning these challenges shouldn’t diminish your motivation. If you have a desire to develop a similar project, hopefully the experiences we’ve shared in this article will be helpful in anticipating challenges and achieving success.
Tiresias provided Odysseus with insights in the underworld that would assist him on his journey. I hope this article will be helpful for you as well.
