The Challenge
Our client was missing the necessary technology to achieve sustainable and scalable growth that would allow them to expand their business into global energy trading markets. To address the technology gap, they needed to build a wholesale trading platform and YLD came in to help assemble and integrate it with the market systems.
The Approach
As part of our partnership, YLD’s primary focus was to help facilitate the development and release of a trading platform and its associated infrastructure. Powered by machine learning, this platform would enable energy traders to engage in trading energy-based products using company assets. Its core components leverage price forecasting to automatically create bids that flow into the market, based on pre-defined parameters.
Since the start of the engagement, YLD has supported the client across multiple technical areas, including backend engineering, data engineering, DevOps and MLOps. Our backend engineering team contributed to the build of the whole platform’s backend, while the DevOps and data platform teams were tasked with ingesting the data, using data pipelines built by data engineering teams. As a result, the data is used to predict energy prices.
Implementing changes to the core architecture
During the initial stage, changes on the backend were needed to pave the way for larger deliverables. Our team assessed the core architecture and introduced changes to the database and ORM solutions. There were some legacy microservices which were no longer useful and were merely consuming resources, prompting the decision to implement architectural changes. The work involved constructing the necessary database models and constraints using Django ORM and shifting microservice access to the database through these models. Additionally, the effort went into providing architectural vision to the product team and changes striving for operational excellence in pipeline efficiency, local development, and unit test improvements.
Building a machine learning-driven energy trading system
YLD contributed to the build of pipelines that would ingest site operations data from third-party data providers. This data would be used to operate one of the power plants and build an ML-driven energy trading system. The pipelines were built with an in-house Python framework with AWS MWAA (Airflow) applied to orchestrate, schedule, and execute them. Our team also improved the robustness of the system by adding unit, integration, and machine learning specific tests.
The infrastructure provisioning was upgraded to be fully automated and declarative, which allowed full deployment to be more predictable and quicker. This automation also enabled proper disaster recovery/ backup plans that reduce the business risk.
Our team also contributed to the creation of the whole process of monitoring and incident management using PagerDuty and DataDog workflows. We chose a FollowTheSun approach to accommodate the team across different time zones, and it included procedures, templates, and automation to reduce stress during tough times.
Engineering approaches to generating bids
As a response to the client’s challenge to ingress high-volume data from the power plants, our team engineered a solution to deliver meaningful and granular data to the different stakeholders. We used IoT Core, Firehose, Lambdas, SQS and EventBridge Pipes to guarantee delivery and proper treatment of the data. To connect the various sites securely and reliably, networking infrastructure needed to take place. This was achieved by leveraging transit gateways and site-to-site VPNs.
Our client realised that Bitbucket was limiting the collaboration between the teams, mainly in PR reviews and on pipelines. After careful analysis and consideration, we migrated all the repositories and CI/CD pipelines to GitHub.
To submit algorithmic-generated and trader-generated bids to an Independent System Operator, a new microservice was needed to tackle multiple stages of the process.
Ahead of the product’s launch, extensive testing, implementation of fixes and various improvements in the overall workflow of the service responsible for submitting orders into the energy markets for multiple products, to ensure a successful release.
The Deliverables
YLD exceeded the client’s expectations by introducing optimisations that translated into almost 120x faster data processing, which was reduced from around 24 minutes to approximately 12 seconds. Our team worked with a test-driven development approach, ensuring appropriate coverage and high-quality delivery throughout, resulting in an expansion of overall test coverage to 70%.
As part of our engagement, we established end-to-end data ingress for a new site in under an hour, a process that originally took months to complete. As a result of our continuous learning and innovative approaches, data can now be easily integrated from new sites, greatly enhancing scalability and fostering significant cost-saving.
Closing the Engagement
YLD helped build services to trade and operate assets in multiple locations with a major release of a trading platform. Our joint effort enabled our client to connect to the market and submit all the data.
By using their new trading platform, our client was able to bring assets online rapidly, and participate in energy markets, realising revenue quickly and meeting contractual obligations.
View More Work
