Search for a command to run...
Why DevOps is Central to DataOps: Faster & Reliable Data Product Development with a Collaborative Mindset
I am passionate about bringing data products to life. Bridging the gap between business and technology to develop data products that makes a different. My Hobbies are: Traveling, Biking and Cooking.
No comments yet. Be the first to comment.
Discover the essential best practices for implementing DataOps architecture, boosting productivity, and achieving error-free deployments
Discover how automated data pipelines revolutionize DataOps, from streamlining workflows to creating a competitive advantage.
Unlock the Full Potential of Your Data with Agile, Secure, and Scalable DataOps Frameworks to build and maintain Customer Centric Data Products
Delivering User-Centric Data Products with Agile Methodologies
DataOps, a fusion of "Data" and "Operations," addresses the challenges of developing data products by combining principles from Agile, DevOps, and Lean Manufacturing. It emphasizes collaboration, automation, and efficiency in handling data pipelines, enabling teams to deliver data products faster and with higher quality (Atwal, 2020, p.xxiii). If you are interested in the definition of DataOps take a look on the following blog post: Click Link
The exponential growth of data in recent years has created both incredible opportunities and formidable challenges for organizations. The ability to harness data efficiently and translate it into actionable insights has become a cornerstone of competitive advantage. However, traditional methodologies for example to develop software product often fail to keep up with the speed and complexity of data products development.
DataOps inherits components from (DataKitchen, 2018):
Lean Manufacturing: Focusing on the value adding processes by eliminating waste leads to a more efficient utilization of resources, with higher quality and lower costs.
Agile: Building the right product for the right people by increasing “the ability to react to unforeseen or volatile requirements regarding the functionality or the content“ of Data Products (Zimmer et al., 2015)
DevOps: Shared commitment towards the Data Product reduces information barriers (Culture of collaboration), automation (e.g., CI/CD pipelines), Infrastructure as Code and automated tests enable fast and reliable deployment of code into production in a high quality (Macarthy & Bass, 2020b)
Lets have a deep dive into the DevOps component:
The performance of software development increased significantly in the last 25 years. A key factor for the increased performance of software development - besides the adaption of agile software development - was the widespread adoption of DevOps philosophy. The software development process is similar complex to the manufacturing process of physical goods. Instead of a physical transformation of raw material to a finished good, software developers create a code (e.g., a new feature for an existing product). This code needs to be tested (quality and security) and to be deployed in the production by IT operations. This process contains different dimensions of complexity, for example: code creation, organizational framework, tools and method.
Traditionally, the release of new versions of a software product were a “high-stress affairs involving outages, firefighting, rollbacks, and occasionally much worse” which was conducted infrequently every few years (Atwal, 2020). There was no matured holistic approach to deal with the complexity involved. As a result, frictions occur in the development process. The software developer and the IT operators had different objectives. Software developers were eager to develop and test new products to fit the customer demands. On the other hand IT, operators were focusing on the stability of the production system. New releases frequently jeopardized the stability of the production system resulting in a negative attitude of its operators to new releases (König & Kugel, 2019). The technical debt of dependent legacy systems (e.g. monolith architecture) increases the workload for IT operations reducing the ability to test new features and increasing the risks of instabilities (Atwal, 2020, p.162). The figur below depicts explanatory the flow of the code in traditional software development:
Traditional flow of code in software development (explanatory)
The software development followed often the water fall principles, which means the work is executed subsequently. The software developer starts to work on the new release. After completing the whole code, it is passed -with some instructions- to the quality and security assurance colleagues in another department. After testing and approving the code, the package is passed to the IT Operations to deploy the code. The IT operations deploy the whole release at once. The infrastructure provision is done manually without significant automation. An extensive amount of time is needed to solve the configurations and dependencies manually.
In the first step the user requirements are specified in a very first phase. Requirement changes or testing MVP (minimum viable products) is rarely possible. Furthermore, the current project progress is difficult to assess since development, testing and deploying are subsequent tasks. Skill centric silo organization results in an extensive alignment and central planning effort without an end-to-end visibility resulting in huge cycle times and delays. The teams were often committed towards the functional organization and not the software product resulting in a dismissive information sharing and poor collaboration (Katal et al., 2019).
From a lean perspective the traditional approach has similarity with the mass production of Ford and Tylor. Several lean principles are not met in the traditional approach. From a flow perspective the code is piling up after each process step resulting in long waiting times. The optimum is an one-piece flow where small chunks of codes go through the process chain frequently. Small chunks of codes could be tested faster and errors will be detected when they were made (right first time). Unfortunately, errors will be detected in the traditional approach late in the process resulting in extensive amount of reworks since error dependencies need to be resolved as well. Rework is one type of waste (Muda). The huge amount of work in progress (code between processes), the unbalanced teams resulting in idle times adding more types of waste to the traditional approach. In the traditional approach it is difficult to manage feedback from the production efficiently since information is shared dismissive between IT Operations and teams, but as well with the customer of the software product resulting in the risk that a software product is built which is not meeting the customer demands. The traditional approach is not compatible with lean ideas resulting in a poor performance. Based on these challenges the DevOps philosophy was developed. „DevOps is described as a software engineering culture and philosophy that utilizes cross-functional teams [Developer, IT-Operations, QA, Security] to build, test, and release software faster and more reliably through automation“ (Macarthy & Bass, 2020b). Companies applying DevOps show frequently outperformance in key success indicators in comparison to the traditional approach. Key success factors are for example: Deployment frequency, lead time for changes, change failure rate, time to restore services (Portman, 2020). The main idea is to facilitate rapid software development and deployment by bringing small releases constantly into production ready status (Lwakatare et al., 2016). There is no conclusive definition of DevOps. In general, there are four pillars of DevOps (CAMS): Culture, Automation, Measurement and Sharing (Katal et al., 2019). DevOps embraces a culture to bridge the conflicting interests for Developer and IT Operator. Small, self-organized and autonomous cross-functional teams (Developer, It Operator and Quality/Security assurance) with a shared commitment towards the product and not the function are built. Thus, the time to share information is reduced significantly, end-to-end visibility is given and the collaboration is strengthened. Learning and innovation cycles are improved through continuous feedback. An explanatory example of releasing and deploying code is depicted in Figure 4. Through continuous integration and continuous delivery/deployment - also known as CI/CD Pipeline – a commit can be brought into a production ready status through automation in a reliable and in high quality fast way. Thus, small pieces of code can be shipped fast into production reducing the work in progress and the cycle times significantly (“One-Piece-Flow”). Failures are detected right after their emergence and can be solved immediately reducing the amount of reworking and thus eliminating waste (“Right First Time”). This procedure enables to test a feature quickly and to adapt to the customer needs. MVP (minimum viable products) can be presented in an early phase of the projects. Agile methods can be applied like working in sprints with SCRUM. Working in sprints is a key enabler to refine the customer requirements during the development phase in order to build the right product for the right people. For explanatory reasons the CI/CD pipeline is broken down in process steps (in reality it is a continuous process). One sprint follows several phases: Plan, code, build (building a deployable package like a docker image), test, release to a repository, deploy, operate, monitor. It is very important that the production system is monitored and errors or warnings are continuously feedbacked. Continuous feedback assures that the necessary information is brought to the developers to react fast and preventive to possible failures (Continuous Improvement, Right First Time).
The processes for software development are highly automated inclusive the quality testing and the provision of infrastructure. An explanatory example of automated infrastructure provision (infrastructure as code) is shown in the figure below:
Containerization (e.g., with Docker), microservice architecture and tools to configure and provide the infrastructure enabling infrastructure as code. Automated and replicable infrastructure reduce significantly technical debt.
The blog discusses the evolution of software development from traditional, siloed, and waterfall-based approaches to modern, collaborative, and automated workflows enabled by DevOps. Key challenges with the traditional approach include long cycle times, poor collaboration between developers and IT operations, and technical debt from legacy systems.
DevOps addresses these challenges by fostering a culture of collaboration through cross-functional teams, automating processes like testing and infrastructure provisioning, and implementing Continuous Integration and Continuous Delivery (CI/CD) pipelines. These practices enable faster deployment, quicker detection and resolution of errors, and the delivery of small, incremental releases. This aligns with Lean principles, such as minimizing waste and enabling one-piece flow, and Agile principles, such as flexibility and responsiveness to customer needs.
What are your thouhts about DevOps? Does it help you in your daily work?
Atwal, H. (2020). Practical DataOps. In Practical DataOps. Apress. Practical DataOps: Delivering Agile Data Science at Scale[Book]
DataKitchen. (2018). DataOps is NOT Just DevOps for Data. DataOps is NOT Just DevOps for Data | by DataKitchen | data-ops | Medium
Katal, A., Dahiya, S. ., & Bajoria, V. (2019a). DevOps: Bridging the gap between Development and Operations. DevOps: Bridging the gap between Development and Operations | IEEE Conference Publication | IEEE Xplore
König, G., & Kugel, R. (2019). DevOps – Welcome to the Jungle. s40702-019-00507-8.pdf
Lwakatare, L., Oivo, M., & Kuvaja, P. (2016). An Exploratory Study of DevOps Extending the Dimensions of DevOps with Practices. ICSEA 2016 : The Eleventh International Conference on Software Engineering Advances.
Macarthy, R. W., & Bass, J. M. (2020a). An Empirical Taxonomy of DevOps in Practice. 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 221–228. https://doi.org/10.1109/SEAA51224.2020.00046
Portman, D. (2020). Are you an Elite DevOps performer? Find out with the Four Keys Project.
Zimmer, M., Kemper, H., & Baars, H. (2015). The impact of Agility Requirements on Business intelligence Architectures.
