As a result of my work at Lojika, i've prepared a conference paper that explains a high level end to end solution framework of Lojika's software/product development. In this paper, i denoted and described fundamental approaches and various technologies to clarify how we reached the optimal level of responsiveness and effectiveness.
I divided this whole study into 2 parts to make audience not to bore. First post is all about the introduction stuff and the second part will be elaborated by explaining flaws and bottlenecks of communication flow in Lojika. Some methods / technologies to resolve these problems are also described. Furthermore, the most important part is about how we've applied continuous delivery practices / principles of agile software development to overcome all those problems in the course of my work.
You can also download this paper.
Abstract—Consistency, responsiveness and reliability are some of common issues for companies in online business that deliver value for their customers. They need to be able to bright out new ideas and changes to production without minimum technical errors. As an agile development methodology, continuous delivery presents best practices for creating reasonable and reliable builds without any special effort. No doubt about that this brings along efficiency and effectiveness. The aim of this paper is to investigate observed effects of continuous delivery on a startup company, Lojika. Due to the importance of it’s early users contribution, well-structured feedback and communication mechanism is a must requirement in order to communicate the feedback throughout the teams including product, development and marketing as quickly as possible. For that reason, probing into the development life cycle of Lojika will be very helpful to enhance the concurrent approaches with new case studies.
Keywords—case study, continuous delivery, continuous communication, build pipeline, agile development, automated testing,devops.
Rapidity, in the new era of software development is a significant necessity in the dynamic, fast growing and changing markets. Hot companies like Google, Facebook, Uber, Airbnb, Tesla are able to react changes quickly, This makes them serious competitors in the market. Many Google services see releases multiple times a week and Facebook releases to production twice a day . Even in aerospace industry, responsiveness and rapid changing is an emerging factor to become more effective and efficient in their business. SpaceX, a space transport company founded in 2002, has been succeeded in integrating agile development practices into their development processes. Kevin Venner, CIO of SpaceX expresses that ”we release at least one a week, sometimes midweek” .
Continuous delivery is not just for startups and lean organizations. Large scale businesses and enterprises can be guided by agile practices to adopt agility on their development processes in an effective and efficient way. Huawei is one of the enterprises which have been successful in applying continuous delivery agile practices. Huawei is $40B company delivering communications technologies for telecom carriers, enterprise and consumers. Analytics regarding R&D of Huawei are tremendous; working 2000 developers worldwide, has 1000 applications, more than 2000 releases per day, more than 100.000 compile&builds per day, more than 1 million test cases run per day and so on. . Another success story is about HP LazerJet firmware team. They build the firmware and the operating system that HP LazerJet printers run on. After they discovered the slowness and ineffectiveness in their operations, they revealed that ten percent of the team were spending their time on code integration. Additionally, other process were more time consuming than it should be. They re-architected their entire system from the ground. Code integration mechanism was made by continuous integration servers and they put a large amount of automated testing including both unit tests and 30.000 automated functional tests in place. After rebuilding everything, they report that their time on continuous integration is 2% and they spend 40% of their time to build new feature .
Despite the fact that some specific differences between all types of software development life cycles (SDLC), the most known phases are requirements gathering and planning, designing and development, testing and deployment . Unlike the others, in conjunction with increasing value of the agile principles, ”efficiency in delivering software” becomes the main focus amongst them. Considering fundamental principles in agile manifesto: ”Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”. The main purpose of continuous delivery is to reduce the risks associated with delivering new versions and increasing feedback and improving collaboration between development, testing and operational responsible for delivery.
The continuous delivery and deployment practices have been proposed to enable accelerated value delivery, reduce the risk of failure, increase productivity, improve visibility, feed- back and quality . In many organizations, release frequency is measured in weeks or months, and the release process is certainly not repeatable or reliable. It is manual and often requires a team of people to deploy the software even into a testing or staging environment .
In this study, we introduce an implementation of practices of continuous delivery in a car-pooling application TAG (Tek Araba Gidelim) which is developed by Lojika. We investigate observed effects of continuous delivery on our communication and feedback structure within the organization consisting of our product, development and marketing team which are all involved in our continuous delivery process. Rest of the paper is organized below:
In the next section, some important details of continuous delivery practices are described to understand what it must be considered to apply it successfully. In the third section, the reason of why Lojika prefers continuous delivery practices rather than other approaches is described. In the forth section of the paper, implementation details of a successful adoption of continuous delivery approach within Lojika are introduced. The results of the study are presented in section 5. Section 6 concludes our study by summing up all sections and results.
Understanding Continuous Delivery
Continuous delivery is accepted as a subset of agile which the software is ready for release while development is continuing, which means that there should be no extra effort to make reasonable builds and that sets continuous delivery apart from ”traditional agile” .
Another well known definition of continuous delivery; ”a software development discipline where you build software in such a way that the software can be released to production at any time” .
In essence, whole story begins with continuous integration. Firstly, ”integration” needs to be clarified. Development in a team is a kind of process that has to contain the integration activity on every changes throughout the development. Integration phase is unpredictable and can take more time than the programming period . A good rule of thumb and de-facto best practice to sustain integration activities of developed components are to use source control and configuration management tools such as Git, Mercurial or Subversion. As illustrated in figure 1, code pushing phase is the first attempt to trigger the pipeline from the commit stage by developers, the commit stage is automatically triggered process comprising interconnected steps such as compiling code, running tests (unit, acceptance and non-functional performance tests etc), code coverage and building artifacts. Whenever a failure occurs in the automatic process, it may help to ensure that problems leading to integration failures are solved as quickly as possible by those responsible. Continuous integration is primarily focused on asserting that the code compiles successfully and passes unit and acceptance tests. However; it is not enough to create a continuous delivery process. . Continuous delivery is more complicated and is about the processes that have to happen after code is integrated for application changes to be delivered to users. Setting up a continuous integration tool such as Jenkins or Bamboo to automate applying code changes continuously does not indicate that continuous delivery is performed. It is just limited to use continuous integration server.
Nevertheless; continuous integration is a first prerequisite to step up to continuous delivery, that is to say that applying the whole commit stage automatically is the inception process of the continuous delivery. Not only that but also acceptance test stage, non-functional validation stage, manual test stage and release stage must be considered to achieve a success in continuous delivery. These stages are enabled through ”the deployment pipeline”.
The commit stage asserts that the system works in low technical level and validate the rules that pass to be accepted that code base is stable . This stage is the very early stage to provide quick feedback to developers. Developers check in code to the source control management system and continuous integration server (CI server) automatically polls the changes from source control management system. CI server compiles the source code and executes unit and integration (component) tests. Code analysis process is performed to check the quality of the code base. In case of an error in this stage, the pipeline stops and notifies the responsible developers. Otherwise, if everything goes well, the generated artifacts, reports or metadata that will be used later in the pipeline are uploaded to artifact repository server and the pipeline step up to the next stage . This transition between these steps has to be done automatically. As it is illustrated in figure 1, to sustain the feedback cycle after any failure, delivery team has to be informed about the failure. Otherwise, in case of passing successfully the build & test stage, developer team should proceed their coding activities. But this does not mean that developers are done of their work. In every stage of the delivery pipeline, it is possible to get negative feedback by QA team.
According to Humble and Farley’s book, there are essential principles and practices to perform an effective commit stage;
• Provide fast and useful feedback. Fail fast to identify errors early. In order to do that it is extremely required to put as many unit and component tests as possible
• What should break the commit stage. Even if everything goes well and commit stage is successful, there may be something wrong with the code base quality. There should be a reasonable threshold to achieve a level that the code base is acceptable
• Tend the commit stage carefully
• Give developers ownership
• Use a build master for very large teams
The acceptance test stage is the next automatically executed stage, it is extremely valuable and it can be very expensive to create and maintain. In many organizations, acceptance testing is done by a separate dedicated team, but this strategy can cause a misunderstanding that developers cannot feel ownership and responsibility of the acceptance stage of the product. Instead of manual testing, performing an automatic test process provides quite improvements on communication structure, reliability and responsiveness:
• Manual testing takes a long time and expensive to perform, but executing acceptance tests automatically is a time saving activity. This also helps teams to focus more on critical issues rather than doing less important and repeatable tasks.
• Automating acceptance tests protects application when large-scale changes are made.
• Automating acceptance tests reduces productivity, predictability and reliability
• By automating the process, whole team owns the acceptance tests
As we discussed earlier, the aim of continuous delivery is to be able to deliver software frequently and essential building blocks of a successful implementation of the deployment pipeline are given below:
• Automate everything
• Keep everything in source control
• Build quality in
• Only build your binaries once
• Deploy the same way to every environment
• Smoke-test your deployments
• Deploy into a copy of production
• The process for releasing/deploying software must be repeatable and reliable
• If something is difficult or painful, do it more often
• Done means released
• Everybody has responsibility for the release process
• Improve continuously
Continuous delivery is available for all types of companies. But there may be some procedural differences from the company to the company. Depending on it’s best practices and basic principles, companies can use tailored and differentiated models for themselves .
This is the end of the first part of my study. You can continue reading part 2 .
 J. Humble. (2016) The case for continuous delivery. [Online]. Available: https://www.thoughtworks.com/insights/blog/case-continuous-delivery
 A. Morrison and B. Parker. (2016) An aerospace industry
cios move toward devops and a test-driven development envi-
ronment. [Online]. Available: http://www.pwc.com/us/en/technology-
 T. Z. . A. Walgren. (2016) Huaweis cd transformation journey. [Online]. Available: https://www.youtube.com/watch?v=xsjsWU23k80
 M. Pais. (2016) Continuous delivery stories. [Online].
 Wikipedia. (2016) Systems development life cycle. [Online]. Available: https://en.wikipedia.org/wiki/Systems
 D. F. Jez Humble, Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Addison Wesley, 2011.
 C. L. Juha Itkonen, “Perceived benefits of adopting continuous deliv- ery practices,” 10th International Symposium on Empirical Software Engineering and Measurement (ESEM). Ciudad Real, Spain: ACM, 2016.
 K. Morris. (2016) Continuous delivery vs. traditional agile. [Online]. Available: https://dzone.com/articles/continuous-delivery-vs
 M. Fowler. (2016) Continuousdelivery. [Online]. Available: http://martinfowler.com/bliki/ContinuousDelivery.html
 K. Beck, Extreme programming explained : embrace change,ser. The XP Series. Boston, San Francisco, Paris: Addison-Wesley, 2000, autres tirages : 2001, 2004. [Online]. Available: http://opac.inria.fr/record=b1098028
 B. Fitzgerald and K.-J. Stol, “Continuous software engineering: A roadmap and agenda,” Journal of Systems and Software, no. 25, pp.19–59, 8 2015.
 C. Tozzi. (2016) Continuous integration vs. continuous delivery: Theres an important difference. [Online]. Available: https://devops.com/continuous-integration-vs-continuous-delivery-theres-important-difference/
 L. Chen, “Continuous delivery huge benefits, but challenges too,” IEEE Software, 3 2015.
 J. Allen. (2016) Patterns for continuous delivery. [Online]. Available: https://www.infoq.com/articles/Continous-Delivery-Patterns
 N. Bozic. (2016) Continuous delivery through pipelines.
[Online]. Available: https://www.go.cd/2015/12/28/gocd-continuous-delivery-through-pipelines/
 (2016) Jenkins. [Online]. Available: https://jenkins.io/
 (2016) Appium introduction. [Online]. Available: http://appium.io/introduction.html
 (2016) Apache jmeter. [Online]. Available: http://jmeter.apache.org/
 (2016) Blazemeter. [Online]. Available: https://www.blazemeter.com/
 (2016) Sonarqube. [Online]. Available: https://en.wikipedia.org/wiki/SonarQube
 (2016) Jfrog. [Online]. Available: https://www.jfrog.com/
 (2016) Liquibase db change management. [Online]. Available: http://www.liquibase.org/
 (2016) Docker. [Online]. Available: https://www.docker.com/