Minimizing Toil & Wastage in Software Development

 

Toil

• Definition: Toil refers to tasks that are manual, repetitive, automatable, tactical, and devoid of enduring value. These tasks scale linearly with the growth of the service and do not contribute to long-term improvements.
• Examples: Routine maintenance, manual deployments, repetitive testing, and handling alerts manually.

Wastage

• Definition: Wastage involves any activity that does not add value to the end product or service. This includes inefficient processes, unnecessary steps, and delays.
• Examples: Waiting for approvals, redundant meetings, and excessive debugging due to poor code quality.

Smart CI-CD Pipeline

Continuous Integration (CI)

• Linting: Analyzes source code to flag errors and stylistic issues, ensuring code quality and consistency.
• Unit Test Coverage: Verifies individual components work as intended, detecting issues early.
• Static Application Security Testing (SAST): Analyzes code for security vulnerabilities without execution, identifying security issues early.
• Build: Compiles source code into executable artifacts, ensuring code is ready for testing and deployment.
• Publish: Stores built artifacts in a repository, centralizing management and distribution of artifacts.

Continuous Delivery (CD)

• Deploy to Non-Prod Environments: Deploys application to various environments iteratively, ensuring thorough testing in production-like environments.
• Dynamic Application Security Testing (DAST): Identifies security vulnerabilities in web applications by simulating real-world attacks, providing a realistic assessment of security.
• Regression Testing: Ensures new code changes do not adversely affect existing functionality, maintaining software stability.
• Performance Testing: Evaluates the speed, responsiveness, and stability of a system under a given workload, enhancing user satisfaction.

Left Shift Strategy

• Purpose: Integrates testing, quality assurance early.
• Benefits: Early detection, improved collaboration, enhanced quality.
• Examples: Continuous Integration (CI), Test-Driven Development (TDD), Static Code Analysis, Pair Programming, Automated Unit Testing, optimizing inner loop

Security left shift example

Automate the Code Review

• Purpose: Maintain code quality and consistency using tools and AI.

• Benefits:
• Time Savings: Automates routine checks, freeing up developers' time.
• Consistency: Ensures consistent enforcement of coding standards.
• Improved Quality: Identifies potential issues and optimizations early.

• Bitbucket / Github
• Pull Request Templates: Standardize PRs with necessary information.
• Static Code Analysis Tools: Use SonarQube or CodeClimate for quality and security checks.
• Automate Reviewer Assignment: Assign reviewers automatically based on code changes.
• Merge Checks: Enforce quality requirements before merging PRs.

Bulk Build & Deployment 

Bulk Build & Deployment is utilized in Release Management by some organizations to streamline the release process, where different teams build and deploy a large number of applications on a specific day of the sprint.

Challenges : Coordination Complexity, Build Quality, Resource Allocation, Visibility and Tracking, Error Handling, Environment Consistency, Automated Testing, Rollback Procedures.

Automate Creation of the Branch for Bulk Apps for Every Sprint

• Reduces manual effort and ensures consistency.
• Speeds up the initial setup process.

Automate PR Creation

• Streamlines code integration and reduces administrative tasks.
• Ensures standardized and error-free PRs.

Automate Merging of PR

• Speeds up the integration process and reduces repetitive tasks.
• Ensures consistent and error-free merges.

Scheduled Automate Build and Deployment in Non-Production

• Ensures regular testing and deployment, catching issues early.
• Reduces manual effort and provides continuous feedback.

Bulk Branch Deletion of Not Used Branches After the Release

• Keeps the repository clean and organized.
• Frees up resources and reduces manual cleanup tasks.

Monitoring and Alerting

Purpose: Track the performance and health of applications.

Benefits:

• Proactive Issue Resolution: Detects and resolves issues before they impact users.
• Improved Reliability: Ensures the system remains reliable and available.
• Data-Driven Decisions: Provides insights for informed system improvements.

Scripting and scheduling

• Cleanup Script: Automates removal of temporary files and old logs to free up disk space.
• Space Check Script: Monitors disk space usage and alerts when thresholds are exceeded.
• Installation Script: Automates software installation and configuration for consistency and efficiency.
• Log Management and Rotation: Manages and rotates log files to prevent excessive disk space usage.
• New Repository Creation Script with Predefined Rules: Automates creation of new repositories with predefined structures and rules.