Incidents (on call)
Roles:
- Lead: asks questions and keeps the investigation moving (but doesn’t actually investigate)
- As the issue evolves, can suggest role changes - e.g. if a new main investigator makes sense
- Scribe: realtime investigation updates in the incident channel (for channel members)
- really helpful for people who arrive on the call late or have to hop off and return later
- Communicator: wider, non-technical updates for users not involved in the incident
- Main investigator: drills into technical hunting + debugging
Incident.io
- The practical guide to incident management • Incident.io 📚
- incident.io • Incident management 🛠️
PagerDuty
- PagerDuty • Incident response and on-call management 🛠️
- Simulated Incident Call Recording • PagerDuty 📺
- pagerduty: Alerts
- pagerduty: intro: On-Demand eLearning
- pagerduty: PagerDuty Services Example: Split High and Low Severity Alerts to Different Escalation Policies - The Ops Community ⚙️
- Dynamic Notifications • “Add severity levels to incidents to support effective incident triage” • PagerDuty 📚
- Useful when different alerts have different support hours
- e.g. business hours for most, but 24 hrs for “service failed healthcheck”
- In alert policy (e.g. on Google Cloud), add a “severity” metadata label to the alert PagerDuty will receive (e.g. on Google Cloud, add a “severity” label to “user labels”)
- On PagerDuty, go to Service > Settings > Assign and Notify and update “Notifications” to “Dynamic notifications based on alert severity”
- Useful when different alerts have different support hours
Hosting
App Hosting:
- 📖 Netlify • Develop & deploy the best web experiences in record time
- Fly.io
- GitHub - MightyMoud/sidekick: Bare metal to production ready in mins; your own fly server on your VPS. • Open source fly.io clone you host yourself • Mahmoud Mousa 🛠️
- 📖 Render • Cloud Application Hosting for Developers
- CloudFlare Workers:
- Workers • “CloudFlare Workers lets developers deploy serverless code written in JavaScript, Rust, C and C++ to Cloudflare’s edge, closer to the end user. This enables lower latency and better performance for your users. Workers is built on the same technology that powers Cloudflare’s global network, so you can deploy a sub-30ms function to more than 200 cities in less than 30 seconds” • Cloudflare 🛠️
- 🌲 cloudflare worker link tree • Walks through setting up a worker
- Cloudflare Workers • Cloudflare Workers docs
- Adding Caching to a Cloudflare Worker • How to add basic caching to a Cloudflare Worker serverless worker.
- Setting up a production ready VPS is a lot easier than I thought • Extremely detailed walkthrough of how to securely deploy a web app yourself for a fraction of what services like Vercel would charge • Dreams of Code 📺
Continuous Integration
- Dependency management:
- Pros/cons of Dependabot vs Renovate?
- Renovate, a Dependabot alternative • “I won’t introduce Dependabot. Lots and lots of developers use it daily on GitHub. I do use it as well. However, it suffers from two drawbacks: While it’s perfectly integrated with GitHub, integrations with other platforms are less seamless.It’s limited in the list of ecosystems it supports For example, I generally use Docker Compose files for my demos. When necessary, I use Kubernetes. Dependabot supports none” • Nicolas Fränkel 📖
- Automate Dependency Management With Renovate From Mend • DevOps Toolkit 📺
- Try to update your dependencies with Renovate or Dependabot • Oles Maiboroda 📖
- Dependabot alerts triaging in GitHub • Rob Bos 📖
Continuous Delivery
- Don’t write software like you write books: the value of delivering early and frequently • Lucas F. Costa 📖
Feature Flags
- LaunchDarkly has no free tier. Is there a solid free alternative for open source users? A reliable way to create this myself (e.g. with a “Feature Flags” Notion DB)?
- can set the flag to a different value in each environment
- e.g. off in prod, on in local dev
- why flag?
- avoid long-lived feature branches that grow large
- support continuous delivery of small changes even if they are pieces of incomplete features that won’t be shown yet
- can deploy code without releasing the feature (decouples those two things)
- flag/toggle categories (see Feature Toggles (aka Feature Flags):
- release flags (for deploying incomplete/new features via CD)
- experiment flags (for A/B tests)
- ops flags (to rollback in case of perf issues)
- permissioning flags (to control who sees a feature)
- how to implement?
- inversion of control — instead of calling the flag provider inside the logic, pass in a boolean result from outside the function (way easier to test)
- place flag at UI level whenever possible rather than sprinkling throughout the logic
- e.g. hide a user input so you don’t have to disable the logic it triggers
- add a “flag removal” task whenever one is added
- consider setting an expiry date that would cause tests to fail
- how to test
- test with current prod config + any flags planned to turn on now both off and on
- also test with all flags on and all flags off
Links
- Welcome to LaunchDarkly • LaunchDarkly docs 📚
- LaunchDarkly React SDK • LaunchDarkly docs 📚
- Feature Flags Facilitate Fabulous Fast Flexible Features • Builder.io 📖
Security
- Sniffnet — comfortably monitor your Internet traffic 🕵️♂️
- One Stop Shop Security Hub For Developers! • Aikido • DevOps Toolbox 📺
Inbox
-
It works on localhost • Fireship 📺
- I didn’t know about Cloudflare Tunnels until this video
- When to use, How to set up and use, why not ngrok
- A Boring Announcement: Free Tunnels for Everyone • Cloudflare 📖
- Tunnel | Zero Trust App Connector • Cloudflare 🛠️
-
security: Input Validation - OWASP Cheat Sheet Series - for text inputs
-
Escalation Policies and Schedules - PagerDuty
-
Instead of “auth”, we should say “permissions” and “login” | nicole@web
-
OPA Guidebook - Sangkeon Lee 📕
-
cautionary tales: CrowdStrike Unofficial Retro
-
Safeguarding Web Applications Against Token Theft: Fortifying Security and Trust by Shikhar Kapoor • Introduces DPoP (Demonstrating Proof of Possession) standard for generating and storing tokens used in web requests more securely than Oath 2.0 • Shikhar Kapoor 📺
-
Creating naming conventions for flags - LaunchDarkly docs
-
The Complete DevOps Roadmap [2024] • Great outline of all subtopics in this area • Programming with Mosh 📺