Lately I’ve been doing a lot of thinking on development processes and quality, specifically for large-scale, professional PHP-based projects.
I’ve got some decent experience writing unit tests with PHPUnit, using a smoke testing script, and even writing use cases (it’s been a while, though).
Generally speaking, my perception is that software development shops that really care about and emphasize quality have processes that consist of things such as:
- Writing use cases
- Performing code reviews
- Unit testing
- Continuous integration
- Writing test cases and test plans (ala a Quality Assurance department)
…and so on.
For those of you that happen to read my blog, what do you think? Do you use these types of quality control measures? If so, have you used them for a long time, or did you one day wake up and realize “Gee, I need to be doing more for quality control”? If you’re not using them, why aren’t you? Lack of time, budgetary constraints, etc.? Or are you just not aware of them?
Just some food for thought. Please leave comments here or email me (brian at deshong dot net) if you’d like to provide some input.
We are a 4 man team. We do peer code reviews. Our official policy is to have someone other than the developer of the code push that code change out to production. That allows for a second set of eyes to review the code. We don’t do unit testing. I can see some benefits. But, I would say that 75% of our problems would not be caught by unit testing. Most of the errors we see in production are either data related or happen because of unexpected input. So, checking that function foo always returns 1 when it gets correct input is not so helpful.
@Brian D.: Quality Assurance and Quality Control are two different things. Quality Assurance consists of the following steps:
1. Define the requirements for your app
2. Implement the app
3. Validate that the app does what you said it would do (and nothing else)
The third part is Quality Control, and this includes most of what you listed. Testing, code reviews, CI, etc. But for this to be a Quality Assurance process, you must first have a definition of the expected system. You can’t know how to test unless you know what’s correct and what’s incorrect.
That first part of defining the system is also useful for part two — how do you know you’re done coding, unless you have spelled out the requirements? All these parts of the process are linked in this way.
@Brian M.: You should also write ‘negative’ unit tests to ensure classes behave appropriately when given invalid input.
I’m proud to say that we’re doing all five things you mentioned (including negative unit tests) to a small degree, it’s been worth it and we hope to implement them further.
@Brian M: With regards to the invalid input/data, you’ll never catch it all first time, but every time you find another problem, you write another test, that 75% will slowly start creeping down.
Unit testing is for testing that function foo doesn’t return 1 when it gets “bad” input – not for testing that everything is ok…
We are a team of 7, and we are using unit testing/test driven development for the past 6 months, and i should mention that the integration has become extremely easier.
In here we are using:
1. Define the requirements
2. Write the tests for the requirements
3. Write the code that makes the tests pass
4. Re-Factor the code
5. Send the result to a internal, but independent QA department, that will ensure that the requirements were all full filled by the development team
Until now it’s working extremely good, although, makes the process of develop->production more delayed.
As a standalone developer, I find that unit testing tends to stifle development. The main piece of potentially testable library code I maintain only needs to support a handful of applications at this point, and since I’m the only one supporting those applications, I often indulge in making major, radical, breaking changes, sometimes completely changing behaviors as I figure out better ways to do things.
This lets me keep things clean and fresh, but it also makes supporting code, such as tests, impractical to write and keep up to date.
Rather, my quality assurance is based on actually using the code. If it produces correct results in 5 different apps, and continues to do so in production, one can only assume it’s working right. That’s good enough for me.
@Ilia: Yeah, I do the same churning to keep things clean and fresh, and it’s really beneficial. But in my case I do this in my garden compost bin. I suppose the material is the same in both cases!
)
Thanks for all of the feedback, everyone! It’s helpful and much appreciated. Keep it coming. I’ll re-read all of it and formulate some responses sooner rather than later.
@Ilia: I am one of those developers that like to improve old code as much as I like writing new code. For me, Unit Tests provide a solid background to make sure that when I decide to rip the guts out of my application to “try something” I can be sure that my changes still meet the expectations and requirements that my application was designed to fulfill. Unit Tests are my safety net so that if I break something that I didn’t know I could break, the little red bar tells me the real story before I commit – not a trouble ticket from the client after I have pushed to production…
On unit testing: recently, I started writing unit tests for almost all code I write. I used to be the kind of developer who thinks that unit tests are simply some extra code you have to write and it was annoying and like @Ilia said, whenever I made any significant changes to my application, it’d be a pain to keep use unit tests up to date. What’s helped me were two things:
1. Realizing that there is a difference between a unit test and an integration test. Checking whether your e-mail validator class validates correctly is a unit test. Checking whether the registration process on your site works end-to-end, including database interaction – that’s an integration test.
2. Someone once said to me (sorry if I don’t remember who – if you let me know, I’ll credit you properly) that you should write a unit test for any situation where you find yourself adding debugging code in your application. If you do a lot of error_log(‘i am here’) or error_log(“value x is $x”), this is a great candidate for a unit test.
I find that this approach really helps me realize when unit tests are needed and I end up writing a lot of them during my development, not as an afterthought. Being able to re-run these tests later on to see if I haven’t broken anything is kind of a bonus.
A lot of times the “bugs” in our code result not from poor code practices, though, but from poor documentation and communication about what we’re actually building. Having a really good system for tracking what you should be making is crucial and I’m really glad you mentioned use cases in your post.
Personally, I think use cases are great and I love them dearly, but boy do I hate writing them. They are the most boring thing in the universe to write. It’s like pseudo-pseudo-code. And whenever a project does use use cases, it is the developers who end up writing them – I find that it’s really difficult for non-developers to think in terms of edge cases. Also, modifying use cases is even worse, and while I’d love to ever work on a project that has “finalized feature set”, that rarely happens and changes keep sneaking in.
Writing great code is easy when you have all the time in the world and the requirements never change. The challenge lies within realizing that such a situation is kind of ideal and has a very very low chance of happening. It is my belief that requirements will continue changing, no matter what we say so should roll with it. Perhaps having some other way to communicate and track requirements would be better? Some sort of a wiki for discussion and a task tracking system that can plug into it? It would be great to somehow also write automated feature testing (user-level testing, not code-level) that confirms that all features are eventually developed to spec.
@Maggie: Agreed on all points! And yes, I also agree that one doesn’t REALLY see the benefits of extensive unit tests/integration tests until they need to use them during refactoring or other sorts of debugging exercises.
However, yeah…writing use cases can be annoying, but it just depends on the person. I tend to like them because it allows me to do all of the thinking on what I’m building upfront, which I really enjoy. But time is not always a luxury that developers get, of course.
Anyways, my biggest argument for use cases is that, at their foundation, they’re a textual representation of requirements, which everyone can read and understand. They cover the normal success scenario, and all of the edge cases.
Having this level of granularity lends itself to a few niceties: 1) having complete, accurate wireframes that also cover edge cases, 2) informing a QA department in writing test cases, and 3) easily assigning individual units of work out to developers.
But yes, I totally get that one major challenge here is how you deal with constantly changing requirements. That’s a whole ‘nother issue that I have no solution for, but yes, updating them is a bit awkward and kind of a pain.
Anyways, yeah…good stuff!