Discussions

News: Article: An Architect's Perspective on Application Quality

  1. According to Allen Stoker, problems arise when we continue to build applications with increasing levels of complexity and don't effectively plan to manage that complexity. In this article, Stoker discusses implementation and testing for two quality-related goals: - Unit testing is an early stop point to fix problems before they impact a broader audience, but typically have little impact on the end users of a system. - Real-Time monitoring is a critical aspect of ensuring long-term quality. Most applications are deployed to perform work that cannot be 'simulated.' This confers a requirement that live transactions be monitored and reported on. There are tools that will provide runtime monitoring, but they may require extensive knowledge and setup efforts, and may require application hooks that need to be considered from the beginning of the project.
    Quality begins in the team - not the application. Proper planning, communication and processes are essential to any successful project. Projects that lack these fundamentals will likely produce problematic applications. I'm a firm believer that large teams with diverse skill sets need a Quality Architect - a highly skilled technical person on your team who has no assignment but to support or 'enable' the other team resources. Such a resource can mean the difference between project success and failure.
    Read An Architect's Perspective on Application Quality. What do you think of this article? How do you strive for quality in your applications?

    Threaded Messages (26)

  2. Maybe a link?[ Go to top ]

    This sounds like a reasonably interesting premise for an article. Maybe a link to the article would be helpful?
  3. Is there an article link missing? Anyways, I have seen a number of beautifully architected projects which are a total mess down in the trenches. I think quality should come both bottom up and top down. It also depends on what quality we are referring to. Having few bugs is a measure of good quality, for the end user, while having clean code would be a good measure for the developer.
  4. Sounds good[ Go to top ]

    When I read through this it makes sense, and I know it's good practice but in my projects sometimes the pressure of delivery pushes them aside.
  5. The quality measures indicated in the article are all outward-facing from the project team (e.g. Reduce the number of defects found in functional testing, Shorten the duration of the QA testing cycle). Quality control starts much much earlier. What about measures such as: * Developer turnaround (time between a developer making a change and being able to test that change locally) * Test coverage * Maintainability of code * a culture of excellence within the development team All of these points contribute significantly to the quality perceived by the environment of the project. It is way too narrow a view to think that external measures will necessarily improve quality. Quality is embedded within the project team and the processes the team follows. Yes, these measures are valid and can provide very useful feedback to the project team, but they do not *control* quality. Max
  6. Thanks doc[ Go to top ]

    But symptomatic analysis wont make anything better. the "appropriate goals" arent to reduce the junk. The appropriate goals are to not code junk in the first place. But that was a good freshman year writ on misplacing problem analytics with the actual goals which a professional engineer would solve. Good JOB!
  7. Finally an article that touches on the approach to system testing and execution behavior monitoring that has largely driven the design of JDBInsight and JXInsight - execution path runtime analysis in production. Being so close to the problem (and hopefully solution) for a number of years is that it is hard to explain to others without getting very excited and frustrated at the same time with the lack of system wide testing in enterprise Java deployments. I am excluding stress/load testing here. In the operations world we see a rapid growth in the adoption of ITIL best practices across many companies in diverse markets. An important management domain in ITIL is "Change and Configuration Management". In this area we have seen the large management software vendors IBM, HP, BMC and CA rushing to deliver a viable CMDB solution that actively supports "Incident" and "Problem" management. One area of that has been sadly neglected in all solutions to date has been in the modeling of application execution patterns as "ConfigItem" themselves. My view is that a contextual (data, parameters) execution path that involves interaction with multiple components and technologies should be modeled and captured and that change management should focus on the degree of change (new execution paths) as a means to aid operations staff in providing effective "Incident" and "Problem" management. Sadly the current approach in most Java enterprise environments is to focus on "Incident" management in a rather brutal manner - RESTART THE SERVER EVERY DAY OR WHEN A ERROR IS REPORTED. There is hardly any effort made to capture runtime state (component and service object state) as well as recent execution patterns (traces, transactions, request, invocations). No one can truly be certain why the failure happened and blaming a recent change in the software and platform is an easy option but probably misguided in that change in one component can indirectly affect other collocated components via shared resource usage (threads, memory, cpu...). Unit testing and code coverage suffer from one big issue and that is the artificially limit the context of the execution. Having complete (100%) code coverage does not imply the software has been executed all the possible execution paths that a JVM runtime will execute when in production when operating with real production data, workload patterns, external integrations, resource consumption patterns... We seem to forget the meaning of "Unit" and attribute a high level of confidence in the quality of the software just by passing unit tests. regards, William Louth JXInsight Product Architect CTO, JInspired "JEE tuning, testing, tracing, and monitoring with JXInsight" http://www.jinspired.com
  8. First of all, You're discussion has nothing to do with Java or server side software. But Im glad your posting on TSS so we all know how paranoid software testers are and how utterly irrelevant this testing methodology is. Second, your psuedo technical assessment of software "robustness" is so off that Im going to have to go and watch every batman video on youtube.com to forget this ever happened. Youtube may go down just because I searched on batman. How would you test that?
  9. BC, I was hoping someone would bite. I think you fail to understand that context is extremely important in "application" testing especially when one looks around and notices that more and more application run times are being driven by meta data (NOT CODE!!) and contextual data (SESSIONS, WORKFLOW STATES). Simply testing a piece of code with "artificial" values ignores that most systems and components have runtime state that can alter the code branches executed. Of course I understand your fear which manifests in such an naive attack (as well as the want to run to the cape crusader) because accepting this means that the complexity of testing grows tremendously and the confidence in our tests today reduces significantly. I think you maybe simplifying the problem just to suit your approach to testing whereas I might complicating the problem just to suit the solution I have created. I let others be the judge but let me try once more with an example. I was recently on site with a customer that was having performance and stability problems in production with a commercial portal solution built on top of the vendors J2EE container. Everything worked fine in pre-production when the product was tested using load test scripts but when it went live within days it was constant fire fighter. What had happened? Many, many things including: 1. Portal administrators making changes to portal themes and layouts resulting in memory capacity issues, peformance degradation due to addition (and excessive) data access traffic, and shared portlet session data access issues. 2. Users performaning "unusual" (in terms of tested scripts) query formations (parameters, operands, intervals) and workflow patterns (switching between bussiness transactional operations). 3. Untested (and un-monitored) user customizations such as business rule definitions, triggers, view definitions.... Combining this with other differences between the pre-production and production environment and it was always going to be a recipe for disaster. BC, I am not a software tester and I am not paranoid. Can you leave the comics aside for now and try to contribute to the thread in a constructive manner. I feel strongly about runtime execution behavior analysis but I am always open to different views that show errors in my own analysis. By the way I did not say that all values must be tested but that we must understand that some parameters do drive the application runtime and that this values are very much part of the software code itself. Kind regards, William Louth CTO, JInspired "JEE tuning, testing, tracing, and monitoring with JXInsight" http://www.jinspired.com
  10. Follow-up: Up to lately most profilers when recording performance measurements for enterprise Java applications only reported on the JDBC API. So when someone looked a profile snapshot the could see performance issues with PreparedStatement.execute() and ResultSet.next() but the context was missing. The context that would be important at least in terms of performance management and error diagnostics would have included 1. The transaction history up to the point of the slowdown and error 2. The transaction history for all concurrent transactions (think: deadlocks, transaction integrity/isolation). 3. The SQL statement (a parameter to the method but very much part of the execution runtime). Also derived from this are database schemas, tables and columns. 4. The parameters passed into the statement 5. The parameters higher up in the context trace stack the had a bearing on the eventual SQL statement created 6. Resource metrics (current JVM memory and process CPU times) 7. The volume of data being retrieved because of the statement and parameters. If that does not convince then have a look at the HP OpenView Service 4.5 product. The call stack at any moment in time is nearly identical for every single transaction because the servers execution behavior is driven by the request data (commands are embedded within RPC requests) and even when being processed in the server the workflow is largely meta driven. There is hardly any real code specific to the workflow - the workflow is driven by current runtime state, database state, command state as well as session parameters. It would be extremely easy to have 100% code coverage but for such a generic application runtime but would the quality of the product be assured - NO. Even worse for the R&D and testing teams is the fact the customers have a high degree of customization which greatly impacts the performance test and capacity planning for such an application - this is the case today and why HP OpenView and its customers use JXInsight for performance management as well as post change management analysis. Regards, William Louth CTO, JInspired "JEE tuning, testing, tracing, and monitoring with JXInsight" http://www.jinspired.com
  11. Actually I never attacked at all. joking you and attacking you are very different. And for all your verbosity I have actually done more testing of application load and optimization than you could ever describe with your little user scenarios and how SUPRISING that was to you. Since I havent developed an app in the last ten years that had less than 10K users and had up to 1 million I really dont need a lesson in testing from you when I have that many users and in 5 years Ive had about bug reports < 35 total over the same number of years with total system availability. Play your little excel spreadsheet testing with your pals.
  12. And for all your verbosity .... Play your little excel spreadsheet testing with your pals.
    +1 (vewy funny indeed) You Wascal you!
  13. Hi BC, The only aspect I see consistent in your posting on this website is the mannerism of your replies. I cannot really comment much on the content because after reading and re-reading your posts I find it hard to see any valid arguments other than extraordinary boosts that would shadow any of the super heroes you routinely watch while all your applications exhibit "total system availability". Now that Mr Fuud has come to your rescue I can ignore all hidden arguments you might have had - a kindred spirit? I do see that you understand execution context and its impact on testing though for the life of me I still have not got a firm understanding of what you are trying to say. http://www.theserverside.com/news/thread.tss?thread_id=41109#212960 A reply with real-world insights into effective testing practices would help us get back on track. Regards, William Louth JXInsight Product Architect CTO, JInspired "JEE tuning, testing, tracing, and monitoring with JXInsight" http://www.jinspired.com
  14. BC, I had a look at some other posted "Advanced Java" where you mention the impracticalities of application isolation in production environments. So I do not understand why we have a problem with my postings where I try to place more focus on real-time monitoring of applications in production to capture different behavioral patterns that cannot easily be reproduced in unit or system testing. My argument is that testing does not stop once an application is deployed. Operations need to be able to continually monitor the availability and in the event of issues capture significant diagnostic "contextual" information for post processing by architects and testers. Where is the spreadsheet in my proposition? JXInsight is a contextual runtime analysis solution that traces patterns: resource transactions, metric trends, system/component state, inter component communication.....How would you get this in a excel spreadsheet? Some background to my postings: http://blog.jinspired.com/ http://www.jinspired.com/products/jxinsight/insights.html William Regards, William L
  15. Louth (is it pronounced like lout)? You forgot part of your signature on that last post...so here it is. You wascal you!
    William Louth JXInsight Product Architect CTO, JInspired "JEE tuning, testing, tracing, and monitoring with JXInsight" http://www.jinspired.com
  16. Huh ?[ Go to top ]

    I am glad to see a meaningful and realistic piece on TSS for the first time in a long time. The writer is clearly talking about a 'first steps' approach and is clearly writing for an audience that does not yet believe in the value of these first steps. That's not some of us here, but IMO criticism should keep that in mind. Yes, there is a lot more to talk about and do. The article does present good ideas, however. It's simply too short to address things like developing a culture of excellence. In my personal view, that's what is really most desirable. Let the individuals select their own tools and policies once they share that culture; you'll be fine. One of the hard parts, in my experience, is defining quality in that culture in such a way that it makes sense to the business (as opposed to just make defect-free software, which is largely irrelevant). Anyway I wanted to say "good show" to the writer ! Maybe you can talk in more detail later. ... and then there are the sales trolls :-)
    Finally an article that touches on the approach to system testing and execution behavior monitoring that has largely driven the design of JDBInsight and JXInsight - execution path runtime analysis in production.
    The article does not mention 'execution path runtime analysis' at all. Tools like this work on problems at the most expensive point of the cycle. They may or may not be valuable, but tools that help at requirements and design time would be more valuable. As for BC, whoever you are ... stop the BS. The fact that you worked with some project with lots of users does not mean you know anything about QA or testing software - in fact, in my experience, lots of people build lots of systems with poor QA and testing methodologies.
  17. Re: Huh ?[ Go to top ]

    Hi Ethan, I am not "sales troll" as being overly passionate about a product or engineering domain does not make you a great sales person - it might help make a great product designer but great product companies do not necessarily make successful companies. I worked previously for Inprise/Borland so I am painfully aware of this when I look around at the current market leaders in the middle-ware space. In our product as well as in the newer entries into the APM & APTM space a execution path equates to a "business transaction pattern" we also have the concept of a "resource transaction path" which is the execution path in terms of database access. When someone says transaction I immediately think of an execution path because transactions cannot always being distinguished by the entry point but by the actual sequence of steps. The ECPerf benchmark has common transaction entry points but the actual transaction steps are different because the benchmark driver drives the execution by parameters as well. Those that have worked with me at Borland and HP know from the early days that I have always being more interested in traffic analysis than unit testing because I recognized very quickly that is was incredibly hard to create comprehensive test suites for complex distributed systems with diverse deployment topologies and interaction styles and that usage patterns and execution behavior did not always correlate to the expected execution pattern in our unit tests. For a long time I was sent on site to resolve problems that could not be reproduced in the test labs. When I decided that something had to be done to speed up the problem analysis and resolution I designed a prototype product that was called Borland AppSimulator. The prototype focused on how to model distributed execution patterns and performance metrics under different configuration of the Borland AppServer so that we could assess whether load balancing was operating efficiently and that degree of fault tolerance (reliability) was visible to architects and operations. Unfortunately at that time in Borland everything had to be called JBuilder to get funding. My vision was viewed as too "out in space". I left. One thing I did learn at Borland was that you should always try to use your own creation (eat your own dog food). JXInsight's performance management console has built in tracing to collect execution patterns for the console when analysing customer snapshot models. The console profiles itself in real-time and you can connect another management to a console to analyse the behavior of the console. When errors occur diagnostic snapshots are generated by the console itself. We created a special Swing trace extension to help collect the paths (transactions) especially as communication is across disconnected threads. http://www.jinspired.com/products/jxinsight/swingtracing.html I hope you now see that I practice what I preach. I would be posting on this thread even if I did not have a commercial product. We need to show some respect for individuals within the profession that have commercial and non-commercial labors of love that look to address issues facing the software industry across all application life cycle phases. Kind regards, William
  18. Re: Huh ?[ Go to top ]

    One of the hard parts, in my experience, is defining quality in that culture in such a way that it makes sense to the business
    I find this problem also. In many cases, they can't or won't understand why some things in IT take so long or even why they're done at all. I've read a few strategies on how to address this, but it's not an easy task -- especially given how complex systems are today. At the end of the day, businesses seem to want quality on their own terms, without having to pay or staff for it. At least that's my experience.
  19. Re: Huh ?[ Go to top ]

    One of the hard parts, in my experience, is defining quality in that culture in such a way that it makes sense to the business


    I find this problem also. In many cases, they can't or won't understand why some things in IT take so long or even why they're done at all. I've read a few strategies on how to address this, but it's not an easy task -- especially given how complex systems are today.

    At the end of the day, businesses seem to want quality on their own terms, without having to pay or staff for it. At least that's my experience.
    Well, I agree in part - that's half of what I meant. Certainly the practice of talking about software quality without wanting to pay for software quality is a common problem. The other half of what I meant is less comfortable for me in my IT role. It's this : businesses actually have the right to define quality on their own terms. That doesn't mean, obviously, that business management should pretend that we can attain a certain level of quality in software without an appropriate level of expenditure in time and money. What I was trying to get at is that business management sometimes - perhaps most of the time - means something different from what software developers mean by 'quality'. For instance, in a product management role, I don't usually care about defect density or path coverage. I care about working features. I don't necessarily care, even, whether all features work exactly as discussed, if there is a workaround that gets the customer what he wants. My bottom line question for quality is if it is 'good enough'. I have this view partly because I (think I) know the customer and partly because my view is that we need to weigh the benefit of each improvement in quality against the value of getting something in the customer's hands before the competitor does. I know that the customer will likely stay with us once he starts working with our stuff, even if our stuff isn't 100% there. So in some ways I as the 'business' end will be satisfied with lower quality than I will be satisified with from the 'developer' end. (The '' is there because, of course, the division into categories is spurious - we are every one of us the business end. But you know what I mean, although that, in itself, is a problem.) And my point was that the 'business' has every right to act this way - it's not unethical in any way IMO. What matters is the cusomer satisfaction. For example, I'm working on a product right now that my client ships knowing that some features have not worked for at least two years. Nobody cares - at least, nobody cares enough not to buy. The things that do work are worth the cost to the customer. As a developer, this bugs me. But it doesn't bug the customer ! So software quality should be defined in terms of the way 'the business' (again this artifical separation!) views quality, not the way I as a developer see quality ... in terms of good enough feature set and time to market rather than (for instance) in terms of path coverage or defect per SLOC. BTW I didn't mean to insult the other guy, unnecessarily at least, with my sales troll comment. Your sales function is worthy and legitimate. It's just that you came on line talking about your product, which had at best a tangential relationship with the original article and in some ways represents the last way to go about quality assurance if not testing. I viewed your post as an advertisement; sorry if I was wrong. I'm not sure if I am anonymous either ;-0 -don't intend to be at any rate.
  20. Re: Huh ?[ Go to top ]

    Hi Ethan, It is exactly for the reasons that you mention that there is a need for tooling that tackles testing (error diagnostics, scalability, performance, reliability) of production deployed applications. I think one of the reasons that business management "appear" to make "undesirable" trade-offs in software quality is that in many cases developers focus on the wrong areas for quality improvements - wrong because at early stages they have insufficient information. This is very similar to what happens with performance tuning where developers spend time tuning areas that in the large scheme of things do not make that much improvement. I have even falling for this mistake a few times in the past. The problem with all of this is that when things do go wrong in production it is operations management that will start screaming for immediate resolutions - talking about previous trade offs in software quality offers little respite. No one in business management will want to discuss previous decisions as users become more irritated and vocal in their views of the software's quality. It is important to note that users have a natural tendency to remember the more negative aspects of a software product - after a few server downtimes the positive aspects of the software are all but forgotten. Lets not forget that the leading application server vendors continue to release hot fixes, patches and service packs on a routine basis. There will always be areas of disconnect between design and development (implementation). We need (different) tools that help assess the architectural qualities at each stage in the application life cycle. Kind regards, William Louth JXInsight Product Architect CTO, JInspired "JEE tuning, testing, tracing, and monitoring with JXInsight" http://www.jinspired.com
  21. Timely editorial in the latest Java Performance Tuning Newsletter. http://www.javaperformancetuning.com/news/news068.shtml "Performance of most complex systems is critically constrained by the flow of data. How it flows, where it flows, how much of it flows to particular places, and what delays it from getting to where it needs to be next....." Regards, William Louth JXInsight Product Architect CTO, JInspired "JEE tuning, testing, tracing, and monitoring with JXInsight" http://www.jinspired.com
  22. William Louth, You do have some good ideas but unfortunately you are far too verbose. If it hadn't been for the funny debate between you and BC, I wouldn't even have had the patience to read so far. You may want to explore one of the following: 1. Start being terse and to the point. 2. Explore a career in sales and marketing (& believe me, I don't mean this in a derogatory way but honestly think with your capacity to make a 5 line tech talk into a 2 hour speech, you would give a lot of folks run for their money) 3. Avoid the use of the signature in every single post. We all know you are a CTO - calm down now, take it easy. Thanks, shanky
  23. Ethan has coded up the correct! In the end, its the business motivation that drives quality (read: "bottom line"). Any approach or technology that can help by automating or providing more complete coverage/analysis anywhere along the QA/Testing path or in the production environment - without adding too much overhead/cost - will be investment worth considering. But to be sure - Quality will only be pursued in as much as it is affecting the bottom line. Jamie Wetzel ***************************** OPNET Technologies (http://www.opnet.com/products/panorama/) APM Solutions Group - Testing Solutions 760.510.1787 jwetzel at opnet dot com *****************************
  24. a few questions?[ Go to top ]

    a few questions: 1. Suggestions are that all production systems be monitored at runtime and all parameters, context data, flow sequence, etc.....should be recorded and analyzed to enhance software quality. Good idea but what kind of performance bottlenecks do you anticipate due to this level of detailed monitoring? What might be the possible non-intrusive ways of acheiving this? Would such level of detailed monitoring work well in asynchronous, distributed, multiple technology and loosely coupled scenario? 2. Perhaps good code quality and not unit testing alone may be one way to enhance software quality. Example - thorough unit testing will not catch the performance problems introduced by boxing/autobxing associated with getting large amounts of integer data from a DB in an ArrayList as opposed an Array for manipulation. 3. Most critical systems go through a parallel run with real data and real users. In my opinion many problems are identified and fixed at this stage - call it the beta release phase. Isn't this a good idea? Thanks, Shashank
  25. First rule of the Internet[ Go to top ]

    William, don't reply to anonymous people. You will just look like a fool while you attempt to be professional and they act like an arse. They are mosquitos, just ignore them.
  26. Re: First rule of the Internet[ Go to top ]

    Hi Mike, if I had seen your post before typing my last lengthy reply maybe I would refrained after realizing that my efforts where futile and that these individual would never understand even if they wished it. I am reminded of Stanislaw Lem's "The Electronic Bard". I understand the rule but acceptance is still hard. Regards, William
  27. Re: First rule of the Internet[ Go to top ]

    William, don't reply to anonymous people. You will just look like a fool while you attempt to be professional and they act like an arse. They are mosquitos, just ignore them.
    Right on!