HP Software provides a vast portfolio of solutions for professional IT, starting with capacity and project planning, on through definition and management of test cases, the actual roll-out, until ongoing operations and management of the IT environment. R&D is comprised of more than 3000 software developers located in 20 sites all over the world, with Böblingen being the 5th largest.
Business Service Management (BSM) combines the application-centric end-user view with the service-oriented ITIL processes of the user support helpdesk and the detailed, technical monitoring in the Operations Management of the back-end.
BSM is a product line including more than 20 products (mostly legacy HP OpenView as well as products from Mercury Interactive and other acquisitions) organized in 3 main centers: Business Availability Center, Operations Center and Network Management Center, with about US$ 350M total license revenue in 2008.
The proven and leading Operations Manager (OM) (ex OpenView Operations, ex VantagePoint Operations, ex IT/Operations, ex OperationsCenter) software was being integrated into the HP Business Availability Center application that was acquired through Mercury Interactive in 2006 in order to provide both a business (top-down) and operational (bottom-up) view of the IT environment in one application.
The Operations Manager i (OMi) product is offering a new Operations user interface which is an integral part of the BSM console. In its first release, this new product still depended upon an Operations Manager installation for the deployment of the Agent infrastructure and management policies; alerts, metrics and status messages were forwarded into a new, web-based Event and Health view. The major theme of the second release was added integration capabilities, so that 3rd-party domain managers or other legacy sources could feed management data into the Event subsystem, profiting from the advanced event correlation capabilities that promise increased efficiency of the IT staff. For a future release, it was suggested to add deployment and policy management capabilities to the product to make it a replacement for its mature predecessor, finally achieving the long-desired technology refresh of the OM product. (A previous attempt based on a home-grown Java + native component architecture had made it up to beta status, and was canceled literally at the last minute, citing problems selling it into the main development platform, Linux. After a short one-release intermezzo, the same developers were recruited for the OMi development.)
HP Software BSM Software Development Engineer
Jul 2009 - …
Just as I was taking my parental leave, my team ramped up its engagement for the first release of Operations Manager i, working as technical consultants and facilitators between the local developers and the offshore content development teams. After a year, I returned just in time to see that effort being shut down (a team dedicated to catalyzing development and shielding the dev teams from each other was not deemed necessary any more), and our team in a quest for a new role. The two on-site development teams (with about 30 developers) had struggled with keeping the schedule (the original release had been delayed for many months, partially due to the tight coupling of three major development sites scattered around the globe, but also caused by a crooked combination of attempted Agile development with traditional, bureaucratic management practices, and the inevitable cultural differences), and three months into the new project were again running late. Thus, after some state of limbo, the implementation of the Southbound Interface was moved to our team, handing us a crucial area of functionality that was new and encapsulated enough to allow us a quick start, without having to delve too deep into the existing source code and stepping onto the other teams' areas.
Our team started development with a re-evaluation of the use cases and existing prototype in August, and then came up with a design that fit the preexisting opinions fairly well. Due to the delay caused by the ownership change of that feature, it was initially planned that the Integration Adapter would release late next summer, a couple of months after the BSM 9.0 MR in July; the reasoning being that customers would at first be occupied with migration of the platform, anyway, and would look into the new integration capabilities only after that. Except that… a few months into the implementation, someone in management recognized that the entire 9.0 release was marketed for its integration capabilities, and that meant that we had to accommodate the early MR date and switch gears in the middle of the project. The re-scheduling left us with only two more months until the functional complete milestone, and we had to drastically cut features. Fortunately, or own development proceeded smoothly, and we were able to make the date, whereas the rest of the BSM platform stubbornly ignored the ominous signs of high defect counts and somber feedback, and finally had to go through a debilitating round of last-minute feature cuts, leaving it running only on Windows, and without support for upgrades from previous versions. One could say that in a face-saving move, they got something out on the promised (and publicized) date, but it's far from what was initially envisioned and planned. This was more than underlined by phrases like
The theme for the Service Packs is 'Make the product ready not only for proof-of-concepts' that floated around in emails.
For the next minor version refresh of the Integration Adapter, I extended both Java backend and Flex frontend to implement a policy editing lock, so that multiple users could work with our user interface in parallel. This work had to be integrated with the other parallel development streams that went on in back- and frontend. I also participated in the design of another crucial new feature, the backsync. But I was mostly occupied in the project with the packaging and installer area, where we had to incorporate the requirement that the underlying Agent infrastructure component be transparently included in our product installer.
This area is traditionally shunned by developers, because of the complexity (abstractions around platform-specific install technology to support installation on Windows, Linux and Unixes from a single source definition), and the resulting huge stack of often old and home-grown technologies (Imake-based custom build scripts, an install technology developed in-house). But, because of my long history with the company, this was a good fit for me, and a challenge that I took up with vigor. My company connections and experience allowed me to untangle the ownerships (most of the shared components had been moved to low-cost geographies, sometimes going through many hands), and design a way to incorporate this difficult requirement into the existing architecture. During the course of this, I filed dozens of defects, surfacing problems and inconsistencies, and also documenting the neglect that some internal shared components had suffered from. Even though most of the components never reacted to the issues in a timely way, I managed to implement the requirement in a robust way, and then started tutoring our remote but brilliant QA team on the installer area, passing on a summary of the history and architecture, and putting forward my opinion on what problems are due to historic baggage unlikely to change, and where the testing effort instead should be directed to, because it is under our control to change.
On my initiative, and because of the good working relationship I had established with our build engineering counterpart, I started an unsolicited side project to refactor our central build scripts, because this had a high overlap with the packaging-related tasks I was working on. Whereas the build team's focus is on short-term corrections to keep the build running, I took the long-term view and cleaned up the many glaring duplications and inconsistencies that had crept into our build scripts. This resulted in a LOC reduction from 2000 to 950 in the central build scripts alone. Extracting build fragments into Imakerules that can be re-used across the software organization greatly reduced the complexity of the scattered build scripts, and did away with many inconsistencies that had grown from the prevalent copy-and-paste mentality. (But getting the reusable code accepted by a reluctant and shortsighted organization was a challenge way harder than the technical difficulties!)
I carried on with the build refactorings throughout the eventual (again delayed) release of the Integration Adapter 9.10 in April 2011, as they would be helpful for the planned move of our source code into a dedicated repository, up until June, where I moved from working on the Integration Adapter (which was about to be merged and re-branded with another related BSM component done in Israel). Whereas some of my teammates started building integrations into management solutions like Microsoft System Center Operations Manager (using the Integration Adapter we had created), my assignment was to help the colleagues that had come from OMi into my team (not because it made sense technically, but for span of management control; one manager was promoted up and away, and my boss had the least people, so he benefited from the eventual re-balancing).
The Content Manager covers the definition, import, and export of management packs; foremost for OMi, but subsequently BSM components started writing their own providers and plugged into that extensible component, shedding their legacy configuration mechanisms.
Ingo Karkat; last update 26-Apr-2012