With its project idea “Itsy Bitsy Technology Crawler,” Schaltzeit GmbH is one of 23 winners of this year’s Grants4Apps, Bayer HealthCare’s ideas competition for innovative software solutions. The prize money is helping us develop a trend crawler that enables an optimized search for developments and trends in databases. Schaltzeit GmbH (SZ) talked about developing the solution with Thomas Stark (TS), whose company cisLogic has long been Schaltzeit’s partner in developing software solutions:
SZ: What can a trend crawler do?
TS: To begin with, you can think of a crawler as a kind of meta search engine. A central search form lets you query several connected database systems at once. Instead of running one query after another in dozens of databases when searching for new developments and trends, the individual queries are processed in parallel in the background, and the results are combined in a list, categorized, and sorted. An application like this saves a lot of time and also helps assess the relevance of the search results. Unlike ordinary meta search engines, our trend crawler is a specialized tool that queries selected databases depending on the field of work and can also legally include password-protected systems. The trend crawler does not build its own index; it acts as a bridge between worlds.
SZ: What is particularly difficult from a developer’s perspective?
TS: Using special adapters, we also connect sites that don’t have a dedicated interface of their own. These search queries are controlled by a robot, mimicking human–browser interaction. For this we use a special phantom browser that has no user interface of its own. The difficulty lies in correctly reading the content returned by the search query, even on poorly programmed sites. In addition, a lot of background knowledge about the content being searched for is needed to steer the sub-queries sensibly. Some of the connected sites don’t provide enough text information about individual hits in their result lists. Here, the crawler has to “crawl” through the listed results manually, open the pages one by one, and read the content there.
SZ: What do you find particularly exciting about it?
TS: From a developer’s perspective, the event-driven multithreading architecture is exciting. While dozens of subprocesses are busy in the background, the individual search queries have to “push” their results to the user’s web browser on their own. What the user sees is a dynamic page that, as if by magic, gradually fills up with content.
SZ: How can users work with the various search results afterward?
TS: The trend crawler gets really interesting when it runs searches on its own and notifies the user when new hits appear in the result lists. Like a radar, the trend crawler reports whenever something happens in the monitored databases for a specific search term. Users can be notified via an alert function.
So how do we differ from other meta search engines?
Unlike meta search engines, which only draw on freely available sources on the internet, we offer searches in password-protected databases as a service. The selection of connected databases and the dedicated configuration of the interfaces are tailored to the client’s requirements. The right selection and configuration determine the quality of the result.
SZ: How will the trend crawler be integrated into the PEACOQ scouting software?
TS: The trend crawler is part of the PEACOQ software and can be integrated there as a module. Results can then be transferred into our database system’s workflow and “refined” into new trend descriptions.
SZ: cisLogic has now been Schaltzeit’s IT partner for more than six years. Which joint projects stood out for you?
TS: Besides the many smaller joint software projects, the development of the PEACOQ software is certainly the main project that cisLogic is carrying out together with Schaltzeit. In developing PEACOQ, the Software Campus, EIT ICT Master School, and Telekom Innovation Contest projects were particularly important, as this is where our collaboration with Schaltzeit really came into its own.
SZ: How is the work usually divided during development?
TS: Schaltzeit handles communication with the client, the definition of requirements, and project management and planning, while cisLogic concentrates on the architecture and technical implementation of the software. Running tests on our test systems and creating design templates also fall within Schaltzeit’s remit.
SZ: What leaps forward do you see for software support in innovation management?
TS: The internet continues to grow at high speed, and the number of relevant specialist information sources and freely available databases is rising rapidly. Monitoring trends via the internet is a challenge that tools like the trend crawler can help master. But that is only where the research work begins. At this point, software support means providing technically supported workflows through which experts can evaluate and select the trends found. In addition, automated analyses and visualizations can help to spot trends and developments earlier and respond to them.
SZ: Thank you very much for the interview. Now get typing! 🙂
