Ideation: What do you do?
Generating ideas is not trivial. There is a huge literature in the philosophy of science dealing with where and how ideas originate. These are not only ideas for science, poetry, literature, and art but also, and more importantly for this book, ideas for new technologies and products. In all cases, idea generation, at worst, is considered mysterious, involving a black box (i.e., the brain) from which something just appears; pure creativity. At best, it is considered to be the result of hard work and deep concentration. In neither case is it considered to involve data or deep data analytics.
Since the topic of idea generation is so broad and diverse, I can only hope to describe a few practical approaches relevant for new product development. These are described in the following section. See Johnson [2011| for a discussion about where ideas come from. Also see Couger  for an extensive discussion of different approaches to creative problem solving (CPS) which is another way to describe idea generation.
There are nine sections in this chapter. Traditional sources for new ideas, such as brainstorming, are discussed in Section 1 but then the sources are extended in the second and third sections to include Big Data and especially text data as a component of Big Data. In Section 3, I review the key points about text analysis in some detail because I believe this is an untapped area for new product ideas. I then narrow the focus for text analysis in Section 4 to call center logs and warranty claims analysis as special cases of text analysis. Sentiment analysis and opinion mining as possible sources of ideas are mentioned in Section 5 but these are not developed in detail since I will discuss them in Chapter 7 when I focus on post-launch tracking. Section 6 contains more traditional market research approaches while Section 7 is concerned with some machine learning methodologies that could be used in the ideation stage. Section 8 is a review of software. Section 9 is a summary. An
Appendix presents some higher-level math concepts mentioned in the chapter and that are referred to in later chapters. It is not necessary to be proficient in these concepts, but if you intend to pursue a more data analytics approach to new product development then these concepts are a must to know. Other chapters will have their own technical appendices with more information needed for a data analytics approach.
Sources for ideas
I categorize idea generation into two broad approaches, traditional and modern, to make a discussion of idea generation more systematic. The traditional approach consists of time-honored ways of finding ideas: brainstorming, market research surveys, and focus groups. The modern approach is more systematic involving data and statistical, econometric, and machine learning methodologies. I will discuss the traditional approaches in the next subsection followed by the modern approach.
There are three (and probably many more) traditional ways to generate ideas: brainstorming, market research surveys, and focus groups. Brainstorming is an old technique that has been in use for 70 years and is applied in many forms depending on the interviewer’s training, expertise, and preferences. It was introduced by Osborn 11953] and subsequently used and abused by many in academics, business, and government. There are two central precepts to the approach:
- 1. idea generation must be separated from idea evaluation; and
- 2. brainstorming in a group context supplements, not replaces, individual ideation.2
Osborn  outlined four guidelines for the effective application of brainstorming:
Brainstorming, despite its wide use, has its detractors and critics who believe it is a waste of time, energy, and money. These are weak criticisms usually proposed by those who do not want to be involved in a brainstorming session; they are basically trite rationalizations because these critics often believe they already “know the answers” or that nothing new will be proposed. Better criticisms have focused on situations with more merit because they get to core psychological reasons participants may not function well in a brainstorming session. Working in groups can be inhibiting and intimidating due to “evaluation apprehension” and “uniformity pressure.” The former is a fear of being evaluated and criticized while the latter is a fear of being pressured to conform with the crowd. These are inherent fears many people share but are not reasons for dismissing the process. Other issues are “social loafing” and an associated “free-rider effect.” The former occurs when people let others in a group lead the way while the latter occurs when people gain some benefit from a group activity while expending little effort. The free-rider problem is an issue in economics and political science regarding the provisioning of public goods. Basically, a free-rider problem exists when someone benefits from an action or situation without cost. In the case of brainstorming, a team member could simply not participate but yet receive accolades and rewards as a team member if a good idea emerges. If a bad or no idea emerges, that person could simply say “I had no part in this.” See Hardin  for a good discussion of the free-rider problem in general. Other reasons include the “sucker effect”, the “matching effect” and the “blocking effect.” See Couger  for discussions. These criticisms obviously go against two of Osborne’s guidelines.
A legitimate issue with brainstorming is a quantity-quality tradeoff. The objective of brainstorming is to generate ideas - lots of them. This is Osborn’s third guideline. During a typical session, dozens, if not hundreds, of ideas could conceivably be listed. This is a quantity problem. Someone, or preferably the team, must wade through the list which would definitely be time consuming and thus costly, not to mention fatiguing and discouraging. Buried inside this list could (hopefully) be the gem of an idea that leads to a new product, but finding it may be as daunting as initially compiling the list. This is a quality problem. Further efforts may be needed to cull that single idea or basis for an idea from the list which means other sessions or teams have to be engaged to review and narrow the list.
See Isaksen , Isaksen and Gaulin , and Couger  for reviews of the brainstorming technique and critical appraisals. There are many other techniques that can be used to generate ideas, both individually and in groups. Couger  lists 22 methods, one of which is brainstorming.
Market research methods consist of quantitative research and qualitative research, or a combination of both in a two-phase approach: qualitative research done first to identify key issues and problems, followed by quantitative research building on and supported by the qualitative research results. Sometimes the quantitative research is followed by another round of qualitative work to verify or further clarify quantitative conclusions.
Included in the qualitative research are in-depth interviews with Subject Matter Experts (SMEs), Key Opinion Leaders (KOLs), major customers, and company executives. The quantitative research includes extensive questioning of customers using survey research techniques. In either case, the objective is to “hear the voice of the customer” although with SMEs and KOLs the voice is indirect since you do not speak to the customers per se. In-depth interviews with major customers, of course, give you access to that voice, but at the same time the voice might be muddled. In a business-to-business context (i.e., B2B), for example, there are two levels of customers. One is the Key Decision Makers (KDM) who authorizes buying the product or service for use in his/her company. This person may not be the actual user of the product, the second level of customer. Users may have opinions which, perhaps because of organizational issues or restrictions, were never conveyed or expressed to the KDMs. Consequently, the KDMs interviewed may not be able to fully express or articulate problems and requirements. The end- users may be buried deep inside the organization and they may be so dispersed throughout the organization or so anonymous that finding a representative sample for interviewing may be more than a challenge. The in-depth interviews are then biased.
A modern approach
A modern approach to idea generation heavily relies on a combination of data and statistical/econometric methods to identify customer - business-to-business (B2B and business-to-customer B2C) - needs, problems, issues, and concerns. These may be known to the customers so they can directly express them, or they may be unaware of them and so they can only express them indirectly. Also, they may not be able to fully and adequately articulate their needs, problems, issues, and concerns because of language issues or a belief that whatever they say or write is “clear enough” even though it is not. This last point introduces what Liu and Lu  refer to as noise and misleading information. In general, methods are still needed to extract the essence of what customers are trying to say that could lead to new product ideas.
Two items are needed for this essence extraction: data and analytical approaches. There are three data sources:
- 1. Big Data - External
- • Social media and product reviews.
- 2. Big Data - Internal
- • Service calls
- • Warranty claims
- 3. Market Research - Voice of the Customer (КОС)
- • Surveys
- • Ethnographic research
The analytical methods applied to the data include the trio of statistics, econometrics, and machine learning, each consisting of a wide array of tools. Within this trio, the actual tools range from the simple to the complex. For example, with the statistical methodologies, means and standard errors are simple while logistic regression is complex. Depending on the data, its magnitude and structural complexity, some methodologies and tools are more appropriate than others.
I will discuss these data sources and the analytical approaches for each in the following sections.
-  No criticism of other participants’ ideas. Judgement is reserved for the endof a brainstorming session. 2. Free thinking is encouraged. The objective is to generate ideas, so anythingand everything is on the table. 3. Generate quantity. The more ideas listed, the better the odds of finding awinning one. 4. Combine ideas. Part of the creative process is combination and connectivity.Ideas can be formed by combining several already stated ideas as well as byfinding a connection between two or more ideas. One idea in a vacuum fromothers may be insufficient but several in a combination may be a gem.