Building an Enterprise Chatbot (eBook)

Work with Protected Enterprise Data Using Open Source Frameworks

Abhishek Singh, Karthik Ramasubramanian, Shrey Shivam (Autoren)

eBook Download: PDF

2019 | 1st ed.
XXII, 385 Seiten
Apress (Verlag)
978-1-4842-5034-1 (ISBN)

Infos

Explore the adoption of chatbots in business by focusing on the design, deployment, and continuous improvement of chatbots in a business, with a single use-case from the banking and insurance sector. This book starts by identifying the business processes in the banking and insurance industry. This involves data collection from sources such as conversations from customer service centers, online chats, emails, and other NLP sources. You'll then design the solution architecture of the chatbot. Once the architecture is framed, the author goes on to explain natural language understanding (NLU), natural language processing (NLP), and natural language generation (NLG) with examples. 

In the next sections, you'll design and implement the backend framework of a typical chatbot from scratch. You will also explore some popular open-source chatbot frameworks such as Dialogflow and LUIS. The authors then explain how you can integrate various third-party services and enterprise databases with the custom chatbot framework. In the final section, you'll discuss how to deploy the custom chatbot framework on the AWS cloud.

By the end of Building an Enterprise Chatbot, you will be able to design and develop an enterprise-ready conversational chatbot using an open source development platform to serve the end user.

What You Will Learn

• Identify business processes where chatbots could be used
• Focus on building a chatbot for one industry and one use-case rather than building a ubiquitous and generic chatbot 
• Design the solution architecture for a chatbot
• Integrate chatbots with internal data sources using APIs
• Discover the differences between natural language understanding (NLU), natural language processing (NLP), and natural language generation (NLG) 
• Work with deployment and continuous improvement through representational learning

Who This Book Is For

Data scientists and enterprise architects who are currently looking to deploy chatbot solutions to their business.

Abhishek Singh is on a mission to profess the de facto language of this millennium, the numbers. He is on a journey to bring machines closer to humans, for a better and more beautiful world by generating opportunities with artificial intelligence and machine learning. He leads a team of data science professionals solving pressing problems in food security, cyber security, natural disasters, healthcare, and many more areas, all with the help of data and technology. Abhishek is in the process of bringing smart IoT devices to smaller cities in India so that people can leverage technology for the betterment of life.

He has worked with colleagues from many parts of the United States, Europe, and Asia, and strives to work with more people from various backgrounds. In 7 years at big corporations, he has stress-tested the assets of U.S. banks at Deloitte, solved insurance pricing models at Prudential, and made telecom experiences easier for customers at Celcom, and core SaaS Data products at Probyto. He is now creating data science opportunities with his team of young minds.

He actively participates in analytics-related thought leadership, authoring, public speaking, meetups, and training in data science. He is a staunch supporter of responsible use of AI to remove biases and fair use of AI for a better society.

Abhishek completed his MBA from IIM Bangalore, a B.Tech. In Mathematics and Computing from IITGuwahati, and a PG Diploma in Cyber Law from NALSAR University, Hyderabad.

Karthik Ramasubramanian has over seven years of practice and leading Data Science and Business Analytics in Retail, FMCG, E-Commerce, Information Technology for a multi-national and two unicorn startups. A researcher and problem solver with a diverse set of experience in the data science lifecycle, starting from a data problem discovery to creating a data science prototype/product.

On the descriptive side of data science, designed, developed and spearheaded many A/B experiment frameworks for improving product features, conceptualized funnel analysis for understanding user interactions and identifying the friction points within a product, designing statistically robust metrics and visual dashboards. On the predictive side, developed intelligent chatbots which understand human-like interactions, customer segmentation models, recommendation systems, identifying medical specialization from a patient query for telemedicine, and many more.

He actively participates in analytics related thought leadership, authoring blogs & books, public speaking, meet-ups, and training & mentoring for Data Science.

 Karthik completed his M.Sc. in Theoretical Computer Science at PSG College of Technology, India, where he pioneered the application of machine learning, data mining, and fuzzy logic in his research work on the computer and network security.

Shrey Shivam extensive experience in leading the design, development, and delivery of solutions in the field of data engineering, stream analytics, machine learning, graph databases, and natural language processing. In his seven years of experience, he has worked with various conglomerates, startups, and big corporations and has gained relevant exposure to digital media, e-commerce, investment banking, insurance, and a suite of transaction-led marketplaces across music, food, lifestyle, news, legal and travel.

 

He is a keen learner and is actively engaged in designing the next generation of systems powered by artificial intelligence-based analytical and predictive models. He has taken up various roles in product management, data analytics, digital growth, system architecture, and full stack engineering. In the era of rapid acceptance and adoption of new and emerging technologies, he believes in strong technical fundamentals and advocates continuous improvement through self-learning.

 

Shrey is currently leading a team of machine learning & big data engineers across the US, Europe, and India to build robust and scalable big data pipelines to implement various statistical and predictive models. Shrey has completed his BTech in Information Technology from Cochin University of Science Technology, India.

Inhalt

Table of Contents 5
About the Authors 12
About the Technical Reviewer 15
Acknowledgments 16
Introduction 18
Chapter 1: Processes in the Banking and Insurance Industries 20
Banking and Insurance Industries 20
A Customer-Centric Approach in Financial Services 25
Benefits from Chatbots for a Business 28
Chatbots in the Insurance Industry 29
Automated Underwriting 31
Instant Quotations 32
AI-Based Personalized Experience 32
Simplification of the Insurance Buying Process 32
Registering a Claim 32
Finding an Advisor 32
Answering General Queries 33
Policy Status 33
Instant Notifications 33
New Policy or Plan Suggestions 33
Conversational Chatbot Landscape 33
Summary 36
Chapter 2: Identifying the Sources of Data 38
Chatbot Conversations 38
General Conversations 39
Specific Conversations 39
Training Chatbots for Conversations 40
Self-Generated Data 41
Customer Interactions 42
Phone 43
Emails 43
Chat 43
Social Media 43
Customer Self-Service 44
Mobile 44
Customer Service Experts 44
Open Source Data 45
Crowdsourcing 45
Personal Data in Chatbots 46
Introduction to the General Data Protection Regulation (GDPR) 48
Data Protected Under the GDPR 48
Data Protection Stakeholders 49
Customer Rights Under the GDPR 49
Chatbot Compliance to GDPR 51
Summary 52
Chapter 3: Chatbot Development Essentials 53
Customer Service-Centric Chatbots 53
Business Context 54
Policy Compliance 56
Security, Authentication, and Authorization 57
Accuracy of User Input Translation to Systems 59
Chatbot Development Approaches 60
Rules-Based Approach 61
Advantages of the Menu-Based Approach 62
Disadvantages of the Menu-Based Approach 63
AI-Based Approach 63
Advantages of the AI-Based Approach 64
Disadvantages of the AI-Based Approach 65
Conversational Flow 65
Key Terms in Chatbots 67
Utterance 67
Intent 68
Entity 68
Channel 69
Human Takeover 69
Use Case: 24x7 Insurance Agent 70
Summary 71
Chapter 4: Building a Chatbot Solution 72
Business Considerations 72
Chatbots vs. Apps 73
Growth of Messenger Applications 74
Direct Contact vs. Chat 74
Business Benefits of Chatbots 75
Cost Savings 75
Customer Experience 76
Success Metrics 77
Customer Satisfaction Index 77
Completion Rate 77
Bounce Rate 78
Managing Risks in Chatbots Service 78
Third-Party Channels 78
Impersonation 79
Personal Information 79
Confirmation Check 80
Generic Solution Architecture for Private Chatbots 80
Workflow Description 81
Key Features 84
Technology Stack 85
Maintenance 85
Summary 86
Chapter 5: Natural Language Processing, Understanding, and Generation 87
Chatbot Architecture 89
Popular Open Source NLP and NLU Tools 92
NLTK 93
spaCy 93
CoreNLP 95
gensim 96
TextBlob 97
fastText 98
Natural Language Processing 98
Processing Textual Data 99
Reading the CSV File 99
Sampling 100
Tokenization Using NLTK 101
Word Search Using Regex 101
Word Search Using the Exact Word 102
NLTK 103
Normalization Using NLTK 103
Noun Phrase Chunking Using Regular Expressions 104
Named Entity Recognition 108
spaCy 110
POS Tagging 110
Dependency Parsing 112
Dependency Tree 113
Chunking 115
Named Entity Recognition 116
Pattern-Based Search 118
Searching for Entity 120
Training a Custom NLP Model 120
CoreNLP 122
Tokenizing 123
Part-of-Speech Tagging 123
Named Entity Recognition 124
Constituency Parsing 124
Dependency Parsing 126
TextBlob 126
POS Tags and Noun Phrase 127
Spelling Correction 127
Machine Translation 128
Multilingual Text Processing 129
TextBlob for Translation 129
POS and Dependency Relations 129
Named Entity Recognition 131
Noun Phrases 132
Natural Language Understanding 132
Sentiment Analysis 133
Polarity 133
Subjectivity 134
Language Models 134
Word2Vec 135
Neural Network Architecture 137
Using the Word2Vec Pretrained Model 138
Performing Out-of-the-Box Tasks Using a Pretrained Model 142
Word Pair Similarity 144
Sentence Similarity 144
Arithmetic Operations 145
Odd Word Out 146
fastText Word Representation Model 147
Information Extraction Using OpenIE 149
Topic Modeling Using Latent Dirichlet Allocation 152
Collection of Documents 152
Loading Libraries and Defining Stopwords 153
Removing Common Words and Tokenizing 153
Removing Words That Appear Infrequently 153
Saving the Training Data as a Dictionary 154
Generating the Bag of Words 155
Training the Model Using LDA 155
Natural Language Generation 157
Markov Chain-Based Headline Generator 158
Loading the Library 159
Loading the File and Printing the Headlines 159
Building a Text Model Using Markovify 160
Generating Random Headlines 160
SimpleNLG 161
Loading the Library 162
Tense 162
Negation 163
Interrogative 163
Complements 164
Modifiers 164
Prepositional Phrases 165
Coordinated Clauses 165
Subordinate Clauses 166
Main Method 167
Printing the Output 167
Deep Learning Model for Text Generation 168
Loading the Library 171
Defining the Training Data 171
Data Preparation 172
Creating an RNN Architecture Using a LSTM Network 178
Defining the Generate Text Method 180
Training the RNN Model 181
Generating Text 184
Applications 184
Topic Modeling Using spaCy, NLTK, and gensim Libraries 185
Tokenizing and Cleaning the Text 186
Lemmatization 187
Preprocessing the Text Method for LDA 187
Reading the Training Data 188
Bag of Words 189
Training and Saving the Model 189
Predictions 190
Gender Identification 191
Loading the NLTK Library and Downloading the Names Corpus 191
Loading the Male and Female Names 192
Common Names 192
Extract Features 193
Randomly Splitting into Train and Test 193
Training the Model 194
Model Prediction 194
Model Accuracy 194
Most Informative Features 195
Document Classification 195
Loading Libraries 196
Reading the Dataset into the Categorized Corpus 196
Computing Word Frequency 197
Checking the Presence of Frequent Words 198
Training the Model 199
Most Informative Features 199
Intent Classification and Question Answering 200
Intent Classification 200
Setting tensorflow as the Back End 201
Building the Model 201
Classifying the Intent 203
Question Answering 204
Building the Model 204
Context and Question 204
Serving the DeepPavlov Model 206
Summary 207
Chapter 6: A Novel In-House Implementation of a Chatbot Framework 209
Introduction to IRIS 210
Intents, Slots, and Matchers 211
Intent Class 213
IntentMatcherService Class 214
The getIntent Method of the IntentMatcherService class 217
Intent Classification Service 220
General Query Intent 220
Matched Intent Class 221
Slot Class 223
IRIS Memory 228
Long- and Short-Term Sessions 228
Long-Term Attributes 228
Short-Term Attributes 229
The Session Class 229
Dialogues as Finite State Machines 235
State 237
Shields 238
Transition 239
State Machine 240
Building a Custom Chatbot for an Insurance Use Case 246
Creating the Intents 249
CustomNumericSlot 250
BooleanLiteralSlot 254
AccTypeSlot 255
IPinSlot 256
AlphaNumericSlot 257
IrisConfiguration 259
Adding States 261
Shields 263
DontHaveAccTypeShield 263
DontHaveQuoteDetailsShield 264
HaveAccTypeShield 265
HaveClaimIdShield 265
HaveQuoteDetailShield 266
Adding Execute Methods 266
Exit State 267
FindAdvisorState 267
GetAccountBalanceState 268
GetAccTypeState 269
GetClaimIdState 271
AskForQuote State 272
GetQuote State 275
Start State 278
GeneralQuery State 278
Adding State Transitions 281
Managing State 287
Exposing a REST Service 289
ConversationRequest 289
ConversationResponse 290
ConversationService 290
ConversationController 293
Adding a Service Endpoint 293
Summary 294
Chapter 7: Introduction to Microsoft Bot, RASA, and Google Dialogflow 296
Microsoft Bot Framework 296
Introduction to QnA Maker 297
Introduction to LUIS 305
Introduction to RASA 307
RASA Core 309
RASA NLU 310
Introduction to Dialogflow 311
Summary 316
Chapter 8: Chatbot Integration Mechanism 318
Integration with Third-Party APIs 318
Market Trends 319
Stock Prices 325
Weather Information 331
Connecting to an Enterprise Data Store 336
Integration Module 340
Demonstration of AskIris Chatbot in Facebook Messenger 353
Account Balance 353
Claim Status 354
Weather Today 355
Frequently Asked Questions 356
Context Switch Example 357
Summary 359
Chapter 9: Deployment and a Continuous Improvement Framework 360
Deployment to the Cloud 360
As a Stand-Alone Spring Boot JAR on AWS EC2 361
As a Docker Container on AWS EC2 364
As an ECS Service 367
Smart IRIS Alexa Skill Creation in Less Than 5 Minutes 372
Continuous Improvement Framework 383
Intent Confirmation (Double-Check) 384
Predict Next Intent 386
A Human in the Loop 388
Summary 390
Index 391

Produkt-Details

• Erscheint lt. Verlag: 13.9.2019
• Zusatzinfo: XXII, 385 p. 102 illus.
• Sprache: englisch
• Themenwelt: Informatik ► Programmiersprachen / -werkzeuge ► Java
• Themenwelt: Informatik ► Theorie / Studium ► Künstliche Intelligenz / Robotik
• Schlagworte: Amazon Lex • Banking • BOT • Chat • Chatbot • Enterprise • Google Dialogflow • insurance • Microsoft Bots • NLP • NLTK • Python
• ISBN-10: 1-4842-5034-6 / 1484250346
• ISBN-13: 978-1-4842-5034-1 / 9781484250341