Samenvatting

The Proactive Experience Center is a practice within Atos, that focuses on proactive workplace management to measure the employee experience of customers. They do this for the following reasons:
• Improving the digital workplace;
• Creating digital inclusion;
• Caring for employee well-being;
• Enhancing customer satisfaction;
• Increasing business outcome.

To achieve this, employee experience needs to be measured. One data source for this, is the technical support ticket of the employee from a customer. This provides insight into which problems need to be addressed, to improve the employee experience. However, currently, the analysis of the ticket data is done on a small portion of ticket data. This is because the category, which is essential in providing the context, is lacking or otherwise often inaccurate and it is impractical for an agent to correctly categorise every ticket for analysis. That means that a solution is needed that can effectively categorise tickets based on the written text. To solve this problem, the main question is formulated as follows:

’How effective is a text classification model in classifying the category of technical support tickets, based on descriptive data provided by tickets?’

First of all, to reduce the complexity and limit the number of models, the scope is on English written tickets and three models, where at least one model is from the domain machine learning.

Secondly, background information is gathered to get insight in why this is important and how to approach this problem. At the same time, the possible data sets that Atos has to offer are inspected. This provided the necessary information to keep the focus of the research on predicting the category of tickets that were written by the end-user sent to a support team.

Literature provided three models; (1) support vector machines, (2) deep neural network, and (3) recurrent neural network.

The text preprocessing is identical for each model. Where it differs is the feature engineering. To compare them, deep neural network uses both word representations. Support vector machines and deep neural network use term frequency-inverse document frequency. Recurrent neural network and another deep neural network use sequences in combination with an embedding layer.

To validate the research, the hyperparameters are tuned with cross-validation, and the effectiveness of the model is scored based on the weighted F1 score, which takes class imbalance into account.

This concluded that recurrent neural networks are the best in classifying the category of tickets. The model scored a weighted F1 score of 85%. However, the results indicated that overlapping categories and/or class imbalance affect the model in predicting the correct label. The comparison between the deep neural networks revealed that retaining context and semantics is beneficial, indicating that word embedding is the way forward.
The research exposed problems with the data and provided a solution for them. However, certain topics are not addressed. The following tasks are recommended to address the problems and topics:

• Improve the definition of the categories;
• Keep the non-English written tickets;
• Measure the individual steps to calculate the business value;
• Assess the effect of out-of-vocabulary words;
• Research models that are similar in complexity as recurrent neural networks