IBM Debater® Claim Sentences Search¶
Data Source: https://developer.ibm.com/exchanges/data/all/claim-sentences-search/
Claims are short phrases that an argument aims to prove. The goal of the Claim Sentence Search task is to detect sentences containing claims in a large corpus, given a debatable topic or motion. The dataset contains results of the q_mc query – sentences containing a certain topic, as described in the paper – containing 1.49M sentences. In addition, the dataset contains a claim sentence test set containing 2.5k top predicted sentences of our model, along with their labels. The sentences were retrieved from Wikipedia 2017.
The dataset includes:
- readme_mc_queries.txt – Readme of the claim sentence search results
- readme_test_set.txt – Readme of the test set
- q_mc_train.csv – Sentences retrieved by the q_mc query on 70 train topics
- q_mc_heldout.csv – Sentences retrieved by the q_mc query on 30 heldout topics
- q_mc_test.csv – Sentences retrieved by the q_mc query on 50 test topics
- test_set.csv – Top predictions of our system along with their labels
In [1]:
from IPython.display import clear_output
In [2]:
!pip install wordcloud
clear_output()
In [3]:
!pip install gensim
clear_output()
In [4]:
!pip install --user -U nltk
clear_output()
In [5]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from wordcloud import WordCloud
import gensim
from gensim.utils import simple_preprocess
from gensim.parsing.preprocessing import STOPWORDS
from nltk.stem import WordNetLemmatizer, SnowballStemmer
from nltk.stem.porter import *
stemmer = SnowballStemmer('english')
import nltk
nltk.download('wordnet')
Out[5]:
In [6]:
!wget https://dax-cdn.cdn.appdomain.cloud/dax-claim-sentences-search/1.0.2/claim-sentences-search.tar.gz -O claim-sentences-search.tar.gz && tar -zxf claim-sentences-search.tar.gz; rm claim-sentences-search.tar.gz
In [7]:
ls
In [8]:
# Load the training data
train_data = pd.read_csv('q_mc_train.csv')
Data Exploration and Data Visualization¶
In [9]:
# Shape of the training data
train_data.shape
Out[9]:
In [10]:
# Training data column names
train_data.columns
Out[10]:
Column name description¶
- id - the topic id, as specified in the appendix of the paper
Towards an argumentative content search engine using weak supervision Ran Levy, Ben Bogin, Shai Gretz, Ranit Aharonov and Noam Slonim COLING 2018 - topic - the motion topic
- mc - the main Wikipedia concept of the topic
- sentence
- suffix - the suffix of the sentence following the mention of the mc
- prefix - the prefix of the sentence preceding the mention of the mc
- url - link to source Wikipedia article
In [11]:
# Get unique topic id
print(sorted(pd.unique(train_data['id'])))
In [12]:
# Number of unique ids
len(pd.unique(train_data['id']))
Out[12]:
In [13]:
# Get unique topics
sorted(pd.unique(train_data['topic']))
Out[13]:
In [14]:
# Number of unique topics
len(pd.unique(train_data['topic']))
Out[14]:
In [15]:
# Number of data points under specific id and topic
train_data.groupby(['id','topic']).size().reset_index().rename(columns={0:'count'}).head()
Out[15]:
In [16]:
# Total number of words in the document
train_data['sentence'].apply(lambda x: len(x.split(' '))).sum()
Out[16]:
In [17]:
# Get unique main content
main_content = pd.unique(train_data['mc'])
In [18]:
main_content
Out[18]:
In [19]:
plt.figure(figsize=(20,10))
train_data.mc.value_counts().plot(kind='bar');
Explore HTTP Cookie topic¶
In [20]:
# Get all data points under the topic HTTP Cookies
http_cookies_data = train_data[train_data['mc'] == 'HTTP cookie']
In [21]:
# Total data points under the topic
http_cookies_data.shape
Out[21]:
In [22]:
http_cookies_data.head()
Out[22]:
In [23]:
# Total number of words
print(http_cookies_data['sentence'].apply(lambda x: len(x.split(' '))).sum())
In [24]:
def input_preprocess(inp_sentence):
processed_result = []
for token in gensim.utils.simple_preprocess(inp_sentence):
if token not in gensim.parsing.preprocessing.STOPWORDS and len(token) > 3:
processed_result.append(token)
return processed_result
In [25]:
# Removing stop words from the sentences
processed_http_cookies = http_cookies_data['sentence'].map(input_preprocess)
In [26]:
token_http_cookies = []
for i in processed_http_cookies:
for j in i:
token_http_cookies.append(j)
In [27]:
# Number of unique words after preprocessing
len(set(token_http_cookies))
Out[27]:
In [28]:
unique_tokens_http = set(token_http_cookies)
WordCloud creation¶
In [29]:
unique_string=(" ").join(unique_tokens_http)
wordcloud = WordCloud(width = 1000, height = 500, background_color="white").generate(unique_string)
plt.figure(figsize=(15,8))
plt.imshow(wordcloud)
plt.axis("off")
plt.show()
plt.close()
Topic Modelling¶
In [30]:
# Pre-process documents to remove stop words
processed_docs = train_data['sentence'].map(input_preprocess)
In [31]:
processed_docs[:5]
Out[31]:
In [32]:
# Create dictionary which contain mapping between words and integer ids
dictionary = gensim.corpora.Dictionary(processed_docs)
In [33]:
count = 0
for k, v in dictionary.iteritems():
print(k, v)
count += 1
if count > 10:
break
In [34]:
# Total number of unique words in the dictionary
count = 0
for k, v in dictionary.iteritems():
count += 1
print('Total number of words in the dictionary: ', count)
In [35]:
# Filter out tokens in the dictionary by the frequency
dictionary.filter_extremes(no_below=15, no_above=0.5, keep_n=10000)
In [36]:
count = 0
for k, v in dictionary.iteritems():
count += 1
print('Total number of words in the dictionary after filtering process: ', count)
In [37]:
# Convert document into bag-of-words format. This results in a tuple containing token_id and token_count
# The output indicates that a token with a specific id appears specified number of time
# For example: (7,1) indicates token with id 7 appears 1 time.
bow_corpus = [dictionary.doc2bow(doc) for doc in processed_docs]
bow_corpus[100]
Out[37]:
In [ ]:
# LDA model
# parameters provided are document vector, number of requested latent topics, dictionary containing mapping from words IDs to word,
# and number of passes through the corpus.
lda_model = gensim.models.LdaMulticore(bow_corpus, num_topics=5, id2word=dictionary, passes=1)
In [ ]:
for idx, topic in lda_model.print_topics(-1):
print('Topic: {} \nWords: {}'.format(idx, topic))