Posted on Dec 7, 2021

HOW TO USE NLP AND APPLY UNSUPERVISED ASPECT SCRAPING ON AMAZON COSMETICS REVIEWS?

In this blog, we will apply a deep-learning-based unsupervised aspect scraping method on Amazon cosmetics reviews. This method is named ABAE (Attention-based Aspect Extraction) as well as it was started by He et al. during 2017. Calling the technique deep-learning-based could be some stretch. This network has only a few layers and calculations in a network have a few measure of difficulty though.

In the theory section the workings of the model will be discussed. Note that this section is challenging; or perhaps the model is confusing.

Introduction of a Data Set
The Amazon consumer’s feedback data was issued by He as well as McAuley during 2016. The example data is given below.

data-set
Note that an example has other products category than makeups. The data-set has 9+ million reviews about cosmetic products. Here, only reviewText characteristic will utilized to scrape aspects.

Getting some intuitions on aspects

The finest way to get some intuitions on the aspects is maybe to review different aspect scraping methods. Aspect scraping methods could be largely categorized as: supervised, unsupervised, and rule based. The technique under study is unsupervised.

Let’s take an example of the rule-based characteristic extraction. Take seed adjectives for discovering an extensive range of adjectives, in a mean time, finding the associated nouns. By taking the seed set of {“great”, “fantastic”, “awful”, “very bad”}, one could resolve nouns adjacent to all these adjectives as well as resolve adjectives, which are contiguous to all these nouns at other places. That’s how, both lists of adjectives as well as listing of nouns increase. In addition, WordNet can be used for finding antonyms of adjectives. The related nouns prove to be good options for aspects and one can utilize regular expressions to do the job. Here is the pair of samples, made using spaCy makes it clear that with a formal NLP technique like reliance parsing could be both effective and easier. We welcome you to the world of NLP (Natural Language Processing)!

graph
graph
In first example, an adjective used “fantastic” is openly linked with a noun called “humus”. In second example an adjective “fantastic” is associated with the word humus through an auxiliary verb. It might be tough to solve it with general expressions. A word “humus” could be more accurately designated as a characteristic term. This aspect-term might in case of restaurant reviews become a part of aspect category called food.

Getting touched on the nouns as fine candidates for different aspects, van Cranenburgh and Tulkens (2020) actually available that for datasets in the study, the most common nouns were good aspects; any additional constraint result into worse performances in the training set.

Supervised models might use, for instance a blend of dependency parsing as well as labelled characteristics to scrape the aspects, making the rules spontaneously. The disadvantage to supervised models is they need labelled data and labeled data is limited. Provided a single domain, this could be feasible to produce some labels, however, chances are there that some kinds of characteristics are ignored; the training set might also need maintenance. For that reason, the initiation of unsupervised techniques for aspect scraping made a huge difference.

Theory: An Unsupervised Neural Attention Model for Aspect Extraction
As ABAE He et al. (2017) had introduced the initial unsupervised neural aspect scraping method. A graphical demonstration of a neural network, got from an original article, would help in digging into workings.

graph
Let’s get some insights on an ABAE model. Presume that matrix T has a set of recognized aspect category embeddings; all the aspect categories are entrenched in the similar space like the words. This model tries to imprecise a synopsis z of the sentence using linear combinations of an aspect embeddings within T, with aspects weights within pt. Through using the attention mechanism words associated to the aspects embeddings are highlighted in the sentences summary z, whereas unrelated words get de-emphasized.

Remember that the ABAE model is not very big, however, its workings are intricate. This might not be easy; this is a part of this story.

The ABAE model isn’t Big However, Its Workings are Intricate.
Starting at the lowest, with fixed words embeddings e of the sentence, as well as attention-based weights a summary z of the input sentence is made. The words embeddings are attained using word2vec. That part is named as attention-based encoder and these equations define an attention-based encoder part given below.

graph
In the given formulas, y is an average sentence embedding, as well as M is the matrix, which shapes the last attention values called a. M is learned with training.

Continuing towards top of a graphical representation: a vector r tries to recreate summary z with the linear combination of aspect category embeddings within T. The weights about a linear combination are dogged by p. This is a model, how could this model get optimized?

The lost function, which allows optimization of a model does not get labels: keep in mind that it is an unsupervised model. A loss function includes a triplet-loss function. The triplet-loss functions associate “truthy” examples with “falsy” examples. Here, the loss functions force the rebuilding r of the sentence to get closer to a summary z of the sentences as well as to other randomly sampled z, within the margin. The loss function indicates that reconstruction of the arbitrary summary needs to be more detached than original summary on which the restoration was based. The loss function could be found here. The n span is amongst the m arbitrarily sampled “falsy” values, just like the nature to z.

graph
Finally, regularization gets applied the forces with aspects to become rather independent.

graph
Analyze Cosmetic Products Reviews
For implementation of an ABAE model, another original article got followed. Text was changed into under case, split into sentences as well as non alpha-numerical characters got removed. Finally, stop words got removed. 20 negative samples got introduced for each positive example. A regularization parameter was reserved a 1. An Adam optimizer got applied with the learning proportion of 0.001. The word2vec settings, with minimal word counts of 10, were similar.

Total aspect categories got set to 5, 10, as well as 20 respectively. At the original article, total aspects were set using earlier knowledge.

During this repetition study, the extreme sentence length got set to 35. In this model, the padding was made using special padding tokens having a zero embedding.

A picture of decreasing loss functions of the initial epoch of an ABAE model, occupied from TensorBoard, is given below. The period is around 9000 batches with 1042 sentences. The lost function ends extremely near to zero after different 4 epochs.

graph
Taking the outputs of a model having 20 aspect groups, an effort was done to summarize as well as name the aspects’ categories:

Acquisition
Band-Aids and Sanding
Beauty Appliances and Their Parts
Color
Constant Experimentation
Dryness
Economics and Purchasing
Fragrance
Great
Historical Time
Irritations
It works
No Side Effects
Omission, or What Haven’t Occurred
Product Application
Recommendation and Trustworthiness
Short time
Skin, Complexion, and Lips
Social Aspects
Supplements and Lotions
The main question is, how useful are the aspect categories? Do all these categories imitate the most common aspects discussed, as well as are these categories comprehensive? A post-hoc analysis could provide you more insights to this.

Post-Hoc Analysis Setup
Let’s go through some ideas about the post-hoc analysis.

As for each sentence, the linear combination about aspect categories is made, Shannon’s entropy might provide insights where sentences are comparatively flat according to aspect categories (higher entropy, more surprises), and that sentences are comparatively spiky according to aspect categories (less surprise, low entropy). Not all sentences are equally useful as per the discussed products, or the provided aspects; an entropy score might reflect it. The max entropy for different 20 categories is log2 (20) and that is 4.32.

To get insights in some kind of precision, provided an example of sentences one might evaluate in case the aspects given in a sentence are properly reflected in aspect categories; perhaps considering the entropy score about a sentence. The sentence ratio, which are reflected good enough in all aspects might render an accuracy score. In the similar way, a notion for recall could get introduced: are a few aspect categories lost, i.e. do the scraped aspect categories covering complete spectrum? Finally, this data might offer insights into significance of scraped aspect categories.

Using a sample of initial 100.000 sentences, 100 sentences having maximum entropy scores got chosen for review. All these sentences comparatively fat with the scraped aspect categories. Below given are top 10 showing entropy, with 3 most applicable categories as well as a sentence itself.

Entropy score: 0.9422962069511414
Aspect categories:
0.1645447462797165 - Trustw. and recom.
0.07130391895771027 - Purchasing and economics
0.0629160925745964 - Cont. exp.
little goes long way

Entropy score: 0.9417917728424072
Aspect categories:
0.16252847015857697 - Trustw. and recom.
0.07755423337221146 - Sanding and band-aids
0.07012781500816345 - Purchasing and economics
really no fragrance though

Entropy score: 0.9401639103889465
Aspect categories:
0.1677447110414505 - Trustw. and recom.
0.07264218479394913 - Purchasing and economics
0.06493180245161057 - Lotions and supplements
oily skin powder users unite

Entropy score: 0.940011203289032
Aspect categories:
0.17124606668949127 - Trustw. and recom.
0.07094670832157135 - Purchasing and economics
0.06432071328163147 - Cont. exp.
younger looking skin already young looking man face

Entropy score: 0.940011203289032
Aspect categories:
0.17124606668949127 - Trustw. and recom.
0.07094670832157135 - Purchasing and economics
0.06432071328163147 - Cont. exp.
id rate product 4

Entropy score: 0.9396255016326904
Aspect categories:
0.16920916736125946 - Trustw. and recom.
0.07293383777141571 - Purchasing and economics
0.06363774091005325 - Cont. exp.
feel makeup misrepresented

Entropy score: 0.9394322037696838
Aspect categories:
0.15950456261634827 - Trustw. and recom.
0.09560395777225494 - Sanding and band-aids
0.06892047822475433 - Purchasing and economics
lectric shave adds little bit lubrication skin help razor glide smoothly

Entropy score: 0.9388947486877441
Aspect categories:
0.15905912220478058 - Trustw. and recom.
0.10041727870702744 - Sanding and band-aids
0.06838632375001907 - Purchasing and economics
something twinge spicier others gives clubman advantage brands not mention scent long lasting rarethis always rotation classic aftershaves daily use would highly recommend anyone likes bay rum wants something something better

Entropy score: 0.9387928247451782
Aspect categories:
0.15888644754886627 - Trustw. and recom.
0.10159197449684143 - Sanding and band-aids
0.06842176616191864 - Purchasing and economics
originally reviewed paraffin pearls 2000 bought 2003 still using 2011
As anticipated, the given values to aspect categories are lower, comparatively flat. A few data points do not look associated to cosmetics products.

Below are the top 5 lowermost entropies given:

Entropy score: 0.1903418004512787
Aspect categories:
0.8898552060127258 - Trustw. and recom.
0.03991604223847389 - Social aspects
0.020657163113355637 - Acquisition
discovered conditioners could finally grow hair long

Entropy score: 0.1903418004512787
Aspect categories:
0.8898552060127258 - Trustw. and recom.
0.03991604223847389 - Social aspects
0.020657163113355637 - Acquisition
need replace 2 bottles

Entropy score: 0.1903418004512787
Aspect categories:
0.8898552060127258 - Trustw. and recom.
0.03991604223847389 - Social aspects
0.020657163113355637 - Acquisition
obviously get blood fingers could put gloves bothers

Entropy score: 0.21359199285507202
Aspect categories:
0.8747269511222839 - Trustw. and recom.
0.04306989908218384 - Social aspects
0.02173822745680809 - Acquisition
added bonus breath stays fresher longer dont feel gross dont shower every day shirt dont anymore deodorant stains im lot less gassy ever

Entropy score: 0.24772478640079498
Aspect categories:
0.8509187698364258 - Trustw. and recom.
0.03546708822250366 - Acquisition
0.03343984857201576 - Lotions and supplements
purchased cute conair name

Entropy score: 0.27374812960624695
Aspect categories:
0.8198038339614868 - Trustw. and recom.
0.04119117930531502 - Lotions and supplements
0.04065270721912384 - Beauty appliance and parts
conair word ismisleading implying market special travel accessories

Entropy score: 0.27374812960624695
Aspect categories:
0.8198038339614868 - Trustw. and recom.
0.04119117930531502 - Lotions and supplements
0.04065270721912384 - Beauty appliance and parts
way faster easier use trying use blow dryer round styling brush

Entropy score: 0.27374812960624695
Aspect categories:
0.8198038339614868 - Trustw. and recom.
0.04119117930531502 - Lotions and supplements
0.04065270721912384 - Beauty appliance and parts
soft gentle sponge applicator could kept
In the given examples of low as well as high entropy cases, one category rules the selection.

Here is the sample of a few randomly selected sentences. After studying 30 sentences, around 50% of aspect categories have proved to be informative; at times missing a clear category. Possibly the entropies in the random samples are very high?

Entropy score: 0.6920719146728516
Aspect categories:
0.26366573572158813 - Application of the product
0.2539910674095154 - Trustw. and recom.
0.18494568765163422 - Color
rather stay way old wrinkeledand looking gooood

Entropy score: 0.6996863484382629
Aspect categories:
0.3106836974620819 - Application of the product
0.15334966778755188 - Color
0.13878107070922852 - Social aspects
redness forehead get little better next day

Entropy score: 0.781614363193512
Aspect categories:
0.19744877517223358 - Lotions and supplements
0.14799079298973083 - Application of the product
0.147004172205925 - Purchasing and economics
used 9 appears discontinued

Entropy score: 0.7271698713302612
Aspect categories:
0.24797487258911133 - Lotions and supplements
0.1878710389137268 - Application of the product
0.14395123720169067 - Social aspects
theres still flashes old plasmatics quotpig pigquot quotfast food servicequot even lyricless quotplasma jamquot certain ramoneslikability quotsummernitequot quotmasterplanquot less bsides redundant quotliving deadquot sounds like discarded outtake original version

Entropy score: 0.6703991293907166
Aspect categories:
0.4293365776538849 - Trustw. and recom.
0.1401335746049881 - Application of the product
0.09364698827266693 - Lotions and supplements
failoni chamber orchestra charming joyfulservile romero degrandis good roles

Entropy score: 0.6442851424217224
Aspect categories:
0.3380078971385956 - Cont. exp.
0.17049644887447357 - Purchasing and economics
0.16352525353431702 - Acquisition
meant disposable bics

Entropy score: 0.5233464241027832
Aspect categories:
0.4165324866771698 - Trustw. and recom.
0.2911195158958435 - Beauty appliance and parts
0.13291838765144348 - Acquisition
fun read etchings vinyl dont remember done beforeshe wasnt americas little sweetheart

Entropy score: 0.7314188480377197
Aspect categories:
0.25480103492736816 - Trustw. and recom.
0.2042699158191681 - Social aspects
0.11465466767549515 - Acquisition
bottle sells approximately 50

Entropy score: 0.7414893507957458
Aspect categories:
0.24682772159576416 - Great
0.18355034291744232 - No side effects
0.15264792740345 - Cont. exp.
usually get fairly cheap dollar tree nice 100 pack quality nice

Entropy score: 0.7384803295135498
Aspect categories:
0.22678133845329285 - Social aspects
0.1833328753709793 - Purchasing and economics
0.17800934612751007 - Color
apply go without shiny look
Discussion
Not unexpectedly, the consistency of aspect categories is very good. However, this coherence needs to be attributed for word2vec rather than ABAE. Nevertheless, reading the aspect categories gets a narrative immediately about the topics or aspects of cosmetics reviews. However, are these early observations effective?

What is upsetting is, the last aspects categories initiate largely overlapping with the early cluster centers given as a starting point. These clusters were produced by taking words of first 250.000 sentences. It underscores the perception behind a model and challenge its validity. Why run a model, as well as why would these centers get aspects?

Any manual review of validity aspects in the post-hoc analysis of results indicates a mixed result. A line between the aspects as well as not is at times hard to pull, inspiring subjective interpretation. This does indicate that ABAE includes the most significant topics. The topics’ assignment of the sentences however are tough to understand, as well as some assured assignments got missed. Were LSA and LDA render a mix bag on a topic level, the ABAE looks to render a mix bag on an assignment level.

Conclusion
The results about unsupervised aspect scraping with neural networks were extremely promising initially. It could be attributed to cohesion of aspect terms within aspect categories. Related to LSA and LDA, this is promising.

The assurance of an ABAE model is, this can scrape aspects without any labels; in the unsupervised as well as tentative way. Here, the results are weaker to claim any success. A superior method of digging into aspects of sentences might perhaps be raising a few aspect categories depending on the domain knowledge as well as label a few data.

If you still have any confusion, or want to know more about this, contact X-Byte Enterprise Crawling today!