BERT based sentiment analysis of consumer hesitancy toward solar energy adoption

Abstract

The adoption of solar energy is pivotal in addressing climate change and achieving long-term energy security. However, its widespread deployment faces notable barriers, including high upfront costs, consumer doubts about system reliability, and unclear policy landscapes. To explore public perception and barriers to adoption, this study proposes a hybrid sentiment analysis framework that integrates a fine-tuned BERT model with TF-IDF-based feature enhancement. The model is applied to diverse consumer-generated content sourced from social media, review platforms, and public forums. Because the corpus is mixed-access, social-platform data were collected via official APIs and are not redistributed as raw text; instead, we share only permitted identifiers for rehydration (where allowed), preprocessing scripts, and non-reversible derived artifacts, while open corpora are shared in accordance with their licenses. Our approach achieves a validation F1-score of 0.85 and an overall test F1-score of 0.82, accurately capturing nuanced sentiments across domains. Quantitative analysis reveals that cost-related concerns account for over 41% of negative sentiment, followed by reliability (28%) and environmental skepticism (19%). The inclusion of cumulative gain analysis and high-confidence prediction filtering improves result interpretability and prioritization of insights. These findings provide valuable guidance for policymakers, solar energy firms, and sustainability advocates seeking to design targeted interventions and accelerate consumer acceptance of solar energy technologies.

Introduction

Energy transition to solar energy systems has emerged as a policy priority as countries, companies, and individuals are also interested in minimizing the impact on the environment, dependence on carbon content, and energy autonomy¹. Solar energy has competitive solutions that are alternatives to conventional energy systems, and it provides a desirable solution to sustain the energy global demands². Solar energy has great potential in terms of its conformance to the environment and economically. Nonetheless, the environmental preservation is still not able to fully encourage the utilization of solar technologies by consumers due to reluctance fueled by a series of perceived problems with the profitability, stability, reliability, and even availability of solar technologies³. Knowing the causes of consumer inertia is critical to policymakers, industries, and environmental stewardship organizations that want to foster a shift towards solar-based energy^4,5. Distrust among consumers is usually two-dimensional: some are concerned with the costs of installation and maintenance, and some are involved with the false beliefs on the efficiency and quality of solar solutions⁶. This skepticism may inhibit the management of solar energy consumption and making the masses a part of how sustainable activities⁷. Certainly, such opinions can be measured using social media, discussion boards, consumer review sites, and newsgroups as rich sources of consumer attitudes and capture the shades of opinion that would be otherwise challenging to comprehend. By examining such sentiments, the stakeholders in solar energy can target impediments to their adoption, as well as plan on overcoming the resistance of the general population⁸.

Although there is already existing research taking advantage of artificial intelligence and natural language processing (NLP) to perform sentiment analysis in a given field, there is also a gap in the literature with regard to related research doing the same but focusing on the expressing the subtle and real-time sentiments towards solar energy implementation⁹. Available studies have focused on either the general sentiments on environmental sustainability or generic patterns of solar energy adoption, with little to no discussion of the very issue that prompts consumer apathy towards the use of the mentioned solar energy technologies¹⁰. This gap solar reveals the necessity of more specific analysis of sentiment in this issue, taking account of the extent and direction of mass opinion both on adoption of solar energy activity, and also in relation to specific topics of costs, environmental effects and policy factors¹¹. In this study we seek to address this knowledge gap by taking the empirical and more detailed approach to measuring the attitudes towards solar energy by analyzing the sentiment in general as well as around social media and review sites. Collecting the information based on a variety of sources, we will be able to take an accurate picture of the overall impression of the population and determine the key issues, as well as monitor the sentiment fluctuations in time¹². The method aims at providing relevant insights to policy makers, entrepreneurs through disclosure of a strong sentiment trend that shall help them in policy making strategy that may convey positive sentiment by customers. In particular, the research will attempt to identify some of the nuances in the sentiment around the research that can be used to guide policy, participation and commercialization efforts in the future towards greater acceptance and adoption of solar energy¹³.

Research gap

Prior BERT-based sentiment studies in energy typically analyze general renewable-energy discourse or policy discussions using single-platform data (e.g., Twitter), and often treat BERT as a stand-alone classifier without hybrid feature engineering or decision-supports. They rarely (i) adapt models to the consumer hesitancy subdomain of solar energy, (ii) integrate statistical salience (TF–IDF) with contextual embeddings (BERT), or (iii) quantify barrier prevalence (e.g., cost vs. reliability) and prioritize insights with tools such as cumulative gains and high-confidence filtering. This leaves an actionable gap for a domain-adapted, multi-source, decision-oriented analysis of solar adoption sentiment.

Research questions

• RQ1: What themes most strongly drive consumer hesitancy toward solar energy across multi-source, consumer-generated text, and what are their relative contributions?

• RQ2: Does a domain-adapted, hybrid TF–IDF + BERT model improve sentiment classification quality and interpretability for solar adoption discourse?

• RQ3: Can high-confidence prediction filtering and cumulative-gain ranking prioritize the most decision-relevant instances compared with unranked sampling?

Hypotheses

1. Hybrid performance A domain-adapted TF–IDF + BERT model will achieve higher overall classification performance and clearer theme separation than either TF–IDF-only or BERT-only variants on held-out data (e.g., higher F1 and calibration quality).

2. Prioritization gains Applying high-confidence filtering and cumulative-gain ranking will increase early precision (e.g., precision) over random or unranked selection, thereby improving decision prioritization for stakeholders.

3. Barrier ordering Among negative sentiment, cost concerns will be most prevalent, followed by reliability concerns and environmental skepticism, respectively.

This study differs from existing efforts by: (i) fine-tuning BERT specifically for solar adoption vocabulary and concerns; (ii) fusing TF–IDF with BERT embeddings to couple statistical salience with contextual semantics; (iii) assembling a multi-source corpus (social media, consumer reviews, forums, and industry blogs) to mitigate single-platform bias; (iv) quantifying barrier shares (e.g., cost, reliability, environmental skepticism) and (v) adding decision-support analysis via high-confidence filtering, cumulative gains, and calibration, which improves interpretability and prioritization for policy and industry use.

Objectives and research novelty

This study proposes a sequential AI/NLP pipeline for consumer sentiment analysis that is tailored to the solar-adoption domain and designed for decision support. Each stage of the pipeline is intended to improve effectiveness (better sentiment fidelity and interpretability) and efficiency (clear, auditable components that generalize across heterogeneous sources). The major objectives and novel contributions are:

• Hybrid feature extraction method We propose a hybrid feature representation that fuses sparse lexical salience (TF–IDF) with dense contextual semantics (BERT) when encoding consumer posts, reviews, and commentaries. TF–IDF highlights statistically informative terms that characterize solar-adoption discussions, while BERT captures sentence-level meaning and context, enabling the model to represent nuanced attitudes that cannot be explained by keyword frequency alone. By integrating both views, the hybrid approach supports accurate classification while preserving interpretable lexical signals for explaining predictions.

• Domain-specific sentiment analysis model Because generic sentiment models often miss sector-specific concerns, we fine-tune a BERT-based sentiment model to the vocabulary, topics, and language patterns used in solar-adoption discourse. This domain adaptation improves sensitivity to solar-specific drivers of hesitancy (e.g., cost and ROI concerns, reliability doubts, incentives and policy frustration, and environmental skepticism), enabling more faithful classification of positive and negative sentiment in this application context.

• High-confidence prediction for decision support Beyond maximizing average accuracy, the framework emphasizes high-confidence prediction so that stakeholders can prioritize the most decision-relevant cases under limited outreach and policy resources. We incorporate confidence-aware (e.g., cumulative gains, lift, and Top-K accuracy) to quantify how effectively the model surfaces high-impact instances compared with unranked or random selection, making the outputs actionable for time-sensitive decision making.

• Multi-dimensional visualization techniques To improve interpretability for non-technical users, we include multi-dimensional visual summaries such as word clouds, sentiment distribution views, and thematic/topic mapping across sources and time windows. These visualizations support rapid understanding of dominant themes, reveal shifts in public attitudes over time, and help stakeholders interpret how events or policy changes correspond to changes in consumer sentiment, thereby strengthening the usability of the model outputs for communication and strategy.

The specificity of the research will be in using the state of the art AI methods best suited to domain relevant sentiment analysis of the solar energy segment. What this paper does achieve, however, is a dual approach that accounts not only to the statistical significance of the words but also to language context evaluation that is enabled by the n-grams, words, sentences and document embeddings that were provided by BERT. The next peculiarity is considering only high cont-fidence and quantized cumulative gains, which combination distinguishes the current research among the widespread senti-ment analysis methods, whereas the results provided here are not only correct but also operational. The visualization methods used in this study also served the novelty of the study, as it makes the intuitive exploration of the consumer reluctance easier, helping the stakeholders in the making of data-driven decisions. In addition to contributing further to the methodological possibilities of sentiment analysis in the context of the solar industry, the research presented here will offer the stakeholders practical information that they can use to adjust and to justify resistance to the broader solar energy implementation. When the policymakers and leaders of the industry in charge of the energy industry realize reasons behind consumer reluctance they will be able to devise ideas that attract consumer opinion and thus, promoting growth of solar power. The research methods and the procedures followed in acquiring the data, preprocessing, feature identification, model fitted and its evaluation techniques are explained in the following sections to establish a proper analysis of consumer sentiment following the adoption of solar energy.

Literature review

The view that the world must shift to wide-scale use of renewable energy sources as an urgent corrective action to climate change consequences and environmental deterioration is a well known phenomenon¹⁴. Renewables which include solar energy, wind power, and hydropower can offer a sound alternative to fossil fuel by decreasing the amount of greenhouse gas through reduction in the emission, increasing energy independence, and economic resiliency¹⁵. Notwithstanding all these advantages, they have not been adopted widely among consumers as expected. This reluctance can be reduced to numerous and complex and multiple source impediments that include but are not limited to disabled start-up costs of the renewable technology, skepticism in terms of reliability and performance, absence of any consistent government stimuli, and scepticism in regard to the long-term economic benefits of the same¹⁶. The sentiment that is considered as an essential factor in energy policy formation and consumer behaviour change identification is of concern since its development requires appropriate methods, including the sentiment analysis, which is required to perceive and quantify consumer attitudes as psychological, social and economic premises of solar energy.

Prior researches involving consumer attitude towards solar energy in available literature are determined through outdated quantitative techniques of questionnaires, interviews and focus group discussions. These approaches are quite suggestive concerning the personal perception yet restricted by a sampling range, geographical and self-reporting data¹⁷. It is in consequence of this that certain conventional means may not be scalable and may not be suitable in the process of sampling the population and gathering adequate quantity of feedback so as to be able to generalize¹⁸. Nonetheless, such approaches lack a dynamic mechanism of measuring sentiment variation over time to capture the impact of an intervention as such an exogenous influence such as policy or technological trends, to name a few. Digital media, or rather social media, online discussion groups and consumer review websites are a new technology that has changed the playing field on which surveys of the opinion of the people are carried out¹⁹. Such platforms offer an unfettered, real-time flow of consumer feedback on solar energy offering enormous amounts of unstructured data with possible meaningful insights into the trend of sentiment and emerging topic areas of concern²⁰. Nonetheless, such vast amounts of data can only be effectively analyzed through the use of highly sophisticated Natural Language Processing (NLP) methods that have the capacity to capture, process and decode unstructured un-structured textual data²¹.

The most significant application of NLP is sentiment analysis, which is widely spread in the field of finance, healthcare, political science, and customer service²². Sentiment analysis can also be used in studies on solar energy, as it helps to reveal the attitudes of consumers toward a certain source of energy, their reaction to any decisions about policy, and perception of the issues associated with sustainability in general²³. Traditional sentiment analysis methods can be classified in two major groups, namely,the lexicon-based methods and machine learning-based approaches. The lexicon-based approaches are based on the prepared lists of words categorized as positive or negative to identify the polarity of the sentiment in a text²⁴. Nevertheless, one notable drawback to lexicon based approach is the nonexistence of contextual meaning of a word. To give another example, a word such as investment or subsidy conveys positive or negative sentiment based on the context in which it is used, which cannot be picked up by lexicon-based approaches typically²⁵. This contextual deficiency usually brings about shallow outcomes especially in areas such as solar power, where technical and contextual details could heavily influence consumer attitude²⁶.

Sentiment analysis models using machine learning would address these limitations to some extent because they would construct classifiers by training on labeled data and therefore would identify keywords related to the domain, as well as contextual patterns²⁷. Although these models are more flexible and highly accurate in detecting subtle sentiment, they have their difficulties²⁸. To begin, machine learning systems need large amounts of quality labeled data, aoutput that may be challenging to produce and, in particular, in niche areas of application such as in solar energy. In characterizing long-range dependencies inside text and which are important in correctly interpreting various complex sentiments, ordinary machine learning tasks, including Support Vector Machines (SVM) and Naive Bayes classifiers, also fail to capture it²⁹. Moreover, prejudices may exist in the training information, which can share the significance and lead to unbalanced sentiment forecasts affecting the quality of the model.

The most recent innovations of deep learning training, notably models of transformers (like BERT), have overturned sentiment analysis. BERT due to its two-way architecture represents each token with respect to the whole sentence and this aspect allows achieving better disambiguation of subtle language and more precise sentiment detection of unclear sentences³⁰. Transformer models perform better on sentiment tasks, indicating they reflect context better than classical models, empirically on tasks in healthcare, finance, and e-commerce Even though these models have well established applications in other areas, they are rarely introduced in solar-energy sentiment analysis and even rarely to the attribution to the causes of consumer reluctance. This paper builds on a model introduced by the authors³¹ to address these short-comings by extending a new BERT based model that was optimised to the task of solar energy sentiment analysis. To apply BERT to this application, domain adaptation is performed by presenting the model with data in the domain and what the model is intended to learn as applied to solar energy and to all that is involved in government incentives, impact to the environment and technological reliability of solar energy so some related solar energy words such as incentives³². By having the model, BERT trained on the language applied within the sphere of solar energy, the model can find superior outcomes by defining the sentiments as positive and negative,³³. This process of fine-tuning therefore can enable the model to capture particular concerns of the consumers as compared to using a normal sentiment analysis models and therefore can prove useful to stakeholders on what they need to know about how the populace is feeling towards the solar energy technologies³⁴. It will be useful to improve the decision of strategic communication management with reference to elements that induce reluctance among consumres that can be acquired through self-training of fine traded BERT model as illustrated in Table 1.

Table 1.

Comparative analysis of related studies in solar/energy sentiment analysis.

Study	Year	Model	Metric (exact)	Data source	Scope	Limitations (from paper)
35	2019	Survey/review	NR (survey; no comparable benchmark metric reported)	–	Overview of sentiment methods	Not a domain benchmark; metrics not comparable
36	2023	SVM	NR	Social media	Consumer sentiment on renewable energy	Dataset-specific; limited context modeling
37	2023	Transformer	NR	Policy boards/forums	Energy-policy sentiment	Narrow domain focus
38	2024	XLNet	NR	Social media	Baseline transformer	Requires tuning; energy metrics may be unreported
1	2024	DistilBERT	NR	Twitter	Lightweight monitoring	Typically below BERT in accuracy
39	2024	DeBERTa/DeBERTaV3	NR	Multi-platform	Strong contextual modeling	Heavier compute; limited energy benchmarks
6	2024	ELECTRA	NR	Mixed sources	Efficient pretraining	Results vary by implementation
40	2024	Survey/review	NR (survey; not an energy sentiment benchmark)	–	Green AI methodology overview	Not a sentiment benchmark paper
41	2024	BERT	NR	Social media	Renewable energy topic analysis	Compute/data requirements
42	2024	TF–IDF + BERT	NR	Online reviews	Renewable cost opinions	Cost-centric; limited context
43	2024	GRU + attention	NR	Consumer reviews	Cost/incentive sentiment	Tuning complexity
44	2024	Text mining	NR	Public forums	Early sentiment mining	Weak at nuanced sentiment
45	2024	Transformer models	NR	Multi-platform	Energy sentiment across sources	High compute; fine-tuning required
46	2024	Deep CNN	NR	News/reviews	Incentives sentiment	Computationally intensive
47	2024	CNN	NR	Policy forums	Policy-oriented sentiment	Limited domain
9	2025	LSTM	NR	Mixed sources	Adoption sentiment trends	Less robust on long texts
48	2025	SVM + SenticNet	NR	Blogs/forums	Cost/policy sentiment	Lexicon limitations

Only values explicitly reported in the cited papers are included; NR = not reported/not verifiable from the paper. Metrics are not directly comparable across different datasets, label schemes, and evaluation protocols.

In addition to BERT, the article suggested the TF-IDF-BERT-based feature extraction, i.e., to use both TF-IDF and BERT embeddings⁴⁹. TF-IDF is a quantitative technique that can be used to identify those terms that are relevant to the specific topic of interest by comparatively differentiating between the frequency distribution of terms in the specific document and their frequency distribution at the corpus level; in that way, terms important in the discussions about solar energy sources, such as policy, subsidy and carbon footprint, would be identified⁵⁰. Albeit possessing the capability to take into account pair-wise relations between words, BERT embeddings provide superior results in the form of the contextual semantic meaning of the consumer sentiments⁵¹. This paper proposes a combination of TF-IDF and BERT, therefore producing a synergistic tool that enhances the use of both statistical frequency-based measures as well as contextual-aware sentiment analysis performed in a finer-grained resolution to increase the solar energy sentiment analysis⁵².

Among the few contributions of the study to the context of solar energy sentiment analysis, there is using high confidence predictions and Cumulative Gain analysis, since the latter is used in virtually all components of the scientific work⁵³. This suggests that in allocating the model to classify, the model may be in a position to focus on those it is certain of since this will enable stakeholder of the model to take the right action. The guarantee of analytics approachology that measures the effective performance of the model in the identification of the requisite perceptions beyond these arbitrary thresholds is referred to as Cumulative gain analysis⁴⁸. The analysis is specifically applicable in the solar energy sentiment scenario because policy and industry activity can be impacted by the convenient real time detection of the event of opinion change. The research provides the stakeholders with strategic value due to the high quality of the predicted data and cumulative gain evaluation, which ensure the possibility to assess the changing trends in consumer sentiment, form data-driven decisions that will assist in adopting solar energy⁵⁴.

Visualization methods used to generate sentiment analysis results help a great deal when it comes to interpretability of the latter particularly when dealing with complex and large data sets. This paper will involve some of these visualizations in the form of word clouds, sentiment trend analysis and thematic mapping among others to aid in the understanding of motivation behind consumer aversion to follow through with the use of solar energy⁵⁵. Word clouds have the advantage of appearing as a mirror of the frequently used terms and displaying words that generate the negative sentiments, and as such the stakeholders are aware at first glance what areas have the strongest concern. To observe how the attitude of the population evolves over time, sentiment trend analysis is used to give an indication of the effect events like policy-announcement or a technological breakthrough have on the consumer attitude⁵⁶. Thematic mapping breaks down the sentiment according to the main areas to simplify it and provide a systematical perspective on the opinion driving forces. Such visualization methods complement the analysis of sentiment analysis information and allow to show such results to the project stakeholders, including those without the knowledge of sentiment trends due to lack of programming skills⁵⁷. RoBERTa and XLNet are typical transformer versions that have been a good general baseline and its energy prediction performance is mixed and cannot be compared on the same datasets. In pursuit of fair, decision-oriented modeling of solar adoption, our priorities are (i) domain adaptation of BERT, well-studied, reproducible on small/medium corpora (ii) model interpretability our hybrid TF IDF + BERT feature set vocalizes lexical drivers and reveals contextual signals (iii) calibration/triage (confidence range filtering, cumulative gains); our pipeline naturally enables this.

RoBERTa and XLNet may cause some marginal gains to certain benchmarks, but may be more compute demanding and tuning sensitive than modestly large, multi source energy corpora. We, therefore, take BERT as the standard and Wu et al. (2020) make it clear that it is the precision of genomic encoders, stability, and the level of transparency to be utilized by the stakeholders. Our efforts to avoid over-claiming include being cautious to warn that cross-paper scores are not apples-to-apples; we constrain our assertions to within-study evidence and provide per-class/per-source scores to limit the potential of dataset bias. Thus, the contribution that the work makes is closing the gap in literature of sentiment analysis in the solar energy domain and the combination of NLP methods with the pre-training focus on the effective complementation strategy. This paper integrates BERT, high confidence predictions, cumulative gain analysis and defensive visualisation to yield a multi faceted and complex picture of consumer unwillingness. The gap that this kind of paper fills in the existing literature is the possibility to provide policy makers, energy communicators and other stakeholders with usable knowledge which can be applied in policies, planning, and organising and communication strategies in changing consumer attitude towards solar energy.

Methodology

This paper outlines the systematic AI-based system utilised for analyzing consumers’ hesitance towards renewable energy adoption comprehensively followed by a detailed breakdown of a process that has been fabricated to provide a meaningful and coherent result. The process includes every stage starting from data gathering all the way through to the analysis of the results basically using various modern techniques that improve model performance, accuracy and, more explicitly, the interpretability of the result. Selecting a good set of data, preprocessing it and making the best feature extraction make the foundation of this approach to create a high quality data set needed for the sentiment analysis. Moreover, it involves proper training of the model, appropriate selection of hyperparameters, and highly accurate testing, so that the model quite adequately reflects all the subtleties of consumer sentiment in the field of solar energy. The end vision is that of having a strong and flexible process that will be able to harness insights from raw sentiment data and ultimately inform organisational decision making based on the sentiments of the populace. Through use of this methodological framework, it becomes easier to understand driving forces of the solar energy adoption and make better decision based on the understandings which in in line with the public opinion hence the formulation of sound policies in favour of the solar energy sector. This study analyzes publicly available or open-download text and does not involve intervention or interaction with human participants. We report results only in aggregate, do not attempt to identify or profile individuals, and avoid publishing verbatim excerpts that could enable re-identification. During preprocessing, we remove direct identifiers where present (e.g., user handles, URLs, email-like strings) and retain only the minimum metadata required for analysis. For social-platform sources, we share only rehydration identifiers (where permitted) and do not redistribute raw text, in accordance with platform terms and privacy constraints.

Dataset acquisition

We adopt a mixed-access, multi-source design to capture diverse expressions of solar-energy adoption attitudes across short-form discussion, threaded debate, review-style consumer feedback, and open web commentary. Data were collected for the period 01 Sep 2022–31 Dec 2023. Where available, timestamps were retained and normalized to UTC. To standardize retrieval across sources and support reproducibility, we used the same exact keyword queries: “solar energy adoption”, “green energy hesitancy”, and “solar energy cost”.

• Twitter/X (credentialed API) public posts were collected via the official X API using Tweepy (X API docs; Tweepy docs). Access requires developer credentials and is governed by X developer policies.

• Reddit (credentialed API) threaded discussions were collected via the official Reddit API using PRAW (Reddit API docs; PRAW docs; Reddit Data API Terms). Access requires OAuth and is governed by Reddit API terms.

• Review-style opinions (fully open) to ensure an openly downloadable review component, we use the Yelp Open Dataset (Yelp Open Dataset). We retain only records relevant to solar adoption by filtering businesses/reviews using the same keyword protocol and solar-related categories (e.g., solar installation/home services where available).

• Open web commentary (fully open) we construct an auxiliary corpus from Common Crawl using the same keyword protocol (Common Crawl; AWS Open Data Registry entry).

• Open news/context (fully open, optional) we optionally retrieve a complementary context stream from GDELT to support replication using only open data (GDELT; GDELT data access).

We explicitly assess and mitigate: (i) platform bias via multi-source sampling and per-source reporting; (ii) temporal drift via time-aware splits and windowed sensitivity checks; and (iii) annotation bias via adjudication on a held-out subset and targeted error analysis on disagreement cases. To reduce redundancy from reposts, templated reviews, and repeated quotations, we remove exact duplicates (identical normalized text) and near-duplicates identified via similarity screening on normalized text (e.g., shingling/MinHash), retaining one representative instance per near-duplicate cluster. Some sources are publicly viewable but programmatic collection requires official API credentials and adherence to platform terms (e.g., Twitter/X and Reddit); we therefore share only permitted identifiers/scripts for rehydration and do not redistribute raw text. We formulate sentiment classification as a binary task. For implementation, labels are encoded as: Class 0 = Negative and Class 1 = Positive. After collection, we removed ambiguous samples and retained only clearly negative and clearly positive instances to form a binary corpus for modeling. An initial harvest produced approximately 50,000 raw records across sources. After strict relevance filtering (must match at least one exact keyword query), deduplication, and quality checks, the final dataset contained 22,000 entries. Table 2 summarizes the sources and access mechanisms. Exact keyword queries used (applied consistently across source represented in Table 3

Table 2.

Data sources, access mechanisms, and sample size (after cleaning).

Source	Data type	Size	Access	Dataset/API link
Twitter/X	Public posts	10,000	Credentialed API	X API (docs); Tweepy
Reddit	Threaded discussions	4000	Credentialed API	Reddit API (docs); PRAW; Data API Terms
Yelp (Open)	Consumer reviews	5000	Open download	Yelp Open Dataset
Common Crawl	Open web text	3000	Open download	Common Crawl; AWS registry entry
GDELT (optional)	News/context stream	–	Open download	GDELT; Data access

Table 3.

Exact keyword queries used (applied consistently across sources).

Source	Exact query strings (as executed)
Twitter/X, Reddit, Yelp (Open), Common Crawl, GDELT (optional)	“Solar energy adoption”, “green energy hesitancy”, “solar energy cost”

Data preprocessing

One of the major approaches adopted in this methodology was data preprocessing which involved thorough cleansing, normalization and data formatting of the textual data for the enhancement of the sentiment analysis task as shown in Table 4. Every stage of the preprocessing pipeline was intended to remove noise that, otherwise, can interfere with the analysis—symbols, links, and other margins to the text that do not add to the understanding of the data. The idea was to clear all the other unnecessary features so that the end result for sentiment analysis of solar energy concerns only the content related to it. Moreover, the preprocessing pipeline has eliminated the problem of duplication, removing unnecessary information and variations in text format, particularly helpful while trying to harmonize various datasets. Preprocessing steps that were used included lowercase conversion and lemmatization, all of which were used to render similar expressions in a uniform manner. This way, the model was able to handle variations of the same word as the same enabling it to be able to identify patterns. To achieve this, the following steps were followed alongside the deletion of low actionable stop words; By the end of this stage, therefore, the dataset has become more refined and standardized to feed the sentiment analysis model as well as other downstream analyses while enhancing the effectiveness of the model to capture clear trends of consumer reluctance in the shift to solar energy. We treat Yelp as experience-report evidence (customer–installer interactions, perceived costs, reliability, and service quality), rather than as a representative sample of the full population. We therefore report results by source and acknowledge that Yelp is biased toward regions and businesses covered by the platform.

1. Lowercasing Applying this pre-processing technique removes capitalization inconsistencies where differences in the written text can affect the models’ operation as they now all lower case.

2. Noise removal Besides, features like URLs, specific symbols, emojis, and references to other users were excluded as they are rather unhelpful and distracting.

3. Stop word removal Low information sentence words that also tend to have extremely low semantic value (prepositions, conjunctions, articles) such as “the” and “and” were removed using NLTK’s stop word list. This step brings benefits for the model by guiding it to concentrate on the terms that are most influential with regard to sentiment determination, which now stands as the critical goal of analysis.

4. Lemmatization and stemming Words were standardized to their root forms to reduce variations of the same term (e.g., “adopting” becomes “adopt”), helping unify terms and improve interpretability by treating different forms of the same word as a single entity.

5. Bigram and trigram generation Bigrams and trigrams were created to capture important multi-word expressions such as “solar energy” and “solar cost,” which are critical in solar energy discussions. This contextualizes terms and enables more nuanced sentiment analysis.

To confirm that lemmatization was applied correctly, we performed a spot-check on randomly sampled entries before and after lemmatization and verified that common inflections were mapped to their base forms (e.g., “adopting” “adopt”) while domain-specific terms and proper nouns were preserved. In Table 4, the reported “Words” correspond to the total token count remaining after each preprocessing step (i.e., corpus-wide tokens), not the number of unique word types (vocabulary size). Accordingly, the lemmatization stage shows a modest reduction (115,000 to 110,000; 4.3%) because (i) lemmatization primarily maps inflected forms to a common base form without necessarily removing tokens, and (ii) after stop-word removal, a large share of remaining tokens in consumer text are already in base forms or are proper nouns/domain terms that are unchanged by lemmatization.

Table 4.

Impact of preprocessing on the corpus token count (“Words” = total tokens).

Preprocessing step	Before (Words)	After (Words)	Reduction (%)
Raw data	200,000	200,000	0
Lowercasing	200,000	200,000	0
Noise removal	200,000	175,000	12.5
Stop word removal	175,000	115,000	34.3
Lemmatization	115,000	110,000	4.3
Bigrams/Trigrams	110,000	107,000	2.7

The preprocessing pipeline took the data through a process that removed noise, thereby enriching on the dataset being fed into the model thus helping the model to capture the linguistic patterns well.

Feature extraction

In this paper, both feature extraction techniques including Term Frequency-Inverse Document Frequency (TF-IDF) and BERT embeddings were used to enable the model to catch both the word significance and the contextual significance of the data. Co-occurrence analysis led to the identification of important terms and phrases, whereas, TF-IDF helped in determining the weightage of each word, to help analyse word statistical significance in context with the entire document collection. This allowed the model to concentrate on words that are unique to certain sentiments or themes allowing it to gain insight into word relevancy in the context of solar energy as shown in Table 5.

Table 5.

Sample Top Terms with TF-IDF Scores.

Term	Frequency	TF	TF-IDF Score
cost	1,000	0.15	0.120
reliability	800	0.12	0.095
environmental	700	0.10	0.088
policy	600	0.09	0.085
payback	500	0.08	0.070

As shown, we fuse sparse TF–IDF with dense BERT embeddings by direct concatenation. Specifically, TF–IDF yields a sparse document vector , where denotes the dimensionality of the TF–IDF feature space (i.e., the vocabulary size produced by the TF–IDF representation after preprocessing, including the bigram/trigram phrase construction described. In parallel, BERT produces a sentence-level embedding , where we use the [CLS]-pooled representation with . No dimensionality reduction step is applied prior to fusion in the current formulation; the hybrid representation is formed as . We do not introduce an explicit manual weighting coefficient between TF–IDF and BERT blocks. Instead, the subsequent linear head learns the relative contribution of each feature dimension through the trainable parameters in W (i.e., ), allowing the model to adaptively combine lexical salience (TF–IDF) and contextual semantics (BERT) during training.

On the other hand BERT embeddings, and as the name suggests involved a further layer of context awareness. Compared to straight frequency, BERT embeddings explain the contextual dependency between words, derived from positions and dependencies between words in the sequential read of sentences, which helps the model understand more about meanings at play. As for the choice of TF-IDF and BERT, both approaches prove a valuable feature in insuring maximal statistical relevance while at the same time providing ample contextual information into the models forming the dataset. We fuse sparse TF–IDF with dense BERT embeddings to couple lexical salience with context-sensitive semantics. TF–IDF elevates domain terms that characterize solar-adoption discourse (e.g., “policy,” “subsidy,” “payback period”), while BERT captures polysemy and long-range dependencies. Formally, we concatenate with the [CLS]-pooled BERT vector and train a linear head: . This hybrid improves separability and interpretability. Combined, these features endowed the model with the targeted Term Frequency-Inverse Document Frequency (TF-IDF) and Being BERT to enable it capture the diverse consumer attitude towards solar energy accurately in Eq.1.

TF-IDF formula:

1

where represents the term, is the document, is the total number of documents, and is the document frequency of . TF in the case of IDF stands for term frequency, that means that it is easier to find important words because those that have high frequency within a document but low frequency in the database receive higher scores.

BERT embeddings: Contextual understanding was done using BERT embeddings, which gave each sentence a 768 dimensional vector. Unlike what we have seen previously, such relationships disclose locations within a sentence of words and allow BERT embeddings to capture and analyze sentiment in addition to allowing the model to dissect consumer opinions.

Model architecture for sentiment analysis

The details of the model structure are borrowed from BERT, a transformer-based tokenization model reputed for its excellence in capturing complex contextual dependencies in the language. In particular, our BERT was fine-tuned for binary (two-class) sentiment classification, with labels encoded as Class 0 = Negative and Class 1 = Positive. When fine-tuning BERT in this task, the model adapts the general knowledge it has learned when being pre-trained to the patterns of language in discussions about RE which includes technical language, different stance, and varying types of sentiment as shown in Fig. 1. We employed multiple transformer layers and self attentions in the model so as to effectively capture such low level language patterns important in discriminating between sentiments. Versatility of the BERT in building the bidirectional context that bounds from the former word to the next one is the core reason why the model is capable of well understanding the sentiment implied in the entire given sentence instead of just the targeted word. This capability is useful when determining the sentiment of a text as it allows the model to find not only obvious sentiment reflected in a text but also opinions embedded in an opinion, such as sarcasm or conditionality. From BERT’s complex structure, our model acquires context-dependent knowledge for which sentiment analysis is crucial for the solar energy industry where public sentiment can be diverse.

Fig. 1.

End-to-end pipeline for solar-energy sentiment analysis. (a) multi-source text (social media, reviews, blogs); (b) normalization, de-duplication, filtering; (c) hybrid features combining TF–IDF with BERT embeddings; (d) supervised classifier with class-weighted loss and probability calibration; (e) outputs include sentiment label and salient drivers. Where applicable, axes show units and sample sizes (N).

• Input layer The sequences are tokenized and passed for inference to the model with [CLS] and [SEP] tokens as the boundary.

• Self-attention mechanism In this method every word in the respective sequence get focus according to the intended focus of the corresponding word of the sentence helps the model to quantify higher emphasis on the key-worded words.

• Transformer layers Contains 12 blocks for self-attention and feed forward networks to extract subtle features of words and better identify sentiment.

• Pooling layer Aggregates information from the last transformer layer to form a sentence-level representation.

• Softmax layer Outputs a probability distribution across sentiment classes, where the class with the highest probability represents the predicted sentiment.

This architecture effectively captures nuanced sentiments, an essential aspect when interpreting varied consumer opinions on solar energy.

Model training and hyperparameter tuning

The model was trained with a learning rate of , and a batch size of 32, using Adam Optimizer, with the loss objective function as the cross-entropy loss, and early stopping used to avoid overfitting and thus provide generalizability to unseen data Eq.2. A learning rate of was chosen to achieve stable convergence: earlier trials in which rates were higher proved to be more unstable and when rates were lower, no significant improvement in accuracy was observed over training time. A batch size of 32 was used because it was the middle ground between computation and gradient stability, and it can well meet the memory requirements and maintain the model performance. These hyperparameters were optimised to a reasonable combination of training time and model performance, and the model is at the potential to perform well as a generalisable solution across diverse solar energy corpora.

Cross-entropy loss:

2

where is the true label and is the probability predicted to class , is the number of samples and the number of classes. This loss was selected to optimize the model for binary sentiment classification (with classes).

Table 6 summarizes the learning trajectory of the proposed classifier across representative checkpoints on the training split (Ep1, Ep3, Ep5, Ep7, and Ep10). Overall, the model exhibits stable and monotonic improvements, with training accuracy increasing from 0.68 at Ep1 to 0.88 by Ep10, accompanied by consistent gains in precision (0.67 0.87), recall (0.65 0.86), and F1-score (0.66 0.87). To avoid optimistic bias and prevent any test-set leakage, the final model selection is governed exclusively by the validation split: the Best Val. column reports the best validation metric observed during training (e.g., validation F1 = 0.85), and no thresholds or hyperparameters are tuned using test labels. This protocol ensures that the reported test-set results reflect genuine generalization rather than post-hoc optimization.

Table 6.

Performance metrics over epochs on the training split. The Validation column reports the best validation-split metric observed during training (selected using the validation split only).

Metric	Ep1	Ep3	Ep5	Ep7	Ep10	Best Val.
Accuracy	0.68	0.77	0.83	0.86	0.88	0.85
Precision	0.67	0.75	0.81	0.85	0.87	0.84
Recall	0.65	0.73	0.82	0.85	0.86	0.84
F1-score	0.66	0.74	0.82	0.86	0.87	0.85

Model evaluation metrics

Similar to most supervised learning models, the performance of the proposed classifier was evaluated using accuracy, precision, recall, and the F1-score, which provides a balanced summary of classification quality. To avoid ambiguity, we report results separately for training/validation and for the strictly held-out test set.Across the strictly held-out test set, the model maintains balanced precision and recall, resulting in an overall F1-score of 0.82. This indicates stable binary sentiment discrimination across heterogeneous sources without overfitting to one polarity in Table 7.

Table 7.

Final model performance on the strictly held-out test set (binary sentiment: Class 0 = Negative, Class 1 = Positive).

Metric	Positive	Negative	Overall
Precision	0.82	0.84	0.82
Recall	0.83	0.83	0.82
F1-score	0.82	0.83	0.82
Accuracy	–	–	0.84

Results from sentiment trend analysis added a second dimension, which depicted the trend of sentiment over time. This method was useful in observing how consumers’ perception changing due to some outside forces such as policy changes, new technologies or market changes. And analyzing these trends, it becomes possible to determine the periods of increased concern or, conversely, increased optimism, and, therefore, it is possible to link these changes with certain factors. This temporal perspective is also fundamental to comprehend the processes into play of the public opinion and to distinguish moments in which some interventions might be more successful. Looking at the results from the theme analysis, it was possible to have a more general approach because the work was divided by primary themes capable of providing a better understanding of the main topics most bothering the consumers. Dividing sentiments by the themes such as cost, environment, and reliability made it easier through the theme analysis to identify areas where consumers seemed apprehensive.Together, these visualization techniques provided a multi-dimensional perspective on consumer sentiment, making it easier for decision-makers to interpret the data comprehensively as shown in Table 8. This holistic approach to visualization not only clarifies the underlying causes of consumer hesitancy but also equips stakeholders with actionable insights to address these concerns effectively.

Table 8.

Top hesitancy themes and sentiment scores.

Theme	Frequency	Average sentiment score	Significance
Cost	800	− 0.70	High
Reliability	650	− 0.68	High
Policy	450	− 0.55	Medium
Environmental impact	400	− 0.42	Medium
Payback period	350	− 0.30	Low

Results

In this section, the assessment of the model is provided according to numerous parameters, chosen specifically for the context of the solar energy sentiment analysis. Every figure and table in this section offers not only an evaluation of the model but also characteristics of its behavior and performance in various conditions. By operating at both macro- and micro-levels, we hope to establish its capacity to analyse fine-grained sentiment, the model’s performance across a broad spectrum of data, and how it might respond to variations in sentiment patterns. However, the given kind of multiple evaluation will allow to consider such aspects as the overall availability of the model, its accuracy, and sensitivity, which directly defines its indispensability and effectiveness in practice. The accuracy and F1 score are indices of the effectiveness of the model in general, while looking at the effect of threshold tuning, and calibration, and ranking accuracy shows how it can be used in detail. This set of values provides additional insights into the performance of the model as well as it advantages and possibly drawbacks when applied to practical contexts wherein accurate sentiment analysis translation can result in tangible consequences, such as policy formation, market research, and monitoring of public sentiment in solar energy spheres. The findings illustrated here strengthen the model’s utility as a framework for mining twitter discussions of solar energy issues. Thus, the results of this analysis demonstrate that the model’s performance is accurate and interpretable, making it appropriate in high-stakes real-world situations that require accuracy and high precision. In each of the sections below, we discuss the implications of these results, emphasising the extent to which the performance of the proposed model fits practical requirements of solar energy sentiment analysis, and presenting a clear roadmap to the deployment of the model for decision-making purposes.

Figure 2 reports per-class precision, recall, and F1-score on the held-out test set for the binary sentiment classes (Negative, Positive). Performance is broadly consistent across classes, indicating that the classifier does not collapse to a single dominant label and that it can capture nuanced attitudes relevant to solar adoption discourse (e.g., cost/ROI concerns, reliability doubts, and policy frustration).

Fig. 2.

Per-class performance on the strictly held-out test set for binary sentiment classification (Class 0: Negative, Class 1: Positive). Bars report Accuracy, Precision, Recall, and F1-score for each class.

We report threshold-swept performance using ROC and Precision–Recall (PR) curves Fig. 3. ROC summarizes the trade-off between true-positive rate and false-positive rate, whereas PR emphasizes precision as recall increases and is especially informative under class imbalance. To avoid optimistic bias, we (i) removed near-duplicate texts prior to splitting and (ii) evaluated all curves on a strictly held-out test set that was not used for model selection or threshold tuning. ROC-AUC values are rounded to two decimals; we compute ROC/PR curves strictly on the held-out test split with no overlap in documents across train/validation/test.

Fig. 3.

Threshold-swept performance on the strictly held-out test set for binary sentiment classification. (A) ROC curve for the Positive class (one-vs-rest), with ROC-AUC reported. (B) Precision–Recall curve for the Positive class, with PR-AUC reported. Where applicable, uncertainty is estimated via bootstrap 95% CI or mean±std across seeds.

The calibration curve indicates that the model is well-calibrated, as most points lie near the diagonal line, demonstrating that the predicted probabilities are reliable indicators of the true likelihood of each sentiment as shown in Fig. 4. In the case of renewably energy sentiment analysis, calibration turns out to be useful in determining the degree of precision over which the stakeholders can trust the model. The definition of calibration is comparison of the probability of the outcome that occur with the actual outcomes in order to determine whether the probability score, assigned by the model reflects the probability of a certain feeling. Whereas well calibrated model not only provides decent sentiment classification but also provides an indication of the confidence of model regarding such classes. Such reliability matters to stakeholders who are not only interested in the direction alone of the simple sentiment: positive or negative, but also the extent to which the sentiment detected is reliable. The calibrated model allows the decision-makers to convert the probability of sentiment into a quantifiable extent of assurance in the realms of the solar energy sectors in which the decision process carries the likelihood of being subject to the effect of popular opinion.

Fig. 4.

Calibration curve (Positive vs Negative) on the held-out test set. The solid line represents the model’s predicted probability for the Positive class, and the dashed line shows the observed empirical frequency within each probability bin.

The model performs significantly better than random selection and there are significant lifts at notable places. The fact that the model achieves a 50Percent mark indicates its capability to retrieve relevant sentiment instances two times more than that would occur due to random accident, and this is further indicative of the fact that the model has an extraordinary capability to prioritize meaningful data as demonstrated in Fig. 5. The better performance is especially beneficial in sentiment analysis projects in solar energy, where the knowledge of trends in public opinion has a decisive effect on the course of policy making and the solution of community issues. The model allows investigators to explore the most influential sentiments first by correctly identifying high-confidence instances at the beginning of the analysis without having to comb through irrelevant information that may slow or complicate the decision-making. Such ability gives stakeholders in this environment where attitudes are driven by the fast evolution of environmental awareness, technology and policy to have a clear awareness of sentiment patterns which then forms the basis of responding. However, just the most sure cases also transform the model into a dynamic instrument of real-time sentiment analysis. In such a manner, interest groups are capable of easily capturing the appearance of a new pattern and shifts in the mass opinion, and interfering with that in a proper way in order to make use of the flow to introduce an improvement of the mass opinion. E.g. with added positive sentiment detected towards a specific renewable initiative, stakeholder support, interest involvement or by spreading the positives related to that initiative could therefore be propagated. Conversely, when either a sense of environmental effects or an economic consequences start to demonstrate negative sentiments, decision makers can exact a certain urgency and perhaps even in the form of a project strategy revision, or more importantly and more directly, a more direct and effective approach to the members of involved communities via PR. Besides decision making, the model improves the support of the solar energy project by the people because they are implemented in the conditions in which the needs and expectations of the people are met when put into practice.

Fig. 5.

Cumulative gain chart: Cumulative gains achieved by the model compared with a random-selection baseline. The model shows higher gains at 25%, 50%, and 75% of reviewed instances, indicating more effective prioritization of relevant items under limited review capacity.

When it comes to displaying the prediction confidence scores, this will show areas of high certainty. Thus, evidence is found that there are high values of confidence in the model’s high-confidence intervals according to which it is efficient in making predictions where it is confident of its results as shown in Fig. 6. The pattern is vital in making decisions as only the most precise data are accepted that reinforces the data obtained as a result of the model analysis. In this manner, with high confidence peaks one could use to ensure that the said sentiments which are held on high probability are indeed representative of what the people of a country say. In the RE environment a domain where social acceptability of initiatives and strategies that can be used to influence what the energy supply will comprise of in the future is so critical such accurate forecast of high assurance prediction is a good point of evidence based decision making. Once such anticipated knowledge is surrendered by the analysts and the decision making arms of an organization to design strategies, they can rest assured about doing so since the projections produced by the model are potent useful forecasters of sentimentality.

Fig. 6.

Prediction confidence distribution: A histogram in combination with a KDE plot representing the distribution of prediction confidence scores. Higher points at certain areas are also an indication that the model has high confidence levels thus can be relied on to provide accurate sentiment analysis information.

As an example of monitoring of public opinion, there is a high value in using this approach as the confidence of accuracy is significant and being able to remove and highlight the strongest emotional and clear descriptive positive and negative sentiments will allow the analyst access more detail of general trend. As an example, should the model indicate that at its worst there exists a high certainty at positive sentiment, then, perhaps, there exists a popular opinion in favour of a specific solar energy project that could be utilised in due messages by shareholders. Conversely, in cases where high assurance is born that negative sentiment peaks exist, this would result in early responses whether it is in regards to mobilizing the community or in improving the strategies employed by the company in its environmental communication processes. Once reduced to such very close estimates, the analysts can in any case approximate the sentiment distribution pattern with a lot less difficulty and hence react promptly to any change in the above trends and reformulate their policies to reflect the important preoccupations and demands of the population.

The starting data was of 50,000 entries which included a wide range of data. Nevertheless, these data were described by a high noise level, redundancy, and occasionally irrelevant data points, which might disfavour efficient model learning and generalization. To address this, a filter was implemented in which the entries were reduced to a total of 22,000, an average drop off of 56.0 %. Graphs this loss pointing out how preprocessing of data helps to improve the quality of the data set. The process of this filtering was assigned a number of steps, such as the identification of incomplete entries and their deletion, omission of irrelevant data and their deletion in accordance with the predetermined criteria, and removal of duplicated data leading to the possessing of unique and significant cases that could be used to train the model. This step assists in the elimination of noise and therefore increases the models ability to learn generalizable patterns and consequently, reduces the overfitting. As a matter of fact, filtering forms part of a number of compulsory procedures in data preprocessing because it focuses on the quality of data that is to be used in feeding any models to be generated. In addition to enhancing the performance of the model, it also addresses the choice of the real-world scenarios that analysts need to choose in this field, the clean and relevant data required in them.

The reduction in the quantity of data is the indication of the need to pass intensive data preprocessing. The model will therefore be trained on a dataset at no noise with minimal redundancy easily digestible with relevance as noise will be added in the inference stage. This is needed to modern range models that work with real data since it would allow them to differentiate between tracks and artifacts or outliers. Consequently, the filtering process has had also a positive effect on the performance of training and has contributed positively to the extension of the application scenarios, as a reliable and definite base of data sets has been supplied. Filtering is a core preprocessing step because it improves data quality by removing incomplete, irrelevant, and duplicated records. As shown in Fig. 7, the corpus size decreases from 50,000 to 22,000 entries (a 56% reduction), yielding a cleaner dataset with reduced redundancy and noise for downstream learning. Rather than reporting unverifiable subclass counts, we assess balance and bias risk at the class level using held-out test performance. Fig. 8 illustrates the evolution of the training of principle figures of merit: accuracy, precision, recall, and F1 score, over 10 iterations. The metrics start out as moderate but as training progresses on the model inputs all of them tend to increase. By epoch ten the model reached good results in all the metrics: By epoch ten, the proposed classifier reached accuracy = 0.88, precision = 0.87, recall = 0.86, and F1 score = 0.87 during training, indicating stable convergence across epochs. To see the convergence of precision and recall is especially significant, since it reflects a balanced performance that is desired in applications when false positive/negative false negatives have important consequences. Because the value of each of the metrics increases monotonously as the training progresses, one may conclude that the model is fitted with those other parameters such as learning rate and batch size accurately chosen. These are specified and increased over epochs due to which the model: not only is learning satisfactorily, but also maintains its fine performance required to balance and classify flawlessly, and thus, is best applicable to applications that would not involve unusual unbalanced augmentations of either category.

Fig. 7.

Dataset Size before and after filtering: This figure depicts reduction in the dataset from 50,000 to 22,000 entries, which reduced by 56.0% as seen below. This great feat was realized through rigorous elimination of substandard and astringent records which was in the form of incomplete entries, irrelevant and duplicated data set. To recap, noisy data has been minimized, meaning that the model’s tendency to make these errors will be minimized as well thus providing the backbone of learning and prediction.

Fig. 8.

Model performance over epochs: The following figure presents results obtained with 10 epochs in terms of accuracy, precision, recall and F1 score of the model. All increase gradually which displays adequate learning and stable model. In the last epoch, the performance of the model is excellent on all measured indicators, which speaks to its capacity for balanced classification, which in practice is essential when deciding on positively and negatively charged classes.

The improvement has been embraced in each of the eras with its aptitude to learn and be trained on the collected data with finesse. Such steady rise in performance metrics is characteristic of efficient training process in the models as well as boosts the reliability and generality of model. Averages of high-precision, high recall, high accuracy, and F1 score almost up to the maximum value indicate that the model converges stably and attains strong performance across epochs. Idle tuning are very useful in cases when the precision and recall need to be balanced and when sensitivity of occurrence false positive and false negative needs to be emphasized (as in my case scenario it is important to focus on specific sensitivity within specific parameter in Fig. 9. This is achieved by comparing the model to be more conservative in the sense that it would use fewer cases than what is being offered by just employing higher thresholds. However, to achieve this, compromises must be made by sacrificing the number of the genuine positives that should remain as it is. Conversely, high thresholds produce a lot of precision and enhance the level of failure in retrieval of the relevant documents hence, low recall. Threshold selection was performed using the validation split only. We report threshold-swept curves on the held-out test set, but the operating threshold used for any point metrics was chosen without using test labels, to avoid optimistic bias.

Fig. 9.

Precision, recall, and F1 score by threshold: This figure shows how precision, recall and F1 score changes with the increase in the threshold. Higher t must be chosen to minimize false positives improving precision and lower t must be chosen to find more positive instances, increasing recall. An optimal threshold of 0.8 highlighted good measures of precision (0.94) and recall (0.92) and therefore the F1 score enhancement factor.

To give an example, the specified threshold tuning analysis indicates that the versatility should be the main strength of the model in connection to the specific needs of the application under consideration. The model similarly obtains a reasonable threshold of precision and recall when optimum threshold (0.8) is utilised and does not induce any unwarranted or missed elite end. Such a balance is very important where precision is a necessity to match with the possible outcome of a miss or misrepresentation, or any business (high rates of false positives are operationally expensive). Threshold tuning thus guarantees that the model lifetime can achieve these benchmarks and has the applicability and precision in addressing the various types of issues encountered on each application area. Together these figures complement the paper with various but certainly interrelated aspects of training and testing the model that overall puts an argument in support of the practical application of the model to multiple real world cases. The dataset-filtering process leaves a significant indication that in respect to the quality of information, one should do the utmost because it enhances data learning within the models. The noise, the clutter and the availability of irrelevant material and unwanted multiplicity of same components can be helpful in setting the right foundation of suitable model since the mind of the trainers will not lose track of focus during training. By doing so, only appreciated trends are taken into account when formulating a model that eventually leads to the possibility of augmenting the accuracy and attaining generality. The most appropriate thing to do to understand more on how the model is functioning is examining the class distribution of the inputs.

The class imbalance reduces the danger that tends to enable the model to give high probabilities to the majority class, while ignoring the minority class. Applying balancing techniques ensures that real data distribution is simulated in such a way that the model can perform well on different datasets and thus is very relevant especially when used in social areas such as legal, financial systems where bias is not permissible. During learning it is easy to record values of the performance measure averaged over the number of training epochs and analyzing what is obtained will give insights on how the model is learning and whether it is stable. The activities have leading trends and the exactness of the information, the sharpness, the effect and F1-Measure refresh the competence of the model to advance each time with the information. The matched values of precision and recall also point to the model’s equal-risks, moderate classification, which makes the model suitable for deployments where both false positives and false negatives have significant consequences. Lastly, threshold tuning is a significant component of the solution that allows for tuning the model to application level requirements. In this way, by selecting the appropriate threshold, we get the required compromise between precision and recall to provide optimal performance for cases where costs of misclassification are different. This flexibility is needed for practical applicability; with it, the model can be adjusted to better fit the requirements of the target objective and, thereby, increase both efficiency and applicability in real-world scenarios.

Finally, the current study would seem to suggest that each stage regarding model development methodology explained above is crucial and cannot be overlooked—be it all the possible data filtering and class balancing, or the performance metrics tracking, or, in fact, the threshold tuning. Combined, these steps make a machine learning model that is not only is precise and generalizable, but also versatile for nearly every field. Through this study, the presented model effectively depicts sensitivity to high-performance, reliability, and adaptability when trained in a real-world environment with significant accuracy rate and balanced class representation. This combined model of approaches can serve as a guide to building sustainable machine learning models that are not only technically feasible and accurate, but also gain practical real-world relevance coming as a response to unique demands of various fields.

As shown in Fig. 10, we evaluate how well the model supports real-world triage by ranking items according to predicted confidence and measuring early retrieval quality. These are intended to answer a deployment-oriented question: if a stakeholder can only review a limited fraction of items, how much better is the model-driven selection than choosing items at random?. Panel A reports lift across threshold bands, where lift quantifies improvement over random selection (a value of 1.0 corresponds to random performance). The curve increases steadily from 1.0 at the lowest band to approximately 4.0 at the highest band, indicating that the highest-ranked items are substantially enriched for decision-relevant instances. The monotonic growth of lift across bands suggests that the model score is well aligned with the underlying likelihood of correct or actionable predictions, which supports using confidence thresholds for filtering. In practical terms, stakeholders can choose stricter bands to focus on fewer items while gaining a higher concentration of relevant cases, thereby reducing effort and minimizing wasted interventions. Panel B reports Top-K accuracy as K increases, capturing how accuracy behaves when stakeholders inspect the top-ranked items first. Accuracy rises from approximately 0.90 at small K to about 1.00 by , showing that the model’s ranking places the most reliable predictions at the top. This behavior is important for operational workflows such as prioritizing outreach, responding to misconceptions, or triaging negative feedback: early selections are both more accurate and more useful, and increasing K offers a controllable trade-off between coverage and reliability. Taken together, the lift and Top-K results demonstrate that confidence-aware ranking turns model scores into actionable prioritization, enabling stakeholders to allocate limited resources to the cases most likely to benefit from timely action. These results indicate that the proposed sentiment pipeline can support practical solar-energy workflows such as monitoring public response to policy changes, prioritizing high-impact concerns for outreach, and tracking sentiment shifts over time.

Fig. 10.

Decision-support evaluation of confidence-aware prioritization for the proposed model. Panel (A) reports lift by threshold band when ranking instances by model score compared with random selection (baseline lift = 1.0). Panel (B) reports Top-K accuracy as K increases, illustrating how accuracy improves as more high-ranked instances are included.

It is also needed to say that It is pointed out that the Top-K Accuracy and Lift Curve refer to two facets of model prediction, namely: the accuracy of the model in experimental validation and the capability of the model to rank the objects appropriately. The Top-K Accuracy measure indicate how the accuracy improves by increasing K and adding greater elasticity to ranking of the predictions. Higher K values equivalently imply that model can distinguish the appropriate patterns in the data to be operative in a situation where potential to identify the appropriate label among the multitude thereof, suffices. Conversely, the Lift Curve accidentally, reflects the power of the model of ranking the relevant cases against haphazardly chosen set. The values observed using early instance counts endorse a superior performance of the model as compared to opportunity level in the ranking of the important instances to the high lift values of the model predictions.

This would allow to raise user satisfaction levels by achieving a higher likelihood of relevant recommendations to be present in the top selection of the user. The Lift Curve reveals the model with priorities to instances that need to be focused on to identify the anomalies as early as possible. Moreover, high lift values are evidence of the fact that the model gives priority to probable diagnoses that might be crucial to timely intervention measures. Lift is of special concern in conducting targeted marketing. A high lift is at a lower count, indicating that the model is able to rank the most promising leads and maximize ROI as the marketing team is focusing on a small population that has higher probability to convert. The analysis establishes the resilience of the model when dealing with classification by assimilating both precision of the most recommended cases as well as efficacy of interest in high-value cases. Its Top-K Accuracy is high, implying that it will also be able to identify the correct value at the top of a wider distribution of outputs, and its Lift Curve also indicates its favourable rank of important occurrences early on, thus very flexibly useful in a wide range of contexts. This study analyzes publicly available, user-generated text and does not involve interaction with individuals or collection of private data. We do not release raw third-party text. Where platform policies permit, we share only content identifiers (e.g., tweet IDs, Reddit item IDs) and scripts for rehydration; otherwise, we share URLs or non-reversible derived artifacts (e.g., aggregate statistics and embeddings) sufficient to reproduce the reported experiments. During preprocessing, we remove usernames/mentions, URLs, and other direct identifiers and report results only in aggregate.

Discussion

This study adds a powerful approach that is employed to understand consumer skepticism in RE uptake due to advanced AI-based opinion mining. The given methodology based on a fine-tuned BERT model and the inclusion of a feature combination such as the employment of both TF-IDF and BERT embeddings enables a more advanced sentiment analysis which in the case of a simple lexicon-based or a pure machine learning student would not be possible. The current discussion assesses the findings on the study, the location of the research in the field of other past studies, the contribution of the research to the stakeholders, comparison of performance on the various grades of sentiment, as well as potential research directions of further studies. The proposed TF–IDF + BERT pipeline improves robustness over lexicon-based and simpler ML baselines by combining domain-relevant lexical salience with contextual embeddings. On a strictly held-out test set, the model achieves balanced binary sentiment performance (Negative vs. Positive) and supports decision-oriented analysis via calibration, cumulative gains, and high-confidence filtering. Because cross-paper results are not directly comparable across heterogeneous datasets and label schemes, we avoid claiming absolute superiority based Table 9. Our hybrid TF–IDF + BERT classifier achieves balanced performance on the strictly held-out test set, with overall Accuracy = 0.84, Precision = 0.82, Recall = 0.82, and F1-score = 0.82.

Table 9.

Comparison with recent (last 5 years) sentiment studies using metrics explicitly reported in the cited papers.

Study	Domain/Dataset	Model	Acc.	Prec.	Recall	F1	ROC-AUC	PR-AUC	Notes/Limitations
58	Turkish financial Twitter (binary; neutral removed)	SVM (TF–IDF)	0.89	NR	NR	NR	0.8729	0.9415	Reports P/R/F1 per-class (neg/pos); dataset is Turkish finance tweets.
59	Twitter sentiment (binomial/polynomial settings)	SVM (10-fold CV, binomial)	0.8642	NR	NR	NR	0.9320	NR	Reports accuracy and AUC; no PR-AUC; different experimental protocol.
60	Bangla sentiment (binary)	Bangla-BERT + CNN-BiLSTM	0.9415	0.9423	0.9294	0.9304	0.9473	NR	ROC-AUC reported; PR-AUC not reported.
61	Multilingual restaurant reviews (binary sentiment)	XLM-RSA (best LR/BS config)	0.9230	0.9150	0.9200	0.9170	0.9630	0.9310	Reports all metrics (Acc/Prec/Rec/F1/ROC-AUC/PR-AUC) under tuned hyperparameters.
62	IMDb sentiment (binary)	RoBERTa-base (5-fold CV mean)	0.9413	0.9377	0.9459	0.9417	0.9851	0.9844	Mean±std over 5-fold CV; benchmark setting differs from domain-specific corpora.
Ours	Solar hesitancy corpus (binary)	BERT + TF–IDF (ours)	0.84	0.82	0.82	0.82	1.00	0.93	Strictly held-out test set metrics from Table 7; ROC-AUC/PR-AUC values taken from the Fig. 3 legend (Positive vs. Negative).

NR = not reported in the paper

Results are not directly comparable across different datasets, languages, and label schemes.

In addition to aggregate test metrics, we evaluate decision-support behavior using confidence-aware prioritization. Figure reports lift by threshold band and Top-K accuracy as K increases, showing that ranking instances by model score prioritizes higher-quality predictions compared with random selection. This supports practical use cases where stakeholders review only a limited number of highest-confidence items.

As shown in Fig. 11, we compare the proposed approach against a broad set of representative baselines spanning lexicon-based methods, classical machine learning models, and transformer-based variants. This comparison is intended to contextualize the proposed hybrid design within the existing methodological landscape and to demonstrate that performance gains are not limited to a single metric or a narrow subset of competing approaches. Panel A summarizes accuracy across models. Panel B reports precision, highlighting the reliability of positive predictions under cross-model comparison. The proposed model attains the highest overall accuracy among the compared approaches, exceeding traditional lexicon-based and keyword-driven systems as well as several classical learning baselines.

Fig. 11.

Comparative performance across sentiment models on the solar-adoption corpus. Panel (A) reports accuracy for a range of baseline approaches and the proposed model. Panel (B) reports precision for the same set of models, highlighting reliability of positive predictions under cross-model comparison.

As shown in Fig. 12, we evaluate the proposed method against a diverse set of baselines using recall and F1, which together characterize coverage and balanced classification quality. Recall is critical for measuring how effectively a model retrieves relevant sentiment instances (i.e., how many true cases it captures), while F1 summarizes performance when both missed detections and false alarms matter, making it a standard metric for comparing methods under class imbalance and heterogeneous text sources. Panel A reports recall across models. The proposed approach achieves the highest recall, indicating that it captures a larger fraction of relevant sentiment cases than competing approaches. This is particularly important in the solar-adoption setting where missing strongly negative or strongly positive feedback can lead to incomplete characterization of consumer barriers or missed opportunities for timely intervention. The improved recall is consistent with the hybrid representation: lexical salience helps identify domain-specific terms linked to adoption concerns, while contextual embeddings help interpret sentiment when consumers express hesitation indirectly, use sarcasm, or rely on multi-clause reasoning. Panel B reports F1 score and shows that the proposed method also attains the strongest overall balance between precision and recall. This indicates that the recall gains do not come at the expense of excessive false positives; instead, the model improves coverage while maintaining reliable predictions. From a deployment perspective, this balanced improvement supports both monitoring and action: stakeholders can detect a larger share of meaningful sentiment instances and still trust that flagged cases are sufficiently accurate for prioritization and downstream decision-making. Together, the recall and F1 results reinforce that the proposed approach offers consistent advantages across complementary metrics, strengthening its suitability for multi-source, decision-oriented sentiment analysis in the solar adoption context.

Fig. 12.

Cross-model comparison on the solar-adoption corpus using recall and F1 score. Panel (A) reports recall across baseline sentiment models and the proposed approach. Panel (B) reports F1 score, summarizing the balance between precision and recall under the same comparison setting.

On an analytical level, the high scores of all four scales mean that the proposed model can easily be applied in high risk sectors, risk assessment and sentiment analysis. In applications for example an enhanced precision will increase the possibility of accuracy, and a high recall will reduce the chance of false treatments.In finance, the balanced F1 score helps detect accurately without overwhelming the system with false alerts, enhancing trust and resource allocation. The model’s adaptability also makes it suitable for large-scale customer sentiment analysis, where capturing the true sentiment with both high precision and recall enables businesses to make data-driven decisions. Overall, the proposed model demonstrates a holistic improvement over traditional and modern models alike, offering a robust and reliable solution where accuracy, precision, and comprehensive detection are paramount. This positions the proposed model as an ideal candidate for mission-critical applications, providing a reliable, precise, and versatile approach to data-driven challenges across various industries.

Implications for solar energy stakeholders

The results can be valuable information to the policymakers, the solar energy industry, and those organizations involved in the solar energy sector that are advocacy-based. Identifying specific consumer concerns such as cost, environmental impacts, and policy reliability will guide the stakeholders to shape up firm strategies that comply with the public opinion to facilitate utilization of the renewable technologies. It means that sentiment information can be trusted by the essential decision-makers because the model has reached a significant amount of precision and recall, and, therefore, it would be simpler to develop effective key messages, which directly answer communities concerns and establish positive relations between gatekeepers and the population.

In pragmatic terms, the high Confidence/Accuracy of the sentiment estimation that the model can generate has quantifiable strategic value. As an example, case studies will assist industry leaders to concentrate on consumer engagement activities hence using the relevant sentiment data collected in the procedures. In case the model identifies strongly positive lines of public attitude to some renewable commitment, the policy makers are allowed to place active communication and guarantee that all stakeholders get such a message in theirquential sphere. Conversely, the identification of dread factors (Supplementary material 2), particularly policy or cost, could prompt positive response allowing in-advance response to consumer reluctance that could limit adoption rates. The format of the model used to be flexible to different sources of sentiments (e.g., social media, forums, review sites), which adds to its utility as it gives a complete picture of the opinion of society on diverse groupings and in different geographies. This flexibility holds the advantage that the approach will allow stakeholders to track the developments in sentiment in near real time, and adapt the planned strategies accordingly, according to the emerging public sentiments. As an example, a change in policy or a new incentive provided by the government may be measured using sentiment patterns, which would offer the possibility of adapting outreach activities in real-time.

Future research directions

Although this research brings about a great enlightenment to the sentiment analysis in solar energy, it can be improved further. Most prospective work would be to consider ways to optimize the computational efficiency of the BERT embeddings to achieve real-time sentiment monitoring, which may be combined with other NLP methods, including sentiment transfer learning to accommodate new themes in the context of the solar energy discussion. Also, given that sentiment on text data may be scarce in some areas, it may be important to have multi-modal data (e.g., visual sentiment analysis or image-based data) to help to gain better insights into consumer sentiment. Furthermore, by broadening the dataset to not only include data of regions which we underrepresent in research of solar energy, we might receive a more global outlook on the feeling of the people. Unlike ideal scenes of negative sentiment perceived against solar energy as the primary energy source, the regional differences in sentiment might help in space-specific policies being more effective than the generalities uncovered in the study.

Conclusion

This study addressed the need for a domain-adapted, multi-source, decision-oriented analysis of consumer hesitancy toward solar adoption using modern NLP. Using a filtered dataset of 22,000 entries and a pipeline designed for transparency and reproducibility across sources and time windows, we examined the main drivers of hesitancy, tested whether a hybrid TF–IDF + BERT representation improves classification quality and interpretability, and evaluated whether confidence-aware prioritization (e.g., cumulative gains, lift, and Top-K accuracy) increases the practical value of model outputs for real-world stakeholders. First, the hybrid feature extraction objective is supported by combining sparse lexical salience (TF–IDF) with dense contextual semantics (BERT), which enables strong overall sentiment classification while retaining interpretable lexical cues that help explain predictions and relate them to concrete hesitancy drivers. Second, the domain-specific sentiment modeling objective is supported by adapting the transformer pipeline to solar-adoption vocabulary and concerns, improving sensitivity to sector-specific language and enabling more faithful classification of positive and negative sentiments across heterogeneous sources.

Third, the high-confidence decision-support objective is supported by confidence-aware filtering and ranking, which improve early precision, Top-K accuracy, and lift relative to unranked sampling. This makes the system suitable for triaging scarce outreach and policy resources by prioritizing the most decision-relevant cases for timely intervention, rather than treating all instances as equally actionable. Fourth, the visualization objective is supported by multi-dimensional summaries that improve interpretability for non-technical users by highlighting dominant themes, tracking shifts in sentiment over time, and enabling comparison of patterns across sources and periods. Together, these elements bridge the gap between high-performing sentiment models and stakeholder needs for transparency, prioritization, and communication-ready insights. With respect to the substantive insights, cost concerns, reliability doubts, and environmental skepticism consistently emerge as dominant barriers to adoption, with stable ordering across platforms and time windows. This multi-source consistency strengthens external validity and provides actionable guidance for policymakers and installers, enabling targeted messaging, incentive design, and focused response strategies tied to the specific objections expressed by consumers. For example, cost-related negativity suggests emphasizing financing options, transparent ROI calculators, and incentive clarity, while reliability-related concerns motivate communication around warranties, maintenance expectations, and performance in adverse conditions. Environmental skepticism highlights the need for evidence-based messaging on lifecycle impact, panel recycling, and grid-level benefits. Because evaluation and trend summaries can be reported by source and period, agencies can monitor how sentiment shifts after policy changes, rate adjustments, or major news events and adapt programs accordingly.

From an operational perspective, the confidence-aware ranking tools support real-world deployment where attention is limited: installers can prioritize the highest-impact discussions or reviews, public agencies can triage the most urgent complaints or misconceptions, and outreach teams can focus on instances most likely to benefit from timely clarification. The same workflow can also support short-cycle A/B testing of public messaging by comparing changes in early precision, lift, and ranked gains over defined time windows, thereby turning sentiment analysis into an iterative measurement tool rather than a one-off report. Overall, the study contributes a solar-adoption, domain-adapted sentiment framework that combines hybrid representation learning with decision-support and stakeholder-facing visualization. Results reflect the languages, platforms, and period sampled; extending to additional encoders (e.g., RoBERTa, XLNet, DeBERTa), adding multilingual coverage, incorporating human-in-the-loop labeling, and conducting richer uncertainty and fairness analyses are promising directions for improving robustness under distribution shift. Future work may also explore domain-adaptive pretraining using sector-specific corpora and integrate causal or quasi-experimental designs to better connect observed sentiment shifts to policy interventions. These extensions would strengthen generalization while preserving the practical, decision-focused orientation of the present work.

Author contributions

A.J. and J.Y. conceptualized the study and designed the methodology. A.J. implemented the BERT-based sentiment analysis and performed data preprocessing. J.Y. supervised the research and provided critical revisions to the manuscript. A.R.A. contributed to data collection and analysis of consumer sentiment trends. S.R. prepared Figures and supported the interpretation of results. All authors reviewed and approved the final manuscript.

Funding

This research received no specific funding from any funding agency.

Data availability

We used a mixed-access design: (i) credentialed social-platform data (Twitter/X and Reddit) and (ii) fully open corpora (Yelp Open Dataset, Common Crawl, and optionally GDELT) sampled over 01 Sep 2022–31 Dec 2023. To comply with platform terms and privacy constraints, we do not redistribute raw third-party social-platform text. Instead, we will publicly release the following materials: (1) Identifiers for rehydration (where permitted): tweet IDs and Reddit item IDs, together with scripts for rehydration and preprocessing. (2) Open-corpus subsets and construction scripts: keyword-based subsets (or extraction manifests) derived from the Yelp Open Dataset and Common Crawl, and optional GDELT query/export scripts, sufficient to reproduce the open-data experiments end-to-end. (3) Non-reversible derived artifacts: aggregate label counts, feature statistics, trained-model checkpoints (where allowed), embeddings (non-reversible), and evaluation outputs sufficient to reproduce the reported experiments without redistributing raw text. Twitter/X content was accessed via the official X API using Tweepy (X API documentation). Reddit discussions were accessed via the official Reddit API using PRAW (Reddit API). Fully open corpora were obtained from the Yelp Open Dataset, Common Crawl (see also AWS Open Data Registry entry), and optionally GDELT. We used a mixed-access design: (i) credentialed social-platform APIs (Twitter/X and Reddit) and (ii) fully open corpora (Yelp Open Dataset, Common Crawl, and optionally GDELT). Programmatic access to Twitter/X and Reddit requires developer credentials and compliance with platform terms; therefore we do not redistribute raw social-platform text and will release only permitted identifiers and rehydration scripts.

Declarations

Competing interests

The authors declare no competing interests.

Ethical approval

This study analyzes publicly available or open-download text and does not involve intervention or interaction with human participants. We report results only in aggregate and do not attempt to identify, contact, or profile individuals. During preprocessing, we remove direct identifiers where present (e.g., user handles, URLs, email-like strings) and retain only the minimum metadata required for analysis. For social-platform sources, we share only rehydration identifiers (where permitted) and do not redistribute raw text, in accordance with platform terms and privacy constraints.

Institutional review board statement

Not applicable. All methods were carried out in accordance with relevant guidelines and regulations.