H3-MOSAIC: multimodal generative AI for semantic place detection from high-frequency GPS on H3 grids in mental health geomatics

Abstract

Background

Mental-health geomatics require reliable ways to convert high-frequency GPS trajectories into meaningful place types that support indicators such as homestay, location entropy, and spatial extent of daily activities. Raw coordinates are typically noisy and carry little semantic information. We introduce H3-MOSAIC(H3-based Multimodal OSM-and-Satellite AI for Classification), a multimodal generative framework that fuses OpenStreetMap (OSM) building text and satellite imagery on H3 grids to infer place semantics from high-frequency GPS.

Methods

Raw GPS was smoothed by minute-level speed filtering, then assigned to Level 10 H3 hexagons. Cells were retained if the mean speed was ≤ 1.2 m/s and the cumulative duration was ≥ 15 min, contiguous cells were merged, and home was defined as the cell with the longest dwell from 23:45 to 06:00. We compared text-only OSM classification with image-based and fused approaches across open-source models (DeepSeek, CLIP, LLaVA, Qwen-VL) and proprietary models (GPT-4o-mini, Gemini-2.5-flash-lite). Performance was assessed by accuracy, Cohen’s kappa, precision, recall, F-measure, and confusion matrices. Day level associations between H3 semantic exposures and stress were examined by a random forest model and explainable methods.

Results

Multimodal methods outperformed single-modality baselines. In the 11-class task, accuracies were: CLIP 0.179, LLaVA 0.269, Qwen-VL 0.565, GPT-4o-mini 0.779, and Gemini-2.5-flash-lite 0.790. In the 5-class consolidation, accuracies rose to 0.702 (Qwen-VL), 0.849 (GPT-4o-mini), and 0.858 (Gemini-2.5-flash-lite). Text-only OSM baselines were lower (≈ 0.60–0.68). Across 3,845 hexagons with OSM text, closed-source models agreed on 79% of labels; disagreements concentrated in mixed-use, office, and green classes. Error modes reflected area-dominant versus keyword-triggered reasoning, hybrid-parcel ambiguity, tag sparsity, and symbolic artifacts. Stabilized semantics support more robust computation of homestay, entropy, and activity space and are suitable for privacy-aware, cross-city reuse. In a day-level case study, minutes at Home related to lower stress; Green showed a U-shaped pattern.

Conclusions

H3-MOSAIC provides a scalable, auditable pipeline for semantic place detection from high-frequency GPS. Multimodal fusion markedly improves accuracy and consistency. Proprietary models are most robust on hard classes and open-source models are practical for coarse taxonomies. H3 day level exposures show stress patterns consistent with established mental health pathways, supporting face validity. The framework enables downstream exposure analyses with reduced misclassification and improved interpretability.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12942-025-00423-9.

Background

Mental health conditions contribute substantially to the global burden of disease, motivating new approaches to screen, assess, and diagnose these illnesses outside of the clinic and across daily life [1–3]. Work in smartphone-based digital phenotyping shows that passively sensed mobility (e.g., GPS tracks) and context can capture everyday functioning and underscores the need for reproducible pipelines and clearer clinical interpretation [4]. Measures of spatial behavior such as homestay, location entropy, and activity space have emerged as informative digital phenotypes with translational potential when measured transparently [5]. Despite the potential and continuous evolution for using these solutions as medical decision support tools, this research area needs more work to overcome technology and methodological rigor issues to adopt proposed solutions in real clinical settings [6]. Recent contributions stress linking mobility-derived indicators to clinical outcomes while addressing generalizability and equity [7].

Yet, extracting interpretable place-based measures from high-frequency GPS remains challenging. Continuous location streams are often noisy and incomplete due to signal loss, indoor drift, energy-saving policies, and participant behavior, producing gaps and artifacts that require imputation or filtering [8]. Even when trajectories are clean, raw coordinates are semantically sparse: converting stay points into meaningful “places” typically relies on clustering plus point-of-interest (POI)/OpenStreetMap (OSM) lookups or rule-based labeling, which can be brittle in the presence of hybrid land uses, sparse or inconsistent tags, or short, infrequent visits. Differences in how studies define places and activity spaces lead to divergent exposure estimates and limit comparability [9]. Mixed-methods work underscores that adding contextual interpretation is essential to understand why mobility changes occur [10]. Practical guidance exists for GPS handling in psychological research, but standardized, generalizable pipelines for robust semantic place detection are still lacking [11]. These constraints hinder the validity of core indicators, and they ultimately limit the translational value of GPS phenotypes for mental health.

Advances in AI are already visible across mental health workflows. Generative and conversational agents are being tested as therapy supports and mental-health chatbots, with early reports on safety, user experience, and cognitive behavioral therapy (CBT) aligned tooling [12–14]. Large Language Models (LLMs) are also being explored for clinician decision support and triage-style reasoning, including head-to-head comparisons with experts and alignment evaluations [15, 16], as well as emotion/affect interpretation and daily-level prediction of negative affect in everyday life [17, 18]. Building on this momentum, advances in generative AI provide a path to strengthen semantic inference from mobility data. LLMs and vision–language models (VLMs) offer opportunities for improved inference while requiring transparency and safety in sensitive behavioral contexts [19]. Recent studies describe how multimodal models can integrate POI text with map and satellite imagery to enrich place semantics while maintaining interpretability [20], and broader assessments encourage systematic benchmarking and explicit reporting of error modes to guide adoption [21–23].

To address these gaps, we present H3-MOSAIC (Multimodal OSM-and-Satellite AI for Classification; hereafter H3-MOSAIC), a multimodal generative framework that fuses high-frequency GPS with OSM building attributes and satellite imagery on H3 grids to infer semantic place types. The approach combines text-based reasoning and image-based inference and audits rationale consistency, aligning with evidence that AI can recover clinically relevant patterns from longitudinal location data [24] and with recent progress in generative methods for mental health [25]. We also establish a baseline comparison across open source and proprietary LLMs and VLMs, and we relate H3-derived day-level semantic exposures to stress measured with ecological momentary assessment. Taken together, these advances reduce semantic sparsity, stabilize indicators such as homestay, entropy, and activity space, improve reliability for downstream analyses, clarify model trade-offs through direct comparison, and support comparability and privacy via grid aggregation and a nighttime home rule, enabling deployment across populations and cities.

Three research questions structure this study and guide the subsequent sections: [] (1) how can OpenStreetMap text and satellite imagery be fused on H3 grids to derive reproducible place semantics for high-frequency GPS data []? (2) what are the performance profiles and error patterns across eleven classes and five functional domains when using text only models, image only models, and fused models []? (3) how do day-level semantic place exposures relate to mental health status?

Literature review

Spatial behavior and mental health: evidence from GPS studies

Primary GPS studies consistently link everyday mobility to mental health symptoms and functioning. In serious mental illness, reduced movement radius and increased homestay correlate with negative symptoms and diminished participation [26]. Longitudinal analysis shows that week-to-week changes in sensor-derived mobility accompany or precede shifts in depression and anxiety, with fewer distinct locations and lower diversity indicating worsening mood [27]. Across European cohorts, intra-individual variability in mobility relates to daily affect and behavior, reinforcing the sensitivity of mobility features to mental states [28]. Smartphone mobility features also capture social motivation and day-level behavior in schizophrenia [29, 30]. Studies combining GPS with bio-signals identify real-world “stress hotspots” during older-adult walks, pointing to specific environmental triggers [31, 32]. Beyond clinical samples, exposure to natural environments along daily paths relates to higher positive effects and better well-being in students and broader community samples [33, 34]. Emerging passively sensed phenotypes further connect mobility with avoidance and distress in social anxiety, illustrating condition-specific signatures [35].

Practical challenges: data quality, semantic sparsity, and contextual validity

Turning raw trajectories into clinically interpretable places is difficult in practice. Device and field studies document missingness, indoor attenuation, and irregular sampling that require careful preprocessing even when phones outperform wearables [36–38]. Methodological guidance for psychological GPS analysis underscores drift, gaps, and the sensitivity of stop detection to temporal and spatial thresholds, emphasizing transparent parameterization [39, 40]. When data are incomplete, trajectory imputation can recover continuity but introduces modelling assumptions that warrant validation [41]. Beyond measurement error lies semantic sparsity: OSM/POI tags are inconsistent and city-specific, so inferring “home,” “work,” or “park” is highly parameter-dependent. Classic demonstrations of the uncertain geographic context problem show that activity-space definitions and spatial scales shift exposure estimates and associated health effects [9]. Mixed-methods work further illustrates why context is essential: combining GPS with in-situ prompts or interviews reveals motives and circumstances behind mobility that coordinates alone cannot capture [10]. Finally, sensing is not missing at random, sociodemographic factors often predict data loss and irregularity, risking biased inference if unaddressed [42, 43].

LLMs and VLMs for POI and land-use semantics

Recent geospatial AI (GenAI) research shows that language and vision–language models can improve POI-based and OSM-based semantic inference. Building-function classifiers that augment physical/spatial features with LLM embeddings of OSM tag strings achieve higher F score and better cross-city transfer than heuristic tag parsing, thus mitigating sparsity and heterogeneity in volunteered geographic information (VGI) [44]. Two-stage, reasoning-aware LLM pipelines for geographic text classification—which combines data augmentation with label selection before predictionstabilize decisions under label ambiguity and class imbalance [45]. Agentic LLM systems convert natural-language intents into executable GIS workflows, operationalizing buffers, overlays, and land-use queries on OSM layers for scalable experiments [46]. LLMs are also being used to assemble geospatial datasets at scale by extracting, geocoding, and normalizing events from unstructured text, producing shareable corpora for training and validation [47]. Complementary work embeds POI contexts and fuses map tiles or remote-sensing imagery with textual cues to handle hybrid or tag-scarce areas, improving area-level land-use prediction [48]. Collectively, this literature supports a multimodal strategy that combines OSM attributes, visual context, and explicit spatial features to deliver more reliable and portable place semantics for downstream health applications [49]. In parallel, imagery-only land-cover/use benchmarks underscore limits in heterogeneous urban textures when text anchors are absent [50], and comparative evaluations across geo-reasoning tasks generally find proprietary models slightly more robust while open-source systems approach usability at coarser taxonomies [51].

Data and methodology

Data source and cohort

We used data from a global cohort of patients with mental illness and healthy controls collected by the Division of Digital Psychiatry at Beth Israel Deaconess Medical Center (BIDMC) [52]. The sensing stream includes high-frequency GPS sampled at 1 Hz from 377 participants across multiple countries (e.g., Spain, the United Kingdom, the United States, and South Korea). The archive comprises 8,400 person-day files (one participant per day per file). After basic quality checks, 329 participants remained in the analytic sample, each with a minimum of 2 hours of usable GPS data.

GPS trajectory data and significant region identification

Figure 1 summarizes the structured, multi-step workflow used to derive behaviorally meaningful regions from raw trajectories and to assign semantic place types. It begins by processing raw GPS trajectories with timestamped latitude and longitude coordinates. These raw points are often subject to noise, including signal drifting and implausible jumps, which are common in high-frequency logging environments. To correct this, we compute segment-wise speeds between consecutive points using the Haversine distance and time difference. GPS points are grouped by minute, and within each minute, outlier speeds are removed using the interquartile range (IQR) method. The smoothed mean speed of each minute is then applied to all points within that time window. This procedure ensures that transient spikes or drifting behavior do not bias downstream mobility analysis.

Fig. 1.

Overview of GPS data processing and location type identification. (a) Raw GPS trajectories. (b) Segment-level speed calculation and temporal smoothing. (c) Spatial assignment of GPS points using H3 grid. (d) Behavioral filtering using thresholds on average speed and total duration. (e) Aggregation of adjacent stay regions with centroid computation. (f) Semantic enrichment of selected regions via multimodal classification models.

After temporal smoothing, we discretize the cleaned GPS data using the Level 10 resolution of the H3 hexagonal grid system, whose cells have an average edge length of ~ 115.6 meters and flat-to-flat diameter of ~ 231 meters. Each GPS point is mapped to a unique hexagonal cell, enabling standardized spatial aggregation across individuals and days. This spatial resolution is comparable to the commonly used 100-meter buffer in mental health research and helps mitigate the impact of small-scale fluctuations or jitters in GPS signals.

To extract behaviorally meaningful areas, we evaluate each H3 cell using two mobility thresholds. First, we compute the average smoothed speed of all points within each cell. Second, we calculate the total duration the individual spent in that cell. Cells are retained if the mean speed is less than or equal to 1.2 m per second (m/s) and the cumulative duration is greater than or equal to 15 min (min). This dual condition filters out transient visits and captures spatial units reflecting deliberate walking or staying behavior. Due to the fixed boundaries imposed by H3, edge effects may artificially split continuous activity into adjacent cells, even when the individual is essentially in the same place. Conversely, people often move slightly around a fixed location, such as home or work, resulting in behaviorally similar points spanning multiple H3 cells. Therefore, adjacent H3 cells that meet the above criteria are aggregated to form larger behavioral regions. The centroid of each aggregated region is calculated as the duration-weighted average of all adjacent H3 areas.

We additionally identify a home location for each participant based on nighttime behavior. Specifically, among all visited H3 cells, the one with the longest cumulative stay during the period from 11:45 PM to 6:00 AM is selected as the home cell. This definition prioritizes regular nighttime residence and is robust to short-term variation in movement during daytime hours.

Semantic place classification with multimodal language models

After identifying meaningful stay and home regions, we apply a multimodal semantic classification framework to infer the functional types of these locations (Fig. 2). The classification integrates both image-based and text-based contextual information using state-of-the-art multimodal language models.

Fig. 2.

Multimodal semantic classification framework. (a) Region-level image tiles from satellite imagery and OSM map renderings are input to vision-language models for type prediction. (b) Building metadata extracted from OSM is parsed and interpreted via language models. (c) Model comparison and validation. (d) Random Forest on H3 semantics and stress. Eleven land-use types are grouped into five functional categories for structured interpretation

For each H3 cell corresponding to a stay region, we generate two types of map tiles: one satellite image tile from Esri World Imagery and one OSM rendering with building overlays. These map images are used as input to vision-language models including Llava 1.5, Qwen2.5-VL, and CLIP ViT, which perform zero-shot classification and generate natural language explanations. Each model outputs a predicted location type such as residential, healthcare, park, or commercial, along with a rationale.

Simultaneously, we retrieve building footprint data from OSM using the building=* tag. The area and label of each building within the hexagon are extracted and organized into a structured JSON format. From this list, the top-ranked buildings by area and semantic relevance are selected. The building metadata is input into text-based LLMs such as GPT-4o Mini and Gemini 2.5 flash, which infer the region’s primary land use based on building names, functions, and areas. The workflows were created in KNIME to ensure reproducibility [53, 54]. Additionally, we extract keywords from the model-generated explanations to understand which features contributed to the decision.

All regions are ultimately classified into 11 base land-use types, organized into 5 functional domains for interpretability (Table 1). Each category is grounded in peer-reviewed evidence on mental-health–relevant exposure pathways, ensuring the taxonomy is theory- and evidence-informed. This classification process balances model accuracy with interpretability. Predictions from multiple models are compared and resolved via voting, confidence ranking, and text reasoning overlap. When inconsistencies are observed, manual annotations are used as reference to benchmark performance and calibrate model weight.

Table 1.

Classification of land-use types and five functional domains

Category I	Category II	Mental-health relevance	Exemplar references
Residential	Residential	Housing stability, homestay, nighttime exposure (house, apartments, dorm)	Housing disadvantage/instability is linked to worse mental health in longitudinal studies and health-system data [55].
Essential	Healthcare	Healthcare accessibility, behavioral health access (hospital, clinic, pharmacy, rehab)	Spatial accessibility to behavioral health care is a salient determinant of outcomes [56].
	Educational	Social engagement, access to campus psychological services (school, kindergarten)	Educational settings are major social participation sites and gateways to campus mental-health services; see social-ties literature [57].
	Religious	Social support/cohesion, rituals and well-being (church, mosque, temple)	Religious/spiritual participation shows associations with lower depression risk/severity [58].
Leisure	Green	Nature contact, stress recovery, walking opportunities (park, garden, forest)	Exposure to greenspace is associated with better mental well-being and lower depression/anxiety risk [59].
	Recreational	Physical activity, social engagement, emotion regulation (gym, theatre, stadium)	Access to recreation supports physical activity, and exercise interventions reduce depressive symptoms in RCT/meta-analyses [60].
Mobility	Hospitality	Business trip/travel stress, routine disruption (hotel, hostel, motel)	Work-related travel and frequent overnight stays are linked to elevated stress, anxiety, and fatigue [61].
	Transportation	Commuting delays and stress, accessibility (station, terminal, airport)	Commuting time and mode relate to subjective well-being and mental health; longer stressful commutes tend to be detrimental [62].
Commerce	Office	Work rhythms, social/commuting indirect effects (office, headquarters)	Indirect pathways via commuting, work routines, and social exposure [62].
	Commercial	Alcohol and food environment exposure, consumption-related stress (shop, cafe, bar)	Alcohol-outlet exposure and harmful alcohol use are associated with anxiety/depression; food environments relate to diet-linked mental health [63]
	Mixed use	Functional diversity and walkability (commercial + residential)	Land-use mix and walkability show associations with mental-health outcomes in recent reviews [64].

Model evaluation and consistency assessment

To evaluate the performance of the semantic classification framework, we applied a combination of standard and explanatory evaluation metrics. Confusion matrix, overall accuracy, and Cohen’s kappa were used to assess classification agreement between predicted labels and human annotations. Accuracy reflects the proportion of correctly classified H3 cells, while kappa adjusts for agreement expected by chance and accounts for class imbalance.

We further applied keyword extraction and word cloud visualization to summarize dominant decision cues used by the models. Word frequency distributions were visualized per category to support model interpretation and error diagnosis. These metrics provide a comprehensive assessment of both classification accuracy and semantic coherence, balancing quantitative performance with explainability.

Results

Semantic classification based on OSM Building data

OSM Building data extraction

Among the 4,931 Level-10 H3 hexagons included in the study, valid OSM building information was obtained for 3,950 hexagons, covering a total of 55,393 individual building records. The rank-size distribution of building areas followed a Zipf-like power law, with a double-logarithmic R² of 0.91. Of these, 268 (0.5%) buildings had an area of zero square meters, and 2,357 (4.3%) buildings exceeded 5,000 m² in footprint area.

Across all extracted records, 1,065 unique OSM key-value fields were identified. Only five fields had non-null values in more than 1% of records: building (100%), name (12%), amenity (5%), shop (3%), and cuisine (1%) (Fig. 3A). Despite the sparsity of other attributes, several low-prevalence fields (e.g., shop, hotel) are important for semantic interpretation. To systematically assess the relevance of these fields, we employed Gemini 2.5-Flash-Lite and GPT-4o-Mini to score field importance using a structured prompt.

Fig. 3.

(A) Field-level importance scores for semantic classification by Gemini and GPT. (B) Classification results by OSM text features. (C) Sankey diagram of classification discrepancies between Gemini and GPT

You are an urban analytics expert. You are given a combination of OSM columns separated by comma. You need to rank these column names based on their importance for semantic location or building type recognition and assign an importance score between 0 and 1.

Gemini assigned importance scores to all 1,065 fields, identifying 155 fields with an importance score > 0.4. GPT-4o extracted only 585 fields, with a correlation of 0.46 on importance score compared to Gemini. Figure 3B shows the top-ranked fields (importance > 0.85 in either model), sorted by GPT’s scores, with notable discrepancies across fields such as “school”, “leisure”, and “apartment”.

To streamline semantic processing and reduce token length in prompt-based classification, we constructed a simplified feature set containing all fields with importance score ≥ 0.4 from either model (n = 207). H3 cells containing only a generic “building = yes” tag with no additional semantic fields were excluded, resulting in 3,845 hexagons for classification.

Semantic classification based on OSM information

For each of these hexagons, we aggregated building areas and retained the simplified OSM tags, which were then passed to Gemini and GPT to get both a predicted type and a rationale using the following prompt:

You are a researcher studying mental health and human mobility. Below is OSM building data for a small area, including building types and their areas (in m²). Based on all buildings and their total areas, identify the main functional category of the area. Choose one from the list below: healthcare (hospital, clinic, pharmacy, rehab), religious (church, mosque, temple), green (park, garden, forest), educational (school, kindergarten), recreational (gym, theatre, stadium), hospitality (hotel, hostel, motel), transportation (station, terminal, airport), office (office, headquarters), commercial (shop, cafe, bar, bank), residential (house, apartments, dorm), mixed-use (galleria, commercial + residential).

Across all predictions, the two models agreed on classification for 3,032 of 3,845 hexagons (79%), indicating high consistency. 625 hexagons (16%) showed divergent results. As visualized in the Sankey diagram (Fig. 3C), a substantial portion of these disagreements involved Gemini classifying a region as residential (n = 428) while GPT labeled it as commercial or mixed-use (n = 471). A detailed review of representative discrepancies revealed five main types of disagreement (Table 2):

Table 2.

Exemplary H3 hexagons with divergent classifications from gemini and GPT

Type Result	Reason by GEMINI	Reason by GPT4O
H3 Index 1 8a1943268a0ffff Gemini Residential GPT Mixed_use	The total area of residential buildings (3403 + 2932 + 1817 + 1252 + 1113 = 10517 m) is the largest among all functional categories. While there is a hotel (7965 m) and a supermarket (10107 m), the combined residential area surpasses them. The ‘construction’ and ‘yes’ buildings are not categorized by function	The area contains a significant mix of residential buildings (Block 2D, Block 2B, Block 2 A, Block 8 A) and commercial establishments (Sainsbury’s supermarket and Hyatt Centric hotel), indicating a blend of living and commercial spaces
H3 Index 2 8a1eb52ac24ffff Gemini residential GPT green	The area is predominantly residential, with a significant portion dedicated to residential buildings (190 m). While there is also a greenhouse (417 m), its primary function is not recreational or a public park, and the residential component is the most substantial for a functional category	The greenhouse is the largest building type, indicating a significant green space presence in the area
H3 Index 3 8a1f99786777fff Gemini residential GPT unknown	The provided data contains only one building with an area of 177 m and a generic ‘building: yes’ tag. Without more specific building type information, it’s impossible to definitively categorize the area’s main function. However, given the limited data, ‘residential’ is the most general and common building use	Insufficient data to determine the main functional category
H3 Index 4 8a2664d882c7fff Gemini residential GPT office	The total area of residential buildings (apartments) is 10411 m, which is the largest among all functional categories. Other buildings like ‘yes’ and ‘retail’ are not specific enough to determine a primary function, and the single ‘office’ building has a smaller area compared to residential buildings	The area is predominantly characterized by office buildings, with the largest building being an office at 6515 m², significantly larger than residential and other types of buildings
H3 Index 5 8a2a14b98237fff Gemini green GPT mixed-use	The area is predominantly composed of ‘deck’ structures, which are often associated with outdoor recreational spaces like parks or gardens	The area consists of decks, which can serve multiple purposes, indicating a potential mixed-use functionality

[1] Area dominance vs. keyword-triggered classification: Gemini relied heavily on total floor area dominance (e.g., residential footprint exceeding others), whereas GPT was more sensitive to distinctive terms such as “supermarket” or “hotel”, leading to mixed-use labels.

[2] Semantic overinterpretation: GPT occasionally misclassified based on single-structure tags (e.g., labeling a greenhouse as green space), while Gemini correctly focused on the dominant building category.

[3] Sparse or generic metadata handling: In cases with minimal data (e.g., “building = yes” only), Gemini defaulted to residential, while GPT returned unknown.

[4] Literal keyword prioritization: GPT sometimes prioritized specific named entities (e.g., brand offices) over aggregated land-use dominance, leading to divergent office vs. residential classifications.

[5] Ambiguous nonstandard structures: For regions with nonstandard labels (e.g., “deck”), Gemini assigned green space while GPT opted for mixed-use.

We also analyzed the models’ rationale texts using keyword frequency and word cloud visualizations (Fig. 4). The top 100 keywords from Gemini’s residential classifications and GPT’s mixed-use classifications overlapped by 64 terms. The primary differences arose from GPT’s greater emphasis on the co-occurrence of commercial or retail features within predominantly residential areas, which frequently led to a mixed-use classification.

Fig. 4.

Gemini residential results for (A) word frequency ranks and (B) word clouds, and GPT mixed-use results for (C) word frequency ranks and (D) word clouds

The per-class results in Table 3, derived from OSM building attributes, highlight systematic similarities and differences between GPT and Gemini. For the residential class, GPT achieved higher recall (0.483 vs. 0.421) but lower precision (0.835 vs. 0.954), yielding a slightly higher F-measure (0.612 vs. 0.584). This pattern reflects GPT’s greater sensitivity to residential cues, while Gemini adopted a more conservative, area-dominant approach that reduced false positives. For commercial, transportation, educational, and recreational categories, both models showed balanced recall and precision with F-measures consistently above 0.55, demonstrating that OSM tags and names in these domains provide sufficient semantic information. Gemini slightly outperformed GPT in commercial, transportation, and recreational, while GPT was marginally better in educational.

Table 3.

Per-class and overall evaluation for GPT and gemini in semantic classification based on OSM Building data

Type	GPT			GEMINI
	Recall	Precision	F	Recall	Precision	F
Commercial	0.784	0.73	0.756	0.794	0.735	0.763
Residential	0.483	0.835	0.612	0.421	0.954	0.584
Mixed-use	0.294	0.301	0.298	0.077	0.002	0.004
Educational	0.78	0.575	0.662	0.736	0.596	0.659
Recreational	0.632	0.497	0.557	0.725	0.508	0.597
Green	0.778	0.04	0.076	0.667	0.068	0.124
Transportation	0.798	0.558	0.657	0.757	0.632	0.689
Healthcare	0.733	0.389	0.508	0.702	0.407	0.516
Religious	0.643	0.293	0.402	0.703	0.423	0.528
Hospitality	0.515	0.5	0.507	0.457	0.6	0.519
Office	0.41	0.348	0.376	0.276	0.587	0.375
Overall	11 Class	11 Class*	5 Class	11 Class	11 Class*	5 Class
Accuracy	0.596	0.637	0.665	0.601	0.683	0.647
Cohen’s kappa	0.490	0.53	0.509	0.493	0.58	0.506

* Indicates results calculated with mixed-use categories excluded

In minority classes, performance diverged more strongly. For office, both models performed poorly, with F-measures below 0.38, but GPT recalled more cases (0.41 vs. 0.276) whereas Gemini achieved higher precision (0.587 vs. 0.348). For religious, Gemini achieved better balance (F = 0.528 vs. 0.402), again reflecting its reliance on dominant building area rather than sparse textual cues. For healthcare and hospitality, both models achieved moderate accuracy (F around 0.51), with Gemini slightly higher in both. For green space, recall was high but precision extremely low for both models, leading to very low F values. Gemini attained a higher F (0.124 vs. 0.076), suggesting that it was less prone to false positives compared to GPT, which often misclassified single tags such as “greenhouse.” The mixed-use category was the most challenging: GPT achieved an F of 0.298 by capturing distinctive commercial–residential co-occurrence, while Gemini almost failed entirely (F = 0.004), consistent with the earlier observation that Gemini heavily prioritizes dominant building areas and underestimates hybrid land-use patterns.

At the overall level, the two models were comparable in the 11-class setting. When mixed-use was excluded, both improved, but Gemini benefited more, confirming that mixed-use is the major source of error in OSM-based labeling. When categories collapsed into five functional domains, GPT slightly outperformed Gemini, though differences were marginal. These results align with the qualitative discrepancies summarized in Table 1: Gemini’s strength lies in conservative, area-weighted classification with high precision in dominant categories, while GPT captures more variation and performs better in mixed-use and certain minority classes but at the cost of additional false positives.

Semantic classification based on OSM and satellite imagery

Overall comparison

This study further compared the performance of multiple open-source and closed-source VLMs applied to combined OSM building map tiles and ESRI satellite imagery. We prioritized closed source LLMs for larger effective capacity and longer context windows, and included strong open source VLMs to assess reproducible boundaries in imageryheavy settings. We evaluated four open-source models (LLaVA-1.5-7b-hf, CLIP-ViT-B/32, Qwen2.5-VL-7b-instruct, Deepseek-VL-7b-chat) and two closed-source models (GPT-4o-mini, Gemini-2.5-flash-lite). Ground truth labels were produced by three trained annotators for 4,695 H3 level 10 cells using a shared guideline that defines class rules, edge cases, and mixed parcel handling. The prompts for all the models are presented in Supplement Table S1.

The results in Table 4 demonstrate clear differences across models. Deepseek-VL produced systematic errors by misclassifying hexagons containing church symbols, leading to extremely low accuracy (0.137). CLIP, while able to generate probability distributions over 11 categories, achieved only 0.179 accuracy in the 11-class task. LLaVA improved slightly (0.269), but both remained limited in their ability to utilize combined map and image cues. In contrast, Qwen achieved substantially better performance (0.565 accuracy, κ = 0.465), showing stronger ability to follow prompts and exploit OSM-text and image information. Among closed-source models, Gemini (accuracy = 0.79, κ = 0.745) and GPT (0.779, κ = 0.734) significantly outperformed open-source alternatives. When categories were aggregated into five functional domains, Gemini reached 0.858 accuracy (κ = 0.804) and GPT 0.849 (κ = 0.794), while Qwen achieved 0.702 (κ = 0.581), confirming the clear advantage of closed-source models. Excluding the mixed-use class (11 Class*), performance improved further for all models, indicating that mixed-use labeling remains the principal source of error.

Table 4.

Overall accuracy and Cohen’s kappa of multimodal models (open-source and closed-source) using OSM map tiles and Esri satellite imagery

	11 Class		5 Class		11 Class*
Models	Accuracy	k	Accuracy	k	Accuracy	k
CLIP	0.179	0.109	0.43	0.225	0.127	0.093
LLAVA	0.269	0.127	0.38	0.169	0.294	0.144
QWEN	0.564	0.464	0.702	0.580	0.608	0.51
GEMINI	0.790	0.746	0.858	0.804	0.852	0.815
GPT	0.779	0.734	0.849	0.794	0.8	0.754

* Indicates results calculated with mixed-use categories excluded, k for Cohen’s kappa

Per-class performance

Table 5 compares Qwen, GPT, and Gemini across the 11 functional categories. In the major categories where OSM tags and building footprints provide clearer semantic cues, both closed-source models consistently out-performed Qwen. Commercial, residential, educational, and transportation categories all achieved F-measures above 0.79 for GPT and Gemini, whereas Qwen’s performance ranged between 0.685 and 0.742. For instance, Gemini achieved an F of 0.887 for commercial and 0.838 for residential, compared with Qwen’s 0.742 and 0.685, respectively. GPT displayed a similar pattern, with the highest F of 0.841 in educational and particularly high precision in healthcare (0.966), reflecting its capacity to leverage distinctive OSM tags such as “clinic” and “hospital.”

Table 5.

Per-class recall, precision, and F-measure for Qwen, GPT, and gemini in semantic classification based on OSM Building attributes and satellite imagery

	Qwen			GPT			GEMINI
	Recall	Precision	F	Recall	Precision	F	Recall	Precision	F
Commercial	0.675	0.822	0.742	0.890	0.795	0.840	0.872	0.903	0.887
Residential	0.794	0.603	0.685	0.960	0.783	0.863	0.937	0.758	0.838
Mixed-use	0.234	0.186	0.208	0.458	0.598	0.519	0.672	0.254	0.369
Green	0.735	0.053	0.098	0.784	0.783	0.784	0.720	0.880	0.792
Educational	0.691	0.667	0.679	0.791	0.898	0.841	0.692	0.928	0.793
Recreational	0.292	0.688	0.410	0.683	0.728	0.705	0.851	0.588	0.696
Transportation	0.654	0.249	0.360	0.812	0.863	0.837	0.789	0.951	0.863
Healthcare	0.474	0.657	0.551	0.523	0.966	0.679	0.769	0.933	0.843
Religious	0.429	0.686	0.527	0.748	0.871	0.805	0.463	0.936	0.619
Hospitality	0.350	0.244	0.288	0.750	0.802	0.775	0.760	0.849	0.802
Office	0.156	0.086	0.111	0.917	0.190	0.314	0.369	0.948	0.531

However, performance diverged markedly in minority or semantically ambiguous classes. The mixed-use category remained the most challenging: Qwen yielded the lowest F (0.208), while GPT (0.519) and Gemini (0.369) performed better but still failed to reach the levels of major categories. This reflects the inherent ambiguity of OSM tags in hybrid areas where commercial and residential functions co-occur. For green space, Qwen achieved very high recall (0.735) but extremely low precision (0.053), resulting in a poor F of 0.098, suggesting over-classification of tags such as “greenhouse” or “deck.” By contrast, GPT (0.783) and Gemini (0.792) showed more balanced performance, producing substantially higher F values. The office category was also difficult: Qwen nearly failed (F = 0.111), whereas GPT (0.314) and Gemini (0.531) produced modest but more reliable results, likely benefiting from better recognition of organizational names and building size dominance.

For institutional and less frequent classes such as religious, healthcare, hospitality, and recreational, the two closed-source models again demonstrated clear advantages. GPT obtained the highest F for religious (0.805), while Gemini scored lower (0.619) due to lower recall but extremely high precision (0.936), indicating a more conservative approach. In healthcare, Gemini achieved the best overall F (0.843), reflecting strong performance in detecting medical facilities, while GPT achieved 0.679 with excellent precision but more missed cases. In recreational, both closed models performed better than Qwen, with Gemini’s F of 0.696 slightly lower than GPT’s 0.705. For hospitality, Gemini reached 0.802 compared with GPT’s 0.775 and Qwen’s 0.288, suggesting that Gemini captured features such as “hotel” or “hostel” tags more effectively.

Taken together, these per-class comparisons show that Qwen provides a usable baseline but is limited in minority and ambiguous categories due to sparse or noisy OSM labels. GPT excels in recall-oriented performance, especially in educational, religious, and transportation categories, whereas Gemini demonstrates more balanced precision–recall trade-offs, with particular strength in healthcare, hospitality, and office detection. The consistently poor results in mixed-use and green highlight the structural limitations of OSM-based semantic classification, where label ambiguity and inconsistent tagging practices constrain even advanced models.

Case-based analysis of classification divergence

A detailed review of six representative H3 hexagons (Fig. 5, supplement Table 1) highlighted four recurrent modes of classification divergence across models.

Fig. 5.

Exemplary H3 hexagons illustrating common patterns of model divergence from non-clinical data

[1] Stable performance in clear residential and commercial areas: In well-defined neighborhoods such as H3 8a2a14b98237fff (residential) and H3 8a2664d882c7fff (commercial), GPT, Gemini, and Qwen consistently produced correct classifications, whereas CLIP and LLaVA occasionally misclassified them as mixed-use.

[2] Transportation hubs identified mainly through text cues: In H3 8a392e32c757fff, GPT and Gemini correctly recognized a bus station by using the OSM label “Estación de autobuses de Villalpando,” while Qwen returned “unknown” and CLIP produced a mixed classification, showing that text parsing is crucial for transportation recognition.

[3] Ambiguity in mixed-use and sparse metadata contexts: In hybrid or metadata-sparse areas such as H3 8a1943268a0ffff and H3 8a1eb52ac24ffff, GPT tended to classify as mixed-use or educational, Gemini defaulted to commercial or green, and Qwen often anchored on a single token (e.g., “Farmacia Comunale”), producing healthcare. These examples illustrate different strategies in the face of incomplete or conflicting signals.

[4] Misclassification from symbolic or nonstandard features: In H3 8a1f99786777fff, CLIP and LLaVA over-interpreted visual textures (recreational), while GPT, Qwen, and Gemini leaned toward residential. Ambiguous cues such as “deck” or greenhouse structures also triggered errors, underscoring the challenge of symbolic OSM features.

Across these cases, the four models showed systematic differences. CLIP relied purely on probabilistic visual embeddings, yielding unstable classifications in ambiguous contexts and often misinterpreting symbolic forms (e.g., churches, decks). LLaVA produced natural language rationales but tended to describe visual context without mapping it to precise functional categories, underperforming in cases like commercial centers and transport hubs. Qwen2.5-VL was the strongest open-source model, able to combine map and image cues with structured reasoning, but prone to over-anchoring on single OSM tokens, leading to semantic overinterpretation in mixed-use areas. In contrast, the closed-source models GPT-4o-mini and Gemini-2.5-flash-lite consistently outperformed the others, integrating map text and imagery to produce accurate classifications with more nuanced rationales. GPT was particularly strong in recall-oriented classes such as religious, educational, and transportation, while Gemini emphasized footprint dominance and achieved higher precision in healthcare and commercial categories.

A comparison of pure OSM text versus multimodal OSM + imagery further confirmed these tendencies. In H3 8a1943268a0ffff, Gemini consistently labeled the area as residential, while GPT shifted from mixed-use with text-only input to commercial with multimodal input, showing its sensitivity to cues like “Sainsbury’s” and “Hyatt.” In H3 8a1eb52ac24ffff, Gemini remained residential, whereas GPT changed from green in the text-only setting to educational with imagery, amplifying its reliance on isolated tokens such as “greenhouse” or “library.” In sparse contexts like H3 8a1f99786777fff, Gemini stayed residential, but GPT vacillated between unknown and mixed-use, underscoring instability under limited metadata. In H3 8a2664d882c7fff, Gemini moved from residential (text-only) to commercial (with imagery), while GPT shifted from office to commercial, reflecting how imagery corrected some textual misclassifications. Finally, in H3 8a2a14b98237fff, Gemini switched from green (text-only) to residential (with imagery), while GPT labeled it mixed-use in both cases.

Overall, these case-based analyses demonstrate a consistent trade-off between the two leading models. Gemini’s conservative reliance on dominant building footprints yields higher precision and greater stability, and benefits further from imagery cues. GPT’s greater sensitivity to distinctive textual features makes it more flexible in detecting mixed or minority classes, but also more prone to semantic drift in sparse or symbolic contexts.

Day-level associations between stress and H3 semantic locations exposure

Stress data and descriptive statistics

We used ecological momentary assessment (EMA) from BIDMC to capture same-day stress. Participants received a smartphone prompt at 8:00 PM (available until 2:00 AM) to complete an approximately 30 item survey designed to take less than five minutes. Perceived stress was measured by the item “Today I felt stressed” on a 7-point Likert scale (1 = not at all, 7 = extremely); analyses use the daily score. GPS trajectories were segmented into stay/walk regions and mapped to H3 cells labeled by our semantic classifier. For each participant-day (N = 6,787), we computed minutes in Home and in 11 non-home classes (daily total ≈ 1,440 min). To avoid conflating contexts, Residential denotes residential areas other than the participant’s own home cell.

Table 6 reports the distributions (minutes/day), prevalence (percent of days with > 0 min), and Pearson correlations with stress. Correlations were small in magnitude but directionally informative: negative for Residential (non-home) and Home, and positive for Healthcare, Hospitality, Green, Mixed-use, and Recreational.

Table 6.

Descriptive statistics and correlation with stress score

Variables	Visit (%)	Min	Max	Mean	Std. deviation	correlation	p value
Educational	0.222	0.0	1438.7	33.188	0.073	−0.01	0.563
Home	0.866	0.0	1440.0	777.112	0.327	−0.056	0.001***
Mixed use	0.169	0.0	1256.216	19.272	0.072	0.034	0.043*
Residential	0.393	0.0	1439.983	93.479	0.192	−0.074	< 0.001
Healthcare	0.073	0.0	1142.833	11.977	0.049	0.064	< 0.001***
Religious	0.034	0.0	662.949	3.242	0.026	−0.011	0.51
Transportation	0.038	0.0	554.414	1.957	0.02	0.017	0.327
Commercial	0.321	0.0	745.159	27.402	0.059	−0.003	0.865
Recreational	0.096	0.0	1044.862	15.173	0.055	0.033	0.048*
Green	0.112	0.0	1439.647	25.326	0.084	0.041	0.016*
Hospitality	0.022	0.0	1321.9	5.19	0.04	0.055	0.001***
Office	0.018	0.0	706.592	1.866	0.024	−0.033	0.052

*** for p ≤ 0.001; ** for p ≤ 0.01; * for p ≤ 0.05

Random Forest model and explainable methods

To explore potential non-linearities we fitted a Random Forest regressor to minutes. Model fit was R² = 0.521. We further applied explainable artificial intelligence (XAI) methods, including feature importance and individual conditional expectation (ICE) profiles (Fig. 6). Feature importance ranked categories as: Home > Residential > Commercial > Educational > Mixed-use > Green > Recreational > Healthcare > Hospitality > Transportation > Office > Religious.

Fig. 6.

Global feature importance and ICE profiles smoothed with generalized additive models for the Random Forest

The GAM (generalized additive model) smoothed ICE profiles refine the correlation picture as follows [1]. Monotonic decrease: home, office, and commercial. More time at home or in offices may reflect stable routines, predictable schedules, and clear social or role structure, contexts that are typically associated with lower daily stress. For commercial, the decreasing trend suggests that shopping or errand contexts are not stress inducing on average and may even be restorative with low arousal, consistent with the near zero correlation [2]. Monotonic increase: mixed use and Healthcare. Mixed use areas combine retail, services, and circulation; more minutes may indicate busy, multi-purpose days. Time in healthcare likely captures care seeking or treatment episodes, aligning with the positive correlation [3]. Decrease then increase (U shaped): residential outside the participant’s home and Green. Short stays in other residential areas may coincide with low demand social or errand visits and lower stress, whereas extended time away from one’s own home could signal instability or caregiving and thus higher stress. For green, modest exposure may be restorative, but very long durations may reflect travel to the edge of the city or strenuous outdoor activity, with stress trending back up [4]. Increase then decrease (inverted U): recreational and religious. Moderate minutes in these contexts may involve social engagement or events with slightly higher arousal, while longer durations may represent settled participation or recovery and therefore easing stress [5]. Decrease then increase: hospitality and transportation. Brief exposure such as checking in or short rides is associated with low stress, but extended overnight stays or long commutes or travel correspond to routine disruption, circadian misalignment, and commuting strain, which increase stress.

Discussion

This study tackles a central bottleneck in health geomatics: turning high-frequency GPS trajectories into stable, comparable, and interpretable place semantics. We propose H3-MOSAIC, which fuses OSM building text and satellite imagery on H3 grids and systematically compares two proprietary and five open-source multimodal generative AI models. Overall, multimodal methods substantially improve classification accuracy and consistency in both 11-class and 5-class tasks. After class consolidation, open-source solutions become practically usable for cost-sensitive, large-scale deployment, while proprietary models remain more stable on hard classes and edge cases. This reusable H3 baseline resources (level 10) and processing workflow can enable cross-city, cross-country comparability and privacy-aware sharing. With this foundation, stabilized place semantics for homestay, entropy, and activity space can directly support time-weighted NDVI (Normalized Difference Vegetation Index) exposure, further reducing exposure misclassification and strengthening inference.

Compared with prior work, H3-MOSAIC advances the field in three ways. First, it moves beyond the “POI/rules only” versus “imagery only” dichotomy and delivers an interpretable, multimodal semantic inference pipeline. Recent LLM-for-text studies show that parsing OSM tags with language models can replace rule-based parsing and markedly improve transferability and F1 for building function recognition, alleviating POI/OSM semantic sparsity and heterogeneity at the source [44]. We go further by adding satellite tiles and behavioral priors (stay/walk thresholds and a 23:45–06:00 nighttime home rule), which reduce ambiguity in mixed parcels and stabilize downstream indicators such as homestay, location entropy, and activity space. This text-plus-imagery-plus-behavior design yields clear gains where text is sparse or contradictory, for example in mixed-use, office, and green classes.

Second, it provides model-diagnosable and governable fusion strategies. Building on the two-stage reasoning pipelines that combat label ambiguity and class imbalance [45], we introduce explanation-consistency auditing (keyword/term frequency) to handle systematic divergences across models, most notably the tension between “area-dominant” and “keyword-triggered” decisions. Beyond that, work that fuses map tiles or remote-sensing imagery with text has improved regional land-use prediction under label scarcity [48]. Our head-to-head study extends this line in a health-geomatics setting by quantifying the recall–precision trade-offs of proprietary and open-source VLMs/LLMs on health-relevant classes and by diagnosing error modes with human-readable rationales.

Third, it shifts the focus from “object/land-use recognition” to “behavioral semantics.” By binding place categories to individual behavior through stay/walk thresholds and a nighttime home rule, H3-MOSAIC makes homestay, entropy, and activity space more stable, interpretable, and reusable. This is critical for mental-health research: core outcome indicators depend on correct delineation of residence, major destinations, and functional environments [26]. Collapsing mixed places into a single dominant class can underestimate exposure diversity, whereas over-emphasizing mixed use can inflate diversity and complicate interpretation. Our multimodal fusion approach stabilizes these indicators without sacrificing sensitivity to clinically important but less frequent environments such as healthcare, religious, and transportation settings. This aligns with evidence that passive sensing supports behavioral monitoring, mobility and environmental exposure related to mood and well-being [33, 34].

On model comparison, we benchmark cost-viable proprietary models (GPT-4o-mini, Gemini-2.5-flash-lite) and mainstream open-source baselines (Qwen-VL, LLaVA, CLIP), and we run comparable experiments for text-only and multimodal-fusion routes, quantifying strengths and error typologies in mixed-use/label-sparse settings. Notably, DeepSeek-VL systematically misread the red H3 hexagon frame as a religious symbol in this task, collapsing accuracy; once those systematic errors are excluded, its overall performance approaches LLaVA. Consistent with recent imagery-only land-cover/use studies (e.g., LoveDA-style scene classification), heterogeneous urban form constrains visual models when text anchors are absent, making them prone to failures on mixed parcels and symbol/texture artifacts [50]. Our results show that under the same mixed-use and label-scarce conditions, fusing OSM text with satellite imagery markedly improves stability and interpretability and reduces false triggers seen in text-only approaches. External comparative evaluations conclude that proprietary models are generally more robust while open-source models can approach usability on coarse classes [51]. We reproduce that ranking in a health-geomatics setting and further pinpoint when and why each model fails—e.g., GPT is more sensitive to salient POI tokens such as “supermarket” and “hotel,” whereas Gemini emphasizes footprint-area dominance, which informs rule-based fusion and human review.

Across the eleven class and five domain tasks, multimodal fusion of OpenStreetMap text and satellite tiles delivers the most stable performance, particularly in mixed use, office, and green where either text or imagery alone is ambiguous. Within this study’s models and versions, proprietary multimodal systems were more stable on rare or difficult classes than the open source VLMs we tested. Vision-only models underperform when text anchors are sparse, and text-only routes miss visual cues of form and texture. This observation is model- and version-specific and should not be generalized to all proprietary or open-weight models. Given fast model turnover, the reported numbers should be read as a time stamped baseline rather than a fixed ranking.

The associations between daily stress and H3 defined locations are consistent with established mental health pathways and support the face validity of our semantics. More minutes at Home relate to lower stress, consistent with evidence on housing stability and regular homestay [55]. Green shows a U-shaped pattern where brief exposure appears restorative [59], whereas very long durations likely reflect travel or demanding activity and stress rises again. Recreational and Religious show an inverted U-shaped pattern in which moderate minutes coincide with events and social participation, while longer stays align with recovery and lower stress [58, 60]. These patterns clarify why the learned semantics are informative for mental health. Future work will test within person fixed effects models, stratify by weekday and weekend and baseline symptom levels, and refine green and recreational typologies to separate restorative from demanding contexts.

Conclusion

We introduced H3-MOSAIC, a multimodal generative AI framework that fuses high-frequency GPS with OSM building text and satellite imagery on H3 grids to produce stable, comparable, and interpretable place semantics for mental-health geomatics. Across both 11-class and 5-class tasks, multimodal fusion outperformed single-modality baselines and clarified where open-source models are practically usable and where proprietary models remain more robust. H3 day level exposures show stress patterns consistent with established mental health pathways, supporting face validity. By coupling semantic outputs with behavior-aware rules (stay/walk thresholds and a nighttime home definition), H3-MOSAIC strengthens downstream indicators such as homestay, location entropy, and activity space, enabling more reliable exposure assessment and cross-city comparability while supporting privacy through grid-level aggregation.

This work has limitations. OSM coverage and label quality vary by place and time; satellite imagery can be temporally misaligned with GPS; class imbalance and single-label targets complicate mixed-use areas; closed-source models constrain reproducibility and cost transparency; and high-frequency sensing data may be missing non-randomly across individuals. Future work will include sensitivity analyses across H3 resolutions and dwell thresholds; disagreement-guided active learning and lightweight instruction-tuning for open-source models; explicit uncertainty calibration; integration of richer temporal context (e.g., opening hours, weather, street-level imagery) to improve interpretability; In addition, future work will derive sleep periods from smartphone screen-state (on/off) and accelerometer data, enabling joint analyses of circadian rhythms and mobility-derived place semantics for mental health applications. Finally, we will explore privacy-preserving deployments that couple stable semantics with time-weighted NDVI and longitudinal outcomes in generalized estimating frameworks.

For model comparison, this study adopts a unified zero-shot protocol to compare multiple LLMs/VLMs, avoiding attributing improvements to extra-model prompt tricks or corpus retrieval. Complex prompting (e.g., CoT or RAG) substantially increases context length and inference cost, and the gains are often model-specific, which undermines cross-model fairness. Under the same evaluation protocol, future work will systematically assess constrained outputs, few-shot CoT, geographic RAG, and lightweight instruction fine-tuning of open-source models.

Author contributions

L.L. (Lingbo Liu): conceptualization, methodology, formal analysis, writing & editing. R.F. (Rachel Franklin)*: supervision, writing—review & editing. J.T. (John Torous)*: supervision, writing—review & editing. J.C. (Jiaee Cheong), T.C. (Tianyue Cong), J.S.B. (Andrew Jin Soo Byun), A.Y.O. (Allie Yubin Oh): data acquisition, preprocessing, validation, resources. All authors read and approved the final manuscript. (*Corresponding authors: * L.L., R.F., J.T *.*).

Funding

This work was supported by the National Science Foundation (NSF) Award No. 181143.

Data availability

Derived datasets (H3 grids index, aggregated semantic labels, model predictions, and evaluation outputs) is deposited in Harvard Dataverse ([https://doi.org/10.7910/DVN/SZQEHP](https:/doi.org/10.7910/DVN/SZQEHP)). Due to participant privacy and data use agreements, raw GPS trajectories are not publicly shareable. The replicable workflow (H3-MOSAIC-Multimodal Generative AI for Semantic Place Detection on H3 Grids) is shared on KNIME Hub ([https://hub.knime.com/s/qXKPJq41jazq9DfZ](https:/hub.knime.com/s/qXKPJq41jazq9DfZ)).

Declarations

Ethics approval and consent to participate

This study analyzes de-identified high-frequency GPS data from research studies conducted by the Division of Digital Psychiatry at BIDMC with the open-source mindLAMP smartphone app. The secondary analysis presented here was conducted on de-identified records aggregated to H3 grids to enhance geo-privacy and all images shared are derived from simulated data to protect privacy.

Consent for publication

Not applicable. This manuscript does not contain any individual person’s data (images, videos, or identifying details).

Competing interests

The authors declare no competing interests.