Newest Questions
36,324 questions
0
votes
0
answers
5
views
Parsing and chunking conversations from chats
I want to chunk conversations from chat streams, both public group chats and direct messages. The goal is to pick up messages on a specific topic, group them into a single conversation with the ...
- 111
0
votes
0
answers
6
views
GAN: Label and image as input
I'm trying to train a GAN to generate new images. I tried training a StyleGAN, but the results were not good. The FID was too high, and the new images didn't look realistic. The dataset contains ...
- 1
0
votes
1
answer
5
views
When is the normalization or standardization is needed?
I have a complex data with wide range and skewed distribution, but i'm planning to use Regression Tree. So do I need to normalize or standardize my dataset?
- 195
0
votes
0
answers
11
views
Document Classification Task for Review Paper References
everyone!
I'm doing for the first time research on how well LLMs and DL models can structuralize scattered data, through NER and RE. We are using a review paper on a domain that has no ontologies or ...
0
votes
0
answers
9
views
Unable to improve model performance
I am building a model which contains data from multiple clients packed into a single dataset. Model predicting success rate for every row of event. However, there is a particular feature call it ...
- 1
0
votes
0
answers
4
views
Why is Logi Symphony missing the "Define text formatting" option when importing data from a CSV?
Issue
In Logi Symphony 25.2.0.1001-w, when configuring a measure in a Metric Set, there is normally a "Define text formatting" option to control how a numeric value is displayed. (e.g., ...
- 181
0
votes
0
answers
11
views
Stylegan preprocess
i have a dataset and in each picture there are many things. What should i do for train of GANs (styleGAN) for Preprocessing that the model distinguish the things in the Picture. Now the result is not ...
- 1
3
votes
0
answers
26
views
Why can zero-inflated generalized Poisson model not capture the overdispersion in the count data?
I have fitted a zero-inflated Poisson (ZIP) model to my count data with an excess of zeros (figure 1). Since it didn't capture the overdispersion in the data, and overdispersion is indeed ...
- 185
2
votes
0
answers
32
views
Kaggle TPUv5e8 7 times slower than v3
I am trying to learn Kaggle TPU and I am migrating a Flower Classification notebook from an older TPU v3-8 environment to the new TPU v5e-8 (TPU VM) environment on Kaggle. I was trying to migrate this ...
2
votes
0
answers
31
views
Convolution Kernels - Understanding dilation
I have a fairly good understanding of the basic convolution operation and how padding & stride affect its output, but dilation is something new to me.
So I am going to make some broad statements ...
3
votes
2
answers
47
views
What if my model performs well even on small training set?
I was working on a dataset which is available on kaggle. At first, I split my data with a train-test ratio of 90:10. Then I fit 24 different models (6 different regressors with 4 different ...
- 31
1
vote
0
answers
11
views
Combining (interrupted) time-series analysis and machine learning to predict how an intervention will perform
I am working on a project that requires machine learning analysis, but I'm new to the field and still learning about different models. I just wanted to ask the community about the best model or ...
- 11
5
votes
1
answer
48
views
When should we avoid balancing an imbalanced dataset?
I am working on a network security-related project, in which I have to build a deep learning model to detect a specific attack. It's about detecting whether a network system of an organisation is a ...
- 53
2
votes
0
answers
25
views
How can I evaluate an LLM’s reliability for use in high-stakes, risk-sensitive decision support?
I’m working with a large language model that has been configured to behave conservatively in high-stakes contexts:
it refuses unsafe or ambiguous user requests,
prioritizes client welfare over ...
- 21
2
votes
1
answer
16
views
Possible to Improve Reconstruction Quality and Accuracy with VAE?
I am training a VAE architecture on microscopy images. Dataset of 1000 training images, 253 testing images. Images are resized to 128x128 input or 256x256 input from original resolution which is ...
- 21
0
votes
0
answers
9
views
What is the problem with symmetry in GloVE motivation?
I am currently studying GloVe paper about word embeddings. link
In Section 3 The GloVe Model this model is derived from several desiderata, one of which confuses me. It is around Equation 3 which ...
- 203
0
votes
0
answers
7
views
Sequence generation model produces incorrect, but coherent outputs
My model takes in an image of a handwritten equation and converts it into its LaTeX representation. In order to do this, it uses a ResNet50 pre-trained model for feature extraction and a transformer ...
- 71
0
votes
0
answers
12
views
Unexpected Feature Importance Pattern in Random Forest Classification of MNIST Digits 0 and 1
I performed Random Forest–based feature importance analysis on the MNIST dataset, focusing only on digits 0 and 1.
When I visualize the importance map (see image below), it doesn’t resemble the ...
0
votes
1
answer
16
views
Is there a fast method from sampling from document embeddings to *maximize* pairwise distances?
I have a large set of document embeddings, and I would like to sample a subset where the median or average pairwise distance is maximized. The idea here is to get a more balanced sample set where long ...
- 173
0
votes
0
answers
19
views
Can I use point biserial for data that violates the assumptions for Pearson's correlation?
Since point biserial is just a special case of Pearson's correlation, can I use it for data that violates the assumptions for Pearson's correlation (outliers, non normal distribution, etc)?
If not, ...
- 1
0
votes
0
answers
10
views
How can I group transcribed phrases into meaningful chunks without using complex models?
I have a large set of phrases obtained via Azure Fast Transcription, and I need to group them into coherent semantic chunks (to use later in a RAG pipeline).
Initially, I tried grouping phrases based ...
- 1
0
votes
0
answers
17
views
How to extract my fingerprint from my laptop's finger sensor
So like I have a bunch of fingerprint as a data set (my college gave me). Now I want to use these fingerprint as datasets and train a model to understand the different things. That is beside the point....
- 1
1
vote
0
answers
27
views
Word cloud - tools to generate a WordCloud?
i need to generate a word-cloud based on a free text dataset. I have tried several options, and i find the most to be of poor quality. Tools i tried:
PowerBI
https://www.freewordcloudgenerator.com/...
- 141
1
vote
0
answers
46
views
MMM model vs Monte Carlo
I was given a project where only using Net Media Value and possibily audience considered , I have to try to estimate sales and unit return of media investment. I was asked to try to apply a Monte ...
- 11
8
votes
1
answer
2k
views
Use of training data that has been labeled by the AI model itself
I'm a software engineer working with medical device AI models that predict diseases and other conditions. For the most part, I don't design the models but I help with getting FDA clearance for them. ...
- 183
0
votes
0
answers
14
views
How to prepare training data to predict churn when contacts have varying months of pre-inactivation?
Problem statement
I need to model contact inactivation, defined as a contact having 12 consecutive months with no touchpoints. At any given scoring date, contacts in the base can have different ...
0
votes
0
answers
16
views
How to handle unstable best_iteration in LightGBM when using Optuna for hyperparameter optimization?
I'm using Optuna to optimize LightGBM hyperparameters, and I'm running into an issue with the variability of best_iteration across different random seeds.
Current ...
0
votes
1
answer
28
views
What could be a dataset in which the presence of an outlier or a null value dramatically affects the performance of the decision tree?
I am tasked with giving an example of a dataset in which the presence of an outlier or a null value dramatically affects the performance of
a decision tree. I've searched and searched the web and I ...
- 1
1
vote
0
answers
13
views
How can we learn a single model that handles causal inference, fairness, and domain shifts together?
I want to know if it’s possible to make one machine learning model that can:
Find causal relationships,
Stay fair to different groups, and
Work well when the data changes over time or across domains.
...
0
votes
0
answers
7
views
Java heap space - even only 1.5Gb of 5.8 used and df size is 3Gb
Why am i getting java.lang.OutOfMemoryError: Java heap space even when i have a plenty of memory.
So my simple code that create dataframe from input data, so no ...
- 101
0
votes
0
answers
7
views
Sales territory optimization
Hi Currently I am working on sale territory optimization. I am using kmean but can not deal with some constrain that are set by business such as planning territory to deal with constrain such as
max ...
1
vote
1
answer
51
views
Metric Learning - MLKR - Fail to reproduce known result
I am trying to replicate the results in this Metric Learning for Kernel Regression, using the metric-learn python package to perform the MLKR computation.
I am completely failing to replicate one ...
- 81
0
votes
0
answers
7
views
RStudio (Multiple Correlation Analysis)
I have a population of customers (n=50,000) that I would like to analyze (in RStudio) and predict which product is the Next Most Likely (NML) to be bought, based on the current populations active ...
0
votes
1
answer
57
views
Which ML model for Medicine Dosage Prediction?
I want to develop a machine learning model that predicts the correct medicine dosage required to keep a specific lab value within the target range of 5 to 7. I also have several other predictor ...
- 21
3
votes
1
answer
101
views
K Fold Cross Validation - Manual Exploration or Use the result?
I have a dataset which I split into training, testing, and out-of-time sets. Then I feed my training set into K Fold CV.
I understand that K Fold Cross Validation is used as a method to select the &...
- 131
2
votes
2
answers
125
views
Is Logistic Regression actually used for regression?
This is a question asked in my homework assignment, the full question is "Is Logistic Regression actually used for regression (predicting a continuous value)? If not, state what task it really ...
- 23
0
votes
1
answer
44
views
Correlated Features In Classificatification Problem
I'm working on binary classification problem to identify struggling students in university. I have some features that are correlated such as high_school_grade_1 that represents 75% of ...
3
votes
1
answer
56
views
LDA linearly separates 2 out 3 classes, what insight does it provide?
My dataset consists of board games data: each board game is rated with a categorical variable (low, medium, high).
I've plotted the LDA projection to check whether classes are linearly separable. The ...
- 33
0
votes
0
answers
11
views
How to calculate weights for two parallel transformer outputs
I have a model where I incorporate additional input to language sequence data. I put these two data into two different transformers then combine them using addition. Simply, I combine them with a ...
- 121
0
votes
1
answer
21
views
Should Hyperparameter Optimization Be Equalized by Trials or Compute Time?
Say I have two different models with different hyperparameters and I want to compare the performance of both models on some dataset.
One model is much simpler than the other and, therefore, if I were ...
0
votes
1
answer
40
views
Regarding text preprocessing in BERT
Hi i had a doubt regarding the nlp part in transformers that
is there any preprocessing required in Bert models?
if preprocessing requires what type of preprocessing requires could any one clarify ...
1
vote
0
answers
35
views
Combining Embeddings and Ontology (DAG) in Visualisation
How can I visualise a hierarchical ontology of items in embedding space, combining text embeddings with the graphical structure? (Something similar to the example below)
I have a hierarchical ...
- 111
1
vote
0
answers
34
views
How to identify and quantify main tendencies across participants from cluster membership heatmaps?
I'd appreciate your thoughts on the following problem.
I've created a heatmap plot (attached) showing the cluster membership ratio for each participant (in separate subplots) and condition (η).
Now, I'...
1
vote
1
answer
27
views
Reinforcement Learning vs Unsupervised Learning
I am trying to understand what are the main differences between reinforcement learning and unsupervised learning when training ML models.
I know the main difference is that reinforcement learning aims ...
- 21
0
votes
0
answers
14
views
Comparing self-selected populations
Suppose I am a free consultant for a single industry of 300 total companies, offering operational consultations. In 2024, I consulted for 100 companies. To expand my practice, I'd like to show that my ...
- 101
0
votes
0
answers
17
views
Reporting results with a little high standard deviation within Nested CV
I'm working on a binary classification problem to identify struggling students, my dataset contains 10 features and 200 samples, I implement Nested CV, the distribution of the target variable is 58%/...
3
votes
1
answer
41
views
How Do You Balance Feature Search Strategy and HP Optimization Cost?
What I’m trying to figure out
I'm working on a machine learning project and would love to hear your thoughts on two things:
A. How to prioritize feature exploration
B. Whether to fix hyperparameters (...
- 31
0
votes
0
answers
57
views
Why bias value is critical to successful learning?
Given basic elements of a neuron(as below) with a bias value:
I learnt that, a bias value allows you to shift the activation function(say sigmoid function) to the left or right, which may be critical ...
- 199
1
vote
0
answers
13
views
Scalar versus 2-element output for binary classification models
When building a binary classification model using a neural network, you have two options for outputs: output a single number from 0 to 1 using sigmoid activation, or output a probability distribution ...
- 21
2
votes
0
answers
12
views
What is the best framework or ML architecture for animal social behavior?
i want to build models to identify over 30 different social and non-social behaviors in pairs and groups of co-housed mice, based on markerless motion capture of their movements in top-down video ...
