5,697 questions
0
votes
0
answers
29
views
How to stream from an merged layer (apply_changes) table into a downstream silver layer as stream and not Materialized View (MV)? [closed]
The Architecture: I am implementing a Delta Live Tables (DLT) pipeline following the Medallion architecture.
Landing: Auto Loader ingesting raw files (JSON/CSV).
Bronze Layer: Uses dlt.apply_changes() ...
-6
votes
0
answers
49
views
sonar scanner not working says its not a file or directory | sonarqube setup on local VM [closed]
I am learning CI/CD pipeline I made a custom Bitbucket runner on my local VMware-based VM and i set the socarqube server and scanner both on the VM but still it's not working it says it's not a file ...
-1
votes
1
answer
44
views
Getting unrecognized arguments: --federated-token in creating pipeline in microsoft/MLOpsPython
I am using this repo to create a mlops pipeline in Azure Devops.
When i tried to run the CI pipeline, I am getting the unrecognized Federal Token. I asked chatgpt, it says to update the cli version. I ...
0
votes
1
answer
29
views
How to pass variable in Classic ADO pipeline between two Powershell Scripts
I have two tasks:
The 'Set Solution Name does this:
$SolutionName = "UnknownSolution"
Write-Host "::vso[task.setvariable variable=SolutionName;]$SolutionName"
Write-Host "...
2
votes
2
answers
188
views
how to get PowerShell to gracefully accept a multi line, array, system.object input to a `[string]` parameter?
ff
I have a function foo, it needs to accept a string, this string can be passed to it directly by typing at the terminal, from a variable or from the clipboard.
function foo{
[CmdletBinding()]
...
0
votes
0
answers
49
views
DLT handling api call exceptions
I am fetching data from multiple gitlab APIs : the repositories I have access to, the members and the associated runners. Some api calls will return a 403 forbidden error.
I need to manage those ...
0
votes
1
answer
92
views
Pull the docker image in GitLab registry from pipeline
I made a Kernel project (v5.15.15) in my GitLab account and setting up build pipeline.
Did some search and made a following .gitlab-ci.yml file.
stages: # List of stages for jobs, and their ...
2
votes
3
answers
172
views
How to pipe a data file with multiples read values to a bash script with a delay between them? It is possible to display the info read from file
This is my main script:
#!/bin/bash
read -p "Name: " NAME
read -p "Age: " AGE
read -p "Country: " COUNTRY
echo "Your name is $NAME. You are $AGE years old. You ...
1
vote
1
answer
52
views
unable to find class for annotation @Field in jenkins pipeline script from scm
I'm encountering an issue with my pipeline script when running it from SCM. The @Field annotation isn't being recognized, even though I'm using the correct import statement (import groovy.transform....
2
votes
2
answers
112
views
PowerShell pipeline adds newline when passing to the next command
I'm trying the the Rust version of coreutils on my Windows 10 system. I renamed coreutils.exe to co.exe so that I can type less. od is the Windows version of the Linux command od [from linux man site =...
0
votes
0
answers
33
views
GitLab CI/CD pipeline Salesforce code coverage test
I'm currently building a CI/CD pipeline in GitLab, and I'm running into trouble with integrating a code coverage check.
What I’ve Got So Far:
A working GitLab Runner (on Windows).
An Apex class and a ...
0
votes
0
answers
40
views
How to process an array as a single object with a Powershell ValueFromPipeline parameter? [duplicate]
I have a task that requires me to collect an array, refine it by removing junk entries, and finally process each line. Breaking each of those tasks into functions is fine, but setting up the script to ...
1
vote
1
answer
137
views
How to pull images from AWS ECR via pull through cache in .gitlab-ci
I'm using a gitlab pipeline, where its base image repo is in AWS ECR. I've configured AWS ECR pull through cache where my upstream registry is docker hub.
In my runners, I've added this
[[...
0
votes
0
answers
48
views
Azure Pipeline Stages to Run show "Unable to load the pipeline's stages."
I checked some netizens' errors.
Currently I've tried change Triggers>YAML>Insert something to "Default agent pool for YAML"
But seems "Unable to load the pipeline's stages" ...
0
votes
0
answers
57
views
Unable to upload artifacts into Nexus for my Jenkins freestyle job
This is the issue that I'm still having it with Nexus (Situation: I can't deploy into nexus - I can't upload artifacts into nexus for any of my freestyle project or pipeline project in Jenkins)
By the ...
0
votes
0
answers
48
views
Jenkins Priority Sorter Plugin Missing Job Priorities Configuration Section
I just installed the Job Priorities plugin for my Jenkins instance, but I can't for the life of me figure out where the Job Priorities configuration section is located.
The plugin is installed, ...
2
votes
0
answers
49
views
iOS security keychain issues in pipeline no valid identity
I am trying to use Fastlane in order to publish the app. In my pipeline script, I’m doing the following steps:
security unlock-keychain -p "$KEYCHAIN_PASSWORD" ~/Library/Keychains/login....
0
votes
0
answers
64
views
How to automate manual data downloading in Python/R?
I work with clinical data at a company that, until I arrived, didn't have a data policy. Currently, raw data extraction relies solely on manually downloading CSV/Excel files from an internal portal ...
3
votes
2
answers
96
views
Is there a way to add more variables to the pipeline from the 'begin' section of a script?
Is there a way to dynamically add more variables to the pipeline in the "begin" section of a script?
I have a script which can accept data in three different ways:
Value as parameter
Pipe ...
4
votes
3
answers
98
views
Evaluate transformations with the same model in scikit-learn
I would like to perform a regression analysis and test different transformations of the input variables for the same model. To accomplish this, I created a dictionary with the different pipelines, ...
0
votes
0
answers
92
views
How to add ID to streaming table in Databricks?
I want to create DLT pipeline like following. I have streaming bronze table which looks like:
Name
City
Country
John
Tokyo
Japan
Mary
New York
USA
Sam
Tokyo
Japan
Now I need two more tables. Main ...
2
votes
1
answer
104
views
How to convert cmd pipe to powershell in context of binary data (ffmpeg and streamlink)
I'd like to convert a pipe of video stream between streamlink and ffmpeg from cmd to Powershell. But from what I searched powershell only supports piping objects.
Is there any way to reproduce this in ...
0
votes
0
answers
35
views
PSORTb Missing output file(s) error in Nextflow process
I'm a beginner here. I've built a few nextflow workflows for other tools before. The command for PSORTb requires you to specify the directory where the output in stored and this is where I feel the ...
0
votes
1
answer
124
views
MAUI with Entity Framework Core unable to build in Azure Pipeline
I am unable to build my MAUI project that uses Entity Framework Core with migrations in Azure; I am able to build it locally.
I get this error when building in Azure:
C:\hostedtoolcache\windows\...
0
votes
1
answer
197
views
Script issue: jobs:run-tests config contains unknown keys: id_tokens, secrets, image, artifacts, coverage
Hello,
I am updating a GitLab CI/CD pipeline to add a new job.
Unfortunately, when I commit and the pipeline run, I got this error message:
jobs:run-tests config contains unknown keys: id_tokens, ...
0
votes
1
answer
71
views
Azure devops yaml pipeline if statement check for empty string
We are trying to run a task conditionally, but the condition always seems to be true, even if it shouldn't. We have 3 files that are passing a parameter from the devops variables down to each file:
...
0
votes
0
answers
38
views
Force fail build on remote agent disconnected
how can I reliably fail a Jenkins pipeline build if the remote agent disconnects during execution?
binding.setVariable('selectedNode', "")
pipeline {
agent {label: LOCAL_NODE}
...
1
vote
0
answers
164
views
Unable to install Azure Self hosted agent as service in Windows server
I am trying to install Azure Pipeline Agent in "Windows Server 2025"
I downloaded the latest agent file "vsts-agent-win-x64-4.255.0" from Azure Devops Page.
I created agent folder ...
2
votes
0
answers
105
views
Any efficient solutions to address the dependency that 'the j-th step of the i-th task is dependent on the (j+1)-th step of the (i-1)-th task'?
I have a workload consisting of a series of tasks and each task can be divided into k steps. So far, the j-th step of the i-th task depends on the j-th step of the (i-1)-th task. This can be easily ...
0
votes
0
answers
44
views
Active Choices - Get all selected checkboxes - Formatted HTML
I have a Jenkins project where I want to display a list of services as checkboxes, then pass the selected services to the pipeline and use an Ansible playbook to handle them (for now, just to restart ...
1
vote
1
answer
62
views
How to remove progress bar in targets package in R
In the targets package in R, does anyone know if it's possible to remove the progress when running tar_make(). I plan to save the logs as a text file but since the progress bar is being printed every ...
0
votes
1
answer
72
views
Get the result of the triggered pipelines
I have a main azure pipeline (pipelineFinal) that should be triggered when a pipelineA is finished.
pipelineA:
trigger: none
jobs:
- job: BuildA
displayName: "Build Pipeline A"
pool:
...
1
vote
1
answer
34
views
Does Boost Redis support Autopipelining
Many redis clients claim to support Auto-pielining. Does Boost redis support that? If yes, how can it be enabled?
1
vote
0
answers
155
views
How to implement timeout and retry for long-running Hugging Face model inference in Python?
I'm working with a locally hosted Hugging Face transformers model (mistral-7b, llama2-13b, etc.), using the pipeline interface on a GPU server (A100).
Sometimes inference takes much longer than ...
0
votes
1
answer
75
views
How to generate unique IDs during S3-to-SingleStore columnstore ingestion (AUTO_INCREMENT not supported in SHARDed table)
I’m working on a data pipeline that ingests JSON data from an S3 bucket into a columnstore table in SingleStore.
The challenge is that I need to assign a unique ID (Primary Key) to each record during ...
0
votes
0
answers
30
views
Output file not getting created/saved - BitBucket Pipeline
This is the definition from my Bitbucket YML file
- step: &check_code_quality
name: Run Quality Check
script:
- echo $BITBUCKET_COMMIT
- git fetch origin --no-tags --depth=100 &...
1
vote
1
answer
466
views
Gitlab multi project pipeline - This job could not start because it could not retrieve the needed artifacts
So I have another project which contains artifacts I want to reuse on my project/job/pipeline. So I try to trigger the "another-project"'s pipeline to build the artifacts which I want to ...
0
votes
0
answers
62
views
Apply Merge from Bronze DLT Table into Silver DLT Table with some light cleaning? (Traditional Upsert style)
I am trying to create a DLT pipeline and this is my first time doing it. What is the best possible way to satisfy my following requirements. I am fully aware that the path I am choosing may not be ...
0
votes
1
answer
59
views
Differential Code Coverage in Azure DevOps (ADO) for logical monorepo
I'm working on a project, where I have a pull request pipeline for a logical monorepo.
For this I'm using the git submodule functionality to pull a few repositories into mine, and building them.
I use ...
1
vote
4
answers
139
views
How to fix the issue: ##[error]Error: No package found with specified pattern: /home/vsts/work/1/s/**/*.zip
I have a pipeline for building and deploying an angular app. It builds successfully but fails during deployment with the error:
##[error]Error:
No package found with specified pattern:
/home/vsts/...
0
votes
0
answers
51
views
Jenkins in Docker Cannot Find NPM: npm: not found (Exit Code 127)
Description:
I am running Jenkins inside a Docker container on Windows 11. I have the Ubuntu app installed from the Microsoft Store, which already has npm version 8.5.1 installed.
However, when I run ...
0
votes
1
answer
62
views
Is there a way to create Snowpipe or Ingestion in Snowflake for a public Azure Blob dataset?
I have done Snowpipe before for a azure Blob before, but since it was private I had access to tenant_id and all authentications. Maybe the question seems backward but when a storage is public, Can we ...
0
votes
1
answer
60
views
When Merging the code, all pipeline getting triggered
I have admin api and public api. 2 seperate branch has been created and 2 different pipelines were created. When i merge the public api code, the admin pipeline also getting triggered, which should ...
1
vote
2
answers
957
views
How to pass {{job.trigger.time.iso_date}} - 1 day as a Databricks pipeline parameter?
I'm running a Databricks Workflow and using {{job.trigger.time.iso_date}} to pass the job execution date as a parameter. However, I need to pass the previous day's date instead (i.e., {{job.trigger....
0
votes
1
answer
97
views
Snakemake - Call multiple scripts in a rule
I have a snakemake rule where I call a python script with the script keyword:
rule merge_results:
input:
[...]
output:
path_results
script:
"merge_results.py&...
1
vote
0
answers
468
views
Can I with SDL3 GPU load vertexes, normals, colours, UVs in different buffers?
gLTF and probably other formats have different "buffers" that are long series of data. Unless the gLTF have a "stride", that is interleaved data, every buffer store things ...
0
votes
3
answers
549
views
Auto-Start Self-Hosted Azure DevOps Agent After Restart
We have a self hosted agent running our builds. We have configured it to auto shutdown everyday. In the morning, we always have to go and run the pipeline agent. Is there a way we can make it run ...
0
votes
0
answers
40
views
Golang & opentelemtry log collection pkg : why is my pipeline not sending logs from input operator to output operator?
recently i started working on log files and handling then i tried to write a simple program trying to tail a log file and send the log lines to a new log file. I used opentelemetry-log-collection ...
0
votes
0
answers
23
views
LinearRegression object has no attr coef_ (in pipeline)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
num_features, bin_features, cat_features = split_features(X)
preprocessor = ColumnTransformer([
('num', ...
1
vote
2
answers
70
views
Have a dict type variable in Azure yml pipelines
I have a yml pipeline and I would like to have a dict variable like the 'files' variable below
variables:
- template: ../../ci-cd/ado_variables/vars.yml
- template: ../../ci-cd/ado_variables/${{ ...