How to Configure Zingg to Export Only Duplicated Results in CSV Files

·Sep 09, 2024 09:28 PM

Question 4: I runned Zingg successfully on my local and got 4 csv file for the results. Wondering is there a setting I can turn on so that it only export the duplicated results in csv file.

4 comments

· Sorted by Oldest

Sonal G.
·
You can run pyspark query on the result and select the rows you care to export

Nicolás N.

·

#!/bin/bash

# Define output file
output_file="combined_output.csv"

# Get the first file header
header=$(head -n 1 tmp/part-00000*.csv)

# Write header to the output file
echo "$header" > $output_file

# Append all files to the output without the header
for file in tmp/part-*.csv; do
    tail -n +2 "$file" >> $output_file
done

echo "Combined CSV created: $output_file"

🙌1

Nicolás N.
·
I had the same requirement, so now I have this bash script to combine them files
👍2
Sonal G.
·
Cool, thanks for sharing

Sonal G.
·
You can run pyspark query on the result and select the rows you care to export

Nicolás N.

·

#!/bin/bash

# Define output file
output_file="combined_output.csv"

# Get the first file header
header=$(head -n 1 tmp/part-00000*.csv)

# Write header to the output file
echo "$header" > $output_file

# Append all files to the output without the header
for file in tmp/part-*.csv; do
    tail -n +2 "$file" >> $output_file
done

echo "Combined CSV created: $output_file"

🙌1

Nicolás N.
·
I had the same requirement, so now I have this bash script to combine them files
👍2
Sonal G.
·
Cool, thanks for sharing