DEV Community

#dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Your Iceberg tables are rotting: we measured it, then built a fix (free, Apache 2.0)
aamir_dataeng profile

Your Iceberg tables are rotting: we measured it, then built a fix (free, Apache 2.0)

#dataengineering #opensource #python #iceberg
Comments
6 min read
Null rate data quality check in SQL, per column
refaeldakar profile

Null rate data quality check in SQL, per column

#sql #postgres #dataengineering #monitoring
Comments
2 min read
Power BI Service on ClickHouse, without an ODBC driver on your gateway
sergeisemenkov profile

Power BI Service on ClickHouse, without an ODBC driver on your gateway

#powerbi #clickhouse #semanticlayer #dataengineering
Comments
5 min read
AI in Data Engineer Interviews: Banned, Allowed, or Graded?
datadriven profile

AI in Data Engineer Interviews: Banned, Allowed, or Graded?

#dataengineering #interview #career #ai
Comments
9 min read
openFDA's count= aggregation shows you the top 1,000 values and never tells you it stopped
fetchsmith profile

openFDA's count= aggregation shows you the top 1,000 values and never tells you it stopped

#webscraping #api #opendata #dataengineering
Comments
6 min read
Your Python Script Works on 1,000 Rows. Then the Real CSV Arrives.
emilywoodsnyc profile

Your Python Script Works on 1,000 Rows. Then the Real CSV Arrives.

#python #dataengineering #programming #interview
Comments 1
8 min read
Your Batch Job Said "Success." Here's How to Check the Data Actually Arrived
askkarenai profile
KAREN

Your Batch Job Said "Success." Here's How to Check the Data Actually Arrived

#dataengineering #database #devops #beginners
Comments
3 min read
Running CDC in Production: The WAL Slot Trap
pipelinewhiteboard profile

Running CDC in Production: The WAL Slot Trap

#postgres #dataengineering #devops #aws
Comments
7 min read
Zero-downtime partition swaps on Athena: build in a new prefix, flip with SET LOCATION
csotelo profile

Zero-downtime partition swaps on Athena: build in a new prefix, flip with SET LOCATION

#aws #dataengineering #python #athena
Comments
5 min read
dlt's Auto-Schema Pipeline: How Python Data Loaders Infer Structure Without Predefined Models
mech_app_ai profile

dlt's Auto-Schema Pipeline: How Python Data Loaders Infer Structure Without Predefined Models

#database #dataengineering #opensource #python
Comments
4 min read
A Green Spark Job Isn't Enough: Building a Fail-Closed Pipeline Health Monitor in Python
rupak_kulkarni_08b45c5846 profile

A Green Spark Job Isn't Enough: Building a Fail-Closed Pipeline Health Monitor in Python

#python #dataengineering #devops #testing
Comments
6 min read
Hadoop Ecosystem Overview
godofgeeks profile

Hadoop Ecosystem Overview

#bigdata #data #dataengineering
Comments
8 min read
Web Scraping vs Official API: Which Should You Use?
aqib_javaid profile

Web Scraping vs Official API: Which Should You Use?

#webscraping #apiintegration #dataengineering #architecture
Comments
6 min read
Equality deletes have no address, and that's why your Iceberg reads crawl
kyien profile

Equality deletes have no address, and that's why your Iceberg reads crawl

#dataengineering #iceberg #sql #flink
Comments
7 min read
Iceberg vs Parquet: do you need a table format?
mohammed_arshadansari_f2 profile

Iceberg vs Parquet: do you need a table format?

#dataengineering #database #data #analytics
Comments
3 min read
👋 Sign in for the ability to sort posts by relevant, latest, or top.