Priyanka Shelar

I never planned to build data infrastructure for trauma surgeries.

I never planned to help thirty million Americans breathe better at night.

I never planned to build systems that track life-saving medical devices from manufacturing facilities to operating rooms across the United States.

But that is exactly what fourteen years in data engineering—built on a foundation of curiosity, persistence, and a willingness to do the unglamorous work that makes everything else possible—has led me to.

I am Priyanka Shelar. I grew up in Mumbai, India, in a family that believed deeply in education and even more deeply in hard work. I earned my undergraduate degree in Information Technology, then a Master’s in Software Engineering from BITS Pilani—one of India’s most competitive engineering institutions. Then I made the decision that would define the next chapter of my life: I came to the United States to build a career in data engineering.

What followed was not a straight line. It never is. But it was a journey that taught me more about resilience, reinvention, and the quiet power of doing technical work that genuinely matters than any career plan could have predicted.

The Work Nobody Sees—and Why It Matters

Data engineering is invisible work.

When a doctor sees that a patient’s sleep therapy compliance has dropped and makes a call that prevents a hospitalization, nobody thinks about the data engineer who built the pipeline that made that insight possible.

When a hospital receives the surgical device it needs for a trauma procedure on time, nobody thinks about the engineer who built the supply chain data infrastructure that tracked that device from manufacturer to operating room.

When a Fortune 500 company makes a marketing investment decision that creates jobs and drives economic growth, nobody thinks about the engineer who built the analytics platform that informed that decision.

We are the invisible layer. The foundation. The infrastructure that makes everything else possible.

And I have spent fourteen years building that infrastructure—not because it makes me famous, but because I genuinely believe that data engineering, done well and applied to the right problems, is one of the most important contributions a technical professional can make to the world.

The Moment That Changed Everything

My first major project in the United States was working on a healthcare informatics platform for one of the world’s leading sleep therapy companies.

Sleep apnea affects more than thirty million Americans. Left untreated—or treated with a device the patient stops using—it contributes to heart disease, stroke, diabetes, and premature death. The clinical challenge is not the therapy itself; the therapy works. The challenge is keeping patients on it.

Our team built a platform that could predict—with approximately 95% accuracy—which patients were at risk of discontinuing therapy before they actually stopped. Clinical staff could see their entire patient population’s compliance status in real time, reach out proactively, and intervene before health deteriorated.

I remember the moment I read feedback from a client stakeholder after the platform launched. She wrote that Territory Managers, Solutions Consultants, and Regional Managers were “amazed” to finally have this visibility—that they had never been able to work this way before.

I printed that email out. I still have it.

Because in that moment, I understood something that has guided every project I have worked on since: data engineering is not about databases, pipelines, or ETL processes. It is about people. It is about the patient who keeps using their CPAP device because a clinical coordinator called them at the right time. It is about the nurse who can finally see which patients need attention most urgently. It is about the family member who doesn’t receive the call that their loved one had a stroke.

That is what we build when we build well.

Learning to Work at Scale

From healthcare, I moved to what I still consider the most technically demanding project of my career: the Johnson & Johnson Enterprise Data Grid.

J&J is one of America’s largest healthcare companies. The Enterprise Data Grid is the data backbone of its global supply chain—processing data from over 65 ERP systems worldwide, handling more than 45 terabytes of data monthly, and tracking pharmaceutical products and medical devices across manufacturing, distribution, and delivery to hospitals and clinical settings across the United States.

I was part of a small, elite team selected specifically for the technical complexity of the work. I was one of the only women on that team, and also the only engineer managing the streaming of over 4,000 data tables across more than 20 critical projects simultaneously.

I will not pretend that was easy. The technical challenges were significant. The learning curve was steep. And navigating those environments as a woman—often the only woman in the room—added a layer of complexity that the technical problems alone did not prepare me for.

But I learned something important in those years: competence speaks louder than anything else. When the systems you build are reliable, when the data you deliver is trustworthy, when the monitoring dashboards you create become tools everyone depends on, the conversations about belonging tend to quiet down on their own.

I also learned that building data infrastructure at national scale—systems that track medical devices used in surgeries, ensure pharmaceutical supply chains remain intact, and give hospitals visibility into critical equipment—is not just a technical achievement. It is a genuine contribution to the people those systems serve.

Reinvention at NYU

In 2021, I made a decision that surprised some people: I went back to school.

I applied to the Master of Science in Computer Science program at New York University’s Tandon School of Engineering. I was accepted with an $8,000 merit scholarship.

People asked me why. I already had fourteen years of experience. A successful career. Why go back?

Because I knew the field was changing. Because I could see that artificial intelligence and machine learning would reshape data engineering in ways that required deeper theoretical foundations. Because I believed the best way to stay at the frontier of my field—and continue building things that matter—was to invest in my own learning, even when it was hard, even when it meant balancing graduate coursework with professional responsibilities, even when I was significantly older than many of my classmates.

NYU gave me more than a degree. It gave me a framework for thinking differently. It gave me a community of researchers and practitioners who challenged my thinking. And it gave me access to opportunities I would not otherwise have had.

One of those opportunities was an internship at Amazon Robotics, where I built data lineage infrastructure for an analytics platform supporting over 500 engineers and data scientists monthly. Another was joining Analytic Partners as a Lead Data Engineer, serving Fortune 500 companies across industries with AI-ready commercial analytics infrastructure.

What I Am Building Next

I have spent fourteen years building data infrastructure for other organizations. The question I am focused on now is: what comes next?

The answer, I believe, lies at the intersection of healthcare, education, and technology—and the communities I want to serve. I am focused on expanding my contributions beyond any single employer or project, applying my data engineering expertise to problems that matter most: improving patient outcomes, strengthening educational systems, and building AI-ready data infrastructure for institutions that increasingly depend on it.

I am also focused on giving back: mentoring the next generation of women in data engineering, writing and speaking in ways that make technical work accessible, and becoming the kind of visible, technically credible woman in this field that I needed—and often struggled to find—when I was starting out.

The work ahead is significant. The opportunity is real. And I am more energized about what is possible than I have been at any point in my career.

What I Would Tell My Younger Self

If I could go back to the young woman in Mumbai deciding whether to take the leap—to leave everything familiar and build a career in a country where nobody knew her name or her family—here is what I would tell her:

The unglamorous work is the important work. Data pipelines are not glamorous. ETL processes are not glamorous. Monitoring dashboards, CDC frameworks, and data lineage systems are not the things that make headlines. But they are the foundation on which everything else is built. Do that work with the same care and rigor you would bring to anything you wanted to be proud of. Because somewhere on the other side of that work, there is a patient who gets the call that keeps them healthy, a surgeon who receives the device that saves a life, a teacher who gets the insight that helps students learn.

Invest in yourself relentlessly. Go back to school when the field demands it. Get the certifications. Write. Speak. Even when you are nervous. Every investment compounds in ways you cannot predict.

Do work that matters. Not every project will change lives. But if you orient your career toward healthcare, education, supply chain, and public service, you will find that the motivation runs deeper than any title or paycheck ever could.

To Every Woman Reading This

You belong in this field.

Not because someone gave you permission, and not because the numbers are improving (though they are, slowly). But because the problems data engineering and AI are solving—in healthcare, education, supply chain, government, and beyond—are too important to leave to any single demographic or perspective.

Your experience as a woman—navigating complexity, managing competing demands, communicating across difference, and persisting in environments not designed for you—is not a liability in this field. It is a qualification.

The world needs more women building the infrastructure the future runs on.

I hope you will join me in building it.

Change Data Capture at Scale: Lessons from Building a National Healthcare CDC Framework

When most engineers think about Change Data Capture (CDC), they think about syncing a few database tables or keeping a data warehouse up to date. That's a perfectly valid use case.

But what happens when CDC isn't just a technical pattern—it’s the backbone of a system that tracks medical devices used in trauma surgeries across an entire country?

That's the reality I worked with as a Data Engineering Specialist at one of America's largest healthcare companies. Over three years, I helped architect and implement a CDC framework that processed 55 TB of data monthly, ingested changes from 64 heterogeneous ERP systems, and served as the critical data layer for national healthcare supply chain operations.

This article shares what I learned—the hard technical lessons, the architectural decisions, and the principles I'd apply again in any large-scale CDC implementation.

What Is CDC and Why Does It Matter at Scale?

Change Data Capture is the process of identifying and capturing changes made to data in a source system—insertions, updates, and deletes—and delivering those changes to downstream systems in near real time.

At small scale, CDC is straightforward. At enterprise scale, across dozens of heterogeneous source systems, it becomes one of the most complex problems in data engineering.

The fundamental challenge: every source system speaks a different language.

SAP has its own change tracking mechanisms. JD Edwards handles deltas differently. Legacy ERPs may have no native CDC support at all. Oracle, Teradata, and SQL Server each have their own approaches to transaction logs, timestamps, and change detection.

When you're ingesting from 64 of these systems simultaneously, you don't just need a CDC solution—you need a CDC framework: one that is system-agnostic, scalable, reliable, and maintainable.

The Architecture: Building a System-Agnostic CDC Framework

The core design principle we followed was separation of concerns. The framework had to work regardless of the source system, the change detection method, or the downstream consumer.

Here's how we structured it:

Layer 1 — Source Connectivity & Change Detection

Different ERP systems required different CDC strategies:

• Log-based CDC for systems that exposed transaction logs (Oracle redo logs, SQL Server CDC)
• Timestamp-based CDC for systems with reliable last-modified timestamps
• Diff-based CDC for legacy systems with no native change tracking—comparing snapshots to identify deltas
• Trigger-based CDC for systems where we could implement database triggers

The key insight here: don't force every source into the same CDC pattern. Respect the capabilities of each source system and build adapters accordingly.

Layer 2 — Ingestion & Standardization

Once changes were captured at the source, they flowed into a standardized ingestion layer built on Apache Kafka for real-time streaming and PySpark on Databricks for batch processing of high-volume historical loads.

Every change event was normalized into a standard envelope format, regardless of source:

{
  source_system: "SAP_ECC",
  table_name: "VBAK",
  operation: "UPDATE",
  timestamp: "2020-06-15T14:32:00Z",
  before_image: { ... },
  after_image: { ... },
  transaction_id: "TXN_ABC123"
}

This standardization was critical. Downstream consumers shouldn't need to know whether a change came from SAP or JD Edwards—they just consume events.

Layer 3 — The Lakehouse (Bronze, Silver, Gold)

All ingested changes landed in a Lakehouse architecture built on Delta Lake:

• Bronze layer — raw change events, exactly as received, immutable
• Silver layer — cleansed, deduplicated, business-key-resolved records
• Gold layer — aggregated, domain-specific datasets ready for analytics and reporting

The Bronze layer was our safety net. If anything went wrong downstream, we could always reprocess from raw captured changes. This proved invaluable during schema evolution events—when a source system changed its table structure, we had the full history to replay.

Layer 4 — Orchestration & Monitoring

At 55 TB/month across 64 source systems, pipeline failures are not a question of if—they're a question of when and how fast you detect them.

We built orchestration using Apache Airflow with custom sensors that monitored:

• Lag between source change and downstream arrival
• Record count reconciliation at each pipeline stage
• Schema drift detection
• Data quality validation gates before Silver and Gold promotion

Every pipeline had an SLA. Breaching an SLA triggered automatic alerts. We maintained 99%+ pipeline reliability across all source systems.

The Hardest Problems We Solved

Problem 1 — Deduplication at Scale

At high ingestion rates, duplicate events are inevitable. Network retries, source system quirks, and at-least-once delivery guarantees all contribute.

Our solution: deterministic event IDs generated from a hash of source system + table + primary key + transaction timestamp. Before writing to Silver layer, we deduplicated against a Bloom filter of recently seen event IDs. This eliminated duplicates with minimal performance overhead.

Problem 2 — Schema Evolution

In a 64-ERP environment, source schemas change. Columns get added, renamed, or removed. Types change.

We handled this with schema registry integration and automated schema evolution policies:

• Backward-compatible changes (adding nullable columns) — automatically handled
• Breaking changes — triggered alerts and required manual review before pipeline resumption
• Full schema history maintained in the Bronze layer for reprocessing

Problem 3 — Handling Late-Arriving Data

Distributed systems produce late-arriving data. A change captured at the source at 2 p.m. might arrive at the ingestion layer at 4 p.m. due to network delays or system backpressure.

We implemented watermarking in our Spark streaming jobs—allowing a configurable late-arrival window before closing a micro-batch. For our use case, a 30-minute late-arrival window balanced correctness with processing latency.

Problem 4 — The Initial Load Problem

CDC captures changes. But when you onboard a new source system, you need the full current state—the initial snapshot—before CDC can meaningfully begin.

For large tables (hundreds of millions of rows), this initial load had to run without impacting the source system. Our approach: parallel chunked extraction with chunk sizes calibrated to source system capacity, running during off-peak hours, with resumability in case of failures.

Data Quality: The Hidden Challenge of CDC at Scale

When data powers national healthcare supply chain decisions—medical device tracking, inventory management, clinical operations—data quality isn't optional. It's a patient safety issue.

We implemented a multi-layer data quality framework:

• At ingestion: schema validation, null checks on critical fields, referential integrity validation
• At Silver promotion: business rule validation, cross-system reconciliation, statistical anomaly detection
• At Gold promotion: domain-specific validation, SLA compliance checks, executive dashboard readiness gates

The result: an 85% reduction in data quality incidents compared to the prior batch-based approach. Stakeholders went from discovering data issues days later to being alerted within minutes.

Why Strong Data Foundations Matter in the Age of Artificial Intelligence

Artificial intelligence is transforming the way organizations operate. From healthcare and financial services to marketing and supply chain management, AI is helping businesses make faster decisions, improve efficiency, and unlock new opportunities.

As AI adoption continues to accelerate, many organizations are focusing heavily on models, algorithms, and automation. However, one critical factor is often overlooked:

AI is only as trustworthy as the data that powers it.

Throughout my career in data engineering and analytics, I have learned that successful AI initiatives do not begin with machine learning models. They begin with something much more fundamental: reliable, high-quality data.

The AI Revolution Is Built on Data

Today, organizations generate massive amounts of information from numerous systems, applications, and external sources. This data fuels analytics platforms, business intelligence solutions, and, increasingly, AI systems.

While artificial intelligence has the potential to deliver extraordinary value, poor data quality can quickly undermine those efforts.

Common challenges include:

• Inconsistent business definitions across teams
• Missing or inaccurate information
• Duplicate records
• Lack of visibility into data sources
• Limited governance and documentation

When these issues exist, organizations often experience conflicting reports, reduced confidence in analytics, and unreliable AI outputs.

Simply put, if the input is flawed, the outcome will be flawed.

Why Trust Matters More Than Ever

AI systems are increasingly being used to support important decisions. Healthcare providers rely on analytics to improve patient outcomes. Financial institutions use AI to detect fraud and manage risk. Businesses use predictive models to optimize operations and better serve customers.

In each of these scenarios, trust becomes essential.

Decision-makers need confidence that:

• The data is accurate.
• The information is current.
• Metrics are consistently defined.
• The underlying processes are transparent.

Without trust, even the most advanced AI systems can struggle to deliver meaningful business value.

Building Strong Data Foundations

In my experience, organizations that achieve success with AI typically invest in strong data foundations first.

These foundations include:

Data Quality

Reliable information begins with maintaining accuracy, consistency, and completeness across systems.

Data Governance

Clearly defined ownership, standards, and processes help ensure that data remains trustworthy and secure.

Scalable Data Architecture

Modern data platforms enable organizations to efficiently process growing volumes of information while maintaining performance and reliability.

Data Lineage and Transparency

Understanding where data originates and how it is transformed creates confidence in analytics and simplifies troubleshooting when issues arise.

Collaboration Across Teams

Technology alone is not enough. Business stakeholders, analysts, engineers, and leaders must work together to establish a culture that values data integrity.

Responsible AI Requires Responsible Data Practices

As AI becomes increasingly integrated into business processes, responsible AI practices are becoming more important than ever.

Responsible AI is not only about algorithms. It also involves:

• Transparency
• Accountability
• Ethical data usage
• Security and privacy
• Continuous monitoring

Organizations that prioritize these principles are better positioned to create AI systems that people can trust.

The Human Side of Technology

One lesson I have learned throughout my journey in technology is that successful digital transformation is never solely about tools or platforms.

People remain at the center of innovation.

Behind every dashboard, every analytics platform, and every AI model are teams of individuals working together to solve problems and create value.

Building trust in technology ultimately begins with building trust among people.

Why Diversity Matters in AI

As we shape the future of artificial intelligence, diverse perspectives are essential.

Women and underrepresented groups bring unique experiences and insights that contribute to more inclusive, ethical, and effective technology solutions.

Encouraging more women to pursue careers in data and AI is not only important for representation—it is essential for innovation.

By supporting one another, sharing knowledge, and mentoring future leaders, we can help create a stronger technology community for generations to come.

Looking Ahead

Artificial intelligence will continue to reshape industries and redefine how organizations operate.

However, long-term success will not be determined solely by the sophistication of AI models. It will be determined by the strength of the data ecosystems that support them.

Before organizations can trust AI, they must first trust their data.

Because, ultimately, building trust in AI starts with building trust in data.