It was super fun talking to Max about my journey through geospatial / climate and where I think I this field is heading.

🌎 Why do we need a cloud-native data lake for geospatial data? In the latest episode of The Infra Pod, @tnachen & @ianlivingstone chat with the cofounders of @EarthmoverHQ, @rabernat & @_jhamman, about the future of data in climate and earth sciences. 🎧 Link in 🧵

🌤️ #AMS2025 is just around the corner! We are taking AMS by storm with an exhibitor booth (booth 353), two talks from @_jhamman and @rabernat , and hosting a @pangeo_data Community Happy Hour (register here: lu.ma/ddtba5f5)!

@rabernat is rounding out the week of #AGU24 with his talk, “How can we make cloud computing actually accessible to all scientists?” on Friday at 5:10 PM agu.confex.com/agu/agu24/mee…

Monday through Thursday, I'll be hanging out with @rabernat at the @EarthmoverHQ booth in the exhibit hall. Swing by to say hello or to snag some swag/stickers/etc. We'll also be demoing #icechunk all week.

Will you be at @theAGU next week? Earthmover is exhibiting! @rabernat and @_jhamman are participating in panel discussions and giving talks👇.

Checked out the other site. 🟦 Seems much better. Gonna be over there more from now on. 👋

I take it all back. At #SatSummit in Lisbon we finally discovered the holy grail of cloud optimized file formats

Are you heading to #AGU24 next month? Consider joining us for a bonus day of hacking on @pangeo_data. I'll be there representing @EarthmoverHQ and helping folks work with #icechunk and @zarr_dev. Details and signup here: discourse.pangeo.io/t/post-a…

🚀 New blog post: Nomadic Compute - The Future of Distributed AI Workloads 🌐 In a fast-paced AI world, flexibility & resilience are key. "Nomadic Compute" pattern lets workloads move dynamically across clouds for peak performance, cost, & availability: tigrisdata.com/blog/nomadic-…

Come learn about recent @xarray_dev GroupBy improvements at tomorrow's (Wed, Nov 13) Pangeo Showcase! discourse.pangeo.io/t/pangeo…

Most developers today wouldn't dream of not using version control for their code... However, the same principles can be applied to data! @EarthmoverHQ's new open source project--Icechunk--includes version control features built specifically for the @zarr_dev data model, brining powerful data version control to the world of massive multidimensional arrays. Features include * All updates occur atomically in isolated snapshots * Tags - immutable pointers to snapshots * Branches - mutable pointers to snapshots With Icechunk, you can safely experiment with changes to your data on a "dev" branch before propagating those changes to "main." You can publish an immutable version of your dataset (tag) while continuing to evolve towards the next version. Or you can simply revert incorrect changes back to an earlier version of your data. These capabilities make life so much easier for data scientists and teams using array data in production. I've been using data version control with Zarr for the past year via our Arraylake platform, and I'm thrilled that these capabilities are now fully open source. I can't imagine going back to the old way of working. Learn more at icechunk.io/

Xarray v2024.10.0 has just been released, including support for xarray.DataTree and zarr-python v3 !!! github.com/pydata/xarray/rel… @xarray_dev @zarr_dev

It’s a real honor to be part of this amazing collection of experts. Looking forward to helping spread the word about Cloud Native Geospatial!

"Large GeoSpatial Benchmarks: First Pass" Last month we asked for TiB scale geo workloads to form a benchmark suite. We got strong response. Since then we've built out these into a public suite. This post goes over what's implemented and early results docs.coiled.io/blog/geospati…

Join us tomorrow Oct 23 at 4 PM EDT for a Pangeo showcase on Icechunk: An Open-Source Transactional Storage Engine for Zarr by @rabernat! discourse.pangeo.io/t/pangeo…

It's been a blast learning rust and working with the @EarthmoverHQ team on Icechunk! Come see what it's all about. Absolutely worth it, I promise.

⚡️ Icechunk is fast! What does this mean for users? Reduced cost for all data-intensive compute jobs and enhanced productivity for the data scientists who work with data all day long. Icechunk, @EarthmoverHQ's new transactional cloud-native storage engine for array / tensor data, works together with @zarr_dev , augmenting the Zarr core data model with features that enhance performance, collaboration, and safety in a multi-user cloud-computing context. Reading data through Icechunk is 36x faster than trying to read HDF5 files from cloud object storage, 6x faster than regular Zarr alone, and 2.5x faster than regular Zarr + Dask. Most importantly, Icechunk can achieve throughput on par with the compute instance network bandwidth, the "hardware limit" for I/O bound workloads. Want to learn more about this benchmark? Come to our Icechunk informational webinar tomorrow, Tuesday, October 22nd from 12 - 1 PM EST. Registration link: share.hsforms.com/1SCOFqe2kT…

We’re hosting a webinar on Tuesday, October 22 from 12- 1 PM EST to discuss what Icechunk means for the scientific data community and answer questions from attendees. Register here: share.hsforms.com/1SCOFqe2kT…

There has been so much interest and investment in launching new EO satellites in the past 3-5 years. But, there has been relatively limited interest and investment into solving the boring problems of standardization, interoperability and analysis-ready data* in EO.