#Egnyte desktop sync limit code
#Egnyte desktop sync limit Offline

Caching (distributed and in-memory) and Cache invalidation.

Real-time sync of files to desktop clients.

Some of the biggest architecture challenges are: All services are stateless and use various databases or our own object store for storage.Ī 10,000 foot overview of Egnyte Connect services: This also allows us to move some of the backend services to be hosted in the public cloud. We use a service-oriented architecture based on REST, which allows us to scale each service independently.

Millions of desktop clients accessing files interactively.

#Egnyte desktop sync limit Offline

Millions of desktop clients syncing files with the cloud for offline access.Top 5 file extensions stored by Egnyte are pdf, doc/docx, xls/xlsx, jpeg, and png.Multiple terabytes of extracted content indexed in Elasticsearch.A million+ metadata API requests per minute at peak and Billions of files and tens of petabytes of data stored in our servers and other object stores such as GCS, S3 and Azure Blobstore.50+ text extraction instances(autoscaled).500+ storage nodes powered by Tomcat/Nginx.3 primary regions with one in Europe connected to respective GCP regions using Google Interconnect.Native desktop and server hosted clients that allow both interactive as well as hybrid sync access to the entire namespace.Python used to power client-side code, certain microservices, migration scripts, internal scripts.

#Egnyte desktop sync limit code

Java used to power core file system code.ReactJS/Backbone/Marionette/JQuery/npm/Nightwatch.To add elasticity, reliability, and durability, these data centers are connected to the Google Cloud platform using high speed, secure Google Interconnect network. The major parts of the system are: CFS (cloud file system), EOS (Egnyte object store), Content Security, Event Sync, Search, and user behavior-based recommendations.Įgnyte Connect platform employs 3 data centers to fulfill requests from millions of users across the world. It can be accessed from a variety of endpoints like FTP, WebDAV, mobile, public API, and browsers, and features strong audit and security components. It is built for hosting a variety of file services and to cloud-enable their existing file repositories. It is a Secure Content Platform used by thousands of customers for their document management needs. In this blog post, I will discuss one of Egnyte’s products: Egnyte Connect. Over time, costs for S3 and GCS became reasonable and, with Egnyte’s storage plugin architecture, our customers can now bring in any storage backend of their choice for the content. Our only option was to roll up our sleeves and build basic cloud file system components, such as object store, ourselves and then build a content platform on top of it. It’s a roller coaster ride, attempting to scale this system to meet the growth and SLA requirements while providing the stringent consistency and durability characteristics we all have come to expect from our laptops.Įgnyte is a secure Content Collaboration and Data Governance platform, founded in 2007, before Google drive was born and when AWS S3 was cost-prohibitive. Let’s take this problem and transform it into a cloud-native file system used by millions of paid users spread across the globe. It chokes if you run a few I/O intensive processes or try collaborating with 100 other users. It’s limited by the disk space and can’t expand storage elastically. There are a few challenges if you are looking to use it to support tens of thousands of users, working on hundreds of millions of files simultaneously containing petabytes of data. Your laptop has a filesystem used by hundreds of processes.

Let’s start with a brief introduction to the platform we develop and scale on a daily basis. In this article, I will share the current architecture and some of the lessons we learned in scaling it to pay it forward to the developer community. I have learned a lot by reading the unique insights of architecture and technical blog posts, written by large platforms companies. Together with my team, we’re responsible for scaling our platform - a content platform backed by a large, distributed file system. I’m Kalpesh Patel, a Distinguished Engineer at Egnyte with 20+ years of experience. The first version of this blog post was originally published at “ ” Egnyte Architecture: Lessons learned in building and scaling a multi-petabyte content platform