Scientific Workflows with EWOKS: Research Pipelines Join the Suite
Research data rarely arrives analysed. It arrives as raw instrument output that has to be reduced, calibrated, and processed through the same sequence of steps every time — by different people, on different days, reproducibly. To serve that need, openEduSuite now integrates EWOKS, the open-source scientific workflow framework developed at the European Synchrotron (ESRF), as a suite service with unified single sign-on.
What EWOKS Is
EWOKS — the Extensible Workflow System, developed by the Software group of the European Synchrotron (ESRF) — is a meta workflow system: an abstraction layer between workflow definition and the execution engine that runs it. A workflow is expressed as a graph of tasks and can then be executed on any supported engine, currently including pypushflow, Orange, Dask, and the internal EWOKS engine. Key properties:
- Python ecosystem — tasks are plain Python, so existing analysis code can be wrapped instead of rewritten.
- Engine portability — the same workflow definition runs on different execution engines, without changing the workflow.
- Web interface and REST API — workflows are created, edited, and monitored in the browser (ewoksweb) and can be driven programmatically.
- Serialisable definitions — workflows are stored as JSON or YAML and can serve as data-provenance documentation for publications.
EWOKS was built to handle the data pipelines of large-scale experiments, where reproducibility is not optional. Those are exactly the properties research groups at universities need as well.
Joining the Suite on Equal Terms
The integration follows the same pattern as every other suite service:
- Single sign-on — the service sits behind the standard authentication gateway and authenticates users against the central Keycloak identity provider. Researchers log in with the accounts they already have; no local user management.
- Verified redirect chains — the SSO path (gateway start, identity-provider login, redirect back into the application) is exercised by automated browser tests, so a broken login flow is caught before users find it.
- A portal entry — EWOKS appears in the main service grid of the portal as "Scientific Workflows", discoverable next to mail, files, and the other everyday services.
Where It Fits
The suite now covers three complementary layers of automation, each with a distinct job:
| Layer | Service | Job |
|---|---|---|
| General automation | n8n | Connecting web services, notifications, administrative glue |
| Scientific workflows | EWOKS | Reproducible research data pipelines as task graphs |
| Interactive compute | JupyterHub, Dask | Exploration, notebooks, distributed compute |
No layer replaces another. A research group may prototype a reduction step in a notebook, productionise it as an EWOKS task, and have an n8n workflow notify a channel when the pipeline finishes.
What's Next
- Group-based authorisation — refining which workflows and results are visible to which research groups, building on the existing role model.
- Example workflows — a small library of ready-to-adapt pipelines for common analysis patterns.
- Documentation — a suite integration guide so institutions can deploy the same pattern.
Call to Action
- Learn EWOKS: the framework documentation lives at ewoks.readthedocs.io.
- Bring a pipeline: wrap your group's existing Python analysis as a first task graph and try it in the suite.
- Get involved: join the openEduSuite community and shape the research-data roadmap.
From instrument to insight — reproducible pipelines are now part of the digital workplace.
Attribution: EWOKS is developed at the European Synchrotron (ESRF) and is used here as open-source software. This article is independent and is not affiliated with or endorsed by the ESRF. All other product names are trademarks of their respective owners.
