Groupware #1994

pbleser-oc · 2025-12-09T08:30:50Z

Description

Ongoing implementation of the Groupware backend service, which initially happened on its own branch but should now occur on the main branch.

Changes are almost exclusively on packages that are distinct to the Groupware backend, namely

pkg/jmap, pkg/jscalendar and pkg/jscontact: contain the implementation of the JMAP protocol and data models for Core, Mail, Contacts, Calendars, Blobs, ...
services/groupware: contains the Groupware backend service which is currently configured to not be started by default, and must be explicitly included in the START_ADDITIONAL_SERVICES property

Changes to common areas include:

devtools/deployments/opencloud_full: addition of a container of the Stalwart mail server which is used for Groupware functionality
pkg/structs: add more helper functions that are then used in pkg/jmap and services/groupware

Note that it is not fully functional yet and is going to be under continued and ongoing development along with its UI counterparts.

Specifically, the following aspects are lacking and only implemented in a skeletal way as a proof of concept:

API documentation: originally developed using go-swagger and redocly-cli, but which is having severe limitations and might be superseded by an ongoing implementation of a custom source code parser which renders OpenAPI
unit tests: very little coverage if at all, which is mostly due to most of the implementation of the Groupware backend doing relatively little logic and is mostly implementing the JMAP model
integration tests: some coverage exists that performs tests against a Stalwart container using testcontainers, but will be made more exhaustive
k6 black box tests: none implemented yet
system documentation: none implemented yet, too early to do so

The Groupware backend is not meant to be used productively yet.

butonic · 2025-12-10T13:22:16Z

services/auth-api/pkg/service/http/v0/service.go

+
+	svc, err := NewAuthenticationApi(options.Config, &options.Logger, options.Metrics, options.TraceProvider, m)
+	if err != nil {
+		panic(err) // TODO p.bleser what to do when we encounter an error in a NewService() ?


The authentication api should be passed in as an option so the creation cannot fail here.

butonic · 2025-12-10T15:52:22Z

services/groupware/DEVELOPER.md

+Run the following command on your host (requires the `ldap-tools` package with the `ldapsearch` CLI tool), which should output a list of DNs of demo users:
+
+```bash
+ldapsearch -h localhost -D 'cn=admin,dc=opencloud,dc=eu' \


this only works with the url for me

Suggested change

ldapsearch -h localhost -D 'cn=admin,dc=opencloud,dc=eu' \

ldapsearch -H ldap://localhost -D 'cn=admin,dc=opencloud,dc=eu' \

butonic · 2025-12-10T16:06:38Z

services/groupware/DEVELOPER.md

+curl -ks -D- -X POST \
+"https://keycloak.opencloud.test/realms/openCloud/protocol/openid-connect/token" \
+-d username=alan -d password=demo -d grant_type=password \
+-d client_id=groupware -d scope=openid


when working from top to bottom this fails. using a client_id=web works ... so I need to add a client id for the goupware somewhere ...

butonic · 2025-12-10T16:08:15Z

services/groupware/DEVELOPER.md

+to which one should receive the following response:
+
+```java
+A OK [CAPABILITY IMAP4rev2 ...] Authentication successful


hm, I need to dig into:

stalwart-1 | 2025-12-10T16:07:05Z TRACE Raw IMAP input received (imap.raw-input) listenerId = "imaptls", localPort = 993, remoteIp = 172.39.0.1, remotePort = 45034, size = 19, contents = "A LOGIN alan demo\r\n" stalwart-1 | 2025-12-10T16:07:05Z ERROR LDAP error (store.ldap-error) listenerId = "imaptls", localPort = 993, remoteIp = 172.39.0.1, remotePort = 45034, reason = "I/O error: Connection refused (os error 111)", causedBy = "crates/directory/src/core/dispatch.rs:25", id = "A"

butonic · 2025-12-11T13:47:51Z

devtools/deployments/opencloud_full/.env

+STALWART_DOMAIN=
+


Suggested change

STALWART_DOMAIN=

STALWART_DOMAIN=

# LDAP config to use. Can either be idmldap (the built in IdP) or ldap (when using keycloak).

STALWART_AUTH_DIRECTORY=idmldap

butonic · 2025-12-11T13:49:18Z

services/groupware/DEVELOPER.md

+## Services
+
+### Stalwart
+


add a section about the STALWART_AUTH_DIRECTORY env var

butonic · 2025-12-11T13:52:52Z

services/groupware/pkg/groupware/groupware_api.go

@@ -0,0 +1,19 @@
+package groupware


I think we should drop the groupware_ prefix from files in this package

butonic · 2025-12-11T13:53:56Z

pkg/jmap/jmap_api.go

@@ -0,0 +1,49 @@
+package jmap


I think we should drop the jmap_ prefix from all files in this package

butonic · 2025-12-11T14:46:38Z

devtools/deployments/opencloud_full/opencloud.yml

+      AUTH_BASIC_LDAP_BIND_PASSWORD: "admin"
+      USERS_LDAP_BIND_PASSWORD: "admin"
+      GROUPS_LDAP_BIND_PASSWORD: "admin"
+      IDM_LDAPS_ADDR: 0.0.0.0:9235


I configured everything on .compose.test to prevent collisions with my .opencloud.test deployment. The stalwart URL then needs to be set for the groupware:

Suggested change

IDM_LDAPS_ADDR: 0.0.0.0:9235

IDM_LDAPS_ADDR: 0.0.0.0:9235

GROUPWARE_JMAP_BASE_URL: https://${STALWART_DOMAIN:-stalwart.opencloud.test}

oh and we need to set FRONTEND_GROUPWARE_ENABLED: "true" and enable the mail app in the web config when stalwart is enabled ... but that connot be configured with a simple env var ... 😞 we need to replace the config file then ...

butonic

Ok, I still don't have a menu entry for mail in the web UI even though the config.json contains the mail app. I assume that is a problem with my setup/config. in any case, I can see the mail UI when navigation to {$OC_URL}/mail manually. \o/

I did run into some minor issues when following the docs. If you could address them ~~(and enlighten me on how to get menu to show the groupware apps)~~ I'm happy to merge this. I mostly want others to be able follow the DEVELOPMENT.md and have a working setup. Kudos for that, btw.

The groupware service itself follows our ... boilerplate ... service code ... and implements the JMAP handling. I assume that will have to evolve, but we can merge it and iterate.

Tip

the web repo has a compose file with all the apps enabled. That gave me the final hint to get the menu entries

…enLDAP container as a directory for user authentication

…ince 'admin' exists as a regular user in LDAP and thus won't have access to the administration

…80 directly to access the userinfo endpoint using HTTP since the certificates in traefik are self-signed and end up being rejected by Stalwart with no option to bypass the certificate check

…h API endpoint with it

…ernal authentication API for third party services such as Stalwart

* primitive implementation to demonstrate how it could work, still to be considered WIP at best * add new dependency: MicahParks/jwkset and MicahParks/keyfunc to retrieve the JWK set from KeyCloak to verify the signature of the JWTs sent as part of Bearer authentication in the /auth API * (minor) opencloud/.../service.go: clean up a logging statement that was introduced earlier to hunt down why the auth-api service was not being started

…es to the model to conform with draft-ietf-calext-jscalendarbis-10 and fields that are currently not implemented in Stalwart

@type

* upgrade Stalwart image for devtools/full to 0.14.1 * re-assert which features are implemented or not in 0.14.1 * refactor the integration tests yet again to make it clearer and easier to see those "features-or-not" * get rid of old tests that are now better covered by integration tests * rewrite how we compare expected and actual objects in integration tests, finally having found a way to ignore the @type attribute properly instead of having to mutate all objects to remove it

* implement Request.AllAccountIds() to generalize the fetching (and uniqifying) of all account IDs, which will allow us to implement things such as "subscribed" accounts, or limiting the number of accounts in one request * add Account-Id response header * add Object-Type response header

… without email bodies

* fix a bug in how email summaries are flattened across multiple accounts, which was previous resulting in empty email objects * allow negative offset in email pagination * make all /emails endpoints return emails without bodies

…gration test

…f they are missing

* jmap/EmailCreate: add more attributes that were omitted: Headers, InReplyTo, References, Sender * add jmap GetEmailSubmissionStatus * improve email integration tests by adding a thorough test for email submission * jmap integration tests: provision principals and domains using the Stalwart Management API, switching from an in-memory to an internal directory

…ion properties

…efaults and config sub-structures

* add example generator infrastructure, with some examples for pkg/jmap and pkg/groupware, with more needing to be done * alter the apidoc Makefile to stop using go-swagger but, instead, use the openapi.yml file that must be dropped into that directory using groupware-apidocs (will improve the integration there later) * add Makefile target to generate examples * bump redocly from 2.4.0 to 2.14.5 * introduce Request.PathParam() and .PathParamDoc() to improve API documentation, as well as future-proofing * improve X-Request-ID and Trace-Id header handling in the middleware by logging it safely when an error occurs in the middleware

…gger

… and remove defunct files

…neration tooling

pbleser-oc self-assigned this Dec 9, 2025

pbleser-oc added Status:Needs-Review Type:Organizational Type:Maintenance E.g. technical debt, packaging, etc. and removed Type:Organizational labels Dec 9, 2025

pbleser-oc requested a review from butonic December 10, 2025 08:32

pbleser-oc marked this pull request as ready for review December 10, 2025 08:32

butonic reviewed Dec 10, 2025

View reviewed changes

butonic reviewed Dec 11, 2025

View reviewed changes

services/groupware/DEVELOPER.md

## Services

### Stalwart

Copy link

Contributor

butonic Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a section about the STALWART_AUTH_DIRECTORY env var

butonic reviewed Dec 11, 2025

View reviewed changes

pkg/jmap/jmap_api.go

@@ -0,0 +1,49 @@

package jmap

Copy link

Contributor

butonic Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should drop the jmap_ prefix from all files in this package

butonic reviewed Dec 11, 2025

View reviewed changes

butonic requested changes Dec 11, 2025

View reviewed changes

pbleser-oc force-pushed the groupware branch from a9ef8f3 to 3d8cad1 Compare December 17, 2025 08:34

pbleser-oc added 12 commits January 22, 2026 09:36

Add Stalwart container to the opencloud_full deployment, using the Op…

c10f04b

…enLDAP container as a directory for user authentication

WIP: initial implementation of the groupware service

abd01d2

add missing routing for /groupware (currently unprotected for testing)

fd327ad

rename Stalwart fallback admin username from 'admin' to 'mailadmin' s…

ef888ea

…ince 'admin' exists as a regular user in LDAP and thus won't have access to the administration

add an OIDC Directory to Stalwart, requires exposing Keycloak port 80…

1b45acf

…80 directly to access the userinfo endpoint using HTTP since the certificates in traefik are self-signed and end up being rejected by Stalwart with no option to bypass the certificate check

WIP: restructure the Jmap client, and implement the /me/messages Grap…

2b55edc

…h API endpoint with it

move services/groupware/pkg/jmap to pkg/jmap

d59d498

add an auth-api service to make an exemplary implementation of an ext…

8ef4d0a

…ernal authentication API for third party services such as Stalwart

minor corrections to the Stalwart configuration

89d07b0

upgrade Stalwart to 0.12

6f35e9c

groupware and jmap improvements and refactoring

e09014a

pbleser-oc added 20 commits January 22, 2026 09:42

groupware: finalize JMAP events integration test, with multiple chang…

deee89d

…es to the model to conform with draft-ietf-calext-jscalendarbis-10 and fields that are currently not implemented in Stalwart

groupware: response payload /groupware/accounts/{id}/emails should be…

f346b98

… without email bodies

groupware: fix email summaries and allow negative offsets

6cf7ed8

* fix a bug in how email summaries are flattened across multiple accounts, which was previous resulting in empty email objects * allow negative offset in email pagination * make all /emails endpoints return emails without bodies

groupware: JMAP WS push notifications support

349bd20

groupware: WS push improvements, add getting email changes to WS inte…

b5d3819

…gration test

groupware: upgrade the Stalwart image in devtools from 0.14.0 to 0.14.1

b4cce7d

groupware, auth-api: remove tracing and tracing configuration

746738a

groupware: fix missing casting to jmap.State after changes in pkg/jmap

3c5fa0f

groupware: minor: remove network declaration in stalwart.yml

5bee8ce

groupware: refactor response objects to take a list of accountIds

f487376

groupware: add retrieving and adding mailboxIds for drafts and sent i…

0b2bfb8

…f they are missing

groupware: add description and version annotations for env configurat…

0c299ad

…ion properties

groupware: fix failing pkg/jscalendar unit tests

4d0c8e9

groupware: fix failing pkg/jscontact unit tests

4f10e18

groupware: shift some attributes of the Groupware object around, in d…

b9ce1fe

…efaults and config sub-structures

groupware: update to Stalwart 0.15.0

073eec1

pbleser-oc force-pushed the groupware branch from d7995f9 to 5d5dd03 Compare January 22, 2026 08:42

fix go.mod by tidying after a significant rebase

0fce463

pbleser-oc force-pushed the groupware branch from 284aa10 to 0fce463 Compare January 22, 2026 08:47

pbleser-oc added 7 commits January 22, 2026 09:48

groupware: update pnpm 10.28.0 -> 10.28.1

e9a49f7

groupware: migrate to cobra instead of urfave

502d084

update redocly 2.14.5 -> 2.14.6

8457109

groupware: change Makefile to use groupware-apidocs instead of go-swa…

1b2e55f

…gger

groupware: more API examples

1408c3d

groupware: update README to new API documentation generation process,…

4778cd5

… and remove defunct files

groupware: also update DEVELOPER.md with new OpenAPI documentation ge…

e6b59c8

…neration tooling

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Groupware #1994

Groupware #1994

Uh oh!

pbleser-oc commented Dec 9, 2025 •

edited

Loading

Uh oh!

butonic Dec 10, 2025

Uh oh!

butonic Dec 10, 2025

Uh oh!

butonic Dec 10, 2025

Uh oh!

butonic Dec 10, 2025

Uh oh!

butonic Dec 11, 2025

Uh oh!

butonic Dec 11, 2025

Uh oh!

butonic Dec 11, 2025

Uh oh!

butonic Dec 11, 2025

Uh oh!

butonic Dec 11, 2025 •

edited

Loading

Uh oh!

butonic Dec 11, 2025

Uh oh!

butonic left a comment •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	ldapsearch -h localhost -D 'cn=admin,dc=opencloud,dc=eu' \
	ldapsearch -H ldap://localhost -D 'cn=admin,dc=opencloud,dc=eu' \

	IDM_LDAPS_ADDR: 0.0.0.0:9235
	IDM_LDAPS_ADDR: 0.0.0.0:9235
	GROUPWARE_JMAP_BASE_URL: https://${STALWART_DOMAIN:-stalwart.opencloud.test}

Groupware #1994

Are you sure you want to change the base?

Groupware #1994

Uh oh!

Conversation

pbleser-oc commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

butonic Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

butonic left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pbleser-oc commented Dec 9, 2025 •

edited

Loading

butonic Dec 11, 2025 •

edited

Loading

butonic left a comment •

edited

Loading