feat(backend): state persisted objects (#9870)

This PR initiates the rework of Zitadel's backend to state-persisted
objects. This change is a step towards a more scalable and maintainable
architecture.

## Changes

* **New `/backend/v3` package**: A new package structure has been
introduced to house the reworked backend logic. This includes:
* `domain`: Contains the core business logic, commands, and repository
interfaces.
* `storage`: Implements the repository interfaces for database
interactions with new transactional tables.
  * `telemetry`: Provides logging and tracing capabilities.
* **Transactional Tables**: New database tables have been defined for
`instances`, `instance_domains`, `organizations`, and `org_domains`.
* **Projections**: New projections have been created to populate the new
relational tables from the existing event store, ensuring data
consistency during the migration.
* **Repositories**: New repositories provide an abstraction layer for
accessing and manipulating the data in the new tables.
* **Setup**: A new setup step for `TransactionalTables` has been added
to manage the database migrations for the new tables.

This PR lays the foundation for future work to fully transition to
state-persisted objects for these components, which will improve
performance and simplify data access patterns.

This PR initiates the rework of ZITADEL's backend to state-persisted
objects. This is a foundational step towards a new architecture that
will improve performance and maintainability.

The following objects are migrated from event-sourced aggregates to
state-persisted objects:

* Instances
  * incl. Domains
* Orgs
  * incl. Domains

The structure of the new backend implementation follows the software
architecture defined in this [wiki
page](https://github.com/zitadel/zitadel/wiki/Software-Architecturel).

This PR includes:

* The initial implementation of the new transactional repositories for
the objects listed above.
* Projections to populate the new relational tables from the existing
event store.
* Adjustments to the build and test process to accommodate the new
backend structure.

This is a work in progress and further changes will be made to complete
the migration.

---------

Co-authored-by: Copilot <[email protected]>
Co-authored-by: Iraq Jaber <[email protected]>
Co-authored-by: Iraq <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>

backend/v3/storage/database/dialect/postgres/embedded/start.go

@@ -0,0 +1,50 @@
+// embedded is used for testing purposes
+package embedded
+
+import (
+	"net"
+	"os"
+
+	embeddedpostgres "github.com/fergusstrange/embedded-postgres"
+	"github.com/zitadel/logging"
+
+	"github.com/zitadel/zitadel/backend/v3/storage/database"
+	"github.com/zitadel/zitadel/backend/v3/storage/database/dialect/postgres"
+)
+
+// StartEmbedded starts an embedded postgres v16 instance and returns a database connector and a stop function
+// the database is started on a random port and data are stored in a temporary directory
+// its used for testing purposes only
+func StartEmbedded() (connector database.Connector, stop func(), err error) {
+	path, err := os.MkdirTemp("", "zitadel-embedded-postgres-*")
+	logging.OnError(err).Fatal("unable to create temp dir")
+
+	port, close := getPort()
+
+	config := embeddedpostgres.DefaultConfig().Version(embeddedpostgres.V16).Port(uint32(port)).RuntimePath(path)
+	embedded := embeddedpostgres.NewDatabase(config)
+
+	close()
+	err = embedded.Start()
+	logging.OnError(err).Fatal("unable to start db")
+
+	connector, err = postgres.DecodeConfig(config.GetConnectionURL())
+	if err != nil {
+		return nil, nil, err
+	}
+
+	return connector, func() {
+		logging.OnError(embedded.Stop()).Error("unable to stop db")
+	}, nil
+}
+
+// getPort returns a free port and locks it until close is called
+func getPort() (port uint16, close func()) {
+	l, err := net.Listen("tcp", ":0")
+	logging.OnError(err).Fatal("unable to get port")
+	port = uint16(l.Addr().(*net.TCPAddr).Port)
+	logging.WithFields("port", port).Info("Port is available")
+	return port, func() {
+		logging.OnError(l.Close()).Error("unable to close port listener")
+	}
+}