Thanks to @Seba-T's work with Orca, we were able to identify a number of performance issues in certain high load conditions.
This commit contains fixes for the following issues
* Unbounded Memory Growth - due to how ARCache works, to maintain temporal consistency it must retain copies of keys (not values) in a special data set for tracking. The Filter Resolve Cache was using unresolved filters as keys. This caused memory explosions when refint or memberof were updating a group with a large number of members because they would emit a query with hundreds of filter terms that would only be used once and never again, causing the ARCache haunted set to grow without bound. To limit this, we no longer cache large/complex queries for resolution, and in future we may implement some other methods to reduce this like sha256/hmac of the queries.
* When creating a new account, dyngroups would be engaged to add the account as a member due to the matching scope. However the change to the dyngroup was triggering an update of all the dyngroups *members* related memberof attributes. This would mean that adding an account would trigger every other account to be loaded an updated.
* When memberof would iterate over leaf entries and update them one at a time. This mean a large number of small fragmented queries in the case of a lot of leaf entries being updated. Now leaf entries are updated in a single stripe once groups are stabilised.
* Member of would always trigger it's members to always update. Instead, we should only update members where a difference is observed, or all members if the group's memberof itself has changed since this needs to propogate to all leaf entries. This significantly reduces the amount of writes and operations to examine the changed member of set.
* Referential integrity would examine all reference uuids on entries for validity rather than just the reference uuids that were altered within the transaction. This change means that only uuids that were *added* are validated during an operation.
* During async write backs (delayed actions) these were performed one at a time. Instead, when possible this should be done in a single transaction as the write transaction caches all writes in memory until the commit meaning that by batching we reduce overall latency.
* In the server there can only be one write transaction and many readers. These are guarded by tokio semaphores that act as fair queues - first in gets the lock next. Due to the design of the server readers would be blocked on the *database* semaphore, and writers would block on the write semaphore and THEN the database semaphore. This arrangement was creating a situation which unfairly advantaged readers over writers, as any write would first have to become the head of it's queue, and then compete with all readers to access a db transaction. Instead, we now have a reader semaphore with size threads minus 1, clamped at a minimum of 1. This means that provided there are two or more threads, then a writer will *always* have a database handle available, and readers will pre-queue with each other before queueing on the db ticket. If there is only one thread, then writes and reads will alternate between each other fairly.
While basking under the shade of the coolabah tree, I was overcome by an intense desire to improve the performance and memory usage of Kanidm.
This pr reduces a major source of repeated small clones, lowers default log level in testing, removes some trace fields that are both large and probably shouldn't be traced, and also changes some lto settings for release builds.
* kanidm cli logs on debug level - Fixes#2745
* such clippy like wow
* It's important for a wordsmith to know when to get its fixes in.
* updootin' wasms
* otel can eprintln kthx
* started python integration tests, features
* more tests more things
* adding heaps more things
* updating docs
* fixing python test
* fixing errors, updating integration test
* Add models for OAuth2, Person, ServiceAccount and add missing endpoints
* Alias Group to GroupInfo to keep it retrocompatible
* Fixed issues from review
* adding oauth2rs_get_basic_secret
* adding oauth2rs_get_basic_secret
* Fixed mypy issues
* adding more error logs
* updating test scripts and configs
* fixing tests and validating things
* more errors
---------
Co-authored-by: Dogeek <simon.bordeyne@gmail.com>
* adding extra_attributes field to BuiltinGroup, migrating more things.
* checkpoint 3 - ACP, easy as 1,2,3
* codespell
* now throwing error on dyngroup with defined members
* Starting to chase down testing
* commenting out unused/inactive endpoints, adding more tests
* clippyism
* making clippy happy v2
* testing when things are not right
* moar checkpoint
* splitting up testkit things a bit
* moving https -> tide
* mad lad be crabbin
* spawning like a frog
* something something different spawning
* woot it works ish
* more server things
* adding version header to requests
* adding kopid_middleware
* well that was supposed to be an hour... four later
* more nonsense
* carrying on with the conversion
* first pass through the conversion is DONE!
* less pub more better
* session storage works better, fixed some paths
* axum-csp version thing
* try a typedheader
* better openssl config things
* updating lockfile
* http2
* actually sending JSON when we say we will!
* just about to do something dumb
* flargl
* more yak shaving
* So many clippy-isms, fixing up a query handler bleep bloop
* So many clippy-isms, fixing up a query handler bleep bloop
* fmt
* all tests pass including basic web logins and nav
* so much clippyism
* stripping out old comments
* fmt
* commenty things
* stripping out tide
* updates
* de-tiding things
* fmt
* adding optional header matching ,thanks @cuberoot74088
* oauth2 stuff to match #1807 but in axum
* CLIPPY IS FINALLY SATED
* moving scim from /v1/scim to /scim
* one day clippy will make sense
* cleanups
* removing sketching middleware
* cleanup, strip a broken test endpoint (routemap), more clippy
* docs fmt
* pulling axum-csp from the wrong cargo.toml
* docs fmt
* fmt fixes
logging changes:
* Offering auth mechanisms -> debug
* 404's aren't really warnings
* double tombstone message, one goes to debug
other changes:
* CSP changes to allow the bootstrap images to load
* more testing javascriptfile things, I R
* it's nice to know where things are
* putting non-rust web things in static/ instead of src/
* RequestCredentials::SameOrigin is the default, also adding a utility function to save dupe code. Wow this saved... kilobytes.
* removing commented code, fixing up codespell config
* clippyisms
* wtf, gha
* dee-gloo-ing some things
* adding some ubuntu build test things
* sigh rustwasm/wasm-pack/issues/1138
* more do_request things
* packaging things
* hilarious dev env setup script
* updated script works, all the UI works, including the experimental UI for naughty crabs
* deb package fixes
* fixed some notes
* setup experimental UI tweaks
