During initial development of the revised metrics series this field
was sourced by 'memfree' from /proc/meminfo as parsed by our ProcFS
tools, but as memavailable got added for PVE 8.4, as it much better
shows the state of actually available memory [0], I suggested to Aaron
to move the new RRD format also to that metric. While that was done,
the RRD definition was still using the old name, while I fixed in in
pmxcfs, I forgot to do so in the upgrade migration tool, which then
created an RRD file with the old name.
Thus map the old name to the new one here in case there is no existing
entry for the new one already, if one would be parsed later it would
override this fallback anyway.
[0] as while the kernel tries to use unused memory to improve
performance, it has no issue to release that again quickly on memory
pressure.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
`filename_pve2` is not accessed anymore after assigning the copy to
`filename`, so we can simply 'transfer' the ownership to `filename`
and set `filenname_pve2` to NULL.
The call to `g_free(filename_pve2)` later can handle a NULL arg, it will
simply do nothing in this case.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Link: https://lore.proxmox.com/20250801095431.117317-5-l.wagner@proxmox.com
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This should be a tiny bit more efficient.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Link: https://lore.proxmox.com/20250801095431.117317-3-l.wagner@proxmox.com
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Otherwise we leak the memory that was already allocated for `filename`.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Link: https://lore.proxmox.com/20250801095431.117317-1-l.wagner@proxmox.com
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Those files get migrated with pve-network >= 0.9.9 - so we do not need
to observe them anymore via pmxcfs.
Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
Link: https://lore.proxmox.com/20250801121029.201766-2-s.hanreich@proxmox.com
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Storage metrics and node metrics use a nested directory structure, e.g.:
pve-storage-9.0
somenode
local
local-lvm
The second level ('somenode') was not created correctly, leading to
errors when trying to update the RRD database.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Link: https://lore.proxmox.com/20250801083415.64421-1-l.wagner@proxmox.com
g_free can be passed a NULL pointer according to the docs [1], it does
nothing in that case.
[1] https://docs.gtk.org/glib/func.free.html
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
With PVE9 we introduced a new RRD format that has different aggregation
steps, similar to what we use in the Backup Server.
We therefore need to adapt the functions that get data from RRD
accordingly.
The result is usually a finer resolution for time windows larger than
hourly.
We also introduce decade as a time window. In case existing RRD files
have not yet been converted to the new RRD format, we need keep using
the old time windows. Additionally, since they only store data up to a
year, we catch the situation where a full decade might be requested and
pin it to a year.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
With PVE9 now we have additional fields in the metrics that are
collected and distributed in the cluster. The new fields/columns are
added at the end of the existing ones. This makes it possible for PVE8
installations to still use them by cutting the new additional data.
To make it more future proof, the format of the keys for each metrics
are now changed:
Old pre PVE9: pve{version}-{type}/{id}
Now with PVE9: pve-{type}-{version}/{id}
This way we have an easier time to handle new versions in the future as
we initially only need to check for `pve-{type}-`. If we know the
version, we can handle it accordingly; e.g. pad if older format with
missing data. If we don't know the version, it must be a newer one and
we cut the data stream at the length we need for the current version.
This means of course that to avoid a breaking change, we can only add
new columns if needed, but not remove any! But waiting for a breaking
change until the next major release is a worthy trade-off if it allows
us to expand the format in between if needed.
The 'rrd_skip_data' function got a new parameter defining the sepataring
character. This then makes it possible to use it also to determine which
part of the key string is the version/type and which one is the actual
resource identifier.
We add several new columns to nodes and VMs (guest) RRDs. See futher
down for details. Additionally we change the RRA definitions on how we
aggregate the data to match how we do it for the Proxmox Backup Server
[0].
The migration of an existing installation is handled by a dedicated
tool. Only once that has happened, will we store data in the new
format.
This leaves us with a few cases to handle:
data recv → old new
↓ rrd files
-------------|---------------------------|-------------------------------------
none | check if directories exists:
| neither old or new -> new
| new -> new
| old only -> old
--------------|---------------------------|-------------------------------------
only old | use old file as is | cut new columns and use old file
--------------|---------------------------|-------------------------------------
new present | pad data to match new fmt | use new file as is and pass data
To handle the padding we use a buffer. Cutting can be handled as we
already do it in the stable/bookworm (PVE8) branch by introducing a null
terminator in the original string at the end of the expected columns.
We add the following new columns:
Nodes:
* memfree
* arcsize
* pressures:
* cpu some
* io some
* io full
* mem some
* mem full
VMs:
* memhost (memory consumption of all processes in the guests cgroup, host view)
* pressures:
* cpu some
* cpu full
* io some
* io full
* mem some
* mem full
[0] https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=src/server/metric_collection/rrd.rs;h=ed39cc94ee056924b7adbc21b84c0209478bcf42;hb=dc324716a688a67d700fa133725740ac5d3795ce#l76
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
We get the following error without this patch:
```
$ pvesh set /cluster/options
400 validation error in '/etc/pve/datacenter.cfg'
replication: type check ('string') failed - got HASH
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Link: https://lore.proxmox.com/20250718081349.125228-2-m.sandoval@proxmox.com
For PVE9 we plan to add additional fields in the metrics that are
collected and distributed in the cluster. The new fields/columns are
added at the end of the current ones. This makes it possible for PVE8
installations to still use them by cutting the new additional data.
To make it more future proof, the format of the keys for each metrics
are changed:
Old: pve{version}-{type}/{id}
New: pve-{type}-{version}/{id}
This way we have an easier time to handle new versions in the future as
we initially only need to check for `pve-{type}-`. If we know the
version, we can handle it accordingly; e.g. pad if older format with
missing data. If we don't know the version, it must be a newer one and
we cut the data stream at the length we need for the current version.
This means of course that to avoid a breaking change, we can only add
new columns if needed, but not remove any! But waiting for a breaking
change until the next major release is a worthy trade-off if it allows
us to expand the format in between if needed.
Since the full keys were used for the final location within the RRD
directory, we need to change that as well and set it manually to
'pve2-{type}' as the key we receive could be for a newer data format.
The 'rrd_skip_data' function got a new parameter defining the sepataring
character. This then makes it possible to use it to determine which part
of the key string is the version/type and which one is the actual
resource identifier.
We drop the pve2-vm schema as the newer pve2.3-vm has been introduced
with commit ba9dcfc1 back in 2013. By now there should be no cluster
where an older node might still send the old pve2-vm schema.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
Link: https://lore.proxmox.com/20250715143218.1548306-8-a.lauterer@proxmox.com
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the newer pve2.3-vm schema has been introduced with commit ba9dcfc1 back
in 2013. By now there should be no cluster where an older node might
still send the old pve2-vm schema.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
Link: https://lore.proxmox.com/20250715143218.1548306-7-a.lauterer@proxmox.com
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Adds replication settings similar to the migration settings. This allows
specifying different networks and it is ultimately more straightforward
than using the migration settings for replication jobs.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Link: https://lore.proxmox.com/20250612132951.449798-2-m.sandoval@proxmox.com
In a previous commit we already added the openfabric / ospf
configuration files, but the configuration format changed since then,
so we replace them with the single configuration file used by the
fabrics now.
Also add a postinst script that removes the leftover folder from that
change.
Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
Link: https://lore.proxmox.com/20250716130837.585796-34-g.goller@proxmox.com
The size of a symlink should be the length of its target (name, not
content), some tools like bsdtar seem to actually rely on this
information for allocating buffers..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Link: https://lore.proxmox.com/20250606085049.127928-1-f.gruenbichler@proxmox.com
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
in case initializing fails with CS_ERR_LIBRARY, the connection to
corosync failed - add that context to make the error message a bit
less cryptic.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Link: https://lore.proxmox.com/20250605141720.734781-2-f.gruenbichler@proxmox.com
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this has always been a bit annoying to manually map via the header
file, and while the helper does not do much more at the moment than
translate them back to the enum keys, it is better than nothing.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Link: https://lore.proxmox.com/20250605141720.734781-1-f.gruenbichler@proxmox.com
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Improves readability and avoids bugs, as it's really explicit which
statements are guarded by a control statement and what isn't.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
80s is not a nice number in neither binary nor decimal nor time (base
12) number systems, so add a comment for why it was chosen.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
There was a long time mishap, that while stemming from another commit
is quite definitively a mental hiccup presented due to the changes
from commit 05a31bf ("makefile: convert to use simple parenthesis")
being done a bit to eagerly and then committed as two separate
commits.
Fixes: 864b236 ("buildsys: rework doc-gen cleanup and makefile inclusion")
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
through the new format target, throw in the cpg-test file manually
while at it.
One can transform a file using this style also by CLI only, like for
example:
clang-format -i --style="{BasedOnStyle: llvm, IndentWidth: 4,
ColumnLimit: 100, AlignAfterOpenBracket: BlockIndent,
BinPackParameters: false}" FILE1 FILE2 ...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We are in the process of auto-formatting our Perl (using perltidy) and
JavaScript (using biome) code bases, just like we adopted rustfmt
already a few years ago. So lets also format our C code bases while at
it, while they see less activity compared to perl/js/rust at the
moment, there is still some development going on and using
clang-format works well with a rather minimal config, so just go for
it.
Add a make 'format' target with an 'tidy' alias as we use that for the
perl stuff (mostly due to the tool being named perltidy there). Using
format as main target is chosen in a similar spirit, as for clang the
tool is named clang-format and there is a clang-tidy which is a linter
though.
Check in a .clang-format config here, basing of the LLVM style but
with 4 spaces as indentation and up to 100 character columns text
width, and also try to avoid packing arguments to much into less
lines, this hurts readability and line-based editing can be nicer
w.r.t conflicts and blame/history tracking in git anyway.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Using 'g_string_free(outbuf, FALSE)', which is equivalent to the newer
‘g_string_free_and_steal’ function, the management structure from the
GLib string will be freed while the underlying "actual" char * C
string won't, and thus the caller of this needs to take care of that
freeing that string when not needed anymore.
This is used in a few places in pmxcfs by design, as newer compiler
and glib gained intrinsics to help detecting missuse one must now use
the result of that function, which is the underlying char pointer, as
heuristic for the caller to signal that one is aware of that contract.
As we pulled out the pointer earlier directly and then called this
partial free method, all worked fine but the compiler couldn't be sure
about this. Adapt the code accordingly, there should be semantic
change whatsoever, but it gets actually slightly shorter, which is
nice.
While at it also replace some uses of 1 with the TRUE constant, it's
the same but it's more telling for those accustomed to code using the
Glib.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
See pve-common's commit 5ae1f2e ("buildsys: add tidy make target")
for details about the chosen xargs parameters.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
See pve-common's commit 5ae1f2e ("buildsys: add tidy make target")
for details about the chosen xargs parameters.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Add it to both the perl side (PVE/Cluster.pm) and pmxcfs side
(status.c).
This config file will be used to map directory IDs to paths on
selected hosts.
Signed-off-by: Markus Frank <m.frank@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Reviewed-by: Laurențiu Leahu-Vlăducu <l.leahu-vladucu@proxmox.com
Reviewed-by: Daniel Kral <d.kral@proxmox.com>
Tested-by: Laurențiu Leahu-Vlăducu <l.leahu-vladucu@proxmox.com>
Tested-by: Daniel Kral <d.kral@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
[TL: s/dir/directory/ for verbosity]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Add the sdn fabrics config files. These are split into two, as we
currently support two fabric types: ospf and openfabric. They hold the
whole configuration for the respective protocols. They are read and
written by pve-network.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
