Last modification: 03-Apr-2019
Author: R. Koucha
rsys: another way to make system() more efficient
Foreword
This article has been extracted from this larger study on the solutions to optimize the C library's system() service.
1. Introduction
In embedded environments, the cost of the hardware is an important consideration. As a consequence, the memory is often very limited. The memory as well as the CPU time are critical resources which must be used with care and as efficiently as possible not only for response time and robustness purposes but also for hardware cost reduction purposes. Several applications need to call shell commands to trigger various tasks that would be tough to accomplish with languages like C. Hence, to make it, the C library provides the system() service which is passed as parameter the command line to run:
The "command" parameter may be a simple executable name or a more complex shell command line using output redirections and pipes.
system() hides a call to "/bin/sh -c" to run the command line passed as parameter.
From Linux system point of view, in the simplest case, system() triggers at least two pairs of fork()/exec() system calls: one for "sh -c" and another for the command line itself as depicted in Figure 1.
Moreover, fork() triggers a duplication of some resources (memory, file descriptors...) of the calling process (the father) to make the forked process (the child) inherit them. If the calling process is big from a memory occupation point of view or the overall memory occupation is high, the system()call may fail because of a lack of free memory. Even tough Linux benefited multiple enhancements like the Copy On Write (i.e. COW) to make the fork() more efficient and less cumbersome, this may lead to a memory over consumption which triggers Linux defense mechanisms like Out Of Memory (OOM) killer.
In the presentation of isys, we described a solution to optimize the C library's system() service. It is possible to go farther to reduce the number of running background shells by sharing them with all the running applications as proposed in this paper.
2. Rsys
To go farther in the isys implementation, we propose to share running shells with all the application processes. The principle consists to setup a daemon process managing one or more background shells (static configuration or dynamic setting on demand for example). Let's call it rsystemd (i.e. RSYSTEM Daemon) to comply with Unix naming scheme. It is started before any application (at system startup for example) and waits for commands to run on a named socket. It submits the command to one of the shells that it manages and reports the result to the originating application processes. To make it, rsystemd relies on libpdip.so to interact with the shells as explained in isys. On application process side, an API named rsystem() behaves the same as system() but actually it hides the interaction with rsystemd through the named socket: the command line passed as argument is written into the socket to make rsystemd run it and return the displays and the command status. The principle is depicted in Figure 2.
In the source tree of the PDIP package, the rsys sub-directory contains a variant of system() using the above principle (cf. rsystem.c embedded in a shared library called librsys.so which implements rsystem() API and rsystemd.c which implements the daemon part). § 4 presents some details about this library. This proposal not only saves CPU time as we do not continuously fork()/exec() and terminate shell processes but it also saves memory space as the running shells are shared with several processes.
By the way, we must not forget that this solution differs from original system() service from a user interface point of view as the shells are running in separate processes which are not children of the application processes: they are childs of rsystemd. As a consequence, the father to child inheritance mechanism does not operate here (file descriptors, environment variables, signal disposition...). But most of the time it is not required by the users of system().
Another point, if rsystemd is designed with a fixed number of running background shells we may face some starvation problems as the shell command requests may not be satisfied immediately if their number is bigger than the running background shells. So, this introduces some possible latency. Moreover we may also face some deadlocks if there are dependencies between shell commands: a command waits for the setting of some resource by another command which can't get an available background shell. But if rsystemd is designed to launch brand new background shells to satisfy pending command requests when all its configured running background shells are busy, the latter problems won't occur.
3. Performances
In this article a little test program is used to compare the performances of system() and rsystem():
| system() |
$ tests/system_it 2000 tests/scrip.sh Running command 'tests/scrip.sh' 2000 times... Elapsed time: 5 s - 612918826 ns |
| rsystem() |
$ sudo rsys/src/rsystemd & $ tests/rsystem_it 2000 tests/scrip.sh Running command 'tests/scrip.sh' 2000 times... Elapsed time: 4 s - 156234186 ns |
We can see that rsystem() is faster than system(). As a consequence, it is a good alternative to system().
4. Download, build and installation
4.1. Build from the sources
Unpack the source code package:
$ tar xvfz pdip-xxx.tgz
Go into the top level directory of the sources and trigger the build of the DEB packages:
$ cd pdip-xxx
$ ./pdip_install -P DEB
4.2. Installation from the packages
RSYS depends on PDIP. So, PDIP must be installed prior to install RSYS otherwise you get the following error:
$ sudo dpkg -i rsys_xxx_amd64.deb
Selecting previously unselected package rsys.
(Reading database ... 218983 files and directories currently installed.)
Preparing to unpack rsys_xxx_amd64.deb ...
Unpacking rsys (xxx) ...
dpkg: dependency problems prevent configuration of rsys:
rsys depends on pdip (>= xxx); however:
Package pdip is not installed.
dpkg: error processing package rsys (--install):
dependency problems - leaving unconfigured
Errors were encountered while processing:
rsys
Install first the PDIP package:
$ sudo dpkg -i pdip_xxx_amd64.deb
Selecting previously unselected package pdip.
(Reading database ... 218988 files and directories currently installed.)
Preparing to unpack pdip_xxx_amd64.deb ...
Unpacking pdip (xxx) ...
Setting up pdip (xxx) ...
Processing triggers for man-db (2.7.5-1)...
Then install the RSYS package:
$ sudo dpkg -i rsys_xxx_amd64.deb
(Reading database ... 219040 files and directories currently installed.)
Preparing to unpack rsys_xxx_amd64.deb ...
Unpacking rsys (xxx) over (xxx) ...
Setting up rsys (xxx)
Installation from the packages is the preferred way as it is easy to get rid of the software with all the cleanups by calling:
$ sudo dpkg -r rsys
(Reading database ... 219043 files and directories currently installed.)
Removing rsys (xxx)
To display the list of files installed by the package:
$ dpkg -L rsys
/.
/usr
/usr/local
/usr/local/include
/usr/local/include/rsys.h
/usr/local/lib
/usr/local/lib/librsys.so
/usr/local/sbin
/usr/local/sbin/rsystemd
/usr/local/share
/usr/local/share/man
/usr/local/share/man/man3
/usr/local/share/man/man3/rsys.3.gz
/usr/local/share/man/man3/rsystem.3.gz
/usr/local/share/man/man8
/usr/local/share/man/man8/rsystemd.8.gz
4.3. Installation from cmake
It is also possible to trigger the installation from cmake:
$ tar xvfz pdip-xxx.tgz
$ cd pdip-xxx
$ cmake .
-- The C compiler identification is GNU 6.2.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Building PDIP version xxx
The user id is 1000
-- Configuring done
-- Generating done
-- Build files have been written to: ...
$ sudo make install
Scanning dependencies of target man
Building pdip_en.1.gz
Building pdip_fr.1.gz
Building pdip_configure.
-- Installing: /usr/local/lib/librsys.so
-- Installing: /usr/local/sbin/rsystemd
-- Set runtime path of "/usr/local/sbin/rsystemd" to ""
4.4. Manual
When the RSYS package is installed, on line manuals are available in section 3 (API) and 8 (rsystemd daemon).
4.4.1. rsystem()
$ man 3 rsystem
NAMErsys - Remote system() service
SYNOPSIS
#include "rsys.h"
int rsystem(const char *fmt, ...);
int rsys_lib_initialize(void);
The RSYS API provides a system(3)-like service based on shared remanent background shells managed by rsystemd(8) daemon. This saves memory and CPU time in applications where system(3) is heavily used.
rsystem() executes the shell command line formatted with fmt. The behaviour of the format is compliant with printf(3). Internally, the command is run by one of the remanent shells managed by rsystemd(8).
rsys_lib_initialize() is to be called in child processes using the RSYS API. By default, RSYS API is deactivated upon fork(2).
By default, the server socket pathname used for the client/server dialog is /var/run/rsys.socket. The RSYS_SOCKET_PATH environment variable is available to specify an alternate socket pathname if one needs to change it for access rights or any test purposes.
rsystem() returns the status of the executed command line (i.e. the last executed command). The returned value is a "wait status" that can be examined using the macros described in waitpid(2) (i.e. WIFEXITED(), WEXITSTATUS(), and so on).
rsys_lib_initialize() returns 0 when there are no error or -1 upon error (errno is set).
The service does not support concurrent calls to rsystem() by multiple threads. If this behaviour is needed, the application is responsible to manage the mutual exclusion on its side.
EXAMPLE
The following program receives a shell command as argument and executes it via a call to rsystem().
#include <stdio.h>
#include <assert.h>
#include <stdlib.h>
#include <libgen.h>
#include <stdlib.h>
#include <string.h>
#include <rsys.h>
int main(int ac, char *av[])
{
int status;
int i;
char *cmdline;
size_t len;
size_t offset;
if (ac < 2)
{
fprintf(stderr, "Usage: %s cmd params...\n", basename(av[0]));
return 1;
}
// Build the command line
cmdline = 0;
len = 1; // Terminating NUL
offset = 0;
for (i = 1; i < ac; i ++)
{
len += strlen(av[i]) + 1; // word + space
cmdline = (char *)realloc(cmdline, len);
assert(cmdline);
offset += sprintf(cmdline + offset, "%s ", av[i]);
} // End for
printf("Running '%s'...\n", cmdline);
status = rsystem(cmdline);
if (status != 0)
{
fprintf(stderr, "Error from program (0x%x = %d)!\n", status, status);
free(cmdline);
return(1);
} // End if
free(cmdline);
return(0);
} // main
Build the program:
$ gcc trsys.c -o trsys -lrsys -lpdip -lpthread
Make sure that rsystemd(8) is running. Then, run something like the following:
$ ./trsys echo example
Running 'echo example '...
example
Rachid Koucha
rsystemd(8), system(3).
4.4.2. rsystemd
$ man 8 rsystemd
NAMErsystemd - Remote system() daemon
SYNOPSISrsystemd [-s shells] [-V] [-d level] [-D] [-h]
DESCRIPTIONrsystemd is a daemon which manages several childs processes running shells. It is a server for the rsystem(3) service.
OPTIONS
-s | --shells shell_list
Shells to launch along with their CPU affinity. This may be overriden by the RSYSD_SHELLS environment variable. The content is a colon delimited list of affinities for shells to launch. An affinity is defined as follow:
* A comma separated list of fields
* A field is either a CPU number or an interval of consecutive CPU numbers described with the first and last CPU numbers separated by an hyphen.
* An empty field implicitely means all the active CPUs
* A CPU number is from 0 to the number of active CPUs minus 1
* If the first CPU number of an interval is empty, it is considered to be CPU number 0
* If the last CPU number of an interval is empty, it is considered to be the biggest active CPU number
If a CPU number is bigger than the maximum active CPU number, it is implicitely translated into the maximum active CPU number.
If this option is not specified, the default behaviour is one shell running on all available CPUs.
-V | --version
Display the daemon's version
-D | --daemon
Activate the daemon mode (the process detaches itself from the current terminal and becomes a child of init, process number 1).
-d | --debug level
Set the debug level. The higher the value, the more traces are displayed.
-h | --help
Display the help
By default, the server socket pathname used for the client/server dialog is /var/run/rsys.socket. The RSYS_SOCKET_PATH environment variable is available to specify an alternate socket pathname if one needs to change it for access rights or any test purposes. It is advised to specify an absolute pathname especially in daemon mode where the server changes its current directory to the root of the filesystem. Consequently, any relative pathname will be considered from the server's current directory.
EXAMPLES
The following launches a shell running on CPU number 3 and CPU numbers 6 to 8. We use "sudo" as rsystemd creates a named socket in /var/run.
$ sudo rsystemd -s 3,6-8
The following launches three shells. The first runs on CPU numbers 0 to 3, CPU number 5 and CPU number 6. The second runs on CPU number 0 and CPU numbers 3 to the latest active CPU. The third runs on all the active CPUs.
$ sudo rsystemd -s -3,5,6:0,3-:
The following launches one shell through the RSYSD_SHELLS environment variable. We pass -E option to "sudo" to preserve the environment otherwise RSYSD_SHELLS would not be taken in account. The environment variable overrides the parameter passed to rsystemd. The affinity of the shell are CPU number 1 and 3.
$ export RSYSD_SHELLS=1,3
$ sudo -E rsystemd -s -3,5,6:0,3-:
Rachid Koucha
SEE ALSOrsystem(3).
4.4.3. FSM of rsystemd
The finished state machine describing the main engine of rsystemd is depicted in Figure 3.
4.5. Build facilities
To help people to auto-detect the location of RSYS stuff (libraries, executables, include files), the RSYS package installs a configuration file named rsys.pc to make it available for pkg-config tool.
Moreover, for cmake based packages, a FindRsys.cmake file is provided at the top level of rsys sub-tree to facilitate auto-configuration.
5. About the author
The author is an engineer in computer sciences located in France. He
can be contacted here.